Julia vs Stan performance comparison
Caveats
This page compares the performance of Julia’s and Stan’s log density and log density gradient computations for the implemented posteriors. Several caveats apply:
- The
posteriordbStan implementations were never meant to represent “perfect and best-performant” practices. - The StanBlocks.jl implementations are not line-by-line translations of the Stan implementations. Sometimes small optimizations were applied, to make the implementation fall more in line with common Julia practices, or to make the code more friendly for Julia’s AD packages, e.g. by avoiding mutation.
- Stan often automatically drops constant terms (unless configured differently), see https://mc-stan.org/docs/reference-manual/statements.html#log-probability-increment-vs.-distribution-statement, thus avoiding superfluous (for its purposes) computation, while the StanBlocks.jl implementations do not.
- Stan implements a lot of custom AD rules, while StanBlocks.jl does not at all, and Enzyme.jl does rarely (if ever?). I suspect that adding AD rules for
_glm_type functions would further improve Julia’s performance. - The StanBlocks.jl “sampling” statements try to be clever about avoiding repeated computations. While I am not sure whether Stan applies the same optimizations, in principle it could do that without extra work by the user.
- While preliminary benchmark runs included “all” Julia AD packages, all of them are almost always much slower than Enzyme.jl for the implemented posteriors, which on top of that performance advantage also supports more Julia language features than some of the other AD packages. As such, I am only comparing Enzyme and Stan. Enzyme outperforming every other AD package for these posteriors/loss functions does of course not mean that it will necessarily do as well for other applications.
- Enzyme’s development is progressing quite quickly. While it currently sometimes crashes Julia, or it sometimes errors while trying to compute a gradient, in general Enzyme’s performance and reliability are continuously and quickly improving.
- Stan’s benchmark is done from Julia via
BridgeStan.jl. While I think that any performance penalty should be extremely small, I am not 100% sure. BridgeStan uses the-O3compiler flag by default, but no additional ones. - All benchmarks are happening with a single thread on my local machine.
- There are probably more caveats!
Warning
In general, doing performance comparisons is quite tricky, for more reasons than just the ones mentioned above. The below plot and tables should most definitely NOT be interpreted as “A is X-times faster than B”.
Visualization
Warning
In general, doing performance comparisons is quite tricky, for more reasons than just the ones mentioned above. The below plot and tables should most definitely NOT be interpreted as “A is X-times faster than B”.
The below plot shows the relative primitive runtime (x-axis, Julia vs Stan, left: Julia is faster) and the relative gradient runtime (y-axis, Julia+X vs Stan, bottom: Julia is faster) for the posteriordb models for which the overview table has a value less than 1e-8 in the remaining relative lpdf error column. The color of the points represents the Julia AD framework used, which currently includes Enzyme.jl and Mooncake.jl. Hovering over the data points will show the posterior name, its dimension, the allocations required by Julia during the primitive and gradient run and a short explanation, e.g. mesquite-logmesquite_logvash (D=7, #allocs=0->70) - Julia's primitive is ~4.5 times faster, but Enzyme's gradient is ~16.0 times slower. Horizontal and vertical lines around scatter points represent the 90% central credible interval of the ratio of the runtime means, estimated using 200 iterations of Bayesian Bootstrap. As those intervals are sometimes quite wide, I’m planning to rerun some or all of the benchmarks.
Tabular data
The below table shows information about the implemented posteriors. Will elaborate on the exact meaning of columns.
| posterior name | dimension | remaining relative lpdf error | relative mean primitive Julia runtime | relative mean primitive Stan runtime | relative mean Enzyme runtime | relative mean Mooncake runtime | relative mean Stan gradient runtime | mean primitive Julia allocations | mean Enzyme allocations | mean Mooncake allocations | constant lpdf difference | Enzyme relative gradient error | Mooncake relative gradient error | Enzyme | Mooncake | implementations |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GLMM_Poisson_data-GLMM_Poisson_model | 45 | 2.3e-16 ± 1.9e-17 | 1.0 ± 0.0035 | 3.4 ± 0.012 | 1.0 ± 0.0029 | 3600.0 ± 3400.0 | 2.0 ± 0.0072 | 0.0 | 0.0 | 2293.0 | 1.5e-8 ± 1.2e-9 | 4.5e-16 ± 2.0e-17 | 4.7e-16 ± 2.2e-17 | 0.13.21 | 0.4.60 | |
| GLMM_data-GLMM1_model | 237 | 1.5e-16 ± 1.0e-17 | 1.0 ± 0.003 | 3.2 ± 0.016 | 1.0 ± 0.0029 | 4.6 ± 0.033 | 2.1 ± 0.012 | 0.0 | 0.0 | 11.0 | 2.3 ± 1.8e-14 | 1.2e-17 ± 4.2e-19 | 1.2e-17 ± 4.3e-19 | 0.13.21 | 0.4.60 | |
| GLM_Binomial_data-GLM_Binomial_model | 3 | 2.0e-16 ± 7.0e-18 | 1.0 ± 0.0019 | 1.6 ± 0.0046 | 1.7 ± 0.0067 | 150.0 ± 0.96 | 1.0 ± 0.0025 | 0.0 | 0.0 | 2282.0 | 14.0 ± 3.2e-14 | 3.0e-16 ± 8.7e-18 | 3.3e-16 ± 9.1e-18 | 0.13.21 | 0.4.60 | |
| GLM_Poisson_Data-GLM_Poisson_model | 4 | 3.8e-16 ± 2.7e-17 | 1.0 ± 0.0022 | 2.8 ± 0.0092 | 1.0 ± 0.0035 | 250.0 ± 3.0 | 1.6 ± 0.005 | 0.0 | 0.0 | 2282.0 | 1.6e-9 ± 1.1e-10 | 4.9e-16 ± 2.3e-17 | 4.9e-16 ± 2.3e-17 | 0.13.21 | 0.4.60 | |
| M0_data-M0_model | 2 | 1.6e-16 ± 1.2e-17 | 1.0 ± 0.0015 | 1.2 ± 0.0038 | 1.3 ± 0.0026 | 15.0 ± 0.15 | 1.0 ± 0.003 | 0.0 | 0.0 | 2001.0 | 2.8e-14 ± 2.0e-15 | 3.5e-15 ± 1.3e-16 | 1.0e-15 ± 4.7e-17 | 0.13.21 | 0.4.60 | |
| Mb_data-Mb_model | 3 | 1.2e-16 ± 8.7e-18 | 1.0 ± 0.0035 | 4.1 ± 0.078 | 3.6 ± 0.16 | 81.0 ± 2.5 | 1.0 ± 0.016 | 3.0 | 32.0 | 50285.0 | 1.1e-13 ± 8.0e-15 | 5.1e-15 ± 2.1e-16 | 5.3e-15 ± 2.5e-16 | 0.13.21 | 0.4.60 | |
| Mh_data-Mh_model | 388 | 4.1e-16 ± 1.9e-17 | 1.0 ± 0.0014 | 1.8 ± 0.0048 | 1.0 ± 0.0023 | 1.7 ± 0.0078 | 1.0 ± 0.0033 | 1.0 | 2.0 | 13.0 | 160.0 ± 2.4e-14 | 1.4e-15 ± 4.6e-17 | 6.7e-16 ± 1.9e-17 | 0.13.21 | 0.4.60 | |
| Mt_data-Mt_model | 4 | 1.5e-16 ± 1.1e-17 | 1.0 ± 0.0024 | 2.1 ± 0.0086 | 13.0 ± 7.4 | 110.0 ± 3.3 | 1.0 ± 0.0031 | 1.0 | 26.0 | 24284.0 | 5.7e-14 ± 4.0e-15 | 2.5e-15 ± 8.4e-17 | 2.6e-16 ± 8.3e-18 | 0.13.21 | 0.4.60 | |
| Mtbh_data-Mtbh_model | 154 | 1.7e-16 ± 1.1e-17 | 1.9 ± 1.5 | 1.0 ± 0.0063 | 2.4 ± 0.04 | 57.0 ± 0.55 | 1.0 ± 0.0034 | 6.0 | 38.0 | 23292.0 | 2.3 ± 4.0e-15 | 6.5e-16 ± 1.9e-17 | 3.5e-16 ± 8.5e-18 | 0.13.21 | 0.4.60 | |
| Mth_data-Mth_model | 394 | 1.9e-16 ± 6.3e-18 | 1.0 ± 0.0038 | 2.0 ± 0.022 | 2.9 ± 0.11 | 87.0 ± 3.2 | 1.0 ± 0.0083 | 3.0 | 30.0 | 61329.0 | 2.3e-13 ± 8.0e-15 | 1.6e-15 ± 5.9e-17 | 7.3e-16 ± 2.3e-17 | 0.13.21 | 0.4.60 | |
| Rate_1_data-Rate_1_model | 1 | 0.0 ± 0.0 | 1.0 ± 0.0014 | 2.6 ± 0.0045 | 1.0 ± 0.00067 | 44.0 ± 0.2 | 2.6 ± 0.0026 | 0.0 | 0.0 | 23.0 | 5.5 ± 6.3e-17 | 1.7e-16 ± 1.2e-17 | 2.5e-16 ± 1.4e-17 | 0.13.21 | 0.4.60 | |
| Rate_2_data-Rate_2_model | 2 | 0.0 ± 0.0 | 1.0 ± 0.0023 | 7.7 ± 5.1 | 1.0 ± 0.0042 | 31.0 ± 0.13 | 1.8 ± 0.0085 | 0.0 | 0.0 | 46.0 | 10.0 ± 1.3e-16 | 2.1e-16 ± 1.1e-17 | 2.7e-16 ± 1.3e-17 | 0.13.21 | 0.4.60 | |
| Rate_3_data-Rate_3_model | 1 | 0.0 ± 0.0 | 1.0 ± 0.0016 | 2.0 ± 0.0062 | 1.0 ± 0.0036 | 50.0 ± 0.21 | 2.3 ± 0.0093 | 0.0 | 0.0 | 46.0 | 10.0 ± 1.3e-16 | 2.2e-16 ± 1.6e-17 | 2.2e-16 ± 1.6e-17 | 0.13.21 | 0.4.60 | |
| Rate_4_data-Rate_4_model | 2 | 0.0 ± 0.0 | 1.0 ± 0.002 | 2.9 ± 0.0058 | 1.0 ± 0.0019 | 31.0 ± 0.075 | 3.1 ± 0.013 | 0.0 | 0.0 | 23.0 | 2.7 ± 6.3e-17 | 1.6e-16 ± 1.0e-17 | 1.8e-16 ± 1.0e-17 | 0.13.21 | 0.4.60 | |
| Rate_5_data-Rate_5_model | 1 | 0.0 ± 0.0 | 1.0 ± 0.0015 | 2.2 ± 0.0052 | 1.0 ± 0.0017 | 86.0 ± 0.51 | 2.6 ± 0.0064 | 0.0 | 0.0 | 46.0 | 0.0 ± 0.0 | 1.7e-16 ± 1.2e-17 | 3.0e-16 ± 2.1e-17 | 0.13.21 | 0.4.60 | |
| Survey_data-Survey_model | 1 | 5.4e-16 ± 2.6e-17 | 1.0 ± 0.0036 | 1.1 ± 0.007 | 4.0 ± 0.21 | 83.0 ± 2.4 | 1.0 ± 0.0078 | 1.0 | 40.0 | 87220.0 | 2.1e-14 ± 5.0e-16 | 5.5e-14 ± 3.6e-15 | 5.4e-14 ± 3.5e-15 | 0.13.21 | 0.4.60 | |
| arK-arK | 7 | 2.0e-16 ± 1.1e-17 | 1.0 ± 0.0019 | 6.1 ± 0.018 | 1.0 ± 0.0018 | 6.3 ± 0.017 | 6.7 ± 0.036 | 0.0 | 0.0 | 21.0 | 15.0 ± 4.1e-15 | 1.0e-16 ± 5.0e-18 | 1.4e-15 ± 6.5e-17 | 0.13.21 | 0.4.60 | |
| arma-arma11 | 4 | 2.2e-16 ± 1.6e-17 | 1.0 ± 0.0018 | 4.1 ± 0.013 | 1.0 ± 0.0018 | 7.5 ± 0.019 | 7.2 ± 0.06 | 0.0 | 0.0 | 12.0 | 4.6 ± 2.3e-14 | 1.2e-16 ± 8.1e-18 | 3.5e-16 ± 2.5e-17 | 0.13.21 | 0.4.60 | |
| bball_drive_event_0-hmm_drive_0 | 6 | 1.1e-16 ± 8.1e-18 | 1.0 ± 0.002 | 2.8 ± 0.017 | 4.4 ± 0.07 | 1.3 ± 0.0046 | 1.0 ± 0.0061 | 9.0 | 40.0 | 184.0 | 2.3 ± 6.4e-14 | 3.0e-12 ± 1.7e-13 | 3.0e-12 ± 1.7e-13 | 0.13.21 | 0.4.60 | |
| bball_drive_event_1-hmm_drive_1 | 6 | 8.1e-16 ± 2.9e-17 | 1.0 ± 0.0045 | 3.4 ± 0.061 | 3.0 ± 0.098 | 1.0 ± 0.022 | 1.2 ± 0.074 | 11.0 | 44.0 | 196.0 | 760.0 ± 9.8e-12 | 4.9e-12 ± 3.5e-13 | 4.9e-12 ± 3.5e-13 | 0.13.21 | 0.4.60 | |
| bones_data-bones_model | 13 | 1.3e-16 ± 9.5e-18 | 1.0 ± 0.01 | 3.8 ± 0.14 | 2.2 ± 0.083 | 3.2 ± 1.6 | 1.0 ± 0.011 | 3.0 | 6.0 | 19.0 | 47.0 ± 1.6e-14 | 1.4e-16 ± 2.4e-18 | 1.3e-16 ± 2.8e-18 | 0.13.21 | 0.4.60 | |
| butterfly-multi_occupancy | 106 | 2.1e-15 ± 8.2e-17 | 1.0 ± 0.0057 | 1.4 ± 0.0073 | missing | missing | missing | 1388.0 | missing | missing | 2400.0 ± 6.4e-13 | FAILED | 1.7e-16 ± 6.9e-18 | missing | missing | |
| diamonds-diamonds | 26 | 1.3e-15 ± 6.6e-17 | 1.0 ± 0.34 | 1.2 ± 0.007 | 2.1 ± 0.0059 | 540.0 ± 5.3 | 1.0 ± 0.0036 | 2.0 | 4.0 | 140045.0 | 4600.0 ± 2.2e-11 | 2.0e-15 ± 6.6e-17 | 2.0e-15 ± 6.6e-17 | 0.13.21 | 0.4.60 | |
| dogs-dogs | 3 | 3.9e-16 ± 2.8e-17 | 1.0 ± 0.0025 | 4.7 ± 0.016 | 1.0 ± 0.0027 | 1.7 ± 0.0042 | 3.8 ± 0.031 | 0.0 | 0.0 | 6.0 | 14.0 ± 3.2e-14 | 1.1e-15 ± 5.4e-17 | 1.6e-16 ± 1.1e-17 | 0.13.21 | 0.4.60 | |
| dogs-dogs_hierarchical | 2 | 1.0e-15 ± 4.2e-17 | 1.0 ± 0.002 | 1.8 ± 0.0078 | 2.2 ± 0.017 | 69.0 ± 2.0 | 1.0 ± 0.0042 | 0.0 | 0.0 | 21018.0 | 9.1e-13 ± 4.0e-14 | 1.2e-15 ± 4.1e-17 | 1.2e-15 ± 3.9e-17 | 0.13.21 | 0.4.60 | |
| dogs-dogs_log | 2 | 5.3e-16 ± 7.2e-17 | 1.0 ± 0.001 | 2.6 ± 0.0058 | 1.0 ± 0.0037 | 1.0 ± 0.0035 | 1.5 ± 0.007 | 0.0 | 0.0 | 0.0 | 9.2 ± 5.6e-14 | 1.3e-15 ± 9.6e-17 | 1.3e-15 ± 9.7e-17 | 0.13.21 | 0.4.60 | |
| dogs-dogs_nonhierarchical | 65 | 0.08 ± 0.0043 | missing | missing | missing | missing | missing | missing | missing | missing | 98.0 ± 5.5 | missing | missing | missing | missing | |
| dugongs_data-dugongs_model | 4 | 1.5e-16 ± 1.0e-17 | 1.0 ± 0.0012 | 1.8 ± 0.0021 | 1.0 ± 0.0031 | 1.9 ± 0.0016 | 1.0 ± 0.0031 | 0.0 | 0.0 | 0.0 | 22.0 ± 1.0e-15 | 1.4e-16 ± 8.0e-18 | 2.8e-16 ± 9.8e-18 | 0.13.21 | 0.4.60 | |
| earnings-earn_height | 3 | 8.7e-16 ± 3.7e-17 | 1.0 ± 0.0028 | 8.4 ± 0.032 | 1.0 ± 0.0022 | 1700.0 ± 50.0 | 5.1 ± 0.018 | 0.0 | 0.0 | 33396.0 | 0.00035 ± 2.3e-5 | 3.7e-16 ± 1.5e-17 | 7.9e-16 ± 3.2e-17 | 0.13.21 | 0.4.60 | |
| earnings-log10earn_height | 3 | 8.8e-16 ± 3.2e-17 | 1.0 ± 0.0031 | 8.6 ± 0.032 | 1.0 ± 0.003 | 1100.0 ± 40.0 | 3.5 ± 0.017 | 0.0 | 0.0 | 33396.0 | 9.3e-10 ± 6.5e-11 | 9.9e-16 ± 3.1e-17 | 9.9e-16 ± 3.1e-17 | 0.13.21 | 0.4.60 | |
| earnings-logearn_height | 3 | 8.3e-16 ± 3.8e-17 | 1.0 ± 0.0027 | 8.6 ± 0.031 | 1.0 ± 0.0024 | 1100.0 ± 25.0 | 3.3 ± 0.012 | 0.0 | 0.0 | 33396.0 | 6.7e-10 ± 4.7e-11 | 1.0e-15 ± 3.2e-17 | 1.0e-15 ± 3.2e-17 | 0.13.21 | 0.4.60 | |
| earnings-logearn_height_male | 4 | 8.8e-16 ± 3.3e-17 | 1.0 ± 0.0023 | 2.8 ± 0.011 | 1.0 ± 0.0033 | 920.0 ± 32.0 | 4.3 ± 0.016 | 0.0 | 0.0 | 33396.0 | 9.9e-10 ± 7.0e-11 | 1.0e-15 ± 3.1e-17 | 1.0e-15 ± 3.1e-17 | 0.13.21 | 0.4.60 | |
| earnings-logearn_interaction | 5 | 8.0e-16 ± 3.3e-17 | 1.0 ± 0.0024 | 3.6 ± 0.022 | 1.0 ± 0.0042 | 710.0 ± 22.0 | 5.8 ± 0.048 | 0.0 | 0.0 | 33396.0 | 2.2e-9 ± 1.5e-10 | 9.7e-16 ± 2.4e-17 | 9.6e-16 ± 2.3e-17 | 0.13.21 | 0.4.60 | |
| earnings-logearn_interaction_z | 5 | 6.9e-16 ± 2.7e-17 | 1.0 ± 0.0019 | 3.2 ± 0.014 | 1.0 ± 0.003 | 770.0 ± 23.0 | 5.5 ± 0.024 | 0.0 | 0.0 | 33396.0 | 4.4e-11 ± 3.0e-12 | 8.2e-16 ± 3.3e-17 | 8.2e-16 ± 3.2e-17 | 0.13.21 | 0.4.60 | |
| earnings-logearn_logheight_male | 4 | 8.2e-16 ± 3.2e-17 | 1.0 ± 0.0021 | 2.6 ± 0.008 | 1.0 ± 0.0023 | 910.0 ± 31.0 | 4.4 ± 0.014 | 0.0 | 0.0 | 33396.0 | 3.6e-11 ± 2.5e-12 | 9.5e-16 ± 2.9e-17 | 9.5e-16 ± 3.0e-17 | 0.13.21 | 0.4.60 | |
| ecdc0401-covid19imperial_v2 | 51 | 1.5e-16 ± 6.7e-18 | 1.0 ± 0.014 | 3.9 ± 0.2 | missing | missing | missing | 80.0 | missing | missing | 18.0 ± 1.5e-12 | FAILED | 2.7e-16 ± 5.4e-18 | missing | missing | |
| ecdc0401-covid19imperial_v3 | 51 | 1.4e-16 ± 6.3e-18 | 1.0 ± 0.0084 | 2.9 ± 0.17 | missing | missing | missing | 80.0 | missing | missing | missing | FAILED | 2.9e-16 ± 6.6e-18 | missing | missing | |
| ecdc0501-covid19imperial_v2 | 51 | 1.3e-16 ± 9.0e-18 | 1.0 ± 0.0095 | 2.8 ± 0.17 | missing | missing | missing | 80.0 | missing | missing | 18.0 ± 8.2e-12 | FAILED | FAILED | missing | missing | |
| ecdc0501-covid19imperial_v3 | 51 | 1.4e-16 ± 9.1e-18 | missing | missing | missing | missing | missing | missing | missing | missing | 18.0 ± 8.2e-12 | FAILED | 3.4e-16 ± 5.8e-18 | missing | missing | |
| eight_schools-eight_schools_centered | 10 | 2.2e-16 ± 9.5e-18 | 1.0 ± 0.0014 | 2.4 ± 0.013 | 1.0 ± 0.0019 | 130.0 ± 0.84 | 2.0 ± 0.0046 | 0.0 | 0.0 | 276.0 | 23.0 ± 2.5e-16 | 1.6e-16 ± 8.6e-18 | 1.6e-16 ± 8.6e-18 | 0.13.21 | 0.4.60 | |
| eight_schools-eight_schools_noncentered | 10 | 3.3e-16 ± 1.3e-17 | 1.0 ± 0.0014 | 3.0 ± 0.0053 | 1.0 ± 0.0021 | 130.0 ± 0.49 | 2.5 ± 0.0047 | 0.0 | 0.0 | 274.0 | 23.0 ± 2.5e-16 | 3.7e-17 ± 1.6e-18 | 3.3e-17 ± 1.5e-18 | 0.13.21 | 0.4.60 | |
| election88-election88_full | 90 | 0.0 ± 0.0 | 1.0 ± 0.0088 | 2.6 ± 0.17 | 1.0 ± 0.03 | 41.0 ± 0.0078 | 2.0 ± 0.16 | 10.0 | 10.0 | 323955.0 | 23.0 ± 0.0 | 1.0e-16 ± 1.8e-18 | 1.4e-16 ± 2.8e-18 | 0.13.21 | 0.4.60 | |
| fims_Aus_Jpn_irt-2pl_latent_reg_irt | 531 | 0.0022 ± 8.2e-5 | missing | missing | missing | missing | missing | missing | missing | missing | 44.0 ± 1.4 | missing | missing | missing | missing | |
| garch-garch11 | 4 | 2.3e-16 ± 1.2e-17 | 1.0 ± 0.0028 | 3.3 ± 0.013 | 1.0 ± 0.0028 | 2.2 ± 0.0052 | 2.6 ± 0.0088 | 0.0 | 0.0 | 0.0 | 1.1e-13 ± 8.0e-15 | 4.3e-16 ± 1.2e-17 | 4.5e-16 ± 1.3e-17 | 0.13.21 | 0.4.60 | |
| gp_pois_regr-gp_pois_regr | 13 | 2.2e-16 ± 1.6e-17 | 1.0 ± 0.27 | 1.0 ± 0.0038 | 2.0 ± 0.0062 | 17.0 ± 0.023 | 1.0 ± 0.0023 | 6.0 | 10.0 | 410.0 | 21.0 ± 1.4e-15 | 2.3e-16 ± 1.1e-17 | 3.3e-16 ± 1.6e-17 | 0.13.21 | 0.4.60 | |
| gp_pois_regr-gp_regr | 3 | 1.5e-16 ± 1.1e-17 | 1.0 ± 0.38 | 1.0 ± 0.0033 | 2.1 ± 0.015 | 7.1 ± 2.4 | 1.0 ± 0.0019 | 9.0 | 15.0 | 57.0 | 31.0 ± 5.0e-16 | 1.7e-16 ± 5.0e-18 | 2.0e-16 ± 6.1e-18 | 0.13.21 | 0.4.60 | |
| hmm_example-hmm_example | 4 | 2.3e-16 ± 1.2e-17 | 1.0 ± 0.0019 | 4.9 ± 0.058 | 2.4 ± 0.018 | 1.2 ± 0.0047 | 1.0 ± 0.0051 | 7.0 | 23.0 | 28.0 | 94.0 ± 3.2e-14 | 2.0e-13 ± 9.4e-15 | 2.0e-13 ± 9.4e-15 | 0.13.21 | 0.4.60 | |
| hmm_gaussian_simulated-hmm_gaussian | 14 | 6.7e-16 ± 2.8e-17 | 1.0 ± 0.0041 | 2.9 ± 0.063 | 2.5 ± 0.22 | 1.1 ± 0.025 | 1.0 ± 0.052 | 8.0 | 17.0 | 151.0 | 460.0 ± 1.1e-12 | 2.5e-12 ± 1.7e-13 | 2.5e-12 ± 1.7e-13 | 0.13.21 | 0.4.60 | |
| hudson_lynx_hare-lotka_volterra | 8 | missing | missing | missing | missing | missing | missing | missing | missing | missing | missing | missing | missing | missing | missing | |
| iohmm_reg_simulated-iohmm_reg | 29 | 5.6e-16 ± 2.6e-17 | 1.0 ± 0.0059 | 5.3 ± 0.19 | missing | missing | missing | 512.0 | missing | missing | 420.0 ± 1.6e-13 | 1.1e-12 ± 4.9e-14 | FAILED | missing | missing | |
| irt_2pl-irt_2pl | 144 | 2.3e-16 ± 1.6e-17 | 1.0 ± 0.0029 | 1.4 ± 0.0077 | 1.8 ± 0.012 | 170.0 ± 4.9 | 1.0 ± 0.0036 | 21.0 | 34.0 | 57093.0 | 15.0 ± 3.2e-14 | 4.6e-16 ± 1.9e-17 | 4.7e-16 ± 1.9e-17 | 0.13.21 | 0.4.60 | |
| kidiq-kidscore_interaction | 5 | 4.2e-16 ± 1.9e-17 | 1.0 ± 0.0016 | 3.4 ± 0.0067 | 1.0 ± 0.0023 | 650.0 ± 13.0 | 5.2 ± 0.015 | 0.0 | 0.0 | 12184.0 | 0.92 ± 8.1e-11 | 5.6e-16 ± 1.3e-17 | 5.6e-16 ± 1.3e-17 | 0.13.21 | 0.4.60 | |
| kidiq-kidscore_momhs | 3 | 6.7e-16 ± 2.7e-17 | 1.0 ± 0.0019 | 8.4 ± 0.028 | 1.0 ± 0.0018 | 1400.0 ± 28.0 | 4.7 ± 0.014 | 0.0 | 0.0 | 12184.0 | 0.92 ± 8.2e-11 | 2.2e-16 ± 1.3e-17 | 4.9e-16 ± 2.2e-17 | 0.13.21 | 0.4.60 | |
| kidiq-kidscore_momhsiq | 4 | 4.6e-16 ± 1.8e-17 | 1.0 ± 0.0023 | 3.0 ± 0.0083 | 1.0 ± 0.0059 | 540.0 ± 9.9 | 3.3 ± 0.0096 | 0.0 | 0.0 | 12184.0 | 0.92 ± 3.2e-11 | 5.4e-16 ± 1.6e-17 | 5.5e-16 ± 1.6e-17 | 0.13.21 | 0.4.60 | |
| kidiq-kidscore_momiq | 3 | 4.1e-16 ± 1.9e-17 | 1.0 ± 0.002 | 10.0 ± 0.027 | 1.0 ± 0.0049 | 780.0 ± 20.0 | 2.6 ± 0.0074 | 0.0 | 0.0 | 12184.0 | 0.92 ± 4.2e-11 | 5.5e-16 ± 1.3e-17 | 5.5e-16 ± 1.3e-17 | 0.13.21 | 0.4.60 | |
| kidiq_with_mom_work-kidscore_interaction_c | 5 | 4.4e-16 ± 1.8e-17 | 1.0 ± 0.0019 | 3.4 ± 0.0058 | 1.0 ± 0.0028 | 690.0 ± 15.0 | 5.4 ± 0.013 | 0.0 | 0.0 | 12172.0 | 7.0e-10 ± 4.6e-11 | 4.1e-16 ± 1.5e-17 | 4.2e-16 ± 1.5e-17 | 0.13.21 | 0.4.60 | |
| kidiq_with_mom_work-kidscore_interaction_c2 | 5 | 4.7e-16 ± 2.0e-17 | 1.0 ± 0.0018 | 3.4 ± 0.01 | 1.0 ± 0.0062 | 570.0 ± 17.0 | 4.0 ± 0.012 | 0.0 | 0.0 | 12172.0 | 4.7e-10 ± 3.3e-11 | 4.0e-16 ± 1.7e-17 | 4.0e-16 ± 1.7e-17 | 0.13.21 | 0.4.60 | |
| kidiq_with_mom_work-kidscore_interaction_z | 5 | 5.4e-16 ± 1.9e-17 | 1.0 ± 0.0019 | 3.3 ± 0.007 | 1.0 ± 0.019 | 590.0 ± 13.0 | 4.7 ± 0.016 | 0.0 | 0.0 | 12172.0 | 7.0e-10 ± 4.9e-11 | 3.6e-16 ± 1.5e-17 | 3.6e-16 ± 1.5e-17 | 0.13.21 | 0.4.60 | |
| kidiq_with_mom_work-kidscore_mom_work | 5 | 6.0e-16 ± 2.6e-17 | 1.0 ± 0.0025 | 3.5 ± 0.011 | 1.0 ± 0.0038 | 730.0 ± 21.0 | 5.6 ± 0.023 | 0.0 | 0.0 | 12172.0 | 9.3e-10 ± 6.4e-11 | 4.8e-16 ± 1.9e-17 | 4.8e-16 ± 1.9e-17 | 0.13.21 | 0.4.60 | |
| kilpisjarvi_mod-kilpisjarvi | 3 | 1.8e-16 ± 9.9e-18 | 1.0 ± 0.0031 | 7.6 ± 0.028 | 1.0 ± 0.0027 | 850.0 ± 13.0 | 3.5 ± 0.017 | 0.0 | 0.0 | 1756.0 | 1.2 ± 1.3e-9 | 2.4e-16 ± 7.3e-18 | 2.4e-16 ± 7.3e-18 | 0.13.21 | 0.4.60 | |
| loss_curves-losscurve_sislob | 15 | 2.0e-16 ± 1.2e-17 | 1.0 ± 0.0021 | 1.8 ± 0.0059 | 13.0 ± 5.6 | 58.0 ± 0.43 | 1.0 ± 0.0034 | 3.0 | 48.0 | 1603.0 | 16.0 ± 8.0e-15 | 1.8e-16 ± 5.1e-18 | 1.8e-16 ± 5.2e-18 | 0.13.21 | 0.4.60 | |
| low_dim_gauss_mix-low_dim_gauss_mix | 5 | 6.6e-16 ± 2.5e-17 | 1.0 ± 0.0023 | 2.2 ± 0.005 | 1.0 ± 0.0091 | 2.0 ± 0.02 | 1.2 ± 0.0045 | 4.0 | 9.0 | 36.0 | 920.0 ± 2.3e-13 | 3.5e-16 ± 1.3e-17 | 2.6e-16 ± 8.9e-18 | 0.13.21 | 0.4.60 | |
| low_dim_gauss_mix_collapse-low_dim_gauss_mix_collapse | 5 | 6.0e-16 ± 2.5e-17 | 1.0 ± 0.0014 | 2.1 ± 0.0041 | 1.0 ± 0.0044 | 2.5 ± 0.0042 | 2.0 ± 0.026 | 1.0 | 2.0 | 19.0 | 920.0 ± 8.8e-14 | 4.6e-16 ± 1.6e-17 | 3.4e-16 ± 1.0e-17 | 0.13.21 | 0.4.60 | |
| lsat_data-lsat_model | 1006 | 1.3e-16 ± 9.0e-18 | 1.0 ± 0.31 | 1.1 ± 0.0046 | 1.3 ± 0.0069 | 130.0 ± 0.51 | 1.0 ± 0.0041 | 15.0 | 74.0 | 140188.0 | 28.0 ± 3.2e-14 | 1.1e-15 ± 3.5e-17 | 1.4e-15 ± 3.7e-17 | 0.13.21 | 0.4.60 | |
| mcycle_gp-accel_gp | 66 | 3.4e-16 ± 1.5e-17 | 1.0 ± 0.0039 | 2.4 ± 0.0044 | 1.0 ± 0.0039 | 42.0 ± 0.5 | 1.1 ± 0.0025 | 12.0 | 24.0 | 4561.0 | 180.0 ± 1.6e-12 | 3.7e-16 ± 1.2e-17 | 3.6e-16 ± 1.1e-17 | 0.13.21 | 0.4.60 | |
| mcycle_splines-accel_splines | 82 | 7.4e-16 ± 4.1e-17 | 1.0 ± 0.0048 | 2.0 ± 0.0047 | 3.0 ± 2.0 | 39.0 ± 0.55 | 1.0 ± 0.0024 | 11.0 | 22.0 | 3851.0 | 190.0 ± 3.7e-9 | 5.4e-16 ± 2.2e-17 | 6.0e-16 ± 2.5e-17 | 0.13.21 | 0.4.60 | |
| mesquite-logmesquite | 8 | 1.4e-16 ± 9.7e-18 | 1.0 ± 0.0025 | 4.6 ± 0.015 | 9.7 ± 0.045 | 61.0 ± 0.75 | 1.0 ± 0.0058 | 0.0 | 84.0 | 1308.0 | 5.7e-14 ± 4.0e-15 | 2.0e-16 ± 8.2e-18 | 2.1e-16 ± 8.7e-18 | 0.13.21 | 0.4.60 | |
| mesquite-logmesquite_logva | 5 | 1.6e-16 ± 1.1e-17 | 1.0 ± 0.0031 | 3.5 ± 0.011 | 1.0 ± 0.0021 | 8600.0 ± 7900.0 | 5.0 ± 0.016 | 0.0 | 0.0 | 1308.0 | 1.1e-13 ± 8.0e-15 | 2.3e-16 ± 7.9e-18 | 2.4e-16 ± 8.2e-18 | 0.13.21 | 0.4.60 | |
| mesquite-logmesquite_logvas | 8 | 1.6e-16 ± 8.5e-18 | 1.0 ± 0.0052 | 5.0 ± 0.026 | 11.0 ± 0.044 | 75.0 ± 1.3 | 1.0 ± 0.0032 | 0.0 | 84.0 | 1308.0 | 5.7e-14 ± 4.0e-15 | 2.0e-16 ± 7.0e-18 | 2.0e-16 ± 6.8e-18 | 0.13.21 | 0.4.60 | |
| mesquite-logmesquite_logvash | 7 | 1.5e-16 ± 6.5e-18 | 1.0 ± 0.0026 | 4.4 ± 0.016 | 10.0 ± 0.056 | 70.0 ± 0.86 | 1.0 ± 0.005 | 0.0 | 70.0 | 1308.0 | 5.7e-14 ± 4.0e-15 | 2.3e-16 ± 8.2e-18 | 2.3e-16 ± 8.3e-18 | 0.13.21 | 0.4.60 | |
| mesquite-logmesquite_logvolume | 3 | 1.6e-16 ± 3.7e-18 | 1.0 ± 0.0026 | 9.6 ± 0.031 | 1.0 ± 0.0025 | 920.0 ± 13.0 | 3.3 ± 0.0083 | 0.0 | 0.0 | 1308.0 | 5.7e-14 ± 4.0e-15 | 1.9e-16 ± 8.4e-18 | 1.9e-16 ± 8.1e-18 | 0.13.21 | 0.4.60 | |
| mesquite-mesquite | 8 | 1.8e-16 ± 7.2e-18 | 1.0 ± 0.0045 | 5.0 ± 0.019 | 8.9 ± 0.041 | 62.0 ± 0.81 | 1.0 ± 0.0044 | 0.0 | 70.0 | 1308.0 | 9.3e-10 ± 6.6e-11 | 1.9e-16 ± 1.3e-17 | 1.9e-16 ± 1.3e-17 | 0.13.21 | 0.4.60 | |
| mnist-nn_rbm1bJ100 | missing | missing | missing | missing | missing | missing | missing | missing | missing | missing | missing | missing | missing | missing | missing | |
| mnist_100-nn_rbm1bJ10 | missing | missing | missing | missing | missing | missing | missing | missing | missing | missing | missing | missing | missing | missing | missing | |
| nes1972-nes | 10 | 7.4e-16 ± 3.4e-17 | 1.0 ± 0.002 | 4.4 ± 0.086 | 5.2 ± 0.081 | 53.0 ± 1.9 | 1.0 ± 0.021 | 0.0 | 88.0 | 37260.0 | 1.4e-11 ± 9.6e-13 | 1.1e-15 ± 2.2e-17 | 1.1e-15 ± 2.2e-17 | 0.13.21 | 0.4.60 | |
| nes1976-nes | 10 | 5.6e-16 ± 2.7e-17 | 1.0 ± 0.0025 | 5.1 ± 0.11 | 46.0 ± 41.0 | 52.0 ± 1.7 | 1.0 ± 0.026 | 0.0 | 88.0 | 33172.0 | 1.1e-11 ± 7.6e-13 | 1.1e-15 ± 2.0e-17 | 1.1e-15 ± 2.0e-17 | 0.13.21 | 0.4.60 | |
| nes1980-nes | 10 | 4.1e-16 ± 2.0e-17 | 1.0 ± 0.002 | 4.2 ± 0.044 | 5.7 ± 0.067 | 59.0 ± 2.2 | 1.0 ± 0.0081 | 0.0 | 88.0 | 19648.0 | 5.5e-12 ± 3.8e-13 | 8.3e-16 ± 1.9e-17 | 8.2e-16 ± 1.8e-17 | 0.13.21 | 0.4.60 | |
| nes1984-nes | 10 | 5.8e-16 ± 2.9e-17 | 1.0 ± 0.0027 | 4.4 ± 0.099 | 5.8 ± 0.096 | 54.0 ± 2.5 | 1.0 ± 0.024 | 0.0 | 88.0 | 34348.0 | 2.1e-11 ± 1.4e-12 | 1.1e-15 ± 2.2e-17 | 1.1e-15 ± 2.1e-17 | 0.13.21 | 0.4.60 | |
| nes1988-nes | 10 | 5.9e-16 ± 2.6e-17 | 1.0 ± 0.0022 | 4.3 ± 0.077 | 5.9 ± 0.082 | 57.0 ± 1.9 | 1.0 ± 0.015 | 0.0 | 88.0 | 31184.0 | 1.1e-11 ± 7.4e-13 | 9.9e-16 ± 2.2e-17 | 9.6e-16 ± 2.2e-17 | 0.13.21 | 0.4.60 | |
| nes1992-nes | 10 | 6.2e-16 ± 3.0e-17 | 1.0 ± 0.0022 | 4.5 ± 0.095 | 5.2 ± 0.077 | 54.0 ± 1.5 | 1.0 ± 0.02 | 0.0 | 88.0 | 37820.0 | 1.5e-11 ± 9.6e-13 | 1.2e-15 ± 2.4e-17 | 1.2e-15 ± 2.5e-17 | 0.13.21 | 0.4.60 | |
| nes1996-nes | 10 | 5.7e-16 ± 2.8e-17 | 1.0 ± 0.0024 | 4.3 ± 0.047 | 6.4 ± 0.18 | 51.0 ± 1.2 | 1.0 ± 0.014 | 0.0 | 88.0 | 29224.0 | 1.2e-11 ± 8.1e-13 | 1.0e-15 ± 2.0e-17 | 1.0e-15 ± 2.0e-17 | 0.13.21 | 0.4.60 | |
| nes2000-nes | 10 | 4.0e-16 ± 1.6e-17 | 1.0 ± 0.0043 | 3.9 ± 0.022 | 5.7 ± 0.078 | 56.0 ± 1.4 | 1.0 ± 0.0045 | 0.0 | 56.0 | 13348.0 | 3.6e-12 ± 2.6e-13 | 6.9e-16 ± 1.5e-17 | 6.8e-16 ± 1.6e-17 | 0.13.21 | 0.4.60 | |
| nes_logit_data-nes_logit_model | 2 | 0.0 ± 0.0 | 1.0 ± 0.0032 | 1.0 ± 0.0034 | 2.9 ± 0.011 | 340.0 ± 9.5 | 1.0 ± 0.0031 | 1.0 | 2.0 | 33036.0 | 0.0 ± 0.0 | 6.1e-15 ± 2.3e-16 | 6.1e-15 ± 2.3e-16 | 0.13.21 | 0.4.60 | |
| normal_2-normal_mixture | 3 | 9.1e-16 ± 4.1e-17 | 1.0 ± 0.0015 | 2.7 ± 0.0067 | 1.0 ± 0.0032 | 2.6 ± 0.032 | 2.3 ± 0.021 | 0.0 | 0.0 | 0.0 | 910.0 ± 3.1e-12 | 1.3e-16 ± 3.8e-18 | 1.2e-16 ± 5.3e-18 | 0.13.21 | 0.4.60 | |
| normal_5-normal_mixture_k | 14 | 8.1e-16 ± 3.6e-17 | 1.0 ± 0.0033 | 3.6 ± 0.18 | 5.2 ± 0.53 | 46.0 ± 2.6 | 1.0 ± 0.028 | 2.0 | 10.0 | 270638.0 | 1600.0 ± 3.2e-13 | 7.6e-16 ± 2.4e-17 | 8.2e-16 ± 2.5e-17 | 0.13.21 | 0.4.60 | |
| one_comp_mm_elim_abs-one_comp_mm_elim_abs | 4 | 1.6e-7 ± 1.1e-8 | 9.7 ± 0.75 | 1.0 ± 0.037 | missing | missing | missing | 1.28436e5 | missing | missing | 29.0 ± 2.3e-7 | FAILED | FAILED | missing | missing | |
| ovarian-logistic_regression_rhs | 3075 | 1.5e-13 ± 4.5e-15 | 1.0 ± 0.01 | 1.5 ± 0.034 | 1.8 ± 0.038 | 34.0 ± 2.7 | 1.0 ± 0.039 | 3.0 | 23.0 | 99928.0 | 1700.0 ± 4.0e-11 | 6.7e-11 ± 3.5e-12 | 6.7e-11 ± 3.5e-12 | 0.13.21 | 0.4.60 | |
| pilots-pilots | 18 | 1.3e-16 ± 8.9e-18 | 1.0 ± 0.0054 | 1.9 ± 0.0053 | 1.0 ± 0.005 | 150.0 ± 1.5 | 1.0 ± 0.0027 | 2.0 | 4.0 | 1170.0 | 0.0 ± 0.0 | 1.7e-16 ± 8.0e-18 | 1.7e-16 ± 7.0e-18 | 0.13.21 | 0.4.60 | |
| prideprejudice_chapter-ldaK5 | 7714 | 4.2e-15 ± 2.1e-16 | 1.0 ± 0.025 | 1.7 ± 0.18 | 2.0 ± 0.22 | 42.0 ± NaN | 1.0 ± 0.045 | 3.0 | 24.0 | 5.44648e6 | 1.3e-9 ± 6.0e-11 | 5.2e-16 ± 3.0e-18 | 5.3e-16 ± 3.4e-18 | 0.13.21 | 0.4.60 | |
| prideprejudice_paragraph-ldaK5 | 15570 | 2.9e-15 ± 1.2e-16 | 1.0 ± 0.031 | 1.7 ± 0.16 | 1.6 ± 0.016 | 48.0 ± NaN | 1.0 ± 0.028 | 4.0 | 32.0 | 6.05833e6 | 9.3e-10 ± 3.7e-11 | 3.0e-16 ± 1.6e-18 | 3.0e-16 ± 1.6e-18 | 0.13.21 | 0.4.60 | |
| prostate-logistic_regression_rhs | 11935 | 3.6e-15 ± 2.2e-16 | 1.0 ± 0.064 | 1.4 ± 0.2 | 1.8 ± 0.18 | 15.0 ± 1.0 | 1.0 ± 0.12 | 5.0 | 27.0 | 384796.0 | 6800.0 ± 4.4e-12 | 7.0e-15 ± 4.4e-16 | 6.6e-15 ± 4.1e-16 | 0.13.21 | 0.4.60 | |
| radon_all-radon_county_intercept | 388 | 4.1e-15 ± 2.1e-16 | 1.0 ± 0.0018 | 4.6 ± 0.076 | 1.0 ± 0.0048 | 14.0 ± 0.19 | 39.0 ± 3.5 | 0.0 | 0.0 | 9.0 | 11000.0 ± 1.5e-11 | 5.6e-15 ± 3.0e-16 | 1.2e-13 ± 4.6e-15 | 0.13.21 | 0.4.60 | |
| radon_all-radon_hierarchical_intercept_centered | 391 | 3.9e-15 ± 1.7e-16 | 1.0 ± 0.0039 | 4.7 ± 0.055 | 1.0 ± 0.012 | 17.0 ± 0.7 | 15.0 ± 1.1 | 0.0 | 0.0 | 20.0 | 12000.0 ± 1.3e-11 | 7.1e-15 ± 3.2e-16 | 1.0e-13 ± 4.4e-15 | 0.13.21 | 0.4.60 | |
| radon_all-radon_hierarchical_intercept_noncentered | 391 | 4.0e-15 ± 1.9e-16 | 1.0 ± 0.0035 | 4.7 ± 0.084 | 1.0 ± 0.008 | 11.0 ± 0.27 | 9.3 ± 0.19 | 1.0 | 2.0 | 22.0 | 12000.0 ± 1.7e-11 | 6.2e-15 ± 2.8e-16 | 9.7e-14 ± 3.4e-15 | 0.13.21 | 0.4.60 | |
| radon_all-radon_partially_pooled_centered | 389 | 3.9e-15 ± 1.8e-16 | 1.0 ± 0.0079 | 9.8 ± 0.55 | 1.0 ± 0.0028 | 14.0 ± 0.12 | 12.0 ± 0.48 | 0.0 | 0.0 | 11.0 | 12000.0 ± 1.2e-11 | 8.1e-15 ± 3.9e-16 | 1.3e-13 ± 6.0e-15 | 0.13.21 | 0.4.60 | |
| radon_all-radon_partially_pooled_noncentered | 389 | 3.6e-15 ± 1.5e-16 | 1.0 ± 0.0067 | 3.6 ± 0.05 | 1.0 ± 0.0038 | 10.0 ± 0.13 | 8.7 ± 0.11 | 1.0 | 2.0 | 13.0 | 12000.0 ± 1.7e-11 | 6.9e-15 ± 3.1e-16 | 1.3e-13 ± 4.6e-15 | 0.13.21 | 0.4.60 | |
| radon_all-radon_pooled | 3 | 1.3e-14 ± 5.7e-16 | 1.0 ± 0.0012 | 26.0 ± 0.28 | 1.0 ± 0.0018 | 820.0 ± 4.6 | 20.0 ± 1.4 | 0.0 | 0.0 | 352064.0 | 12000.0 ± 3.9e-11 | 1.6e-14 ± 6.8e-16 | 1.6e-14 ± 6.9e-16 | 0.13.21 | 0.4.60 | |
| radon_all-radon_variable_intercept_centered | 390 | 4.4e-15 ± 1.9e-16 | 1.0 ± 0.0031 | 5.1 ± 0.17 | 1.0 ± 0.008 | 23.0 ± 0.89 | 62.0 ± 4.5 | 0.0 | 0.0 | 11.0 | 12000.0 ± 1.3e-11 | 6.9e-15 ± 2.9e-16 | 8.9e-14 ± 3.9e-15 | 0.13.21 | 0.4.60 | |
| radon_all-radon_variable_intercept_noncentered | 390 | 3.9e-15 ± 1.7e-16 | 1.0 ± 0.0022 | 4.5 ± 0.063 | 1.0 ± 0.0028 | 11.0 ± 0.14 | 12.0 ± 0.11 | 1.0 | 2.0 | 13.0 | 12000.0 ± 1.8e-11 | 5.4e-15 ± 2.3e-16 | 1.1e-13 ± 4.4e-15 | 0.13.21 | 0.4.60 | |
| radon_all-radon_variable_intercept_slope_centered | 777 | 3.7e-15 ± 1.6e-16 | 1.0 ± 0.0016 | 5.4 ± 0.16 | 1.0 ± 0.0082 | 11.0 ± 0.12 | 8.7 ± 0.25 | 0.0 | 0.0 | 22.0 | 12000.0 ± 1.8e-11 | 6.3e-15 ± 3.0e-16 | 1.1e-13 ± 4.6e-15 | 0.13.21 | 0.4.60 | |
| radon_all-radon_variable_intercept_slope_noncentered | 777 | 4.8e-15 ± 1.9e-16 | 1.0 ± 0.0017 | 4.4 ± 0.047 | 1.0 ± 0.069 | 13.0 ± 0.75 | 15.0 ± 1.3 | 2.0 | 4.0 | 26.0 | 12000.0 ± 2.8e-11 | 5.2e-15 ± 2.1e-16 | 7.0e-14 ± 3.2e-15 | 0.13.21 | 0.4.60 | |
| radon_all-radon_variable_slope_centered | 390 | 1.5e-14 ± 6.3e-16 | 1.0 ± 0.0027 | 5.1 ± 0.16 | 1.0 ± 0.0067 | 11.0 ± 0.092 | 12.0 ± 0.74 | 0.0 | 0.0 | 11.0 | 12000.0 ± 6.1e-11 | 9.3e-15 ± 4.3e-16 | 9.5e-14 ± 4.5e-15 | 0.13.21 | 0.4.60 | |
| radon_all-radon_variable_slope_noncentered | 390 | 1.7e-14 ± 7.4e-16 | 1.0 ± 0.0014 | 4.6 ± 0.051 | 1.0 ± 0.0032 | 12.0 ± 0.37 | 11.0 ± 0.22 | 1.0 | 2.0 | 13.0 | 12000.0 ± 1.0e-10 | 8.3e-15 ± 3.4e-16 | 7.1e-14 ± 3.2e-15 | 0.13.21 | 0.4.60 | |
| radon_mn-radon_county_intercept | 87 | 6.9e-16 ± 3.2e-17 | 1.0 ± 0.00092 | 4.5 ± 0.016 | 1.0 ± 0.0012 | 16.0 ± 0.057 | 14.0 ± 0.056 | 0.0 | 0.0 | 9.0 | 650.0 ± 1.8e-13 | 1.1e-15 ± 5.1e-17 | 9.8e-15 ± 3.3e-16 | 0.13.21 | 0.4.60 | |
| radon_mn-radon_hierarchical_intercept_centered | 90 | 7.9e-16 ± 3.0e-17 | 1.0 ± 0.0017 | 5.1 ± 0.02 | 1.0 ± 0.0015 | 16.0 ± 0.053 | 14.0 ± 0.096 | 0.0 | 0.0 | 20.0 | 840.0 ± 2.3e-13 | 9.0e-16 ± 4.0e-17 | 7.4e-15 ± 3.3e-16 | 0.13.21 | 0.4.60 | |
| radon_mn-radon_hierarchical_intercept_noncentered | 90 | 7.5e-16 ± 2.9e-17 | 1.0 ± 0.0022 | 4.1 ± 0.018 | 1.0 ± 0.0028 | 12.0 ± 0.028 | 9.2 ± 0.015 | 1.0 | 2.0 | 22.0 | 840.0 ± 1.9e-13 | 9.0e-16 ± 3.6e-17 | 8.7e-15 ± 2.9e-16 | 0.13.21 | 0.4.60 | |
| radon_mn-radon_partially_pooled_centered | 88 | 8.4e-16 ± 3.4e-17 | 1.0 ± 0.001 | 3.5 ± 0.0032 | 1.0 ± 0.00096 | 14.0 ± 0.026 | 8.9 ± 0.027 | 0.0 | 0.0 | 11.0 | 840.0 ± 1.8e-13 | 9.4e-16 ± 3.9e-17 | 8.6e-15 ± 3.2e-16 | 0.13.21 | 0.4.60 | |
| radon_mn-radon_partially_pooled_noncentered | 88 | 6.4e-16 ± 3.2e-17 | 1.0 ± 0.0016 | 3.9 ± 0.0096 | 1.0 ± 0.0054 | 6.5 ± 0.016 | 5.9 ± 0.031 | 1.0 | 2.0 | 13.0 | 840.0 ± 2.3e-13 | 8.5e-16 ± 3.4e-17 | 8.2e-15 ± 3.1e-16 | 0.13.21 | 0.4.60 | |
| radon_mn-radon_pooled | 3 | 1.1e-15 ± 5.4e-17 | 1.0 ± 0.0025 | 28.0 ± 0.12 | 1.0 ± 0.0025 | 740.0 ± 14.0 | 9.4 ± 0.04 | 0.0 | 0.0 | 25752.0 | 840.0 ± 2.3e-13 | 1.4e-15 ± 4.6e-17 | 1.4e-15 ± 4.7e-17 | 0.13.21 | 0.4.60 | |
| radon_mn-radon_variable_intercept_centered | 89 | 7.6e-16 ± 3.3e-17 | 1.0 ± 0.0034 | 4.2 ± 0.0091 | 1.0 ± 0.0042 | 13.0 ± 0.043 | 13.0 ± 0.12 | 0.0 | 0.0 | 11.0 | 840.0 ± 1.9e-13 | 9.5e-16 ± 4.6e-17 | 9.9e-15 ± 3.6e-16 | 0.13.21 | 0.4.60 | |
| radon_mn-radon_variable_intercept_noncentered | 89 | 8.0e-16 ± 3.4e-17 | 1.0 ± 0.0018 | 5.1 ± 0.016 | 1.0 ± 0.004 | 9.8 ± 0.041 | 9.8 ± 0.038 | 1.0 | 2.0 | 13.0 | 840.0 ± 2.0e-13 | 7.6e-16 ± 3.1e-17 | 9.1e-15 ± 2.8e-16 | 0.13.21 | 0.4.60 | |
| radon_mn-radon_variable_intercept_slope_centered | 175 | 8.1e-16 ± 3.3e-17 | 1.0 ± 0.0011 | 4.0 ± 0.0053 | 1.0 ± 0.0031 | 15.0 ± 0.23 | 8.6 ± 0.04 | 0.0 | 0.0 | 22.0 | 840.0 ± 2.7e-13 | 7.3e-16 ± 3.6e-17 | 6.8e-15 ± 3.3e-16 | 0.13.21 | 0.4.60 | |
| radon_mn-radon_variable_intercept_slope_noncentered | 175 | 9.2e-16 ± 3.3e-17 | 1.0 ± 0.002 | 4.2 ± 0.0061 | 1.0 ± 0.0037 | 7.9 ± 0.014 | 8.0 ± 0.029 | 2.0 | 4.0 | 26.0 | 840.0 ± 2.5e-13 | 7.3e-16 ± 3.0e-17 | 7.6e-15 ± 2.6e-16 | 0.13.21 | 0.4.60 | |
| radon_mn-radon_variable_slope_centered | 89 | 1.2e-15 ± 5.2e-17 | 1.0 ± 0.0013 | 4.0 ± 0.0065 | 1.0 ± 0.0015 | 13.0 ± 0.015 | 10.0 ± 0.012 | 0.0 | 0.0 | 11.0 | 840.0 ± 2.6e-13 | 1.0e-15 ± 4.2e-17 | 6.7e-15 ± 2.9e-16 | 0.13.21 | 0.4.60 | |
| radon_mn-radon_variable_slope_noncentered | 89 | 9.0e-16 ± 3.8e-17 | 1.0 ± 0.00083 | 4.8 ± 0.012 | 1.0 ± 0.0037 | 11.0 ± 0.043 | 8.9 ± 0.035 | 1.0 | 2.0 | 13.0 | 840.0 ± 3.1e-13 | 1.3e-15 ± 5.2e-17 | 8.4e-15 ± 3.2e-16 | 0.13.21 | 0.4.60 | |
| radon_mod-radon_county | 389 | 0.0 ± 0.0 | 1.6 ± 1.2 | 1.0 ± 0.0095 | 1.0 ± 0.012 | 480.0 ± 11.0 | 1.3 ± 0.018 | 2.0 | 2.0 | 352080.0 | 0.0 ± 0.0 | 7.3e-16 ± 3.3e-17 | 7.1e-16 ± 3.4e-17 | 0.13.21 | 0.4.60 | |
| rats_data-rats_model | 65 | 1.3e-16 ± 9.4e-18 | 1.0 ± 0.0032 | 4.0 ± 0.01 | 1.0 ± 0.0026 | 9.7 ± 0.0074 | 7.0 ± 0.026 | 0.0 | 0.0 | 22.0 | 9.2 ± 4.9e-11 | 3.7e-16 ± 1.6e-17 | 2.9e-15 ± 8.2e-17 | 0.13.21 | 0.4.60 | |
| rstan_downloads-prophet | 62 | 1.0 ± 3.2e-15 | missing | missing | missing | missing | missing | missing | missing | missing | 5.4e16 ± 3.8e15 | missing | missing | missing | missing | |
| sat-hier_2pl | 669 | 0.0013 ± 4.8e-5 | missing | missing | missing | missing | missing | missing | missing | missing | 61.0 ± 2.1 | missing | missing | missing | missing | |
| sblrc-blr | 6 | 2.3e-16 ± 9.2e-18 | 1.0 ± 0.36 | 2.3 ± 0.0092 | 1.0 ± 0.0082 | 210.0 ± 2.6 | 1.0 ± 0.0041 | 1.0 | 2.0 | 2842.0 | 97.0 ± 9.9e-11 | 2.4e-16 ± 7.7e-18 | 2.1e-16 ± 8.1e-18 | 0.13.21 | 0.4.60 | |
| sblri-blr | 6 | 2.7e-16 ± 8.6e-18 | 1.0 ± 0.37 | 1.7 ± 0.0029 | 1.0 ± 0.0045 | 260.0 ± 0.54 | 1.3 ± 0.0024 | 1.0 | 2.0 | 2842.0 | 97.0 ± 4.9e-11 | 2.1e-16 ± 9.2e-18 | 2.1e-16 ± 9.4e-18 | 0.13.21 | 0.4.60 | |
| science_irt-grsm_latent_reg_irt | 408 | 1.5e-16 ± 5.5e-18 | 1.0 ± 0.025 | 4.0 ± 0.36 | 2.3 ± 0.23 | 4.3 ± 0.36 | 1.0 ± 0.059 | 10980.0 | 30202.0 | 60700.0 | 15.0 ± 6.4e-14 | 1.5e-16 ± 3.3e-18 | 1.4e-16 ± 3.5e-18 | 0.13.21 | 0.4.60 | |
| seeds_data-seeds_centered_model | 26 | 1.5e-16 ± 7.3e-18 | 1.0 ± 0.00073 | 2.0 ± 0.0017 | 1.0 ± 0.0012 | 81.0 ± 0.11 | 1.1 ± 0.0011 | 1.0 | 2.0 | 1243.0 | 1.1e-13 ± 8.0e-15 | 2.3e-16 ± 6.1e-18 | 2.2e-16 ± 6.0e-18 | 0.13.21 | 0.4.60 | |
| seeds_data-seeds_model | 26 | 1.4e-16 ± 6.9e-18 | 1.0 ± 0.0017 | 1.6 ± 0.004 | 1.0 ± 0.00087 | 66.0 ± 0.13 | 1.0 ± 0.002 | 0.0 | 0.0 | 1227.0 | 35.0 ± 8.0e-15 | 2.6e-16 ± 7.4e-18 | 2.2e-16 ± 6.1e-18 | 0.13.21 | 0.4.60 | |
| seeds_data-seeds_stanified_model | 26 | 1.4e-16 ± 8.8e-18 | 1.0 ± 0.0018 | 1.7 ± 0.0015 | 1.2 ± 0.012 | 88.0 ± 0.11 | 1.0 ± 0.001 | 0.0 | 0.0 | 1239.0 | 1.1e-13 ± 8.0e-15 | 2.3e-16 ± 5.1e-18 | 2.4e-16 ± 6.0e-18 | 0.13.21 | 0.4.60 | |
| sesame_data-sesame_one_pred_a | 3 | 7.4e-16 ± 5.2e-17 | 1.0 ± 0.0048 | 8.1 ± 0.039 | 1.0 ± 0.0022 | 940.0 ± 18.0 | 4.6 ± 0.017 | 0.0 | 0.0 | 6740.0 | 1.7e-13 ± 1.2e-14 | 1.1e-15 ± 3.8e-17 | 1.3e-15 ± 5.3e-17 | 0.13.21 | 0.4.60 | |
| sir-sir | 4 | 0.00019 ± 4.2e-5 | missing | missing | missing | missing | missing | missing | missing | missing | 110.0 ± 2.5 | missing | missing | missing | missing | |
| soil_carbon-soil_incubation | 6 | 3.3e-6 ± 2.1e-7 | 8.7 ± 1.5 | 1.0 ± 0.056 | missing | missing | missing | 38157.1 | missing | missing | 0.0002 ± 1.4e-5 | FAILED | FAILED | missing | missing | |
| state_wide_presidential_votes-hierarchical_gp | 933 | 0.19 ± 0.012 | missing | missing | missing | missing | missing | missing | missing | missing | 2000.0 ± 87.0 | missing | missing | missing | missing | |
| surgical_data-surgical_model | 14 | 1.5e-16 ± 7.4e-18 | 1.0 ± 0.0008 | 2.2 ± 0.0038 | 1.0 ± 0.0014 | 85.0 ± 0.52 | 1.0 ± 0.002 | 0.0 | 0.0 | 723.0 | 14.0 ± 3.2e-14 | 1.4e-16 ± 2.5e-18 | 1.4e-16 ± 2.9e-18 | 0.13.21 | 0.4.60 | |
| synthetic_grid_RBF_kernels-kronecker_gp | missing | missing | missing | missing | missing | missing | missing | missing | missing | missing | missing | missing | missing | missing | missing | |
| three_docs1200-ldaK2 | 7 | 3.4e-16 ± 1.4e-17 | 1.0 ± 0.0027 | 1.6 ± 0.023 | 5.3 ± 0.4 | 50.0 ± 1.7 | 1.0 ± 0.016 | 2.0 | 34.0 | 90434.0 | 6.8e-13 ± 3.2e-14 | 2.5e-16 ± 7.5e-18 | 2.5e-16 ± 7.4e-18 | 0.13.21 | 0.4.60 | |
| three_men1-ldaK2 | 502 | 2.0e-15 ± 7.7e-17 | 1.0 ± 0.018 | 1.8 ± 0.14 | 3.2 ± 0.32 | 37.0 ± 0.85 | 1.0 ± 0.073 | 2.0 | 34.0 | 389357.0 | 7.3e-11 ± 3.1e-12 | 5.3e-16 ± 9.9e-18 | 5.7e-16 ± 1.2e-17 | 0.13.21 | 0.4.60 | |
| three_men2-ldaK2 | 510 | 1.8e-15 ± 7.1e-17 | 1.0 ± 0.0071 | 1.8 ± 0.082 | 3.8 ± 0.4 | 70.0 ± 34.0 | 1.0 ± 0.067 | 2.0 | 34.0 | 389949.0 | 5.1e-11 ± 2.2e-12 | 4.1e-16 ± 4.8e-18 | 4.1e-16 ± 4.1e-18 | 0.13.21 | 0.4.60 | |
| three_men3-ldaK2 | 505 | 1.5e-15 ± 7.8e-17 | 1.0 ± 0.015 | 1.8 ± 0.11 | 3.5 ± 0.36 | 37.0 ± 1.1 | 1.0 ± 0.063 | 2.0 | 34.0 | 389579.0 | 6.7e-11 ± 3.0e-12 | 5.0e-16 ± 7.2e-18 | 5.0e-16 ± 7.5e-18 | 0.13.21 | 0.4.60 | |
| timssAusTwn_irt-gpcm_latent_reg_irt | 530 | 0.0012 ± 5.4e-5 | missing | missing | missing | missing | missing | missing | missing | missing | 28.0 ± 0.72 | missing | missing | missing | missing | |
| traffic_accident_nyc-bym2_offset_only | 3845 | 9.1e-16 ± 3.6e-17 | 1.0 ± 0.0069 | 2.7 ± 0.0073 | 1.0 ± 0.14 | 140.0 ± 5.4 | 1.2 ± 0.0034 | 6.0 | 8.0 | 107654.0 | 0.65 ± 1.1e-9 | 1.5e-15 ± 4.5e-17 | 1.4e-15 ± 4.7e-17 | 0.13.21 | 0.4.60 | |
| uk_drivers-state_space_stochastic_level_stochastic_seasonal | 389 | 2.6e-16 ± 1.0e-17 | 1.0 ± 0.003 | 3.0 ± 0.0049 | missing | missing | missing | 3.0 | missing | missing | 2.9 ± 4.8e-14 | FAILED | 2.1e-16 ± 8.9e-18 | missing | missing | |
| wells_data-wells_daae_c_model | 6 | 1.4e-16 ± 9.8e-18 | 1.1 ± 0.005 | 1.0 ± 0.005 | 2.8 ± 0.017 | 260.0 ± 5.6 | 1.0 ± 0.004 | 2.0 | 4.0 | 84594.0 | 4.5e-13 ± 3.2e-14 | 9.3e-16 ± 2.3e-17 | 9.3e-16 ± 2.3e-17 | 0.13.21 | 0.4.60 | |
| wells_data-wells_dae_c_model | 5 | 1.4e-16 ± 1.0e-17 | 1.1 ± 0.0062 | 1.0 ± 0.0085 | 3.0 ± 0.018 | 260.0 ± 11.0 | 1.0 ± 0.0039 | 2.0 | 4.0 | 84594.0 | 4.5e-13 ± 3.2e-14 | 9.3e-16 ± 2.3e-17 | 9.4e-16 ± 2.2e-17 | 0.13.21 | 0.4.60 | |
| wells_data-wells_dae_inter_model | 7 | 1.6e-16 ± 9.2e-18 | 1.1 ± 0.0047 | 1.0 ± 0.0059 | 2.8 ± 0.035 | 240.0 ± 14.0 | 1.0 ± 0.0051 | 2.0 | 4.0 | 84594.0 | 4.5e-13 ± 3.2e-14 | 1.1e-15 ± 2.2e-17 | 1.1e-15 ± 2.3e-17 | 0.13.21 | 0.4.60 | |
| wells_data-wells_dae_model | 4 | 0.0 ± 0.0 | 1.1 ± 0.0056 | 1.0 ± 0.0049 | 2.8 ± 0.016 | 260.0 ± 6.0 | 1.0 ± 0.0038 | 2.0 | 4.0 | 84594.0 | 0.0 ± 0.0 | 7.3e-16 ± 1.7e-17 | 7.2e-16 ± 1.6e-17 | 0.13.21 | 0.4.60 | |
| wells_data-wells_dist | 2 | 2.9e-13 ± 1.6e-14 | 1.0 ± 0.0067 | 1.9 ± 0.0095 | 1.4 ± 0.0055 | 300.0 ± 180.0 | 1.0 ± 0.0087 | 0.0 | 0.0 | 84578.0 | 4.2e-8 ± 1.3e-9 | 3.9e-11 ± 1.9e-12 | 3.9e-11 ± 1.9e-12 | 0.13.21 | 0.4.60 | |
| wells_data-wells_dist100_model | 2 | 0.0 ± 0.0 | 1.1 ± 0.0039 | 1.0 ± 0.0036 | 3.0 ± 0.012 | 250.0 ± 8.5 | 1.0 ± 0.0035 | 2.0 | 4.0 | 84594.0 | 0.0 ± 0.0 | 9.6e-16 ± 3.6e-17 | 1.0e-15 ± 3.6e-17 | 0.13.21 | 0.4.60 | |
| wells_data-wells_dist100ars_model | 3 | 0.0 ± 0.0 | 1.0 ± 0.0038 | 1.0 ± 0.0041 | 2.8 ± 0.012 | 270.0 ± 13.0 | 1.0 ± 0.004 | 2.0 | 4.0 | 84594.0 | 0.0 ± 0.0 | 7.4e-16 ± 2.2e-17 | 7.5e-16 ± 2.2e-17 | 0.13.21 | 0.4.60 | |
| wells_data-wells_interaction_c_model | 4 | 1.3e-16 ± 9.2e-18 | 1.0 ± 0.0048 | 1.0 ± 0.0051 | 2.5 ± 0.015 | 200.0 ± 6.4 | 1.0 ± 0.0085 | 2.0 | 4.0 | 84594.0 | 4.5e-13 ± 3.2e-14 | 1.0e-15 ± 3.1e-17 | 1.0e-15 ± 2.9e-17 | 0.13.21 | 0.4.60 | |
| wells_data-wells_interaction_model | 4 | 0.0 ± 0.0 | 1.0 ± 0.0039 | 1.0 ± 0.004 | 2.5 ± 0.013 | 480.0 ± 220.0 | 1.0 ± 0.0045 | 2.0 | 4.0 | 84594.0 | 0.0 ± 0.0 | 8.8e-16 ± 2.3e-17 | 8.8e-16 ± 2.5e-17 | 0.13.21 | 0.4.60 |