Skip to content

Send the parallel CV folds only the data that they need - #663

Merged
avehtari merged 10 commits into
masterfrom
loo-fold-args
Nov 19, 2026
Merged

avehtari merged 10 commits into
masterfrom
loo-fold-args

Conversation

@avehtari

Copy link
Copy Markdown
Member

Summary

The parallel LOO and K-fold CV of cv_varsel() sent each worker data that the folds do not need. Parallel K-fold CV now takes 15.6 % to 36.1 % less time, and parallel LOO CV 6.5 % to 22.5 % less, in the cases below. The rankings and elpd values do not change.

Changes

  • LOO CV with validated search: one_obs() read lw[, i] and search_out_rks[[run_index]] from its enclosing environment. Thus each future got the whole S x n matrix lw and all rankings. Each fold now gets its column and its ranking as an element of .X, as in one_fold() of the K-fold CV.
  • Without offsets, mu_offs is the object mu, and with the identity link, mu is the object eta. A serialized reference model holds a copy of each. refmodel_for_folds() now removes mu_offs and eta if they are identical to mu (refmodel_drop_dup_draws()), and one_obs() and one_fold() restore them on the worker (refmodel_restore_draws()). For K-fold CV, this also applies to the reference model of each fold.

Gaussian reference model, S = 4000 draws, 50 predictors, forward search with validate_search = TRUE, 20 clusters, nterms_max = 5, 8 workers; full LOO CV and K-fold CV with K = 10. CSC Roihu (R 4.6.1, future 1.82.0, future.mirai 1.1.0), jobs 3208224 and 3208318, medians of 3 runs. Master and this PR (2a272a6; later commits change only tests and comments) ran one after the other on one node:

CV n MB per future, master this PR multisession (s), master this PR mirai_multisession (s), master this PR
LOO 300 46.2 19.9 2.77 2.59 2.98 2.31
LOO 1000 153.3 65.5 16.56 13.74 14.27 12.98
LOO 2000 306.3 130.8 51.69 47.92 51.23 46.77
K-fold 300 79.7 28.4 2.54 1.70 1.99 1.68
K-fold 1000 264.3 93.4 5.14 3.37 3.57 2.80
K-fold 2000 527.9 186.1 7.96 5.34 5.46 4.27
K-fold 10000 2637.2 928.2 20.38 13.03 18.54 12.62

Limitations

  • The LOO workers still get at least two S x n matrices: mu of the reference model and the log-likelihood matrix of the weight perturbation. The folds need both.
  • With offsets or with a link other than the identity, refmodel_for_folds() removes only the matrices that are identical to mu.

Tests

  • test_parallel.R: The LOO fold function has neither lw nor search_out_rks in its environment. No reference model for the workers holds mu_offs or eta as a copy of mu (all reference models of the test setup, also for the latent and the augmented-data projection). The restore gives back mu, eta and mu_offs. K-fold CV through the parallel code path, but with the folds in this R session, gives the results of the sequential K-fold CV. The first two tests fail without the change (Aalto Triton job 21672365, CSC Roihu job 3210044).
  • Sequential speed (parallel = FALSE; full LOO CV with n = 300, K-fold CV with n = 1000, project() with four submodel types): the medians differ from master by at most 0.33 s, within the run-to-run variation, and the results are identical (CSC Roihu jobs 3211274 and 3212652).
  • Full suite with NOT_CRAN=true at 6345151 on CSC Roihu (job 3209996; R 4.6.1, testthat 3.3.2, lme4 2.0.6, brms 2.23.1, rstanarm 2.32.2, future 1.82.0): 61,243 expectations, 0 failed. None of the 18 warnings comes from the changed code.

This PR was assisted by Claude

@avehtari avehtari added the performance Performance issues (i.e., high runtime or high memory usage). label Nov 18, 2026
Comment thread R/cv_varsel.R Resolved
Comment thread R/cv_varsel.R Resolved
Comment thread R/cv_varsel.R Resolved
@avehtari
avehtari merged commit 584cc23 into master Nov 19, 2026
6 checks passed
@avehtari
avehtari deleted the loo-fold-args branch November 19, 2026 14:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

performance Performance issues (i.e., high runtime or high memory usage).

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants