Skip to content

Build submodel model matrices on projection cache misses from one main-effect model matrix - #654

Open
avehtari wants to merge 6 commits into
masterfrom
cache-miss
Open

avehtari wants to merge 6 commits into
masterfrom
cache-miss

Conversation

@avehtari

Copy link
Copy Markdown
Member

This PR is based on #652; the base changes to master after #652 is merged.

Summary

On a projection cache miss, proj_to_submodl() builds a data frame with the draw-wise responses and calls model.frame() and model.matrix() for the submodel. A miss takes 1.5-2.1 ms. A LOO validated search with D = 500 candidate terms and nloo = 50 makes about 500,000 misses.

This PR builds the model matrix of all main-effect terms once per cache store (design_main_effects()). On a miss, shared_x_from_design() takes the column blocks of the submodel terms. It also builds the rest of the cache entry: the terms object with predvars and dataClasses, factor levels, contrasts and response names. Without interactions, the columns of a term do not depend on the other terms. The cache entry is then identical to that of the old path (now proj_prepare_general()).

The old path stays for:

  • interactions;
  • models without intercept;
  • terms outside the reference model formula;
  • missing values in a candidate column;
  • column names that data.frame() changes;
  • an error in model.frame() or model.matrix() of all candidate terms;
  • a main-effect model matrix above a quarter of projpred.proj_cache_mb.

design_main_effects() builds the matrix at the third miss of a cache store.

Effect

Simulated Bernoulli data (N 100, D candidate terms, brms reference model), forward search with nterms_max = 20. Times in seconds; both versions ran in the same job.

D call before after speedup
100 varsel() 10.07 8.21 1.23
100 cv_varsel(), LOO, nloo = 50 502.94 428.96 1.17
500 varsel() 63.47 49.85 1.27
500 cv_varsel(), LOO, nloo = 50 3148.60 2378.95 1.32

With loo_perturb_alpha = Inf, the LOO speedup is 1.18 (D 100) and 1.52 (D 500). For 5 GLM data sets with 4-30 predictors, the speedup is 0.98-1.28. In all 21 settings, the two versions give identical rankings, suggested sizes and performance summaries.

Tests

tests/testthat/test_proj_design.R checks that:

  • the new cache entries are identical() to those of the old path for each column type and term type in the test file;
  • the cases in the list above use the old path;
  • varsel() and cv_varsel() give identical results with and without the new path.

Without the change, these tests stop with 5 errors. The full test suite with NOT_CRAN=true (R 4.6.1) gives 0 failures and the same 14 warnings as other branches on master.

This PR was assisted by Claude.

@avehtari avehtari added the performance Performance issues (i.e., high runtime or high memory usage). label Nov 17, 2026
Base automatically changed from strip-contrasts-speed to master November 19, 2026 11:32

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

performance Performance issues (i.e., high runtime or high memory usage).

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant