Robust analysis of memory consumption in Chonk #1641

Activity

1. johnathan79717 commented on Mar 5, 2026

   

   johnathan79717

   on Mar 5, 2026

   More actions

   Prover memory profile: where the bytes go

   Profiled the AMM flow (ecdsar1+amm_add_liquidity_1_recursions+sponsored_fpc, 19 circuits) using BB_POLY_STATS=1. Here's the memory breakdown and where savings opportunities exist.

   Overall memory trace (RSS)

   Start:                           172 MiB  (CRS + msgpack parsing)
   After EcdsaRAccount (122K):      305 MiB  (+133 MiB)
   After private_kernel_init (47K): 342 MiB  (+37 MiB)
   ... several small circuits ...   342 MiB  (reusing freed memory)
   After Token:transfer (180K):     559 MiB  (+217 MiB)  ← largest circuit
   After next kernel_inner:         613 MiB  (peak RSS)
   

   RSS is a high-water mark — glibc doesn't return freed pages to the OS. Circuits are processed sequentially; each circuit's ProverInstance is freed after folding. The peak is set by the largest circuit being accumulated.

   Per-polynomial breakdown (EcdsaRAccount, 122K trace, representative)

   Category	Count	Per-poly MB	Total MB	Values fit in
   sigma_1-4	4	3.73	14.9	≤4 bytes (100%)
   id_1-4	4	3.73	14.9	≤4 bytes (100%)
   w_l/w_r/w_o/w_4	4	3.73	14.9	mixed (30-50% ≤4B, rest ≤32B)
   q_m/q_c/q_l/q_r/q_o/q_4	6	3.73	22.4	mixed (sparse, many zeros)
   Gate selectors	8	0-1.3	~3.8	≤4 bytes
   Tables	4	1.10	4.4	mixed
   Other (z_perm, lookup, databus, etc.)	~20	small	~7.2	various
   Total	~55		82.5 MB	85.7% compressible

   The Compr(MB) column in BB_POLY_STATS shows ideal compressed size: 11.8 MB vs 82.5 MB actual. Most of the gap is from storing small values (≤4 bytes) in 32-byte Fr elements, and from zero elements still occupying 32 bytes.

   Where peak memory occurs

   During each circuit's accumulation, the peak is at sumcheck round 0:

   Original prover polynomials:      82.5 MB (100%)
   + PartiallyEvaluatedMultivariates: ~41 MB (50%, half-size copy)
   = Peak during sumcheck:          ~124 MB (150% of polynomial set)
   

   After sumcheck, PEM is freed and batch_polynomials reuses existing polynomial storage (the first polynomial becomes the accumulation target, others are added-scaled into it).

   Concrete reduction proposals

   1. Compact sigma/id storage (uint32_t compression) — WIP in PR #21124

   • Compresses 8 polynomials from 32B to 4B per element during oink+sumcheck
   • Savings: ~26 MB for the largest circuit (122K trace), proportionally more for larger traces
   • Transparent to readers via Polynomial::operator[] returning Fr by value
   • Currently only in UltraProver path; needs to also go in HyperNova folding path

   2. Sparse/zero-aware allocation

   • Many polynomials are >50% zeros (e.g., q_m: 77% zeros, q_c: 80% zeros)
   • Virtual-zero extension already avoids allocating trailing zeros
   • Further savings from run-length encoding or bitmap-based sparse storage

   3. Free early, allocate late

   • Circuit block memory is already freed after copy to prover polynomials (done in chonk::accumulate)
   • Selectors could potentially be freed after commitment (VK construction) since they're not needed for sumcheck in all flavors — need to check relation dependencies
   • z_perm, lookup_inverses are allocated but zero until computed — could defer allocation

   4. malloc_trim between circuits

   • Calling malloc_trim(0) after each circuit's fold returns would release freed pages back to OS
   • Wouldn't reduce peak but would prevent glibc arena inflation from artificially raising RSS
   • Useful for measurement and for reducing memory pressure on constrained devices

   5. Polynomial streaming / on-demand decode

   • Instead of expand→batch, teach batch_polynomials to read compact sources directly
   • add_scaled(other, scalar) already reads via other[index] which handles compact mode
   • Only the accumulation target (first polynomial) needs raw data() — all others can stay compact
2. ledwards2225

   assigned

   johnathan79717

   on Mar 9, 2026
3. ledwards2225 commented on Mar 9, 2026

   

   ledwards2225

   on Mar 9, 2026

   Author

   More actions

   @johnathan79717 FYI it seems that some of the changes/optimizations we're considering in the near future may effect the memory behavior a bit, e.g. where the peak lies etc. Just more motivation to think about some kind of tooling to make it easy for us to continually reassess the memory situation in the near term
4. johnathan79717 commented on Mar 27, 2026

   

   johnathan79717

   on Mar 27, 2026

   More actions

   /claudebox suggest tooling options to make it easy for us to continually reassess the memory situation, especially about the peak memory usage, how high it is, what contributes to the peak, etc.

   👀React with 👀1Reacted by Aztec Bot
5. AztecBot commented on Mar 27, 2026

   

   AztecBot

   on Mar 27, 2026

   Collaborator

   More actions

   ClaudeBox: suggest tooling options to make it easy for us to continually reassess the memory situation, especially about the peak m ... workflow run
6. johnathan79717

   added 5 commits that reference this issue on Mar 30, 2026

   feat: add --memory_profile_out flag for Chonk memory profiling

   ...

   6183043

   feat: add plot_memory.py for visual RSS timeline of bb prove runs

   ...

   bd5edaf

   feat: add --memory_profile_out flag for Chonk memory profiling (#22145)

   ...

   ad82386

   feat: switch memory profiling from peak RSS to live heap usage (#22229)

   ...

   e22f2cd

   feat: switch memory profiling from peak RSS to live heap usage (#22266)

   ...

   e7adaa6