Skip to content

perf(spec): fold product chains, contract sums and add a sparse grouped sum - #1196

Draft
FabianHofmann wants to merge 5 commits into
masterfrom
perf/spec-all
Draft

FabianHofmann wants to merge 5 commits into
masterfrom
perf/spec-all

Conversation

@FabianHofmann

@FabianHofmann FabianHofmann commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

Note

The following content was generated by AI.

Speeds up Model.from_spec for sparse models. Found while profiling PyPSA/PyPSA#2128, which builds ~2x slower than PyPSA master. Draft: parks the work; the structural fix (a deferred term table, see below) is still open.

Changes proposed in this Pull Request

  • Product chains fold: in term * d1 * d2, the data factors multiply first and the term is scaled once.
  • Sums of products contract: sum(term * data, over=…) reads only the cells both sides hold, when at most half are live. This avoids broadcasting a term over a data-only dim, e.g. global_constraint.
  • cases and quadratics: a single case covering every row returns its term as is. A quadratic whose data is zero everywhere becomes its constant, without being built.
  • Sparse grouped_sum: CSR rows scatter straight into their groups. Before, sparse terms went groupby → reindex → fillna.
  • Core linopy: LinearExpression.where takes a CSR fast path when the mask lies on the grid. CSRStore skips term-free operands on add and bins sorted rows on aggregate.

Built models are identical to before: the full matrices, bounds and objective of 3 PyPSA example networks match at rtol 1e-12. Folding the data factors first changes rounding by about 1e-16.

Benchmarks and caveats

n.optimize.create_model(sparse=True), PyPSA PR branch at 654e6a1, best of 5 runs over two rounds, in seconds. Master is PyPSA master on the same linopy commit.

network PyPSA master PR, linopy 912a8c9 PR, this branch
carbon_management 3.5–4.0 6.4–6.9 6.0
scigrid_de 0.84 2.02 1.35–1.5
model_energy 0.73 1.66–1.85 1.47–1.6
ac_dc_meshed 0.43 1.07 0.97–1.0

Per phase on carbon_management, PR before → after: nodal balance + KVL 2.29 → 1.42 (master 1.65), objective 0.91 → 0.56 (master 0.17). The machine was noisy: master alone moved about 15% between rounds.

Behaviour change: inside a product chain, refuse_made now checks the folded data. If inf × 0 is made in a built row, the build raises SpecDataError. Before, it silently left a NaN coefficient.

Open questions:

  • The contraction threshold ("at most half the cells live") is untuned.
  • The where fast path changes core linopy, not just the spec code.

What is still slow:

  • PyPSA's post-build _relabel, about 0.65 s.
  • Storage weights stored as hard zeros instead of absent, about 0.4 s.
  • One full-size linopy object per spec node. That needs a deferred term table, like fluxopt/specsolve's polars terms, rather than more local patches.

Tests: pytest test gives 5536 passed and 3 failed. The 3 are the SOS piecewise tests, which also fail on master because HiGHS does not support SOS. ruff passes.

Checklist

  • AI-generated content is marked (see AGENTS.md).
  • Code changes are sufficiently documented; i.e. new functions contain docstrings and further explanations may be given in doc.
  • Unit tests for new features were added (if applicable).
  • A note for the release notes doc/release-notes.md of the upcoming release is included.
  • I consent to the release of this PR's code under the MIT license.

Generated by Claude Code

…ed sum

- fold the data factors of a product chain before scaling the term once
- contract a sum of a product onto the cells both sides hold
- a sparse grouped sum scatters CSR rows straight into their groups
- LinearExpression.where takes a CSR fast path for a mask on the grid
- CSRStore skips term-free operands on add and bins sorted rows on aggregate
@codspeed

codspeed Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 181 untouched benchmarks
⏩ 181 skipped benchmarks1


Comparing perf/spec-all (95e7c7c) with master (9b1b998)

Open in CodSpeed

Footnotes

  1. 181 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants