Repository navigation
perf(spec): fold product chains, contract sums and add a sparse grouped sum - #1196
Draft
FabianHofmann wants to merge 5 commits into
Draft
FabianHofmann wants to merge 5 commits into
FabianHofmann wants to merge 5 commits into
Conversation
…ed sum - fold the data factors of a product chain before scaling the term once - contract a sum of a product onto the cells both sides hold - a sparse grouped sum scatters CSR rows straight into their groups - LinearExpression.where takes a CSR fast path for a mask on the grid - CSRStore skips term-free operands on add and bins sorted rows on aggregate
Merging this PR will not alter performance
Comparing Footnotes
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note
The following content was generated by AI.
Speeds up
Model.from_specfor sparse models. Found while profiling PyPSA/PyPSA#2128, which builds ~2x slower than PyPSA master. Draft: parks the work; the structural fix (a deferred term table, see below) is still open.Changes proposed in this Pull Request
term * d1 * d2, the data factors multiply first and the term is scaled once.sum(term * data, over=…)reads only the cells both sides hold, when at most half are live. This avoids broadcasting a term over a data-only dim, e.g.global_constraint.casesand quadratics: a single case covering every row returns its term as is. A quadratic whose data is zero everywhere becomes its constant, without being built.grouped_sum: CSR rows scatter straight into their groups. Before, sparse terms wentgroupby → reindex → fillna.LinearExpression.wheretakes a CSR fast path when the mask lies on the grid.CSRStoreskips term-free operands on add and bins sorted rows on aggregate.Built models are identical to before: the full matrices, bounds and objective of 3 PyPSA example networks match at rtol 1e-12. Folding the data factors first changes rounding by about 1e-16.
Benchmarks and caveats
n.optimize.create_model(sparse=True), PyPSA PR branch at 654e6a1, best of 5 runs over two rounds, in seconds. Master is PyPSA master on the same linopy commit.Per phase on carbon_management, PR before → after: nodal balance + KVL 2.29 → 1.42 (master 1.65), objective 0.91 → 0.56 (master 0.17). The machine was noisy: master alone moved about 15% between rounds.
Behaviour change: inside a product chain,
refuse_madenow checks the folded data. If inf × 0 is made in a built row, the build raisesSpecDataError. Before, it silently left a NaN coefficient.Open questions:
wherefast path changes core linopy, not just the spec code.What is still slow:
_relabel, about 0.65 s.Tests:
pytest testgives 5536 passed and 3 failed. The 3 are the SOS piecewise tests, which also fail on master because HiGHS does not support SOS. ruff passes.Checklist
AGENTS.md).doc.doc/release-notes.mdof the upcoming release is included.Generated by Claude Code