Repository navigation
perf(spec): build sparse declarations from term tables by default - #1197
Draft
FabianHofmann wants to merge 24 commits into
Draft
FabianHofmann wants to merge 24 commits into
FabianHofmann wants to merge 24 commits into
Conversation
5 tasks
Collaborator
Author
|
Note The following content was generated by AI. Merge conflict with Generated by Claude Code |
Squash of perf/spec-term-table onto master. A node carrying a variable is evaluated as a long term table (integer positions, labels, pending factors) and assembled once per declaration into the CSR store.
Drop LINOPY_SPEC_TERMS and LINOPY_SPEC_TERMS_SCOPE. The builder takes every declaration of a sparse model from the term path and falls back to the eager evaluator in one place, _from_terms, where the path raises Unsupported. Constraints, the objective and named expressions all go through _built. terms.eager_only() turns the path off for tests and debugging.
… store CSRStore.canonical returns the store itself once it is canonical, and a store with_grid or prune derive from it stays so. dataclasses.replace resets the flag, so any store with new terms checks again. prune returns the store itself when no coefficient is small enough, as the dense store does. take on the identity rows only swaps the grid. coo_to_csr and aggregate count sorted rows with bincount in linear time, not a binary search per row.
Grid.isel picks the cells and coordinates of an integer or slice selection with numpy, as Dataset.isel gives them, drop or not, auxiliary coordinates included. Store.isel and LinearExpression.isel on a CSR-backed expression use it; other indexers take the xarray path. This replaces the fast path for a scalar drop selection.
Constraints.sanitize_zeros no longer exists (#1183).
_left builds the left side and finds the rows it leaves without a term in one pass, so a side from the term path is not checked twice.
pandas' Index.equals already short-cuts on identity.
…k and one sum A relation sum, a lookup, a shift and a factor over new dims all join a block to rows through Evaluator.matched; a region and an absence both filter through Evaluator.where; Add, Cases and a constraint's difference share _plus. A number is a factor without columns, so scaling is times. A node the tables do not hold now leaves its whole declaration to the eager evaluator: the conversion of an eager result to a table is gone.
…heck makes dead mathspec refuses a sum, a lookup or a shift along a dimension its operand lacks and a grouped sum onto one it carries, so the six guards for them never fired. Name the join's pairs for what they are, and list in the module docstring what leaves a declaration to the eager evaluator.
…ives Covers the branches the spec suite and the PyPSA networks left unread: a sum of a term without the summed dimension, a factor over new dims, a lookup and a grouped sum beside a constant, a shift with an edge value, the sorted join for a key space too large to index, and the fall back to the eager evaluator.
… no longer needs For the merge with b/clean-integrate, whose builder asks terms.active before it calls the term path and no longer appends to FALLBACKS. On b/clean the builder still does both, so this commit is not part of b/clean-engine.
# Conflicts: # linopy/spec/terms.py
Fold test_spec_term_tables.py into test_spec_terms.py. Rows built from term tables are compared with the eager build of the same sparse model, through the one assert_built_alike helper. The rows the term path leaves to the eager path share the row program of the table tests; the window sum is tested once. The table recorder no longer sets the removed LINOPY_SPEC_TERMS switch, and assert_alike narrows both sides for mypy.
Re-measured on a quiet run against linopy 912a8c9: 1.9-3.0x on the four PyPSA networks, carbon_management 8.2 s -> 3.3 s.
FabianHofmann
force-pushed
the
perf/spec-term-table
branch
from
November 21, 2026 21:42
1ba2ade to
5110806
Compare
Merging this PR will not alter performance
Comparing Footnotes
|
4 of 5 tasks
…t hold (#1198) * feat(spec)!: Model.from_spec always builds a sparse model from_spec forwards Model keyword arguments and now sets sparse=True itself; sparse=False raises a ValueError. The builder tries the term path for every declaration outside terms.eager_only(), so terms.active no longer reads the model. Tests that ran a dense spec build compare the term path with the eager path instead. * fix(spec)!: keep a constraint row without a variable term where it holds A row the data leaves without a term of nonzero coefficient is the constant row 'lhs sense rhs'. linopy keeps such a row since #1183, so from_spec now keeps it where it holds, also in a constraint all of whose rows are constant, and raises a SpecDataError naming the row only where it does not; the warning for a row whose sides are equal goes. The term path and the eager path judge a row alike, a zero coefficient counting as no term, so the eager fallback for such a row goes too. * docs: release note PR number for #1198 * refactor(spec): drop the dense branch of constant_term Model.from_spec builds a sparse model only, and a term without a variable is made only while it builds, so constant_term never meets a dense model. * test(spec): run the eager path where dense builds covered it The dense spec builds that went with sparse-only from_spec were the only ones to reach a few eager-path branches: a sum over a dimension without members, cases that leave only data, and a dimension dim_order spreads onto a term. Those tests now also build with terms.eager_only(). A new test frames a quadratic expression, which the term path leaves to the eager path, in the name order of its dims. --------- Co-authored-by: Claude <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes proposed in this Pull Request
Note
The following content was generated by AI.
Model.from_specnow evaluates each declaration as a term table by default. A node that carries a variable becomes a long table of integer positions, variable labels and pending factors, and the declaration's CSR matrix is assembled once. The eager evaluator stays as the fallback for nodes the tables do not cover.from_specbuilds sparse models only (merged in from #1198).linopy/spec/terms.py(new) holds the term-table engine. One join serves the grouped sum, lookup, shift and data gather;+,sum,by=and*datamap onto it. Nodes the tables do not cover raiseUnsupported, and the whole declaration is then built eagerly: window sum, power of a term, product of two live terms, division by a term.builder.py: constraints, the objective and named expressions take one route, with one hand-off to the eager path.linopy.spec.terms.eager_only()turns the term path off, for tests and debugging.csr.py: a store that is already canonical stays canonical;pruneandtakekeep the store when nothing changes; a sorted COO's indptr is built by counting.Grid.iselselects integer and slice indexers on CSR-backed stores and expressions without going through xarray.from_specis unreleased):from_specalways buildsModel(sparse=True)and refusessparse=False. A row the data leaves without a term of nonzero coefficient is the constant rowlhs sense rhs: kept where it holds (0 <= 5), refused with aSpecDataErrornaming the row where it does not (0 >= 5). Both build paths judge such a row alike, so it no longer falls back to the eager path.Wall time of PyPSA's
create_model(sparse=True)with the mathspec formulation (PyPSA#2128 @654e6a1, same PyPSA in both columns), in seconds, best of 3 interleaved rounds:The built models are identical to master's eager path on these four networks: same matrices, bounds, labels, names and dims.
test/test_spec_terms.pybuilds spec programs both ways and covers each fallback.Notes
Constraints.sanitize_zeros, which fix: a constraint row without a nonzero term reaches the solver #1183 removed. The benchmark restores that method locally; PyPSA#2181 drops the call, and PyPSA#2188 drops PyPSA's dense spec builds.SpecDataError, as on master.operators.binaryuses xarray's privateDataArray._replace(3 µs against about 118 µs for the public constructor).eager_only()sets a module global and is not thread-local.Checklist
AGENTS.md).doc.doc/release-notes.mdof the upcoming release is included.Generated by Claude Code