Skip to content

perf(spec): build sparse declarations from term tables by default - #1197

Draft
FabianHofmann wants to merge 24 commits into
masterfrom
perf/spec-term-table
Draft

FabianHofmann wants to merge 24 commits into
masterfrom
perf/spec-term-table

Conversation

@FabianHofmann

@FabianHofmann FabianHofmann commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

Changes proposed in this Pull Request

Note

The following content was generated by AI.

Model.from_spec now evaluates each declaration as a term table by default. A node that carries a variable becomes a long table of integer positions, variable labels and pending factors, and the declaration's CSR matrix is assembled once. The eager evaluator stays as the fallback for nodes the tables do not cover. from_spec builds sparse models only (merged in from #1198).

  • linopy/spec/terms.py (new) holds the term-table engine. One join serves the grouped sum, lookup, shift and data gather; +, sum, by= and *data map onto it. Nodes the tables do not cover raise Unsupported, and the whole declaration is then built eagerly: window sum, power of a term, product of two live terms, division by a term.
  • builder.py: constraints, the objective and named expressions take one route, with one hand-off to the eager path. linopy.spec.terms.eager_only() turns the term path off, for tests and debugging.
  • csr.py: a store that is already canonical stays canonical; prune and take keep the store when nothing changes; a sorted COO's indptr is built by counting.
  • Grid.isel selects integer and slice indexers on CSR-backed stores and expressions without going through xarray.
  • From feat(spec)!: build spec models sparse only and keep constant rows that hold #1198 (breaking, from_spec is unreleased): from_spec always builds Model(sparse=True) and refuses sparse=False. A row the data leaves without a term of nonzero coefficient is the constant row lhs sense rhs: kept where it holds (0 <= 5), refused with a SpecDataError naming the row where it does not (0 >= 5). Both build paths judge such a row alike, so it no longer falls back to the eager path.

Wall time of PyPSA's create_model(sparse=True) with the mathspec formulation (PyPSA#2128 @ 654e6a1, same PyPSA in both columns), in seconds, best of 3 interleaved rounds:

network linopy master this PR
ac_dc_meshed 1.09 0.67
scigrid_de 1.90 0.83
model_energy 1.84 0.77
carbon_management 7.80 3.80

The built models are identical to master's eager path on these four networks: same matrices, bounds, labels, names and dims. test/test_spec_terms.py builds spec programs both ways and covers each fallback.

Notes
  • Rounds on the benchmark VM differ by 10–30 %, so read the ratios, not the last digit.
  • PyPSA#2128 still calls Constraints.sanitize_zeros, which fix: a constraint row without a nonzero term reaches the solver #1183 removed. The benchmark restores that method locally; PyPSA#2181 drops the call, and PyPSA#2188 drops PyPSA's dense spec builds.
  • Eager fallbacks over PyPSA's test suite: 414 of 19,601 declarations, all window sums (409) or products of two variables (5).
  • The term path stores no zero-coefficient terms where the eager path stores explicit zeros. The solver model is the same.
  • A data-only objective raises SpecDataError, as on master.
  • operators.binary uses xarray's private DataArray._replace (3 µs against about 118 µs for the public constructor).
  • eager_only() sets a module global and is not thread-local.
  • History: one squash commit of the earlier prototype, three merged clean-up branches, then feat(spec)!: build spec models sparse only and keep constant rows that hold #1198.

Checklist

  • AI-generated content is marked (see AGENTS.md).
  • Code changes are sufficiently documented; i.e. new functions contain docstrings and further explanations may be given in doc.
  • Unit tests for new features were added (if applicable).
  • A note for the release notes doc/release-notes.md of the upcoming release is included.
  • I consent to the release of this PR's code under the MIT license.

Generated by Claude Code

Copy link
Copy Markdown
Collaborator Author

Note

The following content was generated by AI.

Merge conflict with master, in linopy/constraints.py only. #1183 removed Constraints.sanitize_zeros, and this branch changes that method. PyPSA#2128 still calls m.constraints.sanitize_zeros() at its current head and pins linopy 912a8c9, so this branch stays on 912a8c9 for now. The conflict resolves once #2128 drops that call; then this branch merges master.


Generated by Claude Code

Squash of perf/spec-term-table onto master. A node carrying a variable is
evaluated as a long term table (integer positions, labels, pending factors)
and assembled once per declaration into the CSR store.
Drop LINOPY_SPEC_TERMS and LINOPY_SPEC_TERMS_SCOPE. The builder takes
every declaration of a sparse model from the term path and falls back
to the eager evaluator in one place, _from_terms, where the path raises
Unsupported. Constraints, the objective and named expressions all go
through _built. terms.eager_only() turns the path off for tests and
debugging.
… store

CSRStore.canonical returns the store itself once it is canonical, and a
store with_grid or prune derive from it stays so. dataclasses.replace
resets the flag, so any store with new terms checks again. prune returns
the store itself when no coefficient is small enough, as the dense store
does. take on the identity rows only swaps the grid.

coo_to_csr and aggregate count sorted rows with bincount in linear time,
not a binary search per row.
Grid.isel picks the cells and coordinates of an integer or slice
selection with numpy, as Dataset.isel gives them, drop or not, auxiliary
coordinates included. Store.isel and LinearExpression.isel on a
CSR-backed expression use it; other indexers take the xarray path. This
replaces the fast path for a scalar drop selection.
Constraints.sanitize_zeros no longer exists (#1183).
_left builds the left side and finds the rows it leaves without a term
in one pass, so a side from the term path is not checked twice.
pandas' Index.equals already short-cuts on identity.
…k and one sum

A relation sum, a lookup, a shift and a factor over new dims all join a
block to rows through Evaluator.matched; a region and an absence both
filter through Evaluator.where; Add, Cases and a constraint's difference
share _plus. A number is a factor without columns, so scaling is times.
A node the tables do not hold now leaves its whole declaration to the
eager evaluator: the conversion of an eager result to a table is gone.
…heck makes dead

mathspec refuses a sum, a lookup or a shift along a dimension its operand
lacks and a grouped sum onto one it carries, so the six guards for them
never fired. Name the join's pairs for what they are, and list in the
module docstring what leaves a declaration to the eager evaluator.
…ives

Covers the branches the spec suite and the PyPSA networks left unread: a
sum of a term without the summed dimension, a factor over new dims, a
lookup and a grouped sum beside a constant, a shift with an edge value,
the sorted join for a key space too large to index, and the fall back to
the eager evaluator.
… no longer needs

For the merge with b/clean-integrate, whose builder asks terms.active
before it calls the term path and no longer appends to FALLBACKS. On
b/clean the builder still does both, so this commit is not part of
b/clean-engine.
# Conflicts:
#	linopy/spec/terms.py
Fold test_spec_term_tables.py into test_spec_terms.py. Rows built from
term tables are compared with the eager build of the same sparse model,
through the one assert_built_alike helper. The rows the term path leaves
to the eager path share the row program of the table tests; the window
sum is tested once. The table recorder no longer sets the removed
LINOPY_SPEC_TERMS switch, and assert_alike narrows both sides for mypy.
Re-measured on a quiet run against linopy 912a8c9: 1.9-3.0x on the four
PyPSA networks, carbon_management 8.2 s -> 3.3 s.
@FabianHofmann FabianHofmann changed the title perf(spec): build linear terms as long-format term tables perf(spec): build sparse declarations from term tables by default Nov 21, 2026
@codspeed

codspeed Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 181 untouched benchmarks
⏩ 181 skipped benchmarks1


Comparing perf/spec-term-table (d04e922) with master (9b1b998)

Open in CodSpeed

Footnotes

  1. 181 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

…t hold (#1198)

* feat(spec)!: Model.from_spec always builds a sparse model

from_spec forwards Model keyword arguments and now sets sparse=True
itself; sparse=False raises a ValueError. The builder tries the term path
for every declaration outside terms.eager_only(), so terms.active no
longer reads the model. Tests that ran a dense spec build compare the term
path with the eager path instead.

* fix(spec)!: keep a constraint row without a variable term where it holds

A row the data leaves without a term of nonzero coefficient is the
constant row 'lhs sense rhs'. linopy keeps such a row since #1183, so
from_spec now keeps it where it holds, also in a constraint all of whose
rows are constant, and raises a SpecDataError naming the row only where
it does not; the warning for a row whose sides are equal goes. The term
path and the eager path judge a row alike, a zero coefficient counting
as no term, so the eager fallback for such a row goes too.

* docs: release note PR number for #1198

* refactor(spec): drop the dense branch of constant_term

Model.from_spec builds a sparse model only, and a term without a variable
is made only while it builds, so constant_term never meets a dense model.

* test(spec): run the eager path where dense builds covered it

The dense spec builds that went with sparse-only from_spec were the only
ones to reach a few eager-path branches: a sum over a dimension without
members, cases that leave only data, and a dimension dim_order spreads
onto a term. Those tests now also build with terms.eager_only(). A new
test frames a quadratic expression, which the term path leaves to the
eager path, in the name order of its dims.

---------

Co-authored-by: Claude <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants