Repository navigation
Conversation
Add `zarr_metadata.testing.strategies`, behind a new `testing` extra.
It ships Hypothesis strategies that draw valid Zarr v3 metadata fields,
one for every codec, data type, chunk grid and chunk key encoding the
package defines that holds no other field.
- Each kind's strategy takes what its configurations depend on: a data
type's fill values take its configuration, a chunk grid takes the
array shape, and a codec takes the `Chunk` it is handed.
- `fields` draws every envelope spelling the spec allows, or only the
canonical one.
- Strategies are a mapping keyed by definition, so an extension author
can add strategies for their own definitions.
The tests read every draw back in `CORE_AND_EXTENSIONS` and require no
problems. They also check:
- that every definition in scope has a strategy, apart from the three
that hold nested fields;
- reach, with targets derived from each configuration's TypedDict:
each optional key present and absent, each `Literal` value, and a
non-canonical spelling where one exists;
- fill-value spellings and envelope forms.
Strategies follow the spec where the package is looser. Core data types
are drawn only as bare strings, and regular-grid chunk lengths are never
zero ("Chunk sizes must be greater than zero"), although the package's
rules accept both today.
Assisted-by: ClaudeCode:claude-opus-5-5
…414 Assisted-by: ClaudeCode:claude-opus-5-5
github-actions
Bot
added
needs release notes
Port the testing package to the definitions API from zarr-developers#4490: shared names come from `zarr_metadata.definition`, a field read is an `AcceptedField`, a definition's own scope is a `ZarrV3Context` of one slot, and the drift test compares against the spec's extension points only, since the scope now also files document members. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Documentation build overview
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #4575 +/- ##
=======================================
Coverage 94.84% 94.84%
=======================================
Files 96 96
Lines 13998 13998
=======================================
Hits 13276 13276
Misses 722 722 🚀 New features to boost your workflow:
|
…PR 4575 The fragment was numbered after fork PR 414, so the changelog linked to an unrelated upstream PR. Assisted-by: ClaudeCode:claude-opus-5-5
`st.just({})` (crc32c) and `st.just(EmptyConfiguration())` (every data
type with nothing to configure) handed out one dict for every draw, and
`fields` puts the configuration in the field as it is. A test that
changed the configuration it drew changed every later draw, which
Hypothesis reports as a flaky strategy. Both are now
`st.builds(EmptyConfiguration)`.
Export `Configurations` and `NOTHING_KNOWN` from
`zarr_metadata.testing.strategies`: they appear in the signatures of
`DefinitionStrategy.unconstrained` and `codec_configurations`.
Assisted-by: ClaudeCode:claude-opus-5-5
d-v-b
marked this pull request as ready for review
October 31, 2026 22:07
Contributor
Author
|
internal to zarr-metadata and so self-merging |
…t values `fields` read a field only when asked for the canonical spelling, so a caller's configurations the definition refuses were drawn silently in the default spelling. Every field is now read, in a scope built once per call rather than once per draw, and a refused one raises in both spellings. A definition no Zarr v3 scope holds, such as a v2 codec, is a `TypeError` when `fields` is called, not a `StopIteration` on a draw. A float `scale_offset` codec's `offset` and `scale` are fill values of its type, so they are now also drawn as the hex string of their bits. The docstring no longer claims a non-zero scale makes decoding defined: a tiny one still overflows. The codec chunk test now draws chunks of an unknown data type and with unknown lengths, which every codec takes. Assisted-by: ClaudeCode:claude-opus-5-5
…n Strategies keys `scale_offset` drew nothing for complex numbers, numpy times and data types the package does not define, though its rules refuse only data types whose values are not numbers, and a numpy time "is compatible with any codec that supports arrays of signed 64-bit integers". A numpy time is now drawn as that integer, a complex number as a pair of finite floats, and a type the package does not define takes the identity configuration. None now means only what `CodecStrategy` says it means: no configuration takes the chunk. `Strategies` is keyed by `Any`: a mapping's key type is invariant, so a mapping an extension author builds, keyed by a `CodecDefinition`, was not a `Mapping[Definition[Any], ...]` to pyright. A rectilinear dimension of unknown length may now be drawn with no edges, which its rules accept. Assisted-by: ClaudeCode:claude-opus-5-5
…d rank cap `EMPTY_CONFIGURATIONS` and `MAX_RANK` are defined once, in `_strategy`, rather than once in each module that draws them. A numpy time's `scale_offset` values are now drawn finite, never the integer numpy reads as `NaT`, as the docstring says. The `fields` docstring now says that a codec's chunk rules are not checked, since no chunk is known. Assisted-by: ClaudeCode:claude-opus-5-5
…does not define The identity configuration `scale_offset` draws for a data type the package does not define had no test, since the chunk test draws only the package's own data types. A test now reads an extension `bfloat16` in a scope of its own and checks the codec takes every draw. Also splits the `_scope_of` error message across two literals and drops a doubled blank line in `_data_type`. Assisted-by: ClaudeCode:claude-opus-5-5
d-v-b
added a commit
to d-v-b/zarr-python
that referenced
this pull request
Nov 1, 2026
… definitions The Hypothesis strategies merged from main (zarr-developers#4575) imported `CORE`, `CORE_AND_EXTENSIONS` and `CORE_V2` from modules that now hold only the `_CONTEXT` names, and their coverage test requires a strategy for every definition in `CORE_AND_EXTENSIONS_CONTEXT`. Rename the imports, and add strategies for `fixed_length_utf32`, `vlen-utf8` and `vlen-bytes`; each vlen codec draws nothing for a data type it does not read. Assisted-by: ClaudeCode:claude-opus-5-5
d-v-b
added a commit
that referenced
this pull request
Nov 1, 2026
…ONTEXT scope (#4583) * feat(zarr-metadata)!: read what zarr-python writes in a ZARR_PYTHON_CONTEXT scope Add definitions for the extensions zarr-python writes. zarr-extensions registers `fixed_length_utf32` and the `vlen-utf8` and `vlen-bytes` codecs, so they join CORE_AND_EXTENSIONS_CONTEXT. zarr-python's own `null_terminated_bytes` and `variable_length_bytes`, which no specification registers, are only in the new ZARR_PYTHON_CONTEXT, so the default scope stays spec-pure. Rename CORE to CORE_CONTEXT and CORE_AND_EXTENSIONS to CORE_AND_EXTENSIONS_CONTEXT, so that a scope's name says it is a context. Fixes d-v-b#423 Assisted-by: ClaudeCode:claude-opus-5-5 * docs(zarr-metadata): number the changelog fragments for PR 426 Assisted-by: ClaudeCode:claude-opus-5-5 * docs(zarr-metadata): number the changelog fragments for PR 4583 Assisted-by: ClaudeCode:claude-opus-5-5 * fix(zarr-metadata): keep the old scope names and tighten the string rules Keep `CORE` and `CORE_AND_EXTENSIONS` importable from `zarr_metadata.v3` and `zarr_metadata.v3.definition` as the same scopes with a DeprecationWarning, so a consumer such as pydantic-zarr keeps working across the rename. Check that `vlen-utf8` encodes `string` and `vlen-bytes` encodes `bytes` (or zarr-python's `variable_length_bytes`), as zarr-extensions says, instead of accepting any data type whose values vary in size. Do not count trailing null bytes against the size of a `null_terminated_bytes` fill value, since they are padding. Spell a `fixed_length_utf32` fill value without its trailing U+0000 padding, so `"a"` and `"a\u0000"` have one canonical spelling. Assisted-by: ClaudeCode:claude-opus-5-5 * feat(zarr-metadata): read zarr-python's raw_bytes and rename CORE_V2 Add zarr-python's `raw_bytes` data type (NumPy's `V`) to ZARR_PYTHON_CONTEXT, so the scope reads every data type zarr-python writes. Its names start with `ZarrPython`, since `RawBytes` already names the spec's raw bits. A fill value longer than `length_bytes` is a problem, because zarr-python cuts it short, and the canonical spelling pads it with null bytes, as zarr-python writes it. Rename CORE_V2 to CORE_V2_CONTEXT, so every scope's name ends in `_CONTEXT`. The old name keeps working with a DeprecationWarning, through the same `renamed_scope` helper as the v3 names, which moves to `zarr_metadata._renamed`. Share the base64 decoding and spelling of fill values in `_byte.py`, and say that the vlen codecs' `configuration` key is optional, since zarr-python writes an empty object. Assisted-by: ClaudeCode:claude-opus-5-5 * refactor(zarr-metadata): declare the v2 scope's old name once Move CORE_V2_CONTEXT and its table of old names into `v2/_core.py`, as `v3/_core.py` holds the v3 scopes, so `zarr_metadata.v2` and `zarr_metadata.v2.definition` share one table. Say why a `fixed_length_utf32` fill value counts its trailing U+0000 code points, as the spec requires, while `null_terminated_bytes` follows zarr-python and does not, and test a value that only its padding makes too long. Assisted-by: ClaudeCode:claude-opus-5-5 * fix(zarr-metadata): give main's strategies the renamed scopes and new definitions The Hypothesis strategies merged from main (#4575) imported `CORE`, `CORE_AND_EXTENSIONS` and `CORE_V2` from modules that now hold only the `_CONTEXT` names, and their coverage test requires a strategy for every definition in `CORE_AND_EXTENSIONS_CONTEXT`. Rename the imports, and add strategies for `fixed_length_utf32`, `vlen-utf8` and `vlen-bytes`; each vlen codec draws nothing for a data type it does not read. Assisted-by: ClaudeCode:claude-opus-5-5 * fix(zarr-metadata): draw zarr-python's own data types in the testing strategies `DEFAULT_STRATEGIES` holds a strategy for each definition the package defines, but `null_terminated_bytes`, `variable_length_bytes` and `raw_bytes` had none, so `fields()` raised `LookupError` for them. Draw each, and check the strategies against `ZARR_PYTHON_CONTEXT`, which holds every definition. Say that the documents in test_zarr_python_scope.py leave out the compressors zarr-python adds. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(zarr-metadata): pin the registered extensions to CORE_AND_EXTENSIONS_CONTEXT The strategies are now checked against `ZARR_PYTHON_CONTEXT`, which no longer proves that `fixed_length_utf32`, `vlen-utf8` and `vlen-bytes` are in `CORE_AND_EXTENSIONS_CONTEXT`. Assert it directly. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(zarr-metadata): let the vlen strategies ask their codec's rules Each vlen codec strategy named its compatible data types a second time; draw nothing for exactly the chunks the codec's own chunk rules refuse, so the two cannot drift. Draw `fixed_length_utf32` fill values with trailing U+0000 padding too, which the canonical spelling removes. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Hypothesis testing strategies for metadata fields behind a new
testingextra. One strategy for codecs, data types, chunk grids, and chunk key encodings. Nested codecs and data types (sharding, cast_value, struct) will get their strategies added in a follow-up.There will be a series of these PRs. Once they land in
zarr-metadata, we can remove some of the same logic fromzarritself.For reviewers
[What would you most value a second look at? What are you already confident in? For a refactor, say whether behavior is meant to be unchanged.]
Author attestation
TODO
docs/user-guide/*.mdchanges/