Skip to content

fix(zarr-metadata): derive each model's key, canonicalize and refines from one normal form - #4648

Draft
d-v-b wants to merge 22 commits into
zarr-developers:mainfrom
d-v-b:fix/zarr-metadata-refines-across-scopes
Draft

d-v-b wants to merge 22 commits into
zarr-developers:mainfrom
d-v-b:fix/zarr-metadata-refines-across-scopes

Conversation

@d-v-b

@d-v-b d-v-b commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

🤖 AI text below 🤖

refines on every model now reads the other model's document in this model's scope and compares the two, so a document read in a richer scope refines the same document read in a poorer one, whatever the richer scope learns of the chunk each codec receives. Before, refined_in of a uint8 array whose bytes codec writes an endian, read in a scope that claims no bytes codec, did not refine the model it came from. A model no longer refines one whose document its own scope reads with a problem: a gzip codec with a configuration key its definition does not declare, read in a scope that claims no gzip, is not refined by the model of the same codec without that key. codec_for_chunk now writes the codec as JSON in its simplest spelling for the chunk, as one step of the walk that canonical_of makes, and returns None for a codec with no simplest spelling. A codec's chunk_canonical receives each nested declaration in its simplest spelling.

Why: one normal form per model

This PR first fixed the cross-scope refines regression from #4640 with a new function, codec_refines. That implementation is replaced here, on the same branch, by a design that removes the cause the defects fixed in #4641, #4647 and this PR share.

Before this branch, what a document means was decided in several separately written walks over the same tree, kept in agreement by hand: the key that == and hash compare (array_key_of, each model's _key_of, extra_fields_key), the canonical document (canonical_array_document_v3 and five others), and refinement (six refines methods, the declaration-level refines, _gained, own_key, and, in this PR's first version, codec_refines). Each of the three defects was one walk missing something another walk had.

On this branch each model writes its document in its simplest equivalent spelling in one function, _canonical_document, and everything else is derived in model/_keyed.py:

  • == and hash compare the claims the scope made and the normal form as JSON text.
  • canonicalize() reads the normal form.
  • refines(other) holds when this scope read every name the other scope read, with the same definition, and the other document read in this scope equals this model.

Member order means nothing by construction, because JSON text sorts the members of an object. The cross-scope case this PR was opened for holds by construction, because one reader, with one scope's knowledge of the chunks, reads both documents.

The same holds one level down. canonical_declaration is the one walk that writes a declaration in its simplest spelling, and a pipeline takes part in it instead of walking the codec a second time: it gives the spelling of each codec it placed, and a step that simplifies the codec's own members for the chunk. So codec_for_chunk returns JSON, nothing builds a second declaration to read again, and canonical is called from one function.

What changed in the source

Against #4642 and #4647 merged, the source is 378 lines added and 530 removed.

Removed: ZarrV3ArrayMetadata._plain_key, ZarrV2ArrayMetadata._plain_key and _fill_value_key, extra_fields_key, ZarrV2ConsolidatedMetadata._other_entries_text, _node_refines, _v2_node_refines, the body of each model's _key_of and refines, and from definition/_scope.py the declaration-level refines, _gained and _Claimed, with own_key from definition/_definition.py. None of these was exported from a public module.

Also removed: codec_refines and read_as_claimed, which this PR's first version added. Neither was released.

In the pipeline: respelled and _inside are removed, with the tests of respelled. codec_for_chunk returns JSONValue | None in place of a declaration. The checks #4647 put on what chunk_canonical gives are all kept, in _asked_for_chunk and canonical_declaration.

canonical_declaration no longer reads a configuration again when the definition's canonical returned what it was given. The key now goes through that function, and without this a listing called each definition's rules more than once per declaration.

chunk_canonical now receives each nested declaration in its simplest spelling, where before it received them as written. Neither #4640 nor #4647 is released.

Behaviour difference found

refines used to judge a gain by reading the unclaimed declaration again and dropping the problems of that read. So the model of {"name": "gzip", "configuration": {"level": 5}} refined a model, read in a scope without a gzip, of the same codec with an extra configuration key. refined_in on that second model raises MetadataValidationError, so the two methods disagreed. On this branch refines returns False there. tests/v3/test_scope.py has the case as no-gain-over-a-problem.

Tests
  • Every test of feat(zarr-metadata): return each model with its document in its simplest equivalent spelling #4642 and fix(zarr-metadata): simplify a codec for its chunk in its canonical spelling, and check the result #4647 passes unchanged, except the three in tests/v3/test_scope.py that called the removed declaration-level refines. Those are ported to models with the same cases. Two cases have no model equivalent and were dropped, because they compare refused declarations and no model holds one.
  • The tests of this PR's first version are kept, except the one that called codec_refines: the six refined_in cases, tests/model/test_invariants.py, and the additions to tests/v3/test_definitions.py.
  • A fuzz over 404,218 pairs of v3 array models (180 documents in 7 scopes) gives the same == and refines answers as this PR's first version: 2,810 equal pairs and 6,187 refinements.
  • Against this PR's first version, a first == of two sharded arrays takes 81 µs against 101 µs, a first refines across scopes 190 µs against 195 µs, and a first == of two listings of 200 arrays 11.0 ms against 15.2 ms.
  • A first hash of an array whose bytes codec sits inside 80 nested shards takes 1.35 ms, against 19.5 ms before the per-chunk step was folded in: the codec is no longer rebuilt at each level.
  • tests/v3/test_pipelines.py checks that chunk_canonical is asked once, with the configuration fully simplified.
What this does not do
  • A standalone declaration still has its own key walk, declaration_key, next to canonical_declaration. Models no longer use it, but AcceptedDeclaration.__eq__ does. Deriving it from the same walk needs a normal form for refused declarations.
Stack

This branch holds the commits of #4642 and #4647, since the design needs canonicalize from the first and the checked chunk_canonical from the second. Merge #4647 and #4642 first, then merge main into this branch, and the diff shows only this PR. Before this change the branch held #4647 alone.

The per-chunk step could not be folded in on #4647's own branch: without #4642 a model's key and refines compare declarations, so codec_for_chunk has to return one there. So #4647 adds respelled and this PR removes it.

The design was first tried as d-v-b#465, which is closed in favour of this PR.

🤖 Generated with Claude Code

d-v-b added 15 commits November 16, 2026 15:28
…the chunk it receives

A definition's `canonical` reduces two spellings of a configuration to
one when they mean the same thing wherever the declaration is used. It
receives the configuration alone, so it cannot reduce spellings that
mean the same thing only for some chunks. The `endian` of a `bytes`
codec is one: it means nothing when each value takes one byte, and is
required when a value takes several.

`ArrayCodecDefinition` gains `chunk_canonical`, a function of the
configuration, the nested declarations and the chunk, and
`codec_for_chunk` returns the codec of a `Stage` in that spelling, with
each codec of the pipelines it contains simplified for the chunk that
one receives. The result is a declaration, so `declaration_key`,
`canonical_of` and `refines` work on it as on any other. Its
configuration is checked again: the definition must allow it, the chunk
rules must find no problem with it, and each nested declaration must
stay as it is.

`bytes` declares a `chunk_canonical` that drops `endian` for a data type
stored in single bytes, and a v3 array model compares, hashes and
refines by its codecs as `codec_for_chunk` returns them. So an `int8`
array is one array with either `endian` or with none, as zarr-python 3.0
and 3.1 write it, while a `bytes` codec after a `cast_value` to a
multi-byte data type is still compared by its `endian`.

Assisted-by: ClaudeCode:claude-fable-5-2
…est equivalent spelling

Two models are equal when their documents mean the same thing, and
`to_json` writes each document as it was given, so equal models can
write different documents. A consumer that needs one spelling for all of
them, to compare documents field by field or to hash them, had to
normalize each member itself.

Each model gains `canonicalize`: the v3 and v2 array, group and
consolidated metadata models. It returns the model read from its
document in its simplest equivalent spelling, written from the parts
that `==` compares: each declaration as `canonical_of` writes it, each
v3 codec as `codec_for_chunk` returns it for the chunk it receives, and
the fill value in the canonical spelling of its data type. Empty
`attributes`, empty v3 `storage_transformers` and a v2
`dimension_separator` of "." are left out, since a document without them
means the same. Everything else is kept as written.

The result equals the model, and equal models in one scope give the
same document apart from the order of the members of an object, which a
`JSONWriter(sort_keys=True)` sorts. A group keeps the documents its
`consolidated_metadata` lists as written, since it does not read them;
the consolidated metadata models canonicalize each node they list.

Assisted-by: ClaudeCode:claude-fable-5-2
…canonical-documents

Main now carries the chunk-aware canonical hook (zarr-developers#4640) and the nested declaration order fix (zarr-developers#4641). The one conflict was in model/_array_v3.py, where this branch adds canonical_array_document_v3 next to codecs_for_chunks; both are kept.

Assisted-by: ClaudeCode:claude-fable-5-2
…s canonical gives

chunk_canonical received the configuration as written, and canonical ran afterwards. A codec whose canonical output its chunk_canonical reduces further then had two keys for one meaning: array models that differed only in such spellings were unequal, and the simplest spelling was not a fixed point.

_for_chunk now hands chunk_canonical the configuration as canonical spells it, and checks that chunk_canonical has nothing left to simplify in its own result. A definition that leaves chunk_canonical at its default is skipped without calling either.

Assisted-by: ClaudeCode:claude-fable-5-2
Assisted-by: ClaudeCode:claude-fable-5-2
…at are not the default

A review found that the Hypothesis tests passed with a canonicalize that returned the model's own document, that no test noticed a model losing its scope, and that four behaviours had no case: a claimed extra field, a v2 fill value, a chunk grid and a storage transformer that are respelled.

The parametrized cases are now read in scopes that are not the default ones. The generated documents are compared with the same document reordered at every depth and, for arrays, with each declaration in its own simplest spelling. The reordering helper reverses every object, which the fix in zarr-developers#4641 allows.

Assisted-by: ClaudeCode:claude-fable-5-2
…g claims

Its configuration is kept, but its envelope is written in the shortest form: an empty configuration and a must_understand of true are left out.

Assisted-by: ClaudeCode:claude-fable-5-2
… document it becomes

A review of zarr-developers#4640 found gaps in how _for_chunk checks the result of chunk_canonical:

- a result that added a nested declaration passed, and the respelled codec then held a declaration its nested mapping did not;
- a result that was not JSON raised an error that did not name the definition;
- an array written as a list was reported as a changed nested declaration;
- a problem inside a nested declaration, which is the document's, was held against the result.

The result is now refined as any JSON is, its declaration locations are compared with the codec's, and problems under a nested location are left out. respelled refuses a location that holds no nested declaration with the TypeError its docstring promises.

Assisted-by: ClaudeCode:claude-fable-5-2
…nows when judging a refinement

Since zarr-developers#4640 an array model's refines compared each side's codecs as simplified for that side's own chunks. A richer scope can know more of a chunk than a poorer one: it claims the data type, a codec that comes before, or the codec itself. It then dropped a bytes codec's endian that the poorer reading kept, and refined_in(...) no longer refined the model it came from.

codec_refines compares the codec of a stage with another reading of it, simplifying both for the chunks the stage knows. A gain is judged by what the unclaimed codec reads as with the stage's claims, simplified the same way. ZarrV3ArrayMetadata.refines uses it for each codec.

Assisted-by: ClaudeCode:claude-fable-5-2
Assisted-by: ClaudeCode:claude-fable-5-2
@github-actions github-actions Bot added needs release notes Automatically applied to PRs which haven't added release notes zarr-metadata Specific to the zarr-metadata sub-package labels Nov 18, 2026
@read-the-docs-community

read-the-docs-community Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

Documentation build overview

📚 zarr-metadata | 🛠️ Build #35982921 | 📁 Comparing 94fa728 against latest (b0020f2)

  🔍 Preview build  

3 files changed
± models/index.html
± api/model/index.html
± api/v3/definition/index.html

…nonicalize branch

Combines the branches of zarr-developers#4642 and zarr-developers#4647 as the
base for a prototype.

Assisted-by: ClaudeCode:claude-fable-5-2
…fines from one normal form

Each of the six models writes its document in its simplest equivalent
spelling in one function. The key that == and hash compare is that
document as JSON text with the claims the scope made, canonicalize reads
it, and refines reads the other model's document in this model's scope
and compares. The per-model key functions, the per-model refines
traversals and the declaration-level refines are removed.

A model no longer refines one whose document its scope reads with a
problem, such as a configuration key that the definition does not
declare under a name the other scope left unclaimed.

Assisted-by: ClaudeCode:claude-fable-5-2
…for every generated document

Carries the tests of zarr-developers#4648 that do not name
codec_refines, which this branch does not have.

Assisted-by: ClaudeCode:claude-fable-5-2
Assisted-by: ClaudeCode:claude-fable-5-2
… from one normal form

Merges refactor/zarr-metadata-one-normal-form, which holds the branches of
zarr-developers#4642 and zarr-developers#4647, and takes its tree: refines
reads the other model's document in this model's scope, so codec_refines
and read_as_claimed are no longer needed and are removed. The changelog
fragment is rewritten for the new behaviour.

Assisted-by: ClaudeCode:claude-fable-5-2
@d-v-b d-v-b changed the title fix(zarr-metadata): compare codecs for the chunk the richer reading knows when judging a refinement fix(zarr-metadata): derive each model's key, canonicalize and refines from one normal form Nov 18, 2026
@codecov

codecov Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.91%. Comparing base (b0020f2) to head (94fa728).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #4648   +/-   ##
=======================================
  Coverage   94.90%   94.91%           
=======================================
  Files          96       96           
  Lines       14055    14070   +15     
=======================================
+ Hits        13339    13354   +15     
  Misses        716      716           

see 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…lk that writes its simplest spelling

canonical_declaration writes a declaration in its simplest spelling and
lets a caller that knows more take part: a pipeline gives the spelling of
each codec it placed, and a step that simplifies the codec's own members
for the chunk it receives. codec_for_chunk returns that spelling as JSON,
so nothing builds a second declaration to read again, and respelled is
removed. canonical is called from one function.

Assisted-by: ClaudeCode:claude-fable-5-2

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs release notes Automatically applied to PRs which haven't added release notes zarr-metadata Specific to the zarr-metadata sub-package

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant