Repository navigation
Conversation
…the chunk it receives A definition's `canonical` reduces two spellings of a configuration to one when they mean the same thing wherever the declaration is used. It receives the configuration alone, so it cannot reduce spellings that mean the same thing only for some chunks. The `endian` of a `bytes` codec is one: it means nothing when each value takes one byte, and is required when a value takes several. `ArrayCodecDefinition` gains `chunk_canonical`, a function of the configuration, the nested declarations and the chunk, and `codec_for_chunk` returns the codec of a `Stage` in that spelling, with each codec of the pipelines it contains simplified for the chunk that one receives. The result is a declaration, so `declaration_key`, `canonical_of` and `refines` work on it as on any other. Its configuration is checked again: the definition must allow it, the chunk rules must find no problem with it, and each nested declaration must stay as it is. `bytes` declares a `chunk_canonical` that drops `endian` for a data type stored in single bytes, and a v3 array model compares, hashes and refines by its codecs as `codec_for_chunk` returns them. So an `int8` array is one array with either `endian` or with none, as zarr-python 3.0 and 3.1 write it, while a `bytes` codec after a `cast_value` to a multi-byte data type is still compared by its `endian`. Assisted-by: ClaudeCode:claude-fable-5-2
…est equivalent spelling Two models are equal when their documents mean the same thing, and `to_json` writes each document as it was given, so equal models can write different documents. A consumer that needs one spelling for all of them, to compare documents field by field or to hash them, had to normalize each member itself. Each model gains `canonicalize`: the v3 and v2 array, group and consolidated metadata models. It returns the model read from its document in its simplest equivalent spelling, written from the parts that `==` compares: each declaration as `canonical_of` writes it, each v3 codec as `codec_for_chunk` returns it for the chunk it receives, and the fill value in the canonical spelling of its data type. Empty `attributes`, empty v3 `storage_transformers` and a v2 `dimension_separator` of "." are left out, since a document without them means the same. Everything else is kept as written. The result equals the model, and equal models in one scope give the same document apart from the order of the members of an object, which a `JSONWriter(sort_keys=True)` sorts. A group keeps the documents its `consolidated_metadata` lists as written, since it does not read them; the consolidated metadata models canonicalize each node they list. Assisted-by: ClaudeCode:claude-fable-5-2
Assisted-by: ClaudeCode:claude-fable-5-2
…data-canonical-documents
Assisted-by: ClaudeCode:claude-fable-5-2
…canonical-documents Main now carries the chunk-aware canonical hook (zarr-developers#4640) and the nested declaration order fix (zarr-developers#4641). The one conflict was in model/_array_v3.py, where this branch adds canonical_array_document_v3 next to codecs_for_chunks; both are kept. Assisted-by: ClaudeCode:claude-fable-5-2
…s canonical gives chunk_canonical received the configuration as written, and canonical ran afterwards. A codec whose canonical output its chunk_canonical reduces further then had two keys for one meaning: array models that differed only in such spellings were unequal, and the simplest spelling was not a fixed point. _for_chunk now hands chunk_canonical the configuration as canonical spells it, and checks that chunk_canonical has nothing left to simplify in its own result. A definition that leaves chunk_canonical at its default is skipped without calling either. Assisted-by: ClaudeCode:claude-fable-5-2
Assisted-by: ClaudeCode:claude-fable-5-2
…at are not the default A review found that the Hypothesis tests passed with a canonicalize that returned the model's own document, that no test noticed a model losing its scope, and that four behaviours had no case: a claimed extra field, a v2 fill value, a chunk grid and a storage transformer that are respelled. The parametrized cases are now read in scopes that are not the default ones. The generated documents are compared with the same document reordered at every depth and, for arrays, with each declaration in its own simplest spelling. The reordering helper reverses every object, which the fix in zarr-developers#4641 allows. Assisted-by: ClaudeCode:claude-fable-5-2
…g claims Its configuration is kept, but its envelope is written in the shortest form: an empty configuration and a must_understand of true are left out. Assisted-by: ClaudeCode:claude-fable-5-2
… document it becomes A review of zarr-developers#4640 found gaps in how _for_chunk checks the result of chunk_canonical: - a result that added a nested declaration passed, and the respelled codec then held a declaration its nested mapping did not; - a result that was not JSON raised an error that did not name the definition; - an array written as a list was reported as a changed nested declaration; - a problem inside a nested declaration, which is the document's, was held against the result. The result is now refined as any JSON is, its declaration locations are compared with the codec's, and problems under a nested location are left out. respelled refuses a location that holds no nested declaration with the TypeError its docstring promises. Assisted-by: ClaudeCode:claude-fable-5-2
…nows when judging a refinement Since zarr-developers#4640 an array model's refines compared each side's codecs as simplified for that side's own chunks. A richer scope can know more of a chunk than a poorer one: it claims the data type, a codec that comes before, or the codec itself. It then dropped a bytes codec's endian that the poorer reading kept, and refined_in(...) no longer refined the model it came from. codec_refines compares the codec of a stage with another reading of it, simplifying both for the chunks the stage knows. A gain is judged by what the unclaimed codec reads as with the stage's claims, simplified the same way. ZarrV3ArrayMetadata.refines uses it for each codec. Assisted-by: ClaudeCode:claude-fable-5-2
Assisted-by: ClaudeCode:claude-fable-5-2
Assisted-by: ClaudeCode:claude-fable-5-2
…metadata-refines-across-scopes
Assisted-by: ClaudeCode:claude-fable-5-2
This was referenced Nov 18, 2026
Documentation build overview
|
…nonicalize branch Combines the branches of zarr-developers#4642 and zarr-developers#4647 as the base for a prototype. Assisted-by: ClaudeCode:claude-fable-5-2
…fines from one normal form Each of the six models writes its document in its simplest equivalent spelling in one function. The key that == and hash compare is that document as JSON text with the claims the scope made, canonicalize reads it, and refines reads the other model's document in this model's scope and compares. The per-model key functions, the per-model refines traversals and the declaration-level refines are removed. A model no longer refines one whose document its scope reads with a problem, such as a configuration key that the definition does not declare under a name the other scope left unclaimed. Assisted-by: ClaudeCode:claude-fable-5-2
…for every generated document Carries the tests of zarr-developers#4648 that do not name codec_refines, which this branch does not have. Assisted-by: ClaudeCode:claude-fable-5-2
Assisted-by: ClaudeCode:claude-fable-5-2
… from one normal form Merges refactor/zarr-metadata-one-normal-form, which holds the branches of zarr-developers#4642 and zarr-developers#4647, and takes its tree: refines reads the other model's document in this model's scope, so codec_refines and read_as_claimed are no longer needed and are removed. The changelog fragment is rewritten for the new behaviour. Assisted-by: ClaudeCode:claude-fable-5-2
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #4648 +/- ##
=======================================
Coverage 94.90% 94.91%
=======================================
Files 96 96
Lines 14055 14070 +15
=======================================
+ Hits 13339 13354 +15
Misses 716 716 🚀 New features to boost your workflow:
|
…lk that writes its simplest spelling canonical_declaration writes a declaration in its simplest spelling and lets a caller that knows more take part: a pipeline gives the spelling of each codec it placed, and a step that simplifies the codec's own members for the chunk it receives. codec_for_chunk returns that spelling as JSON, so nothing builds a second declaration to read again, and respelled is removed. canonical is called from one function. Assisted-by: ClaudeCode:claude-fable-5-2
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖 AI text below 🤖
refineson every model now reads the other model's document in this model's scope and compares the two, so a document read in a richer scope refines the same document read in a poorer one, whatever the richer scope learns of the chunk each codec receives. Before,refined_inof auint8array whosebytescodec writes anendian, read in a scope that claims nobytescodec, did not refine the model it came from. A model no longer refines one whose document its own scope reads with a problem: agzipcodec with a configuration key its definition does not declare, read in a scope that claims nogzip, is not refined by the model of the same codec without that key.codec_for_chunknow writes the codec as JSON in its simplest spelling for the chunk, as one step of the walk thatcanonical_ofmakes, and returnsNonefor a codec with no simplest spelling. A codec'schunk_canonicalreceives each nested declaration in its simplest spelling.Why: one normal form per model
This PR first fixed the cross-scope
refinesregression from #4640 with a new function,codec_refines. That implementation is replaced here, on the same branch, by a design that removes the cause the defects fixed in #4641, #4647 and this PR share.Before this branch, what a document means was decided in several separately written walks over the same tree, kept in agreement by hand: the key that
==andhashcompare (array_key_of, each model's_key_of,extra_fields_key), the canonical document (canonical_array_document_v3and five others), and refinement (sixrefinesmethods, the declaration-levelrefines,_gained,own_key, and, in this PR's first version,codec_refines). Each of the three defects was one walk missing something another walk had.On this branch each model writes its document in its simplest equivalent spelling in one function,
_canonical_document, and everything else is derived inmodel/_keyed.py:==andhashcompare the claims the scope made and the normal form as JSON text.canonicalize()reads the normal form.refines(other)holds when this scope read every name the other scope read, with the same definition, and the other document read in this scope equals this model.Member order means nothing by construction, because JSON text sorts the members of an object. The cross-scope case this PR was opened for holds by construction, because one reader, with one scope's knowledge of the chunks, reads both documents.
The same holds one level down.
canonical_declarationis the one walk that writes a declaration in its simplest spelling, and a pipeline takes part in it instead of walking the codec a second time: it gives the spelling of each codec it placed, and a step that simplifies the codec's own members for the chunk. Socodec_for_chunkreturns JSON, nothing builds a second declaration to read again, andcanonicalis called from one function.What changed in the source
Against #4642 and #4647 merged, the source is 378 lines added and 530 removed.
Removed:
ZarrV3ArrayMetadata._plain_key,ZarrV2ArrayMetadata._plain_keyand_fill_value_key,extra_fields_key,ZarrV2ConsolidatedMetadata._other_entries_text,_node_refines,_v2_node_refines, the body of each model's_key_ofandrefines, and fromdefinition/_scope.pythe declaration-levelrefines,_gainedand_Claimed, withown_keyfromdefinition/_definition.py. None of these was exported from a public module.Also removed:
codec_refinesandread_as_claimed, which this PR's first version added. Neither was released.In the pipeline:
respelledand_insideare removed, with the tests ofrespelled.codec_for_chunkreturnsJSONValue | Nonein place of a declaration. The checks #4647 put on whatchunk_canonicalgives are all kept, in_asked_for_chunkandcanonical_declaration.canonical_declarationno longer reads a configuration again when the definition'scanonicalreturned what it was given. The key now goes through that function, and without this a listing called each definition's rules more than once per declaration.chunk_canonicalnow receives each nested declaration in its simplest spelling, where before it received them as written. Neither #4640 nor #4647 is released.Behaviour difference found
refinesused to judge a gain by reading the unclaimed declaration again and dropping the problems of that read. So the model of{"name": "gzip", "configuration": {"level": 5}}refined a model, read in a scope without a gzip, of the same codec with an extra configuration key.refined_inon that second model raisesMetadataValidationError, so the two methods disagreed. On this branchrefinesreturnsFalsethere.tests/v3/test_scope.pyhas the case asno-gain-over-a-problem.Tests
tests/v3/test_scope.pythat called the removed declaration-levelrefines. Those are ported to models with the same cases. Two cases have no model equivalent and were dropped, because they compare refused declarations and no model holds one.codec_refines: the sixrefined_incases,tests/model/test_invariants.py, and the additions totests/v3/test_definitions.py.==andrefinesanswers as this PR's first version: 2,810 equal pairs and 6,187 refinements.==of two sharded arrays takes 81 µs against 101 µs, a firstrefinesacross scopes 190 µs against 195 µs, and a first==of two listings of 200 arrays 11.0 ms against 15.2 ms.hashof an array whosebytescodec sits inside 80 nested shards takes 1.35 ms, against 19.5 ms before the per-chunk step was folded in: the codec is no longer rebuilt at each level.tests/v3/test_pipelines.pychecks thatchunk_canonicalis asked once, with the configuration fully simplified.What this does not do
declaration_key, next tocanonical_declaration. Models no longer use it, butAcceptedDeclaration.__eq__does. Deriving it from the same walk needs a normal form for refused declarations.Stack
This branch holds the commits of #4642 and #4647, since the design needs
canonicalizefrom the first and the checkedchunk_canonicalfrom the second. Merge #4647 and #4642 first, then mergemaininto this branch, and the diff shows only this PR. Before this change the branch held #4647 alone.The per-chunk step could not be folded in on #4647's own branch: without #4642 a model's key and
refinescompare declarations, socodec_for_chunkhas to return one there. So #4647 addsrespelledand this PR removes it.The design was first tried as d-v-b#465, which is closed in favour of this PR.
🤖 Generated with Claude Code