Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
e3e1c42
feat(zarr-metadata): compare each codec in its simplest spelling for …
d-v-b Nov 16, 2026
1be8677
feat(zarr-metadata): return each model with its document in its simpl…
d-v-b Nov 16, 2026
070840f
docs(zarr-metadata): name the changelog fragment after the upstream PR
d-v-b Nov 16, 2026
ac649e0
Merge branch 'feat/zarr-metadata-chunk-canonical' into feat/zarr-meta…
d-v-b Nov 16, 2026
fa27e38
docs(zarr-metadata): name the changelog fragment after the upstream PR
d-v-b Nov 16, 2026
c29f15c
Merge remote-tracking branch 'upstream/main' into feat/zarr-metadata-…
d-v-b Nov 17, 2026
6af625b
fix(zarr-metadata): simplify a codec for its chunk in the spelling it…
d-v-b Nov 17, 2026
c5f7a4b
docs(zarr-metadata): add the changelog fragment
d-v-b Nov 17, 2026
64c89a7
test(zarr-metadata): check what canonicalize simplifies, in scopes th…
d-v-b Nov 17, 2026
34c3378
docs(zarr-metadata): say how canonicalize writes a declaration nothin…
d-v-b Nov 17, 2026
cb67d4b
fix(zarr-metadata): check what a codec's chunk_canonical gives as the…
d-v-b Nov 17, 2026
f1e40f6
fix(zarr-metadata): compare codecs for the chunk the richer reading k…
d-v-b Nov 17, 2026
e1cad28
docs(zarr-metadata): add the changelog fragment
d-v-b Nov 17, 2026
ab8ec2e
docs(zarr-metadata): name the changelog fragment after the upstream PR
d-v-b Nov 18, 2026
e756ffc
Merge branch 'fix/zarr-metadata-chunk-canonical-order' into fix/zarr-…
d-v-b Nov 18, 2026
4cee555
docs(zarr-metadata): name the changelog fragment after the upstream PR
d-v-b Nov 18, 2026
201687d
chore(zarr-metadata): merge the chunk_canonical order fix into the ca…
d-v-b Nov 18, 2026
b2a1c3d
refactor(zarr-metadata): derive each model's key, canonicalize and re…
d-v-b Nov 18, 2026
37141df
test(zarr-metadata): check refinement across scopes and member order …
d-v-b Nov 18, 2026
6f86bac
docs(zarr-metadata): add the changelog fragment
d-v-b Nov 18, 2026
5d662c4
fix(zarr-metadata): derive each model's key, canonicalize and refines…
d-v-b Nov 18, 2026
94fa728
refactor(zarr-metadata): simplify a codec for its chunk inside the wa…
d-v-b Nov 18, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions packages/zarr-metadata/changes/4642.feature.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Each model has a `canonicalize` method that returns the model with its document in its simplest equivalent spelling: `ZarrV3ArrayMetadata`, `ZarrV3GroupMetadata`, `ZarrV3ConsolidatedMetadata`, `ZarrV2ArrayMetadata`, `ZarrV2GroupMetadata` and `ZarrV2ConsolidatedMetadata`. The result equals the model. Every model equal to it, in the same scope, gives the same document, apart from the order of the members of an object, so `to_key_value` with `JSONWriter(sort_keys=True)` writes the same bytes for equal models. The document is written from the parts `==` compares: each declaration as `canonical_of` writes it, each v3 codec as `codec_for_chunk` returns it for the chunk it receives, and the fill value as `canonical_fill_value` writes it. Empty `attributes`, empty v3 `storage_transformers` and a v2 `dimension_separator` of `"."` are left out, and everything else is kept as written.
1 change: 1 addition & 0 deletions packages/zarr-metadata/changes/4647.bugfix.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
A v3 codec's `chunk_canonical` receives the configuration as the codec's `canonical` spells it, so two spellings that `canonical` reduces to one are simplified for a chunk in the same way. Before, it received the configuration as written, and array models that differed only in such spellings were unequal. Its result is checked more closely: one that it would simplify again, or that holds a declaration the codec did not hold, raises `ValueError`, and one that is not JSON raises `TypeError` naming the definition. A result may write an array as a list, and a problem inside a nested declaration is no longer held against it. A codec that leaves `chunk_canonical` at its default is no longer copied each time an array model is compared or hashed.
1 change: 1 addition & 0 deletions packages/zarr-metadata/changes/4648.bugfix.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
`refines` on every model now reads the other model's document in this model's scope and compares the two, so a document read in a richer scope refines the same document read in a poorer one, whatever the richer scope learns of the chunk each codec receives. Before, `refined_in` of a `uint8` array whose `bytes` codec writes an `endian`, read in a scope that claims no `bytes` codec, did not refine the model it came from. A model no longer refines one whose document its own scope reads with a problem: a `gzip` codec with a configuration key its definition does not declare, read in a scope that claims no `gzip`, is not refined by the model of the same codec without that key. `codec_for_chunk` now writes the codec as JSON in its simplest spelling for the chunk, as one step of the walk that `canonical_of` makes, and returns `None` for a codec with no simplest spelling. A codec's `chunk_canonical` receives each nested declaration in its simplest spelling.
81 changes: 80 additions & 1 deletion packages/zarr-metadata/docs/models.md
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,8 @@ To change a model, use one of these methods:
- `refined_in` also reads the document in another context. The new context
must claim what this context left unclaimed and contradict nothing.
Otherwise it raises `ScopeConflictError`.
- `canonicalize` returns the model with its document in its simplest
equivalent spelling. See [Canonical documents](#canonical-documents).

## Writing store bytes

Expand Down Expand Up @@ -226,7 +228,9 @@ These are each declaration, and the fill value read against its data type.
So in a `uint8` array, a `bytes` codec is the same codec with either
`endian`, or with none. The codec is compared for the chunk it
receives: after a `cast_value` to `float32`, the two `endian` values
are two codecs.
are two codecs. `codec_for_chunk` writes a codec in the spelling that
is compared, as JSON, which `canonical_of` cannot give, since it reads
the codec alone.

A v2 document's
[dtype](https://zarr-specs.readthedocs.io/en/latest/v2/v2.0.html#data-type-encoding),
Expand Down Expand Up @@ -264,3 +268,78 @@ equal to itself.

Equal models have equal hashes. They may still write different documents,
because `to_json` writes each document as it was given.

## Canonical documents

`canonicalize` returns a model with its document in its simplest
equivalent spelling. The result equals the model it came from. Every model
equal to that one, read in the same context, gives the same document.

```python test="true" session="models"
from zarr_metadata.model import ZarrV3ArrayMetadata

written = ZarrV3ArrayMetadata.from_json(
{
"zarr_format": 3,
"node_type": "array",
"shape": [4],
"data_type": "uint8",
"chunk_grid": {"name": "regular", "configuration": {"chunk_shape": [4]}},
"chunk_key_encoding": {"name": "default", "configuration": {"separator": "/"}},
"fill_value": 0,
"codecs": [
{"name": "bytes", "configuration": {"endian": "little"}},
{"name": "zstd", "configuration": {"level": 1, "checksum": False}},
],
"attributes": {},
}
)
canonical = written.canonicalize()
assert canonical == written
print(canonical.to_json()["chunk_key_encoding"])
#> {'name': 'default'}
print(canonical.to_json()["codecs"])
#> ({'name': 'bytes'}, {'name': 'zstd', 'configuration': {'level': 1}})
print("attributes" in canonical.to_json())
#> False
```

The canonical document is written from the same parts that `==` compares:

- Each declaration is written as `canonical_of` writes it.
- Each v3 codec is written for the chunk it receives, as `codec_for_chunk`
writes it. Above, each `uint8` value takes one byte, so the `endian`
says nothing.
- The fill value is written as `canonical_fill_value` writes it for the
data type.
- A member that says what its absence says is left out. These are empty
`attributes`, empty v3 `storage_transformers`, and a v2
`dimension_separator` of `"."`.
- Everything else is kept as written. This includes the attributes, and
the configuration of a declaration that nothing in the context claims,
because only its own definition could simplify it. Such a declaration
is still written in the shortest form every reader accepts, without an
empty `configuration` or a `must_understand` of `true`.

A v2 codec is written with each numcodecs default it leaves out, as its
definition's `canonical` writes it. So a canonical v2 document can be
longer than the one it came from.

The order of the members of an object is not part of the spelling, because
JSON gives it no meaning. Two equal models can give canonical documents
whose members are in different orders. A `JSONWriter` that sorts them
writes the same bytes for both, so those bytes are a key for what the
document means:

```python test="true" session="models"
import hashlib

stored = canonical.to_key_value({"sort_keys": True})["zarr.json"]
digest = hashlib.sha256(stored).hexdigest()
```

The other models work the same way. A v3 group does not read the documents
its `consolidated_metadata` lists, so its `canonicalize` keeps them as
written. `ZarrV3ConsolidatedMetadata.canonicalize` and
`ZarrV2ConsolidatedMetadata.canonicalize` write each node they list in its
simplest spelling.
151 changes: 90 additions & 61 deletions packages/zarr-metadata/src/zarr_metadata/definition/_definition.py
Original file line number Diff line number Diff line change
Expand Up @@ -1200,16 +1200,13 @@ def declaration_key(declaration: ResolvedDeclaration[Any]) -> tuple[object, ...]
configuration: JSONValue = dict(declaration.configuration)
# Each nested declaration is compared by its own key, so its location is
# cleared before the definition's `canonical` sees the rest, in the
# same order as `_canonical_declaration`. `canonical` simplifies only the
# same order as `canonical_declaration`. `canonical` simplifies only the
# definition's own members.
for loc in declaration.nested:
configuration = _replaced(configuration, loc, None)
view = read_only(cast("Mapping[str, JSONValue]", configuration))
spelled = asked(
definition,
"canonical",
lambda: json_text(cast("JSONValue", dict(definition.canonical(view)))),
)
_, own = _spelled(definition, view)
spelled = asked(definition, "canonical", lambda: json_text(cast("JSONValue", own)))
return ("read", definition, spelled, _nested_key(declaration.nested))
if isinstance(declaration, UnclaimedDeclaration):
return (
Expand All @@ -1228,11 +1225,6 @@ def declaration_key(declaration: ResolvedDeclaration[Any]) -> tuple[object, ...]
)


def own_key(declaration: AcceptedDeclaration[Any]) -> tuple[object, ...]:
"""The part of `declaration_key` for an accepted declaration that excludes nested declarations: the definition and the canonical spelling of its own members."""
return declaration_key(declaration)[:3]


def _nested_key(nested: Nested) -> tuple[tuple[Loc, tuple[object, ...]], ...]:
"""The key of each nested declaration, paired with its location, in the order of the locations.

Expand Down Expand Up @@ -1277,36 +1269,9 @@ def written_json(declaration: AcceptedDeclaration[Any] | UnclaimedDeclaration) -
return kind._envelope_json(declaration.name, declaration.configuration)


def respelled(
declaration: AcceptedDeclaration[D],
configuration: Mapping[str, JSONValue],
nested: Nested,
) -> AcceptedDeclaration[D]:
"""`declaration` with `configuration` as its configuration, and each declaration in `nested` in place of the nested declaration at its location.

This builds the declaration that another spelling of `declaration`
reads as, without reading it again. The caller answers for the
spelling: `configuration` must be one that the definition allows and
that means the same thing, and it must keep each nested declaration
where it is. `nested` holds only the nested declarations to replace,
each one accepted or unclaimed, and each is written into the
configuration as its `to_json` writes it. A location in `nested` that
does not lead into `configuration` raises `TypeError`. The result is
written as `to_json` writes it, since no document wrote it.
"""
spelled: JSONValue = dict(configuration)
for loc, inner in nested.items():
spelled = _replaced(spelled, loc, written_json(held(inner)))
kind = declaration.read_as
name = declaration.name
carried = kind._filed_under(name)[1] is not None
return AcceptedDeclaration(
json=kind._envelope_json(name, {} if carried else cast("Mapping[str, JSONValue]", spelled)),
name=name,
definition=declaration.definition,
configuration=cast("Mapping[str, JSONValue]", spelled),
nested={**declaration.nested, **nested},
)
def declaration_locations(definition: Definition[Any], configuration: JSONValue) -> tuple[Loc, ...]:
"""The location of each declaration that `configuration` holds, where `definition`'s TypedDict types a member with a declaration alias, in order. `configuration` is JSON with arrays as tuples."""
return tuple(found.loc for found in _typed(definition.configuration, configuration, ())[2])


def item_at(value: JSONValue, path: Loc) -> JSONValue | UNSET:
Expand Down Expand Up @@ -1888,41 +1853,104 @@ def canonical_of(
def _simplest(declaration: ResolvedDeclaration[Any]) -> JSONValue | None:
"""A declaration in its simplest spelling; None when the declaration or one of its nested declarations was refused, because a refused declaration has no simplest spelling."""
if isinstance(declaration, AcceptedDeclaration):
return _canonical_declaration(declaration)
return canonical_declaration(declaration)
if isinstance(declaration, UnclaimedDeclaration):
return declaration.to_json()
return None


def _canonical_declaration(resolved: AcceptedDeclaration[Any]) -> JSONValue | None:
"""An accepted declaration in its simplest equivalent spelling: nested declarations first, then its own members. None when a nested declaration was refused."""
@dataclass(frozen=True, slots=True)
class Further:
"""A simplification of a definition's own members by something `canonical` cannot know, such as the chunk a codec receives.

`ask` receives the configuration in its simplest spelling so far,
read-only: its own members as `canonical` spells them, and each
nested declaration in its own simplest spelling. It returns a simpler
configuration, as JSON with arrays as tuples, or None when it
simplifies nothing. It answers for its result: it raises when the
result does not hold. `what` names the function of the definition
that `ask` calls, for the error raised when its result is not the
simplest spelling.
"""

what: str
ask: Callable[[Mapping[str, JSONValue]], dict[str, JSONValue] | None]


def canonical_declaration(
resolved: AcceptedDeclaration[Any],
spelled: Mapping[Loc, JSONValue | None] | None = None,
further: Further | None = None,
) -> JSONValue | None:
"""An accepted declaration in its simplest equivalent spelling: nested declarations first, then its own members. None when a nested declaration has no simplest spelling.

This is the one walk that writes a declaration in its simplest
spelling. A caller that knows more than the declaration says, as a
pipeline knows the chunk each codec receives, takes part in it
instead of walking the declaration again. `spelled` gives the
simplest spelling of the nested declarations the caller wrote
itself, by location, with None for one that has none. Each other
nested declaration is written as this function writes it. `further`
simplifies the declaration's own members once more, after
`canonical`. What it gives is spelled by `canonical` again, and must
then be the simplest spelling: asked about it, `further` must
simplify nothing, or this raises `ValueError`, which indicates a bug
in the definition.
"""
definition, name = resolved.definition, resolved.name
configuration: JSONValue = dict(resolved.configuration)
for loc, inner in resolved.nested.items():
simplest = _simplest(inner)
simplest = spelled[loc] if spelled is not None and loc in spelled else _simplest(inner)
if simplest is None:
return None
configuration = _replaced(configuration, loc, simplest)
# Every function of a definition receives this read-only view. The
# result of `canonical` (the view itself when nothing is simplified) is
# converted to a dict.
view = read_only(cast("Mapping[str, JSONValue]", configuration))
simplified = asked(
definition,
"canonical",
lambda: dict(cast("Mapping[str, JSONValue]", definition.canonical(view))),
)
_, refused = definition.read_configuration(simplified)
simplified = _spelled_and_checked(definition, cast("Mapping[str, JSONValue]", configuration))
if further is not None:
given = further.ask(read_only(simplified))
if given is not None:
simplified = _spelled_and_checked(definition, given)
left = further.ask(read_only(simplified))
if left is not None:
# The simplest spelling has nothing left to simplify. Otherwise
# a document written from it would read as another declaration.
msg = (
f"{definition.name!r}: its {further.what} gave {given!r}, which is not the "
f"simplest spelling: asked again, it gave {left!r}"
)
raise ValueError(msg)
carrying = definition._carrying_name(simplified)
if carrying is not None:
return carrying
return resolved.read_as._envelope_json(name, simplified)


def _spelled_and_checked(
definition: Definition[Any], configuration: Mapping[str, JSONValue]
) -> dict[str, JSONValue]:
"""`configuration` as `definition`'s `canonical` spells it. A result that the definition does not allow raises `ValueError`, which indicates a bug in the definition."""
kept, simplified = _spelled(definition, read_only(configuration))
# A `canonical` that returns what it received simplified nothing, so
# there is nothing of its own to check.
refused = () if kept else definition.read_configuration(simplified)[1]
if len(refused) != 0:
msg = (
f"{definition.name!r}: its canonical gave {simplified!r}, which does not hold: "
f"{list(refused)!r}"
)
raise ValueError(msg)
carrying = definition._carrying_name(simplified)
if carrying is not None:
return carrying
return resolved.read_as._envelope_json(name, simplified)
return simplified


def _spelled(
definition: Definition[Any], view: Mapping[str, JSONValue]
) -> tuple[bool, dict[str, JSONValue]]:
"""Whether `definition`'s `canonical` returned `view` itself, and what it returned, as a dict. This is the one place that calls `canonical`. Every function of a definition receives a read-only view."""

def spelled() -> tuple[bool, dict[str, JSONValue]]:
given = cast("Mapping[str, JSONValue]", definition.canonical(view))
return given is view, dict(given)

return asked(definition, "canonical", spelled)


def _replaced(value: JSONValue, path: Loc, new: JSONValue) -> JSONValue:
Expand All @@ -1943,6 +1971,7 @@ def _replaced(value: JSONValue, path: Loc, new: JSONValue) -> JSONValue:
"C",
"Context",
"Definition",
"Further",
"Nested",
"Problems",
"RefusedDeclaration",
Expand All @@ -1951,12 +1980,14 @@ def _replaced(value: JSONValue, path: Loc, new: JSONValue) -> JSONValue:
"WithFillValue",
"as_kind",
"asked",
"canonical_declaration",
"canonical_fill_value",
"canonical_of",
"canonicalize",
"configuration_of",
"declaration_key",
"declaration_kind",
"declaration_locations",
"declarations_of",
"document_json",
"filed_under",
Expand All @@ -1970,11 +2001,9 @@ def _replaced(value: JSONValue, path: Loc, new: JSONValue) -> JSONValue:
"kind_of",
"named_configuration",
"no_rules",
"own_key",
"read_only",
"resolve",
"resolve_refined",
"respelled",
"ruled",
"spelled_canonically",
"unchanged",
Expand Down
Loading
Loading