Skip to content

perf(indexing): radix-sort chunk indices and document repeated-index writes as unspecified - #4643

Open
d-v-b wants to merge 11 commits into
zarr-developers:mainfrom
d-v-b:fix/stable-chunk-grouping
Open

d-v-b wants to merge 11 commits into
zarr-developers:mainfrom
d-v-b:fix/stable-chunk-grouping

Conversation

@d-v-b

@d-v-b d-v-b commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Use a stable radix sort for permuting index arrays into chunk order. document that the order of degenerate writes is an implementation detail.

🤖 AI text below 🤖

When an oindex, vindex or __setitem__ index array names an element more than once, the value the element receives is now documented as unspecified, as in NumPy. Remove repeated indices if the result matters. Grouping a fancy selection by chunk is also several times faster.

Closes #2855

What changed
  • A new argsort_chunk_indices replaces main's np.argsort calls in IntArrayDimIndexer and CoordinateIndexer. It sorts chunk indices with NumPy's uint16 radix sort, 16 bits at a time from the lowest bits: one pass up to 2**16 chunks, and up to four passes for larger chunk counts. Below 128 items per pass, NumPy's plain stable sort is faster and is used instead.
  • The user guide gains a "Repeated indices" section, and the set_orthogonal_selection and set_coordinate_selection docstrings state the rule.
  • test_setitem_repeated_index, skipped on main, now runs.
Why no ordering promise

This follows zarr-indexing's rule in #4649: repeated writes promise nothing, which matches NumPy's documented stance on advanced assignment ("if an element is set more than once, it is not possible to predict the final result"), CuPy (undefined) and JAX (implementation-defined). Earlier commits on this branch promised last-wins through a last_writes step that resolved repeats per chunk projection before the codec pipeline. That step cost about 1% on oindex writes and 14-19% (about 5 ms per million points) on vindex writes, and the author decided against the promise.

Every pass of the radix sort is stable, so on CPU NumPy items that share a chunk keep their order, and in practice the result is what NumPy gives (on main, 16 or more repeats in one chunk could give an earlier value). That order is an implementation detail and is not documented as a promise. The author's rule is speed first, then consistency: the stable sort is kept because it is also the fastest. The one exception is a selection of roughly 128 to 2048 items over more than 2**16 chunks, where NumPy's default (unstable) sort would build the indexer up to 6 µs sooner. With an array library that orders repeated assignments differently, the result can differ. GPU buffers were not tested.

Performance

Building an indexer for 10M unordered points (best of 3, Apple Silicon):

case this PR main
oindex, 1k chunks 162 ms 562 ms
vindex, 10k chunks 191 ms 688 ms
oindex, 1M chunks 230 ms 817 ms
oindex, 2**40 chunks 342 ms 838 ms

Sorting 10M chunk indices alone: at 1M chunks the default sort takes 687 ms and the two-pass radix sort 99 ms.

Relation to #4601

#4601 makes repeated writes last-wins by removing duplicates from the whole selection before indexing. With the rule in this PR, the stored value for a repeated index is unspecified, so that deduplication is not needed.

Tests
  • test_setitem_repeated_index covers oindex and vindex, four chunk layouts (including sharded) and four selections. Each case also runs with the chunk sort replaced by one that reverses each chunk's items. It checks that unaddressed elements are untouched, elements addressed once get their value, and repeated elements get one of their values.
  • test_argsort_chunk_indices checks that the sort matches NumPy's stable sort for 1 to 2**70 chunks, with 100 and 2000 items.
  • The property tests test_oindex and test_vindex keep skipping repeated indices, as on main, since they compare against NumPy.
  • Mutation checks (run by hand, then reverted), each failing the tests above:
    • reversing the output positions within a chunk for vindex: 2 failures;
    • reversing them for oindex: 14 failures;
    • too few radix passes: 4 failures;
    • an unstable later pass: 7 failures;
    • an unstable sort for few items: 10 failures;
    • taking a later pass's digits from the unsorted indices: 6 failures.
  • Full suite: 13436 passed. pre-commit and mypy pass.

🤖 Generated with Claude Code

The indexers group selected items by chunk with an unstable argsort, so
repeated indices reached the in-chunk assignment in an arbitrary order
and the stored value depended on the platform, the chunk layout and the
number of selected items. Sorting stably keeps repeats in the order
given, and the in-chunk NumPy assignment then stores the last value.

Refs zarr-developers#2855

Assisted-by: ClaudeCode:claude-fable-5.2
@codecov

codecov Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.91%. Comparing base (caa1536) to head (7610385).

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #4643   +/-   ##
=======================================
  Coverage   94.91%   94.91%           
=======================================
  Files          96       96           
  Lines       14070    14080   +10     
=======================================
+ Hits        13354    13364   +10     
  Misses        716      716           
Files with missing lines Coverage Δ
src/zarr/core/array.py 98.60% <ø> (ø)
src/zarr/core/indexing.py 96.61% <100.00%> (+0.03%) ⬆️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

d-v-b and others added 3 commits November 18, 2026 12:38
…chunk

The stable argsort that keeps repeated indices in order was 2-3x slower than
NumPy's default sort for large unordered selections, on reads as well as
writes. NumPy sorts integers of at most 16 bits with a stable radix sort, so
chunk indices are narrowed to uint16 when there are at most 2**16 chunks.
Building an indexer for 10M unordered points then takes about 165-200 ms,
against 560-690 ms on main and 1.5-2.0 s with the plain stable sort.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
…r chunk

The stable sort kept repeated indices in the order given, but the in-chunk
assignment still left the stored value to the array library: NumPy does not
document the order of repeated assignments and CuPy leaves it undefined.

Following zarr-indexing, `_merge_chunk_array` now resolves a chunk selection
that names an element more than once into one that names each element once,
with the last value given for it in C order. Orthogonal selections are
resolved one axis at a time; coordinate selections jointly, in linear time
via `np.maximum.at` unless the chunk is much larger than the selection.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
…unk order

Repeats are now resolved before any codec sees them: `last_writes` wraps the
indexer in `_set_selection` and, for each chunk projection, keeps the entry
with the highest position in `out_selection` for every element named more
than once. This is the last value in the selection's C order, and it does
not depend on the order items are listed within a chunk, so the sort that
groups items by chunk no longer needs to be stable. Above 2**16 chunks the
indexers return to NumPy's default sort, recovering main's indexer speed.

The per-chunk merge no longer dedupes, and the sharding codec never receives
a repeated index.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
d-v-b and others added 2 commits November 18, 2026 19:29
…t up to 2**32 chunks

Match zarr-indexing (zarr-developers#4649): the value an element receives when an index
array names it more than once is unspecified, as in NumPy. This removes
`last_writes` and its per-chunk resolution from the write path, and
documents the rule in the user guide and the setter docstrings.

Chunk indices are now radix-sorted 16 bits at a time up to 2**32 chunks,
which also covers arrays with more than 2**16 chunks: building an
orthogonal indexer for 10M points over 1M chunks takes 233 ms instead of
817 ms on main.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
…lue is unspecified

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
@d-v-b d-v-b changed the title fix(indexing): keep the last value when setitem gets repeated indices perf(indexing): radix-sort chunk indices and document repeated-index writes as unspecified Nov 18, 2026
d-v-b added a commit to d-v-b/zarr-python that referenced this pull request Nov 18, 2026
NumPy's stable sort is a linear-time radix sort for integers of at most
16 bits. `_group_by_key` shifts keys to start at zero and sorts them one
16-bit digit at a time: one pass when their range fits 16 bits, two
least-significant-first passes when it fits 32 bits, and the default
sort beyond. Grouping 1M random keys over 65,536 chunks drops from about
46 ms to 2 ms. The order within a group stays unspecified.

The approach follows zarr-python zarr-developers#4643's `argsort_chunk_indices`, with a
shift by the minimum key because grids may have negative chunk IDs.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Sort 16 bits at a time for as many passes as the chunk count needs (up
to four), instead of falling back to NumPy's default sort above 2**32
chunks. Building an orthogonal indexer for 10M points over 2**40 chunks
takes 342 ms instead of 838 ms. Below 128 items per pass the plain
stable sort is faster and is used instead, so the sort is stable for
every input.

Assisted-by: ClaudeCode:claude-fable-5-2
Co-Authored-By: Claude Fable 5.2 <[email protected]>
d-v-b added a commit to d-v-b/zarr-python that referenced this pull request Nov 18, 2026
`_group_by_key` fell back to the default sort above a 2**32 key range,
which was slower than radix passes and, as zarr-python zarr-developers#4643 measured,
slower at every key width. It now takes one 16-bit pass per 16 bits of
the shifted key range, up to four, reading the shifted keys as uint64 so
a range above 2**63 stays exact. Inputs under 128 keys per pass take one
stable comparison sort, which beats the passes' fixed overhead. This
mirrors zarr-developers#4643's `argsort_chunk_indices`; the order within a group stays
unspecified.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
d-v-b added a commit that referenced this pull request Nov 18, 2026
…duplicating writes (#4649)

* perf(zarr-indexing): group chunk points with an unstable sort and stop deduplicating writes

Planning grouped each chunk's points with a stable sort so that every
`IndexedSet` and `JointSet` row kept request order, and the writer relied
on that order to make the last repeated write win. Both promises are
dropped: the order of points within a row is unspecified, and a source
cell a write addresses more than once receives one of its values, whole,
without saying which. Callers who need a particular value remove repeats
from the selection first.

Grouping now uses the default sort behind `_group_by_key`, which also
skips the sort for already-grouped joint keys, and the writer no longer
deduplicates targets before its single fancy assignment or `np.put`.

Tests that assumed last-wins now accept any written value, and a new test
runs orthogonal, vectorized and mixed repeated writes with the grouping
replaced by one that reverses each row.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

* docs(zarr-indexing): add changelog fragment for unspecified repeated-write order

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

* docs(zarr-indexing): number the changelog fragment for upstream PR 4649

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

* perf(zarr-indexing): group chunk keys with a 16-bit radix sort

NumPy's stable sort is a linear-time radix sort for integers of at most
16 bits. `_group_by_key` shifts keys to start at zero and sorts them one
16-bit digit at a time: one pass when their range fits 16 bits, two
least-significant-first passes when it fits 32 bits, and the default
sort beyond. Grouping 1M random keys over 65,536 chunks drops from about
46 ms to 2 ms. The order within a group stays unspecified.

The approach follows zarr-python #4643's `argsort_chunk_indices`, with a
shift by the minimum key because grids may have negative chunk IDs.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

* perf(zarr-indexing): radix-sort chunk keys of any width

`_group_by_key` fell back to the default sort above a 2**32 key range,
which was slower than radix passes and, as zarr-python #4643 measured,
slower at every key width. It now takes one 16-bit pass per 16 bits of
the shifted key range, up to four, reading the shifted keys as uint64 so
a range above 2**63 stays exact. Inputs under 128 keys per pass take one
stable comparison sort, which beats the passes' fixed overhead. This
mirrors #4643's `argsort_chunk_indices`; the order within a group stays
unspecified.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

* docs(zarr-indexing): record why repeated write targets get no promised value

Add a "Repeated write targets" design note: the library promises no
value for a cell a write addresses more than once, chooses its internal
order by performance (today a stable radix sort that happens to keep the
last write), carries no order within a planning-table row, and does not
deduplicate, because the pass would cost more than the assignment.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

---------

Co-authored-by: Claude Opus 5.5 <[email protected]>
@d-v-b
d-v-b marked this pull request as ready for review November 18, 2026 20:52
@d-v-b d-v-b added the benchmark Code will be benchmarked in a CI job. label Nov 18, 2026
@codspeed

codspeed Bot commented Nov 18, 2026

Copy link
Copy Markdown

Merging this PR will regress 2 benchmarks

⚡ 8 improved benchmarks
❌ 2 regressed benchmarks
✅ 103 untouched benchmarks
⏩ 37 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Benchmark BASE HEAD Efficiency
❌ test_slice_indexing[None-(slice(None, None, None), slice(0, 3, 2), slice(0, 10, None))-memory_get_latency] 3.1 ms 4.2 ms -25.68%
❌ test_read_array[latency=0.03-batched-memory-Layout(shape=(100000000,), chunks=(100000,), shards=(100000000,))-None-repeated] 428.2 ms 490.2 ms -12.66%
⚡ test_slice_indexing[None-(slice(0, None, 4), slice(0, None, 4), slice(0, None, 4))-memory] 350.2 ms 299.2 ms +17.05%
⚡ test_slice_indexing[None-(slice(10, -10, 4), slice(10, -10, 4), slice(10, -10, 4))-memory] 191.7 ms 163.8 ms +17.02%
⚡ test_slice_indexing[None-(slice(10, -10, 4), slice(10, -10, 4), slice(10, -10, 4))-memory_get_latency] 223.1 ms 191.8 ms +16.32%
⚡ test_slice_indexing[None-(slice(0, None, 4), slice(0, None, 4), slice(0, None, 4))-memory_get_latency] 408.5 ms 352.2 ms +15.97%
⚡ test_slice_indexing[(50, 50, 50)-(slice(0, None, 4), slice(0, None, 4), slice(0, None, 4))-memory_get_latency] 411.8 ms 357.7 ms +15.12%
⚡ test_slice_indexing[(50, 50, 50)-(slice(10, -10, 4), slice(10, -10, 4), slice(10, -10, 4))-memory_get_latency] 214.7 ms 186.9 ms +14.86%
⚡ test_slice_indexing[(50, 50, 50)-(slice(0, None, 4), slice(0, None, 4), slice(0, None, 4))-memory] 406.9 ms 354.4 ms +14.83%
⚡ test_slice_indexing[(50, 50, 50)-(slice(10, -10, 4), slice(10, -10, 4), slice(10, -10, 4))-memory] 215.2 ms 187.5 ms +14.76%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing d-v-b:fix/stable-chunk-grouping (31e9588) with main (d8b8543)2

Open in CodSpeed

Footnotes

  1. 37 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

  2. No successful run was found on main (6eb4563) during the generation of this report, so d8b8543 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

@d-v-b d-v-b removed the benchmark Code will be benchmarked in a CI job. label Nov 18, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

setitem with repeated indices is not guaranteed in practice

1 participant