Repository navigation
Conversation
The indexers group selected items by chunk with an unstable argsort, so repeated indices reached the in-chunk assignment in an arbitrary order and the stored value depended on the platform, the chunk layout and the number of selected items. Sorting stably keeps repeats in the order given, and the in-chunk NumPy assignment then stores the last value. Refs zarr-developers#2855 Assisted-by: ClaudeCode:claude-fable-5.2
Assisted-by: ClaudeCode:claude-fable-5.2
Assisted-by: ClaudeCode:claude-fable-5.2
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #4643 +/- ##
=======================================
Coverage 94.91% 94.91%
=======================================
Files 96 96
Lines 14070 14080 +10
=======================================
+ Hits 13354 13364 +10
Misses 716 716
🚀 New features to boost your workflow:
|
…chunk The stable argsort that keeps repeated indices in order was 2-3x slower than NumPy's default sort for large unordered selections, on reads as well as writes. NumPy sorts integers of at most 16 bits with a stable radix sort, so chunk indices are narrowed to uint16 when there are at most 2**16 chunks. Building an indexer for 10M unordered points then takes about 165-200 ms, against 560-690 ms on main and 1.5-2.0 s with the plain stable sort. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]>
…r chunk The stable sort kept repeated indices in the order given, but the in-chunk assignment still left the stored value to the array library: NumPy does not document the order of repeated assignments and CuPy leaves it undefined. Following zarr-indexing, `_merge_chunk_array` now resolves a chunk selection that names an element more than once into one that names each element once, with the last value given for it in C order. Orthogonal selections are resolved one axis at a time; coordinate selections jointly, in linear time via `np.maximum.at` unless the chunk is much larger than the selection. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]>
…unk order Repeats are now resolved before any codec sees them: `last_writes` wraps the indexer in `_set_selection` and, for each chunk projection, keeps the entry with the highest position in `out_selection` for every element named more than once. This is the last value in the selection's C order, and it does not depend on the order items are listed within a chunk, so the sort that groups items by chunk no longer needs to be stable. Above 2**16 chunks the indexers return to NumPy's default sort, recovering main's indexer speed. The per-chunk merge no longer dedupes, and the sharding codec never receives a repeated index. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]>
…t up to 2**32 chunks Match zarr-indexing (zarr-developers#4649): the value an element receives when an index array names it more than once is unspecified, as in NumPy. This removes `last_writes` and its per-chunk resolution from the write path, and documents the rule in the user guide and the setter docstrings. Chunk indices are now radix-sorted 16 bits at a time up to 2**32 chunks, which also covers arrays with more than 2**16 chunks: building an orthogonal indexer for 10M points over 1M chunks takes 233 ms instead of 817 ms on main. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]>
…lue is unspecified Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]>
d-v-b
added a commit
to d-v-b/zarr-python
that referenced
this pull request
Nov 18, 2026
NumPy's stable sort is a linear-time radix sort for integers of at most 16 bits. `_group_by_key` shifts keys to start at zero and sorts them one 16-bit digit at a time: one pass when their range fits 16 bits, two least-significant-first passes when it fits 32 bits, and the default sort beyond. Grouping 1M random keys over 65,536 chunks drops from about 46 ms to 2 ms. The order within a group stays unspecified. The approach follows zarr-python zarr-developers#4643's `argsort_chunk_indices`, with a shift by the minimum key because grids may have negative chunk IDs. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]>
Sort 16 bits at a time for as many passes as the chunk count needs (up to four), instead of falling back to NumPy's default sort above 2**32 chunks. Building an orthogonal indexer for 10M points over 2**40 chunks takes 342 ms instead of 838 ms. Below 128 items per pass the plain stable sort is faster and is used instead, so the sort is stable for every input. Assisted-by: ClaudeCode:claude-fable-5-2 Co-Authored-By: Claude Fable 5.2 <[email protected]>
d-v-b
added a commit
to d-v-b/zarr-python
that referenced
this pull request
Nov 18, 2026
`_group_by_key` fell back to the default sort above a 2**32 key range, which was slower than radix passes and, as zarr-python zarr-developers#4643 measured, slower at every key width. It now takes one 16-bit pass per 16 bits of the shifted key range, up to four, reading the shifted keys as uint64 so a range above 2**63 stays exact. Inputs under 128 keys per pass take one stable comparison sort, which beats the passes' fixed overhead. This mirrors zarr-developers#4643's `argsort_chunk_indices`; the order within a group stays unspecified. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]>
d-v-b
added a commit
that referenced
this pull request
Nov 18, 2026
…duplicating writes (#4649) * perf(zarr-indexing): group chunk points with an unstable sort and stop deduplicating writes Planning grouped each chunk's points with a stable sort so that every `IndexedSet` and `JointSet` row kept request order, and the writer relied on that order to make the last repeated write win. Both promises are dropped: the order of points within a row is unspecified, and a source cell a write addresses more than once receives one of its values, whole, without saying which. Callers who need a particular value remove repeats from the selection first. Grouping now uses the default sort behind `_group_by_key`, which also skips the sort for already-grouped joint keys, and the writer no longer deduplicates targets before its single fancy assignment or `np.put`. Tests that assumed last-wins now accept any written value, and a new test runs orthogonal, vectorized and mixed repeated writes with the grouping replaced by one that reverses each row. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> * docs(zarr-indexing): add changelog fragment for unspecified repeated-write order Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> * docs(zarr-indexing): number the changelog fragment for upstream PR 4649 Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> * perf(zarr-indexing): group chunk keys with a 16-bit radix sort NumPy's stable sort is a linear-time radix sort for integers of at most 16 bits. `_group_by_key` shifts keys to start at zero and sorts them one 16-bit digit at a time: one pass when their range fits 16 bits, two least-significant-first passes when it fits 32 bits, and the default sort beyond. Grouping 1M random keys over 65,536 chunks drops from about 46 ms to 2 ms. The order within a group stays unspecified. The approach follows zarr-python #4643's `argsort_chunk_indices`, with a shift by the minimum key because grids may have negative chunk IDs. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> * perf(zarr-indexing): radix-sort chunk keys of any width `_group_by_key` fell back to the default sort above a 2**32 key range, which was slower than radix passes and, as zarr-python #4643 measured, slower at every key width. It now takes one 16-bit pass per 16 bits of the shifted key range, up to four, reading the shifted keys as uint64 so a range above 2**63 stays exact. Inputs under 128 keys per pass take one stable comparison sort, which beats the passes' fixed overhead. This mirrors #4643's `argsort_chunk_indices`; the order within a group stays unspecified. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> * docs(zarr-indexing): record why repeated write targets get no promised value Add a "Repeated write targets" design note: the library promises no value for a cell a write addresses more than once, chooses its internal order by performance (today a stable radix sort that happens to keep the last write), carries no order within a planning-table row, and does not deduplicate, because the pass would cost more than the assignment. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> --------- Co-authored-by: Claude Opus 5.5 <[email protected]>
d-v-b
marked this pull request as ready for review
November 18, 2026 20:52
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Use a stable radix sort for permuting index arrays into chunk order. document that the order of degenerate writes is an implementation detail.
🤖 AI text below 🤖
When an
oindex,vindexor__setitem__index array names an element more than once, the value the element receives is now documented as unspecified, as in NumPy. Remove repeated indices if the result matters. Grouping a fancy selection by chunk is also several times faster.Closes #2855
What changed
argsort_chunk_indicesreplaces main'snp.argsortcalls inIntArrayDimIndexerandCoordinateIndexer. It sorts chunk indices with NumPy'suint16radix sort, 16 bits at a time from the lowest bits: one pass up to 2**16 chunks, and up to four passes for larger chunk counts. Below 128 items per pass, NumPy's plain stable sort is faster and is used instead.set_orthogonal_selectionandset_coordinate_selectiondocstrings state the rule.test_setitem_repeated_index, skipped on main, now runs.Why no ordering promise
This follows zarr-indexing's rule in #4649: repeated writes promise nothing, which matches NumPy's documented stance on advanced assignment ("if an element is set more than once, it is not possible to predict the final result"), CuPy (undefined) and JAX (implementation-defined). Earlier commits on this branch promised last-wins through a
last_writesstep that resolved repeats per chunk projection before the codec pipeline. That step cost about 1% onoindexwrites and 14-19% (about 5 ms per million points) onvindexwrites, and the author decided against the promise.Every pass of the radix sort is stable, so on CPU NumPy items that share a chunk keep their order, and in practice the result is what NumPy gives (on main, 16 or more repeats in one chunk could give an earlier value). That order is an implementation detail and is not documented as a promise. The author's rule is speed first, then consistency: the stable sort is kept because it is also the fastest. The one exception is a selection of roughly 128 to 2048 items over more than 2**16 chunks, where NumPy's default (unstable) sort would build the indexer up to 6 µs sooner. With an array library that orders repeated assignments differently, the result can differ. GPU buffers were not tested.
Performance
Building an indexer for 10M unordered points (best of 3, Apple Silicon):
oindex, 1k chunksvindex, 10k chunksoindex, 1M chunksoindex, 2**40 chunksSorting 10M chunk indices alone: at 1M chunks the default sort takes 687 ms and the two-pass radix sort 99 ms.
Relation to #4601
#4601 makes repeated writes last-wins by removing duplicates from the whole selection before indexing. With the rule in this PR, the stored value for a repeated index is unspecified, so that deduplication is not needed.
Tests
test_setitem_repeated_indexcoversoindexandvindex, four chunk layouts (including sharded) and four selections. Each case also runs with the chunk sort replaced by one that reverses each chunk's items. It checks that unaddressed elements are untouched, elements addressed once get their value, and repeated elements get one of their values.test_argsort_chunk_indiceschecks that the sort matches NumPy's stable sort for 1 to 2**70 chunks, with 100 and 2000 items.test_oindexandtest_vindexkeep skipping repeated indices, as on main, since they compare against NumPy.vindex: 2 failures;oindex: 14 failures;🤖 Generated with Claude Code