Skip to content

feat: narrow partial reads of uncompressed chunks within a single row - #4611

Merged
d-v-b merged 5 commits into
zarr-developers:mainfrom
bendichter:partial-decode-single-row
Nov 18, 2026
Merged

d-v-b merged 5 commits into
zarr-developers:mainfrom
bendichter:partial-decode-single-row

Conversation

@bendichter

@bendichter bendichter commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

When a selection touches one index along the first axis, apply the same row window along the next axis, and so on. arr[t, y0:y1, x0:x1] then fetches rows y0 to y1 of frame t instead of the whole frame, and a single item is fetched for a point.

The fetched block keeps the narrowed axes with length 1, so the shifted selection applies to it unchanged.

Summary

We have use-cases such as reading a small patch of a single frame of an image series, where it's possible and useful to extend the partial decode logic to inner dimensions. This identifies that specific indexing pattern and extends the current partial decode logic to efficiently read bytes in this case.

Author attestation

  • I am a human, these are my changes, and I have reviewed and understood every change and can explain why each is correct.

TODO

  • Add unit tests and/or doctests in docstrings
  • Add docstrings and API docs for any new/modified user-facing classes and functions
  • New/modified features documented in docs/user-guide/*.md
  • Changes documented as a new file in changes/
  • GitHub Actions have all passed
  • Test coverage is 100% (Codecov passes)

When a selection touches one index along the first axis, apply the same
row window along the next axis, and so on. arr[t, y0:y1, x0:x1] then
fetches rows y0 to y1 of frame t instead of the whole frame, and a
single item is fetched for a point.

The fetched block keeps the narrowed axes with length 1, so the shifted
selection applies to it unchanged.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@codecov

codecov Bot commented Nov 9, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.91%. Comparing base (b0020f2) to head (e6965ce).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #4611   +/-   ##
=======================================
  Coverage   94.90%   94.91%           
=======================================
  Files          96       96           
  Lines       14055    14070   +15     
=======================================
+ Hits        13339    13354   +15     
  Misses        716      716           
Files with missing lines Coverage Δ
src/zarr/codecs/bytes.py 99.37% <100.00%> (+0.06%) ⬆️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@bendichter bendichter changed the title Narrow partial reads of uncompressed chunks within a single row feat: narrow partial reads of uncompressed chunks within a single row Nov 10, 2026
Comment thread src/zarr/codecs/bytes.py
offset = 0
block_shape = list(shape)
shifted = list(selection)
for axis, (selector, size) in enumerate(zip(selection, shape, strict=False)):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you add a comment declaring why strict=False is required here?

Comment thread src/zarr/codecs/bytes.py
that it indexes an array holding only those rows. Returns None when the
selection touches every row or the rows cannot be determined, in which case
the whole chunk is read.
) -> tuple[int, tuple[int, ...], SelectorTuple] | None:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tuple[int, tuple[int, ...], SelectorTuple]

we could use either a namedtuple or a dataclass here to make things a bit more literate. optional

@d-v-b d-v-b left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

great! I left two comments about nits, they are non-blocking

@d-v-b
d-v-b enabled auto-merge (squash) November 18, 2026 09:31
@d-v-b d-v-b added the benchmark Code will be benchmarked in a CI job. label Nov 18, 2026
@d-v-b
d-v-b merged commit 6eb4563 into zarr-developers:main Nov 18, 2026
49 checks passed
@codspeed

codspeed Bot commented Nov 18, 2026

Copy link
Copy Markdown

Merging this PR will regress 1 benchmark

⚡ 8 improved benchmarks
❌ 1 regressed benchmark
✅ 104 untouched benchmarks
⏩ 37 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Benchmark BASE HEAD Efficiency
❌ test_slice_indexing[None-(slice(None, None, None), slice(0, 3, 2), slice(0, 10, None))-memory_get_latency] 3.1 ms 4.2 ms -25.49%
⚡ test_slice_indexing[None-(slice(10, -10, 4), slice(10, -10, 4), slice(10, -10, 4))-memory] 191.7 ms 163 ms +17.6%
⚡ test_slice_indexing[None-(slice(0, None, 4), slice(0, None, 4), slice(0, None, 4))-memory] 350.2 ms 298.3 ms +17.42%
⚡ test_slice_indexing[None-(slice(10, -10, 4), slice(10, -10, 4), slice(10, -10, 4))-memory_get_latency] 223.1 ms 191.2 ms +16.69%
⚡ test_slice_indexing[None-(slice(0, None, 4), slice(0, None, 4), slice(0, None, 4))-memory_get_latency] 408.5 ms 350.6 ms +16.52%
⚡ test_slice_indexing[(50, 50, 50)-(slice(0, None, 4), slice(0, None, 4), slice(0, None, 4))-memory_get_latency] 411.8 ms 356.8 ms +15.39%
⚡ test_slice_indexing[(50, 50, 50)-(slice(0, None, 4), slice(0, None, 4), slice(0, None, 4))-memory] 406.9 ms 354.1 ms +14.91%
⚡ test_slice_indexing[(50, 50, 50)-(slice(10, -10, 4), slice(10, -10, 4), slice(10, -10, 4))-memory] 215.2 ms 187.4 ms +14.81%
⚡ test_slice_indexing[(50, 50, 50)-(slice(10, -10, 4), slice(10, -10, 4), slice(10, -10, 4))-memory_get_latency] 214.7 ms 188.2 ms +14.12%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing bendichter:partial-decode-single-row (e6965ce) with main (d8b8543)2

Open in CodSpeed

Footnotes

  1. 37 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

  2. No successful run was found on main (b0020f2) during the generation of this report, so d8b8543 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

@d-v-b

d-v-b commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

one perf regression on indexing data with latency. worth investigating but not a blocker

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

benchmark Code will be benchmarked in a CI job.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants