Repository navigation
Conversation
Decouple sequence IDs from physical storage columns so the batched-sequence cache can start with 2 physical columns and grow or shrink at runtime instead of always allocating max_seqs + 1 columns. - CudaKVPool gains resize_columns() with declared vs physical column counts, transactional replacement copies, and binding invalidation on column moves (mirroring grow()). - CudaBatchedSequenceCache tracks IDs and physical columns separately, uploads a 5-wide seq_info whose last field is the runtime pitch, and adds ensure_columns()/compact_columns() behind an inflight guard. - The lowering pass gains manifest version 2 with runtime_columns, the SDPA kernel reads the runtime pitch via INFO_WIDTH, and CudaExecutor reads get_offgraph_kv_sequence_info_abi (1 or 2) from the program. - ABI-1 behaviour is unchanged: the new paths are gated on CacheConfig::runtime_sequence_columns and manifest version 2. Tests: extended test_cuda_kv_cache and test_offgraph_kv_batched_sequence, plus an opt-in Muse-Glimmer runtime KV probe. The SDPA ring-scan logic from the experiment patch was intentionally not ported: main's ring handling is kept and only the runtime-pitch change is included. This is PR 1 of the runtime-columns series; per-owner capacities (ABI 3) build on the ID/column split introduced here.
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/25324
Note: Links to docs will display an error until the docs builds have been completed. ❌ 1 New Failure, 4 Pending, 3 Unclassified FailuresAs of commit b649b06 with merge base c29858d ( NEW FAILURE - The following job has failed:
UNCLASSIFIED FAILURES - DrCI could not classify the following jobs because the workflow did not run on the merge base. The failures may be pre-existing on trunk or introduced by this PR:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
This PR needs a
|
Gasoonjia
left a comment
There was a problem hiding this comment.
Review of the runtime-physical-columns (ABI 2) change. The core pool mechanics look right, but I have one functional concern about failure recovery and a few smaller items.
What looks good
CudaKVPool::resize_columnsruns its steps in a sound order: it allocates every replacement, queues the 2D copies, synchronizes, and only then callsinvalidate_bindings()and discards the old storage. Every failure path before the commit point rolls back to the old allocations, bindings,allocated_bytes_andcolumns_.FailedColumnGrowthKeepsOldStorageAndBindingsUsablecovers this.committedis reported separately from errors after the commit point (the trim sync), andensure_columnspoisons the cache in that case. That is the right split.invalidate_bindings()is shared withgrow(). It reaches every handle (Replay -> recapture, Warmup -> at least one eager step), andContractionInvalidatesEveryGraphAndBindingchecks it.- An ABI mismatch between the program and the runtime fails at load:
check_compiledcompares the side-buffer shape ([S,4]vs[S,5]), andFixedInfoABIIsRejectedcovers it. - The pitch handling is consistent across C++, the Python writes and Triton. Every
seq_inforow, padded segments included, carriescolumns_, so the decode kernel can safely read only row 0 and the prefill kernel can read it per segment. The dummy column moves tocolumns_ - 1with the pitch.
Main concern
- A failed execute leaves
inflight_set, and ABI-2 caches stay unusable untilclear()(inline). In ABI-1 a failed forward does not block later steps.
Other items (inline)
2. Columns grow one at a time. Each admission past the current width runs a device-wide sync, a full KV reallocation and copy, and an allocator trim, all inside prepare_step. Peak memory during a resize is old + new storage, so near max_seqs this can OOM where ABI-1's upfront allocation would have fit.
3. Changing compile_wrapper_opt_level from O0 to O2 in export_solo.py is unrelated and affects ABI-1 exports too.
4. ABI 2 combined with DFlash is rejected only by the export script. The runtime (CudaExecutor::create / the cache factory) still accepts it.
5. The seq_rm guard against step_seq_ids_ applies only in runtime mode. An abandoned declare_step then blocks removal, and ABI-1 still has the latent seqs_.at() throw.
6. The ABI-1 seq_clone failure path changed: a device-wide sync and possible poisoning. So ABI-1 is not strictly untouched (minor).
Nits / docs
make_cuda_batched_sequence_kv_cache's comment still says the step buffers are[segments, 4](cuda_kv_cache.cpp, ~L1576).- The
sdpa_varlendocstring forseq_info(~L410) still saysInt32[S, 4]. - In
LowerOffGraphKVPass,self._manifest.get("runtime_columns", False)is redundant now thatparse_offgraph_kv_manifestcallssetdefault.self._manifest["runtime_columns"]is enough. - When columns grow with an identity mapping, one
cudaMemcpy2DAsyncper layer of widthslot * old_columnscould replace the per-column copies (layers x seqs x 2calls).
Test gaps
- No test covers a step that is prepared and never committed (execute failure) in ABI-2.
- No executor-level test of
get_offgraph_kv_sequence_info_abivalidation (ABI 3; ABI 2 on the cell layout; ABI 2 with a draft method). - The Python test runs
batched_sequence_stepeagerly. No AOTI-compiled ABI-2 program runs in CI; the Muse-Glimmer probe covers that only manually. The probe is ~290 lines of opt-in diagnostics that CI never builds. Consider moving it to a follow-up or a dev-tools location.
This review was generated with AI assistance (Claude Code).
- Revert the unrelated O0 -> O2 change in export_solo.py. - Recover a prepared-but-uncommitted ABI-2 step in the next prepare_step (fence all streams, then clear inflight_) instead of wedging the cache until clear() drops every sequence; add a regression test. - Grow physical columns geometrically and trim the allocator only on the compact path, avoiding one full KV copy per admission. - Reject runtime columns + DFlash in CudaExecutor and in the batched-sequence cache factory. - Gate the seq_clone fence/poison on runtime columns so ABI-1 clone failures are unchanged; refuse seq_rm for a declared sequence in both ABIs. - Fix the lintrunner CMake formatting failure and stale seq_info width comments. - Pool resize comment and lowering .get() redundancy left as-is: resize_columns is only called from ensure_columns, and the .get() keeps LowerOffGraphKVPass usable with manifests not built by the parser.
|
Thanks for the review — addressed in b649b06 (see replies to each inline comment). The one CI failure on the first push was lintrunner CMake formatting only, also fixed in this update. Remaining known gaps, not addressed here: per-column copy merging in resize_columns, and a compiled ABI-2 run in CI (the opt-in probe is not built by CI); both are follow-up work, not blockers for the ABI-2 plumbing itself. |
|
@claude review effort=max |
|
Claude finished @Gasoonjia's task in 53m 50s —— View job PR Review: #25324This PR adds a runtime-pitch Detailed reviewSummaryThe PR lets the batched-sequence cache start at 2 physical columns, grow as sequences are admitted, and shrink with Two problems block it:
"Test CUDA Builds" (run 44647068649) was still queued when I wrote this, so the test failures below come from tracing the code, not from CI. Backward Compatibility
API Design
Infrastructure
Testing
Performance
Code Quality
Specific Comments
RecommendationRequest Changes.
Review generated by Claude Code (AI) using the |
Summary
Decouple sequence IDs from physical
storage columns in the batched-sequence off-graph KV cache, so the cache
can start with 2 physical columns and grow/shrink at runtime instead of
always allocating
max_seqs + 1columns (the compiled maximum).Today three things are glued together:
seq_new()returns a free column).CudaKVPoolcan only growin the rows direction (
grow()), never in the columns direction.from the compiled-in column count, so a narrower runtime allocation
would be read with the wrong stride.
This PR introduces sequence-info ABI 2 (manifest version 2,
runtime_columns), gated so ABI-1 behaviour is unchanged.What changed
CudaKVPool(cuda_kv_pool.{h,cpp}):resize_columns()withdeclared vs. physical column counts, transactional replacement copies
(commit only after every copy passes, mirroring
grow()), and bindinginvalidation on column moves so compiled programs and captured CUDA
graphs rebind/recapture against the new storage.
CudaBatchedSequenceCache(cuda_kv_cache.{h,cpp}): IDs andphysical columns are tracked separately (
free_sequence_id()/free_physical_column()),seq_infois uploaded 5-wide with theruntime pitch in field 4, and
ensure_columns()/compact_columns()grow/shrink the physical width behind aninflight_guard (no clone/declare while a step is prepared butuncommitted). New metrics:
physical_columns,declared_columns,live_sequences,column_resize_count.lower_offgraph_kv.py,sdpa_varlen.py):manifest version 2 validation, runtime pitch in the write path, and
INFO_WIDTH(4 or 5) in both SDPA kernels. Deliberately scoped:main's ring-scan logic is kept untouched — only the pitch change is
ported. The experiment patch's ring rewrite and HEAD_DIM 512 support
were intentionally not included.
CudaExecutor: reads the optionalget_offgraph_kv_sequence_info_abiconstant method (1 or 2), rejects other values and ABI 2 on
non-batched-sequence layouts, and exposes
compact_kv_columns().CacheConfig::runtime_sequence_columns(defaultfalse) is the switch.export_solo.pycan emit an ABI-2 program;the runtime-KV probe is an opt-in CMake target
(
MUSE_GLIMMER_BUILD_RUNTIME_KV_PROBE, OFF by default).Tests
backends/cuda/runtime/test/test_cuda_kv_cache.cpp: +380 lines —pool column resize (grow/shrink/remap/rollback), cache
ownership/growth/rewind with physical columns, compact behaviour.
backends/cuda/tests/test_offgraph_kv_batched_sequence.py: +122 lines —5-wide
seq_info, runtime pitch in writes/attention, ring pitch andsparse-survivor cases.
runtime_kv_columns_probe.cppfor end-to-end checks on areal exported program.
clang-formatclean,ufmt/py_compileclean.CUDA tests were not run locally (no GPU in this environment) —
please rely on CI for the CUDA test targets.
Origin
Ported from the archived experiment patch
(
runtime-columns-20261024, basefca4fb36, 14 files) onto currentmain, with the SDPA ring part reconciled as described above. The
archived experiment results (RTX 5090 reproducibility archive,
Oct 30 2026) motivated the change; no benchmark/orchestration code is included.