Skip to content

[CUDA] Runtime physical columns for batched-sequence off-graph KV (ABI 2) - #25324

Open
Gasoonjia wants to merge 2 commits into
pytorch:mainfrom
Gasoonjia:runtime-columns-abi2-pr1
Open

Gasoonjia wants to merge 2 commits into
pytorch:mainfrom
Gasoonjia:runtime-columns-abi2-pr1

Conversation

@Gasoonjia

@Gasoonjia Gasoonjia commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Decouple sequence IDs from physical
storage columns in the batched-sequence off-graph KV cache, so the cache
can start with 2 physical columns and grow/shrink at runtime instead of
always allocating max_seqs + 1 columns (the compiled maximum).

Today three things are glued together:

  1. A sequence ID is its storage column (seq_new() returns a free column).
  2. The column count is fixed at compile time; CudaKVPool can only grow
    in the rows direction (grow()), never in the columns direction.
  3. The SDPA kernel and the lowering write path compute the K/V row pitch
    from the compiled-in column count, so a narrower runtime allocation
    would be read with the wrong stride.

This PR introduces sequence-info ABI 2 (manifest version 2,
runtime_columns), gated so ABI-1 behaviour is unchanged.

What changed

  • CudaKVPool (cuda_kv_pool.{h,cpp}): resize_columns() with
    declared vs. physical column counts, transactional replacement copies
    (commit only after every copy passes, mirroring grow()), and binding
    invalidation on column moves so compiled programs and captured CUDA
    graphs rebind/recapture against the new storage.
  • CudaBatchedSequenceCache (cuda_kv_cache.{h,cpp}): IDs and
    physical columns are tracked separately (free_sequence_id() /
    free_physical_column()), seq_info is uploaded 5-wide with the
    runtime pitch in field 4, and ensure_columns() /
    compact_columns() grow/shrink the physical width behind an
    inflight_ guard (no clone/declare while a step is prepared but
    uncommitted). New metrics: physical_columns, declared_columns,
    live_sequences, column_resize_count.
  • Lowering + kernel (lower_offgraph_kv.py, sdpa_varlen.py):
    manifest version 2 validation, runtime pitch in the write path, and
    INFO_WIDTH (4 or 5) in both SDPA kernels. Deliberately scoped:
    main's ring-scan logic is kept untouched — only the pitch change is
    ported. The experiment patch's ring rewrite and HEAD_DIM 512 support
    were intentionally not included.
  • CudaExecutor: reads the optional get_offgraph_kv_sequence_info_abi
    constant method (1 or 2), rejects other values and ABI 2 on
    non-batched-sequence layouts, and exposes compact_kv_columns().
    CacheConfig::runtime_sequence_columns (default false) is the switch.
  • Export: Muse-Glimmer export_solo.py can emit an ABI-2 program;
    the runtime-KV probe is an opt-in CMake target
    (MUSE_GLIMMER_BUILD_RUNTIME_KV_PROBE, OFF by default).

Tests

  • backends/cuda/runtime/test/test_cuda_kv_cache.cpp: +380 lines —
    pool column resize (grow/shrink/remap/rollback), cache
    ownership/growth/rewind with physical columns, compact behaviour.
  • backends/cuda/tests/test_offgraph_kv_batched_sequence.py: +122 lines —
    5-wide seq_info, runtime pitch in writes/attention, ring pitch and
    sparse-survivor cases.
  • New opt-in runtime_kv_columns_probe.cpp for end-to-end checks on a
    real exported program.
  • Local checks done here: clang-format clean, ufmt/py_compile clean.
    CUDA tests were not run locally (no GPU in this environment) —
    please rely on CI for the CUDA test targets.

Origin

Ported from the archived experiment patch
(runtime-columns-20261024, base fca4fb36, 14 files) onto current
main, with the SDPA ring part reconciled as described above. The
archived experiment results (RTX 5090 reproducibility archive,
Oct 30 2026) motivated the change; no benchmark/orchestration code is included.

Decouple sequence IDs from physical storage columns so the
batched-sequence cache can start with 2 physical columns and grow or
shrink at runtime instead of always allocating max_seqs + 1 columns.

- CudaKVPool gains resize_columns() with declared vs physical column
  counts, transactional replacement copies, and binding invalidation on
  column moves (mirroring grow()).
- CudaBatchedSequenceCache tracks IDs and physical columns separately,
  uploads a 5-wide seq_info whose last field is the runtime pitch, and
  adds ensure_columns()/compact_columns() behind an inflight guard.
- The lowering pass gains manifest version 2 with runtime_columns, the
  SDPA kernel reads the runtime pitch via INFO_WIDTH, and CudaExecutor
  reads get_offgraph_kv_sequence_info_abi (1 or 2) from the program.
- ABI-1 behaviour is unchanged: the new paths are gated on
  CacheConfig::runtime_sequence_columns and manifest version 2.

Tests: extended test_cuda_kv_cache and
test_offgraph_kv_batched_sequence, plus an opt-in Muse-Glimmer
runtime KV probe. The SDPA ring-scan logic from the experiment patch
was intentionally not ported: main's ring handling is kept and only
the runtime-pitch change is included.

This is PR 1 of the runtime-columns series; per-owner capacities
(ABI 3) build on the ID/column split introduced here.
@pytorch-bot

pytorch-bot Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/25324

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 4 Pending, 3 Unclassified Failures

As of commit b649b06 with merge base c29858d (image):

NEW FAILURE - The following job has failed:

UNCLASSIFIED FAILURES - DrCI could not classify the following jobs because the workflow did not run on the merge base. The failures may be pre-existing on trunk or introduced by this PR:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Nov 20, 2026
@github-actions

Copy link
Copy Markdown

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

@Gasoonjia Gasoonjia left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of the runtime-physical-columns (ABI 2) change. The core pool mechanics look right, but I have one functional concern about failure recovery and a few smaller items.

What looks good

  • CudaKVPool::resize_columns runs its steps in a sound order: it allocates every replacement, queues the 2D copies, synchronizes, and only then calls invalidate_bindings() and discards the old storage. Every failure path before the commit point rolls back to the old allocations, bindings, allocated_bytes_ and columns_. FailedColumnGrowthKeepsOldStorageAndBindingsUsable covers this.
  • committed is reported separately from errors after the commit point (the trim sync), and ensure_columns poisons the cache in that case. That is the right split.
  • invalidate_bindings() is shared with grow(). It reaches every handle (Replay -> recapture, Warmup -> at least one eager step), and ContractionInvalidatesEveryGraphAndBinding checks it.
  • An ABI mismatch between the program and the runtime fails at load: check_compiled compares the side-buffer shape ([S,4] vs [S,5]), and FixedInfoABIIsRejected covers it.
  • The pitch handling is consistent across C++, the Python writes and Triton. Every seq_info row, padded segments included, carries columns_, so the decode kernel can safely read only row 0 and the prefill kernel can read it per segment. The dummy column moves to columns_ - 1 with the pitch.

Main concern

  1. A failed execute leaves inflight_ set, and ABI-2 caches stay unusable until clear() (inline). In ABI-1 a failed forward does not block later steps.

Other items (inline)
2. Columns grow one at a time. Each admission past the current width runs a device-wide sync, a full KV reallocation and copy, and an allocator trim, all inside prepare_step. Peak memory during a resize is old + new storage, so near max_seqs this can OOM where ABI-1's upfront allocation would have fit.
3. Changing compile_wrapper_opt_level from O0 to O2 in export_solo.py is unrelated and affects ABI-1 exports too.
4. ABI 2 combined with DFlash is rejected only by the export script. The runtime (CudaExecutor::create / the cache factory) still accepts it.
5. The seq_rm guard against step_seq_ids_ applies only in runtime mode. An abandoned declare_step then blocks removal, and ABI-1 still has the latent seqs_.at() throw.
6. The ABI-1 seq_clone failure path changed: a device-wide sync and possible poisoning. So ABI-1 is not strictly untouched (minor).

Nits / docs

  • make_cuda_batched_sequence_kv_cache's comment still says the step buffers are [segments, 4] (cuda_kv_cache.cpp, ~L1576).
  • The sdpa_varlen docstring for seq_info (~L410) still says Int32[S, 4].
  • In LowerOffGraphKVPass, self._manifest.get("runtime_columns", False) is redundant now that parse_offgraph_kv_manifest calls setdefault. self._manifest["runtime_columns"] is enough.
  • When columns grow with an identity mapping, one cudaMemcpy2DAsync per layer of width slot * old_columns could replace the per-column copies (layers x seqs x 2 calls).

Test gaps

  • No test covers a step that is prepared and never committed (execute failure) in ABI-2.
  • No executor-level test of get_offgraph_kv_sequence_info_abi validation (ABI 3; ABI 2 on the cell layout; ABI 2 with a draft method).
  • The Python test runs batched_sequence_step eagerly. No AOTI-compiled ABI-2 program runs in CI; the Muse-Glimmer probe covers that only manually. The probe is ~290 lines of opt-in diagnostics that CI never builds. Consider moving it to a follow-up or a dev-tools location.

This review was generated with AI assistance (Claude Code).

Comment thread backends/cuda/runtime/cuda_kv_cache.cpp Resolved
Comment thread backends/cuda/runtime/cuda_kv_cache.cpp Resolved
Comment thread backends/cuda/runtime/cuda_kv_pool.cpp Resolved
Comment thread backends/cuda/runtime/cuda_kv_cache.cpp Resolved
Comment thread backends/cuda/runtime/cuda_kv_cache.cpp Resolved
- Revert the unrelated O0 -> O2 change in export_solo.py.
- Recover a prepared-but-uncommitted ABI-2 step in the next
  prepare_step (fence all streams, then clear inflight_) instead of
  wedging the cache until clear() drops every sequence; add a
  regression test.
- Grow physical columns geometrically and trim the allocator only on
  the compact path, avoiding one full KV copy per admission.
- Reject runtime columns + DFlash in CudaExecutor and in the
  batched-sequence cache factory.
- Gate the seq_clone fence/poison on runtime columns so ABI-1 clone
  failures are unchanged; refuse seq_rm for a declared sequence in
  both ABIs.
- Fix the lintrunner CMake formatting failure and stale seq_info
  width comments.
- Pool resize comment and lowering .get() redundancy left as-is:
  resize_columns is only called from ensure_columns, and the .get()
  keeps LowerOffGraphKVPass usable with manifests not built by the
  parser.
@Gasoonjia

Copy link
Copy Markdown
Contributor Author

Thanks for the review — addressed in b649b06 (see replies to each inline comment). The one CI failure on the first push was lintrunner CMake formatting only, also fixed in this update. Remaining known gaps, not addressed here: per-column copy merging in resize_columns, and a compiled ABI-2 run in CI (the opt-in probe is not built by CI); both are follow-up work, not blockers for the ABI-2 plumbing itself.

@Gasoonjia

Copy link
Copy Markdown
Contributor Author

@claude review effort=max

@claude

claude Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

Claude finished @Gasoonjia's task in 53m 50s —— View job


PR Review: #25324

This PR adds a runtime-pitch seq_info ABI ([S, 5], ABI 2) so the CUDA batched-sequence KV cache can allocate fewer physical columns than the compiled maximum, but ensure_columns widens the storage on every step until it reaches that maximum (by trace, two of the PR's own C++ tests fail), and the new seq_rm refusal leaks sequences in both ABIs, so I'm requesting changes.

Detailed review

Summary

The PR lets the batched-sequence cache start at 2 physical columns, grow as sequences are admitted, and shrink with compact_columns. The kernels get the pitch from seq_info[:, 4].

Two problems block it:

  • The growth policy added in b649b06 resizes on every prepare_step and seq_clone until the width reaches max_seqs + 1, which cancels the memory saving.
  • seq_rm now refuses in states that ordinary forward failures leave behind, and every caller ignores the refusal.

"Test CUDA Builds" (run 44647068649) was still queued when I wrote this, so the test failures below come from tracing the code, not from CI.

Backward Compatibility

  • seq_rm refusals leak sequences (major). backends/cuda/runtime/cuda_kv_cache.cpp:998-1009 now returns false while inflight_ is set or while the id is in step_seq_ids_, in both ABIs.
    • Callers ignore the result. CudaExecutor::close_session calls SessionTable::close (extension/llm/batching/util/session_table.cpp:110-111), which erases the session anyway. SequenceGuard (extension/llm/batching/executor_utils.h:71-75) never retries.
    • So a refusal keeps the sequence, its id and its column until clear(). Once max_seqs ids have leaked, seq_new fails permanently.
    • ABI 1 (existing programs): when prepare_step fails after declare_step, step_seq_ids_ is never cleared (1171). Example failures: an OOM during row growth in pool_.prepare (1121), a failed upload (1154-1157), or the claim check (1164). Closing those sessions then leaks them. On the base branch this seq_rm succeeded.
    • ABI 2: after any failed forward, inflight_ stays set until the next prepare_step. The usual response, closing the failed sessions, leaks every one of them.
    • Both cases contradict the BatchControl contract (extension/llm/cache/cache.h:128-130: "False = an unknown sequence").
    • compact_columns (1226-1230) has the same guards, so memory can't be reclaimed after a failure until another step runs.
    • Suggested fix:
      • Drop the inflight_ check. It isn't needed: prepare_step fences every stream before reusing storage (1059-1071), and seq_clone and compact_columns already refuse while a step is in flight.
      • For a declared id, clear step_seq_ids_ before erasing it. A following prepare_step then returns InvalidArgument (1081-1087) instead of throwing from seqs_.at.
      • Have compact_columns fence and recover the same way prepare_step does. Its step_seq_ids_.empty() guard isn't needed for safety, because prepare_step resolves columns at 1141.
    • Fix this →

API Design

  • seq_new admits a sequence it can't place. When no physical column is free, seq_new (cuda_kv_cache.cpp:936-947) admits the sequence with column -1. Once the Code Quality fix below lands, this happens every time the current columns are full. The column is only allocated by the next prepare_step's ensure_columns call (1119; the loop at 1327-1341), even when that step doesn't include the new sequence.
    • If that resize fails, every prepare_step fails until the new sequence is removed, including steps whose sequences already have columns.
    • This breaks the admission-gate contract (cache.h:104-108: "a step that passes has room for its tokens"). declare_step (899-930) never looks at columns, so it passes and the failure shows up as a forward error.
    • Fix: allocate the column in seq_new as seq_clone does (977-981), return nullopt on failure, and add "insufficient resources" to the seq_new doc (cache.h:114-117), as the seq_clone doc already does.
    • Fix this →
  • The ABI is declared in two places that must agree.
    • The runtime selects ABI 2 only from the constant method "get_offgraph_kv_sequence_info_abi". It is a string literal at backends/cuda/batching/cuda_executor.cpp:375, read with value_or(1).
    • Only examples/models/muse-glimmer/export/export_solo.py:696 emits that method, also as a literal.
    • The pass picks the seq_info width from the manifest's version-2 runtime_columns (backends/cuda/passes/lower_offgraph_kv.py:98-107).
    • So if any other exporter lowers with runtime_columns, the program is configured as ABI 1 and rejected in note_handle. The pool expects [S, 4] (cuda_kv_cache.cpp:847, checked at cuda_kv_pool.cpp:722).
    • Fix: define the name once, next to kMaxSeqsMethod (cuda_executor.h:53), and have one helper in the pass produce both the manifest fields and the constant-method entry.
    • Fix this →
  • A CUDA-only switch sits in the backend-neutral config. CacheConfig::runtime_sequence_columns (extension/llm/cache/cache.h:181-182) only affects the CUDA batched-sequence cache.
    • Only make_cuda_batched_sequence_kv_cache reads it. make_cuda_sequence_kv_cache (cuda_kv_cache.cpp:1526), make_cuda_cell_kv_cache (1556) and every other backend's cache silently ignore true.
    • Fix: reject it in those factories, or make it a parameter of make_cuda_batched_sequence_kv_cache.
    • Fix this →

Infrastructure

  • The probe is never built or run. examples/models/muse-glimmer/CMakeLists.txt:147-170 builds it only under MUSE_GLIMMER_BUILD_RUNTIME_KV_PROBE, which defaults to OFF, and nothing under .ci/ or .github/ turns it on.
    • Yet the RuntimeColumnPitchTest docstring (backends/cuda/tests/test_offgraph_kv_batched_sequence.py:268-270) names "the C++ cache and model probe" as the tests for the smaller allocations.
    • Either build and run the probe in the existing muse-glimmer CUDA CI job, or drop it and that part of the docstring.

Testing

  • Nothing checks that the width stays put or that growth is geometric. GrowthAndSparseIdCompactionPreserveAllKV checks widths only at 1, 2, 4, 8 and 16 live sessions (test_cuda_kv_cache.cpp:2829). At those counts, growing to exactly live + 1 and doubling the usable width give the same widths, so the geometric growth that b649b06 adds is never exercised.
    • Add a test that runs repeated steps over the same sequences and checks that physical_columns and column_resize_count don't change, plus one admission count that isn't a power of two. The Code Quality fix link includes this.
  • Other missing coverage:
    • CudaKVPoolTest.FailedColumnGrowthKeepsOldStorageAndBindingsUsable (test_cuda_kv_cache.cpp:3062) only runs with trim_unused=true. The growth path uses false (cuda_kv_cache.cpp:1353).
    • No test calls seq_rm or seq_clone between a failed step and the next prepare_step, or calls seq_rm on a declared sequence under ABI 1.
    • The executor's ABI checks (cuda_executor.cpp:372-387) have no test: ABI 2 without the sequence layout, ABI 2 with DFlash, and unknown values.
    • RuntimeColumnPitchTest never runs the multi-split path with a 5-wide seq_info. rows is at most 64 (test_offgraph_kv_batched_sequence.py:276), so _num_splits (sdpa_varlen.py:385-388) always returns 1. The "split" path named in the comment at test_offgraph_kv_batched_sequence.py:329 is the decode kernel with a single split.
    • test_lowered_program_matches_the_neutral_op (test_offgraph_kv_batched_sequence.py:477) only lowers the version-1 (4-wide) ABI, so no test checks the numerics of a lowered ABI-2 program.
    • Fix this →

Performance

  • Column resizes copy more than they need to. In backends/cuda/runtime/cuda_kv_pool.cpp:529-564, resize_columns allocates and copies every allocated row of each retained column (logical_rows, lines 530 and 544). grow() copies only live_rows (402-403).
    • With a large initial_capacity, or after the rows have grown, a column resize copies mostly empty rows.
    • When one prepare_step needs both a column resize (cuda_kv_cache.cpp:1119) and row growth (1121-1124), the KV cache is copied twice, each time with old and new storage allocated at once.
    • Fix: pass the live row count, as grow() does. Consider a single combined resize for steps that need both.
    • Fix this → (this link covers the live-rows part only)

Code Quality

  • Blocker: ensure_columns grows on every call. In backends/cuda/runtime/cuda_kv_cache.cpp:1306-1309, the non-compact target never keeps the current width.
    • grown = max(live + 1, 2 * (columns_ - 1) + 1) is always greater than columns_ once columns_ >= 2. So target != columns_ (1342) holds until the width reaches max_seqs_ + 1.
    • Every prepare_step (1118-1120, unconditional under runtime columns) and seq_clone (978) therefore resizes, however many sequences are live.
    • Each resize does a device-wide sync, copies the full KV cache, and calls invalidate_bindings() (cuda_kv_pool.cpp:569), which sends every CUDA graph back to warmup.
    • With one live sequence and max_seqs = 21, the first five steps widen the storage 2 → 3 → 5 → 9 → 17 → 22, and the first step after compact_columns starts the climb again. The memory saving is gone after about log2(max_seqs) steps, and each of those steps pays for a copy and a graph recapture.
    • By trace, two of the PR's own tests fail:
      • GrowthAndSparseIdCompactionPreserveAllKV:
        • The first one-session step widens to 3 columns, but 2845 expects 2 (sessions + 1).
        • before has 22 columns, but 2889 assumes 17.
        • The step after compaction widens from 2 to 3 columns, so the scratch column checked at 2905 is 2 instead of 1, and the pitch checked at 2917 is 3 instead of 2.
      • CloneAndContractionCopyWrappedRingHistory: two single-sequence steps reach 5 columns and the clone reaches 9, but 2948 expects 3.
    • Fix: keep columns_ when live + 1 <= columns_. Otherwise grow to min(max_seqs_ + 1, max(live + 1, 2 * (columns_ - 1) + 1)). With that change, both tests pass by trace.
    • The comment at 1301 ("doubling the live columns") should then say that the usable width doubles.
    • Fix this →

Specific Comments

  • cuda_kv_cache.cpp:802-819: the class comment still describes fixed columns and a 4-field seq_info row.
  • cuda_kv_cache.cpp:1064-1065: the comment says "treat the step as aborted", but the step is kept. Its rows stay claimed (1161-1170), and UncommittedStepRecoversOnNextPrepare checks for pos == 2.
  • cuda_kv_cache.cpp:1518-1519: the poisoned_ comment says the flag is set when a clear() fence fails, but it is now also set at 990, 1067 and 1362.
  • cuda_kv_cache.cpp:1503, 1507: the mutable columns_ and inflight_ sit in the middle of the const members.
  • cuda_kv_cache.cpp:1278: free_sequence_id's loop variable is named column, which is now misleading because ids are no longer columns under ABI 2.
  • cuda_kv_cache.cpp:1312-1318: relocate is dead code. It is only set when compact is true, and compact alone already satisfies the condition at 1342.
  • cuda_kv_cache.h:33-35: the growth_count doc says it identifies the addresses a program was bound to, but column resizes now move the storage without incrementing it. The new fields at 37-40 are undocumented.
  • cuda_kv_cache.h:92: the compact_columns(int) parameter is unnamed.
  • cuda_executor.h:230: compact_kv_columns(int spare_columns = 1) has no doc saying what spare_columns counts or when it may be called.
  • cuda_kv_pool.h:88-90: the doc says the resize "commits only after every replacement copy passes", but with trim_unused it can return an error after committing (cuda_kv_pool.cpp:582-588). Document *committed.
  • lower_offgraph_kv.py:314-315: the slot formula r * columns + c is wrong under runtime_columns, where the pitch is seq_info[0, 4] (329). Line 325 says "ABI-5" where it means the 5-wide ABI 2.
  • sdpa_varlen.py:408-409: with a 5-wide seq_info, key.stride(0) is ignored and rows are addressed at seq_info[:, 4] * stride(1) (94-97, 254-257), but the doc still says only the head dimension must be contiguous.

Fix the Specific Comments →

Recommendation

Request Changes.

  • The growth policy (Code Quality) cancels the PR's memory saving, and by trace it fails two of the PR's own tests.
  • The seq_rm refusal (Backward Compatibility) also leaks sequences in existing ABI-1 programs.
  • The API Design items should be settled before the ABI-2 contract ships.

Review generated by Claude Code (AI) using the /pr-review skill.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant