Skip to content

perf(p2p): verify one KZG batch per block for fetched data columns - #754

Merged
MegaRedHand merged 18 commits into
mainfrom
perf/column-kzg-batch-per-block
Nov 19, 2026
Merged

MegaRedHand merged 18 commits into
mainfrom
perf/column-kzg-batch-per-block

Conversation

@MegaRedHand

@MegaRedHand MegaRedHand commented Oct 30, 2026 •

Copy link
Copy Markdown
Collaborator

🗒️ Description / Motivation

A range or by-root answer, or a batch of parked columns replayed after their parent imports, carries many columns of the same block. Each column was checked on its own blocking task with its own verify_cell_kzg_proof_batch call, and the chain actor heard nothing until every sidecar of the whole call was done, so data availability waited on the slowest block of the batch. Every rule that reads only the block's shared signed header (BLS signature, proposer, finalized ancestry, a state advance past the lookahead) also ran once per sidecar.

What Changed

check_and_forward (p2p/src/beacon/column_checks.rs) groups the sidecars of one call by block (the signed header's root for fulu, beacon_block_root for gloas), keeping one sidecar per (block, index). It then checks each group in two stages on column_check_permits permits:

  1. Prechecks, with the shared rules run once (state_transition/src/beacon/gossip/column.rs). Every rule before the KZG proofs runs sidecar by sidecar through one ChainPrechecks, which judges what the block's sidecars share once per group:

    • for fulu, the parent's post-state (or the head-state signature check before parking), NotAfterParent, the signature, the proposer (including a state advance past the lookahead) and the finalized ancestry, keyed on the header's root and signature;
    • for gloas, the lookup and decode of the named block.

    The shared answer is worked out only when a sidecar first needs it. The sidecars this stage settles (parked or dropped) are forwarded at once.

  2. KZG proofs in sub-batches. The proofs of the sidecars still standing are verified in near-equal sub-batches of at most 16 columns. Each sub-batch is one verify_cell_kzg_proof_batch call (fork_choice::verify_column_cells_batch) on a permit of its own; the first uses the group's permit. A failed batch re-verifies its sidecars one by one. The block's kept sidecars go to the chain actor as one batch.

Gossip's stateful_checks/stateful_checks_gloas keep their order and are not batched. verify_column_cells_batch checks each column's lengths, and the single-column path delegates to it.

Correctness / Behavior Guarantees

  • Verdicts do not change. The proofs are the last rule on the chain path, a failed batch is re-run sidecar by sidecar, and the shared rules are memoized under everything they read of the sidecar (header root and signature, or block root). A test runs a mixed group (good, malformed, stored, forged signature, bad inclusion proof, bad cell) through the group path and checks it against chain_checks_for one sidecar at a time.
  • One deliberate exception: a sidecar repeating a (block, index) already in the same call is dropped unjudged and counted as already_seen. Only a faulty peer sends one, and dropping it is what bounds a group at NUMBER_OF_COLUMNS sidecars.
  • A block's proofs no longer hold one permit from first column to last; a further sub-batch holds one for at most 16 columns. The prechecks still run a whole group in turn on the group's permit; that hold time is not measured. A panicking KZG sub-batch loses only its own sidecars.

Trade-off: wall-clock against CPU

Measured locally, one thread, 9 blobs, fastest of 15 runs: one column takes 3.2 ms, a batch of 8 columns 7.9 ms, of 16 columns 12.6 ms, and of 128 columns 73 ms.

Computed from those, assuming free permits and cores (not measured concurrently):

columns wall-clock now before the PR as one batch CPU now CPU before
8 (a node's sampled set, one batch) 7.9 ms 3.2 ms 7.9 ms 7.9 ms 25.8 ms
128 (supernode, 8 sub-batches) ~13 ms ~26 ms 73 ms ~101 ms ~414 ms

Tests Added / Run

  • Group verdicts equal chain_checks_for one by one (fulu, real block sidecars).
  • A gloas group reads its block once and keeps what verifies.
  • KZG fallback compared with verify_data_column_sidecar_kzg_proofs.
  • A per-column length mismatch is refused even when the totals match.
  • p2p: a block's columns are forwarded as one Checked batch across two sub-batches.
  • p2p: repeats are kept once, and sub-batch sizes are bounded.
  • cargo fmt --all -- --check: clean.
  • cargo clippy --workspace --all-targets -- -D warnings: clean.
  • cargo test --profile release-fast -p ethlambda-state-transition -p ethlambda-p2p: state-transition 647 passed (16 ignored), p2p 375 passed (1 ignored).
  • Not run locally: make test spec suites; CI covers them.

Related Issues / PRs

Part of a beacon import perf batch, each PR independent and based on main: #752 (validation contention metrics, also edits column_checks.rs and docs/metrics.md), #753 (merkle layer hashing). The engine/STF overlap PR also edits fork_choice.rs, in a different region.

✅ Verification Checklist

  • Ran make fmt — clean
  • Ran make lint (clippy with -D warnings) — clean
  • Ran make test (test-consensus plus test-node, at release-fast) — not run locally (spec suites); unit tests of the touched crates pass

A range or by-root answer, or a batch of parked columns replayed after
their parent imports, carries many columns of the same block. Each column
was checked on its own blocking task with its own verify_cell_kzg_proof_batch
call, and the chain actor heard nothing until every sidecar of the whole call
was done, so data availability waited on the slowest block of the batch.

check_and_forward now groups the sidecars by block (the signed header's root
for fulu, beacon_block_root for gloas). A group takes one column_check_permits
permit, runs every sidecar's rules that do not read the cells, then verifies
the proofs of all sidecars still standing in one batch. A batch that fails is
re-run sidecar by sidecar, so exactly the sidecars rejected before are
rejected. The group's Keep and AwaitParent sidecars go to the chain actor as
soon as the group finishes.

Verdicts do not change: the proofs are the last rule on the chain path (the
lookahead rule after them is the gossip path's alone), so chain_checks is now
chain_precheck followed by a KZG step, with gossip's stateful_checks and
stateful_checks_gloas keeping their order through the same split halves.

On this machine a batch over 9-blob columns costs 3.3x less than separate
calls at 8 columns and 4.8x less at 32.
Every fulu sidecar of a block carries the same signed header, so the group
added in the previous commit repeated one BLS verification per sidecar,
now one after another under a single permit. SignatureMemo remembers
header_signature_is_valid's answer for the group, keyed on the header root,
the signature bytes and (for the head-state check made before parking) the
head it read, so a forged signature over a shared header is judged on its
own. Only that answer is remembered; every other rule runs per sidecar in
its order, so verdicts are unchanged. The memo lives in one group's
spawn_blocking closure; gossip and the single-sidecar wrappers use a fresh
one per call.
@github-actions

Copy link
Copy Markdown

🤖 Claude Code Review

Review: PR 754, one KZG batch per block for fetched data columns

I read the diff only. I did not build it or run the tests. Overall this looks good, and I found nothing blocking.

Correctness

  • Verdict equivalence holds. The proofs are the last rule on the chain path. The lookahead rule that follows them is PastLookahead::Queue's alone, and it stays after the KZG step in judge_against_parent (column.rs, around line 485). So chain_precheck followed by kzg_verdicts gives each sidecar the verdict it had before.
  • The fallback is sound. If the batch fails, kzg_verdicts re-verifies each sidecar alone. A malformed entry (Cell::from_bytes error or length mismatch) turns into Err, which also takes the fallback path. Neither a forged column nor a malformed one can get a neighbour rejected or accepted wrongly.
  • Batch inputs are length-checked beforehand. verify_column_cells_batch relies on equal lengths for cells, commitments and proofs. verify_data_column_sidecar (fulu) and gloas_verify_data_column_sidecar (gloas, in gloas_before_kzg) both run before a sidecar can become Pending. A violation would only fail the batch and trigger the fallback, not corrupt a verdict.
  • The SignatureMemo key is sound. The key is (header_root, signature, head). A forged signature over a shared header gets its own entry. The head-state check in queue_unless_forged is keyed on the head it read, so the same header judged against a different state is a separate entry. The test a_groups_shared_header_signature_is_verified_once covers the forged case. A memo entry can also be shared across sidecars whose parent state appeared midway, because the parent-state and head-state paths use different keys (None versus Some(head)).
  • Grouping is safe. Fulu sidecars are grouped by header root, and the inclusion proof ties their commitments to that root. Gloas sidecars are grouped by beacon_block_root and take their commitments from the bid. So the commitments within a group are consistent.
  • Order is preserved. Only Pending sidecars can become Keep, and they are pushed in input order. Keep and AwaitParent order within a group therefore matches arrival order.

Things to consider (non-blocking)

  1. Panic handling is split.
    • In judge_group, a panic in the precheck is caught by catch_unwind per sidecar, so it costs only that sidecar.
    • A panic in kzg_verdicts, or in the batch itself, loses the whole group via the outer spawn_blocking join error (column_checks.rs).
    • Both are reasonable and documented. The catch_unwind relies on panic=unwind, which the old spawn_blocking path also relied on, so nothing changes there.
    • A short comment on the second case in judge_group would help. The existing comment says "costs that one sidecar", which is not true of the KZG step.
  2. A bad column in a block costs more than before. One bad column that passes the precheck makes the group pay one batch plus N single verifications. By the PR's own 3.3–4.8x measurement that is about 1.2–1.3x the old per-sidecar cost, which is acceptable. It might be worth noting that the peer sending the bad column is not penalized here. That is the same as before, so I am not asking for a change.
  3. Metric semantics changed. lean_data_column_kzg_verify_seconds now observes per batch. On failure it also gets one observation per re-verified sidecar. docs/metrics.md documents this, but dashboards or alerts built on per-sidecar latency will shift.
  4. SignatureMemo.verified is a test-only counter in a production struct. It is a cheap usize, so this is fine. If you want to keep it out of the hot struct, #[cfg(test)] would do it.
  5. Memory. The batch copies every cell (c_kzg::Cell::from_bytes, about 2 KiB each) into one Vec. This is bounded by columns times blobs per block and similar to before, but the copies now live at once per group. With several groups in flight (bounded by column_check_permits) that is still modest.

Tests and docs

  • The new tests cover the valid batch, one bad column in a batch, swapped proofs, grouping order, per-block forwarding, and signature-memo reuse. These are the right cases.
  • docs/beacon_wire.md and docs/metrics.md are updated to match.

Verdict: approve. Item 1 is the only point I would act on, and it is just a comment tweak.


Automated review by Claude (Anthropic) · sonnet · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Codex Code Review

I found one substantive issue; otherwise the batching approach looks sound and preserves per-sidecar verdicts.

  • Potential DoS / unbounded task fan-out in crates/net/p2p/src/beacon/column_checks.rs:163. check_and_forward spawns one outer task, then spawns one inner Tokio task per block-group before those tasks enter spawn_blocking. The semaphore limits blocking work, but not the number of queued async tasks or retained groups. A large DataColumnsByRange response with many distinct block roots can therefore allocate one owned task + one moved Vec<DataColumnSidecar> per group and keep them pending indefinitely. For p2p-facing blockchain code, that is a meaningful memory-pressure vector. Consider processing groups in the outer task directly after acquiring each permit, or adding a bounded work queue so permits bound both execution and queued work.

  • Batch verifier relies on a non-local invariant without enforcing it in crates/blockchain/state_transition/src/beacon/fork_choice.rs:4152. verify_column_cells_batch assumes column.len() == commitments.len() == proofs.len() for every entry, but it only precomputes total from column.len() and then independently appends commitments/proofs. Today callers appear to pre-validate shape, so this is probably correct, but the function is pub and a future caller can silently build mismatched vectors and push malformed input into verify_cell_kzg_proof_batch. I’d recommend either making this helper private to the module that preserves the invariant, or asserting/returning an error when lengths differ.

  • Panic handling drops data silently after partial progress in crates/net/p2p/src/beacon/column_checks.rs:104 and crates/net/p2p/src/beacon/column_checks.rs:173. I understand the intent, but catching panics and converting them into dropped sidecars/groups means transient storage bugs or invariant violations can cause permanent data loss rather than backpressure/retry. In consensus-adjacent ingestion, that tradeoff is risky. At minimum I’d want stronger telemetry here (group size / block root / fork) and ideally a clearer distinction between malformed peer input and local internal failure.

  • Performance/readability note: group_by_block in crates/net/p2p/src/beacon/column_checks.rs:75 is fine functionally, but it materializes Vec<Vec<_>> for the whole batch before any work starts. Since forwarding happens per group anyway, a streaming approach would reduce peak memory and simplify the task-fan-out concern above.

  • What looks good

    • kzg_verdicts in crates/blockchain/state_transition/src/beacon/gossip/column.rs:428 preserves correctness by falling back to per-sidecar verification on batch failure, which is the right safety property.
    • Sharing SignatureMemo across sidecars of one block is a sensible optimization and does not appear to weaken signature validation semantics.
    • Grouping by block root/header root matches the batching assumption for shared KZG/signature work.
  • Validation note

    • I attempted a targeted test run, but the environment’s Rust toolchain cache is read-only, so cargo test could not execute here.

Automated review by OpenAI Codex · gpt-5.4 · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Kimi Code Review

I'll review this PR which introduces KZG batch verification for data column sidecars grouped by block. Let me analyze the changes systematically.

Overall Assessment

This is a well-structured optimization that batches KZG proof verification across multiple data column sidecars of the same block, reducing cryptographic overhead. The design is sound but I have several concerns ranging from correctness edge cases to API safety.


Critical Issues

1. SignatureMemo key uses BlsSignature which may not implement Hash/Eq correctly for caching

File: crates/blockchain/state_transition/src/beacon/gossip/column.rs, lines 841-870

let key = (
    header.hash_tree_root(),
    sidecar.signed_block_header.signature,
    head,
);

Problem: BlsSignature is likely a wrapper around byte arrays or cryptographic types. The Hash and Eq implementations must be canonical—bitwise equality of the serialized form. If BlsSignature uses compressed representation with possible non-canonical encodings (e.g., different byte strings that represent the same signature point), memoization could fail to match equivalent signatures or worse, match distinct ones.

Recommendation: Verify BlsSignature's Hash and Eq derive on the compressed serialization, not any internal representation. Add a comment asserting this invariant.


2. cell_indices construction in verify_column_cells_batch may overflow usize on 32-bit systems

File: crates/blockchain/state_transition/src/beacon/fork_choice.rs, lines 4150-4155

let total: usize = columns.iter().map(|column| column.column.len()).sum();

Problem: On 32-bit systems, usize is u32. With many columns × many cells per column, this could overflow. The subsequent Vec::with_capacity(total) would panic or allocate incorrectly.

Recommendation: Use checked_sum() or explicit overflow check, or document the 64-bit platform assumption. Given this is consensus code, defensive programming is warranted:

let total: usize = columns.iter()
    .map(|column| column.column.len())
    .try_fold(0usize, |acc, len| acc.checked_add(len))
    .ok_or_else(|| Error::SpecAssert("column cell count overflow"))?;

3. catch_unwind around chain_precheck_for_with may unwind across FFI boundaries unsafely

File: crates/net/p2p/src/beacon/column_checks.rs, lines 115-122

let precheck = catch_unwind(AssertUnwindSafe(|| {
    column::chain_precheck_for_with(store, &sidecar, now_ms, &mut memo)
}));

Problem: AssertUnwindSafe is used to bypass the UnwindSafe bound, but Store contains internal mutability (likely RwLock, Mutex, or atomic operations). If a panic occurs while memo or store internal state is modified, subsequent use is undefined behavior.

More critically: if c_kzg or BLS verification uses FFI (highly likely), unwinding through FFI is undefined behavior in Rust. The AssertUnwindSafe doesn't fix this—it just suppresses the lint.

Recommendation:

  • Ensure c_kzg operations happen outside this catch_unwind (they don't—they're in kzg_verdicts later, which is fine)
  • But store reads may trigger DB operations that use FFI (RocksDB, LMDB, etc.)
  • Consider moving the catch_unwind to only wrap the pure Rust logic, or document that Store operations must be panic-safe

Actually, re-reading: the catch_unwind is only around chain_precheck_for_with, not kzg_verdicts. The precheck does store reads. If the store's get_state or similar panics, AssertUnwindSafe is claiming this is safe. This is a significant safety claim.


Security Vulnerabilities

4. SignatureMemo does not bound memory consumption—potential DoS vector

File: crates/blockchain/state_transition/src/beacon/gossip/column.rs, lines 841-870

Problem: A malicious peer could send many sidecars with unique (root, signature, head) combinations, causing unbounded HashMap growth within a single spawn_blocking task. While the memo is dropped per-task, a single task processing many malicious messages could OOM.

Recommendation: Add a size limit with LRU eviction, or document why this is bounded (e.g., number of sidecars per block is protocol-limited).


5. KZG batch failure fallback does not preserve timing side-channel resistance

File: crates/blockchain/state_transition/src/beacon/gossip/column.rs, lines 1374-1399

if matches!(batch, Ok(true)) {
    return vec![ChainVerdict::Keep; items.len()];
}
refs.iter()
    .zip(items)
    .map(|(sidecar, (_, pending))| kzg_verdict_ref(sidecar, pending))
    .collect()

Problem: The fallback to individual verification on batch failure creates a timing oracle. A malicious block with one forged column causes:

  1. One fast batch failure (cryptographic rejection)
  2. N individual verifications

An observer measuring lean_data_column_kzg_verify_seconds or wall-clock time can distinguish "all valid" from "one forged" based on timing. This leaks information about which columns are invalid.

Recommendation: This is documented behavior and may be acceptable, but consider:

  • Constant-time fallback (always verify all individually, defeating the optimization)
  • Or document this as a known limitation for non-gossip paths (gossip is single-message)

Actually, re-reading: this is for req/resp, not gossip. The doc says "Gossip keeps one message, one verdict, and is not batched." So this timing channel affects range sync. Acceptable with documentation.


Correctness Issues

6. kzg_verdicts short-circuit for single item bypasses batch path but changes metrics behavior

File: crates/blockchain/state_transition/src/beacon/gossip/column.rs, lines 1374-1376

if let [(sidecar, pending)] = items {
    return vec![kzg_verdict(sidecar, pending)];
}

Problem: For a single item, this calls kzg_verdict which does verify_column_cells_batch(&[...]) with a single-element slice. But the metrics comment says "one observation per sidecar if that batch fails." For the single-item case, there's no batch-then-fallback path, so no double-counting. However, the short-circuit means kzg_verdict uses sidecar.cells() with different lifetime constraints.

Actually, this looks correct. But note that kzg_verdict allocates a new Vec for the single-element slice. The optimization is minor.


7. ColumnRef::cells lifetime annotation may be unsound

File: crates/blockchain/state_transition/src/beacon/gossip/column.rs, lines 1294-1312

fn cells(&self, commitments: &'a [KzgCommitment]) -> fork_choice::ColumnCells<'a> {

Problem: The &'a [KzgCommitment] borrows for 'a, but ColumnRef<'a> already borrows the sidecar for 'a. The returned ColumnCells<'a> captures both. However, in kzg_verdicts:

let refs: Vec<ColumnRef<'_>> = items.iter().map(...).collect();
let columns: Vec<fork_choice::ColumnCells<'_>> = refs
    .iter()
    .zip(items)
    .map(|(sidecar, (_, pending))| sidecar.cells(&pending.commitments))
    .collect();

Here refs has anonymous lifetime tied to items, and columns borrows from refs and items. The borrow checker accepts this, but it's fragile. If refs and columns are reordered or refs is dropped early, this fails.

Recommendation: Add a comment explaining the lifetime dependency chain, or refactor to avoid the two-phase construction.


8. group_by_block uses HashMap without deterministic ordering guarantee for tie-breaking

File: crates/net/p2p/src/beacon/column_checks.rs, lines 173-186

fn group_by_block(sidecars: Vec<DataColumnSidecar>) -> Vec<Vec<DataColumnSidecar>> {
    let mut positions: HashMap<Root, usize> = HashMap::new();
    // ...
}

Problem: The test sidecars_are_grouped_by_block_in_first_seen_order verifies first-seen ordering, but HashMap insertion order is preserved in std::collections::HashMap as of Rust 1.36+. However, this is an implementation detail, not a guarantee. If the hasher changes or hashbrown is updated, ordering may change.

Recommendation: Use IndexMap or document reliance on HashMap's insertion ordering. Or use a Vec with linear search for deterministic behavior.


Performance Issues

9. kzg_verdicts allocates refs and columns vectors unnecessarily

File: crates/blockchain/state_transition/src/beacon/gossip/column.rs, lines 1380-1388

let refs: Vec<ColumnRef<'_>> = items.iter().map(|(sidecar, _)| (*sidecar).into()).collect();
let columns: Vec<fork_choice::ColumnCells<'_>> = refs
    .iter()
    .zip(items)
    .map(|(sidecar, (_, pending))| sidecar.cells(&pending.commitments))
    .collect();

Problem: Two intermediate allocations. For the common case of batch success, only columns is needed. For fallback, both refs and columns are needed.

Recommendation: Could construct columns directly with a single pass if ColumnRef is embedded, but the lifetime trickery makes this hard. Acceptable as-is, but consider:

// Single allocation: Vec with capacity for both
let mut columns = Vec::with_capacity(items.len());
// ... build directly

Actually, ColumnRef is needed for fallback. Keep as-is, but consider SmallVec or similar for typical small N.


10. verify_column_cells_batch eagerly converts all cells before checking lengths

File: crates/blockchain/state_transition/src/beacon/fork_choice.rs, lines 4155-4168

for cell in column.column {
    cells.push(
        c_kzg::Cell::from_bytes(&cell[..])
            .map_err(|_| Error::SpecAssert("len(cell) == BYTES_PER_CELL"))?,
    );
}
commitments.extend_from_slice(column.commitments);
proofs.extend_from_slice(column.proofs);

Problem: If commitments.len() != column.column.len() or proofs.len() != column.column.len(), the error from verify_cell_kzg_proof_batch will be cryptic. The doc comment says "Each column's three lists must have equal length, as verify_data_column_sidecar and its gloas counterpart guarantee"—but this is an internal function with a public API.

Recommendation: Add debug assertions or explicit length checks with clear error messages:

assert_eq!(column.commitments.len(), column.column.len(), "commitments mismatch");
assert_eq!(column.proofs.len(), column.column.len(), "proofs mismatch");

Or return a descriptive Error.


Rust Best Practices

11. SignatureMemo::answers uses Result<(), Outcome> as value type—Outcome is already the error case

File: crates/blockchain/state_transition/src/beacon/gossip/column.rs, lines 841-870

answers: HashMap<(Root, BlsSignature, Option<Root>), Result<(), Outcome>>,

Problem: Outcome includes Reject and Ignore variants. Result<(), Outcome> is semantically odd—success is Ok(()), failure carries Outcome. But Outcome::Accept exists too, making this a tri-state.

Actually, looking at usage: header_signature_is_valid returns Result<(), Outcome> where Ok(()) means "signature valid, continue checks" and Err(outcome) means "verdict decided." The memo caches this exactly.

This is acceptable but the type alias or a custom enum would be clearer:

enum SignatureCheck {
    Valid,           // Ok(())
    Decided(Outcome), // Err(outcome)
}

12. PendingKzg has private field but is pub—intentional?

File: crates/blockchain/state_transition/src/beacon/gossip/column.rs, lines 1270-1274

#[derive(Debug, Clone, PartialEq, Eq)]
pub struct PendingKzg {
    commitments: Vec<KzgCommitment>,
}

Problem: Field is private, struct is public. No constructor is public. But chain_precheck_with returns ChainPrecheck::Pending(PendingKzg { commitments: ... }) within the same module, and kzg_verdicts takes &PendingKzg.

External callers can only use PendingKzg as an opaque type in kzg_verdicts. This seems intentional for API control.

Acceptable, but consider pub(crate) or a constructor if tests need it.


13. judge_group drops sidecars on panic without metrics

File: crates/net/p2p/src/beacon/column_checks.rs, lines 115-122

Err(_) => error!("A data column check panicked; dropping the sidecar"),

Problem: The sidecar is silently dropped with no metrics increment. Other drops are counted via count_drop. Panics are exceptional but should be observable.

Recommendation: Add a metric or at least increment a counter for panicked checks.


Documentation Issues

14. Doc comment typo: "gloas" vs "gloas counterpart"

File: crates/blockchain/state_transition/src/beacon/fork_choice.rs, line 4148

/// [`verify_data_column_sidecar`] and its gloas counterpart guarantee.

Should be "its gloas counterpart" → already says "gloas". Actually "gloas" is the fork name. Not a typo. But "gloas" appears as "gloas" elsewhere. Consistent.

Wait: "gloas" vs "gloas"—the diff shows gloas::DataColumnSidecar. The comment says "gloas counterpart". This is consistent with codebase naming. Acceptable.


15. kzg_verdicts doc comment says "one per entry and in order" but doesn't specify behavior on empty input

File: crates/blockchain/state_transition/src/beacon/gossip/column.rs, lines 1352-1368

pub fn kzg_verdicts(items: &[(&AnyDataColumnSidecar, &PendingKzg)]) -> Vec<ChainVerdict> {

Problem: Empty slice returns empty vec (correct), but not documented. The short-circuit for [(single)] is an optimization leak in docs.

Recommendation: Document: "Returns an empty vec for empty input. For a single item, delegates to kzg_verdict."


Minor Issues

16. settle function could be #[inline]

File: crates/blockchain/state_transition/src/beacon/gossip/column.rs, lines 1320-1326

Trivial function, hot path. Mark #[inline].


17. ColumnRef enum derives no traits, manual From impl

File: crates/blockchain/state_transition/src/beacon/gossip/column.rs, lines 1278-1312

Could derive Clone, Copy since it holds references. Not necessary but idiomatic.


Test Quality

18. bench_kzg_batch_against_separate_calls is #[ignore] but uses println!

File: crates/blockchain/state_transition/src/beacon/gossip/column.rs, lines 1684-1720

Standard practice for benchmarks. Acceptable.

19. real_sidecars test helper uses expect with string literals, not unwrap_or_else

File: crates/blockchain/state_transition/src/beacon/gossip/column.rs, lines 1621-1664

In test code, expect is fine. Acceptable.


Summary

Priority Count Categories
Critical 3 Signature hashing correctness, usize overflow, unwind safety
Security 1 Unbounded memo memory
Correctness 3 Lifetime fragility, HashMap ordering, length validation
Performance 2 Unnecessary allocations, eager conversion
Style 5 Various minor improvements

The PR is well-designed and the batching optimization is valuable. The critical issues should be addressed before merge, particularly the BlsSignature hashing correctness and AssertUnwindSafe soundness claims.


Automated review by Kimi (Moonshot AI) · kimi-k2.6 · custom prompt

MegaRedHand added a commit that referenced this pull request Oct 30, 2026
)

A range or by-root answer, or a batch of parked columns replayed after
their parent imports, carries many columns of the same block. Each column
was checked on its own blocking task with its own verify_cell_kzg_proof_batch
call, and the chain actor heard nothing until every sidecar of the whole call
was done, so data availability waited on the slowest block of the batch.

check_and_forward now groups the sidecars by block (the signed header's root
for fulu, beacon_block_root for gloas). A group takes one column_check_permits
permit, runs every sidecar's rules that do not read the cells, then verifies
the proofs of all sidecars still standing in one batch. A batch that fails is
re-run sidecar by sidecar, so exactly the sidecars rejected before are
rejected. The group's Keep and AwaitParent sidecars go to the chain actor as
soon as the group finishes.

Verdicts do not change: the proofs are the last rule on the chain path (the
lookahead rule after them is the gossip path's alone), so chain_checks is now
chain_precheck followed by a KZG step, with gossip's stateful_checks and
stateful_checks_gloas keeping their order through the same split halves.

On this machine a batch over 9-blob columns costs 3.3x less than separate
calls at 8 columns and 4.8x less at 32.

The branch's second commit verifies a block group's shared header signature
once: every fulu sidecar of a block carries the same signed header, so the
group repeated one BLS verification per sidecar, one after another under a
single permit. SignatureMemo remembers header_signature_is_valid's answer for
the group, keyed on the header root, the signature bytes and (for the
head-state check made before parking) the head it read, so a forged signature
over a shared header is judged on its own. Only that answer is remembered;
every other rule runs per sidecar in its order, so verdicts are unchanged. The
memo lives in one group's spawn_blocking closure; gossip and the single-sidecar
wrappers use a fresh one per call.

Adapted onto nightly's #752: its column_check in-use gauge and blocking-task
timer now wrap one block group's closure instead of one sidecar's, and
docs/metrics.md says a column_check permit and task cover a block's group.

Cherry-picked from perf/column-kzg-batch-per-block at abe310a.
@MegaRedHand

Copy link
Copy Markdown
Collaborator Author

Block import benchmark (offline replay of mainnet blocks, steady state)

The replay feeds blocks only: no data columns are fetched or verified, so this PR's code path does not run there. The leg served as a noise control.

leg STF per block difference
main 012dc4f 157.7 ms
main + #754 156.9 ms -0.8 ms (-0.5%), within the ±1.5% round-to-round noise

Measuring this PR needs a live node or a replay that also fetches columns.

Method

@MegaRedHand MegaRedHand added performance Performance improvements or possible performance improvements beacon Ethereum Beacon Chain client labels Nov 3, 2026
MegaRedHand added a commit that referenced this pull request Nov 3, 2026
…events layout (#765)

Opt-in per-block telemetry: with --telemetry.clickhouse-url (or
ETHLAMBDA_TELEMETRY_CLICKHOUSE_URL), a node writes one ClickHouse row per
point in each block's life, and per data column arrival, in Silver's
block_events table layout, so both clients' timings can be compared block by
block from one table. Off by default: without the flag, emit builds nothing.
See docs/telemetry.md.

Cherry-picked from feat/clickhouse-block-events at 119e70e.

Adapted to three nightly-only changes the PR's base (main 4992d8d) lacks:

- #755 sends engine_newPayload from a task while the state transition runs
  speculatively, so the `engine` pair is only the wait left afterwards. The
  el_sent and el_verdict rows now come from new engine_sent/engine_answered
  instants that bracket the request itself, and the speculative transition
  writes stf_spec_start/stf_spec_end. docs/telemetry.md describes the overlap.
- #760 answers a block whose stored columns are complete with
  DataAvailability::ColumnsVerified, which also counts as columns required,
  so such a block writes da_available rather than da_check_end.
- #754 checks a block's sidecars as one group with a batched KZG, so the
  column rows are written from GroupVerdicts::record, once each sidecar's
  verdict is known.

@pablodeymo pablodeymo left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found no wrong-verdict bug: rule order is preserved on every path, the memo key covers what the signature answer depends on, and a failed batch falls back to per-sidecar verdicts.

The comments are about what moving to one thread and one permit per block costs, then tests and cleanup. Read from the diff; I did not build or run it.

The ones I would act on before merging:

  1. A group's size is unbounded, so one peer answer can force a very large batch and a long serial fallback (group_by_block).
  2. The header-level work other than the signature (advanced proposer, finalized ancestry, gloas block decode) now runs serially per sidecar (judge_before_kzg).
  3. One blocking thread and one long-held permit per block, on a pool shared with envelope checks and gossip-queued sidecars (check_and_forward).

The rest are smaller: panic isolation in the KZG stage, the per-column length check in verify_column_cells_batch, two stale comments outside the diff, test gaps, and a few simplifications.

}

/// Group `sidecars` by block, in the order each block first appears.
fn group_by_block(sidecars: Vec<DataColumnSidecar>) -> Vec<Vec<DataColumnSidecar>> {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A group has no size bound. The column response reader accepts up to max_request_data_column_sidecars() = 16,384 chunks, retain_requested_columns filters on root and index only, and the by-range handler on slot only, so neither removes a repeated (block, index). has_data_column cannot catch a repeat inside a group either, since nothing of the group is stored until the group finishes.

A peer can answer with 16,384 sidecars of one block: a real header, commitments and inclusion proof, with low-entropy cells that compress to very little on the wire. They all land in one group. verify_column_cells_batch then copies every cell into one Vec (about 700 MB at 21 blobs, on top of the decoded response), the batch fails, and the fallback re-verifies all 16,384 serially on one thread before any honest column of that block is forwarded. Before, at most 16 sidecars' cells were copied at a time.

Suggest capping a batch at a bounded size (NUMBER_OF_COLUMNS at most) and handling a repeated (block, index) within one answer explicitly.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 0ec7ceb and e775465. group_by_block now keeps the first sidecar of each (block, index) and counts the rest under already_seen in lean_data_columns_rejected_total. The prechecks refuse an index past NUMBER_OF_COLUMNS before the KZG stage, so at most 128 sidecars of one block reach it, and those are verified in sub-batches of at most 16 (KZG_BATCH_COLUMNS). That bounds both the cell copy into one batch and a failed batch's one-by-one re-check at 16 columns. Keeping the first copy is the one verdict this changes: a valid copy that follows an invalid copy of the same column in the same answer used to be kept and is now dropped unjudged. Only a faulty peer sends a repeat, and the chain actor's parking already dedups the same way (first wins). a_repeated_block_and_index_is_kept_once covers it with 1,000 copies.

@@ -287,38 +532,49 @@ fn judge_against_parent(
(None, PastLookahead::Queue) => None,
(None, PastLookahead::Advance) => {
let Some(expected) = advanced_proposer(parent_state, header.slot, &config) else {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only the signature answer is memoized, but everything judge_before_kzg decides up to the inclusion proof depends on the signed header alone, which the whole group shares. When the slot is outside the parent state's proposer lookahead, each sidecar clones the parent state and runs process_slots here. finalized_ancestry below repeats its LiveChain scan per sidecar, and on the gloas side gloas_before_kzg decodes the full block (get_signed_block) once per sidecar.

All of that was per sidecar before too, but it ran up to 16 wide. It is now serial on the group's one thread, so 8 custody columns wait for 8 state advances before the block's first column reaches the chain actor (128 for a supernode), and that happens exactly when the chain has gaps.

Judging the header once per group (parent state, NotAfterParent, signature, expected proposer, ancestry) and leaving structure, AlreadyStored, inclusion proof and KZG per sidecar would fix this. It would also remove the SignatureMemo parameter from the six functions it is threaded through, including the gossip path where it can never hit.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in b3f541b. ChainPrechecks now judges everything a fulu signed header decides once per group: the parent's post-state (or the head-state check before parking), then NotAfterParent, signature, proposer including advanced_proposer, and finalized ancestry. It keys that on the header's root and signature, so a forged signature over the same header is still judged on its own. A gloas block is decoded once per root. Structure, clock, finality, AlreadyStored, the inclusion proof and the gloas slot and shape rules stay per sidecar, in the same order. The shared answer is only worked out once a sidecar needs it, so a group that is all already stored reads no state. SignatureMemo is gone from all six functions, gossip included. Tests:

  • a_fulu_group_gets_the_verdicts_each_sidecar_gets_alone checks that a mixed group gets exactly what chain_checks_for gives each sidecar alone, with the header judged once per signature.
  • a_group_that_stops_early_never_judges_its_header covers the lazy part.

for group in group_by_block(sidecars) {
// The semaphore is never closed, so this only returns once a
// permit is free.
let Ok(permit) = permits.clone().acquire_owned().await else {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A permit is now held for a whole block's prechecks, batch and fallback instead of one sidecar, and the pool is shared. Envelope checks take one permit per envelope from it (envelope_checks.rs:100), and range sync holds its next batch until those are forwarded. Every gossip Queue/Overloaded sidecar also comes through here as a group of one (verdict.rs:358).

Permits used to turn over in a few milliseconds. With 16 large groups in flight, or one group in its serial fallback, each waiter now waits for a whole group to finish. A separate pool for envelopes might be worth it, or at least a measurement of how long a permit is held during a supernode range batch.

COLUMN_CHECK_PERMITS' doc in lib.rs still says "How many data column sidecars may be in the chain checks at once ... A sidecar arriving with none free waits for one", which is no longer what 16 bounds.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Partly addressed in e775465. The proofs now hold a permit for one sub-batch of at most 16 columns, where they used to hold one for a whole block plus its fallback. One sub-batch measures 12.6 ms on one thread at 9 blobs. The prechecks still run a whole group in turn on the group's permit, and I haven't measured how long that holds it on a live node. Nothing waits for a permit while holding one, and tokio's semaphore serves waiters in order, so an envelope waits behind only what was queued before it. I rewrote COLUMN_CHECK_PERMITS' doc to say what a permit covers now. I didn't split out an envelope pool: the proofs' share of a hold is bounded, and I'd rather decide on numbers than guess. lean_data_column_kzg_verify_seconds now times exactly one sub-batch, which is the hold for every sub-batch after a block's first. A supernode follower (or #752's contention metrics) would show whether the precheck hold makes a separate pool worth it.

continue;
let blockchain = blockchain.clone();
tokio::spawn(async move {
let checked = tokio::task::spawn_blocking(move || {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One blocking thread per block, however many permits are free. By the description's own measurement a batch is 3.3x cheaper at 8 columns and 4.8x at 32, but the per-sidecar checks it replaces ran up to 16 wide.

For a single-block call with idle permits (a by-root answer, a parked-column replay), 8 columns took about one verification of wall-clock and now take 8 / 3.3, about 2.4. 32 columns go from 2 to about 6.7, and 128 from 8 to about 27 if the 4.8x holds, plus the serial prechecks.

Total CPU is lower, but these are the tip-following paths where a held block or envelope is waiting on the columns. Splitting a large group into a few sub-batches across the free permits would keep most of both gains. If the wall-clock at 8 columns is not a concern this is fine to leave, but the trade is worth stating in the description.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in e775465 for blocks with more than 16 pending columns. They're split into near-equal sub-batches of at most 16, the first on the group's permit and each further one on its own. Measured on one thread at 9 blobs, a batch costs about 3 ms plus 0.55 ms per column (3.2 ms for one column, 7.9 ms for 8, 12.6 ms for 16, 73 ms for 128). Computed from those, assuming free permits and cores, 128 columns now take about 13 ms of wall-clock against 26 ms before the PR and 73 ms as one batch, for 101 ms of CPU against 414 ms and 73 ms. I kept the trade at the small end: 16 columns or fewer stay one batch, so a node's 8 sampled columns take 7.9 ms where separate checks would take 3.2 ms, for a third of the CPU. That's stated in the description and in beacon_wire.md. The bench now covers 1 to 128 columns (6805df6).

.iter()
.map(|(sidecar, kzg)| (sidecar, kzg))
.collect();
let kzg_verdicts = column::kzg_verdicts(&items);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

kzg_verdicts runs outside the catch_unwind, so a panic here unwinds through verdicts and takes the whole group with it. That includes sidecars already settled as AwaitParent, which are then never parked, and the drop outcomes, which are never counted.

The comment on the JoinError arm below says losing the group "is what dropping it would have done anyway", which is not true of those. It is unlikely to fire, but either wrap this call too and forward what was already settled, or say so in the comment.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in e775465. The KZG proofs now run on their own spawn_blocking per sub-batch. By then, the prechecks' parked and dropped sidecars have already been forwarded and counted. A panicking sub-batch loses only its own sidecars, uncounted, and the group's other sub-batches are unaffected. The comments on both JoinError arms now say exactly what each one loses.


/// Run the chain checks on `sidecars`, then send the chain actor the ones that
/// passed as one batch and the ones waiting on a block as another.
/// passed as one batch per block and the ones waiting on a block as another.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two comments outside the diff still describe the old contract.

handle_data_column_sidecars_range_response in handlers.rs justifies its check_and_forward call with "One batch through the checks, which forward what passes as one message ... a message per sidecar would put that many mailbox hops between the answer and the imports waiting on it". A 128-slot range answer is now up to 256 chain-actor messages, one keep batch and one awaiting-parent batch per block.

CLAUDE.md's data-columns bullet still says these sidecars go through column::chain_checks, with no mention of the per-block grouping or the precheck/batch split.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both fixed in 5f8c9c6. The range handler's comment now describes forwarding per block, one message for what passes and one for what waits on a block, and says why that is still the point. The CLAUDE.md bullet now covers the grouping, ChainPrechecks, the sub-batches and the verdict guarantee. I also rewrote this PR's own beacon_wire.md paragraph and the metrics.md KZG row, which the rework had made stale.

ChainVerdict::Keep,
]
);
let one_by_one: Vec<_> = items

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This compares kzg_verdicts with kzg_verdict, which is also verify_column_cells_batch (over one column), so it shows the fallback agrees with itself rather than with the old check. Comparing against fork_choice::verify_data_column_sidecar_kzg_proofs, which gossip still uses, would pin "verdicts do not change".

Two more gaps:

  • No p2p test reaches Forwarded::Checked: every check_and_forward test ends in AwaitingParent or a drop. So judge_group's pairing of pending sidecars with the batch's verdicts is unexercised.
  • These tests build PendingKzg by hand, so nothing runs chain_precheck's own Pending output into the batch, and there is no gloas batch case.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in b3f541b and e775465:

  • The fallback test now compares against fork_choice::verify_data_column_sidecar_kzg_proofs (and the gloas one), not against kzg_verdict.
  • a_fulu_group_gets_the_verdicts_each_sidecar_gets_alone builds a real block's sidecars (verified_sidecars, signed by the expected proposer, finalized ancestry passing). It mixes in malformed, already stored, forged-signature, bad-inclusion-proof and bad-cell columns, runs them through ChainPrechecks and one kzg_verdicts batch, and asserts the result equals chain_checks_for one by one, with the good ones Keep.
  • a_gloas_group_reads_its_block_once_and_keeps_what_verifies is the gloas batch case.
  • In p2p, a_blocks_columns_that_verify_are_forwarded_together_across_sub_batches reaches Forwarded::Checked: 18 gloas columns in two sub-batches, a forged column in the second, and every other column back in one batch, in order.


/// The block a sidecar belongs to: a fulu sidecar's signed header's root, a
/// gloas sidecar's `beacon_block_root`.
fn block_of(sidecar: &DataColumnSidecar) -> Root {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DataColumnSidecar::block_root() already does this with the same match, and retain_requested_columns uses it. Calling it here avoids a second definition of which block a sidecar belongs to.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 0ec7ceb. group_by_block uses block_root() and the local helper is gone.


/// What [`chain_precheck`] found out without reading a cell.
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum ChainPrecheck {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: this is Result<PendingKzg, ChainVerdict> under another name. The module already writes "verdict or carry on" as a Result with ? (judge_before_kzg, gloas_before_kzg). A Result here would let chain_precheck_with and chain_precheck_gloas use ? instead of repeating return ChainPrecheck::Settled(ChainVerdict::Drop(Outcome::...)) at every early return, and would remove settle's match.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in b3f541b. ChainPrechecks::check returns Result<PendingKzg, ChainVerdict>, the early returns are plain Errs, and settle is gone.

}
// [IGNORE] The sidecar's slot is outside the parent state's proposer
// lookahead window (MAY queue).
if past_lookahead == PastLookahead::Queue

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After the split, judge_against_parent has one caller, stateful_checks, which always passes PastLookahead::Queue (chain_precheck_with calls judge_before_kzg directly). So this guard is always true and the parameter is dead in this function.

The doc above still says the function is "shared by stateful_checks and chain_checks". Dropping the parameter, or inlining this tail into stateful_checks, would make the gossip-only path read as gossip-only.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in b3f541b. judge_against_parent is gone and its tail (inclusion proof, KZG, lookahead) is inlined into stateful_checks, which now reads as gossip-only. The rules both paths share up to the inclusion proof are now judge_header_against_parent, where PastLookahead is live: gossip passes Queue, the chain path Advance. Its doc names its two callers.

…ells_batch

Concatenating the columns left only the totals to compare, so a column
with two cells and one commitment beside one with one cell and two
commitments paired a cell with its neighbour's commitment and could
verify. Each entry's cells, commitments and proofs must now agree, an
Err otherwise, as c-kzg answers for one column on its own.

verify_column_cells now delegates to the batch function over one column,
so gossip and the chain checks share one copy of the cell conversion.
The sub-batch size the chain checks split a block's columns into is read
off this curve, so it now covers every count a block can have, and
reports the fastest of 15 runs rather than the median of 5, which a
loaded machine skewed by more than the differences being measured.
…group

Only the header signature was remembered across a block's sidecars;
everything else that reads nothing but the signed header ran for every
sidecar, serially on the group's one thread: the parent state read, the
head-state check before parking, NotAfterParent, the proposer (a
process_slots on a state clone past the lookahead) and the finalized
ancestry walk. On the gloas side every sidecar decoded the whole block.

ChainPrechecks now runs those once per signed header (root and
signature, so a forged signature over a block's header is still judged
on its own) and the gloas block lookup once per block root, worked out
the first time a sidecar needs them, so a group that stops earlier (all
already stored, say) reads no state. The structure, clock, finality,
AlreadyStored, inclusion proof and gloas slot/shape rules stay per
sidecar, in their place.

Its check returns Result<PendingKzg, ChainVerdict>, replacing the
ChainPrecheck enum and settle. SignatureMemo is gone, and with it the
parameter threaded through the gossip path, where it could never hit.
judge_against_parent is folded into stateful_checks, which is its only
caller, so the gossip tail (inclusion proof, KZG, lookahead) reads as
gossip's; the header rules both paths share are
judge_header_against_parent, where PastLookahead still picks between
them.

Tests: a mixed fulu group (good, malformed, stored, forged signature,
bad inclusion proof, bad cell) judged together equals chain_checks_for
one by one, through a real block's sidecars and a KZG batch that keeps
the good ones; a gloas group reads its block once and keeps what
verifies; a group that stops early judges no header; and the fallback
test now compares against verify_data_column_sidecar_kzg_proofs, the
check gossip runs.
A group had no size bound. The column response reader accepts up to
16,384 chunks, and neither the by-root filter (root and index) nor the
by-range one (slot) drops a repeated (block, index), so one answer could
put thousands of copies of one block's column into one group: all of
their cells copied into one KZG batch, and on failure all re-verified
one by one before any honest column of the block was forwarded.

group_by_block now keeps the first sidecar of each (block, index) and
reports how many repeats it left out, counted under already_seen in
lean_data_columns_rejected_total. A block has NUMBER_OF_COLUMNS columns
and the prechecks refuse an index past that, so no group sends more
than that many sidecars to the KZG stage. The first copy is kept, as
the chain actor's own parking does: an honest answer never repeats a
column, so the only verdict this changes is a faulty peer's later copy.

Groups by DataColumnSidecar::block_root() instead of a second
definition of which block a sidecar belongs to.
One blocking thread and one permit per block meant a block's columns
were verified serially however many permits were free, and a permit,
on a pool the envelope checks share, was held for a whole block's
prechecks, batch and fallback. Measured locally (9 blobs, fastest of
15, one thread), one column costs 3.2 ms, a batch of 8 7.9 ms, of 16
12.6 ms and of 128 73 ms: about 3 ms plus 0.55 ms per column.

A group is now checked in two stages. The prechecks run on the permit
the group took, and the sidecars they settle (parked or dropped) are
forwarded at once. The KZG proofs of the rest are split into near-equal
sub-batches of at most KZG_BATCH_COLUMNS (16): the first runs on the
group's permit, each further one waits for a permit of its own, and the
group holds none while it waits, so the pool cannot deadlock. The kept
sidecars still reach the chain actor as one batch per block.

What this trades, computed from those timings assuming free permits
and cores: a block of 16 columns or fewer is still one batch, so a
node's 8 sampled columns take 7.9 ms of wall-clock where 8 separate
checks 16 wide would take 3.2 ms, for a third of the CPU. 128 columns
are 8 sub-batches: about 13 ms against 73 ms as one batch and 26 ms as
separate checks, at 101 ms of CPU against 73 ms and 414 ms. A block's
proofs no longer hold one permit from first column to last, and a
failed batch's one-by-one re-check covers 16 columns at most. The
prechecks still run a group's sidecars in turn on one permit; how long
that holds it on a live node is not measured.

A panicking KZG sub-batch now loses only its own sidecars: what the
prechecks settled has already been forwarded and counted, and the
group's other sub-batches are unaffected. The comments say exactly
what each panic arm loses.

COLUMN_CHECK_PERMITS' doc says what a permit now covers. The p2p crate
takes ethlambda-state-transition's test-utils as a dev-dependency, as
rpc does, for a stored post-state in the new test, which runs a gloas
block's columns through two sub-batches with a forged one in the second
and expects every other column back in one Checked batch.
The range response handler justified its check_and_forward call with
the old contract, one message for the whole answer; it is now up to two
per block, and the comment says why that is still the point. CLAUDE.md's
data-columns bullet named only column::chain_checks, with no grouping,
no precheck/batch split and no sub-batches.

beacon_wire.md's paragraph on checking fetched columns now covers the
(block, index) repeats, the shared rules judged once per group, the
sub-batches and what their size trades, with the measured figures;
metrics.md's KZG histogram row says it times one sub-batch, which for
every sub-batch after a block's first is how long it holds its permit.
@MegaRedHand

Copy link
Copy Markdown
Collaborator Author

Going through the review body in order:

  1. Unbounded group: a repeated (block, index) is now kept once, the first copy (0ec7ceb), and the KZG stage verifies at most 16 columns per batch (e775465). Dropping later copies unjudged is the one verdict change in this round. It only affects a faulty peer's answer, and the chain actor's parking already dedups the same way.
  2. Serial header-level work: ChainPrechecks judges a fulu signed header (or a gloas block lookup) once per group, lazily, and SignatureMemo is gone (b3f541b).
  3. One thread and a long-held permit per block: the KZG proofs now run in sub-batches of at most 16 columns across permits (e775465).
    • Measured on one thread at 9 blobs: one column takes 3.2 ms, 8 columns 7.9 ms, 128 columns 73 ms.
    • Computed from those, assuming free permits and cores: 128 columns take about 13 ms of wall-clock (26 ms before the PR, 73 ms as one batch), and 8 columns stay one batch at 7.9 ms (3.2 ms before) for a third of the CPU.
    • The prechecks still run a whole group on one permit, and that hold isn't measured.
    • I didn't split out an envelope pool.

The smaller points are answered in their threads: panic isolation in the KZG stage (e775465), the per-column length check (ab21359), the stale comments (5f8c9c6), the tests, and the three simplifications.

@MegaRedHand

Copy link
Copy Markdown
Collaborator Author

/review

@github-actions

Copy link
Copy Markdown

🤖 Codex Code Review

No actionable findings in the diff. The shared checks retain signature-specific validation, and failed KZG batches fall back to individual verification.

Tests could not run because rustup attempted to write to the read-only /home/runner/.rustup/tmp directory.


Automated review by OpenAI Codex · gpt-6.1-sol · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Claude Code Review

Review: batched data-column chain checks (PR 754)

This PR is careful and well tested. The verdict-equivalence tests (a_fulu_group_gets_the_verdicts_each_sidecar_gets_alone and a_gloas_group_reads_its_block_once_and_keeps_what_verifies) show the main claim: batching changes only how the proofs are paid for, not which sidecars pass.

Things I checked and found correct:

  • Rule order on the gossip path. stateful_checks keeps the same order: header, then inclusion proof, then KZG, then lookahead.
  • Cache keys. The header cache uses (root, signature). A forged signature cannot poison the good header's entry, and the test covers this.
  • Panic handling in ChainPrechecks. or_insert_with stores an entry only after judge_header returns, so it is panic-safe under catch_unwind.
  • Per-entry length check. The check in verify_column_cells_batch closes the cross-column pairing hole, and the test that cancels out lengths is a nice touch.
  • Permit flow. A group task never awaits a permit while holding one, so the permit pool cannot deadlock.

Below are a few issues worth considering, then nits.

Issues

  1. Fallback amplification in kzg_verdicts (gossip/column.rs, kzg_verdicts)

    • One forged column per sub-batch turns a single ~12.6 ms batch into 1 failed batch plus 16 single checks. Using your numbers, that is about 12.6 + 16 × 3.2 ≈ 64 ms, held on one permit.
    • A peer answering a range request can place one bad cell per 16 columns per block, which roughly 5× the CPU cost of honest verification.
    • Previously each bad column cost exactly its own check.
    • Suggestions:
      • Bisect on failure instead of doing a linear re-check. With one bad column this costs about 2·log2(16) batch calls of shrinking size, roughly 2× a batch.
      • At minimum, make sure the peer is penalized on any Reject(Kzg) coming from this path.
  2. PendingKzg is not bound to its sidecar (gossip/column.rs, PendingKzg, kzg_verdict, kzg_verdicts)

    • These are pub and accept any (sidecar, pending) pairing.
    • The p2p caller pairs them correctly today. However, a future caller that zips the wrong lists would verify a gloas sidecar against another block's bid commitments, and no check would catch it.
    • Suggested fix: store the block root and index in PendingKzg, and debug_assert! (or return Err) on a mismatch in kzg_verdict_ref.
  3. Redundant commitment copies

    • In gloas_block, the commitments are copied with to_vec(), and gloas_before_kzg then calls kzg_commitments.clone() again.
    • On the gossip path (stateful_checks_gloas), every sidecar therefore pays two Vec allocations.
    • In ChainPrechecks, every sidecar of a group clones the cached Vec.
    • The cost is small, but Arc<[KzgCommitment]> in both GloasBlock and PendingKzg would make all of these copies free.
  4. The cached AwaitParent covers the whole group (check_fulu)

    • The header judgment is a snapshot taken when the group's first sidecar reaches it.
    • If the parent's post-state lands mid-group, every remaining sidecar is parked on that stale answer. Before this PR, only the sidecars checked before the import were affected.
    • The race window existed before, but it is now as long as the whole group's prechecks.
    • Please confirm that data_column_sidecars_awaiting_parent on the chain actor re-checks whether the parent already has a state when it parks. Otherwise these columns wait for a replay that already happened.
  5. First-copy-wins dedup (column_checks.rs, group_by_block)

    • A malformed or forged first copy shadows a valid later copy of the same (block, index) in the same call.
    • This is fine when a call is one peer's answer.
    • Please verify that no caller of check_and_forward merges sidecars from different sources into one call (parked replays plus gossip-queued, for example). If one does, an attacker's copy could suppress an honest one.
  6. Metric label (metrics.rs)

    • DATA_COLUMN_REJECT_REASONS now seeds "already_seen".
    • Please confirm that Outcome::Ignore(IgnoreReason::AlreadySeen).labels().1 == "already_seen". A mismatch would leave the seeded series at zero forever while the counter increments a different label.

Nits

  • lib.rs, COLUMN_CHECK_PERMITS doc comment: one line now runs past 100 columns (...nor all of a block's proofs. A separate pool from [...]). rustfmt does not rewrap comments.
  • column_checks.rs: the blank line between count_drop and #[cfg(test)] mod tests was removed.
  • check_and_forward: for _ in 0..repeats { count_drop(...) } could be a single inc_by(repeats) if the metrics helper exposes one. With up to ~16k repeats, this is a hot loop for a malicious answer.
  • check_group: the break on acquire_owned failure silently drops the remaining sub-batches. That cannot happen today because the semaphore is never closed, but an error! there would make any future closure visible.

Automated review by Claude · claude-opus-5-5 · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Gemini Code Review

The PR is well-structured and thoroughly tested. Splitting the chain checks into per-block ChainPrechecks and sub-batched KZG verification (verify_column_cells_batch / kzg_verdicts) while releasing the group's permit into the first sub-batch avoids both redundant state/header work and semaphore deadlocks.

One minor edge case to consider:

  1. crates/blockchain/state_transition/src/beacon/gossip/column.rs:484 — Handle empty items in kzg_verdicts
    If kzg_verdicts is called with an empty slice (for example, via group_verdicts when every sidecar in a group is settled during prechecks), it currently falls through to verify_column_cells_batch(&[]) and records a spurious observation on lean_data_column_kzg_verify_seconds. Matching the empty slice up front avoids this:
    pub fn kzg_verdicts(items: &[(&AnyDataColumnSidecar, &PendingKzg)]) -> Vec<ChainVerdict> {
        match items {
            [] => return Vec::new(),
            [(sidecar, pending)] => return vec![kzg_verdict(sidecar, pending)],
            _ => {}
        }

Automated review by Gemini · gemini-4-argon · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Kimi Code Review

No blocking correctness or security issues found. The batching design preserves the per-sidecar verdicts: per-column length checks prevent cross-column pairing, fulu header caching includes the signature to avoid cache poisoning, and failed KZG batches fall back to individual verification. Permit ownership in check_group also avoids holding a permit while acquiring another.

Minor, non-blocking:

  • crates/net/p2p/src/beacon/column_checks.rs, sub_batches (around line 110): the “near equal” distribution can leave a substantially smaller final batch. For example, 100 items become 15,15,15,15,15,15,10, rather than six batches of 14 and two of 15. This does not affect correctness or the maximum batch bound, but quotient/remainder distribution would better match the documentation if balanced sub-batches are intended.
  • check_group (around line 220): a panicked KZG sub-batch loses up to 16 sidecars with only an error log. Since the chain actor trusts forwarded sidecars and never sees these, consider an internal drop metric for observability. This follows the existing panic-isolation policy, so it is not a regression requiring changes.

The new tests provide good coverage of mixed verdicts, forged signatures, duplicate suppression, malformed batch entries, and KZG fallback behavior.


Automated review by Kimi (Moonshot AI) · kimi-k3.1-pro · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Grok Code Review

The batching is sound: per-column length checks block the cross-pairing trick, a failed batch falls back to the same per-sidecar verdicts, and header/block work is cached only after it finishes. A few availability and observability gaps are worth fixing.

  1. group_by_block in crates/net/p2p/src/beacon/column_checks.rs drops every later (block, index) before any check, and keeps the first payload even when the cells differ (the new test locks this in). KZG is binding, so at most one payload can pass, but an invalid first copy makes a later valid copy in the same answer already_seen and never judged. That is worse than “peer sent garbage” if this response is what completes the fetch. Dedupe only byte-identical sidecars, or on reject try the next distinct payload with a small cap (one bad prefix must not hide the valid column; unbounded KZG retries are what you correctly wanted to avoid).

  2. In check_group, acquire_owned() failure breaks and drops the remaining sub-batches with no log, metric, or forward. The semaphore is not closed today, but this path silently loses columns that already passed prechecks. Count and log them before dropping.

  3. A panic from the precheck spawn_blocking task drops the whole group (up to NUMBER_OF_COLUMNS), including sidecars the inner catch_unwind never saw, and does not count_drop. The per-sidecar catch is the right granularity; the outer path should at least log how many sidecars were discarded.

  4. docs/beacon_wire.md misstates the 8-column case (“separate checks 16 wide would take 3.2 ms”). Eight separate checks are about 8 × 3.2 ms; 7.9 ms is roughly a third of that CPU, which is the comparison the rest of the paragraph is making.

verify_column_cells_batch’s per-entry length check is the right fix, and a_batch_entry_whose_lengths_disagree_is_refused_even_when_the_totals_match actually exercises the false-accept. Permit handling does not deadlock: the waiter does not hold a permit while the in-flight sub-batch does. Gossip staying one-message-one-verdict is the right call.


Automated review by Grok (xAI) · grok-4.7 · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 DeepSeek Code Review

Overall this is a careful refactor. The per-sidecar verdict parity between chain_checks/stateful_checks and the new batched path looks correct, and the KZG batch fallback is sound.

A few things I’d consider addressing:

  • Stale shared-header/block verdicts in ChainPrechecks (crates/blockchain/state_transition/src/beacon/gossip/column.rs).
    headers and blocks cache the first result for a block/signature while precheck_group iterates sidecars sequentially. The Store can change during that loop, so a later sidecar may get an older verdict. The main case is AwaitParent: if the parent state/block arrives between two sidecars, the second will still be parked and replayed later, which is safe but adds liveness delay. The comment says the store is “as it stood when that group was judged,” but it is really as it stood at first lookup per key. Worth confirming that is intended; if not, avoid caching AwaitParent or re-evaluate when the cached result is a queue outcome.

  • group_by_block keeps the first copy and drops repeats unjudged (crates/net/p2p/src/beacon/column_checks.rs).
    A malicious/buggy peer can put an invalid copy of (block, index) first and a valid copy second; the valid copy is then counted as already_seen and never checked. An honest peer should never repeat, so this is mainly a liveness trade-off rather than a consensus bug, but it is an untrusted-input path. If this is accepted, it might be worth a one-line note that a repeat is not judged even if the first copy is malformed.

  • Permit-acquire failure can silently drop pending sub-batches (check_group in crates/net/p2p/src/beacon/column_checks.rs).
    In the extra-sub-batch loop:

    let Ok(permit) = permits.clone().acquire_owned().await else {
        break;
    };

    The loop breaks and later forwards only the batches that were already spawned; the rest of the group’s pending sidecars disappear without a log or metric. The semaphore is documented as never closed, so this should not happen in normal operation, but if that invariant ever changes, this would be silent data loss. At minimum, log an error! in the else branch.

  • Gloas commitments are cloned per sidecar (gloas_before_kzg / check_gloas in crates/blockchain/state_transition/src/beacon/gossip/column.rs).
    gloas_before_kzg returns Ok(kzg_commitments.clone()), and check_gloas puts that into a new PendingKzg for every sidecar. For a block with many columns and blobs this is the same full commitment list copied repeatedly. It is not huge, but an Arc<[KzgCommitment]> or a slice borrowing from the cached block would avoid the duplicated allocations.

  • Panic handling in precheck_group drops the sidecar without a rejection metric (crates/net/p2p/src/beacon/column_checks.rs).
    The old per-sidecar code behaved similarly, so this is not a regression, but with the new group/caching structure it would be useful to include a metric or at least count the panicked sidecar, since it is otherwise invisible except for the error log.

No consensus-critical issues found in the KZG batch verification, inclusion-proof ordering, or the fulu/gloas logic.


Automated review by DeepSeek · deepseek-v4-pro · custom prompt

sub_batches took the ceiling of the per-batch share as every batch's
size, so the last batch took whatever was left: 100 columns came out
as six batches of 15 and one of 10, not the "as near equal as that
allows" its doc promised. It now gives the first n % count batches one
column more than the rest, so 100 columns are two of 15 and five of 14.
The count and the bound of 16 per batch are unchanged.

The sub-batch test now pins the 100-column split.
Three paths in the column checks lost sidecars with no more than an
error log: a sidecar whose own prechecks panicked, a whole group whose
precheck task panicked, and a KZG sub-batch whose task panicked. A
fourth, the sub-batch loop's break when a permit cannot be had, lost
the group's remaining unverified sidecars without even a log. That
cannot happen while the semaphore is never closed, but it would be
silent if it ever were.

All four now count what they drop under internal in
lean_data_columns_rejected_total, and the three task-level ones log
how many sidecars they lost.

The repeats of a (block, index) are counted in one increment rather
than one per repeat, since a hostile answer can carry thousands:
inc_data_column_rejected takes a count.

Also: group_by_block's doc says outright that a repeat is not judged
even when the first copy fails, and that every caller hands over one
source's sidecars; the COLUMN_CHECK_PERMITS doc no longer runs past
100 columns; and the blank line before the tests module is back.
…itment copy

kzg_verdicts with no items fell through to an empty
verify_cell_kzg_proof_batch call, which still added an observation to
lean_data_column_kzg_verify_seconds. It now returns at once. No caller
in the node passes an empty slice (sub-batches are never empty), but
the function is public.

gloas_before_kzg returned an owned copy of the bid's commitments, which
gloas_block had already copied out of the block, so gossip's
stateful_checks_gloas paid two copies per sidecar where main paid one.
It now borrows them from the looked-up block: gossip checks against the
borrow, and only ChainPrechecks copies them into the PendingKzg.

ChainPrechecks' doc said it remembers the store as it stood when the
group was judged; each answer is the store as it stood when that answer
was first worked out. The doc now says so, and that a parent whose
post-state lands mid-group costs the rest of the group a round trip
through park_data_columns, which sends a judgeable sidecar straight
back, not the sidecars.
beacon_wire.md put the batch's 7.9 ms of wall-clock beside the 3.2 ms eight separate checks take side by side, then said "for a third of the CPU" with no CPU figure, which read as eight separate checks costing 3.2 ms in all. It now gives both figures for both shapes: the batch is 7.9 ms of wall-clock and of CPU, the separate checks 3.2 ms of wall-clock and 25.8 ms of CPU.
@MegaRedHand

Copy link
Copy Markdown
Collaborator Author

Round three (the automated review of 5f8c9c6). Fixed in 87c266a, c17eeed, 11b312e and e335380. cargo fmt --all -- --check and cargo clippy --workspace --all-targets -- -D warnings are clean. cargo test --profile release-fast -p ethlambda-state-transition -p ethlambda-p2p passes: state-transition 647 (16 ignored), p2p 375 (1 ignored).

Fixed

  • Kimi 1, uneven sub-batches: sub_batches gave every batch the ceiling share and the last one the rest, so 100 columns came out as six of 15 and one of 10, against its own doc. The first n % count batches now take one column more, so 100 columns are two of 15 and five of 14 (87c266a). sub_batches_are_bounded_near_equal_and_in_order pins the 100-column split.

  • Grok 2 and 3, DeepSeek 3 and 5, Kimi 2, Claude's break nit, lost sidecars that went uncounted: four paths dropped sidecars with no more than an error log:

    • a sidecar whose own checks panicked;
    • a group whose precheck task panicked;
    • a KZG sub-batch whose task panicked;
    • the sub-batch loop's break when no permit can be had. This one did not even log. It cannot happen while the semaphore is never closed.

    All four now count what they drop under internal in lean_data_columns_rejected_total, and the three task-level ones log how many sidecars they lost (c17eeed).

  • Claude nits: (c17eeed)

    • Repeats of a (block, index) are now counted in one inc_by, not once per repeat.
    • The COLUMN_CHECK_PERMITS doc no longer runs past 100 columns.
    • The blank line before mod tests is back.
  • DeepSeek 2 and Claude 5, first copy wins: group_by_block's doc now says outright that a repeat is not judged even when the first copy fails. It also says that every caller hands over one source's sidecars (one gossip message, one peer's answer, one block's parked set, the node's own), which I checked at each call site (c17eeed).

  • Gemini 1, empty kzg_verdicts: an empty call fell through to an empty batch and added an observation to lean_data_column_kzg_verify_seconds. It now returns at once (11b312e).

  • Claude 3 and DeepSeek 4, commitment copies: gloas_before_kzg returned an owned copy of commitments gloas_block had already copied, so gossip paid two copies per gloas sidecar where main paid one. It now borrows them, and only ChainPrechecks copies them into a PendingKzg (11b312e).

  • Claude 4 and DeepSeek 1, the shared answer is a snapshot: the ChainPrechecks doc now says each answer is the store as it stood when that answer was first worked out. A parent importing mid-group leaves the rest of the group parked on the earlier answer. That costs a round trip, not the sidecars: park_data_columns asks is_ready_to_judge for every sidecar it is handed and sends a ready one straight back (11b312e).

  • Grok 4, the 8-column sentence in beacon_wire.md: the figures were right (3.2 ms is the wall-clock of eight checks side by side), but the sentence gave no CPU figure. It now gives both for both shapes: the batch takes 7.9 ms of wall-clock and of CPU, the separate checks 3.2 ms of wall-clock and 25.8 ms of CPU (e335380).

Not changed

  • Claude 1, bisecting a failed batch: with one forged column in 16, the linear re-check costs about 64 ms and bisection about 54 ms. Bisection gets worse once several columns are forged. Main checked the same 16 columns one by one for about 51 ms, so a forger gets about 1.25× main's cost. All of these are computed from measured single-thread batch timings.
  • Claude 1, penalizing the peer: real, but already missing on main, which never scored the sender of a bad fetched column either. Tracked in Penalize peers whose fetched data column sidecars fail the chain checks #815.
  • Claude 2, binding PendingKzg to its sidecar: only ChainPrechecks can build one, from the sidecar it is handed, and p2p carries the two as one tuple from then on. A fulu one is the sidecar's own commitments.
  • Claude 6, the already_seen label: IgnoreReason::AlreadySeen labels as "already_seen" (gossip/mod.rs:207), the string seeded.
  • Claude 3, Arc<[KzgCommitment]>: after the change above, each path copies a few KB once per sidecar, as main did, beside a KZG check of 3 ms or more.
  • Grok 1, first copy wins: settled last round as the one deliberate verdict change. A by-root lookup retires on any non-empty answer before the checks run, so [bad, good] ends it exactly as [bad] would.

Codex found no actionable issues; its tests did not run (read-only rustup directory on the runner).

@MegaRedHand

Copy link
Copy Markdown
Collaborator Author

/review

@github-actions

Copy link
Copy Markdown

🤖 Codex Code Review

No actionable findings. The batching preserves per-sidecar validation, isolates invalid KZG proofs through individual fallback checks, and avoids permit deadlocks.

Tests could not run because Rustup attempted to write to a read-only directory.


Automated review by OpenAI Codex · gpt-6.1-sol · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Gemini Code Review

This PR is well-structured and clean. Grouping column sidecars by (block_root, index), sharing header/block prechecks across a block's sidecars via ChainPrechecks, and splitting KZG cell proof verifications into balanced sub-batches of up to 16 columns is a solid design.

A few highlights that look particularly good:

  • Batch alignment safety: verify_column_cells_batch in crates/blockchain/state_transition/src/beacon/fork_choice.rs (lines 4170–4182) validates len(cells) == len(commitments) == len(proofs) per column before concatenating, preventing misaligned lengths across columns from cancelling out.
  • Cache key correctness: ChainPrechecks in crates/blockchain/state_transition/src/beacon/gossip/column.rs (lines 247, 310) keys Fulu header judgments by (header_root, signature), ensuring a sidecar with a forged signature over a valid header is judged independently without poisoning the cached verdict for valid siblings.
  • Permit lifecycle & deadlock avoidance: In crates/net/p2p/src/beacon/column_checks.rs (lines 245–279), the group's initial permit is moved directly into the first KZG sub-batch's spawn_blocking task before acquiring permits for subsequent sub-batches, so no task ever awaits a permit while holding one.

LGTM!


Automated review by Gemini · gemini-4-argon · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Claude Code Review

Review: PR 754, batched KZG verification for chain-checked data columns

The PR is careful and well tested. Verdict equivalence with chain_checks_for is asserted directly, for both fulu and gloas groups. I found no correctness bugs. The notes below are mostly about performance, metrics and robustness.

Correctness

  • Rule order is preserved.
    • Fulu prechecks run in the same order as before: Malformed, FutureSlot, Finalized, AlreadyStored, the header and parent rules, then InclusionProof.
    • Gloas runs block lookup, SlotMismatch, BlockNotGloas, Malformed, in the old order.
    • Gossip stateful_checks keeps signature, proposer, ancestry, inclusion, KZG, then lookahead. PastLookahead::Queue now just skips the proposer rule, and lookahead_covers moved to the caller. That is equivalent.
  • The per-entry length check is the right guard. It is in verify_column_cells_batch (fork_choice.rs ~L4171). Concatenating columns would otherwise let mismatched entries cancel out. Both the reasoning and the regression test are good.
  • The cache key is correct. ChainPrechecks.headers is keyed by (header_root, signature). judge_header reads only the signed header and the store, so every sidecar sharing that key gets the same answer.
  • The panic claim holds. or_insert_with runs the closure before inserting, so a panic leaves the entry vacant.
  • Permits cannot deadlock.
    • check_group holds the group permit across the precheck .await, then moves it into the first sub-batch.
    • Further sub-batches acquire permits while the task holds none.
    • Blocking tasks never await, so there is no deadlock.

Suggestions

  1. gloas_block adds an allocation on the gossip hot path (column.rs, gloas_block). It calls .to_vec() on the bid commitments, and stateful_checks_gloas now pays that per gossip sidecar where it used to borrow. ChainPrechecks also copies them again into each PendingKzg (commitments.to_vec()), giving up to 128 copies per block. Consider:

    • holding Arc<[KzgCommitment]> in GloasBlock, and
    • making PendingKzg.commitments an Arc<[KzgCommitment]>.

    Gloas sidecars of a block would then share one allocation. Fulu could share one as well, since sidecars of the same block carry identical commitments once the inclusion proof passes.

  2. lean_data_column_kzg_verify_seconds now mixes three populations (docs/metrics.md):

    • single-column gossip checks,
    • 1–16-column chain sub-batches,
    • per-column fallback re-verifies.

    Quantiles from this histogram are no longer interpretable. A path=gossip|batch|fallback label, or a separate batch histogram, would keep dashboards meaningful. A columns-per-batch observation would also help tune KZG_BATCH_COLUMNS.

  3. The fallback has a cost amplifier, small but real (kzg_verdicts). One forged column per sub-batch forces 1 batch plus N singles. At 16 columns that is ~12.6 ms + 16×3.2 ms ≈ 64 ms, against ~51 ms for the old per-sidecar path. The amplification is bounded and acceptable. If it ever matters, bisecting the failed batch (log N batch calls) beats N singles when exactly one column is bad.

  4. catch_unwind in precheck_group depends on panic = "unwind". If any release profile sets panic = "abort", the per-sidecar recovery the comments describe is dead code, and a DB .expect() takes the process down. Please confirm the profile, or note the dependency in the comment.

  5. The group cache can go stale, which you document. A fulu AwaitParent or gloas BlockUnknown answer is reused for the rest of the group even if the block imports mid-group. Relying on park_data_columns to send them back is reasonable. Consider a test that pins this round-trip, since correctness of availability now depends on it.

  6. check_and_forward drops groups silently on permit failure. The outer acquire_owned failure path does a bare return, so remaining groups vanish without the internal count that check_group's equivalent branch records. This is pre-existing and unreachable today, but it is inconsistent with the new accounting. Count groups.iter().map(Vec::len).sum() as internal there too.

  7. The chain actor now gets up to 2 messages per block per call, instead of 2 per call. The handler comment justifies the trade-off. Just confirm that new_data_column_sidecars and the availability path don't do per-message work that scales badly with a 64-block range answer.

Nits

  • In verify_column_cells_batch, cell_count is already bound, so std::iter::repeat_n(column.index, column.column.len()) can use it.
  • ColumnRef::cells has identical Fulu and Gloas arms. A small macro or a shared accessor trait would remove the duplication, but it is fine as is.
  • The benchmark test is #[ignore]d with a reason. Good.

Overall the PR is approve-worthy. Item 1 and Item 2 are the ones I'd address before merging.


Automated review by Claude · claude-opus-5-5 · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Grok Code Review

The batching split is sound: per-column length checks, signature-keyed header cache, and the one-by-one fallback on a failed proof batch match the solo verdicts. A few correctness holes remain.

  1. group_by_block (column_checks.rs) dedups on (block_root, index) only, and keeps the first copy unjudged. Gloas store identity is (slot, beacon_block_root, index), and the slot is not committed to by the root. A wrong-slot sidecar followed by the real one is treated as a repeat: the valid column is dropped as already_seen and never checked. Same for two copies that differ in cells — if the first fails KZG, the second is never tried. Include slot in the key (and only skip byte-identical repeats), or keep scanning copies of a (slot, root, index) until one passes. This is safe only if every caller really passes a single source; a mixed vec makes a bad first copy suppress a good one.

  2. kzg_verdicts treats Ok(true) from verify_column_cells_batch as Keep for every entry. A zero-length column adds no tuples, so it is accepted whenever the others verify, and the fallback never runs. A solo verify_cell_kzg_proof_batch on that column can still fail (BADARGS / Ok(false)). That breaks the “batch success means every column would pass alone” invariant. Reject empty entries in verify_column_cells_batch (~4183 in fork_choice.rs), or do not take the all-Keep shortcut when any column is empty. Structural checks may hide this on the chain path; the public batch helper still has the hole. The mismatched-length guard next to it is the right check and the test covers the canceling-lengths case.

  3. In check_and_forward, a closed column_check_permits semaphore returns and drops every group not yet spawned, with no count_drops(..., Internal). check_group accounts for that case. Same path should count group.len() for the remainder.

Otherwise the permit protocol is consistent (no acquire while holding a permit, FIFO, first sub-batch reuses the group permit), and ChainPrechecks correctly keys fulu on (header_root, signature) so a forged signature is not reused. Caching a parent-unknown answer for the rest of the group is fine given the immediate park replay.


Automated review by Grok (xAI) · grok-4.7 · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Kimi Code Review

No blocking correctness or security issues found. The KZG batching preserves the important invariants: lengths are checked per column before flattening, cells/commitments/proofs remain paired correctly, and a failed batch falls back to per-column verification so valid columns are not rejected alongside a forged one. The fulu cache key also correctly includes the signature, preventing a forged signature from poisoning verdicts for the valid signed header. Permit handoff in check_group avoids waiting for a sub-batch permit while holding another.

Two non-blocking concerns:

  • First-copy-wins deduplication can delay availability — crates/net/p2p/src/beacon/column_checks.rs, group_by_block. If the first (block, index) copy is malformed or has bad proofs and a later copy is valid, the valid copy is discarded unjudged. This is documented and does not give a peer new censorship power—it could simply withhold the column—but it does require a later refetch/replay before the block can become available. Ensure the fetch path reliably retries columns lost this way; alternatively, treat repeated identifiers as a malformed peer response and apply the appropriate response-level handling.

  • Precheck permit occupancy remains the main starvation risk — precheck_group/check_group in the same file. One permit covers up to NUMBER_OF_COLUMNS structural and inclusion-proof checks, plus the shared state lookup and potentially an expensive advanced_proposer state advance. Because this pool is shared with envelope checks, a supernode processing several large groups could delay envelopes substantially even though KZG occupancy is now bounded. The documentation acknowledges that this is unmeasured; consider adding a precheck-duration/permit-hold histogram before relying on the current pool size under load.

The regression tests are particularly strong: mixed-verdict groups are compared against standalone checks, forged-cell attribution is tested, and multi-sub-batch forwarding is covered.


Automated review by Kimi (Moonshot AI) · kimi-k3.1-pro · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 DeepSeek Code Review

⚠️ Review failed: DeepSeek returned an empty content field (finish_reason=length, reasoning_content present=yes). Raise max_tokens or lower reasoning_effort ('none' disables thinking).


Automated review by DeepSeek

@MegaRedHand
MegaRedHand added this pull request to the merge queue Nov 19, 2026
Merged via the queue into main with commit d092cd1 Nov 19, 2026
15 checks passed
@MegaRedHand
MegaRedHand deleted the perf/column-kzg-batch-per-block branch November 19, 2026 19:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

beacon Ethereum Beacon Chain client performance Performance improvements or possible performance improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants