Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
8d41b83
perf(p2p): verify one KZG batch per block for fetched data columns
MegaRedHand Oct 30, 2026
abe310a
perf(p2p): verify a block group's shared header signature once
MegaRedHand Oct 30, 2026
0ee30f2
Merge main into perf/column-kzg-batch-per-block
MegaRedHand Nov 4, 2026
2b20cb7
Merge main into perf/column-kzg-batch-per-block
MegaRedHand Nov 11, 2026
1101593
Merge main into perf/column-kzg-batch-per-block
MegaRedHand Nov 18, 2026
ad33f09
Merge main into perf/column-kzg-batch-per-block
MegaRedHand Nov 18, 2026
5a3aab4
Merge main into perf/column-kzg-batch-per-block
MegaRedHand Nov 18, 2026
c738374
Merge main into perf/column-kzg-batch-per-block
MegaRedHand Nov 18, 2026
ab21359
fix(state-transition): check each column's lengths in verify_column_c…
MegaRedHand Nov 19, 2026
6805df6
test(state-transition): time the column KZG batch from 1 to 128 columns
MegaRedHand Nov 19, 2026
b3f541b
perf(state-transition): judge a block's shared column rules once per …
MegaRedHand Nov 19, 2026
0ec7ceb
fix(p2p): keep one sidecar per block and column index in a check call
MegaRedHand Nov 19, 2026
e775465
perf(p2p): verify a block's column proofs in sub-batches across permits
MegaRedHand Nov 19, 2026
5f8c9c6
docs: describe the per-block column checks where they are explained
MegaRedHand Nov 19, 2026
87c266a
fix(p2p): split a block's columns into sub-batches one size apart
MegaRedHand Nov 19, 2026
c17eeed
fix(p2p): count the column sidecars a panic or a closed pool drops
MegaRedHand Nov 19, 2026
11b312e
refactor(state-transition): skip an empty KZG batch and a second comm…
MegaRedHand Nov 19, 2026
e335380
docs: separate the wall-clock and CPU comparisons for an 8-column batch
MegaRedHand Nov 19, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -347,7 +347,7 @@ actual_slot = finalized_slot + 1 + relative_index
- Mesh size: 8 (6-12 bounds), heartbeat: 700ms
- Beacon wire: `validate_messages()` is on, so every beacon message waits for a verdict (~4.2s before gossipsub's cache evicts it). Rules in `state_transition::beacon::gossip` (cheap half inline, stateful half on a bounded `spawn_blocking` task); plumbing in `p2p/src/beacon/verdict.rs`. Lean gossip still auto-forwards
- Beacon peer scoring: lighthouse's parameters ported in `p2p/src/beacon/scoring.rs` (block, aggregate, 64 attestation subnets, exits, slashings; columns and sync contributions unscored), refreshed every slot from the head's active validator count. Mesh-delivery scoring (P3) is gated off while the head lags more than 4 slots and for an epoch after, because a catching-up node `Ignore`s every aggregate and would otherwise graylist its own mesh. Every held fork digest is parametrized before `transition::apply` joins it (gossipsub never scores a mesh grafted before its topic had parameters); the previous digest's weights are zeroed at the switch, and the switch restarts the P3 warmup. The swarm disconnects peers below the graylist (-16000) every 10 s, and peers gossipsub reported as `SlowPeer` in every check for a minute (a disabled gossipsub handler never drains its send queue, so the queue's message copies stay in memory until the connection closes). Gossipsub reports an expired forward like a full queue, so our own saturated uplink makes every peer look slow: a report counts only above `SLOW_PEER_REPORT_THRESHOLD` failures (Lighthouse's cut), and one check disconnects at most `MAX_SLOW_PEER_DISCONNECTS_PER_CHECK` peers. Lean runs unscored. See [`docs/beacon_wire.md`](docs/beacon_wire.md#peer-scoring)
- Data columns: every check runs in p2p. A column gossip did not accept (`Queue`/`Overloaded`), every fetched column, and parked columns replayed after their parent imports go through `column::chain_checks` in `p2p/src/beacon/column_checks.rs`. The chain actor stores what it gets unchecked; only debug builds re-run `chain_checks` there
- Data columns: every check runs in p2p. A column gossip did not accept (`Queue`/`Overloaded`), every fetched column, and parked columns replayed after their parent imports go through the chain checks in `p2p/src/beacon/column_checks.rs`, grouped by block with each `(block, index)` kept once. A group's rules before the KZG proofs run through one `column::ChainPrechecks`, which judges what the block's sidecars share (the fulu signed header's rules, the gloas block lookup) once; the proofs then go through `column::kzg_verdicts` in sub-batches of at most 16 columns, each on its own `column_check_permits` permit, a failed batch re-verified column by column. Each sidecar gets the verdict `column::chain_checks_for` gives it alone. The chain actor stores what it gets unchecked; only debug builds re-run `chain_checks_for` there
- Beacon subscribes seven global topics plus two node-id-derived subnet families: custody
columns and backbone attestation subnets. One of the seven is
`sync_committee_contribution_and_proof`; the four `sync_committee_{0..3}` subnets are
Expand Down
117 changes: 108 additions & 9 deletions crates/blockchain/state_transition/src/beacon/fork_choice.rs
Original file line number Diff line number Diff line change
Expand Up @@ -4119,7 +4119,9 @@ pub fn verify_data_column_sidecar_kzg_proofs(sidecar: &fulu::DataColumnSidecar)
)
}

/// The batch check both forks' `verify_data_column_sidecar_kzg_proofs` run.
/// The batch check both forks' `verify_data_column_sidecar_kzg_proofs` run:
/// [`verify_column_cells_batch`] over this one column, so a column checked on
/// its own and one checked beside others go through the same code.
///
/// The one thing that differs between them is where `commitments` comes from:
/// fulu's sidecar carries them, and gloas's takes them from the bid of the
Expand All @@ -4131,15 +4133,64 @@ fn verify_column_cells(
commitments: &[KzgCommitment],
proofs: &[KzgProof],
) -> Result<bool> {
let cell_indices = vec![index; column.len()];
let mut cells = Vec::with_capacity(column.len());
for cell in column {
cells.push(
c_kzg::Cell::from_bytes(&cell[..])
.map_err(|_| Error::SpecAssert("len(cell) == BYTES_PER_CELL"))?,
);
verify_column_cells_batch(&[ColumnCells {
index,
column,
commitments,
proofs,
}])
}

/// One column's worth of input to [`verify_column_cells_batch`]: the cell
/// position every cell of the column sits at, the cells, and the commitments
/// and proofs they are checked against.
pub struct ColumnCells<'a> {
pub index: u64,
pub column: &'a [fulu::Cell],
pub commitments: &'a [KzgCommitment],
pub proofs: &'a [KzgProof],
}

/// [`verify_column_cells`] over several columns at once: one
/// `verify_cell_kzg_proof_batch` call whose entries are every cell of every
/// column, in order.
///
/// `Ok(true)` means every cell of every column verifies. Anything else (a
/// cell that fails, or a malformed entry) does not say which column is at
/// fault, so a caller that must tell them apart verifies the columns one by
/// one afterwards.
///
/// A column whose cells, commitments and proofs differ in length is an `Err`,
/// as c-kzg makes it for one column on its own. Checked here, column by
/// column, because once the columns are concatenated only the totals could be
/// compared: a short column beside a long one would pair a cell of one with a
/// commitment of the other, and could verify. [`verify_data_column_sidecar`]
/// and its gloas counterpart already refuse such a sidecar, so this guards a
/// caller that skipped them.
pub fn verify_column_cells_batch(columns: &[ColumnCells<'_>]) -> Result<bool> {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The single-column path gets a per-column length check for free: c-kzg returns Err when one column's cells, commitments and proofs differ in length. Here the columns are concatenated first, so only the totals are compared. A column with 2 cells and 1 commitment next to one with 1 cell and 2 commitments has matching totals, and the first column's second cell is then verified against the second column's first commitment.

It is unreachable through today's callers, since verify_data_column_sidecar and gloas_verify_data_column_sidecar run first, but the function is pub and the precondition lives only in the doc comment. A per-column length check before flattening restores the guard.

verify_column_cells above could then delegate to this with one ColumnCells, so gossip and the chain path share one implementation of the rule instead of two copies of the cell-conversion loop.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in ab21359. verify_column_cells_batch checks that each entry's cells, commitments and proofs have equal length and returns Err otherwise, as c-kzg does for one column. verify_column_cells now delegates to it with one ColumnCells, so gossip and the chain path share one cell-conversion loop. a_batch_entry_whose_lengths_disagree_is_refused_even_when_the_totals_match splits one real 3-blob column into (2 cells, 1 commitment, 2 proofs) and (1 cell, 2 commitments, 1 proof). Flattened, those are exactly the whole column, which the test also asserts verifies. It fails with the check disabled.

let total: usize = columns.iter().map(|column| column.column.len()).sum();
let mut cell_indices = Vec::with_capacity(total);
let mut cells = Vec::with_capacity(total);
let mut commitments = Vec::with_capacity(total);
let mut proofs = Vec::with_capacity(total);
for column in columns {
let cell_count = column.column.len();
if column.commitments.len() != cell_count || column.proofs.len() != cell_count {
return Err(Error::SpecAssert(
"len(cells) == len(commitments) == len(proofs)",
));
}
cell_indices.extend(std::iter::repeat_n(column.index, column.column.len()));
for cell in column.column {
cells.push(
c_kzg::Cell::from_bytes(&cell[..])
.map_err(|_| Error::SpecAssert("len(cell) == BYTES_PER_CELL"))?,
);
}
commitments.extend_from_slice(column.commitments);
proofs.extend_from_slice(column.proofs);
}
kzg::verify_cell_kzg_proof_batch(commitments, &cell_indices, &cells, proofs)
kzg::verify_cell_kzg_proof_batch(&commitments, &cell_indices, &cells, &proofs)
}

/// `verify_data_column_sidecar` (gloas `p2p-interface.md`, modified): the
Expand Down Expand Up @@ -10636,4 +10687,52 @@ mod tests {
);
}
}

/// One real column of a three-blob block, split into two entries whose
/// lengths disagree and cancel out: two cells and proofs with one
/// commitment, then one cell and proof with two commitments. Concatenated,
/// that is the valid column again, so only a per-entry length check can
/// refuse it, as c-kzg refuses either entry on its own.
#[test]
fn a_batch_entry_whose_lengths_disagree_is_refused_even_when_the_totals_match() {
let index = 5;
let mut cells = Vec::new();
let mut commitments = Vec::new();
let mut proofs = Vec::new();
for seed in 1..=3u8 {
// Canonical field elements: the top byte of each is zero.
let blob: Vec<u8> = (0..c_kzg::BYTES_PER_BLOB)
.map(|i| if i % 32 == 0 { 0 } else { (i as u8) ^ seed })
.collect();
commitments.push(kzg::blob_to_kzg_commitment(&blob).expect("commitment"));
let (blob_cells, blob_proofs) =
kzg::compute_cells_and_kzg_proofs(&blob).expect("cells and proofs");
let cell: fulu::Cell =
libssz_types::SszVector::try_from(blob_cells[index].to_bytes().to_vec())
.expect("exact cell size");
cells.push(cell);
proofs.push(blob_proofs[index]);
}
let whole = ColumnCells {
index: index as u64,
column: &cells,
commitments: &commitments,
proofs: &proofs,
};
assert!(matches!(verify_column_cells_batch(&[whole]), Ok(true)));

let short = ColumnCells {
index: index as u64,
column: &cells[..2],
commitments: &commitments[..1],
proofs: &proofs[..2],
};
let long = ColumnCells {
index: index as u64,
column: &cells[2..],
commitments: &commitments[1..],
proofs: &proofs[2..],
};
assert!(verify_column_cells_batch(&[short, long]).is_err());
}
}
Loading
Loading