diff --git a/CLAUDE.md b/CLAUDE.md index 4b4afa4b4..5b40153de 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -347,7 +347,7 @@ actual_slot = finalized_slot + 1 + relative_index - Mesh size: 8 (6-12 bounds), heartbeat: 700ms - Beacon wire: `validate_messages()` is on, so every beacon message waits for a verdict (~4.2s before gossipsub's cache evicts it). Rules in `state_transition::beacon::gossip` (cheap half inline, stateful half on a bounded `spawn_blocking` task); plumbing in `p2p/src/beacon/verdict.rs`. Lean gossip still auto-forwards - Beacon peer scoring: lighthouse's parameters ported in `p2p/src/beacon/scoring.rs` (block, aggregate, 64 attestation subnets, exits, slashings; columns and sync contributions unscored), refreshed every slot from the head's active validator count. Mesh-delivery scoring (P3) is gated off while the head lags more than 4 slots and for an epoch after, because a catching-up node `Ignore`s every aggregate and would otherwise graylist its own mesh. Every held fork digest is parametrized before `transition::apply` joins it (gossipsub never scores a mesh grafted before its topic had parameters); the previous digest's weights are zeroed at the switch, and the switch restarts the P3 warmup. The swarm disconnects peers below the graylist (-16000) every 10 s, and peers gossipsub reported as `SlowPeer` in every check for a minute (a disabled gossipsub handler never drains its send queue, so the queue's message copies stay in memory until the connection closes). Gossipsub reports an expired forward like a full queue, so our own saturated uplink makes every peer look slow: a report counts only above `SLOW_PEER_REPORT_THRESHOLD` failures (Lighthouse's cut), and one check disconnects at most `MAX_SLOW_PEER_DISCONNECTS_PER_CHECK` peers. Lean runs unscored. See [`docs/beacon_wire.md`](docs/beacon_wire.md#peer-scoring) - - Data columns: every check runs in p2p. A column gossip did not accept (`Queue`/`Overloaded`), every fetched column, and parked columns replayed after their parent imports go through `column::chain_checks` in `p2p/src/beacon/column_checks.rs`. The chain actor stores what it gets unchecked; only debug builds re-run `chain_checks` there + - Data columns: every check runs in p2p. A column gossip did not accept (`Queue`/`Overloaded`), every fetched column, and parked columns replayed after their parent imports go through the chain checks in `p2p/src/beacon/column_checks.rs`, grouped by block with each `(block, index)` kept once. A group's rules before the KZG proofs run through one `column::ChainPrechecks`, which judges what the block's sidecars share (the fulu signed header's rules, the gloas block lookup) once; the proofs then go through `column::kzg_verdicts` in sub-batches of at most 16 columns, each on its own `column_check_permits` permit, a failed batch re-verified column by column. Each sidecar gets the verdict `column::chain_checks_for` gives it alone. The chain actor stores what it gets unchecked; only debug builds re-run `chain_checks_for` there - Beacon subscribes seven global topics plus two node-id-derived subnet families: custody columns and backbone attestation subnets. One of the seven is `sync_committee_contribution_and_proof`; the four `sync_committee_{0..3}` subnets are diff --git a/crates/blockchain/state_transition/src/beacon/fork_choice.rs b/crates/blockchain/state_transition/src/beacon/fork_choice.rs index 238804550..db931fdfe 100644 --- a/crates/blockchain/state_transition/src/beacon/fork_choice.rs +++ b/crates/blockchain/state_transition/src/beacon/fork_choice.rs @@ -4119,7 +4119,9 @@ pub fn verify_data_column_sidecar_kzg_proofs(sidecar: &fulu::DataColumnSidecar) ) } -/// The batch check both forks' `verify_data_column_sidecar_kzg_proofs` run. +/// The batch check both forks' `verify_data_column_sidecar_kzg_proofs` run: +/// [`verify_column_cells_batch`] over this one column, so a column checked on +/// its own and one checked beside others go through the same code. /// /// The one thing that differs between them is where `commitments` comes from: /// fulu's sidecar carries them, and gloas's takes them from the bid of the @@ -4131,15 +4133,64 @@ fn verify_column_cells( commitments: &[KzgCommitment], proofs: &[KzgProof], ) -> Result { - let cell_indices = vec![index; column.len()]; - let mut cells = Vec::with_capacity(column.len()); - for cell in column { - cells.push( - c_kzg::Cell::from_bytes(&cell[..]) - .map_err(|_| Error::SpecAssert("len(cell) == BYTES_PER_CELL"))?, - ); + verify_column_cells_batch(&[ColumnCells { + index, + column, + commitments, + proofs, + }]) +} + +/// One column's worth of input to [`verify_column_cells_batch`]: the cell +/// position every cell of the column sits at, the cells, and the commitments +/// and proofs they are checked against. +pub struct ColumnCells<'a> { + pub index: u64, + pub column: &'a [fulu::Cell], + pub commitments: &'a [KzgCommitment], + pub proofs: &'a [KzgProof], +} + +/// [`verify_column_cells`] over several columns at once: one +/// `verify_cell_kzg_proof_batch` call whose entries are every cell of every +/// column, in order. +/// +/// `Ok(true)` means every cell of every column verifies. Anything else (a +/// cell that fails, or a malformed entry) does not say which column is at +/// fault, so a caller that must tell them apart verifies the columns one by +/// one afterwards. +/// +/// A column whose cells, commitments and proofs differ in length is an `Err`, +/// as c-kzg makes it for one column on its own. Checked here, column by +/// column, because once the columns are concatenated only the totals could be +/// compared: a short column beside a long one would pair a cell of one with a +/// commitment of the other, and could verify. [`verify_data_column_sidecar`] +/// and its gloas counterpart already refuse such a sidecar, so this guards a +/// caller that skipped them. +pub fn verify_column_cells_batch(columns: &[ColumnCells<'_>]) -> Result { + let total: usize = columns.iter().map(|column| column.column.len()).sum(); + let mut cell_indices = Vec::with_capacity(total); + let mut cells = Vec::with_capacity(total); + let mut commitments = Vec::with_capacity(total); + let mut proofs = Vec::with_capacity(total); + for column in columns { + let cell_count = column.column.len(); + if column.commitments.len() != cell_count || column.proofs.len() != cell_count { + return Err(Error::SpecAssert( + "len(cells) == len(commitments) == len(proofs)", + )); + } + cell_indices.extend(std::iter::repeat_n(column.index, column.column.len())); + for cell in column.column { + cells.push( + c_kzg::Cell::from_bytes(&cell[..]) + .map_err(|_| Error::SpecAssert("len(cell) == BYTES_PER_CELL"))?, + ); + } + commitments.extend_from_slice(column.commitments); + proofs.extend_from_slice(column.proofs); } - kzg::verify_cell_kzg_proof_batch(commitments, &cell_indices, &cells, proofs) + kzg::verify_cell_kzg_proof_batch(&commitments, &cell_indices, &cells, &proofs) } /// `verify_data_column_sidecar` (gloas `p2p-interface.md`, modified): the @@ -10636,4 +10687,52 @@ mod tests { ); } } + + /// One real column of a three-blob block, split into two entries whose + /// lengths disagree and cancel out: two cells and proofs with one + /// commitment, then one cell and proof with two commitments. Concatenated, + /// that is the valid column again, so only a per-entry length check can + /// refuse it, as c-kzg refuses either entry on its own. + #[test] + fn a_batch_entry_whose_lengths_disagree_is_refused_even_when_the_totals_match() { + let index = 5; + let mut cells = Vec::new(); + let mut commitments = Vec::new(); + let mut proofs = Vec::new(); + for seed in 1..=3u8 { + // Canonical field elements: the top byte of each is zero. + let blob: Vec = (0..c_kzg::BYTES_PER_BLOB) + .map(|i| if i % 32 == 0 { 0 } else { (i as u8) ^ seed }) + .collect(); + commitments.push(kzg::blob_to_kzg_commitment(&blob).expect("commitment")); + let (blob_cells, blob_proofs) = + kzg::compute_cells_and_kzg_proofs(&blob).expect("cells and proofs"); + let cell: fulu::Cell = + libssz_types::SszVector::try_from(blob_cells[index].to_bytes().to_vec()) + .expect("exact cell size"); + cells.push(cell); + proofs.push(blob_proofs[index]); + } + let whole = ColumnCells { + index: index as u64, + column: &cells, + commitments: &commitments, + proofs: &proofs, + }; + assert!(matches!(verify_column_cells_batch(&[whole]), Ok(true))); + + let short = ColumnCells { + index: index as u64, + column: &cells[..2], + commitments: &commitments[..1], + proofs: &proofs[..2], + }; + let long = ColumnCells { + index: index as u64, + column: &cells[2..], + commitments: &commitments[1..], + proofs: &proofs[2..], + }; + assert!(verify_column_cells_batch(&[short, long]).is_err()); + } } diff --git a/crates/blockchain/state_transition/src/beacon/gossip/column.rs b/crates/blockchain/state_transition/src/beacon/gossip/column.rs index 900d83353..33579b9d6 100644 --- a/crates/blockchain/state_transition/src/beacon/gossip/column.rs +++ b/crates/blockchain/state_transition/src/beacon/gossip/column.rs @@ -10,6 +10,8 @@ //! actor stores whatever reaches it without checking it //! again, so these are the only checks such a sidecar gets. +use std::collections::HashMap; + use ethlambda_storage::CacheKey; use super::{ @@ -21,9 +23,9 @@ use crate::beacon::config::Config; use crate::beacon::constants::DOMAIN_BEACON_PROPOSER; use crate::beacon::containers::BeaconState; use crate::beacon::containers::DataColumnSidecar as AnyDataColumnSidecar; -use crate::beacon::containers::SignedBeaconBlock; use crate::beacon::containers::fulu::DataColumnSidecar; use crate::beacon::containers::gloas; +use crate::beacon::containers::{SignedBeaconBlock, SignedBeaconBlockHeader}; use crate::beacon::das; use crate::beacon::fork_choice::{self, Store}; use crate::beacon::helpers::accessors::get_beacon_proposer_index; @@ -34,7 +36,7 @@ use crate::beacon::lean_boundary::lean_block_unreachable; use crate::beacon::precheck; use crate::beacon::preset; use crate::beacon::primitives::HashTreeRoot as _; -use crate::beacon::primitives::{Root, Slot, ValidatorIndex}; +use crate::beacon::primitives::{BlsSignature, KzgCommitment, Root, Slot, ValidatorIndex}; use crate::beacon::stf; use crate::metrics; @@ -81,10 +83,14 @@ pub fn cheap_checks( /// The rules that need the parent's post-state. Runs on a blocking thread, /// and reads states only from the cache, as [`super::block::stateful_checks`] -/// does. The rules themselves are [`judge_against_parent`]'s, a permit held -/// while they run. +/// does. +/// +/// The header's rules first ([`judge_header_against_parent`]), then the +/// inclusion proof and the KZG proofs, and the lookahead last: every +/// REJECT-worthy rule is settled before the one rule that may only queue. pub fn stateful_checks(store: &Store, sidecar: &DataColumnSidecar) -> Outcome { - let header = &sidecar.signed_block_header.message; + let signed_header = &sidecar.signed_block_header; + let header = &signed_header.message; let parent_known = store.has_block(&header.parent_root); let parent_state = if parent_known { store.cached_state(CacheKey::BlockState(header.parent_root)) @@ -101,9 +107,31 @@ pub fn stateful_checks(store: &Store, sidecar: &DataColumnSidecar) -> Outcome { } else { QueueReason::ParentUnknown }; - return queue_unless_forged(store, sidecar, reason); + return queue_unless_forged(store, signed_header, reason); + }; + if let Err(outcome) = + judge_header_against_parent(store, signed_header, &parent_state, PastLookahead::Queue) + { + return outcome; + } + // [REJECT] The commitments are the ones the block committed to. + if !fork_choice::verify_data_column_sidecar_inclusion_proof(sidecar) { + return Outcome::Reject(RejectReason::InclusionProof); + } + // [REJECT] Every cell matches its commitment and proof. + let kzg = { + let _timing = metrics::time_data_column_kzg_verify(); + fork_choice::verify_data_column_sidecar_kzg_proofs(sidecar) }; - judge_against_parent(store, sidecar, &parent_state, PastLookahead::Queue) + if !matches!(kzg, Ok(true)) { + return Outcome::Reject(RejectReason::Kzg); + } + // [IGNORE] The sidecar's slot is outside the parent state's proposer + // lookahead window (MAY queue). + if let Err(outcome) = lookahead_covers(&parent_state, header.slot) { + return outcome; + } + Outcome::Accept } /// What the chain may do with a sidecar [`chain_checks`] judged. @@ -136,45 +164,14 @@ pub enum ChainVerdict { /// than queued. Nothing would ever come along to answer it later. /// /// Runs off both actors. It costs a KZG batch and a BLS verification per -/// sidecar, and those used to run on the chain actor's single thread. +/// sidecar, and those used to run on the chain actor's single thread. The p2p +/// layer judges the sidecars of one block together, through +/// [`ChainPrechecks`] and [`kzg_verdicts`], which give each sidecar the +/// verdict this gives it alone. pub fn chain_checks(store: &Store, sidecar: &DataColumnSidecar, now_ms: u64) -> ChainVerdict { - let config = store.config(); - // [REJECT] The sidecar is structurally valid. First, so a peer answering - // a fetch with garbage cannot make this pay for a state read or a KZG - // batch. - if !fork_choice::verify_data_column_sidecar(sidecar, &config) { - return ChainVerdict::Drop(Outcome::Reject(RejectReason::Malformed)); - } - let header = &sidecar.signed_block_header.message; - // [IGNORE] Not from a future slot. Also what bounds `advanced_proposer`'s - // `process_slots` below: a header naming a slot far ahead would otherwise - // make it advance one slot at a time towards that slot. - if is_future_slot(&config, header.slot, now_ms) { - return ChainVerdict::Drop(Outcome::Ignore(IgnoreReason::FutureSlot)); - } - // [IGNORE] From a slot greater than the latest finalized slot. - if header.slot <= finalized_start_slot(store) { - return ChainVerdict::Drop(Outcome::Ignore(IgnoreReason::Finalized)); - } - // [IGNORE] Already stored. Common during a range sync, whose consecutive - // batches ask for overlapping spans of columns, and everything after this - // is the expensive part. - if store.has_data_column(header.slot, &header.hash_tree_root(), sidecar.index) { - return ChainVerdict::Drop(Outcome::Ignore(IgnoreReason::AlreadyStored)); - } - let Ok(Some(parent_state)) = store.get_state(&header.parent_root) else { - // Parked rather than dropped: the specification's "MAY be queued" - // for a sidecar whose parent has not been seen. The head-state - // signature check keeps a forged header from taking a - // `PendingDataColumns` row while it waits. - return match queue_unless_forged(store, sidecar, QueueReason::ParentUnknown) { - Outcome::Queue(_) => ChainVerdict::AwaitParent, - outcome => ChainVerdict::Drop(outcome), - }; - }; - match judge_against_parent(store, sidecar, &parent_state, PastLookahead::Advance) { - Outcome::Accept => ChainVerdict::Keep, - outcome => ChainVerdict::Drop(outcome), + match ChainPrechecks::new(store, now_ms).check_fulu(sidecar) { + Ok(pending) => kzg_verdict_ref(&ColumnRef::Fulu(sidecar), &pending), + Err(verdict) => verdict, } } @@ -184,9 +181,9 @@ pub fn chain_checks_for( sidecar: &AnyDataColumnSidecar, now_ms: u64, ) -> ChainVerdict { - match sidecar { - AnyDataColumnSidecar::Fulu(sidecar) => chain_checks(store, sidecar, now_ms), - AnyDataColumnSidecar::Gloas(sidecar) => chain_checks_gloas(store, sidecar, now_ms), + match ChainPrechecks::new(store, now_ms).check(sidecar) { + Ok(pending) => kzg_verdict(sidecar, &pending), + Err(verdict) => verdict, } } @@ -204,78 +201,358 @@ pub fn chain_checks_gloas( sidecar: &gloas::DataColumnSidecar, now_ms: u64, ) -> ChainVerdict { - let config = store.config(); - // [IGNORE] Not from a future slot. Also what bounds how far ahead a - // parked row can name. - if is_future_slot(&config, sidecar.slot, now_ms) { - return ChainVerdict::Drop(Outcome::Ignore(IgnoreReason::FutureSlot)); - } - // [IGNORE] From a slot greater than the latest finalized slot, unless it - // is a column of the finalized block itself. Not a gloas rule: gloas has - // no finalized bound on sidecars, and this one stays only to bound what a - // parked row can name. The finalized block is exempt because its envelope - // can still be waiting on these columns: a checkpoint-synced anchor is the - // finalized block, and no child building on its payload imports until its - // sampled columns arrive. Matched by root rather than by slot, since an - // anchor state advanced past empty slots finalizes a block from before its - // epoch's start slot. A column naming that root is never parked: the block - // and its post-state are both stored, so it is judged at once. - let finalized = store.beacon_finalized_checkpoint(); - if sidecar.slot <= compute_start_slot_at_epoch(finalized.epoch) - && sidecar.beacon_block_root != finalized.root - { - return ChainVerdict::Drop(Outcome::Ignore(IgnoreReason::Finalized)); + match ChainPrechecks::new(store, now_ms).check_gloas(sidecar) { + Ok(pending) => kzg_verdict_ref(&ColumnRef::Gloas(sidecar), &pending), + Err(verdict) => verdict, } - // [IGNORE] Already stored, as in [`chain_checks`]. - if store.has_data_column(sidecar.slot, &sidecar.beacon_block_root, sidecar.index) { - return ChainVerdict::Drop(Outcome::Ignore(IgnoreReason::AlreadyStored)); +} + +/// The chain checks up to, and not including, the KZG proofs, for the sidecars +/// of one block: every rule that does not read the cells, in the order +/// [`chain_checks`] and [`chain_checks_gloas`] run them. +/// +/// The rules that read nothing of a sidecar but what every sidecar of its +/// block carries run once per block rather than once per sidecar. For fulu +/// that is everything judged against the signed header ([`judge_header`]): +/// finding the parent's post-state, or the head-state signature check before +/// parking, then `NotAfterParent`, the signature, the expected proposer (a +/// `process_slots` on a state clone past the lookahead) and finalized +/// ancestry. For gloas it is finding and decoding the block the sidecar names +/// ([`gloas_block`]). An answer is worked out the first time a sidecar gets as +/// far as needing it, so a group whose sidecars all stop earlier (all already +/// stored, say) pays for none of it, and is remembered under everything it +/// read of the sidecar: the signed header's root *and* signature, so a forged +/// signature over a block's header is judged on its own, or the block root. +/// Everything else (the structure, the clock, finality, `AlreadyStored`, the +/// inclusion proof, the slot match and shape against the bid's commitments) +/// runs for every sidecar, in its place. +/// +/// `Err` is the verdict whatever the proofs say. `Ok` means every rule before +/// the proofs passed, and the proofs are the last rule on this path, so the +/// sidecar is [`ChainVerdict::Keep`] exactly when they verify: hand it to +/// [`kzg_verdict`], or to [`kzg_verdicts`] with the other sidecars of the same +/// block. +/// +/// Meant to live as long as one group's checks. Each answer it remembers is the +/// store as it stood when that answer was first worked out, not when a later +/// sidecar asks: a parent whose post-state lands partway through a group +/// leaves the rest of the group parked on the answer given before it. That +/// costs a round trip, not the sidecars: the chain actor's +/// `park_data_columns` sends back at once any sidecar whose block can be +/// judged by the time it arrives. +pub struct ChainPrechecks<'a> { + store: &'a Store, + now_ms: u64, + /// [`judge_header`]'s answers, by the signed header's root and signature. + headers: HashMap<(Root, BlsSignature), Result<(), ChainVerdict>>, + /// [`gloas_block`]'s answers, by block root. + blocks: HashMap>, +} + +impl<'a> ChainPrechecks<'a> { + /// Checks against `store`, with `now_ms` as the clock for every sidecar. + pub fn new(store: &'a Store, now_ms: u64) -> Self { + Self { + store, + now_ms, + headers: HashMap::new(), + blocks: HashMap::new(), + } } - match stateful_checks_gloas(store, sidecar) { - Outcome::Accept => ChainVerdict::Keep, - Outcome::Queue(QueueReason::BlockUnknown | QueueReason::BlockNotReady) => { - ChainVerdict::AwaitParent + + /// The verdict, when a rule before the KZG proofs settles it, or what the + /// proofs are checked against. + pub fn check(&mut self, sidecar: &AnyDataColumnSidecar) -> Result { + match sidecar { + AnyDataColumnSidecar::Fulu(sidecar) => self.check_fulu(sidecar), + AnyDataColumnSidecar::Gloas(sidecar) => self.check_gloas(sidecar), } - outcome => ChainVerdict::Drop(outcome), } + + /// [`Self::check`] for a fulu sidecar. + fn check_fulu(&mut self, sidecar: &DataColumnSidecar) -> Result { + let store = self.store; + let config = store.config(); + // [REJECT] The sidecar is structurally valid. First, so a peer + // answering a fetch with garbage cannot make this pay for a state read + // or a KZG batch. + if !fork_choice::verify_data_column_sidecar(sidecar, &config) { + return Err(ChainVerdict::Drop(Outcome::Reject(RejectReason::Malformed))); + } + let signed_header = &sidecar.signed_block_header; + let header = &signed_header.message; + // [IGNORE] Not from a future slot. Also what bounds + // `advanced_proposer`'s `process_slots` in `judge_header`: a header + // naming a slot far ahead would otherwise make it advance one slot at + // a time towards that slot. + if is_future_slot(&config, header.slot, self.now_ms) { + return Err(ChainVerdict::Drop(Outcome::Ignore( + IgnoreReason::FutureSlot, + ))); + } + // [IGNORE] From a slot greater than the latest finalized slot. + if header.slot <= finalized_start_slot(store) { + return Err(ChainVerdict::Drop(Outcome::Ignore(IgnoreReason::Finalized))); + } + // [IGNORE] Already stored. Common during a range sync, whose + // consecutive batches ask for overlapping spans of columns, and + // everything after this is the expensive part. + let header_root = header.hash_tree_root(); + if store.has_data_column(header.slot, &header_root, sidecar.index) { + return Err(ChainVerdict::Drop(Outcome::Ignore( + IgnoreReason::AlreadyStored, + ))); + } + // Everything the signed header and the parent's post-state decide, + // once per signed header. Inserted only once `judge_header` returns, + // so a panic inside it leaves nothing behind for the next sidecar to + // read. + let key = (header_root, signed_header.signature); + let shared = *self + .headers + .entry(key) + .or_insert_with(|| judge_header(store, signed_header)); + shared?; + // [REJECT] The commitments are the ones the block committed to. Per + // sidecar: the commitments and the branch proving them are the + // sidecar's own, not the header's. + if !fork_choice::verify_data_column_sidecar_inclusion_proof(sidecar) { + return Err(ChainVerdict::Drop(Outcome::Reject( + RejectReason::InclusionProof, + ))); + } + Ok(PendingKzg { + commitments: sidecar.kzg_commitments.to_vec(), + }) + } + + /// [`Self::check`] for a gloas sidecar. The commitments its proofs are + /// checked against come out of the block's bid, so they travel in the + /// [`PendingKzg`]. + fn check_gloas( + &mut self, + sidecar: &gloas::DataColumnSidecar, + ) -> Result { + let store = self.store; + let config = store.config(); + // [IGNORE] Not from a future slot. Also what bounds how far ahead a + // parked row can name. + if is_future_slot(&config, sidecar.slot, self.now_ms) { + return Err(ChainVerdict::Drop(Outcome::Ignore( + IgnoreReason::FutureSlot, + ))); + } + // [IGNORE] From a slot greater than the latest finalized slot, unless + // it is a column of the finalized block itself. Not a gloas rule: + // gloas has no finalized bound on sidecars, and this one stays only to + // bound what a parked row can name. The finalized block is exempt + // because its envelope can still be waiting on these columns: a + // checkpoint-synced anchor is the finalized block, and no child + // building on its payload imports until its sampled columns arrive. + // Matched by root rather than by slot, since an anchor state advanced + // past empty slots finalizes a block from before its epoch's start + // slot. A column naming that root is never parked: the block and its + // post-state are both stored, so it is judged at once. + let finalized = store.beacon_finalized_checkpoint(); + if sidecar.slot <= compute_start_slot_at_epoch(finalized.epoch) + && sidecar.beacon_block_root != finalized.root + { + return Err(ChainVerdict::Drop(Outcome::Ignore(IgnoreReason::Finalized))); + } + // [IGNORE] Already stored, as in [`chain_checks`]. + if store.has_data_column(sidecar.slot, &sidecar.beacon_block_root, sidecar.index) { + return Err(ChainVerdict::Drop(Outcome::Ignore( + IgnoreReason::AlreadyStored, + ))); + } + // The block the sidecar names, looked up and decoded once per block. + let block = self + .blocks + .entry(sidecar.beacon_block_root) + .or_insert_with(|| gloas_block(store, &sidecar.beacon_block_root)); + match gloas_before_kzg(sidecar, block) { + Ok(commitments) => Ok(PendingKzg { + commitments: commitments.to_vec(), + }), + Err(Outcome::Queue(QueueReason::BlockUnknown | QueueReason::BlockNotReady)) => { + Err(ChainVerdict::AwaitParent) + } + Err(outcome) => Err(ChainVerdict::Drop(outcome)), + } + } +} + +/// The fulu chain rules that read nothing of a sidecar but its signed header, +/// in [`chain_checks`]'s order: what [`ChainPrechecks`] runs once per signed +/// header. +/// +/// `Err(AwaitParent)` when the parent has no post-state and the head state +/// does not show the signature forged; any other `Err` drops the sidecar. +fn judge_header( + store: &Store, + signed_header: &SignedBeaconBlockHeader, +) -> Result<(), ChainVerdict> { + let header = &signed_header.message; + let Ok(Some(parent_state)) = store.get_state(&header.parent_root) else { + // Parked rather than dropped: the specification's "MAY be queued" + // for a sidecar whose parent has not been seen. The head-state + // signature check keeps a forged header from taking a + // `PendingDataColumns` row while it waits. + return Err( + match queue_unless_forged(store, signed_header, QueueReason::ParentUnknown) { + Outcome::Queue(_) => ChainVerdict::AwaitParent, + outcome => ChainVerdict::Drop(outcome), + }, + ); + }; + judge_header_against_parent(store, signed_header, &parent_state, PastLookahead::Advance) + .map_err(ChainVerdict::Drop) +} + +/// What the KZG proofs of a sidecar that passed [`ChainPrechecks`] are +/// checked against. +/// +/// Fulu's sidecar carries its commitments, so this is a copy of them (a few +/// dozen bytes each, nothing next to the cells). Gloas's come from the bid of +/// the block the sidecar names, which only the precheck's store read could +/// supply. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct PendingKzg { + commitments: Vec, +} + +/// A sidecar of either fork, borrowed: lets [`chain_checks`] and +/// [`chain_checks_gloas`] verify the proofs the way [`kzg_verdict`] does +/// without wrapping (and so cloning) a sidecar into [`AnyDataColumnSidecar`]. +enum ColumnRef<'a> { + Fulu(&'a DataColumnSidecar), + Gloas(&'a gloas::DataColumnSidecar), } -/// What [`judge_against_parent`] does when the parent state's proposer +impl<'a> From<&'a AnyDataColumnSidecar> for ColumnRef<'a> { + fn from(sidecar: &'a AnyDataColumnSidecar) -> Self { + match sidecar { + AnyDataColumnSidecar::Fulu(sidecar) => Self::Fulu(sidecar), + AnyDataColumnSidecar::Gloas(sidecar) => Self::Gloas(sidecar), + } + } +} + +impl<'a> ColumnRef<'a> { + /// The cells and proofs of the column, as [`fork_choice::ColumnCells`] + /// against `commitments`. + fn cells(&self, commitments: &'a [KzgCommitment]) -> fork_choice::ColumnCells<'a> { + match self { + Self::Fulu(sidecar) => fork_choice::ColumnCells { + index: sidecar.index, + column: &sidecar.column, + commitments, + proofs: &sidecar.kzg_proofs, + }, + Self::Gloas(sidecar) => fork_choice::ColumnCells { + index: sidecar.index, + column: &sidecar.column, + commitments, + proofs: &sidecar.kzg_proofs, + }, + } + } +} + +/// Verify one pending sidecar's KZG proofs on their own. +pub fn kzg_verdict(sidecar: &AnyDataColumnSidecar, pending: &PendingKzg) -> ChainVerdict { + kzg_verdict_ref(&sidecar.into(), pending) +} + +fn kzg_verdict_ref(sidecar: &ColumnRef<'_>, pending: &PendingKzg) -> ChainVerdict { + let kzg = { + let _timing = metrics::time_data_column_kzg_verify(); + fork_choice::verify_column_cells_batch(&[sidecar.cells(&pending.commitments)]) + }; + // [REJECT] Every cell matches its commitment and proof. + if matches!(kzg, Ok(true)) { + ChainVerdict::Keep + } else { + ChainVerdict::Drop(Outcome::Reject(RejectReason::Kzg)) + } +} + +/// The KZG verdicts of several pending sidecars, one per entry and in order, +/// from one `verify_cell_kzg_proof_batch` call over all their cells. +/// +/// Every entry is [`ChainVerdict::Keep`] when the batch verifies. When it does +/// not, the batch cannot say which sidecar is at fault, so each is verified +/// again on its own: the verdicts are then exactly [`kzg_verdict`]'s, and a +/// forged column costs the rest of its batch one extra pass rather than their +/// acceptance. Intended for the sidecars of one block, which is where the +/// cost is shared: the batch pays its fixed work once instead of once per +/// sidecar. How many go in one call is the caller's to bound, since the +/// cells are copied into it and a failed batch is re-run one by one. +pub fn kzg_verdicts(items: &[(&AnyDataColumnSidecar, &PendingKzg)]) -> Vec { + match items { + // Nothing to verify, and nothing to time: an empty call must not add + // an observation to the KZG histogram. + [] => return Vec::new(), + [(sidecar, pending)] => return vec![kzg_verdict(sidecar, pending)], + _ => {} + } + let refs: Vec> = items.iter().map(|(sidecar, _)| (*sidecar).into()).collect(); + let columns: Vec> = refs + .iter() + .zip(items) + .map(|(sidecar, (_, pending))| sidecar.cells(&pending.commitments)) + .collect(); + let batch = { + let _timing = metrics::time_data_column_kzg_verify(); + fork_choice::verify_column_cells_batch(&columns) + }; + if matches!(batch, Ok(true)) { + return vec![ChainVerdict::Keep; items.len()]; + } + refs.iter() + .zip(items) + .map(|(sidecar, (_, pending))| kzg_verdict_ref(sidecar, pending)) + .collect() +} + +/// What [`judge_header_against_parent`] does when the parent state's proposer /// lookahead has no answer for the sidecar's slot. #[derive(Debug, Clone, Copy, PartialEq, Eq)] enum PastLookahead { - /// Queue it, as gossip does: advancing a state clone takes longer than a - /// gossip verdict can wait. + /// Skip the proposer rule, as gossip does: [`stateful_checks`] queues the + /// sidecar after its last REJECT rule instead ([`lookahead_covers`]), since + /// advancing a state clone takes longer than a gossip verdict can wait. Queue, - /// Advance a clone of the parent state to the slot and ask it, as - /// [`chain_checks`] does. + /// Advance a clone of the parent state to the slot and ask it, as the + /// chain checks do. Advance, } -/// The rules that read the parent's post-state, shared by [`stateful_checks`] -/// and [`chain_checks`]. +/// The rules that read the parent's post-state and nothing of the sidecar but +/// its signed header, shared by [`stateful_checks`] and [`judge_header`]: in +/// order, `NotAfterParent`, the signature, the expected proposer and the +/// finalized ancestry. /// -/// Signature first, lookahead last, mirroring -/// [`super::block::stateful_checks`]'s own order: a forged header naming a -/// real parent is caught by [`header_signature_is_valid`] before this pays -/// for the finalized-ancestry walk (a full `LiveChain` scan) or the KZG -/// proof. -fn judge_against_parent( +/// Signature first, mirroring [`super::block::stateful_checks`]'s own order: a +/// forged header naming a real parent is caught by +/// [`header_signature_is_valid`] before this pays for the finalized-ancestry +/// walk (a full `LiveChain` scan), and before its caller pays for the +/// inclusion proof or the KZG proofs. +fn judge_header_against_parent( store: &Store, - sidecar: &DataColumnSidecar, + signed_header: &SignedBeaconBlockHeader, parent_state: &BeaconState, past_lookahead: PastLookahead, -) -> Outcome { +) -> Result<(), Outcome> { let config = store.config(); - let header = &sidecar.signed_block_header.message; + let header = &signed_header.message; // [REJECT] The sidecar is from a higher slot than its parent. if header.slot <= parent_state.slot() { - return Outcome::Reject(RejectReason::NotAfterParent); + return Err(Outcome::Reject(RejectReason::NotAfterParent)); } // [REJECT] Signed by a validator the parent's state knows, over this // header. - if let Err(outcome) = header_signature_is_valid(sidecar, parent_state, &config) { - return outcome; - } + header_signature_is_valid(signed_header, parent_state, &config)?; // [REJECT] Proposed by the proposer the parent's state fixes for the // slot. Read from its lookahead when that covers the slot; past it, only // the chain path pays to find out. @@ -287,38 +564,16 @@ fn judge_against_parent( (None, PastLookahead::Queue) => None, (None, PastLookahead::Advance) => { let Some(expected) = advanced_proposer(parent_state, header.slot, &config) else { - return Outcome::Ignore(IgnoreReason::Internal); + return Err(Outcome::Ignore(IgnoreReason::Internal)); }; Some(expected) } }; if expected.is_some_and(|expected| header.proposer_index != expected) { - return Outcome::Reject(RejectReason::WrongProposer); + return Err(Outcome::Reject(RejectReason::WrongProposer)); } // [REJECT] The finalized checkpoint is an ancestor of the sidecar's block. - if let Err(outcome) = finalized_ancestry(store, header.parent_root).verdict() { - return outcome; - } - // [REJECT] The commitments are the ones the block committed to. - if !fork_choice::verify_data_column_sidecar_inclusion_proof(sidecar) { - return Outcome::Reject(RejectReason::InclusionProof); - } - // [REJECT] Every cell matches its commitment and proof. - let kzg = { - let _timing = metrics::time_data_column_kzg_verify(); - fork_choice::verify_data_column_sidecar_kzg_proofs(sidecar) - }; - if !matches!(kzg, Ok(true)) { - return Outcome::Reject(RejectReason::Kzg); - } - // [IGNORE] The sidecar's slot is outside the parent state's proposer - // lookahead window (MAY queue). - if past_lookahead == PastLookahead::Queue - && let Err(outcome) = lookahead_covers(parent_state, header.slot) - { - return outcome; - } - Outcome::Accept + finalized_ancestry(store, header.parent_root).verdict() } /// The proposer `parent_state`, advanced to `slot`, names for it: the answer @@ -328,8 +583,9 @@ fn judge_against_parent( /// Advances a clone, so the caller's copy (the store's cached `Arc`) is left /// exactly as it was. Costs a `process_slots` over every slot between the /// parent and `slot`, including a full state merkleization per empty slot -/// crossed, which is why only [`chain_checks`] calls this, and only for a -/// slot the lookahead does not cover. +/// crossed, which is why only the chain checks call this, once per signed +/// header ([`ChainPrechecks`]), and only for a slot the lookahead does not +/// cover. fn advanced_proposer( parent_state: &BeaconState, slot: Slot, @@ -349,7 +605,11 @@ fn advanced_proposer( /// the *expected* one for the slot. A column queued here is written to the /// chain actor's `PendingDataColumns`, so without this check a forged one /// would sit there rather than being refused up front. -fn queue_unless_forged(store: &Store, sidecar: &DataColumnSidecar, reason: QueueReason) -> Outcome { +fn queue_unless_forged( + store: &Store, + signed_header: &SignedBeaconBlockHeader, + reason: QueueReason, +) -> Outcome { let head_state = store .head() .ok() @@ -357,7 +617,7 @@ fn queue_unless_forged(store: &Store, sidecar: &DataColumnSidecar, reason: Queue let Some(head_state) = head_state else { return Outcome::Queue(reason); }; - match header_signature_is_valid(sidecar, &head_state, &store.config()) { + match header_signature_is_valid(signed_header, &head_state, &store.config()) { Err(Outcome::Reject(RejectReason::BadSignature)) => { Outcome::Reject(RejectReason::BadSignature) } @@ -425,23 +685,43 @@ pub fn cheap_checks_gloas( /// is queued too where the specification rejects it, the same deviation /// fulu's parent rule makes until a bad-block cache exists. pub fn stateful_checks_gloas(store: &Store, sidecar: &gloas::DataColumnSidecar) -> Outcome { - let block = match judgeable_block_gloas(store, &sidecar.beacon_block_root) { - Ok(block) => block, + let block = gloas_block(store, &sidecar.beacon_block_root); + let kzg_commitments = match gloas_before_kzg(sidecar, &block) { + Ok(commitments) => commitments, Err(outcome) => return outcome, }; - // [REJECT] The sidecar's slot matches the slot of the block. - if sidecar.slot != block.slot() { - return Outcome::Reject(RejectReason::SlotMismatch); - } - let kzg_commitments = match &block { - SignedBeaconBlock::Gloas(block) => { - &block - .message - .body - .signed_execution_payload_bid - .message - .blob_kzg_commitments - } + // [REJECT] The sidecar's column data passes KZG verification. + let kzg = { + let _timing = metrics::time_data_column_kzg_verify(); + fork_choice::gloas_verify_data_column_sidecar_kzg_proofs(sidecar, kzg_commitments) + }; + if !matches!(kzg, Ok(true)) { + return Outcome::Reject(RejectReason::Kzg); + } + Outcome::Accept +} + +/// What a gloas sidecar's rules read of the block it names: its slot, and the +/// commitments of its bid, or the verdict for a block that has no bid. +#[derive(Debug, Clone)] +struct GloasBlock { + slot: Slot, + commitments: Result, Outcome>, +} + +/// The block `root` names, as far as its sidecars' rules need it, or the +/// verdict [`judgeable_block_gloas`] gives for why it cannot be judged against +/// yet. Decodes the block, so [`ChainPrechecks`] asks it once per block. +fn gloas_block(store: &Store, root: &Root) -> Result { + let block = judgeable_block_gloas(store, root)?; + let commitments = match &block { + SignedBeaconBlock::Gloas(block) => Ok(block + .message + .body + .signed_execution_payload_bid + .message + .blob_kzg_commitments + .to_vec()), // A block of any other fork has no bid to read commitments from, so // no sidecar of this shape can be valid for it. SignedBeaconBlock::Phase0(_) @@ -450,22 +730,36 @@ pub fn stateful_checks_gloas(store: &Store, sidecar: &gloas::DataColumnSidecar) | SignedBeaconBlock::Capella(_) | SignedBeaconBlock::Deneb(_) | SignedBeaconBlock::Electra(_) - | SignedBeaconBlock::Fulu(_) => return Outcome::Reject(RejectReason::BlockNotGloas), - SignedBeaconBlock::Lean(_) => lean_block_unreachable("column::stateful_checks_gloas"), + | SignedBeaconBlock::Fulu(_) => Err(Outcome::Reject(RejectReason::BlockNotGloas)), + SignedBeaconBlock::Lean(_) => lean_block_unreachable("column::gloas_block"), }; + Ok(GloasBlock { + slot: block.slot(), + commitments, + }) +} + +/// Every rule [`stateful_checks_gloas`] runs before the KZG proofs, in the +/// same order, against `block` as [`gloas_block`] found it. `Ok` carries the +/// commitments the proofs are checked against (the block's bid's), and means +/// "accepted unless the proofs fail"; an `Err` is the verdict whatever the +/// proofs say. The commitments are borrowed from `block`, so gossip checks +/// against them without a copy and only a [`PendingKzg`] takes one. +fn gloas_before_kzg<'a>( + sidecar: &gloas::DataColumnSidecar, + block: &'a Result, +) -> Result<&'a [KzgCommitment], Outcome> { + let block = block.as_ref().map_err(|outcome| *outcome)?; + // [REJECT] The sidecar's slot matches the slot of the block. + if sidecar.slot != block.slot { + return Err(Outcome::Reject(RejectReason::SlotMismatch)); + } + let kzg_commitments = block.commitments.as_ref().map_err(|outcome| *outcome)?; // [REJECT] The sidecar passes structural validation. if !fork_choice::gloas_verify_data_column_sidecar(sidecar, kzg_commitments) { - return Outcome::Reject(RejectReason::Malformed); - } - // [REJECT] The sidecar's column data passes KZG verification. - let kzg = { - let _timing = metrics::time_data_column_kzg_verify(); - fork_choice::gloas_verify_data_column_sidecar_kzg_proofs(sidecar, kzg_commitments) - }; - if !matches!(kzg, Ok(true)) { - return Outcome::Reject(RejectReason::Kzg); + return Err(Outcome::Reject(RejectReason::Malformed)); } - Outcome::Accept + Ok(kzg_commitments) } /// The block a gloas sidecar is judged against, or the verdict that says why @@ -554,24 +848,24 @@ fn lookahead_covers(parent_state: &BeaconState, slot: Slot) -> Result<(), Outcom Ok(()) } -/// Whether `sidecar`'s header carries `state`'s key for its `proposer_index`. +/// Whether `signed_header` carries `state`'s key for its `proposer_index`. /// -/// Checked in [`stateful_checks`] before the proposer-identity check and -/// every later rule, so a forged header is caught before the finalized- -/// ancestry walk or the KZG proof. Also what [`queue_unless_forged`] judges a -/// column on before its parent's own state is even available, against the -/// head state instead of the parent's. Neither caller can answer the +/// Checked in [`judge_header_against_parent`] before the proposer-identity +/// check and every later rule, so a forged header is caught before the +/// finalized-ancestry walk or the KZG proof. Also what [`queue_unless_forged`] +/// judges a column on before its parent's own state is even available, against +/// the head state instead of the parent's. Neither caller can answer the /// *expected*-proposer question the state `state` is judged against here (the /// head state may predate the proposer's own deposit; the parent's state /// knows the proposer but not always the slot's shuffling), so this checks /// only what the signature alone can tell them: an unknown proposer or a bad /// signature, never `WrongProposer`. fn header_signature_is_valid( - sidecar: &DataColumnSidecar, + signed_header: &SignedBeaconBlockHeader, state: &BeaconState, config: &Config, ) -> Result<(), Outcome> { - let header = &sidecar.signed_block_header.message; + let header = &signed_header.message; let Ok(proposer) = state.validator(header.proposer_index) else { return Err(Outcome::Reject(RejectReason::UnknownProposer)); }; @@ -585,11 +879,7 @@ fn header_signature_is_valid( state.genesis_validators_root(), ); let signing_root = compute_signing_root(header.hash_tree_root(), domain); - if !bls::verify( - &proposer.pubkey, - signing_root, - &sidecar.signed_block_header.signature, - ) { + if !bls::verify(&proposer.pubkey, signing_root, &signed_header.signature) { return Err(Outcome::Reject(RejectReason::BadSignature)); } Ok(()) @@ -1109,6 +1399,90 @@ mod tests { ); } + /// The sidecars of one block share a header, so a group's checks judge it + /// once; a sidecar carrying a different signature over the same header is + /// its own entry and fails on its own, with the verdict it has when + /// checked alone. + #[test] + fn a_groups_shared_header_is_judged_once() { + let parent = fulu_parent(3); + let parent_root = Root::repeat_byte(0x99); + let store = store_with_parent(&parent, parent_root); + let config = store.config(); + let slot = parent.slot() + 1; + let now = slot_start_ms(&store, slot); + // Signed by the validator it names, which is not the slot's proposer: + // the signature passes and `WrongProposer` follows. + let good = signed_child(&parent, parent_root, slot, 5, 5, &config); + let mut forged = signed_child(&parent, parent_root, slot, 5, 6, &config); + forged.index = 3; + let columns: Vec = (0..3) + .map(|index| DataColumnSidecar { + index, + ..good.clone() + }) + .collect(); + + let mut prechecks = ChainPrechecks::new(&store, now); + for sidecar in &columns { + assert_eq!( + prechecks.check_fulu(sidecar), + Err(ChainVerdict::Drop(Outcome::Reject( + RejectReason::WrongProposer + ))) + ); + } + assert_eq!(prechecks.headers.len(), 1); + + assert_eq!( + prechecks.check_fulu(&forged), + Err(ChainVerdict::Drop(Outcome::Reject( + RejectReason::BadSignature + ))) + ); + assert_eq!(prechecks.headers.len(), 2); + // The forged one's answer is what it is alone, and does not leak onto + // the sidecars that carry the good signature. + assert_eq!( + chain_checks(&store, &forged, now), + ChainVerdict::Drop(Outcome::Reject(RejectReason::BadSignature)) + ); + assert_eq!( + prechecks.check_fulu(&columns[0]), + Err(ChainVerdict::Drop(Outcome::Reject( + RejectReason::WrongProposer + ))) + ); + assert_eq!(prechecks.headers.len(), 2); + } + + /// A group whose sidecars all stop before the header's rules (here, all + /// already stored) never reads the parent's state or judges the header: + /// the shared answer is worked out only when a sidecar needs it. + #[test] + fn a_group_that_stops_early_never_judges_its_header() { + let parent = fulu_parent(3); + let parent_root = Root::repeat_byte(0x99); + let store = store_with_parent(&parent, parent_root); + let config = store.config(); + let slot = parent.slot() + 1; + let now = slot_start_ms(&store, slot); + let card = signed_child(&parent, parent_root, slot, 3, 3, &config); + let header = &card.signed_block_header.message; + store + .put_data_column_sidecar(slot, &header.hash_tree_root(), card.index, Vec::new()) + .expect("store the column"); + + let mut prechecks = ChainPrechecks::new(&store, now); + assert_eq!( + prechecks.check_fulu(&card), + Err(ChainVerdict::Drop(Outcome::Ignore( + IgnoreReason::AlreadyStored + ))) + ); + assert!(prechecks.headers.is_empty()); + } + /// Where gossip queues, `chain_checks` drops: the parent already has a /// post-state, so no import will ever let the walk finish, and parking the /// sidecar would only wait for an import that has already happened. @@ -1372,4 +1746,423 @@ mod tests { .expect("within the list bound"); assert!(!is_parkable_gloas(&config, &custody, &crowded)); } + + // -- KZG batching ------------------------------------------------------ + + /// A blob of `4096` canonical field elements derived from `seed`: the top + /// byte of each 32-byte element is zero, which keeps it below the modulus. + fn blob(seed: u8) -> Vec { + (0..c_kzg::BYTES_PER_BLOB) + .map(|i| { + if i % 32 == 0 { + 0 + } else { + (i as u8).wrapping_mul(31).wrapping_add(seed) + } + }) + .collect() + } + + /// One column of real KZG material: its index, its cell of every blob, and + /// the proof of each. + type RealColumn = (u64, Vec, Vec); + + /// `blobs` blobs' worth of real commitments, and the cells and proofs of + /// every column in `columns`. Every column carries one cell per blob, as on + /// the wire. + fn real_columns(blobs: usize, columns: &[u64]) -> (Vec, Vec) { + let mut commitments = Vec::new(); + let mut cells = Vec::new(); + let mut proofs = Vec::new(); + for seed in 0..blobs { + let blob = blob(seed as u8); + commitments.push(crate::beacon::kzg::blob_to_kzg_commitment(&blob).expect("commit")); + let (blob_cells, blob_proofs) = + crate::beacon::kzg::compute_cells_and_kzg_proofs(&blob).expect("cells"); + cells.push(blob_cells); + proofs.push(blob_proofs); + } + let columns = columns + .iter() + .map(|&index| { + let column: Vec = cells + .iter() + .map(|blob_cells| { + libssz_types::SszVector::try_from( + blob_cells[index as usize].to_bytes().to_vec(), + ) + .expect("exact cell size") + }) + .collect(); + let column_proofs: Vec = proofs + .iter() + .map(|blob_proofs| blob_proofs[index as usize]) + .collect(); + (index, column, column_proofs) + }) + .collect(); + (commitments, columns) + } + + /// [`real_columns`] as the fulu sidecars of those columns, each with its + /// [`PendingKzg`] built by hand: for the tests of [`kzg_verdicts`] alone. + fn real_sidecars(blobs: usize, columns: &[u64]) -> Vec<(AnyDataColumnSidecar, PendingKzg)> { + let (commitments, columns) = real_columns(blobs, columns); + columns + .into_iter() + .map(|(index, column, column_proofs)| { + let mut sidecar = sidecar(5, 0, index); + sidecar.column = column.try_into().expect("within the per-block limit"); + sidecar.kzg_proofs = column_proofs + .try_into() + .expect("within the per-block limit"); + sidecar.kzg_commitments = commitments + .clone() + .try_into() + .expect("within the per-block limit"); + let pending = PendingKzg { + commitments: commitments.clone(), + }; + (AnyDataColumnSidecar::Fulu(sidecar), pending) + }) + .collect() + } + + fn verdicts_of(items: &[(AnyDataColumnSidecar, PendingKzg)]) -> Vec { + let refs: Vec<_> = items.iter().map(|(s, p)| (s, p)).collect(); + kzg_verdicts(&refs) + } + + #[test] + fn a_batch_of_valid_columns_keeps_every_one() { + let items = real_sidecars(2, &[0, 7, 64]); + assert_eq!(verdicts_of(&items), vec![ChainVerdict::Keep; 3]); + } + + /// The verdict the specification's own per-sidecar check gives, the one + /// gossip runs and the chain checks ran before they batched: what a batch's + /// verdicts must equal. + fn spec_kzg_verdict(sidecar: &AnyDataColumnSidecar, pending: &PendingKzg) -> ChainVerdict { + let kzg = match sidecar { + AnyDataColumnSidecar::Fulu(sidecar) => { + fork_choice::verify_data_column_sidecar_kzg_proofs(sidecar) + } + AnyDataColumnSidecar::Gloas(sidecar) => { + fork_choice::gloas_verify_data_column_sidecar_kzg_proofs( + sidecar, + &pending.commitments, + ) + } + }; + if matches!(kzg, Ok(true)) { + ChainVerdict::Keep + } else { + ChainVerdict::Drop(Outcome::Reject(RejectReason::Kzg)) + } + } + + /// One forged cell fails the whole batch; the fallback must then give the + /// verdicts the specification's per-sidecar check gives: the forged column + /// dropped as a KZG rejection, its neighbours kept. + #[test] + fn one_bad_column_in_a_batch_drops_only_that_column() { + let mut items = real_sidecars(2, &[0, 7, 64]); + let AnyDataColumnSidecar::Fulu(forged) = &mut items[1].0 else { + unreachable!("real_sidecars builds fulu sidecars"); + }; + let mut cell = forged.column[0].to_vec(); + cell[0] ^= 1; + forged.column[0] = libssz_types::SszVector::try_from(cell).expect("exact cell size"); + + let batched = verdicts_of(&items); + assert_eq!( + batched, + vec![ + ChainVerdict::Keep, + ChainVerdict::Drop(Outcome::Reject(RejectReason::Kzg)), + ChainVerdict::Keep, + ] + ); + let one_by_one: Vec<_> = items + .iter() + .map(|(sidecar, pending)| spec_kzg_verdict(sidecar, pending)) + .collect(); + assert_eq!(batched, one_by_one); + } + + /// Proofs swapped between two columns: each proof is a valid proof, for + /// the other column's cell, so only a check of the pairing catches it. + #[test] + fn proofs_swapped_between_columns_are_both_rejected() { + let mut items = real_sidecars(1, &[3, 4]); + let (first, second) = items.split_at_mut(1); + let (AnyDataColumnSidecar::Fulu(a), AnyDataColumnSidecar::Fulu(b)) = + (&mut first[0].0, &mut second[0].0) + else { + unreachable!("real_sidecars builds fulu sidecars"); + }; + std::mem::swap(&mut a.kzg_proofs, &mut b.kzg_proofs); + assert_eq!( + verdicts_of(&items), + vec![ChainVerdict::Drop(Outcome::Reject(RejectReason::Kzg)); 2] + ); + } + + /// The sidecars of a real fulu block with `blobs` blobs, at every column in + /// `columns`: a child of `parent` (stored under `parent_root`) at the next + /// slot, proposed and signed by `proposer`, so that every chain rule, the + /// inclusion proof and the KZG proofs included, can pass. + fn real_block_sidecars( + parent: &BeaconState, + parent_root: Root, + proposer: ValidatorIndex, + blobs: usize, + columns: &[u64], + config: &Config, + ) -> Vec { + let blobs: Vec = (0..blobs) + .map(|seed| libssz_types::SszVector::try_from(blob(seed as u8)).expect("blob size")) + .collect(); + let commitments: Vec = blobs + .iter() + .map(|blob| crate::beacon::kzg::blob_to_kzg_commitment(&blob[..]).expect("commit")) + .collect(); + let cell_proofs: Vec = blobs + .iter() + .flat_map(|blob| { + crate::beacon::kzg::compute_cells_and_kzg_proofs(&blob[..]) + .expect("cells") + .1 + .to_vec() + }) + .collect(); + let mut body = electra::BeaconBlockBody::empty(); + body.blob_kzg_commitments = commitments.try_into().expect("within the block limit"); + let block = electra::SignedBeaconBlock { + message: electra::BeaconBlock { + slot: parent.slot() + 1, + proposer_index: proposer, + parent_root, + state_root: Root::ZERO, + body, + }, + signature: Default::default(), + }; + let sidecars = crate::beacon::data_columns::verified_sidecars(&block, &blobs, &cell_proofs) + .expect("the block's own columns"); + columns + .iter() + .map(|&index| signed(sidecars[index as usize].clone(), parent, config, proposer)) + .collect() + } + + /// The verdicts a group gets the way the p2p layer judges it: every + /// sidecar through one [`ChainPrechecks`], then the pending ones through + /// one [`kzg_verdicts`] batch, put back in order. + fn group_verdicts( + prechecks: &mut ChainPrechecks<'_>, + group: &[AnyDataColumnSidecar], + ) -> Vec { + let prechecked: Vec<_> = group + .iter() + .map(|sidecar| prechecks.check(sidecar)) + .collect(); + let pending: Vec<_> = group + .iter() + .zip(&prechecked) + .filter_map(|(sidecar, precheck)| Some((sidecar, precheck.as_ref().ok()?))) + .collect(); + let mut batch = kzg_verdicts(&pending).into_iter(); + prechecked + .iter() + .map(|precheck| match precheck { + Ok(_) => batch.next().expect("one verdict per pending sidecar"), + Err(verdict) => *verdict, + }) + .collect() + } + + /// A fulu group mixing every way a sidecar of one block can end gets, + /// judged together, exactly the verdicts [`chain_checks_for`] gives each + /// sidecar alone, the good ones kept through the batch, and judges the + /// header once per signature it carries. + #[test] + fn a_fulu_group_gets_the_verdicts_each_sidecar_gets_alone() { + let proposer: ValidatorIndex = 3; + let parent = fulu_parent(proposer); + let parent_root = Root::repeat_byte(0xcc); + let mut store = store_with_parent(&parent, parent_root); + // The parent's own parent is the finalized root (`Root::ZERO`, at + // slot 0), so the finalized-ancestry walk passes. + store.insert_live_chain_entry(parent.slot(), parent_root, Root::ZERO); + store.insert_live_chain_entry(0, Root::ZERO, Root::ZERO); + let config = store.config(); + let now = slot_start_ms(&store, parent.slot() + 1); + let all: Vec = (0..8).collect(); + let mut group = real_block_sidecars(&parent, parent_root, proposer, 2, &all, &config); + + // 2: no commitments. + group[2].kzg_commitments = Default::default(); + // 3: already stored. + let header = &group[3].signed_block_header.message; + store + .put_data_column_sidecar(header.slot, &header.hash_tree_root(), 3, Vec::new()) + .expect("store the column"); + // 4: the block's header under another validator's signature. + group[4] = signed(group[4].clone(), &parent, &config, 6); + // 5: a branch that does not prove the commitments. + let mut branch = group[5].kzg_commitments_inclusion_proof.to_vec(); + branch[0] = Root::repeat_byte(0xee); + group[5].kzg_commitments_inclusion_proof = branch.try_into().expect("exact depth"); + // 6: a cell its proof does not open to. + let mut cell = group[6].column[0].to_vec(); + cell[31] ^= 1; + group[6].column[0] = libssz_types::SszVector::try_from(cell).expect("exact cell size"); + let group: Vec = + group.into_iter().map(AnyDataColumnSidecar::Fulu).collect(); + + let mut prechecks = ChainPrechecks::new(&store, now); + let together = group_verdicts(&mut prechecks, &group); + let alone: Vec = group + .iter() + .map(|sidecar| chain_checks_for(&store, sidecar, now)) + .collect(); + assert_eq!(together, alone); + assert_eq!( + together, + vec![ + ChainVerdict::Keep, + ChainVerdict::Keep, + ChainVerdict::Drop(Outcome::Reject(RejectReason::Malformed)), + ChainVerdict::Drop(Outcome::Ignore(IgnoreReason::AlreadyStored)), + ChainVerdict::Drop(Outcome::Reject(RejectReason::BadSignature)), + ChainVerdict::Drop(Outcome::Reject(RejectReason::InclusionProof)), + ChainVerdict::Drop(Outcome::Reject(RejectReason::Kzg)), + ChainVerdict::Keep, + ] + ); + // The block's own signature, and the other validator's. + assert_eq!(prechecks.headers.len(), 2); + } + + /// A gloas group: the block it names is looked up and decoded once, each + /// sidecar is shaped against that block's bid, and one batch keeps the + /// columns that verify and drops the forged one, as the specification's + /// per-sidecar check and [`chain_checks_for`] each do. + #[test] + fn a_gloas_group_reads_its_block_once_and_keeps_what_verifies() { + let mut store = store(0); + let slot = 5; + let root = Root::repeat_byte(0x5a); + let (commitments, columns) = real_columns(2, &[0, 1, 2]); + let mut block = gloas::SignedBeaconBlock { + message: gloas::BeaconBlock { + slot, + proposer_index: 0, + parent_root: Root::ZERO, + state_root: Root::ZERO, + body: Default::default(), + }, + signature: Default::default(), + }; + block + .message + .body + .signed_execution_payload_bid + .message + .blob_kzg_commitments = commitments.into(); + store + .insert_pending_block(root, SignedBeaconBlock::Gloas(block)) + .expect("insert the block"); + store + .insert_state(root, fulu_parent(3)) + .expect("insert a post-state"); + let mut group: Vec = columns + .into_iter() + .map(|(index, column, proofs)| gloas::DataColumnSidecar { + index, + slot, + beacon_block_root: root, + column: column.into(), + kzg_proofs: proofs.into(), + }) + .collect(); + let mut cell = group[1].column[0].to_vec(); + cell[31] ^= 1; + group[1].column[0] = libssz_types::SszVector::try_from(cell).expect("exact cell size"); + let group: Vec = + group.into_iter().map(AnyDataColumnSidecar::Gloas).collect(); + let now = slot_start_ms(&store, slot); + + let mut prechecks = ChainPrechecks::new(&store, now); + let pending: Vec<(&AnyDataColumnSidecar, PendingKzg)> = group + .iter() + .map(|sidecar| { + let pending = prechecks + .check(sidecar) + .expect("every rule before the proofs passes"); + (sidecar, pending) + }) + .collect(); + assert_eq!(prechecks.blocks.len(), 1); + let items: Vec<_> = pending + .iter() + .map(|(sidecar, pending)| (*sidecar, pending)) + .collect(); + let together = kzg_verdicts(&items); + assert_eq!( + together, + vec![ + ChainVerdict::Keep, + ChainVerdict::Drop(Outcome::Reject(RejectReason::Kzg)), + ChainVerdict::Keep, + ] + ); + let spec: Vec<_> = pending + .iter() + .map(|(sidecar, pending)| spec_kzg_verdict(sidecar, pending)) + .collect(); + assert_eq!(together, spec); + let alone: Vec<_> = group + .iter() + .map(|sidecar| chain_checks_for(&store, sidecar, now)) + .collect(); + assert_eq!(together, alone); + } + + /// Prints the time of one batch over `k` columns against `k` separate + /// calls, the fastest of several runs of each so a loaded machine skews + /// it less. Run with `--ignored --nocapture`. + #[test] + #[ignore = "benchmark: prints timings"] + fn bench_kzg_batch_against_separate_calls() { + use std::time::Instant; + const BLOBS: usize = 9; + let all: Vec = (0..preset::NUMBER_OF_COLUMNS as u64).collect(); + let items = real_sidecars(BLOBS, &all); + for k in [1usize, 2, 4, 8, 16, 32, 64, 128] { + let subset = &items[..k]; + let (mut batched, mut separate) = (Vec::new(), Vec::new()); + for _ in 0..15 { + let start = Instant::now(); + assert!( + verdicts_of(subset) + .iter() + .all(|verdict| *verdict == ChainVerdict::Keep) + ); + batched.push(start.elapsed()); + let start = Instant::now(); + for (sidecar, pending) in subset { + assert_eq!(kzg_verdict(sidecar, pending), ChainVerdict::Keep); + } + separate.push(start.elapsed()); + } + let b = batched.into_iter().min().expect("measured"); + let s = separate.into_iter().min().expect("measured"); + println!( + "k={k} blobs={BLOBS}: batch {b:?}, separate {s:?}, ratio {:.2}", + s.as_secs_f64() / b.as_secs_f64() + ); + } + } } diff --git a/crates/net/p2p/Cargo.toml b/crates/net/p2p/Cargo.toml index 22154e141..c81f4082d 100644 --- a/crates/net/p2p/Cargo.toml +++ b/crates/net/p2p/Cargo.toml @@ -83,3 +83,8 @@ libssz-types.workspace = true sha2 = "0.10" hex.workspace = true + +[dev-dependencies] +# `test-utils` for `helpers::test_state`: a test that needs a stored post-state +# builds one with it. +ethlambda-state-transition = { workspace = true, features = ["test-utils"] } diff --git a/crates/net/p2p/src/beacon/column_checks.rs b/crates/net/p2p/src/beacon/column_checks.rs index 91fbd1efe..b1d494662 100644 --- a/crates/net/p2p/src/beacon/column_checks.rs +++ b/crates/net/p2p/src/beacon/column_checks.rs @@ -13,25 +13,312 @@ //! wait on imported (`BlockChainToP2P::check_data_column_sidecars`): a fulu //! sidecar's parent, a gloas sidecar's own block. //! -//! Each sidecar is checked on a blocking thread, bounded by -//! [`P2PServer::column_check_permits`]. Unlike a gossip verdict, nothing here -//! has a deadline, so a sidecar with no free permit waits for one rather than -//! being dropped. +//! The sidecars of one call are grouped by the block they belong to, each +//! `(block, index)` once ([`group_by_block`]), and each group is checked in +//! two stages on blocking threads, bounded by +//! [`P2PServer::column_check_permits`]: +//! +//! 1. Every rule before the KZG proofs, sidecar by sidecar on one thread, +//! through one [`ChainPrechecks`], so the rules that read only what the +//! block's sidecars share (the signed header's, or the gloas block lookup) +//! run once for the group. The sidecars this settles, parked or dropped, +//! are forwarded as soon as it finishes. +//! 2. The KZG proofs of the sidecars still standing, in sub-batches of at most +//! [`KZG_BATCH_COLUMNS`] columns, each one `verify_cell_kzg_proof_batch` +//! call on a thread of its own. The first runs on the permit the group's +//! prechecks held; each further one waits for a permit of its own, so a +//! block with many columns spreads over the free permits rather than +//! queueing on one thread. Once every sub-batch is back, the sidecars kept +//! go to the chain actor as one batch, so it can count a block's columns +//! toward availability without waiting for the rest of a range answer. +//! +//! A permit is held for one group's prechecks and first sub-batch, or for +//! one further sub-batch. The proofs' share of a hold is bounded by +//! [`KZG_BATCH_COLUMNS`], not by how many columns the block has. The +//! prechecks' share is not bounded that way: they take the group's sidecars in +//! turn, one header judgment (a state read, and past the lookahead a state +//! advance) plus a check per sidecar, and how long that holds a permit on a +//! live node is unmeasured. Nothing awaits a permit while it holds one, so +//! groups and sub-batches cannot deadlock the pool between them. The pool is +//! shared with the envelope checks (`envelope_checks.rs`, one permit per +//! envelope), and tokio serves waiting requests in arrival order, so an +//! envelope waits behind only the requests queued before it. +//! +//! Unlike a gossip verdict, nothing here has a deadline, so a group or +//! sub-batch with no free permit waits for one rather than being dropped. +//! Each call runs on one task, which takes a permit per group in turn, and +//! each group on a task of its own until its last sub-batch is back. A group +//! task waiting for a sub-batch permit holds none, so the group tasks alive at +//! once are bounded by the groups of the calls in flight, whose sidecars are +//! in memory already, rather than by the pool. + +use std::collections::{HashMap, HashSet}; +use std::panic::{AssertUnwindSafe, catch_unwind}; +use std::sync::Arc; -use ethlambda_state_transition::beacon::gossip::column::{self, ChainVerdict}; +use ethlambda_network_api::P2PToBlockChainRef; +use ethlambda_state_transition::beacon::gossip::column::{ + self, ChainPrechecks, ChainVerdict, PendingKzg, +}; use ethlambda_state_transition::beacon::gossip::{IgnoreReason, Outcome}; +use ethlambda_storage::Store; use ethlambda_types::beacon::containers::DataColumnSidecar; +use ethlambda_types::beacon::primitives::Root; use ethlambda_types::time::unix_now_ms; -use tracing::{error, warn}; +use tokio::sync::{OwnedSemaphorePermit, Semaphore}; +use tracing::{debug, error, warn}; use crate::{P2PServer, metrics}; -/// Run the chain checks on `sidecars`, then send the chain actor the ones that -/// passed as one batch and the ones waiting on a block as another. +/// The most columns one KZG batch verifies. A block's columns past this are +/// split into near-equal sub-batches, each on a permit of its own. +/// +/// One batch costs about 3 ms plus 0.55 ms per column (9 blobs, +/// `bench_kzg_batch_against_separate_calls`: one column 3.2 ms, 8 columns +/// 7.9 ms, 16 columns 12.6 ms, 128 columns 73 ms, on one core). Computed from +/// those, at 16: the 8 columns a node samples are one batch, a third of the CPU +/// of eight separate checks; a supernode's 128 are 8 batches, about 13 ms of +/// wall-clock with 8 permits free where one batch takes 73 ms; and a failed +/// batch's one-by-one re-check covers 16 columns at most. +const KZG_BATCH_COLUMNS: usize = 16; + +/// A sidecar every rule before the KZG proofs passed, with what its proofs +/// are checked against. +type Pending = (DataColumnSidecar, PendingKzg); + +/// What some of one block's sidecars came to: the ones to store, the ones to +/// park, and the reason for each one dropped. +#[derive(Default)] +struct GroupVerdicts { + keep: Vec, + awaiting_parent: Vec, + dropped: Vec, +} + +impl GroupVerdicts { + fn record(&mut self, sidecar: DataColumnSidecar, verdict: ChainVerdict) { + match verdict { + ChainVerdict::Keep => self.keep.push(sidecar), + ChainVerdict::AwaitParent => self.awaiting_parent.push(sidecar), + ChainVerdict::Drop(outcome) => self.dropped.push(outcome), + } + } + + fn merge(&mut self, other: GroupVerdicts) { + self.keep.extend(other.keep); + self.awaiting_parent.extend(other.awaiting_parent); + self.dropped.extend(other.dropped); + } +} + +/// Group `sidecars` by block, in the order each block first appears, keeping +/// only the first sidecar of each `(block, index)`. Returns the groups and how +/// many repeats were left out. /// -/// Returns at once: the checks run on a task of their own, so the p2p actor +/// The repeats are what bound a group: a block has `NUMBER_OF_COLUMNS` +/// columns, and the checks refuse an index past that before the KZG stage, so +/// no group sends more than `NUMBER_OF_COLUMNS` sidecars to it, however many +/// copies a peer's answer carries (a range answer may hold 16,384). Nothing +/// else would catch a repeat within one call: the answer filters match on +/// block and index, not on repeats of them, and `AlreadyStored` cannot fire +/// before the group is done. The first copy is the one kept, as the chain +/// actor's own parking keeps the first (`queue_sidecar_awaiting_parent`), and +/// the repeats are not judged even when the first copy fails its checks and a +/// later one would pass: an honest answer never repeats a column, and judging +/// copies in turn until one passes is the cost this bounds. Every caller hands +/// over one source's sidecars (one peer's answer, one gossip message, the +/// parked set of one block, this node's own), so a repeat is that source's. +fn group_by_block(sidecars: Vec) -> (Vec>, usize) { + let mut positions: HashMap = HashMap::new(); + let mut seen: HashSet<(Root, u64)> = HashSet::new(); + let mut groups: Vec> = Vec::new(); + let mut repeats = 0; + for sidecar in sidecars { + let block = sidecar.block_root(); + if !seen.insert((block, sidecar.index())) { + repeats += 1; + continue; + } + let position = *positions.entry(block).or_insert_with(|| { + groups.push(Vec::new()); + groups.len() - 1 + }); + groups[position].push(sidecar); + } + (groups, repeats) +} + +/// Split `items` into the fewest sub-batches of at most [`KZG_BATCH_COLUMNS`], +/// in order, no two differing in size by more than one. +fn sub_batches(items: Vec) -> Vec> { + let count = items.len().div_ceil(KZG_BATCH_COLUMNS); + if count == 0 { + return Vec::new(); + } + // The first `longer` sub-batches take one item more than the rest. + let (base, longer) = (items.len() / count, items.len() % count); + let mut items = items.into_iter(); + (0..count) + .map(|batch| { + let size = base + usize::from(batch < longer); + items.by_ref().take(size).collect() + }) + .collect() +} + +/// The first stage: every rule of one block's sidecars that does not read the +/// cells. Returns what those rules settled and the sidecars whose KZG proofs +/// decide. +/// +/// Each sidecar gets the verdict [`column::chain_checks_for`] would give it: +/// the proofs are the last rule on the chain path, so running the other rules +/// first and the proofs afterwards, batched, changes nothing but how the +/// proofs are paid for. +fn precheck_group( + store: &Store, + sidecars: Vec, + now_ms: u64, +) -> (GroupVerdicts, Vec) { + let mut settled = GroupVerdicts::default(); + let mut pending = Vec::new(); + let mut prechecks = ChainPrechecks::new(store, now_ms); + for sidecar in sidecars { + // A panic inside the checks (a `Store` read's `.expect()` on a DB + // error, say) costs that one sidecar, dropped and counted as + // `internal`. `ChainPrechecks` keeps an answer only once it is + // complete, so the next sidecar works it out again. + let precheck = catch_unwind(AssertUnwindSafe(|| prechecks.check(&sidecar))); + match precheck { + Ok(Ok(kzg)) => pending.push((sidecar, kzg)), + Ok(Err(verdict)) => settled.record(sidecar, verdict), + Err(_) => { + error!("A data column check panicked; dropping the sidecar"); + let internal = ChainVerdict::Drop(Outcome::Ignore(IgnoreReason::Internal)); + settled.record(sidecar, internal); + } + } + } + (settled, pending) +} + +/// The second stage, for one sub-batch: its sidecars' KZG proofs in one +/// batch, each re-verified alone if the batch fails ([`column::kzg_verdicts`]). +fn verify_batch(batch: Vec) -> GroupVerdicts { + let items: Vec<_> = batch.iter().map(|(sidecar, kzg)| (sidecar, kzg)).collect(); + let kzg_verdicts = column::kzg_verdicts(&items); + let mut verdicts = GroupVerdicts::default(); + for ((sidecar, _), verdict) in batch.into_iter().zip(kzg_verdicts) { + verdicts.record(sidecar, verdict); + } + verdicts +} + +/// Check one block's sidecars, `permit` already taken for them, and forward +/// what they come to: the parked and the dropped once the prechecks are done, +/// the kept once every sub-batch is. +async fn check_group( + store: Store, + blockchain: P2PToBlockChainRef, + permits: Arc, + group: Vec, + permit: OwnedSemaphorePermit, +) { + let group_size = group.len(); + let prechecked = + tokio::task::spawn_blocking(move || precheck_group(&store, group, unix_now_ms())).await; + let (settled, pending) = match prechecked { + Ok(prechecked) => prechecked, + // Tokio has already caught the panic. A sidecar's own checks catch + // theirs, so this one came from outside them, and the whole group is + // lost: none of it is forwarded, and all of it is counted as + // `internal`. + Err(err) => { + error!(%err, lost = group_size, "A data column precheck panicked; dropping the group"); + count_drops(Outcome::Ignore(IgnoreReason::Internal), group_size); + return; + } + }; + // Nothing the proofs say changes these, so they go now: a panic in a KZG + // batch below cannot cost them. + forward(&blockchain, settled); + + let mut own_permit = Some(permit); + let mut batches = Vec::new(); + let mut remaining = sub_batches(pending).into_iter(); + while let Some(batch) = remaining.next() { + // The group's own permit carries the first sub-batch. Each further one + // waits for a permit of its own, and the group holds none while it + // waits, the first having moved into its batch. + let permit = match own_permit.take() { + Some(permit) => permit, + None => match permits.clone().acquire_owned().await { + Ok(permit) => permit, + // The semaphore is never closed, so this does not happen + // today. Should it ever, the sidecars not yet sent to a batch + // are lost, and say so rather than vanish. + Err(_) => { + let lost = + batch.len() + remaining.by_ref().map(|rest| rest.len()).sum::(); + error!( + lost, + "The column check permits are closed; dropping a block's unverified sidecars" + ); + count_drops(Outcome::Ignore(IgnoreReason::Internal), lost); + break; + } + }, + }; + let size = batch.len(); + let verified = tokio::task::spawn_blocking(move || { + let _permit = permit; + verify_batch(batch) + }); + batches.push((size, verified)); + } + let mut verdicts = GroupVerdicts::default(); + for (size, batch) in batches { + match batch.await { + Ok(batch) => verdicts.merge(batch), + // Tokio has already caught the panic. That sub-batch's sidecars + // are lost, not forwarded and counted as `internal`; the group's + // other sub-batches and everything settled above are not. + Err(err) => { + error!(%err, lost = size, "A data column KZG batch panicked; dropping its sidecars"); + count_drops(Outcome::Ignore(IgnoreReason::Internal), size); + } + } + } + forward(&blockchain, verdicts); +} + +/// Send the chain actor some of one block's verdicts: the sidecars to store as +/// one batch, and the ones waiting on a block as another, and count the drops. +fn forward(blockchain: &P2PToBlockChainRef, verdicts: GroupVerdicts) { + for outcome in verdicts.dropped { + count_drop(outcome); + } + if !verdicts.keep.is_empty() { + let _ = blockchain + .new_data_column_sidecars(verdicts.keep) + .inspect_err(|err| warn!(%err, "Failed to forward checked data column sidecars")); + } + if !verdicts.awaiting_parent.is_empty() { + let _ = blockchain + .data_column_sidecars_awaiting_parent(verdicts.awaiting_parent) + .inspect_err( + |err| warn!(%err, "Failed to forward data column sidecars awaiting a parent"), + ); + } +} + +/// Run the chain checks on `sidecars`, then send the chain actor, per block, +/// the ones waiting on a block as one batch and the ones that passed as +/// another. +/// +/// Returns at once: the checks run on tasks of their own, so the p2p actor /// goes on handling swarm events while a range batch's KZG proofs are -/// verified. +/// verified. A block's sidecars are forwarded when that block's group +/// finishes, not when every group of the call has. pub(crate) fn check_and_forward(server: &P2PServer, sidecars: Vec) { if sidecars.is_empty() { return; @@ -42,67 +329,49 @@ pub(crate) fn check_and_forward(server: &P2PServer, sidecars: Vec 0 { + debug!( + repeats, + "Dropping data column sidecars that repeat a block and index of the same call" + ); + count_drops(Outcome::Ignore(IgnoreReason::AlreadySeen), repeats); + } + for group in groups { // The semaphore is never closed, so this only returns once a // permit is free. let Ok(permit) = permits.clone().acquire_owned().await else { return; }; - let store = store.clone(); - checks.push(tokio::task::spawn_blocking(move || { - let _permit = permit; - let verdict = column::chain_checks_for(&store, &sidecar, unix_now_ms()); - (sidecar, verdict) - })); - } - - let mut keep = Vec::new(); - let mut awaiting_parent = Vec::new(); - for check in checks { - // A panic inside the checks (a `Store` read's `.expect()` on a DB - // error, say) costs that one sidecar, which is what dropping it - // would have done anyway. Tokio has already caught it. - let (sidecar, verdict) = match check.await { - Ok(checked) => checked, - Err(err) => { - error!(%err, "A data column check panicked; dropping the sidecar"); - continue; - } - }; - match verdict { - ChainVerdict::Keep => keep.push(sidecar), - ChainVerdict::AwaitParent => awaiting_parent.push(sidecar), - ChainVerdict::Drop(outcome) => count_drop(outcome), - } - } - - if !keep.is_empty() { - let _ = blockchain - .new_data_column_sidecars(keep) - .inspect_err(|err| warn!(%err, "Failed to forward checked data column sidecars")); - } - if !awaiting_parent.is_empty() { - let _ = blockchain - .data_column_sidecars_awaiting_parent(awaiting_parent) - .inspect_err( - |err| warn!(%err, "Failed to forward data column sidecars awaiting a parent"), - ); + tokio::spawn(check_group( + store.clone(), + blockchain.clone(), + permits.clone(), + group, + permit, + )); } }); } /// Count a dropped sidecar under its reason. +fn count_drop(outcome: Outcome) { + count_drops(outcome, 1); +} + +/// Count `count` sidecars dropped for the same reason, in one increment. /// /// An already stored one is not counted: consecutive range batches ask for /// overlapping spans of columns, so re-deliveries are routine, and they are -/// duplicates rather than rejections. -fn count_drop(outcome: Outcome) { - if outcome == Outcome::Ignore(IgnoreReason::AlreadyStored) { +/// duplicates rather than rejections. A repeat within one call is counted +/// (`already_seen`): an honest peer never sends one. So is every sidecar a +/// panic in the checks, or a closed permit pool, cost (`internal`). +fn count_drops(outcome: Outcome, count: usize) { + if count == 0 || outcome == Outcome::Ignore(IgnoreReason::AlreadyStored) { return; } let (_, reason) = outcome.labels(); - metrics::inc_data_column_rejected(reason); + metrics::inc_data_column_rejected(reason, count as u64); } #[cfg(test)] @@ -257,4 +526,245 @@ mod tests { drop(server); assert_eq!(received.recv().await, None); } + + /// A call mixing blocks forwards each block's sidecars on their own: the + /// chain actor hears about a block as soon as that block's group is done, + /// and a block's sidecars arrive together. + #[tokio::test] + async fn each_block_of_a_mixed_call_is_forwarded_on_its_own() { + let mut server = unconnected_beacon_server(Config::mainnet(), 0).await; + let (sender, mut received) = mpsc::unbounded_channel(); + server.blockchain = Some(Arc::new(RecordingChain(sender))); + + // Two columns of one block (slot 5) and one of another (slot 6), all + // with a parent the store has never seen, interleaved. A third block + // (slot 7) is malformed and goes nowhere. + let a0 = DataColumnSidecar::Fulu(valid_shaped_sidecar(5, 0)); + let b0 = DataColumnSidecar::Fulu(valid_shaped_sidecar(6, 0)); + let a1 = DataColumnSidecar::Fulu(valid_shaped_sidecar(5, 1)); + let mut malformed = valid_shaped_sidecar(7, 0); + malformed.kzg_commitments = Default::default(); + + check_and_forward( + &server, + vec![ + a0.clone(), + b0.clone(), + a1.clone(), + DataColumnSidecar::Fulu(malformed), + ], + ); + + let mut batches = Vec::new(); + for _ in 0..2 { + let Some(Forwarded::AwaitingParent(batch)) = received.recv().await else { + panic!("expected a batch awaiting its parent"); + }; + batches.push(batch); + } + batches.sort_by_key(|batch| batch.len()); + assert_eq!(batches, vec![vec![b0], vec![a0, a1]]); + drop(server); + assert_eq!(received.recv().await, None); + } + + #[test] + fn sidecars_are_grouped_by_block_in_first_seen_order() { + let a0 = DataColumnSidecar::Fulu(valid_shaped_sidecar(5, 0)); + let b0 = DataColumnSidecar::Fulu(valid_shaped_sidecar(6, 0)); + let a1 = DataColumnSidecar::Fulu(valid_shaped_sidecar(5, 1)); + let gloas_of = |byte, index| { + DataColumnSidecar::Gloas(gloas::DataColumnSidecar { + index, + beacon_block_root: Root::repeat_byte(byte), + ..Default::default() + }) + }; + let (g1, g2, g1_next) = (gloas_of(1, 0), gloas_of(2, 0), gloas_of(1, 1)); + + let (groups, repeats) = group_by_block(vec![ + a0.clone(), + b0.clone(), + a1.clone(), + g1.clone(), + g2.clone(), + g1_next.clone(), + ]); + + assert_eq!( + groups, + vec![vec![a0, a1], vec![b0], vec![g1, g1_next], vec![g2]] + ); + assert_eq!(repeats, 0); + } + + /// A `(block, index)` an answer carries more than once is judged once, + /// on its first copy, however many copies follow: what keeps a group, and + /// so the KZG stage, at one sidecar per column. + #[test] + fn a_repeated_block_and_index_is_kept_once() { + let first = valid_shaped_sidecar(5, 0); + let mut other_cells = first.clone(); + other_cells.column[0] = vec![1u8; ethlambda_types::beacon::preset::BYTES_PER_CELL] + .try_into() + .expect("exact cell size"); + let next = valid_shaped_sidecar(5, 1); + let mut sidecars = vec![DataColumnSidecar::Fulu(first.clone())]; + sidecars.extend(std::iter::repeat_n( + DataColumnSidecar::Fulu(other_cells), + 1000, + )); + sidecars.push(DataColumnSidecar::Fulu(next.clone())); + sidecars.push(DataColumnSidecar::Fulu(first.clone())); + + let (groups, repeats) = group_by_block(sidecars); + + assert_eq!( + groups, + vec![vec![ + DataColumnSidecar::Fulu(first), + DataColumnSidecar::Fulu(next) + ]] + ); + assert_eq!(repeats, 1001); + } + + /// Sub-batches hold at most [`KZG_BATCH_COLUMNS`] each, as few as that + /// allows, no two differing by more than one, and in order. 100 columns + /// are seven sub-batches of 15 and 14, not six of 15 and a short one of 10. + #[test] + fn sub_batches_are_bounded_near_equal_and_in_order() { + let sizes = |count: usize| -> Vec { + sub_batches((0..count).collect::>()) + .iter() + .map(Vec::len) + .collect() + }; + assert_eq!(sizes(0), Vec::::new()); + assert_eq!(sizes(1), vec![1]); + assert_eq!(sizes(8), vec![8]); + assert_eq!(sizes(KZG_BATCH_COLUMNS), vec![KZG_BATCH_COLUMNS]); + assert_eq!(sizes(KZG_BATCH_COLUMNS + 1), vec![9, 8]); + assert_eq!(sizes(128), vec![16; 8]); + assert_eq!(sizes(100), vec![15, 15, 14, 14, 14, 14, 14]); + let flattened: Vec = sub_batches((0..100).collect()).concat(); + assert_eq!(flattened, (0..100).collect::>()); + } + + /// A blob of canonical field elements derived from `seed`: the top byte of + /// each 32-byte element is zero, which keeps it below the modulus. + fn blob(seed: u8) -> Vec { + (0..ethlambda_types::beacon::preset::BYTES_PER_BLOB) + .map(|i| { + if i % 32 == 0 { + 0 + } else { + (i as u8).wrapping_mul(31).wrapping_add(seed) + } + }) + .collect() + } + + /// A gloas block at `slot` whose bid commits to two real blobs, stored + /// under `root` with a post-state, and its sidecars at every column in + /// `columns`, carrying the real cells and proofs: sidecars every chain + /// rule passes. + fn stored_gloas_block_and_sidecars( + store: &mut Store, + slot: u64, + root: Root, + columns: &[u64], + ) -> Vec { + use ethlambda_state_transition::beacon::kzg; + let blobs = [blob(1), blob(2)]; + let commitments: Vec<_> = blobs + .iter() + .map(|blob| kzg::blob_to_kzg_commitment(blob).expect("commit")) + .collect(); + let cells_and_proofs: Vec<_> = blobs + .iter() + .map(|blob| kzg::compute_cells_and_kzg_proofs(blob).expect("cells")) + .collect(); + let mut block = gloas::SignedBeaconBlock { + message: gloas::BeaconBlock { + slot, + proposer_index: 0, + parent_root: Root::ZERO, + state_root: Root::ZERO, + body: Default::default(), + }, + signature: Default::default(), + }; + block + .message + .body + .signed_execution_payload_bid + .message + .blob_kzg_commitments = commitments.into(); + store + .insert_pending_block(root, SignedBeaconBlock::Gloas(block)) + .expect("insert the block"); + let state = ethlambda_state_transition::beacon::helpers::test_state::with_validators_at( + ethlambda_types::beacon::fork::ForkName::Gloas, + 1, + ); + store + .insert_state(root, state) + .expect("insert a post-state"); + columns + .iter() + .map(|&index| { + let column: Vec = cells_and_proofs + .iter() + .map(|(cells, _)| { + cells[index as usize] + .to_bytes() + .to_vec() + .try_into() + .expect("exact cell size") + }) + .collect(); + let proofs: Vec<_> = cells_and_proofs + .iter() + .map(|(_, proofs)| proofs[index as usize]) + .collect(); + gloas::DataColumnSidecar { + index, + slot, + beacon_block_root: root, + column: column.into(), + kzg_proofs: proofs.into(), + } + }) + .collect() + } + + /// A block with more columns than one KZG batch takes is verified in + /// sub-batches on permits of their own, and what passes comes back to + /// the chain actor as one batch, in order: here every column but the one + /// forged in the second sub-batch, which goes nowhere. + #[tokio::test] + async fn a_blocks_columns_that_verify_are_forwarded_together_across_sub_batches() { + let mut server = unconnected_beacon_server(Config::mainnet(), 0).await; + let (sender, mut received) = mpsc::unbounded_channel(); + server.blockchain = Some(Arc::new(RecordingChain(sender))); + + let all: Vec = (0..KZG_BATCH_COLUMNS as u64 + 2).collect(); + let mut sidecars = + stored_gloas_block_and_sidecars(&mut server.store, 5, Root::repeat_byte(0x5a), &all); + let forged = KZG_BATCH_COLUMNS - 1; + let mut cell = sidecars[forged].column[0].to_vec(); + cell[31] ^= 1; + sidecars[forged].column[0] = cell.try_into().expect("exact cell size"); + let sidecars: Vec = + sidecars.into_iter().map(DataColumnSidecar::Gloas).collect(); + let mut expected = sidecars.clone(); + expected.remove(forged); + + check_and_forward(&server, sidecars); + + assert_eq!(received.recv().await, Some(Forwarded::Checked(expected))); + drop(server); + assert_eq!(received.recv().await, None); + } } diff --git a/crates/net/p2p/src/lib.rs b/crates/net/p2p/src/lib.rs index cc9ca2bd1..a108dc97f 100644 --- a/crates/net/p2p/src/lib.rs +++ b/crates/net/p2p/src/lib.rs @@ -227,11 +227,16 @@ const MAX_SYNC_RANGE: u64 = MAX_REQUEST_BLOCKS * 64; // 65,536 slots (~3 days) /// `lean_beacon_gossip_validation_seconds` has data from a follower. const GOSSIP_VALIDATION_PERMITS: usize = 128; -/// How many data column sidecars may be in the chain checks at once (see -/// [`beacon::column_checks`]). A separate pool from +/// How many blocking threads the chain checks of fetched and parked data +/// column sidecars, and of fetched execution payload envelopes, may occupy at +/// once (see [`beacon::column_checks`] and [`beacon::envelope_checks`]). A +/// permit covers one block's column prechecks and first KZG sub-batch, one +/// further KZG sub-batch of at most 16 columns, or one envelope: not one +/// sidecar, as it once did, nor all of a block's proofs. A separate pool from /// [`GOSSIP_VALIDATION_PERMITS`], so a range batch's hundreds of sidecars -/// cannot take every permit and leave gossip reporting `Overloaded`. A sidecar -/// arriving with none free waits for one. Revisit with data, as for gossip. +/// cannot take every permit and leave gossip reporting `Overloaded`. Work +/// arriving with none free waits for one, in arrival order. Revisit with data, +/// as for gossip. const COLUMN_CHECK_PERMITS: usize = 16; /// How many `beacon_aggregate_and_proof` and `beacon_attestation_{subnet_id}` diff --git a/crates/net/p2p/src/metrics.rs b/crates/net/p2p/src/metrics.rs index c8f5f5cba..b53569837 100644 --- a/crates/net/p2p/src/metrics.rs +++ b/crates/net/p2p/src/metrics.rs @@ -408,10 +408,12 @@ static LEAN_BEACON_GOSSIP_VERDICT_EXPIRED_TOTAL: LazyLock = LazyL /// Every reason `beacon::column_checks` can count a sidecar under: the /// `Outcome` reason labels `column::chain_checks` drops with, less -/// `already_stored`, which is a duplicate rather than a rejection. Seeded at +/// `already_stored`, which is a duplicate rather than a rejection, plus +/// `already_seen`, a `(block, index)` repeated within one call. Seeded at /// zero by [`init`], so a reason this node never fires is still visible on a /// dashboard. const DATA_COLUMN_REJECT_REASONS: &[&str] = &[ + "already_seen", "malformed", "future_slot", "finalized", @@ -435,12 +437,12 @@ static LEAN_DATA_COLUMNS_REJECTED_TOTAL: LazyLock = LazyLock::new .unwrap() }); -/// Count one sidecar the chain checks dropped. `reason` is one of -/// [`DATA_COLUMN_REJECT_REASONS`]. -pub fn inc_data_column_rejected(reason: &str) { +/// Count `count` sidecars the chain checks dropped for one reason. `reason` is +/// one of [`DATA_COLUMN_REJECT_REASONS`]. +pub fn inc_data_column_rejected(reason: &str, count: u64) { LEAN_DATA_COLUMNS_REJECTED_TOTAL .with_label_values(&[reason]) - .inc(); + .inc_by(count); } /// Register the metrics that should be visible before their first diff --git a/crates/net/p2p/src/req_resp/handlers.rs b/crates/net/p2p/src/req_resp/handlers.rs index 5801a84f2..9809d59ce 100644 --- a/crates/net/p2p/src/req_resp/handlers.rs +++ b/crates/net/p2p/src/req_resp/handlers.rs @@ -2493,11 +2493,15 @@ async fn handle_data_column_sidecars_range_response( return; } - // One batch through the checks, which forward what passes as one - // message. A range answer is the largest batch this node ever takes - // delivery of, and it arrives precisely when the chain actor is busiest - // draining a backlog, so a message per sidecar would put that many - // mailbox hops between the answer and the imports waiting on it. + // The whole answer through the checks in one call, which forward per + // block: the sidecars that pass as one message and the ones waiting on a + // block as another, so at most two per block, not one per sidecar. A range + // answer is the largest batch this node ever takes delivery of, and it + // arrives precisely when the chain actor is busiest draining a backlog, so + // a message per sidecar would put that many mailbox hops between the + // answer and the imports waiting on it; a message per block lets the + // actor count a block's columns toward availability as soon as that block + // is checked, rather than after the slowest block of the span. column_checks::check_and_forward(server, in_range); } diff --git a/docs/beacon_wire.md b/docs/beacon_wire.md index 2cef98120..cf6428303 100644 --- a/docs/beacon_wire.md +++ b/docs/beacon_wire.md @@ -676,6 +676,53 @@ NUMBER_OF_COLUMNS`) bounds what one request may ask for; an answer over that is truncated, not refused, matching how the block protocols treat a peer built to an older, wider ceiling. +*Checking* what arrives over req/resp, and what gossip left unjudged, is +`beacon/column_checks.rs`. The sidecars of one call are grouped by block (a +fulu sidecar's signed header root, a gloas sidecar's `beacon_block_root`), +keeping the first sidecar of each block and column index: an honest answer +never repeats one, a faulty one's repeats are dropped unjudged and counted as +`already_seen`, and that bounds what one block can send to the KZG proofs at +`NUMBER_OF_COLUMNS` sidecars. Each group is then checked in two stages, on +`column_check_permits` permits. + +First, every rule before the KZG proofs, sidecar by sidecar on one thread, in +the order `chain_checks` has always run them (`ChainPrechecks`). The rules that +read only what the block's sidecars share run once for the group, when the +first sidecar gets that far: for fulu, everything judged against the signed +header (the parent's post-state, or the head-state signature check before +parking, then `NotAfterParent`, the signature, the proposer and the finalized +ancestry), remembered under the header's root and signature, so a forged +signature is judged on its own; for gloas, the lookup and decode of the block +the sidecar names. The sidecars this settles, parked or dropped, are forwarded +at once. + +Second, the proofs of every sidecar still standing, in near-equal sub-batches +of at most 16 columns (`KZG_BATCH_COLUMNS`), each one +`verify_cell_kzg_proof_batch` call (`kzg_verdicts`) on a permit of its own, the +first on the group's. The proofs are the last rule on this path, so each +sidecar's verdict is the one it would have had alone. A batch that fails says +nothing about which sidecar is at fault, so each of its sidecars is verified +again by itself and only the ones that fail are rejected. A block's kept +sidecars go to the chain actor as one batch once its last sub-batch is back, +not when the whole call is. Gossip keeps one message, one verdict, and is not +batched. + +The sub-batch size trades wall-clock against CPU. Measured locally at 9 blobs +(`bench_kzg_batch_against_separate_calls`), one column's proofs take 3.2 ms, a +batch of 8 columns 7.9 ms, of 16 12.6 ms, and of 128 73 ms, each on one +thread. Computed from those, assuming free permits and cores: a node's 8 +sampled columns are one batch, taking 7.9 ms of wall-clock and 7.9 ms of CPU, +where eight separate checks running side by side would take 3.2 ms of +wall-clock but 25.8 ms of CPU between them, so the batch is slower to finish +for a third of the CPU. A supernode's 128 are 8 sub-batches, about 13 ms of +wall-clock against 73 ms as one batch and 26 ms as separate checks, for 101 ms +of CPU against 73 ms and 414 ms. A permit is held for one group's prechecks +and first sub-batch, or for one further sub-batch, so a block's proofs never +hold one permit from first column to last; the prechecks' share of a hold +runs the group's sidecars in turn and is not measured. The envelope checks +that share the pool wait in arrival order behind only what was queued before +them. + *Asking* happens on two paths, bulk and per block. The bulk one rides with range sync: every `BeaconBlocksByRange` batch sends a diff --git a/docs/metrics.md b/docs/metrics.md index c9264b677..f4d70b5f9 100644 --- a/docs/metrics.md +++ b/docs/metrics.md @@ -598,8 +598,8 @@ spec. | Name | Type | Usage | Sample collection event | Labels | Buckets | |------|------|-------|-------------------------|--------|---------| | `lean_data_columns_stored_total` | Counter | Data column sidecars verified and written to `Table::DataColumns` | On the chain actor storing a sidecar the p2p layer verified | | | -| `lean_data_columns_rejected_total` | Counter | Sidecars the p2p layer's chain checks dropped, by reason | On each drop in `beacon::column_checks`, except an already stored sidecar | reason=malformed,future_slot,finalized,not_after_parent,unknown_proposer,bad_signature,wrong_proposer,finalized_not_ancestor,parent_not_ready,inclusion_proof,kzg,internal | | -| `lean_data_column_kzg_verify_seconds` | Histogram | Time spent batch-verifying one sidecar's cells against its own commitments | On each KZG batch verification, in gossip validation or the chain checks | | 0.001, 0.0025, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0 | +| `lean_data_columns_rejected_total` | Counter | Sidecars the p2p layer's chain checks dropped, by reason | On each drop in `beacon::column_checks`, except an already stored sidecar | reason=already_seen,malformed,future_slot,finalized,not_after_parent,unknown_proposer,bad_signature,wrong_proposer,finalized_not_ancestor,parent_not_ready,inclusion_proof,kzg,internal | | +| `lean_data_column_kzg_verify_seconds` | Histogram | Time spent in one KZG cell-proof batch verification: one sidecar's cells in gossip validation, one sub-batch of at most 16 of one block's sidecars in the chain checks, which for every sub-batch after a block's first is how long it holds its `column_check_permits` permit (see [Data column sidecars](beacon_wire.md#data-column-sidecars)), plus one observation per sidecar if that batch fails and each is re-verified alone | On each KZG batch verification, in gossip validation or the chain checks | | 0.001, 0.0025, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0 | | `lean_data_column_fetch_failures_total` | Counter | `DataColumnsByRoot` lookups this node gave up on, by reason | On lookup abandonment | reason=no_peers,max_retries | | | `lean_blocks_held_for_columns` | Gauge | Blocks held out of fork choice pending their custody columns | On every hold, release, and finality eviction of the held-block set | | | | `lean_sidecars_awaiting_parent` | Gauge | Sidecars parked until their block's parent has a post-state | On every park, replay, and finality eviction of the parked set | | | @@ -613,8 +613,12 @@ gossiped one reported `Queue` or `Ignore(Overloaded)`, and a parked one sent back once its parent imported. A gossiped sidecar is judged first by the gossip rules and counted in `lean_beacon_gossip_validation_total{kind="data_column_sidecar"}`; the reasons -here are that counter's reason labels. The chain actor stores what reaches it -without checking it again. +here are that counter's reason labels, except `already_seen`, which here +counts a sidecar repeating a block and column index already in the same call +(one peer's answer, say), dropped without being judged. `internal` also counts +the sidecars a panic inside the checks cost, which are dropped (each one +whose own checks panicked, or a whole group or KZG sub-batch whose task did). +The chain actor stores what reaches it without checking it again. **Watch `lean_blocks_held_for_columns`.** It is the first symptom of a stalled availability gate, and a healthy node returns it to zero within a slot or two