diff --git a/CLAUDE.md b/CLAUDE.md index 5b40153de..9377677dc 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -347,7 +347,7 @@ actual_slot = finalized_slot + 1 + relative_index - Mesh size: 8 (6-12 bounds), heartbeat: 700ms - Beacon wire: `validate_messages()` is on, so every beacon message waits for a verdict (~4.2s before gossipsub's cache evicts it). Rules in `state_transition::beacon::gossip` (cheap half inline, stateful half on a bounded `spawn_blocking` task); plumbing in `p2p/src/beacon/verdict.rs`. Lean gossip still auto-forwards - Beacon peer scoring: lighthouse's parameters ported in `p2p/src/beacon/scoring.rs` (block, aggregate, 64 attestation subnets, exits, slashings; columns and sync contributions unscored), refreshed every slot from the head's active validator count. Mesh-delivery scoring (P3) is gated off while the head lags more than 4 slots and for an epoch after, because a catching-up node `Ignore`s every aggregate and would otherwise graylist its own mesh. Every held fork digest is parametrized before `transition::apply` joins it (gossipsub never scores a mesh grafted before its topic had parameters); the previous digest's weights are zeroed at the switch, and the switch restarts the P3 warmup. The swarm disconnects peers below the graylist (-16000) every 10 s, and peers gossipsub reported as `SlowPeer` in every check for a minute (a disabled gossipsub handler never drains its send queue, so the queue's message copies stay in memory until the connection closes). Gossipsub reports an expired forward like a full queue, so our own saturated uplink makes every peer look slow: a report counts only above `SLOW_PEER_REPORT_THRESHOLD` failures (Lighthouse's cut), and one check disconnects at most `MAX_SLOW_PEER_DISCONNECTS_PER_CHECK` peers. Lean runs unscored. See [`docs/beacon_wire.md`](docs/beacon_wire.md#peer-scoring) - - Data columns: every check runs in p2p. A column gossip did not accept (`Queue`/`Overloaded`), every fetched column, and parked columns replayed after their parent imports go through the chain checks in `p2p/src/beacon/column_checks.rs`, grouped by block with each `(block, index)` kept once. A group's rules before the KZG proofs run through one `column::ChainPrechecks`, which judges what the block's sidecars share (the fulu signed header's rules, the gloas block lookup) once; the proofs then go through `column::kzg_verdicts` in sub-batches of at most 16 columns, each on its own `column_check_permits` permit, a failed batch re-verified column by column. Each sidecar gets the verdict `column::chain_checks_for` gives it alone. The chain actor stores what it gets unchecked; only debug builds re-run `chain_checks_for` there + - Data columns: every check runs in p2p. A column gossip did not accept (`Queue`/`Overloaded`), every fetched column, and parked columns replayed after their parent imports go through the chain checks in `p2p/src/beacon/column_checks.rs`, grouped by block with each `(block, index)` kept once. A group's rules before the KZG proofs run through one `column::ChainPrechecks`, which judges what the block's sidecars share (the fulu signed header's rules, the gloas block lookup) once; the proofs then go through `column::kzg_verdicts` in sub-batches of at most 16 columns, each on its own `column_check_permits` permit, a failed batch re-verified column by column. Each sidecar gets the verdict `column::chain_checks_for` gives it alone. The chain actor stores what it gets unchecked; only debug builds re-run `chain_checks_for` there. A held fulu block's custody columns can also come from the execution client (`engine_getBlobsV2`, `crates/blockchain/src/beacon_blobs.rs`; asked only for a block at most `MAX_ASK_AGE_SLOTS` behind the clock, so not during range sync), put through the same `ChainPrechecks` and `kzg_verdicts` in the chain actor's blocking task (the spec lets a node skip that, this one does not: a stored column keeps every other copy out, and import trusts it), and handed to p2p as `SidecarOrigin::ExecutionLayer` through `check_data_column_sidecars`, which records them seen, publishes them on their subnets and forwards them; see `docs/beacon_engine.md` - Beacon subscribes seven global topics plus two node-id-derived subnet families: custody columns and backbone attestation subnets. One of the seven is `sync_committee_contribution_and_proof`; the four `sync_committee_{0..3}` subnets are diff --git a/crates/blockchain/Cargo.toml b/crates/blockchain/Cargo.toml index cf7b0b8f6..08ccb44b4 100644 --- a/crates/blockchain/Cargo.toml +++ b/crates/blockchain/Cargo.toml @@ -35,6 +35,9 @@ tracing.workspace = true hex.workspace = true [dev-dependencies] +# `test_state`'s validator keys, so a test can sign a block that has to pass +# the proposer-signature precheck before its blobs are asked for. +ethlambda-state-transition = { workspace = true, features = ["test-utils"] } hex = { workspace = true } libssz.workspace = true libssz-types.workspace = true diff --git a/crates/blockchain/src/beacon_blobs.rs b/crates/blockchain/src/beacon_blobs.rs new file mode 100644 index 000000000..fe6f0e776 --- /dev/null +++ b/crates/blockchain/src/beacon_blobs.rs @@ -0,0 +1,895 @@ +//! Data columns from this node's own execution client (`engine_getBlobsV2`). +//! +//! A fulu block held for its custody columns (see +//! `BlockChainServer::hold_block_for_columns`) is usually answered by gossip +//! within a fraction of a second. When it is not, the execution client may +//! already hold every blob the block commits to, since the transactions that +//! carry them went through its pool first. This module asks it once and turns +//! the answer into the custody columns still missing. +//! +//! # Flow +//! +//! ```text +//! hold (first time, recent block) -> spawn_fetch +//! task: engine_getBlobsV2(versioned hashes) 1 s ceiling +//! null / error -> count, give up (gossip and peer asks carry on) +//! every blob -> spawn_blocking: +//! block imported meanwhile? -> stop +//! custody columns still missing, none? -> stop +//! cells + sidecars +//! the chain checks, KZG included (check_built) +//! check_data_column_sidecars(.., ExecutionLayer) +//! p2p: records the columns as seen, publishes the new ones on their +//! subnets, forwards them to the chain actor, which stores them and +//! releases the held block +//! ``` +//! +//! # Why it is shaped this way +//! +//! - **Spec basis.** Fulu `p2p-interface.md`, "Distributed blob publishing +//! using blobs retrieved from local execution-layer client": honest nodes +//! SHOULD query `engine_getBlobsV2` as soon as they receive a valid +//! `beacon_block` or `data_column_sidecar` from gossip, convert a complete +//! answer to data columns and import them as if received on gossip. This +//! node asks on the first data-availability hold of a block at most +//! [`MAX_ASK_AGE_SLOTS`] behind the clock, whether the block came from +//! gossip or a by-root answer, once its proposer signature verifies against +//! its parent's state (see [`signed_by_its_proposer`]); it does not yet ask +//! when a sidecar arrives. Hence the hand-over to p2p rather than a direct +//! store write: publishing on subscribed subnets and updating the +//! anti-equivocation cache are MUSTs, and p2p owns both. +//! - **Recent blocks only.** The specification's trigger is gossip, so the +//! tip. Range sync holds older blocks by the batch, and asking for each +//! would be an unbounded fan-out of prechecks on the chain actor, engine +//! calls and cell computations, for blocks whose columns peers are already +//! being asked for (see [`MAX_ASK_AGE_SLOTS`]). +//! - **One attempt.** The call races gossip delivery of the same columns, the +//! specification's own timeout for the method is 1 s, and a retry after the +//! engine ladder's backoff would land after that race is decided. A miss +//! leaves the block to gossip and the per-slot peer asks, which is where it +//! was anyway. (The execution client can learn a blob a moment later, since +//! blob transactions keep arriving in its pool; a later ask is not useless, +//! just not worth the delay.) +//! - **V2, not V3.** V3 answers partially, but a column needs one cell from +//! every blob, so a partial blob set builds no complete column and this node +//! has no partial-column assembly. +//! - **Checked like a fetched column.** The specification lets a node skip +//! verifying blobs from its own execution client, but this node does not: +//! the built sidecars go through the same chain checks a fetched column does +//! ([`check_built`]), KZG proofs included, before anything is published or +//! stored. Two things depend on it. The proposer precheck below cannot name +//! the expected proposer for a slot past its parent's lookahead, while the +//! chain checks advance the parent's state to find out. And every row in +//! `DataColumns` is one this node verified: a column stored here keeps every +//! other copy of it out (`AlreadyStored`), and block import trusts the stored +//! columns, so a wrong blob or proof taken on trust would either stop the +//! block for good or import it on data nobody checked. The cost is one KZG +//! batch over the custody columns, on the blocking pool. See +//! `docs/spec_deviations.md`. +//! - **Not gloas.** From gloas on, data availability moves from the block to +//! the payload envelope, so a held gloas block has no columns to fill this +//! way. +//! +//! The cell computation is CPU-heavy, so it runs on tokio's blocking pool: off +//! the async runtime and off the chain actor, which only ever sees the +//! finished columns, through p2p. + +use std::time::Instant; + +use ethlambda_engine::EngineClient; +use ethlambda_network_api::{BlockChainToP2PRef, SidecarOrigin}; +use ethlambda_state_transition::beacon::data_columns; +use ethlambda_state_transition::beacon::gossip::column::{self, ChainPrechecks, ChainVerdict}; +use ethlambda_state_transition::beacon::gossip::{IgnoreReason, Outcome}; +use ethlambda_state_transition::beacon::precheck::{Reference, precheck_block}; +use ethlambda_state_transition::beacon::stf::deneb::kzg_commitment_to_versioned_hash; +use ethlambda_storage::Store; +use ethlambda_types::ShortRoot; +use ethlambda_types::beacon::containers::{DataColumnSidecar, SignedBeaconBlock, deneb, electra}; +use ethlambda_types::beacon::preset::CELLS_PER_EXT_BLOB; +use ethlambda_types::beacon::primitives::KzgProof; +use ethlambda_types::primitives::H256; +use ethlambda_types::time::unix_now_ms; +use tracing::{debug, error, info, warn}; + +use crate::metrics; + +/// How many slots behind the store's clock a held block may be and still have +/// the execution client asked for its blobs. +/// +/// Wide enough for what the specification's trigger covers (a block gossip +/// delivered in its own slot or late) and for a by-root answer filling in a +/// parent missed a slot or two back. Range sync is what it cuts off: every +/// block that delivers is older than the slot it arrives in, a catch-up holds +/// them by the batch, and asking for each would cost a parent-state read and +/// a BLS verification on the chain actor ([`signed_by_its_proposer`]), an +/// engine round trip, and on a hit a cell computation, all unbounded. Such a +/// block's columns are asked of peers at once (`hold_block_for_columns`), so +/// it does not wait for a redrive either way. How often the execution client +/// would still hold an older block's blobs is not measured. +pub(crate) const MAX_ASK_AGE_SLOTS: u64 = 4; + +/// Whether a held block at `slot` is recent enough, at `current_slot`, to ask +/// the execution client about (see [`MAX_ASK_AGE_SLOTS`]). +pub(crate) fn recent_enough_to_ask(slot: u64, current_slot: u64) -> bool { + slot.saturating_add(MAX_ASK_AGE_SLOTS) >= current_slot +} + +/// Whether a held block may have its columns built from the execution +/// client's blobs: its proposer signature, and the rest of +/// [`precheck_block`]'s rules, hold against its parent's post-state. +/// +/// A filter, not the guard: [`check_built`] runs every chain check on the +/// built columns before they leave this module, and is what keeps a forged +/// header out of p2p's seen cache and off the wire. This only saves asking the +/// execution client, and computing cells, for a block whose signature already +/// fails here. A block can reach its hold unverified, since range sync and +/// by-root answers never pass gossip validation and the import checks the +/// signature only after data availability. +/// +/// It cannot catch everything [`check_built`] does: past the parent state's +/// proposer lookahead, [`precheck_block`] checks the signature against the +/// key of whichever validator the block names, not that this validator is the +/// slot's proposer. +/// +/// The parent has a post-state whenever the import reaches a hold; a hold +/// without one is not asked about. +pub(crate) fn signed_by_its_proposer( + store: &Store, + block: &SignedBeaconBlock, + block_root: H256, +) -> bool { + let slot = block.slot(); + let Some(parent_state) = store + .get_state(&block.parent_root()) + .expect("DB read should succeed") + else { + debug!( + %slot, + block_root = %ShortRoot(&block_root.0), + "Not asking the execution client for blobs: the block's parent has no post-state" + ); + return false; + }; + let config = store.config(); + precheck_block(block, block_root, Reference::Parent(&parent_state), &config) + .inspect_err(|err| { + let proposer = block.proposer_index(); + warn!( + %slot, + proposer, + block_root = %ShortRoot(&block_root.0), + %err, + "Not asking the execution client for blobs: the block fails its precheck" + ) + }) + .is_ok() +} + +/// Ask the execution client for a held fulu block's blobs on a task of its +/// own, and hand the custody columns built from them to the p2p layer. +pub(crate) fn spawn_fetch( + engine: EngineClient, + p2p: BlockChainToP2PRef, + store: Store, + block: electra::SignedBeaconBlock, + block_root: H256, + custody_columns: Vec, +) { + tokio::spawn(fetch_and_publish( + engine, + p2p, + store, + block, + block_root, + custody_columns, + )); +} + +async fn fetch_and_publish( + engine: EngineClient, + p2p: BlockChainToP2PRef, + store: Store, + block: electra::SignedBeaconBlock, + block_root: H256, + custody_columns: Vec, +) { + let slot = block.message.slot; + let versioned_hashes: Vec = block + .message + .body + .blob_kzg_commitments + .iter() + .map(kzg_commitment_to_versioned_hash) + .collect(); + + let started = Instant::now(); + let answer = engine.get_blobs_v2(&versioned_hashes).await; + metrics::observe_engine_get_blobs_duration(started.elapsed()); + + let entries = match answer { + Err(err) => { + debug!(%slot, block_root = %ShortRoot(&block_root.0), %err, "engine_getBlobsV2 failed"); + metrics::inc_engine_get_blobs("error"); + return; + } + Ok(None) => { + debug!(%slot, block_root = %ShortRoot(&block_root.0), "Execution client is missing blobs for the block"); + metrics::inc_engine_get_blobs("miss"); + return; + } + Ok(Some(entries)) => { + metrics::inc_engine_get_blobs("hit"); + entries + } + }; + + let blob_count = entries.len(); + let work = tokio::task::spawn_blocking(move || { + // The block was imported while the execution client answered (gossip + // completed it first): nothing left to supply. + match store.has_state(&block_root) { + Ok(false) => {} + Ok(true) => return None, + Err(err) => { + error!(%slot, block_root = %ShortRoot(&block_root.0), %err, "Failed to check whether a held block was imported"); + return None; + } + } + let present = match store.data_column_indices_for(slot, &block_root) { + Ok(present) => present, + Err(err) => { + error!(%slot, block_root = %ShortRoot(&block_root.0), %err, "Failed to read a held block's stored data columns"); + return None; + } + }; + let missing: Vec = custody_columns + .iter() + .copied() + .filter(|index| !present.contains(index)) + .collect(); + if missing.is_empty() { + return None; + } + + let converted = Instant::now(); + let mut blobs = Vec::with_capacity(entries.len()); + let mut proofs: Vec = Vec::with_capacity(entries.len() * CELLS_PER_EXT_BLOB); + for entry in entries { + match deneb::Blob::try_from(entry.blob) { + Ok(blob) => blobs.push(blob), + Err(_) => { + warn!(%slot, block_root = %ShortRoot(&block_root.0), "Execution client's blob has the wrong size"); + return None; + } + } + proofs.extend(entry.proofs); + } + let sidecars = match data_columns::unverified_sidecars(&block, &blobs, &proofs, &missing) { + Ok(sidecars) => sidecars, + Err(err) => { + warn!(%slot, block_root = %ShortRoot(&block_root.0), %err, "Execution client's blobs do not fit the block"); + return None; + } + }; + metrics::observe_engine_blobs_to_columns(converted.elapsed()); + let sidecars = sidecars.into_iter().map(DataColumnSidecar::Fulu).collect(); + Some(check_built(&store, slot, block_root, sidecars)) + }) + .await; + + let sidecars = match work { + Ok(Some(sidecars)) if !sidecars.is_empty() => sidecars, + Ok(_) => return, + Err(err) => { + error!(%slot, block_root = %ShortRoot(&block_root.0), %err, "Blob-to-column task failed"); + return; + } + }; + + let columns = sidecars.len(); + if let Err(err) = p2p.check_data_column_sidecars(sidecars, SidecarOrigin::ExecutionLayer) { + error!(%slot, %err, "Failed to hand execution-client columns to P2P"); + return; + } + + info!( + %slot, + block_root = %ShortRoot(&block_root.0), + blobs = blob_count, + columns, + elapsed_ms = started.elapsed().as_millis() as u64, + "Built data columns from the execution client's blobs" + ); +} + +/// The built `sidecars` that pass every chain check a fetched column gets +/// ([`column::chain_checks`]'s rules, in the two stages p2p runs them for one +/// block: [`ChainPrechecks`], then one KZG batch). +/// +/// That is the structure, the clock and finality, `AlreadyStored`, the signed +/// header (parent state, slot order, signature, the expected proposer, past +/// the lookahead by advancing the parent's state), finalized ancestry, the +/// inclusion proof, and the cells against the block's commitments. The +/// sidecars all carry one signed header, so the header is judged once. +/// +/// A refusal drops the column and is counted by reason +/// (`lean_engine_blob_columns_refused_total`). `AlreadyStored` is not a +/// refusal: gossip or a peer answer stored that column while the execution +/// client answered, which is the race this call loses at times. Anything else +/// is a block whose header does not hold (a forged or wrong-proposer block that +/// got past the precheck) or an execution client whose blobs or proofs do not +/// match the block, and none of it may reach p2p's seen cache, the wire or +/// `DataColumns`. +fn check_built( + store: &Store, + slot: u64, + block_root: H256, + sidecars: Vec, +) -> Vec { + let mut prechecks = ChainPrechecks::new(store, unix_now_ms()); + let mut refused: Vec = Vec::new(); + let mut pending = Vec::with_capacity(sidecars.len()); + for sidecar in sidecars { + match prechecks.check(&sidecar) { + Ok(kzg) => pending.push((sidecar, kzg)), + Err(verdict) => refused.push(verdict), + } + } + + let verdicts = { + let items: Vec<_> = pending + .iter() + .map(|(sidecar, kzg)| (sidecar, kzg)) + .collect(); + column::kzg_verdicts(&items) + }; + let mut kept = Vec::with_capacity(pending.len()); + for ((sidecar, _), verdict) in pending.into_iter().zip(verdicts) { + match verdict { + ChainVerdict::Keep => kept.push(sidecar), + verdict => refused.push(verdict), + } + } + + let stored_meanwhile = ChainVerdict::Drop(Outcome::Ignore(IgnoreReason::AlreadyStored)); + for verdict in refused + .iter() + .filter(|&&verdict| verdict != stored_meanwhile) + { + let reason = match verdict { + ChainVerdict::Drop(outcome) => outcome.labels().1, + // The parent has a post-state whenever a block is held, so this + // is not expected; counted rather than asserted. + ChainVerdict::AwaitParent | ChainVerdict::Keep => "await_parent", + }; + metrics::inc_engine_blob_columns_refused(reason); + } + if let Some(verdict) = refused.iter().find(|&&verdict| verdict != stored_meanwhile) { + warn!( + %slot, + block_root = %ShortRoot(&block_root.0), + ?verdict, + refused = refused.len(), + kept = kept.len(), + "The chain checks refused columns built from the execution client's blobs; not publishing them" + ); + } + kept +} + +#[cfg(test)] +mod tests { + use std::sync::Arc; + use std::sync::atomic::{AtomicUsize, Ordering}; + use std::time::Duration; + + use ethlambda_engine::{EngineClient, JwtSecret}; + use ethlambda_state_transition::beacon::kzg; + use ethlambda_types::beacon::containers::SignedBeaconBlock; + use ethlambda_types::beacon::primitives::KzgCommitment; + use ethlambda_types::primitives::HashTreeRoot as _; + use libssz_types::SszList; + use tokio::io::{AsyncReadExt as _, AsyncWriteExt as _}; + use tokio::net::TcpListener; + + use super::*; + use crate::import_timing::ImportTimings; + use crate::tests::{bare_block, bare_state, beacon_server_recording, beacon_store, fulu_block}; + use ethlambda_state_transition::beacon::helpers::test_state; + use ethlambda_types::beacon::config::Config; + use ethlambda_types::beacon::constants::DOMAIN_BEACON_PROPOSER; + use ethlambda_types::beacon::fork::ForkName; + use ethlambda_types::beacon::preset::{self, BYTES_PER_BLOB}; + use ethlambda_types::beacon::signing::{ + compute_domain, compute_epoch_at_slot, compute_signing_root, + }; + + /// `beacon_store(0, 0)` whose anchor, the parent of every block here, has + /// a post-state registering validator 0 under the key [`sign`] uses, so a + /// signed block passes the precheck a hold runs before it asks the + /// execution client. + fn store_with_keyed_parent() -> Store { + store_with_keyed_parent_of(1) + } + + /// [`store_with_keyed_parent`] with `count` validators, each registered + /// under the key [`sign`] uses for its index. + fn store_with_keyed_parent_of(count: usize) -> Store { + let mut store = beacon_store(0, 0); + store + .insert_signed_block(H256::ZERO, bare_block(0, H256::repeat_byte(0xcc))) + .expect("insert"); + store + .insert_state(H256::ZERO, keyed_parent_state(count)) + .expect("insert"); + store + } + + /// `bare_state` with `count` validators keyed as [`sign`] signs. + fn keyed_parent_state(count: usize) -> ethlambda_types::beacon::containers::BeaconState { + let keyed = test_state::with_signing_validators_at(ForkName::Phase0, count); + let mut state = bare_state(); + for index in 0..count { + let validator = keyed.validator(index as u64).expect("keyed").clone(); + state + .push_validator(validator, preset::MAX_EFFECTIVE_BALANCE) + .expect("the validators fit the registry"); + } + state.apply_pending_mutations(); + state + } + + /// `block` signed by validator 0, its proposer, under the mainnet fork + /// schedule `beacon_store` runs on. + fn sign(mut block: SignedBeaconBlock) -> SignedBeaconBlock { + let config = Config::mainnet(); + let fork_version = + config.fork_version(config.fork_at_epoch(compute_epoch_at_slot(block.slot()))); + // `bare_state`'s genesis validators root. + let domain = compute_domain(DOMAIN_BEACON_PROPOSER, fork_version, H256::ZERO); + let signing_root = compute_signing_root(block.message_hash_tree_root(), domain); + let SignedBeaconBlock::Fulu(inner) = &mut block else { + unreachable!("these tests build fulu blocks"); + }; + inner.signature = test_state::sign_for(inner.message.proposer_index as usize, signing_root); + block + } + + /// A blob whose every field element is below the BLS modulus. + fn blob_bytes(seed: u8) -> Vec { + let mut bytes = vec![0u8; BYTES_PER_BLOB]; + for (i, element) in bytes.chunks_mut(32).enumerate() { + element[1] = seed; + element[2] = i as u8; + element[3] = (i >> 8) as u8; + } + bytes + } + + /// The `result` of an `engine_getBlobsV2` answer for `blobs`, with real + /// cell proofs. + fn answer_for(blobs: &[Vec]) -> String { + answer_with_proofs_of(blobs, blobs) + } + + /// An answer carrying `blobs`, each with the cell proofs of the blob at the + /// same position in `proofs_from`: [`answer_for`] when the two are equal, + /// a well-shaped answer with wrong proofs otherwise. + fn answer_with_proofs_of(blobs: &[Vec], proofs_from: &[Vec]) -> String { + let entries: Vec = blobs + .iter() + .zip(proofs_from) + .map(|(blob, proof_blob)| { + let (_, proofs) = kzg::compute_cells_and_kzg_proofs(proof_blob).unwrap(); + let proofs: Vec = proofs + .iter() + .map(|proof| format!("\"0x{}\"", hex::encode(proof.0))) + .collect(); + format!( + r#"{{"blob":"0x{}","proofs":[{}]}}"#, + hex::encode(blob), + proofs.join(",") + ) + }) + .collect(); + format!("[{}]", entries.join(",")) + } + + /// A loopback JSON-RPC server answering every request with `result`, and + /// counting them. + async fn mock_engine(result: String) -> (EngineClient, Arc) { + let requests = Arc::new(AtomicUsize::new(0)); + let listener = TcpListener::bind("127.0.0.1:0") + .await + .expect("a loopback port"); + let addr = listener.local_addr().expect("bound address"); + let counter = Arc::clone(&requests); + let payload = format!(r#"{{"jsonrpc":"2.0","id":1,"result":{result}}}"#); + tokio::spawn(async move { + loop { + let Ok((mut socket, _)) = listener.accept().await else { + return; + }; + let counter = Arc::clone(&counter); + let payload = payload.clone(); + tokio::spawn(async move { + let mut pending: Vec = Vec::new(); + loop { + let (head_end, body_len) = loop { + if let Some(end) = + pending.windows(4).position(|window| window == b"\r\n\r\n") + { + let head = String::from_utf8_lossy(&pending[..end]).to_lowercase(); + let len = head + .lines() + .find_map(|line| line.strip_prefix("content-length:")) + .and_then(|value| value.trim().parse::().ok()) + .unwrap_or(0); + break (end + 4, len); + } + let mut chunk = [0u8; 16 * 1024]; + match socket.read(&mut chunk).await { + Ok(0) | Err(_) => return, + Ok(read) => pending.extend_from_slice(&chunk[..read]), + } + }; + while pending.len() < head_end + body_len { + let mut chunk = [0u8; 16 * 1024]; + match socket.read(&mut chunk).await { + Ok(0) | Err(_) => return, + Ok(read) => pending.extend_from_slice(&chunk[..read]), + } + } + pending.drain(..head_end + body_len); + counter.fetch_add(1, Ordering::SeqCst); + + let response = format!( + "HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nContent-Length: {}\r\n\r\n{payload}", + payload.len() + ); + if socket.write_all(response.as_bytes()).await.is_err() { + return; + } + } + }); + } + }); + let client = EngineClient::new(format!("http://{addr}"), JwtSecret::new([0x0f; 32])) + .expect("a client"); + (client, requests) + } + + /// A fulu block at slot 1 committing to each of `blobs`, signed by its + /// proposer. + fn block_committing_to(blobs: &[Vec]) -> SignedBeaconBlock { + block_by(0, blobs) + } + + /// [`block_committing_to`], naming `proposer` and signed by its key. + fn block_by(proposer: u64, blobs: &[Vec]) -> SignedBeaconBlock { + let mut block = fulu_block(H256::ZERO, 1, 0); + let commitments: Vec = blobs + .iter() + .map(|blob| kzg::blob_to_kzg_commitment(blob).unwrap()) + .collect(); + let SignedBeaconBlock::Fulu(inner) = &mut block else { + unreachable!("fulu_block builds a fulu block"); + }; + inner.message.proposer_index = proposer; + inner.message.body.blob_kzg_commitments = SszList::try_from(commitments).unwrap(); + sign(block) + } + + const CUSTODY: [u64; 3] = [0, 5, 77]; + + /// Polls until `RecordingP2P` has a hand-over, or a few seconds pass. + async fn wait_for_hand_over(p2p: &crate::tests::RecordingP2P) { + for _ in 0..100 { + if !p2p.checks.lock().unwrap().is_empty() { + return; + } + tokio::time::sleep(Duration::from_millis(50)).await; + } + } + + #[tokio::test] + async fn a_full_answer_hands_over_exactly_the_custody_columns() { + let blobs = vec![blob_bytes(1), blob_bytes(2)]; + let block = block_committing_to(&blobs); + let root = block.message_hash_tree_root(); + let (engine, requests) = mock_engine(answer_for(&blobs)).await; + let (mut server, p2p) = beacon_server_recording(store_with_keyed_parent()); + server.engine = Some(engine); + server.custody_columns = CUSTODY.to_vec(); + + server.hold_block_for_columns(block.clone(), 1, ImportTimings::default()); + wait_for_hand_over(&p2p).await; + + let checks = p2p.checks.lock().unwrap(); + assert_eq!(checks.len(), 1, "one hand-over"); + assert_eq!( + *p2p.check_origins.lock().unwrap(), + vec![ethlambda_network_api::SidecarOrigin::ExecutionLayer] + ); + let indices: Vec = checks[0] + .iter() + .map(|sidecar| match sidecar { + DataColumnSidecar::Fulu(sidecar) => sidecar.index, + DataColumnSidecar::Gloas(_) => panic!("a fulu block yields fulu sidecars"), + }) + .collect(); + assert_eq!(indices, CUSTODY.to_vec()); + for sidecar in checks[0].iter() { + let DataColumnSidecar::Fulu(sidecar) = sidecar else { + unreachable!() + }; + let header = &sidecar.signed_block_header.message; + assert_eq!(header.slot, block.slot()); + assert_eq!(header.parent_root, block.parent_root()); + assert_eq!(header.hash_tree_root(), root); + assert_eq!(sidecar.column.len(), blobs.len()); + } + assert_eq!(requests.load(Ordering::SeqCst), 1); + } + + #[tokio::test] + async fn a_stored_custody_column_is_not_handed_over_again() { + let blobs = vec![blob_bytes(3)]; + let block = block_committing_to(&blobs); + let root = block.message_hash_tree_root(); + let (engine, _requests) = mock_engine(answer_for(&blobs)).await; + let (mut server, p2p) = beacon_server_recording(store_with_keyed_parent()); + server.engine = Some(engine); + server.custody_columns = CUSTODY.to_vec(); + server + .store + .put_data_column_sidecar(block.slot(), &root, 0, vec![1]) + .unwrap(); + + server.hold_block_for_columns(block, 1, ImportTimings::default()); + wait_for_hand_over(&p2p).await; + + let checks = p2p.checks.lock().unwrap(); + assert_eq!(checks.len(), 1); + let indices: Vec = checks[0] + .iter() + .map(|sidecar| match sidecar { + DataColumnSidecar::Fulu(sidecar) => sidecar.index, + DataColumnSidecar::Gloas(_) => panic!("a fulu block yields fulu sidecars"), + }) + .collect(); + assert_eq!(indices, vec![5, 77]); + } + + #[tokio::test] + async fn a_null_answer_hands_nothing_over() { + let blobs = vec![blob_bytes(4)]; + let block = block_committing_to(&blobs); + let (engine, requests) = mock_engine("null".to_string()).await; + let (mut server, p2p) = beacon_server_recording(store_with_keyed_parent()); + server.engine = Some(engine); + server.custody_columns = CUSTODY.to_vec(); + + server.hold_block_for_columns(block, 1, ImportTimings::default()); + for _ in 0..40 { + if requests.load(Ordering::SeqCst) > 0 { + break; + } + tokio::time::sleep(Duration::from_millis(25)).await; + } + tokio::time::sleep(Duration::from_millis(300)).await; + + assert_eq!(requests.load(Ordering::SeqCst), 1, "the engine was asked"); + assert!(p2p.checks.lock().unwrap().is_empty()); + } + + #[tokio::test] + async fn without_an_engine_nothing_is_asked_or_handed_over() { + let blobs = vec![blob_bytes(5)]; + let block = block_committing_to(&blobs); + let (mut server, p2p) = beacon_server_recording(store_with_keyed_parent()); + server.custody_columns = CUSTODY.to_vec(); + + server.hold_block_for_columns(block, 1, ImportTimings::default()); + tokio::time::sleep(Duration::from_millis(200)).await; + + assert!(p2p.checks.lock().unwrap().is_empty()); + } + + #[tokio::test] + async fn a_second_hold_of_the_same_root_does_not_ask_again() { + let blobs = vec![blob_bytes(6)]; + let block = block_committing_to(&blobs); + let (engine, requests) = mock_engine("null".to_string()).await; + let (mut server, _p2p) = beacon_server_recording(store_with_keyed_parent()); + server.engine = Some(engine); + server.custody_columns = CUSTODY.to_vec(); + + server.hold_block_for_columns(block.clone(), 1, ImportTimings::default()); + for _ in 0..40 { + if requests.load(Ordering::SeqCst) > 0 { + break; + } + tokio::time::sleep(Duration::from_millis(25)).await; + } + server.hold_block_for_columns(block, 1, ImportTimings::default()); + tokio::time::sleep(Duration::from_millis(300)).await; + + assert_eq!(requests.load(Ordering::SeqCst), 1); + } + + /// A block imported while the execution client answered (gossip + /// completed it first): the answer is a hit, but the re-check before the + /// build finds the block's state and nothing is built or handed over. + #[tokio::test] + async fn a_hit_for_a_block_imported_meanwhile_hands_nothing_over() { + let blobs = vec![blob_bytes(8)]; + let block = block_committing_to(&blobs); + let root = block.message_hash_tree_root(); + let mut store = beacon_store(0, 0); + store + .insert_signed_block(root, block.clone()) + .expect("insert"); + store.insert_state(root, bare_state()).expect("insert"); + let SignedBeaconBlock::Fulu(inner) = block else { + unreachable!("fulu_block builds a fulu block"); + }; + let (engine, requests) = mock_engine(answer_for(&blobs)).await; + let p2p = Arc::new(crate::tests::RecordingP2P::default()); + + fetch_and_publish(engine, p2p.clone(), store, inner, root, CUSTODY.to_vec()).await; + + assert_eq!(requests.load(Ordering::SeqCst), 1, "the engine was asked"); + assert!(p2p.checks.lock().unwrap().is_empty()); + } + + /// A block more than `MAX_ASK_AGE_SLOTS` behind the clock (range sync's) + /// is not asked about; one exactly that far behind still is. + #[tokio::test] + async fn only_a_recent_block_is_asked_about() { + let at_the_edge = block_committing_to(&[blob_bytes(9)]); + let too_old = block_committing_to(&[blob_bytes(10)]); + let (engine, requests) = mock_engine("null".to_string()).await; + let (mut server, _p2p) = beacon_server_recording(store_with_keyed_parent()); + server.engine = Some(engine); + server.custody_columns = CUSTODY.to_vec(); + + let edge = at_the_edge.slot() + MAX_ASK_AGE_SLOTS; + server.hold_block_for_columns(too_old, edge + 1, ImportTimings::default()); + server.hold_block_for_columns(at_the_edge, edge, ImportTimings::default()); + for _ in 0..40 { + if requests.load(Ordering::SeqCst) > 0 { + break; + } + tokio::time::sleep(Duration::from_millis(25)).await; + } + tokio::time::sleep(Duration::from_millis(300)).await; + + assert_eq!( + requests.load(Ordering::SeqCst), + 1, + "only the block at the edge of the window was asked about" + ); + } + + /// An execution client whose proofs do not match its blobs (here, each + /// blob answered with another blob's proofs): the answer has the right + /// shape and the precheck passes, but the KZG batch refuses every built + /// column, so nothing reaches p2p's seen cache, the wire or `DataColumns`. + #[tokio::test] + async fn columns_with_wrong_proofs_are_not_handed_over() { + let blobs = vec![blob_bytes(11), blob_bytes(12)]; + let block = block_committing_to(&blobs); + // Not another `blob_bytes`: two of those differ by the same constant + // in every field element, and a cell proof does not change when a + // constant is added to the polynomial, so their proofs coincide. + let scrambled = |seed: u8| -> Vec { + let mut bytes = blob_bytes(seed); + for element in bytes.chunks_mut(32) { + element[2] = element[2].wrapping_mul(7); + } + bytes + }; + let wrong = answer_with_proofs_of(&blobs, &[scrambled(11), scrambled(12)]); + assert_ne!(wrong, answer_for(&blobs), "the proofs differ"); + let (engine, requests) = mock_engine(wrong).await; + let (mut server, p2p) = beacon_server_recording(store_with_keyed_parent()); + server.engine = Some(engine); + server.custody_columns = CUSTODY.to_vec(); + + server.hold_block_for_columns(block, 1, ImportTimings::default()); + for _ in 0..40 { + if requests.load(Ordering::SeqCst) > 0 { + break; + } + tokio::time::sleep(Duration::from_millis(25)).await; + } + tokio::time::sleep(Duration::from_millis(1500)).await; + + assert_eq!(requests.load(Ordering::SeqCst), 1, "the engine was asked"); + assert!(p2p.checks.lock().unwrap().is_empty()); + } + + /// A block past its parent state's proposer lookahead (a phase0 parent has + /// none), naming a validator that is not the slot's proposer and signed by + /// that validator's own key. The precheck can only check the signature + /// there, so it passes and the execution client is asked; the chain checks + /// advance the parent's state, find the real proposer, and refuse every + /// column. The slot's real proposer's block, from the same blobs, is + /// handed over. + #[tokio::test] + async fn a_block_from_the_wrong_proposer_past_the_lookahead_is_not_handed_over() { + let config = Config::mainnet(); + let mut advanced = keyed_parent_state(2); + ethlambda_state_transition::beacon::stf::process_slots(&mut advanced, 1, &config) + .expect("advance to slot 1"); + let proposer = + ethlambda_state_transition::beacon::helpers::accessors::get_beacon_proposer_index( + &advanced, + ) + .expect("a proposer for slot 1"); + let impostor = 1 - proposer; + + for (named, handed_over) in [(impostor, false), (proposer, true)] { + let blobs = vec![blob_bytes(15)]; + let block = block_by(named, &blobs); + let (engine, requests) = mock_engine(answer_for(&blobs)).await; + let (mut server, p2p) = beacon_server_recording(store_with_keyed_parent_of(2)); + server.engine = Some(engine); + server.custody_columns = CUSTODY.to_vec(); + + server.hold_block_for_columns(block, 1, ImportTimings::default()); + if handed_over { + wait_for_hand_over(&p2p).await; + } else { + for _ in 0..40 { + if requests.load(Ordering::SeqCst) > 0 { + break; + } + tokio::time::sleep(Duration::from_millis(25)).await; + } + tokio::time::sleep(Duration::from_millis(1500)).await; + } + + assert_eq!( + requests.load(Ordering::SeqCst), + 1, + "the precheck let validator {named}'s block through to the engine" + ); + assert_eq!( + !p2p.checks.lock().unwrap().is_empty(), + handed_over, + "validator {named}'s columns handed over" + ); + } + } + + /// An altered copy of a block (its body changed after signing, the + /// signature kept) whose commitments name blobs the execution client + /// holds: the columns would carry a header nobody signed, so the client is + /// not asked and nothing reaches p2p's seen cache or the wire. + #[tokio::test] + async fn a_block_whose_signature_does_not_verify_is_not_asked_about() { + let blobs = vec![blob_bytes(7)]; + let mut block = block_committing_to(&blobs); + let SignedBeaconBlock::Fulu(inner) = &mut block else { + unreachable!("fulu_block builds a fulu block"); + }; + inner.message.body.graffiti = H256::repeat_byte(0x01); + let (engine, requests) = mock_engine(answer_for(&blobs)).await; + let (mut server, p2p) = beacon_server_recording(store_with_keyed_parent()); + server.engine = Some(engine); + server.custody_columns = CUSTODY.to_vec(); + + server.hold_block_for_columns(block, 1, ImportTimings::default()); + tokio::time::sleep(Duration::from_millis(300)).await; + + assert_eq!( + requests.load(Ordering::SeqCst), + 0, + "the engine was not asked" + ); + assert!(p2p.checks.lock().unwrap().is_empty()); + } +} diff --git a/crates/blockchain/src/beacon_columns.rs b/crates/blockchain/src/beacon_columns.rs index 88d035a2a..2ae4fd3c5 100644 --- a/crates/blockchain/src/beacon_columns.rs +++ b/crates/blockchain/src/beacon_columns.rs @@ -18,6 +18,7 @@ use std::collections::{HashMap, HashSet}; +use ethlambda_network_api::SidecarOrigin; use ethlambda_state_transition::beacon::gossip::column::{ is_parkable_gloas, judgeable_block_gloas, }; @@ -141,7 +142,7 @@ impl BlockChainServer { let Some(ref p2p) = self.p2p else { return; }; - let _ = p2p.check_data_column_sidecars(sidecars).inspect_err( + let _ = p2p.check_data_column_sidecars(sidecars, SidecarOrigin::Parked).inspect_err( |err| error!(%err, "Failed to send data column sidecars to the p2p layer for checks"), ); } diff --git a/crates/blockchain/src/lib.rs b/crates/blockchain/src/lib.rs index 77edf2a0e..39617da64 100644 --- a/crates/blockchain/src/lib.rs +++ b/crates/blockchain/src/lib.rs @@ -60,6 +60,7 @@ pub use events::{ChainEvent, EventBus, Topic, UnknownTopic}; pub mod aggregation; mod beacon_aggregates; +mod beacon_blobs; mod beacon_columns; pub mod beacon_engine; mod beacon_envelope; @@ -3277,6 +3278,16 @@ impl BlockChainServer { /// arrives in; leaving it to the redrive cadence instead made a follower /// recovering from a checkpoint sync import roughly one such block per /// slot. + /// + /// The execution client is asked for the block's blobs once, on the first + /// hold of a fulu block that commits to any, is at most + /// [`beacon_blobs::MAX_ASK_AGE_SLOTS`] behind `current_slot`, and whose + /// proposer signature verifies against its parent's state (see + /// [`beacon_blobs`]). Unlike the peer ask, that one is made at the tip: + /// it asks no peer, so it burns no lookup attempt, and a hit hands the + /// custody columns back through the same path gossip uses, so it just + /// races gossip to complete the block. A block further behind (range + /// sync's) is not asked about; its peer ask starts at once below. fn hold_block_for_columns( &mut self, block: SignedBeaconBlock, @@ -3304,6 +3315,30 @@ impl BlockChainServer { "Holding block: its custody columns have not all arrived yet" ); + // Before the block is moved into the store. Only the first hold asks: + // a redelivery of a held root must not re-query the execution client. + // Only a recent block does, checked before the signature so range + // sync pays for neither. Only a block whose proposer signature + // verifies does: the built columns go through every chain check + // before they leave `beacon_blobs`, but a block that already fails + // here is not worth an engine call and a cell computation. + if let (Some(engine), Some(p2p), SignedBeaconBlock::Fulu(inner)) = + (&self.engine, &self.p2p, &block) + && !inner.message.body.blob_kzg_commitments.is_empty() + && !self.blocks_awaiting_columns.contains_key(&block_root) + && beacon_blobs::recent_enough_to_ask(slot, current_slot) + && beacon_blobs::signed_by_its_proposer(&self.store, &block, block_root) + { + beacon_blobs::spawn_fetch( + engine.clone(), + p2p.clone(), + self.store.clone(), + inner.clone(), + block_root, + self.custody_columns.clone(), + ); + } + // Written the same way a parent-missing block is (see // `process_or_pend_block`): no `LiveChain` entry, so it stays // invisible to fork choice until it is re-admitted, but readable back @@ -4800,6 +4835,8 @@ mod tests { #[derive(Default)] pub(crate) struct RecordingP2P { pub(crate) checks: std::sync::Mutex>>, + /// The origin each entry of `checks` was handed over with, in order. + pub(crate) check_origins: std::sync::Mutex>, pub(crate) fetches: std::sync::Mutex>, } @@ -4832,7 +4869,10 @@ mod tests { fn check_data_column_sidecars( &self, sidecars: Vec, + origin: ethlambda_network_api::SidecarOrigin, ) -> Result<(), spawned_concurrency::error::ActorError> { + // Origin first: a test waits on `checks`, then reads the origin. + self.check_origins.lock().unwrap().push(origin); self.checks.lock().unwrap().push(sidecars); Ok(()) } diff --git a/crates/blockchain/src/metrics.rs b/crates/blockchain/src/metrics.rs index 68de7d435..cf3cea573 100644 --- a/crates/blockchain/src/metrics.rs +++ b/crates/blockchain/src/metrics.rs @@ -1715,3 +1715,74 @@ pub fn observe_beacon_aggregate_end_to_end(duration: Duration) { }); LEAN_BEACON_AGGREGATE_END_TO_END_SECONDS.observe(duration.as_secs_f64()); } + +/// Count one `engine_getBlobsV2` attempt for a held fulu block, by what came +/// back: `hit` (every blob), `miss` (the execution client answered `null`, so +/// at least one blob is not in its pool) or `error` (transport, timeout or a +/// malformed answer). +pub fn inc_engine_get_blobs(outcome: &str) { + static LEAN_ENGINE_GET_BLOBS_TOTAL: std::sync::LazyLock = + std::sync::LazyLock::new(|| { + register_int_counter_vec!( + "lean_engine_get_blobs_total", + "engine_getBlobsV2 attempts for held blocks by outcome (hit, miss, error)", + &["outcome"] + ) + .unwrap() + }); + LEAN_ENGINE_GET_BLOBS_TOTAL + .with_label_values(&[outcome]) + .inc(); +} + +/// How long one `engine_getBlobsV2` round trip took, whatever its outcome. +/// +/// The buckets stop at the call's own timeout, so a mass in the last bucket +/// means the execution client is not answering inside the window the +/// specification gives it. +pub fn observe_engine_get_blobs_duration(duration: Duration) { + static LEAN_ENGINE_GET_BLOBS_DURATION_SECONDS: std::sync::LazyLock = + std::sync::LazyLock::new(|| { + register_histogram!( + "lean_engine_get_blobs_duration_seconds", + "Time for one engine_getBlobsV2 round trip", + vec![0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0] + ) + .unwrap() + }); + LEAN_ENGINE_GET_BLOBS_DURATION_SECONDS.observe(duration.as_secs_f64()); +} + +/// How long turning an execution client's blobs into custody column sidecars +/// took: the cell computation plus the sidecar build, off the actor thread. +pub fn observe_engine_blobs_to_columns(duration: Duration) { + static LEAN_ENGINE_BLOBS_TO_COLUMNS_SECONDS: std::sync::LazyLock = + std::sync::LazyLock::new(|| { + register_histogram!( + "lean_engine_blobs_to_columns_seconds", + "Time to build custody data column sidecars from execution-client blobs", + vec![0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.0] + ) + .unwrap() + }); + LEAN_ENGINE_BLOBS_TO_COLUMNS_SECONDS.observe(duration.as_secs_f64()); +} + +/// One column built from the execution client's blobs that the chain checks +/// refused, so it was neither published nor stored. `reason` is the refusing +/// rule's label (`kzg`, `wrong_proposer`, `inclusion_proof`, ...); a column +/// stored meanwhile by another route is not counted. +pub fn inc_engine_blob_columns_refused(reason: &str) { + static LEAN_ENGINE_BLOB_COLUMNS_REFUSED_TOTAL: std::sync::LazyLock = + std::sync::LazyLock::new(|| { + register_int_counter_vec!( + "lean_engine_blob_columns_refused_total", + "Data columns built from execution-client blobs that the chain checks refused, by reason", + &["reason"] + ) + .unwrap() + }); + LEAN_ENGINE_BLOB_COLUMNS_REFUSED_TOTAL + .with_label_values(&[reason]) + .inc(); +} diff --git a/crates/blockchain/state_transition/src/beacon/data_columns.rs b/crates/blockchain/state_transition/src/beacon/data_columns.rs index 26cc347e8..45cac547e 100644 --- a/crates/blockchain/state_transition/src/beacon/data_columns.rs +++ b/crates/blockchain/state_transition/src/beacon/data_columns.rs @@ -148,31 +148,50 @@ pub fn get_data_column_sidecars( ); (0..NUMBER_OF_COLUMNS) .map(|column_index| { - let column: Vec = cells_and_proofs - .iter() - .map(|(cells, _)| { - SszVector::try_from(cells[column_index].to_bytes().to_vec()) - .expect("a c-kzg cell is BYTES_PER_CELL bytes") - }) - .collect(); - let kzg_proofs: Vec = cells_and_proofs - .iter() - .map(|(_, proofs)| proofs[column_index]) - .collect(); - DataColumnSidecar { - index: column_index as u64, - column: SszList::try_from(column) - .expect("a block holds at most MAX_BLOB_COMMITMENTS_PER_BLOCK blobs"), - kzg_commitments: kzg_commitments.clone(), - kzg_proofs: SszList::try_from(kzg_proofs) - .expect("a block holds at most MAX_BLOB_COMMITMENTS_PER_BLOCK blobs"), - signed_block_header: signed_block_header.clone(), - kzg_commitments_inclusion_proof: inclusion_proof.clone(), - } + column_sidecar( + column_index, + signed_block_header, + kzg_commitments, + inclusion_proof, + cells_and_proofs, + ) }) .collect() } +/// The sidecar for one column: that column's cell and proof from every blob, +/// under the block's header, commitments and inclusion proof. The one place a +/// sidecar is assembled, so the full set and a requested subset cannot differ. +fn column_sidecar( + column_index: usize, + signed_block_header: &SignedBeaconBlockHeader, + kzg_commitments: &KzgCommitments, + inclusion_proof: &KzgCommitmentsInclusionProof, + cells_and_proofs: &[(Box, &ProofsPerBlob)], +) -> DataColumnSidecar { + let column: Vec = cells_and_proofs + .iter() + .map(|(cells, _)| { + SszVector::try_from(cells[column_index].to_bytes().to_vec()) + .expect("a c-kzg cell is BYTES_PER_CELL bytes") + }) + .collect(); + let kzg_proofs: Vec = cells_and_proofs + .iter() + .map(|(_, proofs)| proofs[column_index]) + .collect(); + DataColumnSidecar { + index: column_index as u64, + column: SszList::try_from(column) + .expect("a block holds at most MAX_BLOB_COMMITMENTS_PER_BLOCK blobs"), + kzg_commitments: kzg_commitments.clone(), + kzg_proofs: SszList::try_from(kzg_proofs) + .expect("a block holds at most MAX_BLOB_COMMITMENTS_PER_BLOCK blobs"), + signed_block_header: signed_block_header.clone(), + kzg_commitments_inclusion_proof: inclusion_proof.clone(), + } +} + /// The specification's `get_data_column_sidecars_from_block`. /// /// # Panics @@ -205,25 +224,12 @@ pub fn verified_sidecars( blobs: &[Blob], cell_proofs: &[KzgProof], ) -> Result, SidecarError> { - let commitments = &block.message.body.blob_kzg_commitments; - if blobs.len() != commitments.len() { - return Err(SidecarError::Shape( - "the number of blobs differs from the block's commitments", - )); - } - if cell_proofs.len() != blobs.len() * CELLS_PER_EXT_BLOB { - return Err(SidecarError::Shape( - "the number of cell proofs is not blobs * CELLS_PER_EXT_BLOB", - )); - } + check_shape(block, blobs, cell_proofs)?; if blobs.is_empty() { return Ok(Vec::new()); } - - let all_cells: Vec> = blobs - .par_iter() - .map(|blob| kzg::compute_cells(&blob[..]).map_err(|_| SidecarError::InvalidBlob)) - .collect::>()?; + let commitments = &block.message.body.blob_kzg_commitments; + let all_cells = compute_all_cells(blobs)?; let mut batch_commitments = Vec::with_capacity(cell_proofs.len()); let mut batch_indices = Vec::with_capacity(cell_proofs.len()); @@ -246,16 +252,106 @@ pub fn verified_sidecars( return Err(SidecarError::InvalidProofs); } - // The shape check above made the length an exact multiple, so no remainder. - let (proof_chunks, _) = cell_proofs.as_chunks::(); - let cells_and_proofs: Vec<(Box, &ProofsPerBlob)> = - all_cells.into_iter().zip(proof_chunks).collect(); + let cells_and_proofs = pair_cells_with_proofs(all_cells, cell_proofs); Ok(get_data_column_sidecars_from_block( block, &cells_and_proofs, )) } +/// The sidecars for `columns` of `block`, built from blobs and cell proofs +/// this node's own execution client returned (`engine_getBlobsV2`), without +/// verifying the proofs. +/// +/// Not [`verified_sidecars`], which verifies every cell of every blob and +/// builds all `NUMBER_OF_COLUMNS` columns. The caller, the chain actor's +/// `beacon_blobs` module, needs only its custody columns, and runs the chain +/// checks on what this returns (KZG proofs included) before anything is +/// published or stored. Blobs from any other source must go through +/// [`verified_sidecars`]. +/// +/// The shape checks are the same as [`verified_sidecars`]. Only the requested +/// columns are built, one per distinct index in ascending order, whatever the +/// order or repetition in `columns`; an index at or past `NUMBER_OF_COLUMNS` +/// is a [`SidecarError::Shape`]. A block with zero blobs, or an empty +/// `columns`, yields no sidecars and computes no cells. +pub fn unverified_sidecars( + block: &electra::SignedBeaconBlock, + blobs: &[Blob], + cell_proofs: &[KzgProof], + columns: &[u64], +) -> Result, SidecarError> { + check_shape(block, blobs, cell_proofs)?; + let mut columns = columns.to_vec(); + columns.sort_unstable(); + columns.dedup(); + if columns + .last() + .is_some_and(|&last| last >= NUMBER_OF_COLUMNS as u64) + { + return Err(SidecarError::Shape( + "a requested column index is not below NUMBER_OF_COLUMNS", + )); + } + if blobs.is_empty() || columns.is_empty() { + return Ok(Vec::new()); + } + + let cells_and_proofs = pair_cells_with_proofs(compute_all_cells(blobs)?, cell_proofs); + let commitments = &block.message.body.blob_kzg_commitments; + let (inclusion_proof, body_root) = inclusion_proof_and_body_root(&block.message.body); + let header = header_with_body_root(block, body_root); + Ok(columns + .into_iter() + .map(|column_index| { + column_sidecar( + column_index as usize, + &header, + commitments, + &inclusion_proof, + &cells_and_proofs, + ) + }) + .collect()) +} + +/// The count checks both sidecar builders start with, before any KZG work. +fn check_shape( + block: &electra::SignedBeaconBlock, + blobs: &[Blob], + cell_proofs: &[KzgProof], +) -> Result<(), SidecarError> { + if blobs.len() != block.message.body.blob_kzg_commitments.len() { + return Err(SidecarError::Shape( + "the number of blobs differs from the block's commitments", + )); + } + if cell_proofs.len() != blobs.len() * CELLS_PER_EXT_BLOB { + return Err(SidecarError::Shape( + "the number of cell proofs is not blobs * CELLS_PER_EXT_BLOB", + )); + } + Ok(()) +} + +/// Every blob's cells, computed in parallel (one blob per task). +fn compute_all_cells(blobs: &[Blob]) -> Result>, SidecarError> { + blobs + .par_iter() + .map(|blob| kzg::compute_cells(&blob[..]).map_err(|_| SidecarError::InvalidBlob)) + .collect() +} + +/// Blob `i`'s cells next to its `CELLS_PER_EXT_BLOB` proofs. `check_shape` +/// made the proof count an exact multiple, so there is no remainder. +fn pair_cells_with_proofs( + all_cells: Vec>, + cell_proofs: &[KzgProof], +) -> Vec<(Box, &ProofsPerBlob)> { + let (proof_chunks, _) = cell_proofs.as_chunks::(); + all_cells.into_iter().zip(proof_chunks).collect() +} + #[cfg(test)] mod tests { use super::*; @@ -382,4 +478,65 @@ mod tests { Err(SidecarError::InvalidBlob) )); } + + #[test] + fn requested_columns_match_the_verified_set_in_ascending_order() { + let blobs = [blob(1), blob(2)]; + let (block, proofs) = block_with(&blobs); + let all = verified_sidecars(&block, &blobs, &proofs).unwrap(); + let last = NUMBER_OF_COLUMNS as u64 - 1; + let got = unverified_sidecars(&block, &blobs, &proofs, &[last, 5, 0, 5]).unwrap(); + let expected: Vec<_> = [0, 5, last as usize] + .into_iter() + .map(|i| all[i].clone()) + .collect(); + assert_eq!(got, expected); + } + + #[test] + fn no_requested_columns_or_no_blobs_yield_nothing() { + let blobs = [blob(1)]; + let (block, proofs) = block_with(&blobs); + assert!( + unverified_sidecars(&block, &blobs, &proofs, &[]) + .unwrap() + .is_empty() + ); + let (empty_block, empty_proofs) = block_with(&[]); + assert!( + unverified_sidecars(&empty_block, &[], &empty_proofs, &[0, 1]) + .unwrap() + .is_empty() + ); + } + + #[test] + fn unverified_sidecars_refuse_bad_shapes() { + let blobs = [blob(1)]; + let (block, proofs) = block_with(&blobs); + assert!(matches!( + unverified_sidecars(&block, &[], &proofs, &[0]), + Err(SidecarError::Shape(_)) + )); + assert!(matches!( + unverified_sidecars(&block, &blobs, &proofs[..proofs.len() - 1], &[0]), + Err(SidecarError::Shape(_)) + )); + assert!(matches!( + unverified_sidecars(&block, &blobs, &proofs, &[0, NUMBER_OF_COLUMNS as u64]), + Err(SidecarError::Shape(_)) + )); + } + + #[test] + fn a_tampered_proof_still_builds_because_nothing_is_verified() { + let blobs = [blob(1), blob(2)]; + let (block, mut proofs) = block_with(&blobs); + proofs.swap(0, 1); + let swapped = proofs[0]; + let got = unverified_sidecars(&block, &blobs, &proofs, &[0]).unwrap(); + assert_eq!(got.len(), 1); + assert_eq!(got[0].kzg_proofs[0], swapped); + assert!(!verify_data_column_sidecar_kzg_proofs(&got[0]).unwrap()); + } } diff --git a/crates/net/api/src/lib.rs b/crates/net/api/src/lib.rs index 75c1c51b9..d331965ed 100644 --- a/crates/net/api/src/lib.rs +++ b/crates/net/api/src/lib.rs @@ -34,20 +34,45 @@ pub trait BlockChainToP2P: Send + Sync { ) -> Result<(), ActorError>; /// Ask peers for whatever of one block this node is missing. fn fetch_block(&self, request: FetchRequest) -> Result<(), ActorError>; - /// Run the chain checks on sidecars the chain actor had parked, now that - /// the block they are checked against has a post-state: their parent for a - /// fulu sidecar, their own block for a gloas one. + /// Hand sidecars the chain actor holds to the p2p layer, which decides + /// what they need before they are stored by where they came from; see + /// [`SidecarOrigin`]. /// - /// The chain actor keeps a sidecar without checking it, so it hands these - /// back rather than judging them itself: the p2p layer runs every column - /// check, off both actors, and sends the ones that pass back through - /// [`P2PToBlockChain::new_data_column_sidecars`]. + /// For [`SidecarOrigin::Parked`] that is the chain checks: sidecars the + /// chain actor had parked, now that the block they are checked against has + /// a post-state (their parent for a fulu sidecar, their own block for a + /// gloas one). The chain actor keeps a sidecar without checking it, so it + /// hands these back rather than judging them itself: the p2p layer runs + /// every column check, off both actors, and sends the ones that pass back + /// through [`P2PToBlockChain::new_data_column_sidecars`]. fn check_data_column_sidecars( &self, sidecars: Vec, + origin: SidecarOrigin, ) -> Result<(), ActorError>; } +/// Where the sidecars handed to [`BlockChainToP2P::check_data_column_sidecars`] +/// came from, which decides what the p2p layer does with them before they +/// reach [`P2PToBlockChain::new_data_column_sidecars`]. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum SidecarOrigin { + /// From the network, parked by the chain actor unchecked until the block + /// they are judged against had a post-state: run the chain checks. + Parked, + /// Built by this node from the blobs its own execution client returned + /// for a held block (`engine_getBlobsV2`). + /// + /// Already checked: the chain actor runs every chain check on them, KZG + /// proofs included, before handing them over, though the fulu + /// `p2p-interface.md` lets a client skip verifying blobs from its local + /// execution layer. What is left is what it "MUST" do "as if the + /// `data_column_sidecar` had been received via gossip": publish each on + /// its subnet if and only if this node is subscribed to it, and record it + /// in the anti-equivocation (seen) cache. + ExecutionLayer, +} + /// What one block is missing, from the chain actor's point of view. /// /// One message rather than a block fetch and a column fetch, because the diff --git a/crates/net/engine/src/building.rs b/crates/net/engine/src/building.rs index e7ac9559f..d47d6cd97 100644 --- a/crates/net/engine/src/building.rs +++ b/crates/net/engine/src/building.rs @@ -18,7 +18,7 @@ use ethlambda_types::beacon::primitives::{ use serde::{Deserialize, Serialize, Serializer}; use crate::error::EngineError; -use crate::types::{data, quantity}; +use crate::types::{data, parse_data, parse_fixed, quantity}; /// `PayloadAttributesV3`: what the execution client needs to build the payload /// for one slot. @@ -367,21 +367,6 @@ fn decode(message: &str) -> EngineError { EngineError::Decode(message.to_string()) } -/// A `DATA` of any length. -fn parse_data(text: &str) -> Result, EngineError> { - let digits = text - .strip_prefix("0x") - .ok_or_else(|| decode("DATA must be 0x-prefixed"))?; - hex::decode(digits).map_err(|err| EngineError::Decode(format!("DATA is not hex: {err}"))) -} - -/// A `DATA` of exactly `N` bytes. -fn parse_fixed(text: &str) -> Result<[u8; N], EngineError> { - parse_data(text)?.try_into().map_err(|bytes: Vec| { - EngineError::Decode(format!("expected {N} bytes, got {}", bytes.len())) - }) -} - /// A `QUANTITY` that fits in 64 bits. fn parse_quantity(text: &str) -> Result { let digits = text diff --git a/crates/net/engine/src/client.rs b/crates/net/engine/src/client.rs index e34fef2de..a0a2d4ec0 100644 --- a/crates/net/engine/src/client.rs +++ b/crates/net/engine/src/client.rs @@ -1,4 +1,4 @@ -//! The JSON-RPC client, its retry ladder, and the four methods. +//! The JSON-RPC client, its retry ladder, and the methods a follower calls. use std::time::Duration; @@ -10,8 +10,9 @@ use tracing::{debug, warn}; use crate::auth::JwtSecret; use crate::error::EngineError; use crate::types::{ - ClientVersionV1, CustodyColumns, ExecutionPayloadV3, ExecutionPayloadV4, ForkchoiceStateV1, - ForkchoiceUpdatedResponse, PayloadStatusV1, data, + BlobAndProofV2, BlobAndProofV2Json, ClientVersionV1, CustodyColumns, ExecutionPayloadV3, + ExecutionPayloadV4, ForkchoiceStateV1, ForkchoiceUpdatedResponse, PayloadStatusV1, data, + decode_blobs_v2, }; /// Per-attempt timeout. @@ -26,6 +27,10 @@ pub const ENGINE_TIMEOUT: Duration = Duration::from_secs(8); /// for the older methods. pub const ENGINE_NEW_PAYLOAD_V5_TIMEOUT: Duration = Duration::from_secs(6); +/// Per-attempt timeout for `engine_getBlobsV2`, the value `osaka.md` gives that +/// method. +pub const ENGINE_GET_BLOBS_TIMEOUT: Duration = Duration::from_secs(1); + /// Timeout for `engine_getPayloadV5` and `engine_getPayloadV6`, which get this /// one attempt and no retry. /// @@ -381,6 +386,37 @@ impl EngineClient { response.try_into() } + /// `engine_getBlobsV2`: every blob named by `versioned_hashes`, in order, + /// with its cell proofs, or `None` when the execution client is missing + /// any of them. + /// + /// One attempt, bounded by [`ENGINE_GET_BLOBS_TIMEOUT`], and deliberately + /// not through the retry ladder: this call races the gossip delivery of the + /// same data columns, and a retry after the ladder's first backoff would + /// land after that race is lost. A failure leaves the block to gossip and + /// peers, which is where it was anyway. + /// + /// An answer that is not shaped as the specification requires (wrong + /// length, blob size or proof count, or a `null` element) is a + /// [`EngineError::Decode`]. + pub async fn get_blobs_v2( + &self, + versioned_hashes: &[Bytes32], + ) -> Result>, EngineError> { + let hashes: Vec = versioned_hashes + .iter() + .map(|hash| data(hash.as_slice())) + .collect(); + let result: Option>> = self + .call_once( + "engine_getBlobsV2", + &json!([hashes]), + ENGINE_GET_BLOBS_TIMEOUT, + ) + .await?; + decode_blobs_v2(result, versioned_hashes.len()) + } + /// `engine_exchangeCapabilities`. Returns what the execution client says it /// supports. pub async fn exchange_capabilities(&self, ours: &[&str]) -> Result, EngineError> { diff --git a/crates/net/engine/src/lib.rs b/crates/net/engine/src/lib.rs index 742cd0bbf..2ac3c74a9 100644 --- a/crates/net/engine/src/lib.rs +++ b/crates/net/engine/src/lib.rs @@ -1,6 +1,6 @@ //! The Ethereum Engine API, as a consensus-layer follower needs it. //! -//! Wire concerns only: JWT authentication, the JSON-RPC envelope, and the four +//! Wire concerns only: JWT authentication, the JSON-RPC envelope, and the //! methods a follower calls. Nothing here knows what a beacon block is beyond //! the containers it serializes, and nothing here decides what an answer means. //! Assembling a request from a block, and reading a status as a fork choice @@ -17,9 +17,11 @@ //! //! Payload building (`PayloadAttributesV3` on `forkchoiceUpdated`, and //! Osaka's `engine_getPayloadV5`) is in [`building`], for the Beacon API's -//! block production. Two method families a full client would have are -//! deliberately absent: `engine_getBlobs*`, because there is no blob-pool -//! fetch path and data columns come from peers; and +//! block production. `engine_getBlobsV2` is called for held fulu blocks, to +//! rebuild data columns from the execution client's blob pool. `V3` is not: +//! it answers partially, a column needs one cell from every blob, and without +//! partial-column assembly a partial blob set yields no complete column. One +//! method family a full client would have is deliberately absent: //! `engine_getPayloadBodies*`, because nothing consumes them. pub mod auth; @@ -31,7 +33,9 @@ pub mod types; pub use auth::JwtSecret; pub use client::EngineClient; pub use error::EngineError; -pub use types::{CustodyColumns, ForkchoiceStateV1, PayloadStatusV1, PayloadStatusValue}; +pub use types::{ + BlobAndProofV2, CustodyColumns, ForkchoiceStateV1, PayloadStatusV1, PayloadStatusValue, +}; /// The methods this client will call, sent in the `engine_exchangeCapabilities` /// handshake. @@ -45,5 +49,6 @@ pub const ETHLAMBDA_ENGINE_CAPABILITIES: &[&str] = &[ "engine_forkchoiceUpdatedV4", "engine_getPayloadV5", "engine_getPayloadV6", + "engine_getBlobsV2", "engine_getClientVersionV1", ]; diff --git a/crates/net/engine/src/types.rs b/crates/net/engine/src/types.rs index 6e142812e..1d3509583 100644 --- a/crates/net/engine/src/types.rs +++ b/crates/net/engine/src/types.rs @@ -12,10 +12,12 @@ use ethlambda_types::beacon::containers::capella::Withdrawal; use ethlambda_types::beacon::containers::{deneb, gloas}; use ethlambda_types::beacon::preset; -use ethlambda_types::beacon::primitives::{ExecutionBlockHash, Uint256}; +use ethlambda_types::beacon::primitives::{ExecutionBlockHash, KzgProof, Uint256}; use ethlambda_types::beacon::serde_helpers::HexPrefixed; use serde::{Deserialize, Serialize, Serializer}; +use crate::error::EngineError; + /// `PayloadStatusV1.status`. #[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] #[serde(rename_all = "SCREAMING_SNAKE_CASE")] @@ -195,6 +197,21 @@ pub fn data(bytes: &[u8]) -> String { HexPrefixed(bytes).to_string() } +/// Decodes a `DATA` of any length. +pub(crate) fn parse_data(text: &str) -> Result, EngineError> { + let digits = text + .strip_prefix("0x") + .ok_or_else(|| EngineError::Decode("DATA must be 0x-prefixed".to_string()))?; + hex::decode(digits).map_err(|err| EngineError::Decode(format!("DATA is not hex: {err}"))) +} + +/// Decodes a `DATA` of exactly `N` bytes. +pub(crate) fn parse_fixed(text: &str) -> Result<[u8; N], EngineError> { + parse_data(text)?.try_into().map_err(|bytes: Vec| { + EngineError::Decode(format!("expected {N} bytes, got {}", bytes.len())) + }) +} + /// The `custodyColumns` parameter of `engine_forkchoiceUpdatedV4`: a bitarray /// of `CELLS_PER_EXT_BLOB` bits marking the columns the consensus client /// custodies. @@ -255,6 +272,76 @@ pub fn uint256(value: &Uint256) -> String { format!("0x{trimmed}") } +/// `BlobAndProofV2` (Osaka): one blob from the execution client's blob pool, +/// with its `CELLS_PER_EXT_BLOB` cell proofs. +/// +/// Produced only by `decode_blobs_v2`, which checks the wire shape, so a value +/// of this type always has a `BYTES_PER_BLOB` blob and exactly +/// `CELLS_PER_EXT_BLOB` proofs. The proofs are not verified: the specification +/// lets a consensus client skip that for blobs from its own execution client. +#[derive(Debug, Clone)] +pub struct BlobAndProofV2 { + /// The blob, `BYTES_PER_BLOB` bytes. + pub blob: Vec, + /// The blob's cell proofs, exactly `CELLS_PER_EXT_BLOB` of them. + pub proofs: Vec, +} + +#[derive(Deserialize)] +pub(crate) struct BlobAndProofV2Json { + blob: String, + proofs: Vec, +} + +/// Decodes an `engine_getBlobsV2` result for a request of `requested` hashes. +/// +/// `null` is `Ok(None)`: the execution client is missing at least one blob, or +/// cannot serve its pool. A `null` element is a decode error, since V2 never +/// sends one (V3 does, for a partial answer). +pub(crate) fn decode_blobs_v2( + result: Option>>, + requested: usize, +) -> Result>, EngineError> { + let Some(entries) = result else { + return Ok(None); + }; + if entries.len() != requested { + return Err(EngineError::Decode(format!( + "asked for {requested} blobs, got {}", + entries.len() + ))); + } + + let mut blobs = Vec::with_capacity(entries.len()); + for (index, entry) in entries.into_iter().enumerate() { + let json = entry + .ok_or_else(|| EngineError::Decode(format!("blob {index} is null in a V2 answer")))?; + + let blob = parse_data(&json.blob)?; + if blob.len() != preset::BYTES_PER_BLOB { + return Err(EngineError::Decode(format!( + "blob {index} is {} bytes, expected BYTES_PER_BLOB ({})", + blob.len(), + preset::BYTES_PER_BLOB + ))); + } + if json.proofs.len() != preset::CELLS_PER_EXT_BLOB { + return Err(EngineError::Decode(format!( + "blob {index} has {} proofs, expected CELLS_PER_EXT_BLOB ({})", + json.proofs.len(), + preset::CELLS_PER_EXT_BLOB + ))); + } + let proofs = json + .proofs + .iter() + .map(|text| parse_fixed(text).map(KzgProof)) + .collect::, _>>()?; + blobs.push(BlobAndProofV2 { blob, proofs }); + } + Ok(Some(blobs)) +} + #[cfg(test)] mod tests { use ethlambda_types::beacon::containers::bellatrix; @@ -497,4 +584,44 @@ mod tests { format!("0xa{}", "0".repeat(62)) ); } + + fn blob_json(blob_len: usize, proof_count: usize) -> BlobAndProofV2Json { + BlobAndProofV2Json { + blob: format!("0x{}", "00".repeat(blob_len)), + proofs: vec![format!("0x{}", "11".repeat(48)); proof_count], + } + } + + #[test] + fn a_blob_of_the_wrong_size_is_a_decode_error() { + let answer = Some(vec![Some(blob_json( + preset::BYTES_PER_BLOB - 1, + preset::CELLS_PER_EXT_BLOB, + ))]); + assert!(matches!( + decode_blobs_v2(answer, 1), + Err(EngineError::Decode(_)) + )); + } + + #[test] + fn a_null_element_is_a_decode_error() { + let answer = Some(vec![None]); + assert!(matches!( + decode_blobs_v2(answer, 1), + Err(EngineError::Decode(_)) + )); + } + + #[test] + fn a_well_formed_blob_decodes() { + let answer = Some(vec![Some(blob_json( + preset::BYTES_PER_BLOB, + preset::CELLS_PER_EXT_BLOB, + ))]); + let blobs = decode_blobs_v2(answer, 1) + .expect("well formed") + .expect("some"); + assert_eq!(blobs[0].proofs.len(), preset::CELLS_PER_EXT_BLOB); + } } diff --git a/crates/net/engine/tests/wire_smoke.rs b/crates/net/engine/tests/wire_smoke.rs index affd897da..3a66b77a4 100644 --- a/crates/net/engine/tests/wire_smoke.rs +++ b/crates/net/engine/tests/wire_smoke.rs @@ -11,18 +11,20 @@ use std::sync::Arc; use ethlambda_engine::types::{CustodyColumns, ForkchoiceStateV1, PayloadStatusValue}; use ethlambda_engine::{EngineClient, EngineError, JwtSecret}; -use ethlambda_types::beacon::primitives::ExecutionBlockHash; +use ethlambda_types::beacon::preset; +use ethlambda_types::beacon::primitives::{Bytes32, ExecutionBlockHash}; use tokio::io::{AsyncReadExt as _, AsyncWriteExt as _}; use tokio::net::TcpListener; use tokio::sync::Mutex; /// Serves exactly one request, answering with `body`, and hands the caller the /// request it saw. -async fn serve_once(body: &'static str) -> (String, Arc>>) { +async fn serve_once(body: impl Into) -> (String, Arc>>) { let listener = TcpListener::bind("127.0.0.1:0") .await .expect("an ephemeral loopback port"); let addr = listener.local_addr().expect("the bound address"); + let body: String = body.into(); let seen = Arc::new(Mutex::new(None)); let seen_writer = Arc::clone(&seen); @@ -138,8 +140,7 @@ async fn an_rpc_error_envelope_surfaces_typed_and_is_not_retried() { #[tokio::test] async fn new_payload_v5_sends_four_params_under_its_own_method() { use ethlambda_types::beacon::containers::{bellatrix, gloas}; - use ethlambda_types::beacon::preset; - use ethlambda_types::beacon::primitives::{Bytes32, ExecutionAddress, Root, Uint256}; + use ethlambda_types::beacon::primitives::{ExecutionAddress, Root, Uint256}; let (endpoint, seen) = serve_once( r#"{"jsonrpc":"2.0","id":1,"result":{"status":"VALID","latestValidHash":"0x0505050505050505050505050505050505050505050505050505050505050505","validationError":null}}"#, @@ -222,6 +223,82 @@ async fn forkchoice_updated_v4_sends_three_params_and_parses_syncing() { assert!(request.contains(",null,\"0x01020000000000000000000000000000\"]")); } +/// A `getBlobsV2` result body with `blobs` blobs of `proof_count` proofs each. +/// Proof `i` is 48 bytes of `i + 1`, so a reordering shows up. +fn blobs_v2_body(blobs: usize, proof_count: usize) -> String { + let blob = format!("0x{}", "00".repeat(preset::BYTES_PER_BLOB)); + let proofs: Vec = (0..proof_count) + .map(|i| format!("0x{}", format!("{:02x}", i + 1).repeat(48))) + .collect(); + let entry = serde_json::json!({ "blob": blob, "proofs": proofs }); + serde_json::json!({ "jsonrpc": "2.0", "id": 1, "result": vec![entry; blobs] }).to_string() +} + +#[tokio::test] +async fn get_blobs_v2_null_is_none_and_the_request_carries_the_hashes() { + let (endpoint, seen) = serve_once(r#"{"jsonrpc":"2.0","id":1,"result":null}"#).await; + + let hashes = [Bytes32::repeat_byte(1), Bytes32::repeat_byte(2)]; + let answer = client(endpoint) + .get_blobs_v2(&hashes) + .await + .expect("null is an answer, not an error"); + assert!(answer.is_none()); + + let request = seen.lock().await.clone().expect("the mock saw a request"); + let (_, body) = request.split_once("\r\n\r\n").expect("a request body"); + let body: serde_json::Value = serde_json::from_str(body).expect("a JSON body"); + assert_eq!(body["method"], "engine_getBlobsV2"); + assert_eq!( + body["params"], + serde_json::json!([[ + format!("0x{}", "01".repeat(32)), + format!("0x{}", "02".repeat(32)), + ]]) + ); +} + +#[tokio::test] +async fn get_blobs_v2_decodes_a_full_answer_in_order() { + let (endpoint, _seen) = serve_once(blobs_v2_body(1, preset::CELLS_PER_EXT_BLOB)).await; + + let blobs = client(endpoint) + .get_blobs_v2(&[Bytes32::repeat_byte(1)]) + .await + .expect("a well-formed answer") + .expect("every blob is available"); + + assert_eq!(blobs.len(), 1); + assert_eq!(blobs[0].blob.len(), preset::BYTES_PER_BLOB); + assert!(blobs[0].blob.iter().all(|byte| *byte == 0)); + assert_eq!(blobs[0].proofs.len(), preset::CELLS_PER_EXT_BLOB); + for (i, proof) in blobs[0].proofs.iter().enumerate() { + assert_eq!(proof.0, [(i + 1) as u8; 48], "proof {i} out of order"); + } +} + +#[tokio::test] +async fn get_blobs_v2_with_the_wrong_proof_count_is_a_decode_error() { + let (endpoint, _seen) = serve_once(blobs_v2_body(1, preset::CELLS_PER_EXT_BLOB - 1)).await; + + let err = client(endpoint) + .get_blobs_v2(&[Bytes32::repeat_byte(1)]) + .await + .expect_err("a short proof list is malformed"); + assert!(matches!(err, EngineError::Decode(_)), "got {err:?}"); +} + +#[tokio::test] +async fn get_blobs_v2_with_a_different_length_than_requested_is_a_decode_error() { + let (endpoint, _seen) = serve_once(blobs_v2_body(1, preset::CELLS_PER_EXT_BLOB)).await; + + let err = client(endpoint) + .get_blobs_v2(&[Bytes32::repeat_byte(1), Bytes32::repeat_byte(2)]) + .await + .expect_err("one blob for two hashes is malformed"); + assert!(matches!(err, EngineError::Decode(_)), "got {err:?}"); +} + /// Accepts every connection, reads its request, and never answers. Returns /// the endpoint and a count of the connections it took. async fn serve_silently() -> (String, Arc) { diff --git a/crates/net/p2p/src/beacon/engine_columns.rs b/crates/net/p2p/src/beacon/engine_columns.rs new file mode 100644 index 000000000..95e308fcd --- /dev/null +++ b/crates/net/p2p/src/beacon/engine_columns.rs @@ -0,0 +1,249 @@ +//! Data column sidecars this node built from its own execution client's blob +//! pool (`engine_getBlobsV2`), handed over by the chain actor with +//! `SidecarOrigin::ExecutionLayer`. +//! +//! The fulu p2p spec says a client that takes blobs from its local execution +//! layer MUST behave as if the sidecar had arrived by gossip. In particular it +//! MUST publish the sidecar on its `data_column_sidecar_{subnet_id}` topic if +//! and only if it is subscribed to that topic, and MUST update the gossip rule +//! data structures (the anti-equivocation cache, here +//! [`P2PServer::seen_columns`]). +//! +//! That is all this module does, and why no checks run here: the chain actor +//! already ran every chain check on these sidecars, KZG proofs included, +//! before handing them over (`ethlambda_blockchain`'s `beacon_blobs`, the same +//! rules `beacon::column_checks` runs on a fetched column). Debug builds re-run +//! them in the chain actor. +//! +//! Recording comes before publishing on purpose. A copy of the same column that +//! gossip delivers afterwards then finds its key taken and is ignored rather +//! than re-forwarded, and a column gossip already accepted is not published a +//! second time. Gossip records a column only once its verdict is `Accept` +//! (`verdict::Validated::record_seen`), so a copy still being validated when +//! these are recorded is ignored too, as any second valid copy is. + +use ethlambda_state_transition::beacon::das; +use ethlambda_types::ShortRoot; +use ethlambda_types::beacon::containers::DataColumnSidecar; +use ethlambda_types::primitives::HashTreeRoot as _; +use libp2p::gossipsub::IdentTopic; +use tracing::{debug, error, info, warn}; + +use crate::beacon::topics as beacon_topics; +use crate::gossipsub::compress_message; +use crate::{P2PServer, metrics}; + +/// Record each sidecar in the seen cache, publish the newly recorded ones on +/// their subnets, and hand them to the chain actor. +pub(crate) fn publish_and_forward(server: &mut P2PServer, sidecars: Vec) { + let Some(beacon) = server.wire.beacon() else { + error!( + columns = sidecars.len(), + "Execution-layer columns reached a lean node; dropping them" + ); + return; + }; + // Subnets are derived from the custody set (see `build_swarm`), so a column + // is subscribed to exactly when it is custodied. + let custody_columns = beacon.custody_columns.clone(); + + let mut fresh: Vec<(DataColumnSidecar, Option)> = Vec::new(); + let mut already_seen = 0u64; + let mut not_custody = 0u64; + let mut missing_digest = false; + let mut batch_header = None; + for sidecar in sidecars { + let DataColumnSidecar::Fulu(fulu) = &sidecar else { + debug!("Skipping a gloas sidecar from the execution layer"); + continue; + }; + let index = fulu.index; + if !custody_columns.contains(&index) { + not_custody += 1; + continue; + } + let header = &fulu.signed_block_header.message; + if !server + .seen_columns + .record(header.slot, header.proposer_index, index) + { + already_seen += 1; + continue; + } + batch_header.get_or_insert_with(|| (header.slot, header.hash_tree_root())); + let topic = server + .wire + .beacon() + .and_then(|beacon| beacon.publish_digest(header.slot)) + .map(|digest| { + let subnet = das::compute_subnet_for_data_column_sidecar(index); + IdentTopic::new(beacon_topics::data_column_topic_name(digest, subnet)) + }); + missing_digest |= topic.is_none(); + fresh.push((sidecar, topic)); + } + + let mut published = 0u64; + let mut forwarded = Vec::with_capacity(fresh.len()); + for (sidecar, topic) in fresh { + if let Some(topic) = topic { + server + .swarm_handle + .publish(topic, compress_message(&sidecar.to_ssz())); + published += 1; + } + forwarded.push(sidecar); + } + let unpublished = forwarded.len() as u64 - published; + + if let Some((slot, block_root)) = batch_header { + if missing_digest { + warn!( + slot, + "No held fork digest covers these columns' slot; not publishing them" + ); + } + info!( + slot, + block_root = %ShortRoot(&block_root.0), + published, + no_digest = unpublished, + already_seen, + not_custody, + "Execution-layer data columns" + ); + } + metrics::inc_data_columns_from_execution_layer("published", published); + metrics::inc_data_columns_from_execution_layer("no_digest", unpublished); + metrics::inc_data_columns_from_execution_layer("already_seen", already_seen); + metrics::inc_data_columns_from_execution_layer("not_custody", not_custody); + + if forwarded.is_empty() { + return; + } + if let Some(blockchain) = &server.blockchain { + let _ = blockchain.new_data_column_sidecars(forwarded).inspect_err( + |err| error!(%err, "Failed to hand the execution-layer columns to the chain"), + ); + } +} + +#[cfg(test)] +mod tests { + use std::sync::Arc; + + use ethlambda_types::beacon::config::Config; + use ethlambda_types::beacon::fork::ForkName; + + use super::*; + use crate::beacon::decode::decode_data_column_sidecar; + use crate::gossipsub::decompress_message; + use crate::swarm_adapter::{SwarmCommand, SwarmHandle}; + use crate::test_support::{RecordingChain, unconnected_beacon_server, valid_shaped_sidecar}; + + const SLOT: u64 = 5; + const PROPOSER: u64 = 7; + + /// A server whose node custodies `custody`, wired to a recording chain. + async fn server_custodying(custody: &[u64]) -> (P2PServer, Arc) { + let mut server = unconnected_beacon_server(Config::mainnet(), 0).await; + let crate::Wire::Beacon(beacon) = &mut server.wire else { + unreachable!("unconnected_beacon_server builds a beacon wire"); + }; + beacon.custody_columns = custody.to_vec(); + // The test server's digest is a placeholder; a held digest is what + // lets the columns be published. + beacon.topics.fork_digest = beacon.digest_for_slot(SLOT); + let chain = Arc::new(RecordingChain::default()); + server.blockchain = Some(chain.clone()); + (server, chain) + } + + fn sidecar(index: u64) -> DataColumnSidecar { + DataColumnSidecar::Fulu(valid_shaped_sidecar(SLOT, index)) + } + + fn forwarded_indices(chain: &RecordingChain) -> Vec> { + chain + .sidecar_batches + .lock() + .unwrap() + .iter() + .map(|batch| batch.iter().map(DataColumnSidecar::index).collect()) + .collect() + } + + #[tokio::test] + async fn newly_seen_custody_columns_are_forwarded_in_one_batch_and_recorded() { + let (mut server, chain) = server_custodying(&[1, 2]).await; + + publish_and_forward(&mut server, vec![sidecar(1), sidecar(2)]); + + assert_eq!(forwarded_indices(&chain), [vec![1, 2]]); + assert!(server.seen_columns.contains(SLOT, PROPOSER, 1)); + assert!(server.seen_columns.contains(SLOT, PROPOSER, 2)); + } + + /// What goes on the wire is what gossip validation decodes: the fulu + /// container, snappy-compressed, on the column's own subnet topic under + /// the digest held for its slot. Only a newly recorded custody column + /// goes out. + #[tokio::test] + async fn a_new_custody_column_is_published_as_gossip_decodes_it() { + let (mut server, _chain) = server_custodying(&[1, 2]).await; + let (swarm_handle, mut commands) = SwarmHandle::capturing(); + server.swarm_handle = swarm_handle; + assert!(server.seen_columns.record(SLOT, PROPOSER, 2)); + let crate::Wire::Beacon(beacon) = &server.wire else { + unreachable!("unconnected_beacon_server builds a beacon wire"); + }; + let digest = beacon.digest_for_slot(SLOT); + + publish_and_forward(&mut server, vec![sidecar(1), sidecar(2), sidecar(9)]); + + let mut published = Vec::new(); + while let Ok(command) = commands.try_recv() { + if let SwarmCommand::Publish { topic, data } = command { + published.push((topic, data)); + } + } + assert_eq!(published.len(), 1, "only column 1 is new and custodied"); + let (topic, data) = &published[0]; + let subnet = das::compute_subnet_for_data_column_sidecar(1); + let expected = IdentTopic::new(beacon_topics::data_column_topic_name(digest, subnet)); + assert_eq!(topic.hash(), expected.hash()); + let bytes = decompress_message(data).expect("snappy-compressed"); + let decoded = decode_data_column_sidecar(ForkName::Fulu, &bytes) + .expect("the fulu container gossip decodes"); + assert_eq!(decoded, sidecar(1)); + } + + #[tokio::test] + async fn a_column_gossip_already_accepted_is_not_forwarded() { + let (mut server, chain) = server_custodying(&[1, 2]).await; + assert!(server.seen_columns.record(SLOT, PROPOSER, 1)); + + publish_and_forward(&mut server, vec![sidecar(1), sidecar(2)]); + + assert_eq!(forwarded_indices(&chain), [vec![2]]); + } + + #[tokio::test] + async fn a_column_outside_the_custody_set_is_dropped_and_not_recorded() { + let (mut server, chain) = server_custodying(&[1]).await; + + publish_and_forward(&mut server, vec![sidecar(1), sidecar(9)]); + + assert_eq!(forwarded_indices(&chain), [vec![1]]); + assert!(!server.seen_columns.contains(SLOT, PROPOSER, 9)); + } + + #[tokio::test] + async fn a_batch_with_nothing_new_sends_the_chain_nothing() { + let (mut server, chain) = server_custodying(&[1]).await; + + publish_and_forward(&mut server, vec![sidecar(9)]); + + assert!(forwarded_indices(&chain).is_empty()); + } +} diff --git a/crates/net/p2p/src/beacon/mod.rs b/crates/net/p2p/src/beacon/mod.rs index 33944b4f7..25b4d00ac 100644 --- a/crates/net/p2p/src/beacon/mod.rs +++ b/crates/net/p2p/src/beacon/mod.rs @@ -17,6 +17,7 @@ pub mod builder_market; pub mod column_checks; pub mod decode; pub mod encoding; +pub mod engine_columns; pub mod envelope_checks; pub mod fork_schedule; pub mod handler; diff --git a/crates/net/p2p/src/lib.rs b/crates/net/p2p/src/lib.rs index 83e22d945..0835881dd 100644 --- a/crates/net/p2p/src/lib.rs +++ b/crates/net/p2p/src/lib.rs @@ -39,7 +39,7 @@ use std::{ use either::Either; use ethlambda_network_api::{ ConnectedPeer, FetchRequest, InitBlockChain, MetaDataController, P2PToBlockChainRef, - PeerDirection, PeerTableController, + PeerDirection, PeerTableController, SidecarOrigin, block_chain_to_p2p::{ CheckDataColumnSidecars, FetchBlock, PublishAggregatedAttestation, PublishAttestation, PublishBlock, @@ -1979,7 +1979,12 @@ impl Handler for P2PServer { impl Handler for P2PServer { async fn handle(&mut self, msg: CheckDataColumnSidecars, _ctx: &Context) { - beacon::column_checks::check_and_forward(self, msg.sidecars); + match msg.origin { + SidecarOrigin::Parked => beacon::column_checks::check_and_forward(self, msg.sidecars), + SidecarOrigin::ExecutionLayer => { + beacon::engine_columns::publish_and_forward(self, msg.sidecars) + } + } } } @@ -2997,6 +3002,9 @@ pub(crate) mod test_support { Vec, )>, >, + /// Each `new_data_column_sidecars` batch, in arrival order. + pub(crate) sidecar_batches: + std::sync::Mutex>>, } impl ethlambda_network_api::P2PToBlockChain for RecordingChain { @@ -3024,8 +3032,9 @@ pub(crate) mod test_support { } fn new_data_column_sidecars( &self, - _sidecars: Vec, + sidecars: Vec, ) -> Result<(), spawned_concurrency::error::ActorError> { + self.sidecar_batches.lock().unwrap().push(sidecars); Ok(()) } fn data_column_sidecars_awaiting_parent( diff --git a/crates/net/p2p/src/metrics.rs b/crates/net/p2p/src/metrics.rs index f2e9114be..09563fb8c 100644 --- a/crates/net/p2p/src/metrics.rs +++ b/crates/net/p2p/src/metrics.rs @@ -517,6 +517,28 @@ pub fn inc_data_column_fetch_failure(reason: &str) { .inc(); } +/// Count data column sidecars this node built from its own execution client's +/// blobs, by what `beacon::engine_columns` did with them. `result` is +/// `"published"` (newly recorded and sent to its subnet topic), `"no_digest"` +/// (newly recorded and handed to the chain, but no held digest covers its slot +/// so nothing was published),`"already_seen"` (gossip delivered that column +/// first) or `"not_custody"` (not one of this node's columns, so not +/// subscribed and dropped). +pub fn inc_data_columns_from_execution_layer(result: &str, count: u64) { + static LEAN_DATA_COLUMNS_FROM_EXECUTION_LAYER_TOTAL: LazyLock = + LazyLock::new(|| { + register_int_counter_vec!( + "lean_data_columns_from_execution_layer_total", + "Data column sidecars built from the local execution client's blobs, by outcome", + &["result"] + ) + .unwrap() + }); + LEAN_DATA_COLUMNS_FROM_EXECUTION_LAYER_TOTAL + .with_label_values(&[result]) + .inc_by(count); +} + /// Test-only readback of [`inc_data_column_fetch_failure`]'s counter. The /// metric otherwise has no consumer inside the crate itself (Prometheus /// scrapes it), so this is the only way a test can observe that a failure was diff --git a/docs/beacon_engine.md b/docs/beacon_engine.md index 988c90eae..472c34225 100644 --- a/docs/beacon_engine.md +++ b/docs/beacon_engine.md @@ -14,7 +14,7 @@ The wire is ## Which methods, and why so few -Six: +Seven: | Method | Introduced | Why | |---|---|---| @@ -24,14 +24,16 @@ Six: | `engine_forkchoiceUpdatedV4` | amsterdam | The same, from a gloas head on; also carries the node's custody columns | | `engine_exchangeCapabilities` | common | Startup handshake | | `engine_getClientVersionV1` | identification | Log what the execution client is | +| `engine_getBlobsV2` | osaka | Rebuild a held fulu block's custody columns from the execution client's blob pool | Osaka introduces **no new `newPayload`**. Its own document adds only -`engine_getPayloadV5` and `engine_getBlobsV2`/`V3`; `engine_newPayloadV5` belongs +`engine_getPayloadV5` and `engine_getBlobsV2`/`V3`, of which this node calls +`engine_getBlobsV2` (see [Blobs from the execution client](#blobs-from-the-execution-client)); `engine_newPayloadV5` belongs to Amsterdam, where gloas's envelope import uses it. So the Osaka-current call for a payload is prague's V4, and the Osaka-current fork choice notification is cancun's V3. Amsterdam's pair (newPayload V5, forkchoiceUpdated V4) is used from gloas on; the startup handshake warns when the execution client does not advertise either, but only on a network whose schedule includes gloas. -Three method families a full client would have are deliberately absent: +Two method families a full client would have are deliberately absent: - **`engine_getPayload*` outside the gloas path.** The follower itself never asks an execution client to start building a block, so its own @@ -43,8 +45,6 @@ Three method families a full client would have are deliberately absent: `engine_getPayloadV6` (`ExecutionPayloadV4`, with `blockAccessList` and `slotNumber`) from gloas on. The handshake warns when a gloas network's execution client does not advertise `engine_getPayloadV6`. -- **`engine_getBlobs*`.** There is no blob-pool fetch path here; data columns come - from peers over the network. - **`engine_getPayloadBodies*`.** Nothing consumes them. Blocks before electra are not asked about at all. This node checkpoint-syncs onto @@ -315,6 +315,121 @@ costs one retry ladder per slot however many envelopes wait. This is the gloas counterpart of the limitation below, and unlike a block the envelope is not dropped. +## Blobs from the execution client + +A fulu block is held until its custody data columns are stored. Gossip usually +delivers them within a fraction of a second, but the execution client may +already have every blob the block commits to, since the blob transactions went +through its pool first. The fulu p2p spec ("Distributed blob publishing using +blobs retrieved from local execution-layer client") says honest nodes SHOULD +ask `engine_getBlobsV2` on receiving a valid `beacon_block` or +`data_column_sidecar` from gossip, and turn a complete answer into data columns. +The code is `crates/blockchain/src/beacon_blobs.rs`. + +**When it fires.** On the first data-availability hold of a block +(`hold_block_for_columns`), when all of these hold: the block is fulu, it +commits to at least one blob, `--execution-endpoint` is set, the block is at +most `MAX_ASK_AGE_SLOTS` (4) slots behind the store's clock, and it passes +`precheck_block` against its parent's post-state (expected proposer and +proposer signature). Any source qualifies (gossip, or a by-root answer). A +redelivery of a held root does not ask again. + +The age bound is what keeps range sync out. Every block it delivers is older +than the slot it arrives in, and a catch-up holds them by the batch, so asking +for each would be an unbounded fan-out of prechecks on the chain actor, engine +calls and cell computations. Such a block's columns are asked of peers as soon +as it is held, so it does not wait for a redrive either way. How often the +execution client would still hold an older block's blobs is not measured. + +The precheck is a filter, not the guard: the built columns go through every +chain check before they are published (see the build below). It saves an engine +call and a cell computation for a block whose signature already fails, which +range sync and by-root answers can deliver, since they never pass gossip +validation and the import checks the signature only after data availability. +It cannot catch everything: past the parent state's proposer lookahead, +`precheck_block` checks the signature against whichever validator the block +names, not that this validator is the slot's proposer. It costs one BLS +verification per first hold of a block that would be asked about. + +**The call.** One attempt with a 1 s timeout (`ENGINE_GET_BLOBS_TIMEOUT`, the +value `osaka.md` gives the method), outside the retry ladder. The call races +gossip delivery of the same columns, and a retry after the ladder's first +backoff would land after that race is decided. A miss leaves the block to +gossip and the per-slot peer asks, which is where it was anyway. The answer is +all-or-nothing: `null` means the execution client lacks at least one blob (or is +syncing), and counts as a miss. + +**V2, not V3.** V3 answers partially, but a column holds one cell from every +blob, so a partial blob set builds no complete column, and this node has no +partial-column assembly. + +**The build.** On a full answer, a blocking task first re-checks that the block +was not imported meanwhile and which custody columns are still missing from the +store, then builds only those (cells and sidecars) off the async runtime and off +the chain actor. Still in that task, the sidecars then go through the chain +checks a fetched column gets (`ChainPrechecks`, then one `kzg_verdicts` batch): +the signed header, including the expected proposer past the lookahead, +finalized ancestry, the inclusion proof, and the KZG proofs. Only the ones that +pass go on. The spec says a node SHOULD skip verifying blobs from its own +execution client; this one does not (see `docs/spec_deviations.md`). + +**The hand-over.** The sidecars go to p2p through the existing +`check_data_column_sidecars`, with `SidecarOrigin::ExecutionLayer`. p2p runs no +checks of its own on that origin, since the chain actor's task ran them. p2p +records each column in the gossip seen cache first (so a +copy gossip delivers later is ignored, and a column gossip already accepted is +not published twice), publishes the newly recorded ones on their subnet topics +if and only if this node is subscribed to them, which here means its custody +columns, and forwards one batch to the chain actor. The chain actor stores the +columns and releases the held block through the usual completeness check. + +``` +hold (1st, recent) -> engine_getBlobsV2 1 s, one attempt + |-- null / error -> count; gossip and peer asks carry on + '-- every blob -> build missing custody columns + (block imported meanwhile? stop) + chain checks, KZG included + (refused -> count, drop) + | + check_data_column_sidecars(ExecutionLayer) + | + p2p: seen cache -> publish on subscribed subnets -> forward one batch + | + chain actor: store columns -> release the held block +``` + +**Trust.** Nothing the execution client returns is taken on trust. A blob that +is not the one a commitment names, or a wrong proof, fails the KZG batch, and +the columns built from it are counted in +`lean_engine_blob_columns_refused_total` and dropped: never published, never +recorded seen, never stored. That matters beyond peer score. A stored column +keeps every other copy of it out (gossip and the fetch checks answer +`AlreadyStored`), and block import trusts what is stored, so a column taken on +trust would either stop its block for good or import it on data nobody +verified. The block then waits for gossip and the peer asks, as on a miss. The +price is one KZG batch over the custody columns per hit, on the blocking pool, +before the columns are published. + +**Not done.** + +- No ask when a sidecar arrives, only on a block's first hold. +- Gloas: data availability moves to the payload envelope, so a held gloas block + has no columns to fill this way. +- Partial columns (see V3 above). + +**Metrics.** + +| Metric | Reads | +|---|---| +| `lean_engine_get_blobs_total{outcome}` | Attempts: `hit`, `miss`, `error` | +| `lean_engine_get_blobs_duration_seconds` | One round trip, whatever the outcome | +| `lean_engine_blobs_to_columns_seconds` | Cell computation plus sidecar build | +| `lean_engine_blob_columns_refused_total{reason}` | Built columns the chain checks refused, by rule (`kzg`, `wrong_proposer`, ...); any of these points at the execution client or a forged block | +| `lean_data_columns_from_execution_layer_total{result}` | What p2p did with each column: `published`, `no_digest`, `already_seen`, `not_custody` | + +A high `miss` rate means the execution client's pool does not have the blobs. +That is normal while the execution client is syncing. + ## `forkchoiceUpdated` Sent once after every head recompute, which is once per import cascade and once @@ -374,6 +489,8 @@ authentication, so an endpoint without a secret is refused at startup, before anything binds or connects. Clap cannot express "both or neither" across two optional flags, so the pairing is checked once, in `From`. +Blob fetching has no flag of its own: it is on whenever `--execution-endpoint` is set. + With **neither** flag, the follower contacts no execution client and every block gets `PayloadValidity::NotRequired`, which is exactly what it did before any of this existed. diff --git a/docs/metrics.md b/docs/metrics.md index e1a19b128..71d2c8619 100644 --- a/docs/metrics.md +++ b/docs/metrics.md @@ -636,6 +636,18 @@ spec. | `lean_envelopes_awaiting_block` | Gauge | Gloas execution payload envelopes held until their block is imported | On every hold, release, and finality eviction of the held envelopes | | | | `lean_envelopes_awaiting_columns` | Gauge | Gloas execution payload envelopes held until every sampled column is stored | On every hold, release, and finality eviction of the held envelopes | | | | `lean_blocks_awaiting_parent_payload` | Gauge | Gloas blocks held until their FULL parent's payload envelope is verified | On every hold, release, and finality eviction of the held blocks | | | +| `lean_engine_get_blobs_total` | Counter | `engine_getBlobsV2` attempts for held fulu blocks, by outcome | On each answer of the one attempt made at a block's first hold | outcome=hit,miss,error | | +| `lean_engine_get_blobs_duration_seconds` | Histogram | Time for one `engine_getBlobsV2` round trip | On each attempt, whatever its outcome | | 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0 | +| `lean_engine_blobs_to_columns_seconds` | Histogram | Time to build custody data column sidecars from execution-client blobs | On each build that reaches the cell computation, off the actor thread | | 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.0 | +| `lean_engine_blob_columns_refused_total` | Counter | Data columns built from execution-client blobs that the chain checks refused, so neither published nor stored | On each built column refused, once; a column another route stored meanwhile is not counted | reason=kzg,wrong_proposer,inclusion_proof,... (the refusing rule's label) | | +| `lean_data_columns_from_execution_layer_total` | Counter | Data column sidecars built from the execution client's blobs, by what p2p did with them | On each hand-over, once per result, incremented by the column count | result=published,no_digest,already_seen,not_custody | | + +`lean_engine_get_blobs_total` and its siblings cover the execution-client path +(`docs/beacon_engine.md`, "Blobs from the execution client"). A high `miss` +share means the execution client's blob pool lacks the blobs, which is normal +for a syncing execution client; `already_seen` in +`lean_data_columns_from_execution_layer_total` counts columns gossip delivered +first. `lean_data_columns_rejected_total` counts the chain checks, which run in the p2p layer on every sidecar gossip did not accept: every fetched sidecar, a diff --git a/docs/spec_deviations.md b/docs/spec_deviations.md index 9e32cfeb0..870cae6eb 100644 --- a/docs/spec_deviations.md +++ b/docs/spec_deviations.md @@ -687,6 +687,25 @@ verified ancestor: a gloas block's EL parent is the ancestor whose payload hash is its `bid.parent_block_hash`, skipping blocks it built past on their EMPTY node, and a pre-gloas block's is its parent. Both walks stop at finality. +## Columns built from the execution client's blobs are verified + +- **The specification:** fulu `p2p-interface.md`, "Distributed blob publishing + using blobs retrieved from local execution-layer client": a client that + builds data column sidecars from `engine_getBlobsV2`'s answer SHOULD skip + verifying them, and MUST then treat them as received on gossip. +- **ethlambda:** the built sidecars go through every chain check a fetched + column gets, KZG proofs included, before they are published, recorded seen + or stored (`crates/blockchain/src/beacon_blobs.rs`, `check_built`). Columns + that fail are dropped and counted in `lean_engine_blob_columns_refused_total`. +- **Why:** a stored column keeps every later copy of it out (`AlreadyStored`), + and block import trusts the stored columns, so a wrong blob or proof taken + on trust from a faulty execution client would either stop its block for + good or import it on data nobody verified. The same checks also catch a + block from the wrong proposer past its parent state's lookahead, which the + proposer precheck made before the call cannot. +- **Consequence:** one KZG batch over the custody columns per `getBlobsV2` + hit, on the blocking pool, between the answer and publication. + ## `forkchoiceUpdated` for a gloas head - The head hash follows the head node: `bid.block_hash` for a FULL head,