Repository navigation
perf(beacon): back the participation lists and pending deposits with persistent trees - #762
Conversation
…persistent trees On mainnet (about 2.4M validators) the post-block state_root check costs about 27 ms per block, and perf attributes most of BeaconState::hash_tree_root to plain SszList fields re-merkleized from scratch on every call: the two participation lists (2.4 MB each) and pending_deposits. Lighthouse keeps every state list in milhouse trees with cached node hashes. EpochParticipation (altair through fulu), PendingDeposits (electra, fulu) and their gloas progressive counterparts now use ethlambda_ssz_tree lists. Only the current epoch's participation list is written by a block, so the previous epoch's list and an untouched deposit queue reuse their cached roots. The SSZ encoding and roots are unchanged (tests compare against the flat lists), so DB_VERSION is not bumped. Participation writes are buffered densely, not in a BTreeMap: at one slot's worth of scattered attesters the BTreeMap inserts cost several times the hashing they save. A quick ignored benchmark at 2.4M entries and 75k writes measures about 13 ms for the flat lists against about 5 ms for the trees. Reads go through ParticipationRef and ParticipationMut, since a tree has no slice view. The participation scan walks the list in lockstep with the registry instead of a tree descent per validator. Both lists are also rebased onto a resident relative of a state decoded from storage.
|
Combined result for the four Lighthouse ports (#759 pubkey index, #760 column KZG, #761 progressive target balances, #762 tree-backed participation), squashed onto Offline import replay on ethlambda-5, same three mainnet corpora, both legs built at the commit and run with
Per STF step, ordinary blocks (n=162): Live on the ethlambda-2 mainnet follower (nightly 1d06bf9): steady |
…peat Three primitives the participation-list fixes that follow need, kept in their own commit so the fixes read as pure call-site changes. - `List::set` / `ProgressiveList::set` buffer a value without first descending the tree to copy the old one into the write buffer, which `get_mut` does even when the caller already knows the old value. - `sorted_reader()` reads non-decreasing indices keeping the current leaf's slice, descending only when an index leaves it. Pending writes are consulted first, exactly as `get` does, so the answers are identical. - `List::repeat` / `ProgressiveList::repeat` build a list of identical elements by sharing one full leaf and one full inner node per level, each caching its hash once, so building and hashing cost the tree height plus the partial last leaf instead of the whole length. Nodes are immutable and every write copies the path it touches, so a shared leaf can never be mutated; tests write into a repeated list and check its clone and the untouched leaves, and compare roots against the Vec-built list at lengths around every leaf and node boundary.
…rotation Two micro-benchmarks next to `participation_root_after_a_block_of_writes_bench`, at 2.4M validators: the participation part of one `process_attestation` (about 75k ascending reads, then the writes) in each combination of per-index `get`/sorted reader and `get_mut`/`set`, and the epoch rotation's new list built from a zeroed `Vec` against `List::repeat`, with the root that follows. Run with `cargo test --profile release-fast -p ethlambda-types --lib -- --ignored --nocapture attestation_pass_bench participation_rotation_bench`.
The write phase did `get_mut(index)` and `*flags |= new_flags` for every attester. On a tree-backed list `get_mut` descends the tree to copy the old value into the write buffer, although the read phase had already read it as `current_flags`. Micro-benchmark at 2.4M validators and about 74k ascending attesters (best of five, release-fast): the write phase takes 2.16 ms with `get_mut` and 0.94 ms with the blind `set`. The read phase now records `current_flags | new_flags` and the write phase stores it with the new `ParticipationMut::set`. That is exactly what `|=` would leave: the attesting indices are distinct (members of disjoint committees, and the signature set built before the read phase refuses a repeated index), so each index is written once, and nothing writes the list between the two phases, so the stored value is still `current_flags`. Applies to the altair, deneb, electra (electra and fulu) and gloas versions of `process_attestation`.
… reader Once the participation lists became trees, the read phase of `process_attestation` paid a full tree descent per attester (`get` was a slice index before). An attestation carries tens of thousands of attesters. Its indices are sorted ascending and consecutive ones almost always share a leaf, which holds thousands of flags. `sorted_reader()` keeps the current leaf's slice and descends only when an index leaves it, consulting pending writes first exactly as `get` does, so the answers are identical. Micro-benchmark at 2.4M validators and about 74k ascending attesters (best of five, release-fast): the read phase takes 0.70 ms per-index and 0.10 ms with the reader; with the blind write from the previous commit the whole participation pass goes from 2.86 ms to 1.03 ms. Every `process_attestation` (altair, deneb, electra and fulu, gloas) now uses it; the reader debug-asserts that indices do not decrease, which `get_attesting_indices` guarantees by sorting.
…o leaf `process_participation_flag_updates` installed the new current list as `vec![0; validator_count]` converted into a tree: about 2.4M bytes allocated and filled into hundreds of separate leaves, every one of which the next state root then hashed. At epoch boundaries `process_epoch` went from 96.3 ms to 105.5 ms in the offline import replay once the lists became trees. `List::repeat` / `ProgressiveList::repeat` (added with the other ssz-tree primitives) build the list from one full leaf and one full inner node per level, each shared through its `Arc` with its cached hash, so building and hashing cost the tree height plus the partial last leaf. Micro-benchmark at 2.4M validators (release-fast, steady state over three runs): the old build plus the root takes 3.0 to 6.0 ms, the repeated list 0.10 to 0.13 ms. Nodes are immutable and every write copies its path, so sharing is invisible to later writes; the roots and SSZ bytes are tested equal to the flat list's at lengths around every leaf and node boundary, and a write to a repeated list is tested to leave its clone and the other leaves alone. Used by the altair-through-fulu rotation, gloas's own copy of it, and the altair upgrade's two fresh lists. PendingDeposits is rebuilt from a `Vec` once per epoch by `process_pending_deposits`, which only drops a prefix and appends postponed deposits. The tree cannot keep the suffix's cached hashes: leaves are aligned runs, so dropping a prefix that is not a multiple of a leaf shifts every later element into a different leaf, and the API has no operation that re-chunks while carrying cached element roots. An ignored benchmark in `ethlambda-types` puts the cost at about 7 ms for the copy and rebuild plus about 24 ms to hash, for a queue of 200k deposits (about 1 ms plus 3 ms at 20k), once per epoch. The hash was paid on every block before the trees, so this is left as it is.
|
Per-PR attribution on
#760 does not run in the replay (no custody columns), so the second step is #761's. Regressions found by the per-step split, and what the follow-up commits (now pushed to #761 and #762) did about them:
ethlambda-2 (mainnet follower) runs nightly 2b51a33 since 2026-11-03 14:42 UTC. |
🤖 Codex Code ReviewNo actionable findings. The tree-backed list migration appears to preserve participation updates, epoch rotation, and SSZ encoding and hash behavior. The sorted reader’s ordering assumption matches attester-index construction. Tests could not run: the pinned Rust toolchain was unavailable, and the offline fallback lacked the required Automated review by OpenAI Codex · gpt-6.1-sol · custom prompt |
🤖 Claude Code ReviewReview: tree-backed participation lists and
|
🤖 Gemini Code ReviewThis PR looks great. The additions to A few highlights from reviewing the implementation:
No issues found — LGTM. Automated review by Gemini · gemini-4-argon · custom prompt |
🤖 Grok Code ReviewAutomated review by Grok (xAI) |
🤖 Kimi Code ReviewAutomated review by Kimi (Moonshot AI) |
🤖 DeepSeek Code ReviewReviewed the tree-backed migration for participation lists and
The tree-backed sharing and root-equality coverage look otherwise solid. Automated review by DeepSeek · deepseek-v4-pro · custom prompt |
ProgressiveSortedReader::get subtracted the current subtree's start from an index below it, which wraps in release builds and answers None for an element that exists. SortedReader answers right on a step back and only pays a descent. The progressive one now restarts from the first subtree when the index is below the subtree it last read, or after a read past the end, so both readers keep one contract. In-order reads take the same path as before; the existing reader-vs-get test, which reads past the end, covers the restart after an end read.
…rite exact Each fork's process_attestation said its attesting indices are sorted and distinct because of a signature set built above. No such set is built there: is_valid_indexed_attestation, verified just before the read phase, refuses any index list that is not strictly ascending, in release builds too. Both comments now name it, and electra's no longer runs past the line width.
…cument two read invariants rebase_participation_on reached gloas through an else-if that ran only because altair_validator_lists_mut fails on a gloas state. It now matches the gloas pair by name first, then the bounded pair, with the same outcome for every pairing. ParticipationRef's PartialEq compares entries across list kinds; its doc now says so and points to ptr_eq. Tree::Zero's doc says a Zero only ever stands past the end of the data, which Tree::get and leaf_run rely on.
|
Round one (the automated review of 0960567). Fixed in ca69a1d, 46cc758 and 74690a0. Fixed
Not changed
Codex and Gemini found nothing to change. Grok's and Kimi's jobs failed (curl exit 56) and produced no review. |
|
/review |
AI security review: 0 finding(s) to checkReviewed by Opus, GPT, Kimi, GLM, DeepSeek. No findings to check. Housekeeping (not security): 7Stale docs, missing tests, leftovers and inconsistencies, from a separate cheap model. Not checked by the judge.
Cost of this review: ~$5.34 of a $30 budget (estimated from token counts). |
…g deposits where docs still said slices The move to tree-backed lists left seven places describing the old shapes: 1. PendingQueueFields said all three queues deref to [T] and offer slice views. pending_deposits is now a tree-backed list, reached only through push, take and set. 2. The participation module said its scan zips the registry against participation slices. 3. The gloas epoch module doc named libssz_types::ProgressiveList for all five progressive fields, and said the participation steps read plain slices. 4. process_justification_and_finalization said the same. 5. fulu's process_deposit_request named the wrong ProgressiveList for gloas's pending deposits. 6. The rewards spec-test runner's comment, and the same comment in the epoch_processing runner, said the lists deref to [T]. 7. beacon_stf.md spoke of flat participation slices. Each now names ParticipationRef (or InactivityScoresRef) and the ethlambda_ssz_tree lists.
|
Round two (the automated review of 109b8f8). Fixed in 0428fb8, comments and docs only. The review raised no security findings. I checked each of the seven housekeeping items against the code, and all were stale docs from the move to tree-backed lists. Each is fixed in that one commit: Fixed
|
|
/review |
pablodeymo
left a comment
There was a problem hiding this comment.
I read this together with #761, at 0428fb8. The sorted reader and the blind writes are exact for the reasons the comments give, and merged with #761 every fork's process_attestation makes one sorted pass and reads each attester once. Two places where the tree still costs more than it needs to, inline; the second may be part of the process_epoch +7.6 ms the description says is still being profiled.
| } | ||
|
|
||
| /// The flags at `index`, or `None` past the end. A tree descent. | ||
| pub fn get(&self, index: usize) -> Option<&'a ParticipationFlags> { |
There was a problem hiding this comment.
pack_attestations (block_production.rs:265, outside the diff) still calls this once per attester of every candidate to count new attesters, so each call is now a tree descent where it used to be a slice read. It only runs when proposing, but over the whole pool. get_attesting_indices returns sorted indices, so sorted_reader() fits there the same way it does in process_attestation.
| match self { | ||
| PendingQueueFields::Electra(state) => { | ||
| core::mem::take(&mut state.pending_deposits).into_inner() | ||
| core::mem::take(&mut state.pending_deposits).to_vec() |
There was a problem hiding this comment.
The epoch code takes the whole queue out here, and set_pending_deposits builds a fresh tree from the Vec (single_pass.rs:535, stf/epoch/electra.rs:540). So every epoch boundary copies every pending deposit and discards all the cached hashes, even when no deposit was applied or postponed. When remaining is the queue unchanged, leaving the list in place would keep its root. to_vec() here also clones each deposit where into_inner() used to move it.
Could this be part of the process_epoch regression?
AI security review: 0 finding(s) to checkReviewed by Opus, GPT, Kimi, DeepSeek. No findings to check. Cost of this review: ~$5.21 of a $30 budget (estimated from token counts). |
🗒️ Description / Motivation
The post-block state root check cost ~27 ms per block on the mainnet follower. Perf put 87% of
BeaconState::hash_tree_rootin plainSszListfields re-merkleized from scratch on every call: the two participation lists (2.4 MB each) andPendingDeposits; only 13% was in the tree-backedethlambda_ssz_treelists. Lighthouse keeps every state list inmilhousepersistent trees with cached node hashes.What Changed
EpochParticipation(altair through fulu) isethlambda_ssz_tree::List<u8, LIMIT>with the denseVecMapwrite buffer;PendingDeposits(electra, fulu) isList<PendingDeposit, LIMIT, BTreeMap>; gloas's areethlambda_ssz_tree::ProgressiveList. All intree_fields!, soapply_pending_mutationsflushes them, andBeaconState::rebase_onshares them with a resident relative.altair_validator_lists()returnsParticipationRefs,epoch_participation_mut()aParticipationMut.helpers/participation.rs::scanwalks both participation iterators in lockstep withiter_validators()instead of an indexed read per validator.process_attestationvariants, epoch rotation, fork upgrades andprocess_pending_deposits;docs/beacon_stf.mdandCLAUDE.mdupdated.Correctness / Behavior Guarantees
DB_VERSIONbump: new tests compare the old and new representations at lengths 0, 1, 31, 32, 33, 1000, 65541 and 300001 (bounded and progressive participation, pending deposits), plus writes and a push after a warm hash.BTreeMapbuffer was a clear loss for participation (75k scattered writes: 29-32 ms vs 13 ms flat); the denseVecMapbuffer gives ~5 ms for the two lists.Benchmarks
Offline
ethlambda benchmark import replayon ethlambda-5 (Ryzen 9700X), three mainnet corpora, 2 rounds, steady state n=164, paired per block against main012dc4f49:Per round -15.3% / -13.6%. Epoch-boundary blocks (n=2) -4.9%.
Follow-up commits: regressions this PR introduced
Measured on nightly with all four Lighthouse ports, this PR's first commit cost
attestations+1.5 ms per block (a tree descent per attester to read its flags, and another perget_mutwrite) andprocess_epoch+9.1 ms at epoch boundaries. The last five commits address that:ethlambda_ssz_tree:List::set/ProgressiveList::set(a blind buffered write),sorted_reader()(reads non-decreasing indices, descending only when an index leaves the leaf last read), andrepeat(value, len)(one full leaf and one full inner node per level, shared, each hashing once).process_attestation: the read phase uses the sorted reader (attesting indices are sorted) and recordscurrent | new; the write phase stores it withset. That is exact because an attestation's indices are distinct and nothing writes the list between the two phases.repeat.Micro-benchmarks (2.4M validators, ~74k ascending attesters): read 0.56-0.70 -> 0.10 ms, write 1.6-2.2 -> ~1.0 ms; rotation plus the next hash 3-6 ms -> 0.1-0.2 ms.
On nightly, together with #761's follow-up, offline replay on ethlambda-5, paired, steady state n=164:
attestations(ordinary blocks)process_epoch(epoch boundaries)Still above nightly before the ports (2.9 ms and 97.2 ms):
attestations+1.4 ms (shared with #761) andprocess_epoch+7.6 ms per epoch boundary; being profiled.PendingDepositsis left as it is: dropping a processed prefix that is not a multiple of a leaf's element count shifts every later element into another leaf, so the cached hashes cannot be kept without a re-chunking operation. Its per-epoch cost is about 1 ms plus 3 ms of hashing for 20k deposits, against a hash that used to be paid on every block.Tests Added / Run
participation_root_after_a_block_of_writes_bench(cargo test --profile release-fast -p ethlambda-types --lib -- --ignored --nocapture participation_root).set, the sorted reader andrepeatchecked againstgetand against Vec-built lists (roots and SSZ bytes) at lengths around every leaf and inner-node boundary up to 2.4M; writes into a repeated list throughset,get_mut,iter_cow,rebase_onandpushleave the clone and untouched leaves unchanged. Ignored benchesattestation_pass_bench,participation_rotation_bench,pending_deposits_epoch_rebuild_bench.cargo test --profile release-fast -p ethlambda-ssz-tree --lib(80, also in the dev profile),-p ethlambda-types --lib(237),-p ethlambda-state-transition --lib(630),-p ethlambda-storage --lib(210);cargo checkof blockchain, rpc, p2pssz_static/1761,sanity/595,epoch_processing/804,/attestation/356,fork/138,transition/172,/random/208,fork_choice/206,/deposit_request/43,rewards/372,finality/41Related Issues / PRs
✅ Verification Checklist
make fmt— cleanmake lint(clippy with-D warnings) — cleanmake test— not run; the crate lib tests and spec filters above were run instead