Repository navigation
perf(beacon): keep unslashed target balances incrementally for the pulled-up tip - #761
MegaRedHand wants to merge 18 commits into
Conversation
…lled-up tip compute_pulled_up_tip ran on every imported block and, at ~2.4M validators, cloned the whole post-state (~2.5 ms) and scanned the registry for the participation totals (~6 ms), plus an uncached total active balance. Lighthouse avoids both with its ProgressiveBalancesCache. The per-fork states derive SSZ, so the cache lives in the Store, keyed by block root (TargetBalanceCache: raw previous/current unslashed TIMELY_TARGET sums, bounded by capacity and pruned below the finalized epoch). process_attestation in every fork from altair on records into a TargetBalanceDelta, carried in BlockProcessingOptions, when a validator newly earns the target flag. A block in the same epoch as its parent has the parent's totals plus that delta; an epoch boundary, a cache miss or a block carrying slashings takes one registry scan, cached for the children. The justification weighing now runs on a small JustificationState extracted from the state, so the pulled-up tip clones nothing and reads the total active balance from the store's ActiveBalanceCache. weigh_justification_and_finalization delegates to the same logic, so epoch processing is unchanged. Debug builds also run the old full path and assert the checkpoints and totals agree, which the fork_choice fixtures exercise on every block.
|
Combined result for the four Lighthouse ports (#759 pubkey index, #760 column KZG, #761 progressive target balances, #762 tree-backed participation), squashed onto Offline import replay on ethlambda-5, same three mainnet corpora, both legs built at the commit and run with
Per STF step, ordinary blocks (n=162): Live on the ethlambda-2 mainnet follower (nightly 1d06bf9): steady |
…nce delta The progressive target-balance delta recorded every attester that newly earned TIMELY_TARGET with `state.validator(index)`, a second registry tree descent per attester on top of the one the reward computation had already made. Offline import replay at mainnet scale put the `attestations` step at 5.8 ms per block against 2.8 ms before the tree-backed lists and the delta; this is the delta's share of that. The read phase now keeps the effective balance and `slashed` flag it read for the reward, and `TargetBalanceDelta::record_new_target` takes those two values instead of a `&Validator`. Nothing in the read phase writes the registry, so the cached values are the ones a second read would return, and a newly earned target flag implies the loop reached that flag and read the attester first. Applies to the altair, deneb, electra (electra and fulu) and gloas versions of `process_attestation`.
|
Per-PR attribution on
#760 does not run in the replay (no custody columns), so the second step is #761's. Regressions found by the per-step split, and what the follow-up commits (now pushed to #761 and #762) did about them:
ethlambda-2 (mainnet follower) runs nightly 2b51a33 since 2026-11-03 14:42 UTC. |
get_head tallied every validator's latest message on every call, about 1M on mainnet, and a live import pays it twice for the first timely block of a slot (on_block's boost-gate walk, then the actor's get_head). That tally was most of get_head's ~17 ms in the eth-5 replay. The store now keeps the vote weights in a node array (Lighthouse unstable's proto-array layout: one node per block, carrying its PENDING, EMPTY and FULL weights). One attestation is one vote batch: Store::record_latest_messages records the messages and moves the attesters' weight under one lock, and the next read refreshes the subtree totals in one reverse pass over the blocks. The head walk reads them through cached_node_weights. The proposer boost, gloas's previous-slot rule and the walk bound depend on the clock, so they stay read-time. There are no best-child pointers: under gloas the choice moves without any vote, which is why Lighthouse stopped maintaining them. What the cache cannot follow incrementally (finality pruning, a live-window removal, a restart, a new justified balances snapshot, the gloas rule switch, a block stored without its node) marks it stale, and the next read rebuilds it from the latest messages at the old tally's cost, about once per epoch. A rebuild rescans LiveChain under the lock, so an API read that scanned before a removal cannot rebuild the removed blocks back in. compute_node_weights stays, as the reference. Equivalence is checked by a proptest against a naive tally (with the incremental state also equal to a fresh rebuild after every operation), on the random trees at every step of every seed, by a new test that feeds votes, blocks and equivocations through update_latest_messages, on_block's insertion order and insert_equivocating_index, and at every head check of every fork-choice fixture. Local benchmark (1M validators, 73 blocks, laptop): the tally 27 ms; a warm cached read 7 us; a read after a 15.6k-vote batch 49 us; the batch itself 1.3 ms; a rebuild 17 ms. Cherry-picked from perf/beacon-proto-array at b4adbaf. Adapted onto nightly: both vote writers (set_latest_message and the new record_latest_messages) grow the dense vote table through one grow_latest_messages helper, which keeps #740's exact growth plus headroom; the storage and fork-choice imports are merged with #759's and #761's caches. Replaces #767, reverted in the previous commit.
🤖 Grok Code ReviewI'll trace the incremental target-balance path against justification and attestation processing, since a mismatch there would change fork choice. I need the surrounding justification and participation code to check the invariants.I'll check how justification skips early epochs and how participation totals treat slashed validators, so the incremental path can be compared to the spec. Automated review by Grok (xAI) · grok-4.7 · custom prompt |
🤖 Codex Code ReviewNo actionable findings. The incremental target-balance logic correctly falls back to full scans across epoch boundaries, cache misses, and slashings. The justification/finalization refactor preserves the existing rules. Tests could not run because Rustup attempted to write to the read-only Automated review by OpenAI Codex · gpt-6.1-sol · custom prompt |
🤖 Claude Code ReviewReview: incremental target balances for the pulled-up tip (PR 761)The design is sound and well documented. Extracting Correctness / consensus risk
Rust / API
Storage
Tests
Minor
Overall this is a good change. I'd like to see Item 1 (release-mode safety net) addressed before merge, given that the failure mode is silent fork-choice divergence. Automated review by Claude · claude-opus-5-5 · custom prompt |
🤖 Gemini Code ReviewThis PR looks great. The design of
No issues found — LGTM! Automated review by Gemini · gemini-4-argon · custom prompt |
🤖 DeepSeek Code ReviewThe PR is well-structured and the idea of caching unslashed timely-target balances to avoid a registry scan is sound. However, I found a few correctness issues that should be addressed before merge.
Automated review by DeepSeek · deepseek-v4-pro · custom prompt |
🤖 Kimi Code ReviewOverall, this looks correct and well-designed. The incremental totals preserve the raw-sum/floor distinction, newly earned target flags are counted only once, slashing-containing blocks fall back to a full scan, and epoch changes prevent invalid parent extensions. The Two non-blocking robustness concerns:
Test coverage is good for delta accounting and cache eviction. An additional integration test explicitly exercising a slashing block’s full-scan fallback and an epoch-boundary cache miss would help protect the two most important fallback conditions. Automated review by Kimi (Moonshot AI) · kimi-k3.1-pro · custom prompt |
…r, and test the slashing fallback on_block chose between the parent's cached totals and a full scan inline: a block that slashes must scan, since the delta does not see a validator leaving the unslashed set. No fork choice fixture imports a block that carries a slashing (they arrive as their own steps), so the debug-build equivalence check never ran that branch. totals_from_parent now holds the whole decision, and a_block_that_slashes_is_not_extended_from_its_parent checks it. block_has_slashings answered true for a lean block under a comment saying lean blocks carry none. It now panics through lean_block_unreachable, like every other beacon accessor, and is private to the helper.
record_new_target loaded and stored separately, which is correct only while a single thread writes. The type is Sync and nothing enforces that, so each addition is now one fetch_update. Still saturating, like the registry scan's own sums.
…hat block import sets target_balances: a gloas block's pre-state is its parent block's post-state on either payload branch, since an envelope is only verified and a FULL parent's execution requests run inside the child's own block. So the parent's cached entry is the right base for both. BlockProcessingOptions said the default is the only value block import uses; import now sets target_delta.
|
Round one (the automated review of 713e1fd). Fixed in ce1331c, 4271e8f and cadc844. Fixed
Not changed
Codex and Gemini found nothing to change, and Grok's comment ended after its preamble with no findings. |
|
/review |
🤖 Grok Code ReviewI'll inspect the justification and participation paths this PR depends on so the incremental totals stay consistent with the spec.I need the early-epoch justification skip and how participation flags are applied, because those decide whether the incremental totals are safe.The flag-update timing relative to the delta is the risky part; I'll read that next. Automated review by Grok (xAI) · grok-4.7 · custom prompt |
🤖 Codex Code ReviewNo actionable findings. The incremental target balances correctly fall back to a full scan at epoch boundaries, on cache misses, and for blocks containing slashings. Speculative imports preserve the collected delta, and the justification refactor retains the existing rules. Tests could not run: the pinned toolchain was unavailable, and the offline fallback lacked the required Automated review by OpenAI Codex · gpt-6.1-sol · custom prompt |
🤖 Claude Code ReviewReview: incremental target balances for
|
🤖 Gemini Code ReviewThe implementation looks solid and well-reasoned:
No issues found. Automated review by Gemini · gemini-4-argon · custom prompt |
🤖 Kimi Code ReviewNo blocking correctness or security issues found. The incremental calculation appropriately keeps totals unfloored, excludes slashed attesters, and falls back to a full registry scan at epoch boundaries, cache misses, and blocks containing slashings. Retaining the full phase0 path and adding debug differential assertions against the specification path are particularly good safeguards. Non-blocking observations:
Automated review by Kimi (Moonshot AI) · kimi-k3.1-pro · custom prompt |
🤖 DeepSeek Code Review
Automated review by DeepSeek · deepseek-v4-pro · custom prompt |
Keeping the effective balance and slashed for the target-balance delta moved the division into the flag loop of altair's, deneb's and electra's process_attestation, so an attester earning three flags paid it three times, where main hoisted it. The kept tuple now carries the increments too, divided on the validator's first read, beside the two fields the delta reads. Gloas, which divided per flag on main as well, gets the same shape. The arithmetic is unchanged, so the rewards are bit-identical.
the_target_delta_is_what_a_registry_scan_gains named the unslashed share slashed_share and assumed every attester held MAX_EFFECTIVE_BALANCE. It is now unslashed_share, summed from the committee's own effective balances less the slashed member's.
…ug builds pulled_up_checkpoints compared its totals with a full scan in debug builds even when it had just scanned for them, which compared a scan with itself and doubled the registry passes of every fork-choice fixture block without a cached parent. The assertion now runs only for totals extended from the parent's.
|
Round two (the automated review of cadc844, judged against 99aa236, which merged main after it). Fixed in fbff48d, f5716b1 and 89381bd. Fixed
Not changed
Codex and Gemini found nothing to change, and Grok's comment ended after its preamble with no findings. |
|
/review |
pablodeymo
left a comment
There was a problem hiding this comment.
I read this together with #762, at 89381bd. The exactness argument holds as far as I can check: every fork's process_attestation feeds the delta, effective balances only move at epoch boundaries (where this falls back to the scan), blocks with slashings take the scan, and in gloas the parent's execution requests are applied inside the child's own transition. Merged with #762, the two touch the same attestation loops and don't conflict, and each attester is read once. Two points inline, neither blocking.
|
|
||
| /// Whether the block carries an operation that slashes a validator; see the | ||
| /// module documentation for why such a block is not extended incrementally. | ||
| fn block_has_slashings(block: &SignedBeaconBlock) -> bool { |
There was a problem hiding this comment.
The incremental totals are right only while two things stay true: process_attestation is the only place a participation flag is set mid-epoch, and a block-body proposer or attester slashing is the only way a validator is slashed. I checked every slash_validator caller today (the shared proposer slashing, gloas's own, attester slashings), and this match covers them all.
My worry is the next fork. A new slashing path, or anything that changes an effective balance mid-epoch, would have release builds extend stale totals into a wrong pulled-up tip with no error, and only debug builds would notice. Could the slashing case be recorded in slash_validator instead, the way Lighthouse's on_slashing does? That would also remove this per-fork match and the fallback.
| #[cfg(debug_assertions)] | ||
| assert_eq!( | ||
| outcome, | ||
| full_pulled_up_checkpoints(state, config)?, |
There was a problem hiding this comment.
Small one: both debug cross-checks use ?. If the full path returns an error, a debug build fails compute_pulled_up_tip while release returns Ok from the incremental path, and since release-fast keeps debug assertions, the tests would run something production doesn't. Turning the full path's error into an assertion failure (expect) rather than an Err of this function keeps the two builds on the same path.
AI security review: 0 finding(s) to checkReviewed by Opus, GPT, Kimi, GLM. No findings to check. Housekeeping (not security): 1Stale docs, missing tests, leftovers and inconsistencies, from a separate cheap model. Not checked by the judge.
Cost of this review: ~$4.76 of a $30 budget (estimated from token counts). |
Main's one-batch signature verification (#738) and this branch both appended a parameter to the same block-processing functions. Each keeps both, `target_delta` before `signatures`; the three `process_operations` that reach eight arguments allow clippy's too_many_arguments, like their siblings. `transition_block` takes main's `Result<BeaconState>` shape and calls the `*_with_options` entry points. `SpeculativeTransition` holds main's `transition` beside this branch's `target_delta`, and `assert_agrees_with` compares both. The delta test wraps `process_attestation` in `checked`, as main's single-operation tests do.
🗒️ Description / Motivation
compute_pulled_up_tipruns on every imported block. It cloned the whole post-state and ranprocess_justification_and_finalizationon the clone, which for altair and later scans all ~2.4M validators (ParticipationTotals::compute) plus an uncached total active balance. Lighthouse makes this O(1) withProgressiveBalancesCache(unslashed participating balances per flag for the previous and current epoch, updated inprocess_attestationand on slashing) and weighs justification on a smallJustificationAndFinalizationStateinstead of a state clone.What Changed
ethlambda-storage:TargetBalanceCache, held byStore: block root -> raw unslashed TIMELY_TARGET balances of its post-state for the previous and current epoch, plus their epoch. Capped (lowest epoch evicted first) and pruned below the finalized epoch.helpers/target_balances.rs:TargetBalanceDelta, carried inBlockProcessingOptions(target_delta) and recorded in every altair+process_attestationwhen a validator newly earns TIMELY_TARGET while unslashed.stf/epoch/justification.rs:JustificationState::{from_state, weigh, write_back}holds only what weighing reads and writes;weigh_justification_and_finalizationdelegates to it, so epoch processing is unchanged.compute_pulled_up_tipuses it (no state clone) with the cached total active balance. Phase0 keeps the old path.Correctness / Behavior Guarantees
Checked against the code: only
process_attestationwrites participation flags within a block; effective balances change only in epoch processing (and the electra upgrade, at an epoch boundary); deposits, consolidation/withdrawal/exit requests and gloas builder operations touch balances, queues or exit epochs only. A slashing changes the unslashed set, so slashing blocks take the full scan.Benchmarks
Offline
ethlambda benchmark import replayon ethlambda-5 (Ryzen 9700X), three mainnet corpora, 2 rounds, steady state n=164, paired per block against main012dc4f49:Per round -15.6% / -16.2%. Epoch-boundary blocks (n=2) -1.2%, by design (full scan there).
Tests Added / Run
the_target_delta_is_what_a_registry_scan_gains(real signed attestations, previous and current epoch, a slashed attester, a repeated vote); tests for the delta,extend_parent_totals, and cache eviction/pruning.cargo test --profile release-fast -p ethlambda-storage --lib(213),-p ethlambda-state-transition --lib(633),-p ethlambda-blockchain --lib(292)fork_choice/206,sanity/595,epoch_processing/804,/attestation/356,/proposer_slashing/235,/attester_slashing/240,/random/208,finality/41fork_choice/under thedevprofile with the equivalence assertion active: 206 passed (948 blocks took the incremental path, 36 the full scan)Notes
nightly, perf(blockchain): run the beacon state transition while the engine verdict is in flight #755 runs the STF speculatively beforeon_block; there the delta must travel inSpeculativeTransitionand be used exactly when the speculative state is (nightly carries that wiring and two tests). This PR, based on main, does not need it, but it will once perf(blockchain): run the beacon state transition while the engine verdict is in flight #755 lands.attestations+1.4 ms per block on nightly.record_new_targetnow takes the effective balance andslashedthat the reward computation already read. Measured on nightly together with perf(beacon): back the participation lists and pending deposits with persistent trees #762's follow-ups: STF 20.7 -> 19.1 ms per block (-1.5 ms, CI [-1.6, -1.4]),attestations5.8 -> 4.3 ms.Related Issues / PRs
✅ Verification Checklist
make fmt— cleanmake lint(clippy with-D warnings) — cleanmake test— not run; the crate lib tests and spec filters above were run instead