Skip to content

perf(beacon): keep unslashed target balances incrementally for the pulled-up tip - #761

Open
MegaRedHand wants to merge 18 commits into
mainfrom
perf/progressive-target-balances
Open

MegaRedHand wants to merge 18 commits into
mainfrom
perf/progressive-target-balances

Conversation

@MegaRedHand

@MegaRedHand MegaRedHand commented Nov 2, 2026 •

Copy link
Copy Markdown
Collaborator

🗒️ Description / Motivation

compute_pulled_up_tip runs on every imported block. It cloned the whole post-state and ran process_justification_and_finalization on the clone, which for altair and later scans all ~2.4M validators (ParticipationTotals::compute) plus an uncached total active balance. Lighthouse makes this O(1) with ProgressiveBalancesCache (unslashed participating balances per flag for the previous and current epoch, updated in process_attestation and on slashing) and weighs justification on a small JustificationAndFinalizationState instead of a state clone.

What Changed

  • ethlambda-storage: TargetBalanceCache, held by Store: block root -> raw unslashed TIMELY_TARGET balances of its post-state for the previous and current epoch, plus their epoch. Capped (lowest epoch evicted first) and pruned below the finalized epoch.
  • helpers/target_balances.rs: TargetBalanceDelta, carried in BlockProcessingOptions (target_delta) and recorded in every altair+ process_attestation when a validator newly earns TIMELY_TARGET while unslashed.
  • At import, a block's totals are its parent's cached totals plus the delta when the parent's entry is for the block's own epoch. An epoch boundary, a cache miss, a restart, or a block carrying a slashing does one full scan and caches the result.
  • stf/epoch/justification.rs: JustificationState::{from_state, weigh, write_back} holds only what weighing reads and writes; weigh_justification_and_finalization delegates to it, so epoch processing is unchanged. compute_pulled_up_tip uses it (no state clone) with the cached total active balance. Phase0 keeps the old path.
  • Debug builds run the old full path next to the new one in every pulled-up tip and assert the checkpoints and totals match.

Correctness / Behavior Guarantees

Checked against the code: only process_attestation writes participation flags within a block; effective balances change only in epoch processing (and the electra upgrade, at an epoch boundary); deposits, consolidation/withdrawal/exit requests and gloas builder operations touch balances, queues or exit epochs only. A slashing changes the unslashed set, so slashing blocks take the full scan.

Benchmarks

Offline ethlambda benchmark import replay on ethlambda-5 (Ryzen 9700X), three mainnet corpora, 2 rounds, steady state n=164, paired per block against main 012dc4f49:

main this PR change 95% CI
STF per block 158.1 ms 133.0 ms -25.1 ms (-15.9%) [-25.9, -24.4]
wall (import + fork choice) 179.7 ms 156.4 ms -23.3 ms (-13.0%)

Per round -15.6% / -16.2%. Epoch-boundary blocks (n=2) -1.2%, by design (full scan there).

Tests Added / Run

  • the_target_delta_is_what_a_registry_scan_gains (real signed attestations, previous and current epoch, a slashed attester, a repeated vote); tests for the delta, extend_parent_totals, and cache eviction/pruning.
  • cargo test --profile release-fast -p ethlambda-storage --lib (213), -p ethlambda-state-transition --lib (633), -p ethlambda-blockchain --lib (292)
  • Beacon spec fixtures: fork_choice/ 206, sanity/ 595, epoch_processing/ 804, /attestation/ 356, /proposer_slashing/ 235, /attester_slashing/ 240, /random/ 208, finality/ 41
  • fork_choice/ under the dev profile with the equivalence assertion active: 206 passed (948 blocks took the incremental path, 36 the full scan)

Notes

Related Issues / PRs

  • Part of a set of Lighthouse ports from the same profile, each its own PR.

✅ Verification Checklist

  • Ran make fmt — clean
  • Ran make lint (clippy with -D warnings) — clean
  • Ran make test — not run; the crate lib tests and spec filters above were run instead

…lled-up tip

compute_pulled_up_tip ran on every imported block and, at ~2.4M validators,
cloned the whole post-state (~2.5 ms) and scanned the registry for the
participation totals (~6 ms), plus an uncached total active balance. Lighthouse
avoids both with its ProgressiveBalancesCache.

The per-fork states derive SSZ, so the cache lives in the Store, keyed by block
root (TargetBalanceCache: raw previous/current unslashed TIMELY_TARGET sums,
bounded by capacity and pruned below the finalized epoch). process_attestation
in every fork from altair on records into a TargetBalanceDelta, carried in
BlockProcessingOptions, when a validator newly earns the target flag. A block
in the same epoch as its parent has the parent's totals plus that delta; an
epoch boundary, a cache miss or a block carrying slashings takes one registry
scan, cached for the children.

The justification weighing now runs on a small JustificationState extracted
from the state, so the pulled-up tip clones nothing and reads the total active
balance from the store's ActiveBalanceCache. weigh_justification_and_finalization
delegates to the same logic, so epoch processing is unchanged. Debug builds
also run the old full path and assert the checkpoints and totals agree, which
the fork_choice fixtures exercise on every block.
@MegaRedHand

Copy link
Copy Markdown
Collaborator Author

Combined result for the four Lighthouse ports (#759 pubkey index, #760 column KZG, #761 progressive target balances, #762 tree-backed participation), squashed onto nightly b1fc143 as 080bd40..1d06bf9.

Offline import replay on ethlambda-5, same three mainnet corpora, both legs built at the commit and run with --warm-committee-sums, 2 rounds, steady state n=164, paired per block:

nightly b1fc143 + all four (1d06bf9) change 95% CI
STF per block 140.7 ms 20.7 ms -120.0 ms (-85.3%) [-123.5, -117.3]
wall (import + fork choice) 162.5 ms 43.8 ms -118.7 ms (-73.0%) [-122.1, -116.0]

Per STF step, ordinary blocks (n=162): sync_aggregate 71.4 -> 0.9 ms (#759), execution_requests 5.3 -> 0.0 ms (#759), state_root 24.8 -> 4.0 ms (#762), unattributed 28.0 -> 1.3 ms (#761's removed state clone and scan, #762's cheaper clone). Regressions being fixed: attestations +2.9 ms, process_epoch +9.2 ms at epoch boundaries, get_head +1.0 ms; a per-item attribution run is in progress.

Live on the ethlambda-2 mainnet follower (nightly 1d06bf9): steady stf ~30-40 ms per gossip block (nightly before the ports: ~175 ms).

…nce delta

The progressive target-balance delta recorded every attester that newly earned
TIMELY_TARGET with `state.validator(index)`, a second registry tree descent per
attester on top of the one the reward computation had already made. Offline
import replay at mainnet scale put the `attestations` step at 5.8 ms per block
against 2.8 ms before the tree-backed lists and the delta; this is the delta's
share of that.

The read phase now keeps the effective balance and `slashed` flag it read for
the reward, and `TargetBalanceDelta::record_new_target` takes those two values
instead of a `&Validator`. Nothing in the read phase writes the registry, so
the cached values are the ones a second read would return, and a newly earned
target flag implies the loop reached that flag and read the attester first.
Applies to the altair, deneb, electra (electra and fulu) and gloas versions of
`process_attestation`.
@MegaRedHand

Copy link
Copy Markdown
Collaborator Author

Per-PR attribution on nightly, where the four ports are stacked in the order #759, #760 + #761, #762. Offline import replay on ethlambda-5, three mainnet corpora, --warm-committee-sums, 2 rounds, steady state n=164, paired per block:

nightly at STF per block change vs before the ports 95% CI
b1fc143 (before the ports) 140.1 ms
+ #759 (080bd40) 65.1 ms -75.0 ms (-53.5%) [-78.5, -72.3]
+ #760, #761 (33551eb) 39.9 ms -100.2 ms (-71.5%) [-103.7, -97.5]
+ #762 (1d06bf9) 20.8 ms -119.4 ms (-85.2%) [-122.9, -116.6]
+ follow-ups in #761 and #762 (2b51a33) 19.1 ms

#760 does not run in the replay (no custody columns), so the second step is #761's.

Regressions found by the per-step split, and what the follow-up commits (now pushed to #761 and #762) did about them:

step from added after the follow-ups
attestations, ordinary blocks #761: second validator read per attester +1.4 ms 5.8 -> 4.3 ms together (before the ports: 2.9)
attestations, ordinary blocks #762: per-attester tree read and write +1.5 ms
process_epoch, epoch boundaries #762 +9.1 ms 106.2 -> 104.8 ms (before the ports: 97.2)
get_head the #760 + #761 step +1.3 ms unchanged

get_head's code is untouched by either PR; a perf profile puts the extra time in walk_head's own code, so it looks like codegen rather than work. The same profile shows std SipHash in fork choice's vote tally as most of get_head's ~17 ms per block; that is a separate change, not part of these PRs.

ethlambda-2 (mainnet follower) runs nightly 2b51a33 since 2026-11-03 14:42 UTC.

@MegaRedHand MegaRedHand added performance Performance improvements or possible performance improvements beacon Ethereum Beacon Chain client labels Nov 3, 2026
MegaRedHand added a commit that referenced this pull request Nov 4, 2026
get_head tallied every validator's latest message on every call, about 1M
on mainnet, and a live import pays it twice for the first timely block of a
slot (on_block's boost-gate walk, then the actor's get_head). That tally was
most of get_head's ~17 ms in the eth-5 replay.

The store now keeps the vote weights in a node array (Lighthouse unstable's
proto-array layout: one node per block, carrying its PENDING, EMPTY and FULL
weights). One attestation is one vote batch: Store::record_latest_messages
records the messages and moves the attesters' weight under one lock, and the
next read refreshes the subtree totals in one reverse pass over the blocks.
The head walk reads them through cached_node_weights. The proposer boost,
gloas's previous-slot rule and the walk bound depend on the clock, so they
stay read-time. There are no best-child pointers: under gloas the choice
moves without any vote, which is why Lighthouse stopped maintaining them.

What the cache cannot follow incrementally (finality pruning, a live-window
removal, a restart, a new justified balances snapshot, the gloas rule switch,
a block stored without its node) marks it stale, and the next read rebuilds it
from the latest messages at the old tally's cost, about once per epoch. A
rebuild rescans LiveChain under the lock, so an API read that scanned before a
removal cannot rebuild the removed blocks back in.

compute_node_weights stays, as the reference. Equivalence is checked by a
proptest against a naive tally (with the incremental state also equal to a
fresh rebuild after every operation), on the random trees at every step of
every seed, by a new test that feeds votes, blocks and equivocations through
update_latest_messages, on_block's insertion order and
insert_equivocating_index, and at every head check of every fork-choice
fixture.

Local benchmark (1M validators, 73 blocks, laptop): the tally 27 ms; a warm
cached read 7 us; a read after a 15.6k-vote batch 49 us; the batch itself
1.3 ms; a rebuild 17 ms.

Cherry-picked from perf/beacon-proto-array at b4adbaf. Adapted onto
nightly: both vote writers (set_latest_message and the new
record_latest_messages) grow the dense vote table through one
grow_latest_messages helper, which keeps #740's exact growth plus headroom;
the storage and fork-choice imports are merged with #759's and #761's caches.
Replaces #767, reverted in the previous commit.
@MegaRedHand
MegaRedHand marked this pull request as ready for review November 17, 2026 18:52
@github-actions

Copy link
Copy Markdown

🤖 Grok Code Review

I'll trace the incremental target-balance path against justification and attestation processing, since a mismatch there would change fork choice. I need the surrounding justification and participation code to check the invariants.I'll check how justification skips early epochs and how participation totals treat slashed validators, so the incremental path can be compared to the spec.


Automated review by Grok (xAI) · grok-4.7 · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Codex Code Review

No actionable findings. The incremental target-balance logic correctly falls back to full scans across epoch boundaries, cache misses, and slashings. The justification/finalization refactor preserves the existing rules.

Tests could not run because Rustup attempted to write to the read-only /home/runner/.rustup directory.


Automated review by OpenAI Codex · gpt-6.1-sol · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Claude Code Review

Review: incremental target balances for the pulled-up tip (PR 761)

The design is sound and well documented. Extracting JustificationState gives epoch processing and the fork-choice tip one shared weighing implementation, which removes a class of drift bugs. The debug-mode cross-checks against the full path are a good safety net. My main concern is what happens in release builds, plus a few smaller items.

Correctness / consensus risk

  1. Release builds get no check on the incremental path (fork_choice.rs, pulled_up_checkpoints).

    • Both assert_eq!s are #[cfg(debug_assertions)]. In release, a missed write path silently skews unrealized justification. Such a path could be a future fork adding a flag writer, a new slashing route, or an effective-balance write. That skew changes fork choice (pulled-up tips, update_unrealized_checkpoints), which can cause a head divergence from the network.
    • Suggestion: occasionally re-verify in release, either sampled (e.g. 1 in N blocks) or at each epoch's first tracked block. On mismatch, log error!, bump a metric, and fall back to the scanned value. The cost is negligible compared to the risk.
  2. Gloas parent state vs. cache key.

    • The cache entry for parent_root comes from the parent block's post-state, as passed to compute_pulled_up_tip.
    • In Gloas, the child may build on the parent's post-payload (full) state. The module docs argue envelope processing never touches flags, slashed or effective balances, so the totals coincide. That holds today, but it is an implicit cross-fork invariant.
    • Please add an explicit sentence in helpers/target_balances.rs about full vs. empty parent states. Also confirm a fork-choice fixture with full-parent children exercises this path under debug asserts.
  3. block_has_slashings returns true for Lean (target_balances.rs, last match arm).

    • The comment says Lean "never carries one", but returning true means "force a full scan".
    • It is harmless, but it reads as a contradiction. Either use lean_block_unreachable() (matching the rest of the crate) or fix the comment to say the arm deliberately falls back to the scan.
  4. The weigh_justification_and_finalization write-back on error (justification.rs) is a subtle behavior change.

    • The old code left whatever was written before the failing get_block_root. The new code writes back the partially weighed struct, which is equivalent because weigh mutates fields in the same order.
    • The ? on block_root(...) sits between field updates, so this equivalence depends on that ordering. A short note in weigh would keep a future reorder from silently changing semantics. In practice, callers discard the state on error anyway.

Rust / API

  1. TargetBalanceDelta::record_new_target is a non-atomic read-modify-write (load then store).

    • It is correct under the stated single-writer assumption, but the type is Sync and nothing enforces that assumption.
    • fetch_update(Relaxed, Relaxed, |v| Some(v.saturating_add(eb))) costs essentially nothing and removes the footgun. Alternatively, fetch_add: overflow is impossible given total ETH supply.
  2. BlockProcessingOptions<'a> still derives PartialEq, Eq (stf/block.rs).

    • Equality now compares the delta's contents through the hand-written PartialEq, which is surprising for a "what to skip" options struct. The custom PartialEq/Eq on TargetBalanceDelta exists only to satisfy this derive.
    • Consider dropping PartialEq/Eq from the options if nothing needs them.
    • The doc line "The default … is the only value block import uses" is now stale, since on_block sets target_delta.
  3. The *_with_options wrappers (stf/mod.rs) add two more too_many_arguments functions. This is acceptable, but state_transition and apply_block could just take options directly; there are few call sites. Optional.

Storage

  1. Pruning uses SLOTS_PER_EPOCH (store.rs, the prune_below(finalized.slot / SLOTS_PER_EPOCH) call).

    • Make sure this is the active preset's constant (minimal = 8) and not a hard-coded mainnet value. Otherwise, on minimal-preset fixtures the prune epoch is 4× too low. That only costs memory, but it is worth confirming.
  2. Eviction scans every entry under the mutex (TargetBalanceCache::insert).

    • It is O(4096) only when full, so it is fine compared with the registry scan it saves.
    • Lowest-epoch eviction can evict a still-needed same-epoch sibling during a long non-finality period. This is a perf-only consequence, and the doc already says so.

Tests

  • The the_target_delta_is_what_a_registry_scan_gains test covers the important invariants: a slashed attester is excluded, repeated votes count once, and previous and current epochs are both handled.
  • There is no unit test exercising JustificationState::weigh against the state-based path outside fork-choice fixtures. A small table test over bit patterns and checkpoint epochs would pin down the four finalization rules independently of the debug asserts.

Minor

  • pulled_up_checkpoints in debug builds now runs a full scan plus a state clone per block, so debug spec-test runtimes will rise. That is acceptable, but worth noting.
  • TargetBalanceDelta is collected even when crosses_with_empty_slots/epoch-crossing makes it unusable. This is harmless, but you could pass None in that branch to make the intent explicit.

Overall this is a good change. I'd like to see Item 1 (release-mode safety net) addressed before merge, given that the failure mode is silent fork-choice divergence.


Automated review by Claude · claude-opus-5-5 · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Gemini Code Review

This PR looks great. The design of TargetBalanceCache, TargetBalanceDelta, and JustificationState is clean and well-reasoned:

  • Consensus invariants: Within a single epoch (when no epoch boundary is crossed in process_slots) and in the absence of block slashings (block_has_slashings), active validator membership and effective balances are invariant during block processing, and only process_attestation sets new TIMELY_TARGET participation flags. Falling back to a full scan across epoch boundaries, on cache misses, in epochs <= GENESIS_EPOCH + 1, in Phase0, or when a block contains slashings preserves exact equivalence with the spec.
  • State-free pulled-up tip: Extracting JustificationState avoids cloning BeaconState on every on_block while sharing the exact weighing logic with epoch processing, and the #[cfg(debug_assertions)] cross-checks in pulled_up_checkpoints against compute_target_balances and full_pulled_up_checkpoints provide strong continuous verification across test fixtures.
  • Storage & concurrency: Using AtomicU64 in TargetBalanceDelta keeps BlockProcessingOptions Copy + Send + Sync across async proposal paths, and bounding TargetBalanceCache by both capacity (evicting the oldest epoch) and finalization pruning keeps memory usage bounded.

No issues found — LGTM!


Automated review by Gemini · gemini-4-argon · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 DeepSeek Code Review

The PR is well-structured and the idea of caching unslashed timely-target balances to avoid a registry scan is sound. However, I found a few correctness issues that should be addressed before merge.

  1. crates/blockchain/state_transition/src/beacon/fork_choice.rs, function full_pulled_up_checkpoints

    let mut state = state.clone();

    Here state is &BeaconState, so this clones the reference, not the inner BeaconState. This likely fails to compile because stf::epoch::process_justification_and_finalization(&mut state, ...) then expects &mut BeaconState but receives &mut &BeaconState.

    Use:

    let mut state = (*state).clone();
  2. crates/blockchain/state_transition/src/beacon/helpers/target_balances.rs, tracks_epoch

    pub fn tracks_epoch(epoch: Epoch) -> bool {
        epoch > crate::beacon::constants::GENESIS_EPOCH + 1
    }

    This skips current epoch 1 (assuming GENESIS_EPOCH == 0). However, process_justification_and_finalization runs for a state at epoch 1 and can justify epoch 0, and possibly even finalize the genesis checkpoint via the old_current_justified_checkpoint.epoch + 1 == current_epoch finalization rule.

    Because pulled_up_checkpoints gates weigh behind tracks_epoch, blocks in epoch 1 will return the original post-state checkpoints instead of the pulled-up checkpoints in release builds. That is fork-choice-visible. The debug assertion comparing against full_pulled_up_checkpoints would fail for such cases if tests cover epoch 1 with participating attestations.

    The predicate should likely be:

    epoch > GENESIS_EPOCH

    or the special case should be removed entirely. The module comment claiming “justification does not run there” for the first two epochs is incorrect.

  3. crates/blockchain/state_transition/src/beacon/helpers/target_balances.rs, block_has_slashings

    // Not a Beacon Chain block: nothing here ever carries one.
    SignedBeaconBlock::Lean(_) => true,

    The comment says Lean blocks never carry slashings, but the code returns true. This forces a full scan if compute_pulled_up_tip is ever called for a Lean block, and compute_target_balances will likely fail on a Lean state because it has no participation flags. Confirm whether on_block/compute_pulled_up_tip can ever see SignedBeaconBlock::Lean; if not, this is only misleading, but if it can, this returns the wrong value and should be false or the target-balance logic should be gated to Beacon chain only.

  4. Minor: TargetBalanceDelta::record_new_target uses saturating_add.

    These are consensus-critical totals. A saturating add masks overflow if assumptions about max validator count or effective balance change. A checked_add with proper error propagation would be more robust, even though current Ethereum parameters make overflow practically impossible.


Automated review by DeepSeek · deepseek-v4-pro · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Kimi Code Review

Overall, this looks correct and well-designed. The incremental totals preserve the raw-sum/floor distinction, newly earned target flags are counted only once, slashing-containing blocks fall back to a full scan, and epoch changes prevent invalid parent extensions. The JustificationState refactor also appears equivalent to the previous in-place implementation, including partial write-back on errors.

Two non-blocking robustness concerns:

  1. Saturating arithmetic can diverge from the full-scan path
    crates/blockchain/state_transition/src/beacon/helpers/target_balances.rs:91-99 and :121-127 use saturating_add. The registry scan uses checked arithmetic and returns an error on overflow, while the incremental path silently clamps and continues with potentially different justification results. This is unlikely with realistic chain balances, but consensus code should avoid two overflow behaviors. Prefer checked addition with an overflow flag/error that forces the full-scan path or propagates the failure.

  2. Delta accumulation assumes single-writer access without enforcing it
    target_balances.rs:96-100 performs an atomic load followed by a store. This is correct for the current sequential attestation processing, but concurrent writers would silently lose updates despite TargetBalanceDelta being shareable across threads. Consider fetch_update, which can also support checked-overflow handling, or otherwise make the single-writer invariant harder to violate.

Test coverage is good for delta accounting and cache eviction. An additional integration test explicitly exercising a slashing block’s full-scan fallback and an epoch-boundary cache miss would help protect the two most important fallback conditions.


Automated review by Kimi (Moonshot AI) · kimi-k3.1-pro · custom prompt

…r, and test the slashing fallback

on_block chose between the parent's cached totals and a full scan inline:
a block that slashes must scan, since the delta does not see a validator
leaving the unslashed set. No fork choice fixture imports a block that
carries a slashing (they arrive as their own steps), so the debug-build
equivalence check never ran that branch. totals_from_parent now holds the
whole decision, and a_block_that_slashes_is_not_extended_from_its_parent
checks it.

block_has_slashings answered true for a lean block under a comment saying
lean blocks carry none. It now panics through lean_block_unreachable, like
every other beacon accessor, and is private to the helper.
record_new_target loaded and stored separately, which is correct only
while a single thread writes. The type is Sync and nothing enforces that,
so each addition is now one fetch_update. Still saturating, like the
registry scan's own sums.
…hat block import sets

target_balances: a gloas block's pre-state is its parent block's
post-state on either payload branch, since an envelope is only verified
and a FULL parent's execution requests run inside the child's own block.
So the parent's cached entry is the right base for both.

BlockProcessingOptions said the default is the only value block import
uses; import now sets target_delta.
@MegaRedHand

Copy link
Copy Markdown
Collaborator Author

Round one (the automated review of 713e1fd). Fixed in ce1331c, 4271e8f and cadc844. make fmt, make lint and cargo clippy -p ethlambda-state-transition --all-targets --features beacon-spec-tests -- -D warnings are clean. Tests are left to CI.

Fixed

  • Kimi, a test for the slashing fallback: no fork-choice fixture imports a block that carries a slashing (they arrive as their own steps), so the debug-build equivalence check never ran that branch. The choice between the parent's cached totals and a full scan is now one helper, totals_from_parent, which on_block calls. a_block_that_slashes_is_not_extended_from_its_parent checks that a block carrying a proposer slashing is not extended, while the same block without one is.
  • Claude 3, DeepSeek 3, the lean arm of block_has_slashings: it answered true under a comment saying lean blocks carry no slashing. It now panics through lean_block_unreachable, like every other beacon accessor, and is private to the helper.
  • Claude 5, Kimi 2, load then store: record_new_target now adds with one fetch_update, so a second writer could not lose an addition. Still saturating, like the registry scan.
  • Claude 2, gloas FULL parents: documented in target_balances. In this gloas there is no separate FULL state: an envelope is only verified, and a FULL parent's execution requests run inside the child's own block (process_parent_execution_payload), where they set no flag, effective balance or slashed. So the parent's cached entry is the right base on either branch. The gloas fork-choice fixtures with FULL children run under the debug equivalence check.
  • Claude 6, the stale doc: BlockProcessingOptions no longer says the default is the only value block import uses; import sets target_delta. I kept the derive: the delta's PartialEq is also what the speculation's agreement check compares.

Not changed

  • Claude 1, a release-build check: any skew cannot outlive an epoch, since an entry is extended only within its own epoch and every epoch's first block already scans in full. The realized checkpoints come from epoch processing's own scan. In tests every pulled-up tip asserts the incremental path agrees with the full one.
  • Claude 4, write-back ordering: weigh assigns in the spec's order, the order the old code wrote in, and its wrapper's doc already says it writes back on an error. Callers discard a failed transition's state.
  • Claude 7, the _with_options wrappers: kept; optional.
  • Claude 8, SLOTS_PER_EPOCH: it is the compiled preset's (preset::SLOTS_PER_EPOCH, 8 on minimal).
  • Claude 9 and the minor notes: agreed they are fine as they are.
  • Claude, a table test for weigh: it is the body of weigh_justification_and_finalization, so the justification_and_finalization fixtures run it on every fork.
  • DeepSeek 1, state.clone(): on a &BeaconState it calls BeaconState::clone and returns an owned state; it compiles.
  • DeepSeek 2, tracks_epoch and epoch 1: the spec skips justification while current_epoch <= GENESIS_EPOCH + 1, and tracks_epoch mirrors that guard.
  • DeepSeek 4, Kimi 1, saturating vs checked: the registry scan saturates too, so there is one overflow behaviour. Neither can reach u64::MAX Gwei, over a hundred times the whole supply.
  • Kimi, the epoch-boundary miss: covered by a_parent_is_extended_only_within_its_own_tracked_epoch.

Codex and Gemini found nothing to change, and Grok's comment ended after its preamble with no findings.

@MegaRedHand

Copy link
Copy Markdown
Collaborator Author

/review

@github-actions

Copy link
Copy Markdown

🤖 Grok Code Review

I'll inspect the justification and participation paths this PR depends on so the incremental totals stay consistent with the spec.I need the early-epoch justification skip and how participation flags are applied, because those decide whether the incremental totals are safe.The flag-update timing relative to the delta is the risky part; I'll read that next.


Automated review by Grok (xAI) · grok-4.7 · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Codex Code Review

No actionable findings. The incremental target balances correctly fall back to a full scan at epoch boundaries, on cache misses, and for blocks containing slashings. Speculative imports preserve the collected delta, and the justification refactor retains the existing rules.

Tests could not run: the pinned toolchain was unavailable, and the offline fallback lacked the required libssz dependency.


Automated review by OpenAI Codex · gpt-6.1-sol · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Claude Code Review

Review: incremental target balances for compute_pulled_up_tip (PR 761)

This is a well-reasoned change. The module docs in target_balances.rs lay out the invariant clearly: within an epoch, a validator's target contribution changes only through a newly set flag, an effective-balance change at a boundary, or a slashing. The debug-build cross-checks against the full scan are a good safety net for the fixtures.

I checked the main correctness paths and found no bugs:

  • Epoch keying. TargetBalances.epoch is the parent post-state's current epoch, and extension requires parent.epoch == block_epoch. So any process_slots boundary forces a rescan.
  • tracks_epoch cut-off. epoch > GENESIS_EPOCH + 1 matches the spec's early return in process_justification_and_finalization. It also sidesteps the genesis case where the previous and current participation lists are aliased.
  • Slashed attesters.
    • Already-slashed attesters are excluded at record time.
    • Blocks that slash anyone fall back to a full scan.
    • A child of a slashing block extends from that block's freshly scanned entry, so the slashing is accounted for.
  • Speculation. Each path collects into its own TargetBalanceDelta: the speculative transition, the debug fresh_delta, and the non-speculative path. Nothing is double-counted.
  • weigh_justification_and_finalization on error. It writes back exactly the partially updated fields the old in-place version would have left behind.

Findings

  1. Release builds trust the incremental totals forever (fork_choice.rs, pulled_up_checkpoints).

    • Outside debug builds, nothing ever re-checks the extended totals against a scan.
    • Suppose a future fork, or an overlooked path, sets a flag, changes an effective balance, or slashes outside the three enumerated cases. The error would then:
      • silently propagate block-to-block until the next epoch boundary;
      • skew unrealized_justifications, and through them, head selection.
    • Consider a cheap release-mode tripwire, for example:
      • verify with a full scan once per epoch, or on every Nth extension;
      • emit a metric or error! on mismatch, then fall back to the scanned value.
  2. The invariant is enforced only by convention (target_balances.rs, module docs, item 1 of that list).

    • block_has_slashings is an exhaustive match, so a new fork variant forces an update there. Good.
    • Nothing similar forces a new fork's process_attestation (or any new flag-writing operation) to call record_new_target.
    • Suggestions:
      • a test that, per fork, runs a block with attestations and compares before + delta against a rescan, generalizing the altair-only the_target_delta_is_what_a_registry_scan_gains;
      • or a comment at each flag-write site pointing back to this module.
  3. write_back dirties every field unconditionally (justification.rs, JustificationState::write_back).

    • It always writes all three checkpoints and all four bits, even when unchanged.
    • If the state is tree-backed with cached hashes, this may invalidate nodes that previously stayed clean, for example finalized_checkpoint when nothing finalizes.
    • The cost is small (once per epoch), but writing only fields that differ from from_state would keep the old cost profile.
  4. BlockProcessingOptions equality is surprising (stf/block.rs).

    • PartialEq/Eq is derived, and TargetBalanceDelta: PartialEq compares sums.
    • So two options carrying different accumulators compare equal when their sums match.
    • If equality on options is used anywhere meaningful, prefer std::ptr::eq on the reference. Otherwise consider dropping Eq from TargetBalanceDelta (it exists only for the debug assertion) and comparing (previous(), current()) explicitly there.
  5. Eviction is O(n) under the mutex while finality stalls (target_balance_cache.rs, insert).

    • While the table is full, every insert scans all 4096 entries for the minimum epoch.
    • That is negligible today. If the capacity grows, a BTreeMap<Epoch, Vec<Root>> side index would make eviction cheap.
  6. Debug builds scan twice when there is no parent entry (pulled_up_checkpoints).

    • When extended_totals is None, the registry is scanned for totals and again for the assert_eq!.
    • Skip the assertion in that branch, since it compares the scan with itself.
  7. Test naming (stf/altair.rs, the_target_delta_is_what_a_registry_scan_gains).

    • slashed_share is actually the unslashed committee share; rename it, e.g. unslashed_share.
    • The assertion also assumes every validator has MAX_EFFECTIVE_BALANCE. Summing the committee's effective balances, minus the slashed member, would make the test robust to fixture changes.
  8. Minor: block_has_slashings repeats the same check per variant. If a body accessor exists (or is cheap to add), something like block.body().proposer_slashings_len() would collapse the eight arms. This is optional.

None of these block merging; Item 1 is the one I'd most like to see addressed, given the fork-choice impact of a silent divergence.


Automated review by Claude · claude-opus-5-5 · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Gemini Code Review

The implementation looks solid and well-reasoned:

  • Consensus invariants: Within a single epoch, the active validator set for previous_epoch and current_epoch and their effective balances are invariant (exits, deposits, consolidations, and withdrawal requests only affect future epochs or epoch-boundary processing), and blocks with proposer or attester slashings fall back to a full registry scan via block_has_slashings.
  • Attestation flag accumulation: TargetBalanceDelta::record_new_target is only invoked when TIMELY_TARGET_FLAG_INDEX transitions from unset to set for an attester, and participation updates are committed per attestation so duplicate attestations within the same block are not double-counted.
  • State-free pulled-up tip: Extracting JustificationState avoids the full BeaconState clone in compute_pulled_up_tip for Altair and later forks while preserving exact parity with weigh_justification_and_finalization (verified by debug assertions against the full-clone path).
  • Cache lifecycle: TargetBalanceCache is properly bounded by capacity and pruned at finalization.

No issues found.


Automated review by Gemini · gemini-4-argon · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Kimi Code Review

No blocking correctness or security issues found. The incremental calculation appropriately keeps totals unfloored, excludes slashed attesters, and falls back to a full registry scan at epoch boundaries, cache misses, and blocks containing slashings. Retaining the full phase0 path and adding debug differential assertions against the specification path are particularly good safeguards.

Non-blocking observations:

  • Test coverage: stf/altair.rs adds a strong delta-versus-registry-scan test, but equivalent recording logic is duplicated in deneb.rs, electra.rs, and gloas.rs (around lines 438, 805, and 1478). An omitted or incorrectly conditioned record in one fork would silently undercount in release builds. Consider adding direct coverage for Electra and especially Gloas, whose attestation/payment flow differs.
  • Bounded eviction cost: storage/src/target_balance_cache.rs:75-88 scans all 4,096 entries to find the lowest epoch whenever a full cache receives a new root. This is bounded and vastly cheaper than a registry rescan, so it is not a blocker, but during prolonged non-finality it occurs on every import. Epoch buckets or batch eviction could avoid the repeated linear scan if profiling shows it matters.

Automated review by Kimi (Moonshot AI) · kimi-k3.1-pro · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 DeepSeek Code Review

  • Compile blocker: BlockProcessingOptions is now BlockProcessingOptions<'a>, but several signatures still use bare BlockProcessingOptions, which now requires a lifetime argument and will not compile. See transition_block in crates/blockchain/state_transition/src/beacon/fork_choice.rs around line 1087, and state_transition_with_options / apply_block_with_options in crates/blockchain/state_transition/src/beacon/stf/mod.rs around lines 145 and 174. These should be BlockProcessingOptions<'_>. The same likely applies to any existing &BlockProcessingOptions signatures in the fork stf files that were not updated in this diff.

  • Performance nit: in the hot attestation loops in altair.rs, deneb.rs, electra.rs (and gloas.rs already used this shape), the hoisted per-validator value now stores only (effective_balance, slashed) and recomputes effective_balance / EFFECTIVE_BALANCE_INCREMENT once per newly earned flag. The previous code hoisted the division result itself (attester_increments), which the comment in electra.rs explicitly calls out as important at mainnet scale. Consider storing (increments, effective_balance, slashed) together so the division happens once per validator.

  • Check the or-pattern in block_has_slashings in crates/blockchain/state_transition/src/beacon/helpers/target_balances.rs:
    SignedBeaconBlock::Electra(b) | SignedBeaconBlock::Fulu(b).
    This only compiles if both variants bind the same payload type. If Fulu has its own block type, this needs to be split into separate match arms.

  • The incremental balance logic itself looks consistent: the delta only records newly set TIMELY_TARGET flags, skips slashed validators, full-scans on slashings or epoch-crossing, and the extracted JustificationState preserves the previous partial-write-on-error behavior. The storage cache is bounded and pruning is keyed correctly by epoch.


Automated review by DeepSeek · deepseek-v4-pro · custom prompt

Keeping the effective balance and slashed for the target-balance delta
moved the division into the flag loop of altair's, deneb's and electra's
process_attestation, so an attester earning three flags paid it three
times, where main hoisted it. The kept tuple now carries the increments
too, divided on the validator's first read, beside the two fields the
delta reads. Gloas, which divided per flag on main as well, gets the
same shape. The arithmetic is unchanged, so the rewards are
bit-identical.
the_target_delta_is_what_a_registry_scan_gains named the unslashed share
slashed_share and assumed every attester held MAX_EFFECTIVE_BALANCE. It
is now unslashed_share, summed from the committee's own effective
balances less the slashed member's.
…ug builds

pulled_up_checkpoints compared its totals with a full scan in debug
builds even when it had just scanned for them, which compared a scan
with itself and doubled the registry passes of every fork-choice fixture
block without a cached parent. The assertion now runs only for totals
extended from the parent's.
@MegaRedHand

MegaRedHand commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator Author

Round two (the automated review of cadc844, judged against 99aa236, which merged main after it). Fixed in fbff48d, f5716b1 and 89381bd. make fmt, make lint and cargo clippy -p ethlambda-state-transition --all-targets --features beacon-spec-tests -- -D warnings are clean. Tests are left to CI.

Fixed

  • DeepSeek 2, the increments division: keeping the effective balance and slashed for the delta had moved the / EFFECTIVE_BALANCE_INCREMENT into the flag loop of altair's, deneb's and electra's process_attestation. An attester earning three flags paid it three times, where main divided once. The kept tuple now carries the increments too, divided on the validator's first read. Gloas, which divided per flag on main as well, gets the same shape. Same arithmetic, so the rewards are bit-identical.
  • Claude 6, the debug double scan: with no cached parent, pulled_up_checkpoints scanned for its totals and then asserted them against a second scan, comparing the scan with itself. The assertion now runs only for totals extended from the parent's.
  • Claude 7, the delta test: slashed_share is now unslashed_share. It is summed from the committee's own effective balances less the slashed member's, rather than assuming MAX_EFFECTIVE_BALANCE for each.

Not changed

  • Claude 1, a release-build tripwire: settled in round one; a skew cannot outlive an epoch.
  • Claude 2, Kimi 1, per-fork coverage of the delta: every fork's process_attestation records it, and in tests every extended total is asserted equal to a full scan. Whether every fork's fork-choice fixtures reach that check with a non-zero delta (an in-epoch block adding a target vote on a cached parent) is not counted yet; if electra or gloas has no such block, this becomes a missing-test point. A unit test per fork would need electra and gloas attestation builders that do not exist yet, and would repeat that check.
  • Claude 3, write_back: fork choice never calls it. It weighs on the extracted JustificationState. Epoch processing writes back once per epoch into plain container fields, which carry no cached hash to invalidate.
  • Claude 4, options equality: settled in round one; the delta's PartialEq is what the speculation's agreement check compares.
  • Claude 5, Kimi 2, eviction cost: settled in round one.
  • Claude 8, collapsing block_has_slashings: no body accessor spans the forks; the per-variant match stays exhaustive.
  • DeepSeek 1, bare BlockProcessingOptions: an elided lifetime in a type path compiles; make lint builds every target.
  • DeepSeek 3, Electra(b) | Fulu(b): both variants wrap electra::SignedBeaconBlock.

Codex and Gemini found nothing to change, and Grok's comment ended after its preamble with no findings.

@MegaRedHand

Copy link
Copy Markdown
Collaborator Author

/review

@pablodeymo pablodeymo left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I read this together with #762, at 89381bd. The exactness argument holds as far as I can check: every fork's process_attestation feeds the delta, effective balances only move at epoch boundaries (where this falls back to the scan), blocks with slashings take the scan, and in gloas the parent's execution requests are applied inside the child's own transition. Merged with #762, the two touch the same attestation loops and don't conflict, and each attester is read once. Two points inline, neither blocking.


/// Whether the block carries an operation that slashes a validator; see the
/// module documentation for why such a block is not extended incrementally.
fn block_has_slashings(block: &SignedBeaconBlock) -> bool {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The incremental totals are right only while two things stay true: process_attestation is the only place a participation flag is set mid-epoch, and a block-body proposer or attester slashing is the only way a validator is slashed. I checked every slash_validator caller today (the shared proposer slashing, gloas's own, attester slashings), and this match covers them all.

My worry is the next fork. A new slashing path, or anything that changes an effective balance mid-epoch, would have release builds extend stale totals into a wrong pulled-up tip with no error, and only debug builds would notice. Could the slashing case be recorded in slash_validator instead, the way Lighthouse's on_slashing does? That would also remove this per-fork match and the fallback.

#[cfg(debug_assertions)]
assert_eq!(
outcome,
full_pulled_up_checkpoints(state, config)?,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small one: both debug cross-checks use ?. If the full path returns an error, a debug build fails compute_pulled_up_tip while release returns Ok from the incremental path, and since release-fast keeps debug assertions, the tests would run something production doesn't. Turning the full path's error into an assertion failure (expect) rather than an Err of this function keeps the two builds on the same path.

@github-actions

Copy link
Copy Markdown

AI security review: 0 finding(s) to check

Reviewed by Opus, GPT, Kimi, GLM.
DeepSeek (stopped) did not finish; Opus, GPT, Kimi, GLM reviewed the whole PR. Comment /review continue to retry, with a new budget; reviewers that finished are reused.
Malicious-code check (tests, build and install scripts, hooks, editor tasks, dependency sources): nothing found.

No findings to check.

Housekeeping (not security): 1

Stale docs, missing tests, leftovers and inconsistencies, from a separate cheap model. Not checked by the judge.

Kind Item Where Fix
stale-docs process_justification_and_finalization doc still describes the old clone-and-advance path crates/blockchain/state_transition/src/beacon/stf/epoch/mod.rs:113 Update the doc comment to describe the new architecture: for altair+ compute_pulled_up_tip delegates to pulled_up_checkpoints, which uses JustificationState::weigh without cloning the state; only Phase0 falls through to process_justification_and_finalization via full_pulled_up_checkpoints.

Cost of this review: ~$4.76 of a $30 budget (estimated from token counts).

@pablodeymo
pablodeymo added this pull request to the merge queue Nov 20, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to a conflict with the base branch Nov 20, 2026
Main's one-batch signature verification (#738) and this branch both
appended a parameter to the same block-processing functions. Each keeps
both, `target_delta` before `signatures`; the three `process_operations`
that reach eight arguments allow clippy's too_many_arguments, like their
siblings.

`transition_block` takes main's `Result<BeaconState>` shape and calls the
`*_with_options` entry points. `SpeculativeTransition` holds main's
`transition` beside this branch's `target_delta`, and `assert_agrees_with`
compares both. The delta test wraps `process_attestation` in `checked`,
as main's single-operation tests do.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

beacon Ethereum Beacon Chain client performance Performance improvements or possible performance improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants