Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
8529fe0
perf(beacon): back the participation lists and pending deposits with …
MegaRedHand Nov 2, 2026
6de4728
feat(ssz-tree): add a blind set, a sorted reader and a shared-node re…
MegaRedHand Nov 3, 2026
099d568
test(types): add ignored benchmarks for the attestation pass and the …
MegaRedHand Nov 3, 2026
d08f731
perf(beacon): write participation flags blindly in process_attestation
MegaRedHand Nov 3, 2026
c67f224
perf(beacon): read an attestation's participation flags with a sorted…
MegaRedHand Nov 3, 2026
fb7b208
perf(beacon): build the rotated participation list by sharing one zer…
MegaRedHand Nov 3, 2026
ef181c7
Merge main into perf/tree-backed-participation
MegaRedHand Nov 4, 2026
0960567
Merge main into perf/tree-backed-participation
MegaRedHand Nov 11, 2026
16813b0
Merge main into perf/tree-backed-participation
MegaRedHand Nov 18, 2026
57f9e89
Merge main into perf/tree-backed-participation
MegaRedHand Nov 18, 2026
4608ddf
Merge main into perf/tree-backed-participation
MegaRedHand Nov 18, 2026
3a939ad
Merge main into perf/tree-backed-participation
MegaRedHand Nov 19, 2026
6ce4b76
Merge main into perf/tree-backed-participation
MegaRedHand Nov 20, 2026
ca69a1d
fix(ssz-tree): keep a progressive sorted reader right after a step back
MegaRedHand Nov 19, 2026
46cc758
docs(beacon): name the check that makes the sorted reader and blind w…
MegaRedHand Nov 19, 2026
74690a0
refactor(types): match gloas first in rebase_participation_on, and do…
MegaRedHand Nov 19, 2026
109b8f8
Merge remote-tracking branch 'origin/perf/tree-backed-participation' …
MegaRedHand Nov 20, 2026
0428fb8
docs(beacon): describe the tree-backed participation lists and pendin…
MegaRedHand Nov 20, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -671,6 +671,7 @@ existing once per chain.
`[patch.crates-io]` override because nothing outside this workspace depends on
libssz, so there is no second copy to unify; that also keeps the manifest free
of a `[patch]` table, which `shadow/cargo-patch.toml` would collide with.
- **The participation lists and `pending_deposits` are tree-backed too** (`EpochParticipation`, `PendingDeposits`; progressive from gloas), reached through `altair_validator_lists` (`ParticipationRef`, no slice) and `epoch_participation_mut`; scan with `ParticipationRef::iter`, not `get(i)`.
- **`Validators` and `Balances` are `ethlambda_ssz_tree::List`s** (gloas:
`ethlambda_ssz_tree::ProgressiveList`, same method set), persistent
Merkle trees that cache node hashes and share unchanged subtrees between
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1069,9 +1069,8 @@ mod tests {
let BeaconState::Fulu(inner) = &mut state else {
unreachable!("built as fulu")
};
for flags in inner.current_epoch_participation.iter_mut() {
*flags = 0b111;
}
let count = inner.current_epoch_participation.len();
inner.current_epoch_participation = vec![0b111; count].try_into().unwrap();
assert!(pack_attestations(&state, vec![candidate]).is_empty());
}

Expand Down
53 changes: 28 additions & 25 deletions crates/blockchain/state_transition/src/beacon/helpers/electra.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1039,19 +1039,22 @@ fn placeholder_pending_deposit(validator: &Validator, amount: Gwei) -> electra::
/// participation and inactivity entries
/// ([`Self::push_empty_participation_and_inactivity`]), and the
/// three pending queues (`pending_deposits`, `pending_partial_withdrawals`,
/// `pending_consolidations`), both as element-level slice views (`len()`,
/// `iter()`/`get()` through the returned `&[T]`, one-at-a-time `push_*`) and
/// as a whole (`take_*`/`set_*`, for epoch processing's own whole-queue
/// drains).
/// `pending_consolidations`): one-at-a-time `push_*`, the whole queue
/// (`take_*`/`set_*`, for epoch processing's own whole-queue drains), and,
/// for the last two, element-level slice views (`len()`, `iter()`/`get()`
/// through the returned `&[T]`).
///
/// Electra's, fulu's, and gloas's `pending_deposits` (and the other two
/// queues) share one element type each but not one container: electra's and
/// fulu's is the bounded `SszList` (`electra::PendingDeposits` and friends),
/// gloas's is EIP-7688's unbounded [`libssz_types::ProgressiveList`]
/// (`crate::beacon::containers::gloas::PendingDeposits` and friends). Both
/// deref to `[T]` with the same element type, which is what lets the
/// slice-view and element-`push_*` methods below serve all three forks
/// through one match rather than needing a gloas copy of every caller.
/// Electra's, fulu's, and gloas's queues share one element type each but not
/// one container. `pending_deposits` is tree-backed: an
/// `ethlambda_ssz_tree::List` on electra and fulu
/// (`electra::PendingDeposits`), an `ethlambda_ssz_tree::ProgressiveList` on
/// gloas (`crate::beacon::containers::gloas::PendingDeposits`). Neither has a
/// slice, so it is reached only through `push_pending_deposit` and
/// `take_pending_deposits`/`set_pending_deposits`. The other two are the
/// bounded `SszList` on electra and fulu and EIP-7688's unbounded
/// [`libssz_types::ProgressiveList`] on gloas, which both deref to `[T]`, so
/// their slice views serve all three forks. Either way one match per method
/// covers every fork, rather than a gloas copy of every caller.
///
/// Epoch processing's whole-queue drains and the block-level request
/// processors (`crate::beacon::stf::electra`'s `process_deposit_request`
Expand Down Expand Up @@ -1106,14 +1109,14 @@ impl<'a> PendingQueueFields<'a> {

/// Queues one more [`electra::PendingDeposit`], pushed by
/// [`queue_excess_active_balance`], [`queue_entire_balance_and_reset_validator`],
/// and the deposit-request processors. Infallible on gloas's unbounded
/// queue; electra's and fulu's bounded one can still reject a length over
/// its own SSZ limit.
/// and the deposit-request processors. Electra's and fulu's bounded queue
/// can reject a length over its own SSZ limit; gloas's progressive one
/// never does.
pub(crate) fn push_pending_deposit(&mut self, deposit: electra::PendingDeposit) -> Result<()> {
match self {
PendingQueueFields::Electra(state) => state.pending_deposits.push(deposit)?,
PendingQueueFields::Fulu(state) => state.pending_deposits.push(deposit)?,
PendingQueueFields::Gloas(state) => state.pending_deposits.push(deposit),
PendingQueueFields::Gloas(state) => state.pending_deposits.push(deposit)?,
}
Ok(())
}
Expand All @@ -1129,19 +1132,19 @@ impl<'a> PendingQueueFields<'a> {
pub(crate) fn take_pending_deposits(&mut self) -> Vec<electra::PendingDeposit> {
match self {
PendingQueueFields::Electra(state) => {
core::mem::take(&mut state.pending_deposits).into_inner()
core::mem::take(&mut state.pending_deposits).to_vec()

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The epoch code takes the whole queue out here, and set_pending_deposits builds a fresh tree from the Vec (single_pass.rs:535, stf/epoch/electra.rs:540). So every epoch boundary copies every pending deposit and discards all the cached hashes, even when no deposit was applied or postponed. When remaining is the queue unchanged, leaving the list in place would keep its root. to_vec() here also clones each deposit where into_inner() used to move it.

Could this be part of the process_epoch regression?

}
PendingQueueFields::Fulu(state) => {
core::mem::take(&mut state.pending_deposits).into_inner()
core::mem::take(&mut state.pending_deposits).to_vec()
}
PendingQueueFields::Gloas(state) => {
core::mem::take(&mut state.pending_deposits).into_inner()
core::mem::take(&mut state.pending_deposits).to_vec()
}
}
}

/// See [`Self::take_pending_deposits`]. Fallible only for electra's and
/// fulu's bounded `SszList`; gloas's progressive one never rejects a
/// fulu's bounded list; gloas's progressive one never rejects a
/// length.
pub(crate) fn set_pending_deposits(
&mut self,
Expand Down Expand Up @@ -1266,9 +1269,9 @@ impl<'a> PendingQueueFields<'a> {
/// and `inactivity_scores` by one all-zero entry each, keeping them
/// exactly as long as the registry after a new validator is appended.
///
/// Gloas's own three lists are progressive (`libssz_types::ProgressiveList`),
/// so their `push` is infallible, unlike electra's and fulu's bounded
/// `SszList`.
/// Gloas's own lists are progressive, so their pushes cannot hit a length
/// limit; its inactivity scores are still a flat
/// `libssz_types::ProgressiveList`, whose `push` is infallible outright.
pub(crate) fn push_empty_participation_and_inactivity(&mut self) -> Result<()> {
match self {
PendingQueueFields::Electra(state) => {
Expand All @@ -1282,8 +1285,8 @@ impl<'a> PendingQueueFields<'a> {
state.inactivity_scores.push(0)?;
}
PendingQueueFields::Gloas(state) => {
state.previous_epoch_participation.push(0);
state.current_epoch_participation.push(0);
state.previous_epoch_participation.push(0)?;
state.current_epoch_participation.push(0)?;
state.inactivity_scores.push(0);
}
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,10 @@
//! the specification's helpers for that one question at a time reads the
//! tree-backed registry many times over and, per active validator, once more
//! by index (a tree descent each). Here the registry is walked once, with
//! `validators().iter()` zipped against the participation slices, and every
//! later step runs over flat data.
//! `iter_validators()` in lockstep with both participation lists' iterators
//! (through [`ParticipationRef`](crate::beacon::containers::ParticipationRef),
//! since the lists are tree-backed), and every later step runs over flat
//! data.
//!
//! The pieces, from cheapest to most complete:
//!
Expand Down Expand Up @@ -134,7 +136,16 @@ fn scan(
let mut previous_error = None;
let mut current_error = None;

// Walked in lockstep with the registry, not indexed per validator: each
// `get` on a tree-backed list is a descent, which at mainnet size was the
// largest cost of this pass. A list shorter than the registry yields
// `None` from the index it runs out at, the same as `get` did.
let mut previous_flags_iter = previous_list.iter();
let mut current_flags_iter = current_list.iter();

for (index, validator) in state.iter_validators().enumerate() {
let previous_entry = previous_flags_iter.next();
let current_entry = current_flags_iter.next();
let effective_balance = validator.effective_balance;
let active_now = is_active_validator(validator, current_epoch);
let active_before = is_active_validator(validator, previous_epoch);
Expand All @@ -145,7 +156,7 @@ fn scan(

let mut bits = 0u8;
if active_before {
match previous_list.get(index) {
match previous_entry {
Some(&participation) => {
if !validator.slashed {
for (flag_index, total) in previous_flags.iter_mut().enumerate() {
Expand All @@ -172,7 +183,7 @@ fn scan(
}

if current_target && active_now {
match current_list.get(index) {
match current_entry {
Some(&participation) => {
if has_flag(participation, constants::TIMELY_TARGET_FLAG_INDEX)
&& !validator.slashed
Expand Down
39 changes: 24 additions & 15 deletions crates/blockchain/state_transition/src/beacon/stf/altair.rs
Original file line number Diff line number Diff line change
Expand Up @@ -174,15 +174,18 @@ pub fn process_attestation(

let mut proposer_reward_numerator: Gwei = 0;
let mut updates: Vec<(ValidatorIndex, ParticipationFlags)> = Vec::new();
// `is_valid_indexed_attestation` above refused any `attesting_indices` that
// is not strictly ascending, so the reader walks each leaf once rather than
// descending the tree for every attester.
let mut reader = epoch_participation.sorted_reader();
for index in attesting_indices {
let current_flags =
epoch_participation
.get(index as usize)
.copied()
.ok_or(Error::IndexOutOfBounds {
index: index as usize,
len: epoch_participation.len(),
})?;
let current_flags = reader
.get(index as usize)
.copied()
.ok_or(Error::IndexOutOfBounds {
index: index as usize,
len: epoch_participation.len(),
})?;

let mut new_flags: ParticipationFlags = 0;
// The attester's validator is read at most once, on the first flag
Expand Down Expand Up @@ -219,7 +222,7 @@ pub fn process_attestation(
.ok_or(Error::ArithmeticOverflow("proposer_reward_numerator"))?;
}
if new_flags != 0 {
updates.push((index, new_flags));
updates.push((index, current_flags | new_flags));
}
}

Expand All @@ -234,14 +237,20 @@ pub fn process_attestation(
previous_epoch_participation
};
let epoch_participation_len = epoch_participation.len();
for (index, new_flags) in updates {
let flags = epoch_participation
.get_mut(index as usize)
.ok_or(Error::IndexOutOfBounds {
// A blind write of `current_flags | new_flags`, which the read phase
// stored, rather than `|=` on the stored value. That is exact:
// `is_valid_indexed_attestation` above refused a repeated index, so each
// index is written once, and nothing writes this list between the read
// phase and here, so the stored value is still `current_flags`. It saves
// the tree descent `get_mut` makes to copy the old value into the write
// buffer.
for (index, flags) in updates {
if !epoch_participation.set(index as usize, flags) {
return Err(Error::IndexOutOfBounds {
index: index as usize,
len: epoch_participation_len,
})?;
*flags |= new_flags;
});
}
}

// The weights partition `WEIGHT_DENOMINATOR` between the three flags, the
Expand Down
39 changes: 24 additions & 15 deletions crates/blockchain/state_transition/src/beacon/stf/deneb.rs
Original file line number Diff line number Diff line change
Expand Up @@ -397,15 +397,18 @@ pub fn process_attestation(

let mut proposer_reward_numerator: Gwei = 0;
let mut updates: Vec<(ValidatorIndex, ParticipationFlags)> = Vec::new();
// `is_valid_indexed_attestation` above refused any `attesting_indices` that
// is not strictly ascending, so the reader walks each leaf once rather than
// descending the tree for every attester.
let mut reader = epoch_participation.sorted_reader();
for index in attesting_indices {
let current_flags =
epoch_participation
.get(index as usize)
.copied()
.ok_or(Error::IndexOutOfBounds {
index: index as usize,
len: epoch_participation.len(),
})?;
let current_flags = reader
.get(index as usize)
.copied()
.ok_or(Error::IndexOutOfBounds {
index: index as usize,
len: epoch_participation.len(),
})?;

let mut new_flags: ParticipationFlags = 0;
// The attester's validator is read at most once, on the first flag
Expand Down Expand Up @@ -442,7 +445,7 @@ pub fn process_attestation(
.ok_or(Error::ArithmeticOverflow("proposer_reward_numerator"))?;
}
if new_flags != 0 {
updates.push((index, new_flags));
updates.push((index, current_flags | new_flags));
}
}

Expand All @@ -454,14 +457,20 @@ pub fn process_attestation(
previous_participation
};
let epoch_participation_len = epoch_participation.len();
for (index, new_flags) in updates {
let flags = epoch_participation
.get_mut(index as usize)
.ok_or(Error::IndexOutOfBounds {
// A blind write of `current_flags | new_flags`, which the read phase
// stored, rather than `|=` on the stored value. That is exact:
// `is_valid_indexed_attestation` above refused a repeated index, so each
// index is written once, and nothing writes this list between the read
// phase and here, so the stored value is still `current_flags`. It saves
// the tree descent `get_mut` makes to copy the old value into the write
// buffer.
for (index, flags) in updates {
if !epoch_participation.set(index as usize, flags) {
return Err(Error::IndexOutOfBounds {
index: index as usize,
len: epoch_participation_len,
})?;
*flags |= new_flags;
});
}
}

const NON_PROPOSER_WEIGHT: u64 = constants::WEIGHT_DENOMINATOR - constants::PROPOSER_WEIGHT;
Expand Down
44 changes: 26 additions & 18 deletions crates/blockchain/state_transition/src/beacon/stf/electra.rs
Original file line number Diff line number Diff line change
Expand Up @@ -763,15 +763,18 @@ pub fn process_attestation(

let mut proposer_reward_numerator: Gwei = 0;
let mut updates: Vec<(ValidatorIndex, ParticipationFlags)> = Vec::new();
// `is_valid_indexed_attestation` above refused any `attesting_indices` that
// is not strictly ascending, so the reader walks each leaf once rather than
// descending the tree for every attester.
let mut reader = participation.sorted_reader();
for index in attesting_indices {
let current_flags =
participation
.get(index as usize)
.copied()
.ok_or(Error::IndexOutOfBounds {
index: index as usize,
len: participation.len(),
})?;
let current_flags = reader
.get(index as usize)
.copied()
.ok_or(Error::IndexOutOfBounds {
index: index as usize,
len: participation.len(),
})?;

let mut new_flags: ParticipationFlags = 0;
// The attester's validator is read at most once, on the first flag
Expand Down Expand Up @@ -806,7 +809,7 @@ pub fn process_attestation(
.ok_or(Error::ArithmeticOverflow("proposer_reward_numerator"))?;
}
if new_flags != 0 {
updates.push((index, new_flags));
updates.push((index, current_flags | new_flags));
}
}

Expand All @@ -817,15 +820,20 @@ pub fn process_attestation(
let mut fields = block_mut(state, "process_attestation")?;
let participation_mut = fields.epoch_participation_mut(current_epoch_target);
let participation_len = participation_mut.len();
for (index, new_flags) in updates {
let flags =
participation_mut
.get_mut(index as usize)
.ok_or(Error::IndexOutOfBounds {
index: index as usize,
len: participation_len,
})?;
*flags |= new_flags;
// A blind write of `current_flags | new_flags`, which the read phase
// stored, rather than `|=` on the stored value. That is exact:
// `is_valid_indexed_attestation` above refused a repeated index, so
// each index is written once, and nothing writes this list between the
// read phase and here, so the stored value is still `current_flags`. It
// saves the tree descent `get_mut` makes to copy the old value into the
// write buffer.
for (index, flags) in updates {
if !participation_mut.set(index as usize, flags) {
return Err(Error::IndexOutOfBounds {
index: index as usize,
len: participation_len,
});
}
}
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@

use crate::beacon::config::Config;
use crate::beacon::constants;
use crate::beacon::containers::BeaconState;
use crate::beacon::containers::{BeaconState, EpochParticipation};
use crate::beacon::error::{Error, Result};
use crate::beacon::helpers::accessors::get_current_epoch;
use crate::beacon::helpers::altair::get_next_sync_committee;
Expand Down Expand Up @@ -400,7 +400,10 @@ pub fn process_participation_flag_updates(state: &mut BeaconState) -> Result<()>
state.altair_validator_lists_mut()?;

*previous_epoch_participation = core::mem::take(current_epoch_participation);
*current_epoch_participation = vec![0; validator_count].try_into()?;
// `repeat` rather than a zeroed `Vec`: the `Vec` builds hundreds of
// separate leaves that the next state root then hashes one by one, while
// `repeat` shares one zero leaf and its cached hash among all of them.
*current_epoch_participation = EpochParticipation::repeat(0, validator_count)?;

Ok(())
}
Expand Down Expand Up @@ -498,7 +501,8 @@ mod tests {
// none has yet this epoch: only the previous epoch should justify.
{
let (previous_epoch_participation, _, _) = state.altair_validator_lists_mut().unwrap();
for flags in previous_epoch_participation.iter_mut() {
for index in 0..previous_epoch_participation.len() {
let flags = &mut previous_epoch_participation[index];
*flags = add_flag(*flags, constants::TIMELY_TARGET_FLAG_INDEX);
}
}
Expand Down
Loading
Loading