Repository navigation
Description
At the first slot of every epoch, the duty loop runs the whole epoch refresh before it proposes. refresh_epoch makes up to twelve requests to the beacon node, one after another, and the slot's block waits for all of them. The block's budget is whatever is left before the attestation offset once they are done.
Code references are at main @ 5d8628e9.
What happens today
The loop wakes at the slot boundary (crates/validator/src/lib.rs:388-390). On an epoch's first slot it runs refresh_epoch (lib.rs:400-452). Only then does it look up the slot's proposer and call propose (lib.rs:459-461). Inside the refresh, in order:
is_optimistic_or_syncing:GET /eth/v1/node/syncing(lib.rs:1005).validator_indices:POST /eth/v1/beacon/states/head/validatorswith every loaded pubkey (lib.rs:1012).prepare_proposers, when any validator has a fee recipient (lib.rs:1037-1045). The fallback sends it to every node at once and waits for all of them (beacon_node/fallback.rs:224,fallback.rs:640-648).- Attester duties for this epoch (
duties.rs:364). - Attester duties for the next epoch (
duties.rs:366). - Proposer duties for this epoch (
duties.rs:386). - At gloas, PTC duties for this epoch and the next: two requests (
lib.rs:1258-1269). - From altair, sync committee duties for this period and the next: two requests (
lib.rs:1302-1310). - When this client sits on a sync committee, one or two subnet subscriptions (
lib.rs:1314-1327).
That is eight requests on a fulu epoch and ten on a gloas one, before sync committee members are counted. The block needs only the proposer duties, sixth in line. The loop cannot skip the refresh altogether. Proposer duties are fetched for the current epoch only (duties.rs:13-16, duties.rs:375-392), so at the boundary the client does not know yet whether it proposes.
Each request is bounded by REQUEST_TIMEOUT, 8 s (beacon_node/http.rs:63). Nothing bounds the refresh as a whole. The proposal's budget is computed after the refresh, from the clock: clock.until_attestation(slot, now) (lib.rs:928).
- Before gloas that is 4 s into a 12 s slot.
- At gloas it is 3 s (
ATTESTATION_DUE_BPS_GLOASis 2500), and the envelope has to fit in the same budget. - A refresh that runs past the attestation offset leaves a budget of zero. The proposal then ends with
ProposalDeadlineas soon as its first request is pending (proposal.rs:171-186).
Some of the refresh's work already runs after the block:
- The attestation subnet subscriptions and the builder registrations are sent from spawned tasks after the proposal (
lib.rs:472-513,lib.rs:987-989). - The sync committee subscriptions are not:
refresh_syncstill sends them inline (lib.rs:1314-1327).
#804, now merged, re-sends the proposer preparations after every slot's duties (lib.rs:546-554). It keeps the refresh's own send in front of the epoch's first proposal on purpose. Its doc says that send "runs on the duty loop ahead of the epoch's first proposal, where skipping would cost that proposal its preparation, and waiting on a hung node would cost it up to that timeout" (lib.rs:1145-1148). So #804 does not shorten the refresh.
Why it matters
- A slow node costs the epoch's first block. A beacon node is at its slowest at the epoch boundary. It runs epoch processing there, and the next epoch's duties need a state advanced past it. On an ethlambda node, attester duties "walk every committee of the epoch, a full shuffle per request on mainnet" (
docs/rpc.md:308-309), and the refresh asks for two epochs of them.validator_indicesgrows with the number of keys (validator: test the client with a large number of validator keys #797). None of this has been measured. At 300 ms each, a gloas epoch's ten requests use up the whole 3 s its first block gets. - A failing refresh repeats every slot. A refresh can fail at the syncing check, the validator lookup or this epoch's attester duties.
last_epochis then not set (lib.rs:427), so the refresh is retried at the start of each following slot (lib.rs:436-439), in front of that slot's attestations. Until one succeeds, the epoch's proposer duties are never fetched, and every proposal in the epoch is skipped. - One unresponsive node multiplies it. The fallback asks the nodes in their configured order on every request, with no memory of which one failed (
beacon_node/fallback.rs:1-5,fallback.rs:111-140).- A first node that accepts connections and never answers costs 8 s per request before the second node is asked. That is about 80 s for a gloas epoch's ten requests, while the loop sits in one refresh.
- An unreachable first node costs
CONNECT_TIMEOUT, 3 s, per request (http.rs:64-78). - When the refresh returns,
next_slot_to_servemoves on to the current slot, and the slots it ran through get no duties (slot_clock.rs:319-344). - A hung node anywhere in the list still costs the refresh 8 s, because the preparation send waits for every node.
- The epoch's first attestations and sync messages wait too. The sleep to the slot's first duty is measured after the refresh and the proposal (
lib.rs:515-520). The comment above the loop says waking at the boundary gives the refresh "a third of a slot of headroom" (lib.rs:384-387). At gloas that is a quarter of a slot, and it is the block's budget as well.
What other clients do
| Client | What stands between an epoch's first slot and its block | Read at |
|---|---|---|
| Lighthouse | Each duty type polls in its own loop, and the block service is fed over a channel. Proposer duties for the current and the next epoch are polled every slot (duties_service.rs:1882). The loop notifies the block service from its cache before any request (duties_service.rs:1862), so the first slot's proposer is known before the boundary. The function's doc still describes the old cost: the first slot's proposal "will need go through the slow path every time ... it is very important to ensure this function is as fast as possible" (duties_service.rs:1839-1843) |
unstable @ 11813cfb |
| Teku | A production call waits only for its own duty type to load (PendingDuties.java:151). From fulu, proposer duties are loaded one epoch ahead (BlockProposalUtilFulu.java:41-42, BlockDutyScheduler.java:57-59). The validator index lookup runs at the epoch's second slot (ValidatorTimingActions.java:70) and prepare_beacon_proposer at its third (BeaconProposerPreparer.java:71) |
master @ 62a54b96 |
| Prysm | Before gloas it is coupled like ethlambda: one runner runs UpdateDuties for the current and the next epoch at the epoch start, before any role, and a failure skips the slot (runner.go:116-124). From gloas it promotes the cached next-epoch duties with no request (duties.go:253-263), and fetches the next epoch "in the background, past the boundary-heavy, reorg-prone first slots" (duties.go:466-474) |
develop @ 0c1e3d24 |
| Nimbus | Each duty type runs its own loop, and each proposer duty spawns a task that waits for its slot (block_service.nim:700-705). Proposer duties are polled for the current and the next epoch (duties_service.nim:657-661), so the first slot's task exists before the boundary |
unstable @ 9bed158a |
| Lodestar | Separate loops, and proposers are notified from cache at the slot start (blockDuties.ts:172). Before gloas, the next epoch's proposers are polled about 1 s before the boundary (BLOCK_DUTIES_LOOKAHEAD_BPS 9167 of the last slot). From gloas they are pre-fetched with v2 (blockDuties.ts:14-18, :29, :174-178) |
unstable @ 77088eb7 |
| Vouch | The first slot's proposal waits for that epoch's proposer duties fetch (proposer.go:43), after a 200 ms pause (service.go:393), on a goroutine of its own (service.go:412). The next epoch's attester and PTC duties are prepared halfway through the epoch (service.go:449-453). prepare_beacon_proposer runs three quarters through it (proposalspreparer.go:35) |
master @ 691f3652 |
| Vero | Awaits the slot's proposal first, then starts the epoch's duty refresh as a task (block_proposal.py:160-198). The first slot's duty comes from the previous epoch's fetch of the current and the next epoch, v2 for proposers (block_proposal.py:236-242) |
master @ 9d110c1d |
File names are relative to each client's validator client directory, except Teku's BlockProposalUtilFulu.java, which is in ethereum/spec.
Lighthouse, Teku (from fulu), Nimbus, Lodestar and Vero know the first slot's proposer before the boundary. Vouch fetches it at the boundary with nothing else in front of it. Prysm before gloas runs its whole duty update first, as ethlambda does, and from gloas it no longer does.
Proposal
The block at an epoch's first slot needs one thing from the refresh: whether one of this client's validators proposes. So:
- Know the next epoch's proposers before the boundary. Since fulu, the state's
proposer_lookaheadfixes an epoch's proposers one epoch ahead. Their dependent root is the last block before the previous epoch, the same as the attester duties' (beacon_node/mod.rs:435-439).- ethlambda's node answers v1 and v2 proposer duties for the next epoch from that lookahead (
docs/rpc.md:296-298). PreferencesServicealready fetches the next epoch's v2 proposer duties every slot at gloas (preferences.rs:341-374).- The "no lookahead" reasoning in
duties.rs:13-16dates from before fulu. - Fetching the next epoch's proposers during the current epoch, and checking their dependent root again at the boundary, leaves the first slot's block with nothing to wait for.
- ethlambda's node answers v1 and v2 proposer duties for the next epoch from that lookahead (
- Run the rest of the refresh after the block. Validator indices, attester duties, PTC duties, sync duties and the sync subscriptions are not needed before the block. The new epoch's attester and PTC schedules are already held from the previous epoch's lookahead. The refresh can run after
propose, or on its own task like the subnet subscriptions. - Bound the refresh as a whole. Give it a deadline, for example the end of its slot, so a slow node cannot hold the duty loop for several slots.
- Ask for proposers first if anything stays in front of the block. Today they come sixth.
To decide
- Pre-fulu networks. Before fulu, the next epoch's proposers are not fixed in advance. Keep the boundary fetch there, with proposer duties first and nothing else ahead of the block? Lodestar polls them about 1 s before the boundary instead.
- Which proposer schedule the loop keeps. The loop keeps v1 duties, and
PreferencesServicefetches v2 for itself (preferences.rs:13-20). After fulu the two lists agree and only their dependent roots differ. One fetch serving both would save requests. - The refresh's preparation send. fix(validator): re-send proposer preparations every slot, not once an epoch #804 keeps it ahead of the first proposal so the node builds for the right address. With fix(validator): re-send proposer preparations every slot, not once an epoch #804, the previous slot's re-send has already told the node about every validator the last refresh resolved. Before the block, only two cases still need the refresh's send:
- a validator this refresh resolves for the first time, such as a key imported mid-epoch;
- a node that restarted since that re-send.
Is that worth up to 8 s of the block's budget against a hung node, or should the send move after the block?
- Where the rest of the refresh runs. Inline after the proposal keeps
DutiesServiceon one task. A spawned task needs the schedule behind a lock or a hand-off, and the loop must never read a half-updated schedule.
Tests
- A mock node whose duty endpoints wait on a gate, as block production already can (
produce_gate,beacon_node/mock.rs:154). With a proposer duty at an epoch's first slot, assert the block is requested before the gated duty requests are released. - With the next epoch's proposer duties prefetched, assert the boundary slot proposes with no request ahead of
produce_block. - A refresh held past the end of its slot is abandoned, and the next slot's duties run on time.
Activity