Repository navigation
fix(validator): re-send proposer preparations every slot, not once an epoch - #804
Conversation
… epoch A beacon node keeps prepare_beacon_proposer's fee recipients in memory, so a restart forgets them, and until the next call it builds its own payloads for the zero address. The validator client sent them only from the epoch refresh, so a beacon node restarted mid-epoch spent the rest of that epoch paying its local blocks' execution-layer rewards to nobody. The duty loop now re-sends them at the end of every slot (skipping the slot whose refresh just sent them), off the slot loop like the registration sends, as Lighthouse's validator client does. The settings are read on each send, so a keymanager fee-recipient change reaches the node within a slot. Found on a kurtosis devnet: an ethlambda beacon node SIGKILLed 14 slots into an epoch built 6 local blocks for the zero address before the next epoch's refresh (fulu), and 3 more after the same fault on gloas; the validator client logged "Block pays its execution-layer rewards to an address this client did not ask for" for each.
🤖 Gemini Code ReviewThe changes look good. Extracting Automated review by Gemini · gemini-4-argon · custom prompt |
🤖 Claude Code ReviewReview: re-send proposer preparations every slot (PR 804)The PR fixes a real failure. A beacon node restarted mid-epoch builds for the zero address until the next refresh. Factoring out 1. Tasks can pile up if the beacon node hangs (
|
🤖 Grok Code ReviewI'll read the surrounding duty-loop and preparation code so the review is grounded in how this slot path actually runs. Automated review by Grok (xAI) · grok-4.7 · custom prompt |
🤖 Codex Code ReviewNo actionable correctness or security findings in the diff. The per-slot resend reuses the existing preparation logic and sends updated fee recipients to all configured beacon nodes. No consensus processing code changes. Tests could not run because rustup attempted to write to a read-only directory. Automated review by OpenAI Codex · gpt-6.1-sol · custom prompt |
🤖 Kimi Code ReviewFindings
The helper extraction itself looks correct: it preserves the previous empty-preparation behavior, keeps refresh failures non-fatal, and uses Automated review by Kimi (Moonshot AI) · kimi-k3.1-pro · custom prompt |
🤖 DeepSeek Code ReviewOverall this is a clean refactor: extracting A few concerns:
The consensus-facing parts of the PR appear unaffected; this is purely validator fee-recipient preparation behavior. Automated review by DeepSeek · deepseek-v4-pro · custom prompt |
|
The approach looks right and the main flow reads correctly. Two things I'd like fixed before this merges; the rest is non-blocking. Before merge1. A failed per-slot send warns, it does not log at debug. The comment above the spawn says a failure is "only logged at debug", but With one of two nodes down that is a warn every slot instead of once an epoch. With a single node down it is two per slot, the second claiming a missed duty on slots that have none.
2. A key deleted through the keymanager keeps being prepared. The send replays If the key was moved to another client on the same beacon node, the two clients overwrite each other's preparation and the key's block can pay the old client's default address. Before this change nothing was sent for a key after its delete. Filtering Non-blocking
|
…send The per-slot re-send went through FallbackBeaconNode::prepare_beacon_proposer, the loud try_all: with a node down it warned for each refusing node every slot and, with none accepting, claimed a missed duty on slots with no duty. prepare_beacon_proposer_quietly is the same request through try_all_quiet, as submit_proposer_preferences does; its default implementation is the plain call, for a single node.
…ld keys only The per-slot re-send moves into Preparations, whose after_slot is the duty loop's gate: - it sends only the resolved validators the store still holds, so a key deleted through the keymanager is no longer prepared with --suggested-fee-recipient; - it is single-flight (InFlight), so a send that read an older fee recipient cannot land after a newer one; - it goes through prepare_beacon_proposer_quietly and logs its failure at debug. Refreshed carries whether the refresh's own send reached a node, so a failed one is re-sent on its own slot, and its warning no longer says the whole epoch may pay elsewhere. Tests drive the gate with the refresh's real outcome, and cover a deleted key and a re-send still waiting on a slow node. The two that repeated a_validators_own_address_is_registered_for_it and no_configured_address_sends_no_registration are gone.
…ties The keymanager mismatch error, docs/cli.md, and the comments in proposer_settings.rs, preferences.rs and the beacon node's FeeRecipients still described the once-an-epoch cadence. Also says what a DELETE leaves the node with, and when an imported key is first sent. spec_deviations.md gave a signature per proposal as the reason preferences are re-sent less often, but a kept one is re-sent as signed: the difference is one request for every validator against one per preference.
The keymanager's log after a fee recipient is set still said it reaches the beacon node at the next epoch. The mismatch error no longer mentions a validator left with no address, which never reaches that error, and the FeeRecipients comment no longer makes a claim about other validator clients' cadence.
|
Thanks. All of it is in 76d43fc, ec76cf7, 133c4ce and 60650ca, in your order: Before merge
Non-blocking
|
|
/review |
🤖 Gemini Code ReviewThe changes look good. Re-sending Automated review by Gemini · gemini-4-argon · custom prompt |
🤖 Grok Code ReviewI'll inspect the duty-loop integration and the fallback/in-flight helpers this PR depends on before judging correctness. Automated review by Grok (xAI) · grok-4.7 · custom prompt |
🤖 Claude Code ReviewReview: per-slot
|
🤖 Codex Code Review
Automated review by OpenAI Codex · gpt-6.1-sol · custom prompt |
🤖 Kimi Code Review
Otherwise, the change is well structured: deleted-key filtering avoids stale preparations, Automated review by Kimi (Moonshot AI) · kimi-k3.1-pro · custom prompt |
🤖 DeepSeek Code ReviewOverall this is a well-reasoned change and the test coverage for the re-send/skip behavior is good. No consensus-critical or memory-safety issues found in this diff. A few issues worth addressing before merge:
Automated review by DeepSeek · deepseek-v4-pro · custom prompt |
The refresh's own slot skipped the per-slot re-send when the refresh's send reached a node. Two things could then stand past the next slot's proposal: a re-send from the slot before that landed after the refresh's send, carrying the address a keymanager change had replaced, and a change made after the refresh's send. Re-sending after every slot's duties puts the current address back before the next proposal, for one extra request an epoch. Refreshed loses its prepared field. The Preparations doc now says the HTTP client's 8 s request timeout is what keeps a hung node from stopping the re-sends to the healthy ones.
…uest load A change reaches the node a slot later while a slow node holds the last send. docs/cli.md says the addresses go out as one request per slot to each beacon node. The default prepare_beacon_proposer_quietly says why it is the plain call, the mock's preparation_entered notes the permit every call stores, and a paragraph in preferences.rs is re-wrapped.
|
Round three (the automated review of 60650ca). Fixed in e2b58d4 and 2faeed9. Verification:
Fixed
Not changed
Gemini found nothing. Grok posted only its opening line. |
|
/review |
🤖 Gemini Code ReviewThe changes look solid and well-tested. Key highlights:
LGTM! Automated review by Gemini · gemini-4-argon · custom prompt |
🤖 Grok Code ReviewI'll inspect the surrounding validator and beacon-node code so the review is grounded in how preparations, fallback logging, and in-flight gating actually work. Automated review by Grok (xAI) · grok-4.7 · custom prompt |
🤖 Claude Code ReviewReview: per-slot
|
🤖 Codex Code ReviewNo actionable findings. The re-send guard, deleted-key filtering, and fallback broadcasting look sound. No consensus, signature-verification, or SSZ logic changes. Tests could not run: rustup attempted to write to the read-only Automated review by OpenAI Codex · gpt-6.1-sol · custom prompt |
🤖 DeepSeek Code ReviewReviewed the diff. No consensus-layer state transition, fork choice, attestation, or SSZ logic is changed; this is a validator-client fee-recipient resend change. The core design is sound, and the in-flight guard plus tests cover the important races.
Otherwise, the implementation is well-documented and the new tests cover the important skipping and overwrite-recovery behavior. Automated review by DeepSeek · deepseek-v4-pro · custom prompt |
🤖 Kimi Code ReviewNo blocking correctness, security, or performance issues found. The implementation is careful in the important areas:
Residual risk, already documented in Automated review by Kimi (Moonshot AI) · kimi-k3.1-pro · custom prompt |
The keymanager's log after a fee recipient is set, the feerecipient row in docs/cli.md and the proposer_settings.rs module doc said the address goes out after the slot's duties. That holds only for a validator the last epoch refresh resolved; one imported since, or with no index yet, is first sent once a refresh resolves it. The Preparations doc no longer bounds a hung node's cost only for 12 s slots, says a key imported again after a delete goes out with the defaults, and words after_slot's return plainly. The single-flight test asserts the slow re-send is still parked before the gate opens.
|
Round four (the automated review of 2faeed9). Fixed in db66094. Verification:
Fixed
Not changed
Codex, Gemini and Kimi found nothing to change. Grok posted only its opening line. |
|
/review |
🤖 Gemini Code ReviewThis PR looks good. The per-slot Automated review by Gemini · gemini-4-argon · custom prompt |
🤖 Claude Code ReviewReview: per-slot
|
🤖 Codex Code ReviewNo actionable findings. The per-slot resend, deleted-key filtering, and fallback broadcast behavior look sound. This PR does not change consensus processing, signature verification, or SSZ encoding. Tests could not run: rustup attempted to update the toolchain in a read-only directory. Automated review by OpenAI Codex · gpt-6.1-sol · custom prompt |
🤖 Kimi Code ReviewNo blocking issues found. The implementation in
The tests cover the significant cases: all-node failure, deleted keys, refresh-slot resends, fee-recipient changes, and slow in-flight requests. The only residual consideration is operational: request size and beacon-node work now scale as O(resolved validators) per slot, per fallback node. That matches the intended Lighthouse-style behavior, but large validator deployments may benefit from metrics tracking preparation request duration and skipped resends. Automated review by Kimi (Moonshot AI) · kimi-k3.1-pro · custom prompt |
🤖 Grok Code ReviewThe per-slot re-send and the deleted-key filter are the right fix for a restarted beacon node building to the zero address. A few correctness gaps remain.
The quiet fallback path is sound: Automated review by Grok (xAI) · grok-4.7 · custom prompt |
🤖 DeepSeek Code ReviewOverall this is a well-structured change with thorough tests. I have a few substantive comments and some minor clarity issues.
The test coverage is good, especially the cases for deleted keys, same-slot re-send after a late prior send, and the in-flight skip behavior. Automated review by DeepSeek · deepseek-v4-pro · custom prompt |
🗒️ Description / Motivation
A beacon node keeps
prepare_beacon_proposer's fee recipients in memory, and a restart forgets them. Until the next call, it builds its own payloads for the zero address. The validator client sent them only from the epoch refresh, so a beacon node restarted mid-epoch paid the rest of that epoch's locally built blocks to nobody.Found on a kurtosis devnet (ethlambda beacon node + ethlambda validator client, image
sha-f4efc82):What Changed
crates/validator/src/lib.rsPreparations: after every slot's duties, the duty loop re-sends the preparations of the validators the last refresh resolved, for the keys the store still holds, one send at a time (InFlight). Not on the slot whose refresh's send reached a node:Refreshed.preparedcarries that outcomecrates/validator/src/beacon_node/prepare_beacon_proposer_quietly: the same request throughtry_all_quiet, so a node that is down costs adebug!line a slot, not two warningsdocs/cli.md,docs/rpc.md,docs/spec_deviations.md, comments and the mismatch errorDELETEor import does to itCorrectness / Behavior Guarantees
start_proposer_prepare_servicecallsprepare_proposers_and_publishevery slot.DELETEleaving a validator no address sends nothing for it; a key imported mid-epoch is first sent at the next refresh, which is also when its proposer duties are first fetched.Tests Added / Run
the_preparations_go_out_again_after_every_slot_but_the_refreshs: the duty loop's gate, driven by the refresh's real outcome.a_failed_registration_does_not_fail_the_refresh(extended): a failed refresh send is re-sent on its own slot.a_key_deleted_since_the_refresh_is_not_prepared_againa_re_send_still_waiting_on_a_node_makes_the_next_slot_skipthe_quiet_fee_recipient_send_reaches_every_node_too(fallback)cargo fmt --all,cargo clippy --workspace --all-targets -- -D warnings,cargo test --profile release-fast -p ethlambda-validator(622 passed),cargo test --profile release-fast -p ethlambda-rpc.Related Issues / PRs
--fallback-fee-recipientfor the beacon node, the other half of not paying the zero address).✅ Verification Checklist
make fmt— clean-D warningson the workspace — cleanmake test(test-consensusplustest-node, atrelease-fast) —ethlambda-validatorandethlambda-rpconly; CI runs the rest