Repository navigation
Description
With several --beacon-nodes, every request walks the list from the first node, in command-line order, with no memory of how each node did last time. Everything except subscriptions, fee-recipient preparations, builder registrations and proposer preferences goes to the first node that accepts it, blocks and attestations included. A hung first node therefore costs every slot's duties while a healthy node sits behind it, and a block or aggregate reaches the network through one node only.
Code references are at main @ 5d8628e9.
What happens today
- Order is fixed and nothing is remembered.
FallbackBeaconNodekeeps the nodes in command-line order. Its module doc says "First healthy wins, in the order given on the command line. Deliberately not health-scored" and names Lighthouse's ranking as the better design (crates/validator/src/beacon_node/fallback.rs:1-5).- The order "never changes" because a
Produceris a position in it, so a blinded block goes back to the node that produced it (fallback.rs:42-44, used at:401-419). try_eachcalls the nodes one at a time from position 0 on every call (fallback.rs:80, loop at:121). It keeps no state, so a node that failed the last call is asked first again.
- What is sent to every node.
try_allandtry_all_quietcall every node at once (fallback.rs:173,:197,join_allat:224). They are used for:register_validators(:424) andregister_validators_in_batches(:444, its ownjoin_all);submit_proposer_preferences(:508);prepare_beacon_proposer(:640) and the per-slotprepare_beacon_proposer_quietly(:653, from fix(validator): re-send proposer preparations every slot, not once an epoch #804);subscribe_committees(:670) andsubscribe_sync_committees(:733).
- What is sent to one node. Every other call goes through
try_each, including every signed message:publish_block(:381) andpublish_execution_payload_envelope(:560);submit_attestations(:600) andpublish_aggregates(:626);submit_sync_committee_messages(:696) andpublish_contribution_and_proofs(:720);submit_payload_attestations(:590).
- The one health check does not change the order.
is_optimistic_or_syncing(fallback.rs:288) runs once an epoch, fromrefresh_epoch(crates/validator/src/lib.rs:1005). It decides whether duties are refreshed. The nextattestation_datacall still starts at node 1.HttpBeaconNodereads onlyis_syncing,is_optimisticandel_offlinefrom/eth/v1/node/syncing(crates/validator/src/beacon_node/http.rs:643-667). It does not parsehead_slotorsync_distance(crates/validator/src/beacon_node/dto.rs:138-156). - Why publishing is first-wins.
publish_block's doc (fallback.rs:368-380) gives two reasons. A node that accepts a block gossips it, so other nodes get it from the network anyway. And a broadcast makes the outcome ambiguous: with answers 200, 202 and a timeout, there is no single answer to report.
Why it matters
- A hung first node costs every attestation.
- Each request has an 8 s timeout (
REQUEST_TIMEOUT,http.rs:63). A node that refuses connections fails fast. A node that accepts the connection and does not answer (overloaded, stuck rebuilding a state, swapping) fails only after the full 8 s. - The attestation duty starts a third into a 12 s slot, with the rest of the slot as its budget (
lib.rs:747-772).attestation_datawaits 8 s on node 1, so node 2 is asked when the slot is over, and the duty is abandoned. At gloas the duty starts a quarter in, which leaves node 2 about one second. The comment atlib.rs:749-753describes the same timeout arithmetic. - The next slot starts at node 1 again. The client misses every attestation for as long as node 1 hangs, with node 2 healthy.
- A proposal is worse off. Its budget ends at the attestation offset, 4 s into the slot (
propose,lib.rs:928-930). Node 1's 8 s timeout outlasts it, so the proposal is abandoned before node 2 is ever asked.
- Each request has an 8 s timeout (
- A lagging node is used as long as it answers. The Beacon API leaves the
is_syncingthreshold to each client, so a node a few slots behind can still reportfalse. It answersattestation_datafor the requested slot with its own, older head asbeacon_block_root. The only check is that the answer is for the requested slot and has a valid index (validate_attestation_data, called athttp.rs:741). The client votes for a stale head while node 2 is at the tip. This is what ranking by sync distance avoids. - One node carries each signed message to the network. If the accepting node has few peers or gossips slowly, the block, aggregate or sync contribution reaches the network late. A late block is orphaned. A 202 on
publish_block(broadcast without import) also stops the walk, by design.
What other clients do
| Client | How it picks a node | Where signed messages go | Read at |
|---|---|---|---|
| Lighthouse | Once a slot, 2 s before the next one, checks every node's sync distance, optimistic status and el_offline, and sorts the nodes into health tiers. List order breaks ties. Requests try the nodes in that order (validator_client/beacon_node_fallback/src/lib.rs:143-158, :669; beacon_node_health.rs:9-19, :150-170, :215-260). |
--broadcast takes attestations, blocks, subscriptions, sync-committee or none (validator_client/src/cli.rs:45-53, beacon_node_fallback/src/lib.rs:896-905). Without it only subscriptions go to every node (validator_client/src/config.rs:141). A broadcast is an error if any node failed (beacon_node_fallback/src/lib.rs:826-850). |
unstable @ 11813cfb |
| Teku | Checks every node once a slot, at the aggregation deadline: syncing, optimistic, and at least 50 peers. The first endpoint is the primary and is skipped while not ready. Failovers are tried in readiness order (validator/remote/src/main/java/tech/pegasys/teku/validator/remote/BeaconNodeReadinessManager.java:84-88, :104-107, :186-206). |
Blocks, attestations, aggregates and sync messages go to the primary and every ready failover. On by default; the hidden --Xfailovers-publish-signed-duties-enabled turns it off (FailoverValidatorApiHandler.java:286-340, :485-530; validator/api/.../ValidatorConfig.java:48-49). |
master @ 62a54b96 |
| Prysm | With several --beacon-rest-api-provider URLs, reads query every node and keep "the best suited response" (cmd/validator/flags/flags.go:45-51, api/rest/multi_handler.go:38-41). |
Every POST goes to all nodes and succeeds when one accepts (api/rest/multi_handler.go:208-210). |
develop @ 0c1e3d24 |
| Nimbus | Every 2 s re-checks each node not in the Synced state, and sends a request only to nodes whose state allows it (beacon_chain/validator_client/fallback_service.nim:39-40, :280-321; validator_client/api.nim:290-300). The default --beacon-node-mode asks every node for attestation data, aggregates, blocks and contributions and picks the best answer (beacon_chain/conf.nim:1135-1139, validator_client/common.nim:366-386). |
Publishing sends to every online node at once, returns on the first success and lets the others finish (common.nim:366-386, validator_client/api.nim:269-399). |
unstable @ 9bed158a |
| Lodestar | No sync-distance ranking. Each URL has an error score. While the first URL is healthy only it is asked; after it errors, requests race it and the following URLs until it recovers (packages/api/src/utils/client/httpClient.ts:45-50, :174-200). |
The same path as reads. | unstable @ 77088eb7 |
| Vouch | Set per request kind in its strategies config: best asks every node and picks the best answer, majority needs a threshold of nodes to agree, first takes the first answer (docs/configuration.md:144-178, main.go:1397-1408). |
The multinode submitter (docs/configuration.md:115-118) sends to every node concurrently (services/submitter/multinode/submitattestations.go:49-50). |
master @ 691f3652 |
| Vero | Attestation data needs a threshold of nodes to agree. Other reads take the first good answer from all nodes. A score picks best_beacon_node for single-node calls (src/providers/multi_beacon_node.py:1-30). |
Attestations, aggregates, blocks, sync messages, subscriptions and preparations go to every node (multi_beacon_node.py:283-300, :536-561, :700-805). |
master @ 9d110c1d |
Every client here except Lodestar either ranks its nodes by their reported state (Lighthouse, Teku, Nimbus) or asks several at once and picks among the answers (Prysm, Nimbus, Vouch, Vero). All but Lodestar can send signed messages to every node, by default or behind a flag; Lodestar does so only while its first node is recovering from errors.
Proposal
- Keep per-node health state in
FallbackBeaconNode.- Probe every node once a slot, in parallel, from
/eth/v1/node/syncing:is_syncing,is_optimistic,el_offline,head_slot,sync_distance. - Also record the outcome of the last real request to each node, so a timeout lowers its rank before the next probe.
- The same state can serve validator: make /health report whether the client can do its duties #833 and the per-node config check in validator: check every beacon node is on the same chain, and follow config changes #831.
- Probe every node once a slot, in parallel, from
- Walk nodes in rank order.
- Tiers, best first: synced and validated; synced but optimistic or with its execution client offline; syncing or far behind; failing or unreachable. Command-line order breaks ties, so a single healthy node behaves exactly as today.
- Keep
nodes: Vec<B>in command-line order and rank a separate list of positions, so aProducerstill names the node that produced a blinded block. - A node that timed out drops to the last tier until it answers a probe. A hung node then costs at most one timeout per probe interval.
- Optionally broadcast signed messages.
- A flag such as
--broadcast <topics>sends the chosen messages to every node at once, liketry_all. Topics could beattestations(attestations, aggregates, payload attestations),blocks(blocks and envelopes),sync-committee(messages and contributions) andsubscriptions(what is broadcast today). - Blinded blocks stay on their producer whatever the flag says (
fallback.rs:386-419). - For
publish_block, a broadcast succeeds when any node accepts the block. Report the best answer: imported (200) over broadcast-only (202) over refused. That answers the ambiguitypublish_block's doc raises. The other answers are logged per node, astry_alldoes now. - Signing happens once, before the send, so slashing protection is unaffected.
- A flag such as
To decide
- Default broadcast topics. Subscriptions, preparations, registrations and preferences already go to every node unconditionally; keep that whatever the default is. Whether blocks and attestations go to every node by default (Teku, Prysm, Nimbus, Vouch, Vero) or only when asked (Lighthouse).
- What a partial broadcast reports. Lighthouse calls a broadcast failed if any node failed.
try_allcalls it done if any node accepted. The second fits a signed message better, since one accepting node puts it on the network. - Probe cadence and tolerance. The point in the slot to probe (it must not compete with the proposal at the boundary; Lighthouse probes 2 s before the next slot), and how many slots of
sync_distancestill count as synced (Lighthouse: 8, with further tiers at 16 and 64). - Envelopes under broadcast. A gloas envelope is cached on the node that produced the block (
fallback.rs:545-556). Any node that has the block can validate and gossip the envelope, but one that has not imported the block yet will queue or drop it. - Block production. Whether
produce_blockshould go to the best-ranked node only, or ask several nodes and pick the best block as Vouch does. Probably out of scope here. - Related open PR. feat(validator): doppelganger protection #807 (doppelganger protection) adds a liveness call through
try_eachand touchesfallback.rs.
Tests
- A mock node that accepts the request and never answers: with it first and a healthy node second,
attestation_datais answered within the slot after the first probe, and the hung node is not asked again until it answers a probe. - Ranking: a node reporting
sync_distance5 is asked after a synced node listed behind it. Two healthy nodes keep command-line order. - A blinded block produced by a node that is ranked first but listed second goes back to position 1.
- Broadcast:
publish_blockwith answers 200, 202 and a timeout returns the 200, and logs the other two.
Activity