Skip to content

validator: rank beacon nodes by health and broadcast signed messages #830

Open

Description

@MegaRedHand

With several --beacon-nodes, every request walks the list from the first node, in command-line order, with no memory of how each node did last time. Everything except subscriptions, fee-recipient preparations, builder registrations and proposer preferences goes to the first node that accepts it, blocks and attestations included. A hung first node therefore costs every slot's duties while a healthy node sits behind it, and a block or aggregate reaches the network through one node only.

Code references are at main @ 5d8628e9.

What happens today

  • Order is fixed and nothing is remembered.
    • FallbackBeaconNode keeps the nodes in command-line order. Its module doc says "First healthy wins, in the order given on the command line. Deliberately not health-scored" and names Lighthouse's ranking as the better design (crates/validator/src/beacon_node/fallback.rs:1-5).
    • The order "never changes" because a Producer is a position in it, so a blinded block goes back to the node that produced it (fallback.rs:42-44, used at :401-419).
    • try_each calls the nodes one at a time from position 0 on every call (fallback.rs:80, loop at :121). It keeps no state, so a node that failed the last call is asked first again.
  • What is sent to every node. try_all and try_all_quiet call every node at once (fallback.rs:173, :197, join_all at :224). They are used for:
  • What is sent to one node. Every other call goes through try_each, including every signed message:
    • publish_block (:381) and publish_execution_payload_envelope (:560);
    • submit_attestations (:600) and publish_aggregates (:626);
    • submit_sync_committee_messages (:696) and publish_contribution_and_proofs (:720);
    • submit_payload_attestations (:590).
  • The one health check does not change the order. is_optimistic_or_syncing (fallback.rs:288) runs once an epoch, from refresh_epoch (crates/validator/src/lib.rs:1005). It decides whether duties are refreshed. The next attestation_data call still starts at node 1. HttpBeaconNode reads only is_syncing, is_optimistic and el_offline from /eth/v1/node/syncing (crates/validator/src/beacon_node/http.rs:643-667). It does not parse head_slot or sync_distance (crates/validator/src/beacon_node/dto.rs:138-156).
  • Why publishing is first-wins. publish_block's doc (fallback.rs:368-380) gives two reasons. A node that accepts a block gossips it, so other nodes get it from the network anyway. And a broadcast makes the outcome ambiguous: with answers 200, 202 and a timeout, there is no single answer to report.

Why it matters

  1. A hung first node costs every attestation.
    • Each request has an 8 s timeout (REQUEST_TIMEOUT, http.rs:63). A node that refuses connections fails fast. A node that accepts the connection and does not answer (overloaded, stuck rebuilding a state, swapping) fails only after the full 8 s.
    • The attestation duty starts a third into a 12 s slot, with the rest of the slot as its budget (lib.rs:747-772). attestation_data waits 8 s on node 1, so node 2 is asked when the slot is over, and the duty is abandoned. At gloas the duty starts a quarter in, which leaves node 2 about one second. The comment at lib.rs:749-753 describes the same timeout arithmetic.
    • The next slot starts at node 1 again. The client misses every attestation for as long as node 1 hangs, with node 2 healthy.
    • A proposal is worse off. Its budget ends at the attestation offset, 4 s into the slot (propose, lib.rs:928-930). Node 1's 8 s timeout outlasts it, so the proposal is abandoned before node 2 is ever asked.
  2. A lagging node is used as long as it answers. The Beacon API leaves the is_syncing threshold to each client, so a node a few slots behind can still report false. It answers attestation_data for the requested slot with its own, older head as beacon_block_root. The only check is that the answer is for the requested slot and has a valid index (validate_attestation_data, called at http.rs:741). The client votes for a stale head while node 2 is at the tip. This is what ranking by sync distance avoids.
  3. One node carries each signed message to the network. If the accepting node has few peers or gossips slowly, the block, aggregate or sync contribution reaches the network late. A late block is orphaned. A 202 on publish_block (broadcast without import) also stops the walk, by design.

What other clients do

Client How it picks a node Where signed messages go Read at
Lighthouse Once a slot, 2 s before the next one, checks every node's sync distance, optimistic status and el_offline, and sorts the nodes into health tiers. List order breaks ties. Requests try the nodes in that order (validator_client/beacon_node_fallback/src/lib.rs:143-158, :669; beacon_node_health.rs:9-19, :150-170, :215-260). --broadcast takes attestations, blocks, subscriptions, sync-committee or none (validator_client/src/cli.rs:45-53, beacon_node_fallback/src/lib.rs:896-905). Without it only subscriptions go to every node (validator_client/src/config.rs:141). A broadcast is an error if any node failed (beacon_node_fallback/src/lib.rs:826-850). unstable @ 11813cfb
Teku Checks every node once a slot, at the aggregation deadline: syncing, optimistic, and at least 50 peers. The first endpoint is the primary and is skipped while not ready. Failovers are tried in readiness order (validator/remote/src/main/java/tech/pegasys/teku/validator/remote/BeaconNodeReadinessManager.java:84-88, :104-107, :186-206). Blocks, attestations, aggregates and sync messages go to the primary and every ready failover. On by default; the hidden --Xfailovers-publish-signed-duties-enabled turns it off (FailoverValidatorApiHandler.java:286-340, :485-530; validator/api/.../ValidatorConfig.java:48-49). master @ 62a54b96
Prysm With several --beacon-rest-api-provider URLs, reads query every node and keep "the best suited response" (cmd/validator/flags/flags.go:45-51, api/rest/multi_handler.go:38-41). Every POST goes to all nodes and succeeds when one accepts (api/rest/multi_handler.go:208-210). develop @ 0c1e3d24
Nimbus Every 2 s re-checks each node not in the Synced state, and sends a request only to nodes whose state allows it (beacon_chain/validator_client/fallback_service.nim:39-40, :280-321; validator_client/api.nim:290-300). The default --beacon-node-mode asks every node for attestation data, aggregates, blocks and contributions and picks the best answer (beacon_chain/conf.nim:1135-1139, validator_client/common.nim:366-386). Publishing sends to every online node at once, returns on the first success and lets the others finish (common.nim:366-386, validator_client/api.nim:269-399). unstable @ 9bed158a
Lodestar No sync-distance ranking. Each URL has an error score. While the first URL is healthy only it is asked; after it errors, requests race it and the following URLs until it recovers (packages/api/src/utils/client/httpClient.ts:45-50, :174-200). The same path as reads. unstable @ 77088eb7
Vouch Set per request kind in its strategies config: best asks every node and picks the best answer, majority needs a threshold of nodes to agree, first takes the first answer (docs/configuration.md:144-178, main.go:1397-1408). The multinode submitter (docs/configuration.md:115-118) sends to every node concurrently (services/submitter/multinode/submitattestations.go:49-50). master @ 691f3652
Vero Attestation data needs a threshold of nodes to agree. Other reads take the first good answer from all nodes. A score picks best_beacon_node for single-node calls (src/providers/multi_beacon_node.py:1-30). Attestations, aggregates, blocks, sync messages, subscriptions and preparations go to every node (multi_beacon_node.py:283-300, :536-561, :700-805). master @ 9d110c1d

Every client here except Lodestar either ranks its nodes by their reported state (Lighthouse, Teku, Nimbus) or asks several at once and picks among the answers (Prysm, Nimbus, Vouch, Vero). All but Lodestar can send signed messages to every node, by default or behind a flag; Lodestar does so only while its first node is recovering from errors.

Proposal

  • Keep per-node health state in FallbackBeaconNode.
  • Walk nodes in rank order.
    • Tiers, best first: synced and validated; synced but optimistic or with its execution client offline; syncing or far behind; failing or unreachable. Command-line order breaks ties, so a single healthy node behaves exactly as today.
    • Keep nodes: Vec<B> in command-line order and rank a separate list of positions, so a Producer still names the node that produced a blinded block.
    • A node that timed out drops to the last tier until it answers a probe. A hung node then costs at most one timeout per probe interval.
  • Optionally broadcast signed messages.
    • A flag such as --broadcast <topics> sends the chosen messages to every node at once, like try_all. Topics could be attestations (attestations, aggregates, payload attestations), blocks (blocks and envelopes), sync-committee (messages and contributions) and subscriptions (what is broadcast today).
    • Blinded blocks stay on their producer whatever the flag says (fallback.rs:386-419).
    • For publish_block, a broadcast succeeds when any node accepts the block. Report the best answer: imported (200) over broadcast-only (202) over refused. That answers the ambiguity publish_block's doc raises. The other answers are logged per node, as try_all does now.
    • Signing happens once, before the send, so slashing protection is unaffected.

To decide

  • Default broadcast topics. Subscriptions, preparations, registrations and preferences already go to every node unconditionally; keep that whatever the default is. Whether blocks and attestations go to every node by default (Teku, Prysm, Nimbus, Vouch, Vero) or only when asked (Lighthouse).
  • What a partial broadcast reports. Lighthouse calls a broadcast failed if any node failed. try_all calls it done if any node accepted. The second fits a signed message better, since one accepting node puts it on the network.
  • Probe cadence and tolerance. The point in the slot to probe (it must not compete with the proposal at the boundary; Lighthouse probes 2 s before the next slot), and how many slots of sync_distance still count as synced (Lighthouse: 8, with further tiers at 16 and 64).
  • Envelopes under broadcast. A gloas envelope is cached on the node that produced the block (fallback.rs:545-556). Any node that has the block can validate and gossip the envelope, but one that has not imported the block yet will queue or drop it.
  • Block production. Whether produce_block should go to the best-ranked node only, or ask several nodes and pick the best block as Vouch does. Probably out of scope here.
  • Related open PR. feat(validator): doppelganger protection #807 (doppelganger protection) adds a liveness call through try_each and touches fallback.rs.

Tests

  • A mock node that accepts the request and never answers: with it first and a healthy node second, attestation_data is answered within the slot after the first probe, and the hung node is not asked again until it answers a probe.
  • Ranking: a node reporting sync_distance 5 is asked after a synced node listed behind it. Two healthy nodes keep command-line order.
  • A blinded block produced by a node that is ranked first but listed second goes back to position 1.
  • Broadcast: publish_block with answers 200, 202 and a timeout returns the 200, and logs the other two.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions