Skip to content

validator: detect a slashed validator, warn loudly, and optionally stop #836

Open

Description

@MegaRedHand

The validator client never notices that one of its keys has been slashed. It keeps signing for that key, says nothing, and keeps running every other key under the same setup that got the first one slashed. The data that would tell it is already fetched every epoch.

Code references are at bf66e86c.

What happens today

  • refresh_epoch resolves every loaded key once an epoch through POST /eth/v1/beacon/states/head/validators (crates/validator/src/lib.rs:986, crates/validator/src/beacon_node/http.rs:670-698). The response carries each validator's status and validator.slashed, as the DTO test body shows (crates/validator/src/beacon_node/dto.rs:936-950).
  • ValidatorEntryDto keeps index, status and validator.pubkey (dto.rs:282-293), so slashed is dropped on decode. status reaches ValidatorEntry (crates/validator/src/beacon_node/mod.rs:346-351) and is read only by is_registrable, which picks the keys that get builder registrations (lib.rs:1044, :1061-1063). Its starts_with("active") lets active_slashed through, so a slashed key is still registered.
  • On current main (fix(validator): re-send proposer preparations every slot, not once an epoch #804) the duty loop keeps the epoch's resolved entries for the per-slot fee-recipient re-send (resolved in run at 5d8628e9), so the statuses are in memory for the whole epoch.
  • Nothing logs a validator's status, a change of status, or a slashing. No metric carries a per-validator label (crates/validator/src/metrics.rs).
  • The client subscribes to no beacon node events. An ethlambda beacon node would not send proposer_slashing or attester_slashing events anyway: it accepts the topics and never emits them (docs/rpc.md:999-1003).
  • Duties go on for the slashed key. Its proposals are invalid, since process_block_header refuses a slashed proposer (crates/blockchain/state_transition/src/beacon/stf/block.rs:254-257). Its attestations earn nothing, since attestation rewards count only unslashed participants (get_unslashed_participating_indices, crates/blockchain/state_transition/src/beacon/helpers/altair.rs:242).

Why it matters

The slashed key itself has little left to lose. It cannot be slashed a second time (is_slashable_validator requires !slashed, crates/blockchain/state_transition/src/beacon/helpers/predicates.rs:35-39), and its duties earn nothing.

The other keys are what is at risk. A slashing means the setup is broken: the keys are running in a second place, slashing protection was bypassed (a key started with --init-slashing-protection that had history elsewhere, or a record restored from an old backup), or a key leaked. Whatever caused it usually applies to every key that shares the validators directory or the copied setup, and those keys are next. Each slashed validator's epoch-processing penalty grows with the total slashed in the surrounding window (process_slashings, crates/blockchain/state_transition/src/beacon/stf/epoch/electra.rs:279), so every further key slashed raises the cost for all of them. Stopping when the first is detected bounds that. Today the operator finds out from a block explorer or a falling balance.

What other clients do

Client Behavior Read at
Teku --shut-down-when-validator-slashed-enabled, off by default ("If enabled and an owned validator key is detected as slashed, the node will terminate. In this case, the service should not be restarted.") (teku/src/main/java/tech/pegasys/teku/cli/options/ValidatorOptions.java:195-203, default in validator/api/src/main/java/tech/pegasys/teku/validator/api/ValidatorConfig.java:54). When on, an owned key in a proposer or attester slashing event from the beacon node, or with status active_slashed or exited_slashed in a status update, logs a fatal line and exits with the fatal exit code (validator/client/src/main/java/tech/pegasys/teku/validator/client/ValidatorTimingActions.java:128-176, slashingriskactions/SlashedValidatorShutDown.java:34-39, slashingriskactions/SlashingRiskAction.java:30) master @ 62a54b96
Lighthouse The validator client has no slashed handling (no slashed check anywhere under validator_client/). The beacon node's validator monitor exports validator_monitor_slashed per monitored validator (beacon_node/beacon_chain/src/metrics.rs:972-977) and logs a proposer or attester slashing that names one at crit (beacon_node/beacon_chain/src/validator_monitor.rs:1810-1817, :1864-1869) unstable @ 11813cfb
Nimbus The beacon node's validator monitor exports validator_monitor_slashed (beacon_chain/validators/validator_monitor.nim:63) unstable @ 9bed158a
Prysm The validator client exports a per-key validator_statuses gauge in which 5 is SLASHING (validator/client/metrics.go:22-31) develop @ 0c1e3d24
Lodestar The validator client folds active_slashed into "active" in its status summary (packages/validator/src/services/indices.ts:20-27) unstable @ 77088eb7

Teku is the only client read here that stops on a slashed key, and only when asked. Lighthouse and Nimbus surface it on the beacon node.

Proposal

  • Read it. Declare validator.slashed in ValidatorEntryDto and carry it in ValidatorEntry. status alone loses it once a validator reaches withdrawal_possible or withdrawal_done.
  • Say it. In refresh_epoch, log every slashed loaded key at error! (pubkey, index, status) the first time it is seen, and keep a gauge of slashed loaded keys (ethlambda_validator_validators_slashed).
  • Stop signing for that key. Drop its proposer, attester and aggregation duties: a block from it is invalid and its attestations earn nothing. Leave it out of builder registrations (is_registrable) and fee-recipient preparations.
  • Optionally stop the process. A flag (Teku's is --shut-down-when-validator-slashed-enabled) that exits with a distinct, documented exit code and a message saying not to restart until the cause is found.
  • Events later. Polling once an epoch works against any beacon node, and a slashing shows in the head state as soon as a block includes it, so detection takes at most an epoch. Subscribing to proposer_slashing and attester_slashing events would cut that to seconds, against a node that emits them.

To decide

  • The default. Warn, count and drop the slashed key's duties by default, with stopping opt-in as in Teku? Or stop by default, given that the other keys are the ones at risk?
  • Restart loops. A supervisor with Restart=always would restart the client, which would detect the same key and exit again. Document RestartPreventExitStatus= with the exit code, or stay up with every key paused instead of exiting.
  • Trusting one node. With several --beacon-nodes the lookup takes the first node that answers (try_each, crates/validator/src/beacon_node/fallback.rs:325-328), so one faulty node reporting slashed would stop the client. Confirm against a second node, or against the finalized state, before exiting?
  • The slashed key's sync committee duty. Sync committee rewards have no slashed check (process_sync_aggregate, crates/blockchain/state_transition/src/beacon/stf/altair.rs:303), so a slashed member still earns them. Keep that duty, or drop every duty for simplicity?

Tests

  • A DTO test decoding "slashed": true.
  • MockBeaconNode resolving two keys, one slashed: the refresh logs it once, sets the gauge to 1, serves no duties for it and serves the other key's.
  • With the stop flag: run returns the dedicated error, and the binary maps it to its exit code.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions