Repository navigation
fix(fork_choice): walk the filtered block tree with an explicit stack - #758
Conversation
…ead of scanning the whole block index at every node of filter_block_tree. The scan ran once per node of the justified subtree over the full live window since finality, so the head walk was quadratic in the number of unfinalized blocks, exactly what a finality stall or a range catch-up grows. walk_head and invalidate_subtree already build the map this way; this brings the filter in line with them. A new test keeps the spec-literal scan as a reference and checks the two agree on random trees, including ones where a branch is filtered out.
filter_block_tree recursed once per block below the justified checkpoint. That depth grows by a block every slot for as long as the chain does not justify, and get_head runs on the chain actor's thread, which spawned-concurrency starts with std::thread::spawn and so the default 2 MiB stack. On that stack the recursive walk overflowed between 5,500 and 6,000 blocks of depth (release-fast): about 18 to 20 hours of a stall with no new justification. A stack overflow aborts the process, and the next head computation after a restart walks the same tree. The walk now keeps its own stack of the path from the justified block to the block in hand. It visits the same blocks in the same order, asserts `block_root in store.blocks` at the same points and returns the same tree. The leaf check moved unchanged into is_viable_leaf, which returns the verdict and leaves the insertion to the walk.
🤖 Kimi Code ReviewI'll review this PR which converts a recursive SummaryThe PR correctly addresses a critical stack overflow vulnerability in the fork choice Critical Issues1. Potential Logic Bug:
|
| Aspect | Rating | Notes |
|---|---|---|
| Correctness | ✅ Good | Iterative transformation preserves semantics |
| Security | ✅ Improved | Eliminates DoS via stack overflow |
| Performance | ✅ Neutral | Same complexity, heap instead of stack |
| Safety | ✅ Good | No unsafe, proper error handling |
| Maintainability | Frame struct is clear; comments could be sharper |
|
| Testing | ✅ Good | Deep chain test validates the fix |
Overall: Approve with minor suggestions. The core fix is correct and important. Address the comment clarity and consider the test stack size adjustment.
Automated review by Kimi (Moonshot AI) · kimi-k2.6 · custom prompt
🤖 Codex Code ReviewAutomated review by OpenAI Codex |
🤖 Claude Code ReviewReview:
|
#746 was squash-merged, so this branch's copy of its commits no longer matches main. The tree is main plus only this branch's own commit.
…#758) filter_block_tree recursed once per block below the justified checkpoint. That depth grows by a block every slot for as long as the chain does not justify, and get_head runs on the chain actor's thread, which spawned-concurrency starts with std::thread::spawn and so the default 2 MiB stack. On that stack the recursive walk overflowed between 5,500 and 6,000 blocks of depth (release-fast): about 18 to 20 hours of a stall with no new justification. A stack overflow aborts the process, and the next head computation after a restart walks the same tree. The walk now keeps its own stack of the path from the justified block to the block in hand. It visits the same blocks in the same order, asserts `block_root in store.blocks` at the same points and returns the same tree. The leaf check moved unchanged into is_viable_leaf, which returns the verdict and leaves the insertion to the walk. Cherry-picked from fork-choice-iterative-filter at de7bb98 (b102e35; the branch's other commits are #746, already on nightly, and merges of main). Claude-Session: https://claude.ai/code/session_011vghNmndKPkZinHRkVTwVB
pablodeymo
left a comment
There was a problem hiding this comment.
Reviewed at 7464df5. The explicit-stack walk looks correct to me: it visits the same blocks in the same post-order as the recursion, makes the block_root in store.blocks assertion on entry, inserts on exit, and returns the same verdict.
What I checked beyond reading it:
- The new test catches the regression. Under
release-fastit passes in 0.07 s on this branch. With main's recursivefilter_block_treeput back under the same test, it aborts withthread 'deep-chain-head-walk' has overflowed its stack. - The differential test passes (
the_filtered_tree_matches_the_spec_literal_children_scan_on_random_trees). - The 2 MiB premise holds.
Backend::Threadin spawned-concurrency 0.5.0 isstd::thread::spawn, and nothing in the repo sets a stack size orRUST_MIN_STACK.
Small things, none blocking (line numbers are in fork_choice.rs at this head):
- Two doc comments no longer match the code.
- Line 1888 says
indexis "built once byget_filtered_block_tree". That function receivesindexand builds onlychildren_of. The wording predates this PR, but the sentence is edited here. - Line 3389 calls
filter_block_tree"the only caller ofget_checkpoint_blockin this file". The call is inis_viable_leafnow, andvalidate_on_attestation_indexedandon_blockcall it too.
- Line 1888 says
- The test reference can call
is_viable_leafdirectly.filter_block_tree_by_scan(line 10000) gets a leaf's verdict by callingfilter_block_treewith an empty child map, so the reference runs through the loop it is the reference for. Calling the new helper removes that, and with it the "only the spec-literal reference in the tests passes anything else" caveat in the doc. - No test pins the error path. The description says a missing justified root still fails with
SpecAssert("block_root in store.blocks"). I did not find a test asserting that; a short one againstget_filtered_block_treewould cover it. - The description could mention the other callers.
get_head_nodeis also reachable off the chain actor's thread, as a fallback inrpc/src/beacon/validator.rs,rpc/src/beacon/gloas_proposal.rsandgossip/execution_payload_bid.rs. This fix covers those too.
Follow-ups, not for this PR:
- Leaves × depth is the next limit.
is_viable_leafstill walks every leaf back to the finalized epoch throughget_checkpoint_block, which is the 74 ms at 14,851 blocks and 451 leaves from the #746 measurements. The overflow used to cap the depth, and now nothing does. Thepaththis PR adds makes it cheap to fix:- If the justified block is above the finalized epoch's start slot, the answer is the same for every leaf, so one
get_ancestorfrom the base is enough. - Otherwise the checkpoint block is on
path, and apartition_pointby slot finds it.
- If the justified block is above the finalized epoch's start slot, the answer is the same for every leaf, so one
discard_pending_subtreestill recurses (crates/blockchain/src/lib.rs), once per pending descendant, on the same actor thread. I did not measure its frame or check whether a deep pending chain can form, so this is only a pointer.
…checkpoint_block filter_block_tree's doc said get_filtered_block_tree builds index; it receives it from its caller and builds only children_of. gloas_get_ancestor's doc called filter_block_tree the only caller of get_checkpoint_block in the file; the call is in the filter's leaf check (is_viable_leaf), and validate_on_attestation_indexed and on_block call it too.
…f for a leaf filter_block_tree_by_scan got a leaf's verdict by calling filter_block_tree with an empty child map, so the reference ran through the walk it is the reference for. It now calls is_viable_leaf directly and makes the specification's block_root in store.blocks check itself, sharing nothing else with the walk. Nothing passes filter_block_tree a root other than the justified one any more, so its doc note says get_filtered_block_tree is the only caller.
…k is gone
A justified block whose LiveChain row was deleted, with its child still in the index, must fail get_filtered_block_tree on entry with SpecAssert("block_root in store.blocks"), the error update_head_from_fork_choice logs before keeping its head. The child is what makes the walk's entry check the only source of that message: with the check removed, the walk reaches the child and fails its leaf check on block_root in store.block_states instead, and the test fails.
|
Thanks for running the overflow both ways. I've pushed the four small items:
I've left the two follow-ups out of this PR: leaves × depth through I also merged main; it merged without conflicts, and the state-transition tests pass on the merge (641 passed). |
|
/review |
🤖 Gemini Code ReviewThe changes look great. Replacing the recursion in Automated review by Gemini · gemini-4-argon · custom prompt |
🤖 Claude Code ReviewReview: PR 758, iterative
|
🤖 Codex Code ReviewNo actionable findings. The iterative traversal preserves leaf validation, branch viability propagation, and error behavior while removing the recursion-related stack overflow risk. The added regression tests cover deep chains and missing justified roots. Tests could not run: Rustup attempted to update the pinned toolchain but was blocked by the read-only filesystem. Automated review by OpenAI Codex · gpt-6.1-sol · custom prompt |
🤖 Grok Code ReviewI'll compare the iterative walk with the previous recursive filter and check the surrounding fork-choice helpers for semantic drift. Automated review by Grok (xAI) · grok-4.7 · custom prompt |
🤖 Kimi Code ReviewNo issues found. The explicit-stack traversal in The deep-chain regression test appropriately exercises a 2 MiB stack and 20,000 blocks, while the missing-justified-block test covers the important failure path. The walk now uses heap memory proportional to chain depth instead of risking a process-aborting stack overflow, with linear traversal given the prebuilt index and children map. Automated review by Kimi (Moonshot AI) · kimi-k3.1-pro · custom prompt |
🤖 DeepSeek Code ReviewThe iterative rewrite is sound: it preserves the original DFS order and error sites, and moving the path to a heap-allocated One consensus-level concern:
No memory-safety or performance issues found in the stack conversion itself. Automated review by DeepSeek · deepseek-v4-pro · custom prompt |
|
Round three (the automated review of 895a48d). Nothing changed. On 895a48d the fork_choice unit tests pass under Not changed
Gemini, Codex and Kimi found nothing to change. Codex could not run the tests (its toolchain update was blocked). Grok's comment stopped after its opening sentence. |
🗒️ Description / Motivation
filter_block_treerecursed once per block below the justified checkpoint. While the chain does not justify, that depth grows by one block every slot, andget_headruns on the chain actor's thread: spawned-concurrency'sBackend::Threadstarts it withstd::thread::spawn, so it gets the default 2 MiB stack (nothing in the node sets a stack size).On that stack the recursive walk overflowed between 5,500 and 6,000 blocks of depth (
release-fast, measured on an M3 Pro and on a Ryzen 7 9700X). On mainnet that is about 18 to 20 hours of a stall in which nothing new is justified. A stack overflow aborts the process, and the next head computation after a restart walks the same tree.The chain actor is not the only thread that runs this walk.
get_head_nodeis also the fallback, when the store has no head recorded, in three places off the actor:crates/net/rpc/src/beacon/validator.rs(payload_present_index, for gloas attestation data), called inline from the async handler, so on a tokio worker thread;crates/net/rpc/src/beacon/gloas_proposal.rs(prepare, forproduceBlockV4), underspawn_blocking;crates/blockchain/state_transition/src/beacon/gossip/execution_payload_bid.rs(head_view, behindis_bid_compatible_with_head), in p2p's stateful gossip checks, also underspawn_blocking.Tokio's worker and blocking threads default to 2 MiB stacks too, and the node does not change that, so the same walk could overflow there; this fix covers those callers as well. I did not measure the overflow depth on those threads.
Found while timing #746 on synthetic stall trees (results in a comment there). #746 kept the recursion; its parent-to-children map is what makes each step of the explicit stack a lookup rather than an index scan.
What Changed
crates/blockchain/state_transition/src/beacon/fork_choice.rsfilter_block_treekeeps its own stack of the path fromblock_rootto the block in hand (one frame per block: its index entry, its children, the next child to visit, and whether any child so far is viable) instead of recursing. Same signature.is_viable_leaf, which returns the verdict and leaves the insertion to the walk.index, and which functions callget_checkpoint_block.the_head_walk_does_not_overflow_a_default_stack_on_a_stall_deep_chain.the_filter_fails_on_entry_when_the_justified_block_is_gone.filter_block_tree_by_scannow asksis_viable_leaffor a leaf's verdict instead of callingfilter_block_treewith an empty child map, so it no longer runs through the walk it checks.Correctness / Behavior Guarantees
children_ofslice order), so the same filtered tree and return value.block_root in store.blocksassertion runs when a block is entered, where each recursive call made it, so a pruned justified root still fails the walk with the same error (update_head_from_fork_choicelogs it and keeps the head).the_filter_fails_on_entry_when_the_justified_block_is_gonepins this.Vecwith one small frame per level of depth, instead of one stack frame per level.Tests Added / Run
the_head_walk_does_not_overflow_a_default_stack_on_a_stall_deep_chain. It builds a 20,000-block chain with nothing justified past the anchor and runsget_filtered_block_treeandget_head_nodeon a thread with an explicit 2 MiB stack. With perf(fork_choice): build the parent-to-children map once in get_filtered_block_tree #746's recursive walk it aborts with a stack overflow; with this change it passes in 0.07 s. A stack overflow takes down the whole test binary rather than failing one test, which the test's doc comment says.the_filter_fails_on_entry_when_the_justified_block_is_gone. It deletes the justified block'sLiveChainrow (what invalidating it leaves behind) while its child stays, and checksget_filtered_block_treefails withSpecAssert("block_root in store.blocks"). The child makes the walk's entry check the only source of that message: with the check removed, the test fails, because the walk reaches the child and its leaf check fails onblock_root in store.block_statesinstead.the_filtered_tree_matches_the_spec_literal_children_scan_on_random_treescompares the walk with the recursive spec-literal reference on 100 random trees, including trees where branches are filtered out. The reference now shares onlyis_viable_leafwith the walk.make fmt,make lint: cleancargo test --profile release-fast -p ethlambda-state-transition: 641 passed, 0 failed, 15 ignored (fork_choice unit tests: 96 passed)make test-consensus: 1,325 passed, 0 failed (before the review changes; not re-run)--test beacon_spec_tests --features beacon-spec-tests -- fork_choice): 206 passed (before the review changes; not re-run)Not checked: the overflow depth under the
releaseprofile, where LTO can change frame sizes, and on tokio's threads.Related Issues / PRs
✅ Verification Checklist
make fmt: cleanmake lint(clippy with-D warnings): cleanmake test(test-consensusplustest-node, atrelease-fast):test-consensusrun locally;test-nodeleft to CI (unrelated to this change)