Skip to content

Preserve stable configuration snapshot identity across refreshes - #31

Closed
mrayva wants to merge 47 commits into
resetius:masterfrom
mrayva:stable-snapshot-metadata-continuity
Closed

mrayva wants to merge 47 commits into
resetius:masterfrom
mrayva:stable-snapshot-metadata-continuity

Conversation

@mrayva

@mrayva mrayva commented Nov 14, 2026

Copy link
Copy Markdown

Summary

  • Preserve an already known stable configuration identity when installing a newer MRFTS001 snapshot.
  • Reject conflicting MRFTS003 configuration identities on both covered-boundary acknowledgements and new installs.
  • Add regression coverage and document the behavior.

Validation

  • Release network suite: 21/21 passed.
  • ASAN network suite: 21/21 passed.
  • git diff --check passed.

This is a self-reviewed patch; no independent reviewer approval is claimed.

mrayva added 30 commits November 7, 2026 22:10
* Reject corrupt persistence on recovery and fix log truncation state

* Fail-stop on live storage errors instead of escaping into network handlers

* Cover replacement after live truncation without reopening storage
* Add Linux durable state with atomic publication and crash-boundary tests

* Verify Raft election and application replay across durable-state restart
)

* Preserve forwarded client cookies and bind routes to leader and peer lifecycle

* Clarify forwarding marker value and peer lifecycle terminology
* Preserve snapshot chunk identity across retries to avoid ACK starvation

* Protect active reconnects from stale reader failures
* Pin active snapshot transfers across ongoing compaction

* Avoid competing connect deadline wakeups after completion
@mrayva mrayva closed this Nov 14, 2026
@resetius

resetius commented Nov 14, 2026 •

Copy link
Copy Markdown
Owner

Thank you for this work: snapshots, log compaction, InstallSnapshot, learners and membership changes are exactly what this repository needs, and the PR comes with a lot of careful testing. It was closed within a minute, and the same branch then appeared as PR 47 in the fork, so it was probably opened here by accident. I still went through it, because a lot of it would be welcome upstream. Below is a summary and a list of things to adjust when the work is split into smaller PRs. A proposed split plan follows in a separate comment, and there are inline notes on the key lines.

What makes an implementation a good fit here

Any implementation of these features is welcome if it stays generic:

  • The Raft core stays a deterministic state machine. Its inputs are messages, ticks and client requests; its outputs are messages and state/log changes.
  • Storage, snapshots and transport sit behind interfaces, so the library can be embedded anywhere.
  • It is portable (not Linux-only).
  • The library reports errors to the embedder instead of calling std::terminate() itself.

What the PR contains

The title describes the last commit (2463892, about 25 lines in raft.cpp plus a test). The PR as a whole carries the fork's entire divergence from master: 47 commits (46 squash merges of the fork's own PRs plus the final commit), 93 files, +12202/-600.

Area +/- Content
Raft core (miniraft/) +2263 / -190 strict-majority commit, FIFO request completion, fresh-majority strong reads, term-bound pending requests, RPC validation, chunked InstallSnapshot, compaction, learners, voter binding, joint-quorum arithmetic, configuration entry codec and history validator
Persistence (part of the above) TDiskState recovery validation and truncation fixes; TDurableState and TSnapshotStore with atomic publication; snapshot formats MRFTS001/002/003, durable formats MRFTD001/002/003
Network (miniraft/net/) +196 / -123 bounded frame reader/writer, peer-attributed forwarded replies, connection generations, runtime learner targets
Examples +764 / -266 KV/SQL RSMs split out, payload validation, SQL read-only enforcement and read budgets, KV/SQL snapshots, --durable, --snapshots, --learner
Tests +6592 / -13 15 new cmocka suites, 3 Python smoke tests run from ctest, adjusted test_raft inputs
Docs +2342 / -2 41 docs/*.md files (one per fork milestone v0.1.0-alpha.1 to .47), README preamble
CI / build +45 / -6 correctness.yml workflow, MINIRAFT_BUILD_NETWORK option, .gitmodules URL change

There is no generated or vendored code, and the coroio submodule pointer is unchanged.

CI. The first runs never started: first-time-contributor approval was still pending when the PR was closed, so both runs show "failure" with 0 jobs; the workflow files are fine. I re-ran both on the merge commit with current master (a4debbf), and both pass on attempt 2:

Bugs in master that this fixes

Each of these is real, and each would make a great small PR on its own:

  1. TVolatileState::CommitAdvance sorts ascending and takes Indices[nservers / 2], so a 4-node cluster commits with only 2 of 4 replicas. The fix is (nservers - 1) / 2.
  2. TRequestProcessor::ProcessWaiting checks and replies to Waiting.back() but pops the front. When several requests complete at once, the newest one gets several replies and the older ones get none. StrongWaiting has the same problem.
  3. --consistent reads: CommitSeqno advances to the seqno of any single successful AppendEntries reply, so one follower is enough to confirm a read. ReadIndex needs a majority to acknowledge heartbeats sent after the read arrived (Ongaro, PhD thesis 2014, §6.4). ApproveRead also truncates the seqno to int.
  4. ProcessCommitted queues a WriteAnswers entry for every applied entry on every node. Followers never drain the queue, so it grows without bound.
  5. TDiskState:
    • a state file of unexpected size is silently reinitialised as a fresh store, which forgets the term and the vote;
    • after the log is truncated to zero, Append records offset 0 but writes at wherever the stream position was left.
  6. TMessageReader::Read allocates a wire-supplied len without a bound and computes len - sizeof(TMessage) without checking len >= sizeof(TMessage).
  7. SQL example: Read runs arbitrary SQL through sqlite3_exec, so a "read" can change one replica outside the log. QuerySize is also not checked against the message length.
  8. One more I noticed while reading, fixed by the fork's progress-recovery change: ProcessCommitted always assigns Rsm->LastAppliedIndex = commitIndex, even when commitIndex is lower. Right after the SQL example restarts (it persists LastAppliedIndex, while CommitIndex starts at 0), the next pass re-applies entries that were already applied.

Things to adjust when splitting

Ranked by priority:

P1

  • miniraft/net/node_impl.h:46: Disconnect destroys Drainer even when it is suspended in a socket write. If the read error and write readiness arrive in the same poller batch, the destroyed frame is resumed afterwards: a possible use-after-free. I found this by reading the code and have not reproduced it.

P2

  • miniraft/raft.cpp:1037-1041: a follower whose store has no voter binding (for example an older MRFTD001 store) terminates when it receives a valid MRFTS003 snapshot, and terminates again after restart. Checking local preconditions before installing and replying Success = 0 would avoid this.
  • miniraft/raft.cpp:1019-1023 with :923-925: after the follower rejects a snapshot, the leader restarts the transfer from offset 0 with no backoff, forever. It needs a terminal outcome.
  • miniraft/raft.cpp:1024-1030 (the change in the title): the configuration identity is metadata local to each node, and the receiver patches it into the leader's image. For membership changes, the configuration should come from configuration entries in the log, and the snapshot should carry the sender's configuration as of its last included index (Ongaro & Ousterhout 2014, §7 and Fig. 13). Then no receiver-side merging is needed.
  • miniraft/durable.cpp:276-283 (via Publish, line 210): every append re-encodes and fsyncs the whole image (snapshot plus retained log). An append-only log plus a separately published snapshot file, which is the approach TSnapshotStore already takes, would keep the atomicity guarantees at a fraction of the I/O.
  • miniraft/durable.cpp:111, miniraft/snapshot.cpp:165: these are Linux-only, although the calls used are POSIX (openat, renameat, fsync, flock). Enabling them on macOS/BSD, with F_FULLFSYNC on macOS, would make snapshots portable.
  • std::terminate() inside the library (raft.cpp:1040-1041, raft.cpp:1278, persist.cpp:84, durable.cpp:272-273): reporting the error to the embedder (an exception type or callback) would leave the fail-stop policy to the embedder.
  • Scope: smaller PRs, each with its own tests, will be much easier to review and land. The groundwork for membership changes (configuration.h:8, the two-set TQuorum in quorum.h:10, MRFTS002 in snapshot.cpp:18) fits best together with the membership-change PR that uses it.

P3

  • miniraft/raft.cpp:53: the covered-snapshot path has a second, hand-written header parser. A follower that already has the boundary entry in its committed prefix holds everything the image covers, so it can acknowledge without reading the image.
  • miniraft/messages.h:232 (with line 164): Cast now throws, and holders built through the public constructor have Capacity == 0, so they fail at runtime. Keeping Cast compatible and validating at the trust boundaries (frame reader, storage reader) would preserve the embedder API.
  • miniraft/raft.cpp:1296 and :1325: if these branches were ever reached, the loop would not terminate, because PruneRequests() returns early. They are unreachable today, so an assert, or failing the front request, would be safer.
  • miniraft/raft.cpp:1110: forwarding stops for good after 2^32 cookies. Wrapping around and skipping cookies that are still in flight would be enough.
  • Formats: one versioned snapshot format and one versioned storage format, each with a version field, instead of MRFTS001/002/003 and MRFTD001/002/003.
  • Docs: a short doc per feature instead of per-milestone notes; the README preamble and the .gitmodules change are specific to the fork.

What is good

  • Every fix for the bugs listed above comes with focused deterministic tests: majority in all cluster sizes, FIFO, fresh-majority reads, stepdown, disconnect, and the recovery cases.
  • Input from the wire and from disk is validated before it is used: frame and batch bounds, RPC identities, committed-prefix protection.
  • The snapshot transfer design is thought through: images are pinned per transfer, chunks keep their identity across retries, idle transfers expire, and a covered prefix is handled.
  • Storage publication is atomic, with file and directory fsync and crash-stage hooks in the tests.
  • CI is green, including ASan+UBSan.

I hope you will consider sending pieces of this upstream; a proposed order follows below.

@resetius

Copy link
Copy Markdown
Owner

Proposed split into PRs

Every PR targets master, covers one topic, brings its own tests, and follows the conditions from the comment above:

  • the Raft core stays a deterministic state machine;
  • storage, snapshots and transport sit behind interfaces;
  • the code is portable;
  • errors are reported to the embedder, not handled with std::terminate().

Fork commits are referred to by short SHA. "Parallel" means a PR does not wait on the others in its stage.

Stage A: independent fixes for master (all parallel, except 4 after 2)

  1. Strict-majority commit index
    • Scope: CommitAdvance in miniraft/raft.cpp (from 76aae7a).
    • Tests: test_majority_all_cluster_sizes, test_even_cluster_partition_requires_majority, and the small deterministic cluster harness from test/test_correctness.cpp.
    • Depends on: nothing.
  2. FIFO completion of waiting requests
    • Scope: ProcessWaiting uses front/pop for both queues (from 76aae7a).
    • Tests: test_read_fifo, test_strong_read_fifo, test_write_fifo, test_partial_write_completion.
    • Depends on: nothing.
  3. Fresh-majority confirmation for --consistent reads
    • Scope: a read barrier seqno, per-peer acknowledged seqno, a quorum in CommitSeqno, and a current-term commit precondition (from d114d5d, plus the reply correlation it needs from cbbbfa1).
    • Tests: test_strong_reads_require_fresh_majority_in_all_cluster_sizes, test_stale_leader_with_one_follower_cannot_serve_strong_read, test_new_read_cannot_reuse_previous_read_quorum, test_delayed_pre_read_replies_cannot_confirm_new_read.
    • Depends on: nothing (it touches CommitAdvance next to 1, so a trivial rebase).
  4. Pending request lifecycle
    • Scope: keep write replies only for local pending writes, keyed by index and term; fail pending requests on leadership change and on disconnect (from e117c18).
    • Tests: test_follower_application_does_not_retain_unused_replies, test_replaced_write_never_receives_another_commands_reply, test_stepdown_rejects_write_without_waiting_for_application, test_disconnect_*.
    • Depends on: 2.
  5. Application progress never rewinds
    • Scope: ProcessCommitted never lowers LastAppliedIndex, and recovered progress protects the committed prefix (from a0c593d).
    • Tests: test_recovered_application_progress_is_not_rewound, test_application_progress_never_decreases_on_lower_commit_input.
    • Depends on: nothing.
  6. TDiskState recovery and truncation
    • Scope: no silent reinitialisation of a store with unexpected contents, an explicit seekp in Append, offset and term validation on load (from 781ce32, 93f8332). Failures are reported as exceptions.
    • Tests: test/test_persist_recovery.cpp and the test_disk_state* adjustments.
    • Depends on: nothing.
  7. Bounded frame reading
    • Scope: TMessageReader validates type and length before allocating (core of b36e5b1).
    • Tests: the reader cases of test/test_network_frames.cpp.
    • Depends on: nothing.
  8. SQL example: read-only reads and length checks
    • Scope: from b514088 and the SQL part of ca86106; optionally managed transactions (5849d85) and read budgets (f958c43) as follow-ups.
    • Tests: test/test_sql_payloads.cpp.
    • Depends on: nothing.
  9. (optional) Forwarding keeps the client's cookie
    • Scope: a separate EForwarded marker, with replies routed by the forward cookie and the originating peer (from eee8ddc).
    • Tests: test/test_forwarding.cpp.
    • Depends on: nothing.

Stage B: infrastructure (10-13 parallel, once their Stage A dependency is in)

  1. Message limits and wire validation
    • Scope: TMessageLimits, batch caps in CreateAppendEntries, writer-side validation (rest of b36e5b1, and afde6fa). TMessageHolder::Cast stays compatible for embedders; extent checks live at the trust boundaries.
    • Tests: test/test_message_safety.cpp, test/test_network_frames.cpp.
    • Depends on: 7.
  2. RPC validation in the core
    • Scope: ValidateRpc (sender/receiver identity, term and index ranges, committed-prefix protection), AppendEntries reply correlation, leader-only TRaft::Append, ownership of in-memory log entries, term-only lookups (from cbbbfa1, e0a4cb9, 704ae08, 6d9c758).
    • Tests: test/test_rpc_validation.cpp and the proposal tests in test/test_correctness.cpp.
    • Depends on: 3.
  3. Transport connection lifecycle
    • Scope: connection generations, handling of stale reader failures, connect without the unsafe deadline path (network parts of 049907d and eb16ab6). Also covers how a suspended Drainer is retired (see the inline note on node_impl.h:46).
    • Tests: test/test_node_lifecycle.cpp.
    • Depends on: nothing.
  4. Storage and application interfaces for snapshots
    • Scope: optional IState methods with defaults (snapshot index/term, GetSnapshot, InstallSnapshot, RetainedLogTerm) and optional IRsm SaveSnapshot/LoadSnapshot/ValidateSnapshot, so existing backends keep compiling and behaving as before. Plus a portable POSIX durable backend: an append-only log and an atomically published snapshot file instead of a full-image rewrite per append (from 4e706a4 and the storage parts of 4b76d8b and edea429).
    • Tests: test/test_durable.cpp, test/test_persist_recovery.cpp, enabled on every POSIX platform.
    • Depends on: 6.

Stage C: features, in dependency order

  1. Log compaction
    • Scope: one versioned snapshot format (index, term, configuration, application data, checksum), compaction triggered by the embedder, restore at startup, KV example snapshots (from 4b76d8b, the compaction parts of edea429, and 42fb6c0, 91d8a0c, 3126493).
    • Tests: test/test_snapshots.cpp and the compaction cases of test/test_compaction.cpp.
    • Depends on: 13.
  2. InstallSnapshot with chunking
    • Scope: messages 6/7, an image pinned per transfer, chunk identity kept across retries, idle expiry, a cap on pinned generations, the covered-prefix shortcut, and a terminal outcome for a rejected image (from edea429, 049907d, eb16ab6, b20fba6, d202c36, ff79596, 005b14a, 326033e, 3010108, 53e1a0e, c52eb0a).
    • Tests: the transfer cases of test/test_compaction.cpp. A deterministic in-process cluster test is preferable to the Python multi-process smoke test.
    • Depends on: 14 and 11.
  3. SQL example snapshots
    • Scope: from 06d936a.
    • Tests: the snapshot cases of test/test_sql_payloads.cpp.
    • Depends on: 14 and 8. Parallel with 15.
  4. Learners (non-voting members)
    • Scope: a learner role in the core, replication without votes or campaigns, a catch-up check, server-side targets (from 9a5752e, 25890fe, c912788).
    • Tests: test/test_learners.cpp and the learner case in test/test_read_write.cpp.
    • Depends on: 11. A new learner catches up through 15, but development can run in parallel with 14-16.
  5. Quorum abstraction
    • Scope: TQuorum (one or two voter sets) used for elections, commitment and read confirmation. A pure refactor with no behaviour change (from 452377c).
    • Tests: test/test_quorum.cpp plus the existing majority tests.
    • Depends on: 1 and 3. Parallel with Stage C.
  6. Membership changes
    • Scope:
      • configuration entries in the log, which take effect as soon as they are appended;
      • joint consensus (Ongaro & Ousterhout 2014, §6; thesis ch. 4) or single-server changes;
      • learner promotion;
      • the snapshot carries the configuration as of its last included index.
    • Based on a rework of db5794b, 5ae2203, 99efd7d, f510e99, 47218e8, 467bb14, 0fea03f, a2cd136, 2463892. Once the configuration comes from the log, the separate stable-identity format (MRFTS003) and the merging on the receiver side are no longer needed.
    • Tests: test/test_configuration.cpp plus end-to-end reconfiguration tests in the deterministic harness (adding and removing a voter, leader removal, a crash during the joint phase).
    • Depends on: 15, 17, 18. It can itself be two PRs: (a) configuration entries and the joint quorum, then (b) snapshot configuration and promotion.

Parallelism at a glance: 1-9 together (4 after 2); then 10, 11, 12 and 13 together; then 14, followed by 15 and, in parallel, 16; 17 and 18 run alongside 14-16; 19 comes last.

Not needed upstream: per-milestone docs (one short doc per feature is better), the README preamble, the .gitmodules URL change, and a separate correctness.yml (the existing workflow can run the new suites, and a core-only build without networking can be added there).

@resetius resetius left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Inline notes for the summary comment above, ranked P1/P2/P3 the same way. The "Good" notes mark code worth bringing upstream first; the numbers refer to the split-plan comment.

Comment thread miniraft/net/node_impl.h
// still be registered with the poller; never destroy that live coroutine
// because an older reader failed. Established connectors are done here.
Connected = false;
if (Drainer) { Drainer.destroy(); Drainer = {}; }

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 (possible use-after-free; found by reading the code, not reproduced). Disconnect runs inside OutboundServe and can destroy Drainer while it is suspended in TMessageWriter::Write on the same socket, for example when the peer stopped reading and the send buffer is full.

When the peer then resets the connection, epoll/kqueue report read and write readiness in the same batch. coroio queues both handles in ReadyEvents_ (READ first) and resumes them one by one:

  1. The reader fails and calls Disconnect.
  2. Drainer.destroy() frees the writer frames (through the TValueTask destructors).
  3. WakeupReadyHandles then resumes the queued WRITE handle of a destroyed frame.

The RemoveEvent queued by the socket destructor only takes effect at the next poll, so it doesn't help here.

The guard above already protects a live Connector for this reason. Treating a suspended Drainer the same way would close the gap: mark the node disconnected and let the writer fail and unwind on its own, or defer destruction until it is done.

Comment thread miniraft/raft.cpp
State->InstallSnapshot(snapshot); // durable snapshot and suffix before application replacement or ACK
application->LoadSnapshot(snapshot.Data, snapshot.Index);
if (application->LastAppliedIndex != snapshot.Index) std::terminate();
} catch (...) { std::terminate(); }

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2. A valid snapshot from a legitimate leader can end here in std::terminate(). DecodeSnapshot accepts MRFTS003 and the checks above pass. However, TDurableState::InstallSnapshot (and TState::InstallSnapshot) throw "Configuration snapshots require a bound voter set" when the store has no voter binding. The TRaft constructor leaves a non-fresh store without a snapshot and without learners unbound; an MRFTD001 store is one example. After a restart the node is still behind, so it would terminate again.

Suggestion: check every precondition that depends only on local state (binding, index/term ordering) before this block and reply Success = 0. Keep this path for genuine I/O failures, reported to the embedder.

Comment thread miniraft/raft.cpp
if (snapshot.StableConfiguration &&
(snapshot.ConfigurationIndex != knownSnapshot->ConfigurationIndex ||
snapshot.ConfigurationTerm != knownSnapshot->ConfigurationTerm))
throw TMessageError("Stable snapshot configuration identity changed");

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 (liveness). When the follower rejects an image here (a conflicting identity, or ProbeCoveredSnapshot returning Supported = false on the covered path), the leader resets the transfer to offset 0 (lines 923-925) and sets RpcDue to 0 (line 939). It then streams the whole image again; nothing changes on either side, so this repeats forever without backoff or a diagnostic.

Suggestion: if the condition can't happen in a correct cluster, an assert is enough. Otherwise, give it a terminal outcome: stop the transfer to that peer and report the peer to the embedder. A test that injects a permanently rejecting follower would pin this down.

Comment thread miniraft/raft.cpp
if (!snapshot.StableConfiguration) {
// A legacy-format refresh has no configuration identity. Keep
// the receiver's trusted identity when storing the new image.
snapshot.StableConfiguration = true;

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 (design). Here the receiver writes its own ConfigurationIndex/ConfigurationTerm into the image the leader sent. Raft has a simpler rule: a snapshot carries the configuration in effect at its last included index, taken from the sender's log (Ongaro & Ousterhout 2014, §7 and Fig. 13; thesis ch. 5).

Once configuration changes are entries in the log, every node derives the same configuration for the same index. The leader's snapshot then already carries the right configuration, and nothing needs merging or rejecting on the receiving side. For the split, it would be easiest to land this together with membership changes (19 in the split plan) and to drop the separate stable-identity format.

Comment thread miniraft/raft.cpp
// replication progress. Stable metadata identity is checked as well. The
// application payload and CRC are unnecessary because the receiver already
// has this committed log prefix.
const size_t fixedHeaderBytes = stable ? 52 : 36;

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3. This is a second, hand-written header parser (offsets 8/16/24/28/36/44, header sizes 36/52) that has to stay in sync with EncodeSnapshot/DecodeSnapshot in snapshot.cpp. The covered case doesn't need the image at all: when SnapshotIndex <= commitIndex and the terms match, the follower already holds every entry the snapshot covers (Log Matching), so it can acknowledge right away without buffering. That would remove this parser and the extra IncomingSnapshot state for the covered path.

Comment thread miniraft/raft.cpp
auto commitIndex = std::max(CommitIndex, Indices[nservers / 2]);
// Ascending order: a strict majority must have replicated this index.
// For even cluster sizes, the upper median would accept only half.
auto commitIndex = std::max(CommitIndex, Indices[(nservers - 1) / 2]);

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good: a real bug in master. With an ascending sort, Indices[nservers / 2] lets a 4-node cluster commit with 2 of 4 replicas. This line with test_majority_all_cluster_sizes would be a great first standalone PR (1 in the split plan).

Comment thread miniraft/raft.cpp
continue;
}
if (Waiting.front().Index > lastApplied) break;
auto w = Waiting.front(); Waiting.pop();

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good: also a master bug. The old loop checked and replied to Waiting.back() but popped the front, so when several requests completed at once, the newest got several replies and the older ones got none. Front/pop for both queues, with the FIFO tests, makes a small independent PR (2 in the split plan).

Comment thread miniraft/raft.cpp
return VolatileState->CommitSeqno;
if (StateName != EState::LEADER || VolatileState->CommitIndex == 0 ||
State->LogTerm(VolatileState->CommitIndex) != State->CurrentTerm) return 0;
return VotingQuorum->Value([&](uint32_t id) -> uint64_t {

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good. This is the ReadIndex rule from the thesis (§6.4): a commit in the current term, plus a majority acknowledging RPCs sent after the read barrier. In master, one follower's reply is enough to advance CommitSeqno. The barrier, the per-peer ReadAckSeqno and this quorum would make a focused fix (3 in the split plan).

Comment thread miniraft/persist.cpp

// In particular, after truncation to zero the stream may still point into
// the old suffix. Always position the output explicitly.
Entries.seekp(offset);

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good: a real master bug. After the log was truncated to zero, Append recorded offset 0 but wrote wherever the stream position had been left. This explicit seekp, together with refusing to reinitialise a store of unexpected size (line 32), would make a nice small TDiskState PR (6 in the split plan). Errors there could surface as exceptions instead of Fail/std::terminate.

Comment thread miniraft/net/server.h
NNet::TValueTask<TMessageHolder<TMessage>> ReadFrame(bool logOnly, uint64_t& remaining) {
TMessage header;
co_await NNet::TByteReader(Socket).Read(&header, sizeof(header));
ValidateMessageHeader(header.Type, header.Len, Limits);

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good. Validating type and length before allocating fixes master's unbounded NewHoldedMessage(type, len) and the len - sizeof(TMessage) underflow. The reader part alone would be a small standalone PR (7 in the split plan), with the batch limits following later.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants