Repository navigation
feat(load): load single-region TsFiles as whole-file pieces through consensus - #19335
luoluoyuyu wants to merge 4 commits into
Conversation
…onsensus A TsFile whose devices all map to one data region and one time partition no longer has to be split into chunk pieces when the region has multiple replicas. The coordinator copies the source file in 4 MiB whole-file slices, submits each slice as a PIECE in the existing two-phase consensus LOAD transaction, and the receiver reassembles the exact bytes, validates them, and imports the complete TsFile on COMMIT. The local whole-file fast path is kept for a single local replica, and multi-region or time-partition-crossing files keep the split path.
The branch is based on apache/master, whose LoadTsFileAnalyzer still logs LOAD_ANALYSIS_STAGE_ARG_ARG_TSFILES_HAVE_BEEN_ANALYZED_PROGRESS_ARG_PERCENT. The working-tree i18n refactor had renamed it, so the two locales must keep the old key as well until the caller is migrated.
Caideyipi
left a comment
There was a problem hiding this comment.
The whole-file path has two recovery/snapshot correctness gaps. See the inline comments.
|
|
||
| private static void discardUnrecordedStagedFiles(final File snapshotTaskDir) throws IOException { | ||
| // A whole-file task has no chunk records by design: its marker and its bytes are the state. | ||
| if (new File(snapshotTaskDir, TsFileWriterManager.WHOLE_FILE_META_NAME).isFile()) { |
There was a problem hiding this comment.
[P2] Catch up whole-file bytes to the snapshot boundary. copyTask copies staged data before progress logs, then relies on catchUpStagedFiles to repair concurrent appends from recorded ranges. Whole-file writes do not record per-slice ranges, so this early return preserves whatever prefix the copy captured. If a snapshot overlaps a PIECE already covered by its consensus boundary, the restored staging file can miss that slice tail; later slices then fail the offset check in appendWholeFile, or PREPARE rejects the file length. Please record/catch up a whole-file applied offset or otherwise serialize these writes with snapshot copying.
| wholeFileModsChannel = | ||
| FileChannel.open( | ||
| wholeFileMods.toPath(), StandardOpenOption.READ, StandardOpenOption.WRITE); | ||
| wholeFileModsSourceLength = wholeFileMods.length(); |
There was a problem hiding this comment.
[P2] Persist the declared mods length for recovery. ensureWholeFileMods compares this field with each incoming slice sourceLength, but recovery assigns the current staged prefix length here. After a restart during a multi-slice .mods transfer, the next replayed slice carries the original full source length and is rejected, so the transaction cannot resume. Persist the mods kind and declared source length separately from the staged file length, and restore those values here.
Deletions are stored in a .mods sidecar and are not part of the TsFile bytes. The whole-file path therefore no longer defines MODS_V1/MODS_V2 slice kinds: a TsFile with a deletion sidecar falls back to the chunk-splitting path, which already transfers its deletions, and the whole-file path only carries the complete .tsfile.
The deletion sidecar is not part of the TsFile bytes, but it does not need a separate MODS slice kind. The coordinator now reads the source .mods with ModificationFile.readAllModifications and sends every entry as the existing DeletionData in the same two-phase LOAD transaction. The receiver stages those entries in the .mods sidecar of the whole-file staging, and loadNewTsFile applies them on COMMIT.
Description
A TsFile whose devices all map to one data region and one time partition no longer has to be split into chunk pieces when the region has multiple replicas. The coordinator copies the source file in 4 MiB whole-file slices, submits each slice as a PIECE in the existing two-phase consensus LOAD transaction, and the receiver reassembles the exact bytes, validates them, and imports the complete TsFile on COMMIT.
The local whole-file fast path is kept for a single local replica, and multi-region or time-partition-crossing files keep the split path.
This PR has:
for an unfamiliar reader.
for code coverage.
Key changed/added classes (or packages if there are too many classes) in this PR