Skip to content

feat(load): load single-region TsFiles as whole-file pieces through consensus - #19335

Open
luoluoyuyu wants to merge 4 commits into
apache:masterfrom
luoluoyuyu:feat/load-whole-file-2pc
Open

luoluoyuyu wants to merge 4 commits into
apache:masterfrom
luoluoyuyu:feat/load-whole-file-2pc

Conversation

@luoluoyuyu

Copy link
Copy Markdown
Member

Description

A TsFile whose devices all map to one data region and one time partition no longer has to be split into chunk pieces when the region has multiple replicas. The coordinator copies the source file in 4 MiB whole-file slices, submits each slice as a PIECE in the existing two-phase consensus LOAD transaction, and the receiver reassembles the exact bytes, validates them, and imports the complete TsFile on COMMIT.

The local whole-file fast path is kept for a single local replica, and multi-region or time-partition-crossing files keep the split path.


This PR has:

  • been self-reviewed.
    • concurrent read
    • concurrent write
    • concurrent read and write
  • added documentation for new or modified features or behaviors.
  • added Javadocs for most classes and all non-trivial methods.
  • added or updated version, license, or notice information
  • added comments explaining the "why" and the intent of the code wherever would not be obvious
    for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold
    for code coverage.
  • added integration tests.
  • been tested in a test IoTDB cluster.

Key changed/added classes (or packages if there are too many classes) in this PR

…onsensus

A TsFile whose devices all map to one data region and one time partition no longer has to be split into chunk pieces when the region has multiple replicas. The coordinator copies the source file in 4 MiB whole-file slices, submits each slice as a PIECE in the existing two-phase consensus LOAD transaction, and the receiver reassembles the exact bytes, validates them, and imports the complete TsFile on COMMIT.

The local whole-file fast path is kept for a single local replica, and multi-region or time-partition-crossing files keep the split path.
The branch is based on apache/master, whose LoadTsFileAnalyzer still logs LOAD_ANALYSIS_STAGE_ARG_ARG_TSFILES_HAVE_BEEN_ANALYZED_PROGRESS_ARG_PERCENT. The working-tree i18n refactor had renamed it, so the two locales must keep the old key as well until the caller is migrated.

@Caideyipi Caideyipi left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The whole-file path has two recovery/snapshot correctness gaps. See the inline comments.


private static void discardUnrecordedStagedFiles(final File snapshotTaskDir) throws IOException {
// A whole-file task has no chunk records by design: its marker and its bytes are the state.
if (new File(snapshotTaskDir, TsFileWriterManager.WHOLE_FILE_META_NAME).isFile()) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Catch up whole-file bytes to the snapshot boundary. copyTask copies staged data before progress logs, then relies on catchUpStagedFiles to repair concurrent appends from recorded ranges. Whole-file writes do not record per-slice ranges, so this early return preserves whatever prefix the copy captured. If a snapshot overlaps a PIECE already covered by its consensus boundary, the restored staging file can miss that slice tail; later slices then fail the offset check in appendWholeFile, or PREPARE rejects the file length. Please record/catch up a whole-file applied offset or otherwise serialize these writes with snapshot copying.

wholeFileModsChannel =
FileChannel.open(
wholeFileMods.toPath(), StandardOpenOption.READ, StandardOpenOption.WRITE);
wholeFileModsSourceLength = wholeFileMods.length();

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Persist the declared mods length for recovery. ensureWholeFileMods compares this field with each incoming slice sourceLength, but recovery assigns the current staged prefix length here. After a restart during a multi-slice .mods transfer, the next replayed slice carries the original full source length and is rejected, so the transaction cannot resume. Persist the mods kind and declared source length separately from the staged file length, and restore those values here.

Deletions are stored in a .mods sidecar and are not part of the TsFile bytes. The whole-file path therefore no longer defines MODS_V1/MODS_V2 slice kinds: a TsFile with a deletion sidecar falls back to the chunk-splitting path, which already transfers its deletions, and the whole-file path only carries the complete .tsfile.
The deletion sidecar is not part of the TsFile bytes, but it does not need a separate MODS slice kind. The coordinator now reads the source .mods with ModificationFile.readAllModifications and sends every entry as the existing DeletionData in the same two-phase LOAD transaction. The receiver stages those entries in the .mods sidecar of the whole-file staging, and loadNewTsFile applies them on COMMIT.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants