Skip to content

[Pipe] Guard DAL lifecycle and drain producers before region cleanup - #19321

Merged
jt2594838 merged 2 commits into
apache:masterfrom
Pengzna:codex/fix-dal-region-cleanup-warn
Nov 20, 2026
Merged

jt2594838 merged 2 commits into
apache:masterfrom
Pengzna:codex/fix-dal-region-cleanup-warn

Conversation

@Pengzna

@Pengzna Pengzna commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Deleting a DataRegion under IoTConsensusV2 currently obtains its deletion resource manager through a get-or-create API. Cleanup can allocate a DAL buffer, open a file and start a persistence thread just to close them. Normal close then interrupts an idle queue wait and emits a misleading WARN. Cleanup also needs to exclude concurrent initialization and late pipe work so it cannot remove files used by a live manager.

For example, with IoTConsensusV2 configured:

CREATE DATABASE root.dal_warn;
INSERT INTO root.dal_warn.d(time, s) VALUES (1, 1);
DROP DATABASE root.dal_warn;

Before this change, cleanup can initialize DAL resources and emit Interrupted when waiting for taking DeletionResource from blocking queue to serialize. despite no SQL DELETE having been issued. A concurrent or late get-or-create can also leave a manager using a removed directory. After this change, cleanup never constructs a manager, creation and close/removal share a region lifecycle lock, and a deleted-region tombstone prevents recreation until storage starts a new region lifecycle. An idle buffer or one collecting a batch exits normally after draining; a real flush timeout remains visible as a warning.

Implementation

  • Serialize manager creation, registration, close and DAL removal with stable, bounded striped locks. Install the deleted-region barrier before draining producers. Reject registration through a retained old manager with a completed failure result, and prevent old reference-release callbacks from deleting a new lifecycle's files. Reset the barrier only from the storage region creation path; ordinary manager close still permits recreation.
  • Use getInstanceIfPresent for event-binding lookups. Keep lazy creation where historical DAL recovery or a new local deletion needs it, guarded by the lifecycle barrier; historical extraction also checks the DataRegion's deleted state under its write lock.
  • Release the region receiver and stop/drain its realtime assigner before closing the manager and deleting DAL. Prevent late source registration while the region is deleted. Do not hold the DAL lock while waiting for receiver/assigner shutdown.
  • Use bounded queue polls and normal executor shutdown after draining. Retain forced shutdown and an explicit warning after a flush timeout, preserve unexpected interruption warnings, and correct the batch-poll message to refer to DeletionResource. Add new messages in both locales.
  • Run reference-release hooks outside the DeletionResource monitor to avoid inversion with the lifecycle lock while close waits for synchronized persist completion callbacks.

Validation

  • JDK 17: the six relevant test classes passed 33 tests, with 0 failures/errors (DAL lifecycle/buffer/persistence/recovery and realtime listener/assigner).
  • Spotless formatting/checks and English/Chinese compilation passed through DataNode, ConfigNode and the other upstream modules.
  • Both root-reactor invocations (mvn test -Dtest=DeletionResourceManagerCleanupTest,PageCacheDeletionBufferTest,DeletionResourceTest,DeletionRecoverTest,PipeInsertionDataNodeListenerTest,PipeDataRegionAssignerTest -Dsurefire.failIfNoSpecifiedTests=false -DfailIfNoTests=false and mvn test-compile -P with-zh-locale -DskipTests) stopped later at distribution dependency resolution: the local iotdb-server, iotdb-cli and iotdb-confignode 2.0.13-SNAPSHOT zip artifacts are absent. Thus this is not a full-reactor build success. No local cluster IT has been run.

Regression coverage includes initialization overlapping cleanup, creation attempting to enter between manager close and directory removal, rejected late creation/registration, old-manager callbacks after region recreation, actual StorageEngine cleanup ordering, stopped listener admission, normal idle/batch shutdown, unexpected interruption, and the forced flush-timeout path. Existing DAL persistence/recovery and assigner tests are included. No local cluster integration test has been run.

This PR has:

  • been self-reviewed.
  • added comments/Javadoc for lifecycle and shutdown behavior.
  • added unit tests for the affected paths.

Copilot AI balanced review requested due to automatic review settings November 18, 2026 15:20

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues remain for the DAL cleanup and shutdown-interruption logging changes.

No open issues 🎉

What changed in this PR

This PR avoids unnecessary deletion-log (DAL) initialization during IoTConsensusV2 DataRegion cleanup and suppresses expected interruption warnings during normal buffer shutdown.

Changes:

  • Centralize cleanup so it closes existing managers and removes remaining DAL directories without creating a manager.
  • Suppress idle-queue interruption warnings during shutdown while retaining warnings for unexpected interruptions.
  • Add regression tests for manager cleanup, recreation, orphaned directories, and interruption logging.
File Description
iotdb-core/​datanode/​src/​test/​java/​org/​apache/​iotdb/​db/​pipe/​consensus/​deletion/​persist/​PageCacheDeletionBufferTest.java Tests expected and unexpected interruption logging.
iotdb-core/​datanode/​src/​test/​java/​org/​apache/​iotdb/​db/​pipe/​consensus/​deletion/​DeletionResourceManagerCleanupTest.java Tests cleanup without initialization, orphan removal, and manager recreation.
iotdb-core/​datanode/​src/​main/​java/​org/​apache/​iotdb/​db/​storageengine/​dataregion/​DataRegion.java Delegates DAL cleanup to the new manager helper.
iotdb-core/​datanode/​src/​main/​java/​org/​apache/​iotdb/​db/​pipe/​consensus/​deletion/​persist/​PageCacheDeletionBuffer.java Suppresses the queue-interruption warning when the buffer is closed.
iotdb-core/​datanode/​src/​main/​java/​org/​apache/​iotdb/​db/​pipe/​consensus/​deletion/​DeletionResourceManager.java Adds cleanup that avoids manager creation and preserves close-before-removal order.

Review effort: Balanced

manager.close();
removeDAL(groupId, manager.storageDir);
} else {
removeDAL(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

getInstance() builds the manager inside ConcurrentHashMap.computeIfAbsent(), and the entry is not visible until initAndRecover() returns. If region deletion reaches this helper while that initialization is in flight, get() returns null here and this branch deletes the directory the constructor is recovering; it can then finish with a live buffer pointing at a deleted DAL. Please serialize cleanup with per-region manager creation (and cover this overlap in a regression test) before deleting the directory.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in dd1a121. getInstance() and closeAndRemoveDAL() now share a stable region lifecycle lock, including the constructor/recovery and the complete close/remove sequence. Cleanup also leaves a deleted-region tombstone, cleared only by storage region creation, so late tasks cannot recreate DAL after cleanup. Added latch-controlled regressions that pause initialization before the map entry is published and pause cleanup after manager close but before directory removal; competing cleanup/creation must wait, and creation is rejected after deletion.

@luoluoyuyu

Copy link
Copy Markdown
Member

The direction is right and the change is low-risk, but it fixes the symptom rather than the root cause. Three concerns worth addressing before merge:

  1. Lifecycle ownership is still unresolved. DeletionResourceManager is lazily created in a static map, decoupled from DataRegion, so cleanup has to guess whether it exists. closeAndRemoveDAL only patches this one call site; the other getInstance() call sites (PipeInsertionDataNodeListener:157, PipeDataRegionAssigner:390, PipeHistoricalDataRegionTsFileAndDeletionSource:578/1430) are still get-or-create. A late pipe task can therefore still recreate the manager, its directory and its persist thread after cleanup. Worse, the new map.get() + delete-directory sequence is not atomic: a concurrent getInstance() between the two steps gets its directory deleted underneath it, and its FileChannel keeps writing to an unlinked inode (silent loss of deletion records). The old computeIfAbsent at least made creation atomic. Please either bind creation/close to the DataRegion lifecycle, or add getInstanceIfPresent plus a per-region tombstone so post-cleanup creation fails.

  2. Shutdown still uses interrupt as control flow; only the log was hidden. Gating the first catch on isClosed is fine, but the poll() catch at PageCacheDeletionBuffer.java:407 still logs WARN unconditionally (the message constant is even ...TAKING_WALENTRY_FROM, i.e. copy-pasted from WAL and never revisited). If close() reaches shutdownNow() after the 10s flush timeout, the interrupt can land in poll() and the warning is back. Prefer a sentinel/poison pill, or a bounded poll() that checks isClosed, so the thread exits naturally.

  3. Cleanup ordering is untouched. StorageEngine:898-899 deletes the DAL directory before releaseReceiverResource(regionId), so pipe/receiver resources can still be active during cleanup. Please stop/release the region's pipe resources first, then close the manager, then delete the directory.

@Pengzna Pengzna changed the title [Pipe] Avoid DAL initialization and expected interruption warnings during region cleanup [Pipe] Guard DAL lifecycle and drain producers before region cleanup Nov 19, 2026
@Pengzna

Pengzna commented Nov 19, 2026

Copy link
Copy Markdown
Collaborator Author

@luoluoyuyu Thanks for the detailed review. I agreed that the original patch left a lifecycle race, and revised the implementation in dd1a121:

  1. Creation/cleanup ownership: manager creation, registration, ordinary close, reference-release cleanup and close-and-remove now share a stable region lifecycle lock. Region deletion installs a tombstone before stopping producers, and close-and-remove keeps the lock through directory removal. Late get-or-create returns null, a retained closed/deleted manager returns a failed persistence result, and old-manager callbacks cannot touch a new lifecycle's DAL. Only the storage region-creation path clears the tombstone. The assigner and historical event-binding lookups now use getInstanceIfPresent; historical recovery and the local-deletion producer retain lazy creation where needed, behind the barrier. Historical extraction also checks the DataRegion's deleted state under its write lock.
  2. Shutdown: initial queue waiting and batch collection use bounded polls that observe close. After accepted deletions drain, the executor uses shutdown() and exits naturally. The existing forced-shutdown fallback remains after a flush timeout, with an explicit WARN reporting the outstanding count; unexpected interruptions still warn. The batch-poll message now refers to DeletionResource rather than WALEntry.
  3. Ordering: StorageEngine.deleteDataRegion first installs the DAL creation barrier, releases the receiver, and removes/drains the realtime assigner; it then closes the manager and removes DAL. Late listener registration is blocked until a new storage lifecycle starts. Producer shutdown runs without holding the DAL lock. I also moved the atomic reference-release hook outside the DeletionResource monitor to avoid lock inversion with persistence callbacks during close.

Added latch-controlled tests for initialization overlapping cleanup and creation attempting to enter between close and directory removal, plus coverage for late creation/registration, old callbacks after recreation, StorageEngine cleanup order, stopped listener admission, idle/batch shutdown, unexpected interruption and flush timeout.

Validation: JDK 17, all 33 tests in the six relevant classes passed with 0 failures/errors. Spotless and English/Chinese compilation passed through DataNode, ConfigNode and upstream modules. Both root reactor runs stopped later at distribution because the local server/CLI/ConfigNode 2.0.13-SNAPSHOT zip artifacts are absent, so I am not claiming a complete reactor build. No local cluster IT was run; CI for the new commit is in progress.

@jt2594838
jt2594838 merged commit d20ef86 into apache:master Nov 20, 2026
54 of 57 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants