Repository navigation
[Pipe] Guard DAL lifecycle and drain producers before region cleanup - #19321
Conversation
… expected close warnings
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues remain for the DAL cleanup and shutdown-interruption logging changes.
No open issues 🎉
What changed in this PR
This PR avoids unnecessary deletion-log (DAL) initialization during IoTConsensusV2 DataRegion cleanup and suppresses expected interruption warnings during normal buffer shutdown.
Changes:
- Centralize cleanup so it closes existing managers and removes remaining DAL directories without creating a manager.
- Suppress idle-queue interruption warnings during shutdown while retaining warnings for unexpected interruptions.
- Add regression tests for manager cleanup, recreation, orphaned directories, and interruption logging.
| File | Description |
|---|---|
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/pipe/consensus/deletion/persist/PageCacheDeletionBufferTest.java |
Tests expected and unexpected interruption logging. |
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/pipe/consensus/deletion/DeletionResourceManagerCleanupTest.java |
Tests cleanup without initialization, orphan removal, and manager recreation. |
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/DataRegion.java |
Delegates DAL cleanup to the new manager helper. |
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/consensus/deletion/persist/PageCacheDeletionBuffer.java |
Suppresses the queue-interruption warning when the buffer is closed. |
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/consensus/deletion/DeletionResourceManager.java |
Adds cleanup that avoids manager creation and preserves close-before-removal order. |
Review effort: Balanced
| manager.close(); | ||
| removeDAL(groupId, manager.storageDir); | ||
| } else { | ||
| removeDAL( |
There was a problem hiding this comment.
getInstance() builds the manager inside ConcurrentHashMap.computeIfAbsent(), and the entry is not visible until initAndRecover() returns. If region deletion reaches this helper while that initialization is in flight, get() returns null here and this branch deletes the directory the constructor is recovering; it can then finish with a live buffer pointing at a deleted DAL. Please serialize cleanup with per-region manager creation (and cover this overlap in a regression test) before deleting the directory.
There was a problem hiding this comment.
Fixed in dd1a121. getInstance() and closeAndRemoveDAL() now share a stable region lifecycle lock, including the constructor/recovery and the complete close/remove sequence. Cleanup also leaves a deleted-region tombstone, cleared only by storage region creation, so late tasks cannot recreate DAL after cleanup. Added latch-controlled regressions that pause initialization before the map entry is published and pause cleanup after manager close but before directory removal; competing cleanup/creation must wait, and creation is rejected after deletion.
|
The direction is right and the change is low-risk, but it fixes the symptom rather than the root cause. Three concerns worth addressing before merge:
|
|
@luoluoyuyu Thanks for the detailed review. I agreed that the original patch left a lifecycle race, and revised the implementation in dd1a121:
Added latch-controlled tests for initialization overlapping cleanup and creation attempting to enter between close and directory removal, plus coverage for late creation/registration, old callbacks after recreation, StorageEngine cleanup order, stopped listener admission, idle/batch shutdown, unexpected interruption and flush timeout. Validation: JDK 17, all 33 tests in the six relevant classes passed with 0 failures/errors. Spotless and English/Chinese compilation passed through DataNode, ConfigNode and upstream modules. Both root reactor runs stopped later at |
Description
Deleting a DataRegion under IoTConsensusV2 currently obtains its deletion resource manager through a get-or-create API. Cleanup can allocate a DAL buffer, open a file and start a persistence thread just to close them. Normal close then interrupts an idle queue wait and emits a misleading WARN. Cleanup also needs to exclude concurrent initialization and late pipe work so it cannot remove files used by a live manager.
For example, with IoTConsensusV2 configured:
Before this change, cleanup can initialize DAL resources and emit
Interrupted when waiting for taking DeletionResource from blocking queue to serialize.despite no SQL DELETE having been issued. A concurrent or late get-or-create can also leave a manager using a removed directory. After this change, cleanup never constructs a manager, creation and close/removal share a region lifecycle lock, and a deleted-region tombstone prevents recreation until storage starts a new region lifecycle. An idle buffer or one collecting a batch exits normally after draining; a real flush timeout remains visible as a warning.Implementation
getInstanceIfPresentfor event-binding lookups. Keep lazy creation where historical DAL recovery or a new local deletion needs it, guarded by the lifecycle barrier; historical extraction also checks the DataRegion's deleted state under its write lock.Validation
mvn test -Dtest=DeletionResourceManagerCleanupTest,PageCacheDeletionBufferTest,DeletionResourceTest,DeletionRecoverTest,PipeInsertionDataNodeListenerTest,PipeDataRegionAssignerTest -Dsurefire.failIfNoSpecifiedTests=false -DfailIfNoTests=falseandmvn test-compile -P with-zh-locale -DskipTests) stopped later atdistributiondependency resolution: the localiotdb-server,iotdb-cliandiotdb-confignode2.0.13-SNAPSHOT zip artifacts are absent. Thus this is not a full-reactor build success. No local cluster IT has been run.Regression coverage includes initialization overlapping cleanup, creation attempting to enter between manager close and directory removal, rejected late creation/registration, old-manager callbacks after region recreation, actual StorageEngine cleanup ordering, stopped listener admission, normal idle/batch shutdown, unexpected interruption, and the forced flush-timeout path. Existing DAL persistence/recovery and assigner tests are included. No local cluster integration test has been run.
This PR has: