Skip to content

[Backport] Pipe lifecycle resource cleanup - #19327

Open
Caideyipi wants to merge 2 commits into
apache:dev/1.3from
Caideyipi:codex/cp-b2a6c642-dev13
Open

Caideyipi wants to merge 2 commits into
apache:dev/1.3from
Caideyipi:codex/cp-b2a6c642-dev13

Conversation

@Caideyipi

Copy link
Copy Markdown
Collaborator

Summary

Backport the pipe lifecycle resource cleanup from #17666 (b2a6c642e533fef9e0c6e73f5cc81334db8ed03f) onto dev/1.3 at 86f93353021c8d0015f1da3e22a88e10ea8bbcc4.

  • When an in-flight event finishes after its pipe is dropped, it could recreate committer or metric state that the drop already cleaned up. Late events now leave that state cleared.
  • When a lagging schema-region follower installs a snapshot without a pipe listening queue, its old queue could stay open. Snapshot recovery now closes and clears the old queue.

Backport scope

The patch uses the existing 1.3 snapshot and pipe metadata APIs. The completion-metrics lifecycle changes are omitted because dev/1.3 has no corresponding subsystem.

Verification

  • git diff --check, Spotless, and mvn clean test-compile -pl iotdb-core/node-commons,iotdb-core/datanode -am -T 1 passed.
  • An earlier focused run passed PipeTaskAgentTest and IoTDBSinkRateLimiterTest and hit fixture errors in PipeEventCommitManagerTest. The replacement fixture compiles. Its runtime cases and the four DataNode test classes still need to run.

…17666)

* Fix

* leak_fix

* spotless

* Restore schema pipe queue from snapshots

* Pipe: Avoid recreating committer state for late commits

* Pipe: Avoid stale metrics state after drop

* Pipe: Serialize late metric creation with drop

* Declare Logback test dependency in node-commons

* Pipe: prevent stale state after pipe drop

* Pipe: balance late memory accounting and avoid passive listener recreation

* Pipe: clear shared runtime state on pipe drop

* Fix late Pipe cleanup state recreation

* Pipe: reset stale schema queues when loading snapshots

* Pipe: add schema snapshot recovery IT

* Pipe: clean unused schema listener after snapshot load

* Move snapshot recovery test to DataNode suite

(cherry picked from commit b2a6c64)
@Caideyipi

Copy link
Copy Markdown
Collaborator Author

Follow-up review of the DataNode CI failure at 9130bcd16d1a4dd6b7e8f16e7a70ee82748b1691 found that both metric sets used pipe deregistration from unbindFrom. MetricService.restartService() unbinds and then rebinds each set, so this discarded still-live pipe/operator state before metrics could be recreated. The fix removes registered metric instances during unbind while preserving active pipe/operator state for the bind phase.

Validation: Spotless and git diff --check passed. The clean targeted Maven run passed PipeMetricsLifecycleTest and PipeDataNodeSinglePipeMetricsLifecycleTest (4 tests, 0 failures/errors).

Reviewed SHA: 9130bcd16d1a4dd6b7e8f16e7a70ee82748b1691.
Final local and confirmed remote PR head: 298cc7b8f58525239d4a619e8f549f79b61ec92c.
Remote CI for the updated head is in progress.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant