Skip to content

[To dev/1.3] Fix stale query cleanup for abandoned queries and query statements in executeBatch (#19330) - #19334

Open
JackieTien97 wants to merge 2 commits into
apache:dev/1.3from
JackieTien97:ty/cp-19330-dev-1.3
Open

JackieTien97 wants to merge 2 commits into
apache:dev/1.3from
JackieTien97:ty/cp-19330-dev-1.3

Conversation

@JackieTien97

@JackieTien97 JackieTien97 commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Description

Backport of #19330 to dev/1.3. It contains two fixes for query executions that stay registered in the Coordinator after nobody can use them any more.

1. Abandoned queries were never reclaimed

A client runs a query and then neither fetches the rest of the result nor closes the statement, and no fragment instance of the query is left to fail it. Coordinator.cleanUpStaleQueries should reclaim such a query 60 s after its timeout, but it skipped every query for which IQueryExecution.isActive() returned true, and isActive() returned startTimeOfCurrentRpc == -1, which is true when no RPC is in progress. The cleanup therefore skipped exactly the abandoned queries.

  • cleanUpStaleQueries no longer depends on isActive(): every query that exceeded its timeout by more than 60 s is reclaimed.
  • isActive() returns true iff an RPC of the query is in progress, as documented. QueryExecution.cleanUpCoordinatorContextMapIfNeeded had been written against the inverted value, so its check is negated and its behaviour does not change.

What was reclaimed before is still reclaimed, and the queries without an RPC in progress are reclaimed in addition. #19330 explains why the cleanup does not start to skip the queries with an RPC in progress, and shows the effect on a two-DataNode cluster: the query was still registered and fully readable after 90 s, and is now removed 63.5 s after it started with timeout=2000.

2. Query statements of a batch were not run to their end, and not released

A statement that is a query, such as select s1 into root.db.dst(s1) from root.db.src, is sent through executeBatchStatement (JDBC Statement.executeBatch()). The handler only started it and returned its status. A batch has no result set, so nobody fetched the result, and a query only makes progress while its result is consumed. The statement was not run to its end, the batch still reported success, and the execution stayed registered until the stale query cleanup removed it.

The handler now consumes the result of such a statement and then releases its execution. querySchemaTemplate (e.g. Session.showMeasurementsInTemplate) returns its whole result in the response and now releases its execution when it returns.

One consequence to be aware of: a plain SELECT in a batch is now executed to its end before the batch returns, and its result is discarded. The time this takes is bounded by query_timeout_threshold.

Source and target

  • Source: Fix stale query cleanup for abandoned queries and query statements in executeBatch #19330, merged into master as 90e3d967f9799086df50fd33084452b9268f25c0. That commit is the squash of the two commits of the PR, 78bf43b042f34433cfa48fbc47a8707dd04586b2 and 9c3af0cfd3bcca347bba24a550a00a6a57372624, and has the same tree as the second one. The two commits were cherry-picked with -x, so the two fixes stay separate here.
  • Target base: dev/1.3 at 86f9335302.

Adaptations

Main code:

  • QueryExecution.cleanUpCoordinatorContextMapIfNeeded keeps the cleanupQueryExecution(queryId, null, t) call of dev/1.3; only its condition is negated, as in the source.
  • ClientRPCServiceImpl: the surrounding code of both handlers differs on dev/1.3 (no table model branch, the query id is requested at one place), so the changes were re-applied by hand on the dev/1.3 file: the consumeResult helper, consuming the result and releasing the execution per statement in executeBatchStatement, and releasing the execution in executeTemplateQueryStatement.

Tests:

  • CoordinatorTest, QueryExecutionLifecycleTest and IoTDBShowQueriesIT do not exist on dev/1.3. They are added with the tests of this change only, and IoTDBShowQueriesIT with the small fixture these tests need. They are adapted to the Coordinator.execution and SessionInfo signatures, the QueryType values and the ColumnHeaderConstant package of dev/1.3.
  • IoTDBExecuteBatchIT starts a cluster per test on dev/1.3, so the new test has no cleanup block.

Not ported, because they depend on something dev/1.3 does not have:

  • the assertions on the UDF query scope of an execution (dev/1.3 has no UDF query scope);
  • the assertions on the QUERY_TIMEOUT status of a reclaimed query: stopAndCleanup() does not move the query state machine on dev/1.3, so that status stays SUCCESS there. The ported tests assert that the query is removed from the Coordinator. The next section shows what this means for a client;
  • IoTDBInsertQueryIT.testInsertQueryInBatch (table model).

What a client sees when its query is reclaimed on dev/1.3

getStatus() of a reclaimed execution keeps reporting SUCCESS on dev/1.3, because QueryExecution.stopAndCleanup() does not move the query state machine there (master does since #19224). Only the first RPC of a query reads that status. Every later RPC looks the query up in the Coordinator and reports 715 when it is gone.

Measured on a cluster with two DataNodes, with dev/1.3 at 86f9335302 and with this branch. The queries are sent to DataNode 1 with a timeout of 2 s:

Query dev/1.3 This branch
20000 rows on DataNode 2, left alone after the first batch not reclaimed: still registered 84 s later, and the client then reads all 20000 rows reclaimed after 63.5 s; the client reads the 4096 rows it had received and then gets 715: Query is not found, it may be killed by others, timeout or some other runtime errors, ...
20000 rows on DataNode 1, left alone after the first batch no longer registered when the client comes back; it reads 4096 rows and then gets 715 the same
first RPC still dispatching when the cleanup runs (DataNode 2 suspended for 75 s) reclaimed after 63.0 s and 63.9 s (two runs); when DataNode 2 continues, the RPC returns SUCCESS without a result set. Statement.executeQuery throws SQLException: Cannot invoke "java.util.List.size()" because "columnNameList" is null, Session.executeQueryStatement throws a NullPointerException with the same message the same, reclaimed after 62.4 s
  • The queries this PR starts to reclaim (first row) end with an explicit 715 on the next fetch. No result is cut short without an error.
  • The last row is where the SUCCESS status reaches a client. It is existing behaviour of dev/1.3 and is not changed here: the old check skipped only the queries without an RPC in progress, so a query whose first RPC is in progress was reclaimed before this PR as well. On master the same run ends with 720: Current query is time out, ... for both clients, because [Query] Fix scheduler stop races and query attempt context leaks #19224 made stopAndCleanup() fail the query state machine. That change is not part of Fix stale query cleanup for abandoned queries and query statements in executeBatch #19330 and is not included in this backport. The run needed dn_connection_timeout_ms=300000 to keep the dispatch RPC waiting for 75 s. With the default of 60 s the dispatch RPC times out before the cleanup runs, and both clients get 720: Current query is time out, ... after 60.0 s (measured on this branch).

Validation on dev/1.3

All commands were run offline on this branch with JDK 21.

Unit tests:

mvn -o test -pl iotdb-core/datanode -am \
  -Dtest=CoordinatorTest,QueryExecutionLifecycleTest,ConfigExecutionTest,QueryExecutionRetryTest \
  -Dsurefire.failIfNoSpecifiedTests=false
  • With the four main files of the first commit as on dev/1.3: 15 tests, 3 failures:
    • CoordinatorTest.testStaleQueryWithoutRpcInProgressIsCleanedUp:97 expected null, but was:<QueryExecution[...]>
    • ConfigExecutionTest.isActiveTellsWhetherAnRpcOfTheQueryIsInProgress:169
    • QueryExecutionLifecycleTest.isActiveTellsWhetherAnRpcOfTheQueryIsInProgress:48
  • With this branch: 15 tests, 0 failures, with the spotless and checkstyle checks of the build enabled.

Integration tests:

mvn -o clean verify -DskipUTs '-Dit.test=IoTDBShowQueriesIT,IoTDBExecuteBatchIT' \
  -DfailIfNoTests=false -Dfailsafe.failIfNoSpecifiedTests=false \
  -pl integration-test -am -P with-integration-tests
  • With ClientRPCServiceImpl as on dev/1.3, the three new tests fail:
    • IoTDBExecuteBatchIT.testJDBCExecuteBatchWithSelectInto:105 expected:<2> but was:<0>
    • IoTDBShowQueriesIT.queryInBatchReleaseTest: expected:<[]> but was:<[20261119_125950_00007_1]>
    • IoTDBShowQueriesIT.schemaTemplateQueryReleaseTest: expected:<[]> but was:<[20261119_125950_00012_1]>
  • With this branch: IoTDBShowQueriesIT (2 tests) and IoTDBExecuteBatchIT (3 tests) pass with the default profile (1C1D) and with -PClusterIT (1C3D).
  • mvn spotless:check -pl iotdb-core/datanode,integration-test -P with-integration-tests passes.
  • The table above comes from a real 1C2D cluster of each build (query_timeout_threshold=5000, dn_connection_timeout_ms=300000), driven through JDBC and the Session API. The batch measurements of Fix stale query cleanup for abandoned queries and query statements in executeBatch #19330 on two DataNodes were not repeated on dev/1.3; the integration tests above cover that fix.

This PR has:

  • been self-reviewed.
  • added comments explaining the "why" and the intent of the code wherever would not be obvious
    for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold
    for code coverage.
  • added integration tests.

Key changed/added classes (or packages if there are too many classes) in this PR
  • iotdb-core/datanode: org.apache.iotdb.db.queryengine.plan.Coordinator, org.apache.iotdb.db.queryengine.plan.execution.QueryExecution, org.apache.iotdb.db.queryengine.plan.execution.config.ConfigExecution, org.apache.iotdb.db.protocol.thrift.impl.ClientRPCServiceImpl
  • iotdb-core/datanode (test): CoordinatorTest, ConfigExecutionTest, QueryExecutionLifecycleTest
  • integration-test: IoTDBShowQueriesIT, IoTDBExecuteBatchIT

🤖 Generated with Claude Code

Trigger: a client runs a query and then neither fetches the rest of the
result nor closes the statement, and no fragment instance of the query is
left to fail it. Reproduced on a 1C2D cluster with a query sent to
DataNode 1 whose data is on DataNode 2 (20000 rows, timeout 2s): the client
reads the first batch and then stays idle.

Wrong behaviour: Coordinator.cleanUpStaleQueries runs every 2s to reclaim
the queries that exceeded their timeout by more than 60s, and it skipped
every query for which IQueryExecution.isActive() returned true. isActive()
is documented as "there is an active RPC for this query", but QueryExecution
and ConfigExecution returned startTimeOfCurrentRpc == -1, i.e. true when no
RPC is in progress. The cleanup therefore skipped exactly the abandoned
queries and only reclaimed queries with an RPC in progress. The query above
was still registered in the Coordinator 90s later, and the client could
still fetch all of its rows.

Corrected behaviour:
- cleanUpStaleQueries no longer depends on isActive(): every query that
  exceeded its timeout by more than 60s is reclaimed. The query above is
  cleaned up with a QueryTimeoutRuntimeException 63s after it started, and a
  later fetch is told that the query is not found (715).
- isActive() returns true iff an RPC of the query is in progress, as
  documented. QueryExecution.cleanUpCoordinatorContextMapIfNeeded had been
  written against the inverted value, so its check is negated and its
  behaviour is unchanged.

The cleanup deliberately does not start to skip the queries with an RPC in
progress, which is what the isActive() check was meant to do, because the
cleanup of such queries is relied on today:
- executeBatchStatement (for a query statement) and querySchemaTemplate
  register an execution that they never release, and the end of their RPC
  is never recorded. The stale query cleanup is what releases these
  executions (measured: 65-66s after start with a 5s timeout, before and after
  this change). Skipping them would keep them forever.
- an RPC that is still in progress 60s after the timeout is stuck, e.g. in
  a dispatch that does not return, and stopping the query is what unblocks
  it (QueryTerminalCleanupTest).
So what was reclaimed before is still reclaimed, and the queries without an
RPC in progress are reclaimed in addition.

(cherry picked from commit 78bf43b)

Backport of apache#19330 to dev/1.3:
- QueryExecution.cleanUpCoordinatorContextMapIfNeeded keeps the
  cleanupQueryExecution(queryId, null, t) call of dev/1.3; only its
  condition is negated, as in the source commit.
- CoordinatorTest and QueryExecutionLifecycleTest do not exist on
  dev/1.3. They are added with the tests of this change only, adapted to
  the Coordinator.execution and SessionInfo signatures and to the
  QueryType values of dev/1.3.
- Not ported: the assertions on the UDF query scope of an execution and
  on the QUERY_TIMEOUT status of a reclaimed query. dev/1.3 has no UDF
  query scope, and its stopAndCleanup() does not move the query state
  machine, so the status of a reclaimed query stays SUCCESS there. The
  ported tests assert that the query is removed from the Coordinator.
…them

Trigger 1: a statement that is a query is sent through
executeBatchStatement (JDBC Statement.executeBatch), for example
  select s1 into root.db.dst(s1) from root.db.src     (tree model)
  INSERT INTO dst SELECT * FROM src                   (table model)

Wrong behaviour: the handler only started such a statement and returned
its status. A batch has no result set, so nobody fetched the result of
the statement, and a query only makes progress while its result is
consumed:
- with the source data on the DataNode that received the batch, the
  statement wrote nothing (0 of 2000 rows) or only a part (200 of the
  1200 rows of IoTDBInsertQueryIT), and the batch reported success;
- with the source data on another DataNode, the rows were written about
  0.5s after the batch had returned.
In both cases the execution stayed registered in the Coordinator, and
listed by SHOW QUERIES, until the stale query cleanup removed it 60s
after its timeout.

Corrected behaviour: the handler consumes the result of such a statement
and then releases its execution. The statement has been run to its end
when the batch returns, a failure while it runs is reported in the status
of the statement, and nothing stays registered.

Trigger 2: a schema template query (querySchemaTemplate, for example
Session.showMeasurementsInTemplate).

Wrong behaviour: the handler returns the whole result in its response,
but it left the execution registered until the stale query cleanup
removed it 60s after its timeout.

Corrected behaviour: the handler releases the execution when it returns.

(cherry picked from commit 9c3af0c)

Backport of apache#19330 to dev/1.3:
- ClientRPCServiceImpl: the code around both handlers differs on dev/1.3
  (no table model branch, the query id is requested at one place), so
  the changes were re-applied by hand on the dev/1.3 file: the
  consumeResult helper, consuming the result and releasing the execution
  per statement in executeBatchStatement, and releasing the execution in
  executeTemplateQueryStatement.
- IoTDBShowQueriesIT does not exist on dev/1.3. It is added with the two
  tests of this change and the small fixture they need, and uses the
  ColumnHeaderConstant of dev/1.3.
- IoTDBExecuteBatchIT starts a cluster per test on dev/1.3, so the new
  test has no cleanup block.
- Not ported: IoTDBInsertQueryIT.testInsertQueryInBatch and the table
  model example above. dev/1.3 has no table model.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant