Skip to content

Fix stale query cleanup for abandoned queries and query statements in executeBatch - #19330

Merged
JackieTien97 merged 2 commits into
apache:masterfrom
JackieTien97:ty/fix-stale-query-cleanup
Nov 20, 2026
Merged

JackieTien97 merged 2 commits into
apache:masterfrom
JackieTien97:ty/fix-stale-query-cleanup

Conversation

@JackieTien97

Copy link
Copy Markdown
Contributor

Description

Two fixes for query executions that stay registered in the Coordinator after nobody can use them any more.

1. Abandoned queries were never reclaimed

Scenario. A client runs a query and then neither fetches the rest of the result nor closes the statement, and no fragment instance of the query is left to fail it. For example, the query is sent to one DataNode, its data is on another one, and the rest of the result is already buffered on the DataNode that received the query.

Wrong behaviour. Coordinator.cleanUpStaleQueries runs every 2 s to reclaim the queries that exceeded their timeout by more than 60 s. It skipped every query for which IQueryExecution.isActive() returned true. isActive() is documented as "there is an active RPC for this query", but QueryExecution and ConfigExecution returned startTimeOfCurrentRpc == -1, which is true when no RPC is in progress. The cleanup therefore skipped exactly the abandoned queries, and they stayed registered until the statement or the session was closed.

Corrected behaviour.

  • cleanUpStaleQueries no longer depends on isActive(). Every query that exceeded its timeout by more than 60 s is reclaimed: it is removed from the Coordinator and cleaned up with a QueryTimeoutRuntimeException.
  • isActive() returns true iff an RPC of the query is in progress, as documented. QueryExecution.cleanUpCoordinatorContextMapIfNeeded had been written against the inverted value, so its check is negated and its behaviour does not change.

The alternative is to keep the check and let the cleanup skip the queries with an RPC in progress, which is what the isActive() check was meant to do. I did not do that, because such queries are reclaimed today and that is relied on:

  • an RPC that is still in progress 60 s after the timeout is stuck, e.g. in a dispatch that does not return, and stopping the query is what unblocks it (QueryTerminalCleanupTest.staleQueryCleanupDuringDispatchMustUnblockGetStatus);
  • a caller that never reports the end of its RPC through Coordinator.recordExecutionTime would keep its query forever. The two handlers of the second fix were such callers.

So what was reclaimed before is still reclaimed, and the queries without an RPC in progress are reclaimed in addition.

Example. 1 ConfigNode and 2 DataNodes. select s1 from root.remote.d1 (20000 rows stored on DataNode 2) is sent to DataNode 1 with timeout=2000; the client reads the first batch and then stays idle.

before after
the query in the Coordinator of DataNode 1 still registered 90 s later removed 63.5 s after it started (Cleaning up stale query ... timeout duration is: 2000ms)
the client continues reading after 90 s reads all 20000 rows 715: Query is not found, it may be killed by others, timeout or some other runtime errors, ...

This is visible to a client that pauses for longer than the timeout plus 60 s in the middle of a result: its next fetch now fails, while it used to succeed as long as the rest of the result was already buffered. (The same query on data of DataNode 1, whose fragment instance was still waiting to send its result, was released within 10 s before and after this change.)

2. Query statements of a batch were not run to their end, and not released

Scenario. A statement that is a query is sent through executeBatchStatement (JDBC Statement.executeBatch()), for example select s1 into root.bdst.d1(s1) from root.bsrc.d1 in the tree model or INSERT INTO dst SELECT * FROM src in the table model.

Wrong behaviour. The handler only started such a statement and returned its status. A batch has no result set, so nobody fetched the result of the statement, and a query only makes progress while its result is consumed. The statement was therefore not run to its end, the batch still reported success, and the execution stayed registered (and listed by SHOW QUERIES) until the stale query cleanup removed it 60 s after its timeout.

Corrected behaviour. The handler consumes the result of such a statement and then releases its execution. The statement has been run to its end when the batch returns, a failure while it runs is reported in the status of that statement, and nothing stays registered.

querySchemaTemplate (e.g. Session.showMeasurementsInTemplate) had the second half of the problem: it returns its whole result in the response, but left the execution registered until the stale query cleanup removed it. It now releases the execution when it returns.

Example. Same cluster, source tables and series with 2000 rows, query_timeout_threshold=5000.

statement sent in a batch before after
tree SELECT ... INTO, source data on the DataNode that receives the batch status 200, 0 rows written, the query is aborted at its timeout status 200, 2000 rows written when the batch returns
tree SELECT ... INTO, source data on the other DataNode status 200, 0 rows when the batch returns, 2000 rows about 0.5 s later status 200, 2000 rows written when the batch returns
table INSERT ... SELECT, both placements as the two rows above status 200, 2000 rows written when the batch returns
any of them, and a plain SELECT execution still registered after the batch returned, until timeout + 60 s (65-66 s measured for a plain SELECT) no execution registered when the batch returns
Session.showMeasurementsInTemplate execution registered for 65-66 s no execution registered when the request returns

With several data regions the statement could also be executed partly: the new IoTDBInsertQueryIT.testInsertQueryInBatch found 200 of 1200 rows before the fix.

One consequence to be aware of: a plain SELECT in a batch is now executed to its end before the batch returns, and its result is discarded. It used to be started and then left alone. The time this takes is bounded by query_timeout_threshold.

Tests

  • Unit tests for the cleanup: CoordinatorTest (stale queries with and without an RPC in progress, inside the grace period, without a positive timeout, and a killed query), ConfigExecutionTest and QueryExecutionLifecycleTest (isActive()). On the main code of master, 3 of them fail: testStaleQueryWithoutRpcInProgressIsCleanedUp (expected null, but was:<QueryExecution[...]>) and the two isActiveTellsWhetherAnRpcOfTheQueryIsInProgress.

  • Integration tests, all failing before the second fix:

    • IoTDBShowQueriesIT.queryInBatchReleaseTest and schemaTemplateQueryReleaseTest: expected:<[]> but was:<[20261119_122523_00011_1]> (the id of the query that was left registered)
    • IoTDBExecuteBatchIT.testJDBCExecuteBatchWithSelectInto: expected:<2> but was:<0>
    • IoTDBInsertQueryIT.testInsertQueryInBatch: expected:<1200> but was:<200>

    With the fix, IoTDBShowQueriesIT and IoTDBExecuteBatchIT pass with the default SimpleIT profile and with -PClusterIT, and IoTDBInsertQueryIT with -PTableSimpleIT and -PTableClusterIT.

  • The numbers in the two tables were measured on a 1C2D cluster built from master and from this branch.


This PR has:

  • been self-reviewed.
  • added comments explaining the "why" and the intent of the code wherever would not be obvious
    for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold
    for code coverage.
  • added integration tests.
  • been tested in a test IoTDB cluster.

Key changed/added classes (or packages if there are too many classes) in this PR
  • iotdb-core/datanode: org.apache.iotdb.db.queryengine.plan.Coordinator, org.apache.iotdb.db.queryengine.plan.execution.QueryExecution, org.apache.iotdb.db.queryengine.plan.execution.config.ConfigExecution, org.apache.iotdb.db.protocol.thrift.impl.ClientRPCServiceImpl
  • iotdb-core/datanode (test): CoordinatorTest, ConfigExecutionTest, QueryExecutionLifecycleTest
  • integration-test: IoTDBShowQueriesIT, IoTDBExecuteBatchIT, IoTDBInsertQueryIT

🤖 Generated with Claude Code

Trigger: a client runs a query and then neither fetches the rest of the
result nor closes the statement, and no fragment instance of the query is
left to fail it. Reproduced on a 1C2D cluster with a query sent to
DataNode 1 whose data is on DataNode 2 (20000 rows, timeout 2s): the client
reads the first batch and then stays idle.

Wrong behaviour: Coordinator.cleanUpStaleQueries runs every 2s to reclaim
the queries that exceeded their timeout by more than 60s, and it skipped
every query for which IQueryExecution.isActive() returned true. isActive()
is documented as "there is an active RPC for this query", but QueryExecution
and ConfigExecution returned startTimeOfCurrentRpc == -1, i.e. true when no
RPC is in progress. The cleanup therefore skipped exactly the abandoned
queries and only reclaimed queries with an RPC in progress. The query above
was still registered in the Coordinator 90s later, and the client could
still fetch all of its rows.

Corrected behaviour:
- cleanUpStaleQueries no longer depends on isActive(): every query that
  exceeded its timeout by more than 60s is reclaimed. The query above is
  cleaned up with a QueryTimeoutRuntimeException 63s after it started, and a
  later fetch is told that the query is not found (715).
- isActive() returns true iff an RPC of the query is in progress, as
  documented. QueryExecution.cleanUpCoordinatorContextMapIfNeeded had been
  written against the inverted value, so its check is negated and its
  behaviour is unchanged.

The cleanup deliberately does not start to skip the queries with an RPC in
progress, which is what the isActive() check was meant to do, because the
cleanup of such queries is relied on today:
- executeBatchStatement (for a query statement) and querySchemaTemplate
  register an execution that they never release, and the end of their RPC
  is never recorded. The stale query cleanup is what releases these
  executions (measured: 65-66s after start with a 5s timeout, before and after
  this change). Skipping them would keep them forever.
- an RPC that is still in progress 60s after the timeout is stuck, e.g. in
  a dispatch that does not return, and stopping the query is what unblocks
  it (QueryTerminalCleanupTest).
So what was reclaimed before is still reclaimed, and the queries without an
RPC in progress are reclaimed in addition.
…them

Trigger 1: a statement that is a query is sent through
executeBatchStatement (JDBC Statement.executeBatch), for example
  select s1 into root.db.dst(s1) from root.db.src     (tree model)
  INSERT INTO dst SELECT * FROM src                   (table model)

Wrong behaviour: the handler only started such a statement and returned
its status. A batch has no result set, so nobody fetched the result of
the statement, and a query only makes progress while its result is
consumed:
- with the source data on the DataNode that received the batch, the
  statement wrote nothing (0 of 2000 rows) or only a part (200 of the
  1200 rows of IoTDBInsertQueryIT), and the batch reported success;
- with the source data on another DataNode, the rows were written about
  0.5s after the batch had returned.
In both cases the execution stayed registered in the Coordinator, and
listed by SHOW QUERIES, until the stale query cleanup removed it 60s
after its timeout.

Corrected behaviour: the handler consumes the result of such a statement
and then releases its execution. The statement has been run to its end
when the batch returns, a failure while it runs is reported in the status
of the statement, and nothing stays registered.

Trigger 2: a schema template query (querySchemaTemplate, for example
Session.showMeasurementsInTemplate).

Wrong behaviour: the handler returns the whole result in its response,
but it left the execution registered until the stale query cleanup
removed it 60s after its timeout.

Corrected behaviour: the handler releases the execution when it returns.
@JackieTien97
JackieTien97 merged commit 90e3d96 into apache:master Nov 20, 2026
38 of 39 checks passed
@JackieTien97
JackieTien97 deleted the ty/fix-stale-query-cleanup branch November 20, 2026 01:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant