Repository navigation
Fix stale query cleanup for abandoned queries and query statements in executeBatch - #19330
Merged
JackieTien97 merged 2 commits intoNov 20, 2026
Merged
Conversation
Trigger: a client runs a query and then neither fetches the rest of the result nor closes the statement, and no fragment instance of the query is left to fail it. Reproduced on a 1C2D cluster with a query sent to DataNode 1 whose data is on DataNode 2 (20000 rows, timeout 2s): the client reads the first batch and then stays idle. Wrong behaviour: Coordinator.cleanUpStaleQueries runs every 2s to reclaim the queries that exceeded their timeout by more than 60s, and it skipped every query for which IQueryExecution.isActive() returned true. isActive() is documented as "there is an active RPC for this query", but QueryExecution and ConfigExecution returned startTimeOfCurrentRpc == -1, i.e. true when no RPC is in progress. The cleanup therefore skipped exactly the abandoned queries and only reclaimed queries with an RPC in progress. The query above was still registered in the Coordinator 90s later, and the client could still fetch all of its rows. Corrected behaviour: - cleanUpStaleQueries no longer depends on isActive(): every query that exceeded its timeout by more than 60s is reclaimed. The query above is cleaned up with a QueryTimeoutRuntimeException 63s after it started, and a later fetch is told that the query is not found (715). - isActive() returns true iff an RPC of the query is in progress, as documented. QueryExecution.cleanUpCoordinatorContextMapIfNeeded had been written against the inverted value, so its check is negated and its behaviour is unchanged. The cleanup deliberately does not start to skip the queries with an RPC in progress, which is what the isActive() check was meant to do, because the cleanup of such queries is relied on today: - executeBatchStatement (for a query statement) and querySchemaTemplate register an execution that they never release, and the end of their RPC is never recorded. The stale query cleanup is what releases these executions (measured: 65-66s after start with a 5s timeout, before and after this change). Skipping them would keep them forever. - an RPC that is still in progress 60s after the timeout is stuck, e.g. in a dispatch that does not return, and stopping the query is what unblocks it (QueryTerminalCleanupTest). So what was reclaimed before is still reclaimed, and the queries without an RPC in progress are reclaimed in addition.
…them Trigger 1: a statement that is a query is sent through executeBatchStatement (JDBC Statement.executeBatch), for example select s1 into root.db.dst(s1) from root.db.src (tree model) INSERT INTO dst SELECT * FROM src (table model) Wrong behaviour: the handler only started such a statement and returned its status. A batch has no result set, so nobody fetched the result of the statement, and a query only makes progress while its result is consumed: - with the source data on the DataNode that received the batch, the statement wrote nothing (0 of 2000 rows) or only a part (200 of the 1200 rows of IoTDBInsertQueryIT), and the batch reported success; - with the source data on another DataNode, the rows were written about 0.5s after the batch had returned. In both cases the execution stayed registered in the Coordinator, and listed by SHOW QUERIES, until the stale query cleanup removed it 60s after its timeout. Corrected behaviour: the handler consumes the result of such a statement and then releases its execution. The statement has been run to its end when the batch returns, a failure while it runs is reported in the status of the statement, and nothing stays registered. Trigger 2: a schema template query (querySchemaTemplate, for example Session.showMeasurementsInTemplate). Wrong behaviour: the handler returns the whole result in its response, but it left the execution registered until the stale query cleanup removed it 60s after its timeout. Corrected behaviour: the handler releases the execution when it returns.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Two fixes for query executions that stay registered in the
Coordinatorafter nobody can use them any more.1. Abandoned queries were never reclaimed
Scenario. A client runs a query and then neither fetches the rest of the result nor closes the statement, and no fragment instance of the query is left to fail it. For example, the query is sent to one DataNode, its data is on another one, and the rest of the result is already buffered on the DataNode that received the query.
Wrong behaviour.
Coordinator.cleanUpStaleQueriesruns every 2 s to reclaim the queries that exceeded their timeout by more than 60 s. It skipped every query for whichIQueryExecution.isActive()returned true.isActive()is documented as "there is an active RPC for this query", butQueryExecutionandConfigExecutionreturnedstartTimeOfCurrentRpc == -1, which is true when no RPC is in progress. The cleanup therefore skipped exactly the abandoned queries, and they stayed registered until the statement or the session was closed.Corrected behaviour.
cleanUpStaleQueriesno longer depends onisActive(). Every query that exceeded its timeout by more than 60 s is reclaimed: it is removed from theCoordinatorand cleaned up with aQueryTimeoutRuntimeException.isActive()returns true iff an RPC of the query is in progress, as documented.QueryExecution.cleanUpCoordinatorContextMapIfNeededhad been written against the inverted value, so its check is negated and its behaviour does not change.The alternative is to keep the check and let the cleanup skip the queries with an RPC in progress, which is what the
isActive()check was meant to do. I did not do that, because such queries are reclaimed today and that is relied on:QueryTerminalCleanupTest.staleQueryCleanupDuringDispatchMustUnblockGetStatus);Coordinator.recordExecutionTimewould keep its query forever. The two handlers of the second fix were such callers.So what was reclaimed before is still reclaimed, and the queries without an RPC in progress are reclaimed in addition.
Example. 1 ConfigNode and 2 DataNodes.
select s1 from root.remote.d1(20000 rows stored on DataNode 2) is sent to DataNode 1 withtimeout=2000; the client reads the first batch and then stays idle.Coordinatorof DataNode 1Cleaning up stale query ... timeout duration is: 2000ms)715: Query is not found, it may be killed by others, timeout or some other runtime errors, ...This is visible to a client that pauses for longer than the timeout plus 60 s in the middle of a result: its next fetch now fails, while it used to succeed as long as the rest of the result was already buffered. (The same query on data of DataNode 1, whose fragment instance was still waiting to send its result, was released within 10 s before and after this change.)
2. Query statements of a batch were not run to their end, and not released
Scenario. A statement that is a query is sent through
executeBatchStatement(JDBCStatement.executeBatch()), for exampleselect s1 into root.bdst.d1(s1) from root.bsrc.d1in the tree model orINSERT INTO dst SELECT * FROM srcin the table model.Wrong behaviour. The handler only started such a statement and returned its status. A batch has no result set, so nobody fetched the result of the statement, and a query only makes progress while its result is consumed. The statement was therefore not run to its end, the batch still reported success, and the execution stayed registered (and listed by
SHOW QUERIES) until the stale query cleanup removed it 60 s after its timeout.Corrected behaviour. The handler consumes the result of such a statement and then releases its execution. The statement has been run to its end when the batch returns, a failure while it runs is reported in the status of that statement, and nothing stays registered.
querySchemaTemplate(e.g.Session.showMeasurementsInTemplate) had the second half of the problem: it returns its whole result in the response, but left the execution registered until the stale query cleanup removed it. It now releases the execution when it returns.Example. Same cluster, source tables and series with 2000 rows,
query_timeout_threshold=5000.SELECT ... INTO, source data on the DataNode that receives the batchSELECT ... INTO, source data on the other DataNodeINSERT ... SELECT, both placementsSELECTSELECT)Session.showMeasurementsInTemplateWith several data regions the statement could also be executed partly: the new
IoTDBInsertQueryIT.testInsertQueryInBatchfound 200 of 1200 rows before the fix.One consequence to be aware of: a plain
SELECTin a batch is now executed to its end before the batch returns, and its result is discarded. It used to be started and then left alone. The time this takes is bounded byquery_timeout_threshold.Tests
Unit tests for the cleanup:
CoordinatorTest(stale queries with and without an RPC in progress, inside the grace period, without a positive timeout, and a killed query),ConfigExecutionTestandQueryExecutionLifecycleTest(isActive()). On the main code of master, 3 of them fail:testStaleQueryWithoutRpcInProgressIsCleanedUp(expected null, but was:<QueryExecution[...]>) and the twoisActiveTellsWhetherAnRpcOfTheQueryIsInProgress.Integration tests, all failing before the second fix:
IoTDBShowQueriesIT.queryInBatchReleaseTestandschemaTemplateQueryReleaseTest:expected:<[]> but was:<[20261119_122523_00011_1]>(the id of the query that was left registered)IoTDBExecuteBatchIT.testJDBCExecuteBatchWithSelectInto:expected:<2> but was:<0>IoTDBInsertQueryIT.testInsertQueryInBatch:expected:<1200> but was:<200>With the fix,
IoTDBShowQueriesITandIoTDBExecuteBatchITpass with the defaultSimpleITprofile and with-PClusterIT, andIoTDBInsertQueryITwith-PTableSimpleITand-PTableClusterIT.The numbers in the two tables were measured on a 1C2D cluster built from master and from this branch.
This PR has:
for an unfamiliar reader.
for code coverage.
Key changed/added classes (or packages if there are too many classes) in this PR
iotdb-core/datanode:org.apache.iotdb.db.queryengine.plan.Coordinator,org.apache.iotdb.db.queryengine.plan.execution.QueryExecution,org.apache.iotdb.db.queryengine.plan.execution.config.ConfigExecution,org.apache.iotdb.db.protocol.thrift.impl.ClientRPCServiceImpliotdb-core/datanode(test):CoordinatorTest,ConfigExecutionTest,QueryExecutionLifecycleTestintegration-test:IoTDBShowQueriesIT,IoTDBExecuteBatchIT,IoTDBInsertQueryIT🤖 Generated with Claude Code