Repository navigation
[To dev/1.3] Fix stale query cleanup for abandoned queries and query statements in executeBatch (#19330) - #19334
Open
JackieTien97 wants to merge 2 commits into
Open
JackieTien97 wants to merge 2 commits into
JackieTien97 wants to merge 2 commits into
Conversation
Trigger: a client runs a query and then neither fetches the rest of the result nor closes the statement, and no fragment instance of the query is left to fail it. Reproduced on a 1C2D cluster with a query sent to DataNode 1 whose data is on DataNode 2 (20000 rows, timeout 2s): the client reads the first batch and then stays idle. Wrong behaviour: Coordinator.cleanUpStaleQueries runs every 2s to reclaim the queries that exceeded their timeout by more than 60s, and it skipped every query for which IQueryExecution.isActive() returned true. isActive() is documented as "there is an active RPC for this query", but QueryExecution and ConfigExecution returned startTimeOfCurrentRpc == -1, i.e. true when no RPC is in progress. The cleanup therefore skipped exactly the abandoned queries and only reclaimed queries with an RPC in progress. The query above was still registered in the Coordinator 90s later, and the client could still fetch all of its rows. Corrected behaviour: - cleanUpStaleQueries no longer depends on isActive(): every query that exceeded its timeout by more than 60s is reclaimed. The query above is cleaned up with a QueryTimeoutRuntimeException 63s after it started, and a later fetch is told that the query is not found (715). - isActive() returns true iff an RPC of the query is in progress, as documented. QueryExecution.cleanUpCoordinatorContextMapIfNeeded had been written against the inverted value, so its check is negated and its behaviour is unchanged. The cleanup deliberately does not start to skip the queries with an RPC in progress, which is what the isActive() check was meant to do, because the cleanup of such queries is relied on today: - executeBatchStatement (for a query statement) and querySchemaTemplate register an execution that they never release, and the end of their RPC is never recorded. The stale query cleanup is what releases these executions (measured: 65-66s after start with a 5s timeout, before and after this change). Skipping them would keep them forever. - an RPC that is still in progress 60s after the timeout is stuck, e.g. in a dispatch that does not return, and stopping the query is what unblocks it (QueryTerminalCleanupTest). So what was reclaimed before is still reclaimed, and the queries without an RPC in progress are reclaimed in addition. (cherry picked from commit 78bf43b) Backport of apache#19330 to dev/1.3: - QueryExecution.cleanUpCoordinatorContextMapIfNeeded keeps the cleanupQueryExecution(queryId, null, t) call of dev/1.3; only its condition is negated, as in the source commit. - CoordinatorTest and QueryExecutionLifecycleTest do not exist on dev/1.3. They are added with the tests of this change only, adapted to the Coordinator.execution and SessionInfo signatures and to the QueryType values of dev/1.3. - Not ported: the assertions on the UDF query scope of an execution and on the QUERY_TIMEOUT status of a reclaimed query. dev/1.3 has no UDF query scope, and its stopAndCleanup() does not move the query state machine, so the status of a reclaimed query stays SUCCESS there. The ported tests assert that the query is removed from the Coordinator.
…them Trigger 1: a statement that is a query is sent through executeBatchStatement (JDBC Statement.executeBatch), for example select s1 into root.db.dst(s1) from root.db.src (tree model) INSERT INTO dst SELECT * FROM src (table model) Wrong behaviour: the handler only started such a statement and returned its status. A batch has no result set, so nobody fetched the result of the statement, and a query only makes progress while its result is consumed: - with the source data on the DataNode that received the batch, the statement wrote nothing (0 of 2000 rows) or only a part (200 of the 1200 rows of IoTDBInsertQueryIT), and the batch reported success; - with the source data on another DataNode, the rows were written about 0.5s after the batch had returned. In both cases the execution stayed registered in the Coordinator, and listed by SHOW QUERIES, until the stale query cleanup removed it 60s after its timeout. Corrected behaviour: the handler consumes the result of such a statement and then releases its execution. The statement has been run to its end when the batch returns, a failure while it runs is reported in the status of the statement, and nothing stays registered. Trigger 2: a schema template query (querySchemaTemplate, for example Session.showMeasurementsInTemplate). Wrong behaviour: the handler returns the whole result in its response, but it left the execution registered until the stale query cleanup removed it 60s after its timeout. Corrected behaviour: the handler releases the execution when it returns. (cherry picked from commit 9c3af0c) Backport of apache#19330 to dev/1.3: - ClientRPCServiceImpl: the code around both handlers differs on dev/1.3 (no table model branch, the query id is requested at one place), so the changes were re-applied by hand on the dev/1.3 file: the consumeResult helper, consuming the result and releasing the execution per statement in executeBatchStatement, and releasing the execution in executeTemplateQueryStatement. - IoTDBShowQueriesIT does not exist on dev/1.3. It is added with the two tests of this change and the small fixture they need, and uses the ColumnHeaderConstant of dev/1.3. - IoTDBExecuteBatchIT starts a cluster per test on dev/1.3, so the new test has no cleanup block. - Not ported: IoTDBInsertQueryIT.testInsertQueryInBatch and the table model example above. dev/1.3 has no table model.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Backport of #19330 to
dev/1.3. It contains two fixes for query executions that stay registered in theCoordinatorafter nobody can use them any more.1. Abandoned queries were never reclaimed
A client runs a query and then neither fetches the rest of the result nor closes the statement, and no fragment instance of the query is left to fail it.
Coordinator.cleanUpStaleQueriesshould reclaim such a query 60 s after its timeout, but it skipped every query for whichIQueryExecution.isActive()returned true, andisActive()returnedstartTimeOfCurrentRpc == -1, which is true when no RPC is in progress. The cleanup therefore skipped exactly the abandoned queries.cleanUpStaleQueriesno longer depends onisActive(): every query that exceeded its timeout by more than 60 s is reclaimed.isActive()returns true iff an RPC of the query is in progress, as documented.QueryExecution.cleanUpCoordinatorContextMapIfNeededhad been written against the inverted value, so its check is negated and its behaviour does not change.What was reclaimed before is still reclaimed, and the queries without an RPC in progress are reclaimed in addition. #19330 explains why the cleanup does not start to skip the queries with an RPC in progress, and shows the effect on a two-DataNode cluster: the query was still registered and fully readable after 90 s, and is now removed 63.5 s after it started with
timeout=2000.2. Query statements of a batch were not run to their end, and not released
A statement that is a query, such as
select s1 into root.db.dst(s1) from root.db.src, is sent throughexecuteBatchStatement(JDBCStatement.executeBatch()). The handler only started it and returned its status. A batch has no result set, so nobody fetched the result, and a query only makes progress while its result is consumed. The statement was not run to its end, the batch still reported success, and the execution stayed registered until the stale query cleanup removed it.The handler now consumes the result of such a statement and then releases its execution.
querySchemaTemplate(e.g.Session.showMeasurementsInTemplate) returns its whole result in the response and now releases its execution when it returns.One consequence to be aware of: a plain
SELECTin a batch is now executed to its end before the batch returns, and its result is discarded. The time this takes is bounded byquery_timeout_threshold.Source and target
masteras90e3d967f9799086df50fd33084452b9268f25c0. That commit is the squash of the two commits of the PR,78bf43b042f34433cfa48fbc47a8707dd04586b2and9c3af0cfd3bcca347bba24a550a00a6a57372624, and has the same tree as the second one. The two commits were cherry-picked with-x, so the two fixes stay separate here.dev/1.3at86f9335302.Adaptations
Main code:
QueryExecution.cleanUpCoordinatorContextMapIfNeededkeeps thecleanupQueryExecution(queryId, null, t)call ofdev/1.3; only its condition is negated, as in the source.ClientRPCServiceImpl: the surrounding code of both handlers differs ondev/1.3(no table model branch, the query id is requested at one place), so the changes were re-applied by hand on thedev/1.3file: theconsumeResulthelper, consuming the result and releasing the execution per statement inexecuteBatchStatement, and releasing the execution inexecuteTemplateQueryStatement.Tests:
CoordinatorTest,QueryExecutionLifecycleTestandIoTDBShowQueriesITdo not exist ondev/1.3. They are added with the tests of this change only, andIoTDBShowQueriesITwith the small fixture these tests need. They are adapted to theCoordinator.executionandSessionInfosignatures, theQueryTypevalues and theColumnHeaderConstantpackage ofdev/1.3.IoTDBExecuteBatchITstarts a cluster per test ondev/1.3, so the new test has no cleanup block.Not ported, because they depend on something
dev/1.3does not have:dev/1.3has no UDF query scope);QUERY_TIMEOUTstatus of a reclaimed query:stopAndCleanup()does not move the query state machine ondev/1.3, so that status staysSUCCESSthere. The ported tests assert that the query is removed from theCoordinator. The next section shows what this means for a client;IoTDBInsertQueryIT.testInsertQueryInBatch(table model).What a client sees when its query is reclaimed on
dev/1.3getStatus()of a reclaimed execution keeps reportingSUCCESSondev/1.3, becauseQueryExecution.stopAndCleanup()does not move the query state machine there (masterdoes since #19224). Only the first RPC of a query reads that status. Every later RPC looks the query up in theCoordinatorand reports715when it is gone.Measured on a cluster with two DataNodes, with
dev/1.3at86f9335302and with this branch. The queries are sent to DataNode 1 with a timeout of 2 s:dev/1.3715: Query is not found, it may be killed by others, timeout or some other runtime errors, ...715SUCCESSwithout a result set.Statement.executeQuerythrowsSQLException: Cannot invoke "java.util.List.size()" because "columnNameList" is null,Session.executeQueryStatementthrows aNullPointerExceptionwith the same message715on the next fetch. No result is cut short without an error.SUCCESSstatus reaches a client. It is existing behaviour ofdev/1.3and is not changed here: the old check skipped only the queries without an RPC in progress, so a query whose first RPC is in progress was reclaimed before this PR as well. Onmasterthe same run ends with720: Current query is time out, ...for both clients, because [Query] Fix scheduler stop races and query attempt context leaks #19224 madestopAndCleanup()fail the query state machine. That change is not part of Fix stale query cleanup for abandoned queries and query statements in executeBatch #19330 and is not included in this backport. The run neededdn_connection_timeout_ms=300000to keep the dispatch RPC waiting for 75 s. With the default of 60 s the dispatch RPC times out before the cleanup runs, and both clients get720: Current query is time out, ...after 60.0 s (measured on this branch).Validation on
dev/1.3All commands were run offline on this branch with JDK 21.
Unit tests:
mvn -o test -pl iotdb-core/datanode -am \ -Dtest=CoordinatorTest,QueryExecutionLifecycleTest,ConfigExecutionTest,QueryExecutionRetryTest \ -Dsurefire.failIfNoSpecifiedTests=falsedev/1.3: 15 tests, 3 failures:CoordinatorTest.testStaleQueryWithoutRpcInProgressIsCleanedUp:97 expected null, but was:<QueryExecution[...]>ConfigExecutionTest.isActiveTellsWhetherAnRpcOfTheQueryIsInProgress:169QueryExecutionLifecycleTest.isActiveTellsWhetherAnRpcOfTheQueryIsInProgress:48Integration tests:
mvn -o clean verify -DskipUTs '-Dit.test=IoTDBShowQueriesIT,IoTDBExecuteBatchIT' \ -DfailIfNoTests=false -Dfailsafe.failIfNoSpecifiedTests=false \ -pl integration-test -am -P with-integration-testsClientRPCServiceImplas ondev/1.3, the three new tests fail:IoTDBExecuteBatchIT.testJDBCExecuteBatchWithSelectInto:105 expected:<2> but was:<0>IoTDBShowQueriesIT.queryInBatchReleaseTest: expected:<[]> but was:<[20261119_125950_00007_1]>IoTDBShowQueriesIT.schemaTemplateQueryReleaseTest: expected:<[]> but was:<[20261119_125950_00012_1]>IoTDBShowQueriesIT(2 tests) andIoTDBExecuteBatchIT(3 tests) pass with the default profile (1C1D) and with-PClusterIT(1C3D).mvn spotless:check -pl iotdb-core/datanode,integration-test -P with-integration-testspasses.query_timeout_threshold=5000,dn_connection_timeout_ms=300000), driven through JDBC and the Session API. The batch measurements of Fix stale query cleanup for abandoned queries and query statements in executeBatch #19330 on two DataNodes were not repeated ondev/1.3; the integration tests above cover that fix.This PR has:
for an unfamiliar reader.
for code coverage.
Key changed/added classes (or packages if there are too many classes) in this PR
iotdb-core/datanode:org.apache.iotdb.db.queryengine.plan.Coordinator,org.apache.iotdb.db.queryengine.plan.execution.QueryExecution,org.apache.iotdb.db.queryengine.plan.execution.config.ConfigExecution,org.apache.iotdb.db.protocol.thrift.impl.ClientRPCServiceImpliotdb-core/datanode(test):CoordinatorTest,ConfigExecutionTest,QueryExecutionLifecycleTestintegration-test:IoTDBShowQueriesIT,IoTDBExecuteBatchIT🤖 Generated with Claude Code