Repository navigation
[To dev/1.3] Fix success status of a query that is removed while its first RPC starts it - #19336
Open
JackieTien97 wants to merge 1 commit into
Open
JackieTien97 wants to merge 1 commit into
JackieTien97 wants to merge 1 commit into
Conversation
…rts it Removing a query from the Coordinator stops it and releases its result handle, but does not move its state machine on dev/1.3. If cleanUpStaleQueries reclaims a query whose first RPC is still dispatching (the RPC lasted longer than timeout + 60 s) and the dispatch then succeeds, the RPC returned SUCCESS_STATUS without a result set. JDBC executeQuery and Session.executeQueryStatement failed with a NullPointerException on columnNameList; Statement.execute and Session.executeNonQueryStatement reported a statement that succeeded. Coordinator.execution now checks, after the start, whether a statement it registered as a query is still registered. If it is not and the status says success, the status is replaced with QUERY_WAS_KILLED(715) and the message the fetch RPCs already use for a query that is gone. A failure status is returned unchanged, and statements that are not queries are not affected. master reports an error here since apache#19224, which fails the state machine in stopAndCleanup. That change is not ported; this is a dev/1.3-only fix.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
A fix for
dev/1.3only.masterreports an error in this situation since #19224 (see "Relation tomasterand #19334").Scenario
A client sends a query to a DataNode. While the RPC is still starting the query, the query is removed from the
Coordinator, and the start then succeeds.On
dev/1.3this is whatCoordinator.cleanUpStaleQueriesdoes to a query whose first RPC lasts longer thantimeout + 60 s: it reclaims every query that exceeded its timeout by more than 60 s, also one whose first RPC is in progress. So the dispatch of the fragment instances has to hang for that long and then succeed, for example because a DataNode stops responding without closing its connections whiledn_connection_timeout_msis set above that time. With the default of 60 s the dispatch RPC times out first and the statement fails with an error.Incorrect behaviour
Removing a query stops it and releases its result handle, but it does not move the state machine of the query on
dev/1.3(QueryExecution.stopAndCleanup(Throwable)isstop(t); releaseResource(t);). When the start then succeeds,getStatus()reportsSUCCESS_STATUS.Coordinator.executionreturns that status, the RPC handler finds no execution for the query id, and answers with the bare success status: no columns and no data.What the client makes of that depends on the call it used. The calls that do not expect a result set report a statement that succeeded.
Corrected behaviour
The statement fails with
QUERY_WAS_KILLED(715)and the text that the fetch RPCs of this branch already use for a query that is gone:Query is not found, it may be killed by others, timeout or some other runtime errors, you can see more details in server log.Example
A cluster with one ConfigNode and two DataNodes,
query_timeout_threshold=5000,dn_connection_timeout_ms=300000.root.remote.d1is stored on DataNode 2, the statement is sent to DataNode 1:DataNode 2 is suspended with
SIGSTOPright before the statement is sent and continued 75 s later. DataNode 1 reclaims the query in the meantime (Cleaning up stale query with id 64, which has been running for 62025 ms, timeout duration is: 2000ms), and the dispatch succeeds once DataNode 2 continues.dev/1.3at86f9335302Statement.executeQuery(2 s)SQLException: Cannot invoke "java.util.List.size()" because "columnNameList" is nullIoTDBSQLException, error code 715Session.executeQueryStatement(2 s)NullPointerExceptionwith the same message, thrown fromIoTDBRpcDataSet.<init>StatementExecutionException, status code 715Statement.execute(2 s)false,getResultSet()isnull, no exceptionIoTDBSQLException, error code 715Session.executeNonQueryStatement(server default, 5 s)StatementExecutionException, status code 715All four calls return when DataNode 2 continues, 75 s after they were sent, in both builds. With both DataNodes running, the four calls return the 10 rows (or succeed) in both builds.
Design
Coordinator.execution, not in the RPC handlers. Every entry point that executes a statement goes through this method, and it holds the execution object: after the start it checks whether a statement that it registered as a query is still registered. A handler cannot make this distinction. For a handler, "the statement is not a query" and "the query is gone" both look like "there is no execution for this query id", andStatement.isQuery()does not tell them apart, because theCoordinatorregisters every execution whose query type isREAD, which includes for example theSHOWstatements.SUCCESS_STATUSorREDIRECTION_RECOMMEND, which is the condition under which the handlers go on to look up the execution.Coordinatordoes not know why the query was removed, and 715 with this text is what a client of this branch already gets when it fetches from a query that was removed.DataNodeQueryMessages(enandzh), with the name and the texts it has onmaster(there inDataNodeMiscMessages).ClientRPCServiceImpl.NO_QUERY_EXECUTION_ERR_MSGnow refers to it instead of repeating the English text, so both places use one definition.Relation to
masterand #19334masterdoes not need this change. Since [Query] Fix scheduler stop races and query attempt context leaks #19224,QueryExecution.stopAndCleanup(Throwable)moves the state machine of the query to a terminal state, so the first RPC reports the failure: the experiment above ends with720: Current query is time outfor JDBC and the Session API onmaster. This PR does not port [Query] Fix scheduler stop races and query attempt context leaks #19224. It only stopsdev/1.3from reporting success.Not changed
cleanUpStaleQueriesshould spare a query whose first RPC is in progress is a separate question (Fix stale query cleanup for abandoned queries and query statements in executeBatch #19330 explains why it does not).QueryExecution.stopruns while the scheduler of the query is not assigned yet (ClusterScheduler.start()has not returned), so it stops nothing, and the scheduler starts its state tracker after the dispatch succeeded. Counted withjmap -histo:liveon DataNode 1 after the four statements above were answered and all connections were closed, in both builds: 0QueryExecution, 0ClusterScheduler, 5FixedRateFragInsStateTracker, and 4 of them 60 s later (1 before the experiment).masterhandles the stop of a query that is still being scheduled since [Query] Fix scheduler stop races and query attempt context leaks #19224; that part is not ported here.Coordinator.executionreturns, then the handler callsgetQueryExecution) still takes the old path. That window is a few instructions wide instead of the whole dispatch.Coordinatorget the same715status instead of a success status for such a query. For example,ClusterSchemaFetchExecutorthrewFetch Schema failed, because queryExecution is null, and now throwsFetch Schema failed, because Query is not found, ...with code 715.Tests
All commands were run offline on this branch with JDK 21.
New unit test
CoordinatorRemovedQueryTest, which runs a statement throughCoordinator.executionwith a realQueryExecution:testQueryReclaimedWhileItStartsIsReportedAsNotFound: the realcleanUpStaleQueries()reclaims the query while it starts; the status must be 715 with the message above.testQueryThatStaysRegisteredIsReportedAsSuccessfultestStatementThatIsNotAQueryIsReportedAsSuccessfultestFailureOfQueryRemovedWhileItStartsIsKept: a removed query whose start failed keeps its failure status.mvn -o test -pl iotdb-core/datanode -am \ -Dtest=CoordinatorRemovedQueryTest,QueryExecutionRetryTest,ConfigExecutionTest \ -Dsurefire.failIfNoSpecifiedTests=falseCoordinatoras ondev/1.3: 1 failure,CoordinatorRemovedQueryTest.testQueryReclaimedWhileItStartsIsReportedAsNotFound:117 expected:<715> but was:<200>.Further runs on this branch:
queryengine,protocolandstorageengine.load(418 test classes): 999 tests, 0 failures, 7 skipped.mvn -o test-compile -DskipTests -P with-zh-localepasses.IoTDBSimpleQueryIT,IoTDBExecuteBatchIT,IoTDBSelectIntoIT,IoTDBMetadataFetchITandIoTDBQueryDemoIT: 109 tests, 0 failures, 4 skipped.CoordinatorRemovedQueryTest,CoordinatorTest,QueryExecutionLifecycleTest,QueryExecutionRetryTestandConfigExecutionTestpass together (19 tests).dev/1.3at86f9335302and from this branch.This PR has:
for an unfamiliar reader.
for code coverage.
Key changed/added classes (or packages if there are too many classes) in this PR
iotdb-core/datanode:org.apache.iotdb.db.queryengine.plan.Coordinator,org.apache.iotdb.db.protocol.thrift.impl.ClientRPCServiceImpl,org.apache.iotdb.db.i18n.DataNodeQueryMessages(en,zh)iotdb-core/datanode(test):CoordinatorRemovedQueryTest🤖 Generated with Claude Code