Skip to content

[To dev/1.3] Fix success status of a query that is removed while its first RPC starts it - #19336

Open
JackieTien97 wants to merge 1 commit into
apache:dev/1.3from
JackieTien97:ty/fix-reclaimed-query-status-1.3
Open

JackieTien97 wants to merge 1 commit into
apache:dev/1.3from
JackieTien97:ty/fix-reclaimed-query-status-1.3

Conversation

@JackieTien97

Copy link
Copy Markdown
Contributor

Description

A fix for dev/1.3 only. master reports an error in this situation since #19224 (see "Relation to master and #19334").

Scenario

A client sends a query to a DataNode. While the RPC is still starting the query, the query is removed from the Coordinator, and the start then succeeds.

On dev/1.3 this is what Coordinator.cleanUpStaleQueries does to a query whose first RPC lasts longer than timeout + 60 s: it reclaims every query that exceeded its timeout by more than 60 s, also one whose first RPC is in progress. So the dispatch of the fragment instances has to hang for that long and then succeed, for example because a DataNode stops responding without closing its connections while dn_connection_timeout_ms is set above that time. With the default of 60 s the dispatch RPC times out first and the statement fails with an error.

Incorrect behaviour

Removing a query stops it and releases its result handle, but it does not move the state machine of the query on dev/1.3 (QueryExecution.stopAndCleanup(Throwable) is stop(t); releaseResource(t);). When the start then succeeds, getStatus() reports SUCCESS_STATUS. Coordinator.execution returns that status, the RPC handler finds no execution for the query id, and answers with the bare success status: no columns and no data.

What the client makes of that depends on the call it used. The calls that do not expect a result set report a statement that succeeded.

Corrected behaviour

The statement fails with QUERY_WAS_KILLED(715) and the text that the fetch RPCs of this branch already use for a query that is gone: Query is not found, it may be killed by others, timeout or some other runtime errors, you can see more details in server log.

Example

A cluster with one ConfigNode and two DataNodes, query_timeout_threshold=5000, dn_connection_timeout_ms=300000. root.remote.d1 is stored on DataNode 2, the statement is sent to DataNode 1:

select s1 from root.remote.d1 where time <= 10

DataNode 2 is suspended with SIGSTOP right before the statement is sent and continued 75 s later. DataNode 1 reclaims the query in the meantime (Cleaning up stale query with id 64, which has been running for 62025 ms, timeout duration is: 2000ms), and the dispatch succeeds once DataNode 2 continues.

Client call (timeout) dev/1.3 at 86f9335302 This branch
JDBC Statement.executeQuery (2 s) SQLException: Cannot invoke "java.util.List.size()" because "columnNameList" is null IoTDBSQLException, error code 715
Session.executeQueryStatement (2 s) NullPointerException with the same message, thrown from IoTDBRpcDataSet.<init> StatementExecutionException, status code 715
JDBC Statement.execute (2 s) returns false, getResultSet() is null, no exception IoTDBSQLException, error code 715
Session.executeNonQueryStatement (server default, 5 s) returns normally StatementExecutionException, status code 715

All four calls return when DataNode 2 continues, 75 s after they were sent, in both builds. With both DataNodes running, the four calls return the 10 rows (or succeed) in both builds.

Design

  • The check is in Coordinator.execution, not in the RPC handlers. Every entry point that executes a statement goes through this method, and it holds the execution object: after the start it checks whether a statement that it registered as a query is still registered. A handler cannot make this distinction. For a handler, "the statement is not a query" and "the query is gone" both look like "there is no execution for this query id", and Statement.isQuery() does not tell them apart, because the Coordinator registers every execution whose query type is READ, which includes for example the SHOW statements.
  • Only a success status is replaced. If the start of a removed query failed, the failure is returned as before; it says more than "not found". The status counts as success when its code is SUCCESS_STATUS or REDIRECTION_RECOMMEND, which is the condition under which the handlers go on to look up the execution.
  • Statements that are not queries are not affected. They are never registered, so there is nothing to compare.
  • The code is 715, not 720. The Coordinator does not know why the query was removed, and 715 with this text is what a client of this branch already gets when it fetches from a query that was removed.
  • The message is a constant in DataNodeQueryMessages (en and zh), with the name and the texts it has on master (there in DataNodeMiscMessages). ClientRPCServiceImpl.NO_QUERY_EXECUTION_ERR_MSG now refers to it instead of repeating the English text, so both places use one definition.

Relation to master and #19334

Not changed

  • The query is still reclaimed. This PR changes only the status the first RPC returns. Whether cleanUpStaleQueries should spare a query whose first RPC is in progress is a separate question (Fix stale query cleanup for abandoned queries and query statements in executeBatch #19330 explains why it does not).
  • What the removed query leaves behind on the DataNode is the same with and without this PR. QueryExecution.stop runs while the scheduler of the query is not assigned yet (ClusterScheduler.start() has not returned), so it stops nothing, and the scheduler starts its state tracker after the dispatch succeeded. Counted with jmap -histo:live on DataNode 1 after the four statements above were answered and all connections were closed, in both builds: 0 QueryExecution, 0 ClusterScheduler, 5 FixedRateFragInsStateTracker, and 4 of them 60 s later (1 before the experiment). master handles the stop of a query that is still being scheduled since [Query] Fix scheduler stop races and query attempt context leaks #19224; that part is not ported here.
  • A query that is removed between the check and the handler's own lookup (Coordinator.execution returns, then the handler calls getQueryExecution) still takes the old path. That window is a few instructions wide instead of the whole dispatch.
  • Other callers of the Coordinator get the same 715 status instead of a success status for such a query. For example, ClusterSchemaFetchExecutor threw Fetch Schema failed, because queryExecution is null, and now throws Fetch Schema failed, because Query is not found, ... with code 715.

Tests

All commands were run offline on this branch with JDK 21.

New unit test CoordinatorRemovedQueryTest, which runs a statement through Coordinator.execution with a real QueryExecution:

  • testQueryReclaimedWhileItStartsIsReportedAsNotFound: the real cleanUpStaleQueries() reclaims the query while it starts; the status must be 715 with the message above.
  • testQueryThatStaysRegisteredIsReportedAsSuccessful
  • testStatementThatIsNotAQueryIsReportedAsSuccessful
  • testFailureOfQueryRemovedWhileItStartsIsKept: a removed query whose start failed keeps its failure status.
mvn -o test -pl iotdb-core/datanode -am \
  -Dtest=CoordinatorRemovedQueryTest,QueryExecutionRetryTest,ConfigExecutionTest \
  -Dsurefire.failIfNoSpecifiedTests=false
  • With Coordinator as on dev/1.3: 1 failure, CoordinatorRemovedQueryTest.testQueryReclaimedWhileItStartsIsReportedAsNotFound:117 expected:<715> but was:<200>.
  • With this branch: 11 tests, 0 failures, with the spotless and checkstyle checks of the build enabled.

Further runs on this branch:

  • The unit tests of the datanode packages queryengine, protocol and storageengine.load (418 test classes): 999 tests, 0 failures, 7 skipped.
  • mvn -o test-compile -DskipTests -P with-zh-locale passes.
  • Integration tests with the default profile (1C1D), IoTDBSimpleQueryIT, IoTDBExecuteBatchIT, IoTDBSelectIntoIT, IoTDBMetadataFetchIT and IoTDBQueryDemoIT: 109 tests, 0 failures, 4 skipped.
  • This branch merged with the branch of [To dev/1.3] Fix stale query cleanup for abandoned queries and query statements in executeBatch (#19330) #19334: no conflict, and CoordinatorRemovedQueryTest, CoordinatorTest, QueryExecutionLifecycleTest, QueryExecutionRetryTest and ConfigExecutionTest pass together (19 tests).
  • The table in "Example" and the instance counts in "Not changed" come from real 1C2D clusters built from dev/1.3 at 86f9335302 and from this branch.

This PR has:

  • been self-reviewed.
    • concurrent read
  • added comments explaining the "why" and the intent of the code wherever would not be obvious
    for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold
    for code coverage.
  • been tested in a test IoTDB cluster.

Key changed/added classes (or packages if there are too many classes) in this PR
  • iotdb-core/datanode: org.apache.iotdb.db.queryengine.plan.Coordinator, org.apache.iotdb.db.protocol.thrift.impl.ClientRPCServiceImpl, org.apache.iotdb.db.i18n.DataNodeQueryMessages (en, zh)
  • iotdb-core/datanode (test): CoordinatorRemovedQueryTest

🤖 Generated with Claude Code

…rts it

Removing a query from the Coordinator stops it and releases its result
handle, but does not move its state machine on dev/1.3. If
cleanUpStaleQueries reclaims a query whose first RPC is still dispatching
(the RPC lasted longer than timeout + 60 s) and the dispatch then succeeds,
the RPC returned SUCCESS_STATUS without a result set. JDBC executeQuery and
Session.executeQueryStatement failed with a NullPointerException on
columnNameList; Statement.execute and Session.executeNonQueryStatement
reported a statement that succeeded.

Coordinator.execution now checks, after the start, whether a statement it
registered as a query is still registered. If it is not and the status says
success, the status is replaced with QUERY_WAS_KILLED(715) and the message
the fetch RPCs already use for a query that is gone. A failure status is
returned unchanged, and statements that are not queries are not affected.

master reports an error here since apache#19224, which fails the state machine in
stopAndCleanup. That change is not ported; this is a dev/1.3-only fix.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant