Repository navigation
Conversation
Tree-model queries on schema-template devices fetched the schema remotely on every run instead of reading the DataNode schema cache. Two bugs caused it. 1. ClusterSchemaFetcher.fetchSchema looked up template devices with the full series path (root.db.d1.s1), because PartialPath.getDevicePath() returns the path itself. The cache stores them under the device path (root.db.d1), so every lookup missed. Strip the measurement before the lookup, as the normal-series branch already does. 2. On a cluster, a DataNode that runs the fragment of a SELECT LAST coordinated by another DataNode declares the last cache, which creates the device's cache entry with a TreeDeviceNormalSchema. TableDeviceCacheEntry.setDeviceSchema stored the fetched template schema with compareAndSet(null, ...), which never replaces it, so the device missed on that DataNode until restart. Replace a missing or normal schema with the template schema, keep the last-cache field, and adjust the memory estimate by the replaced size.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes #19331.
Problem
Tree-model queries on schema-template devices fetch the schema remotely on every run instead of
reading the DataNode schema cache. On a cluster with
schema_replication_factor=3, that is aRatis read on the schema region per statement. In our deployment (3C3D, 2.0.8), a 40-device
SELECT LASTspent about 180 ms of its 575 ms average in schema fetch. With both fixes below,schema fetch fell to 13 ms and the average to 192 ms.
Two bugs cause it.
1.
ClusterSchemaFetcher.fetchSchemareads the cache with the wrong key. For templatedevices it calls
schemaCache.getMatchedTemplateSchema(pattern.getDevicePath()).patternis aPartialPathfor a full series such asroot.db.d1.s1, andPartialPath.getDevicePath()returns the path itself, so the lookup is for device
root.db.d1.s1. The cache stores thedevice as
root.db.d1, so every lookup misses and the statement goes tofetchSchemaOfPreciseMatchOrPreciseDeviceUsingTemplate. The normal-series branch of the samemethod already strips the measurement.
2.
TableDeviceCacheEntry.setDeviceSchemacannot replace a normal schema. This shows onlyonce (1) is fixed, and only on clusters:
SELECT LASTon template deviced, whose data region is onDataNode B.
OperatorTreeGeneratorcallsdeclareLastCache, which reachesTableDeviceCacheEntry.setMeasurementSchemaand creates the entry with aTreeDeviceNormalSchema.d.getMatchedTemplateSchemamisses because the entry holdsa normal schema. B fetches the template schema and calls
putDeviceSchema.setDeviceSchemausescompareAndSet(null, new TreeDeviceTemplateSchema(...)), which failsbecause the field is not null. Steps 3 and 4 repeat for every query B coordinates on
d,until restart.
A multi-device statement does the remote fetch when any one device misses, so a few stuck
devices per node keep every multi-device
SELECT LASTslow.Fix
pattern.setDeviceSchema, replace a missing schema or aTreeDeviceNormalSchemawith the fetchedTreeDeviceTemplateSchema(getAndUpdate), and return the size delta. A template schema ora table-model
TableAttributeSchemais never replaced. The separatelastCachefield isuntouched, so values cached by the fragment are kept.
Alternative considered for (2): stop
declareLastCachefrom creating a normal schema fortemplate devices. The fragment would then need a template check on the query execution path.
Replacing the normal schema at put time keeps the change inside the template branch of
setDeviceSchema.Reproduction (1 ConfigNode and 3 DataNodes, local Docker, 40 template devices)
Steady
Fetch Schema CostfromEXPLAIN ANALYZE. A device counts as "stuck" when it costs morethan 1 ms; a cache hit is about 0.07 ms.
SELECT LASTfetch costRows were identical across nodes and builds, and new writes were visible in
SELECT LASTonevery node.
This PR has:
for an unfamiliar reader.
Key changed/added classes in this PR
ClusterSchemaFetcherTableDeviceCacheEntryClusterSchemaFetcherTest(new)TreeDeviceSchemaCacheManagerTest