Skip to content

[Query] Fix tree-model schema cache misses for template devices - #19332

Open
mingisme wants to merge 1 commit into
apache:masterfrom
mingisme:fix_template_schema_cache
Open

mingisme wants to merge 1 commit into
apache:masterfrom
mingisme:fix_template_schema_cache

Conversation

@mingisme

Copy link
Copy Markdown

Description

Fixes #19331.

Problem

Tree-model queries on schema-template devices fetch the schema remotely on every run instead of
reading the DataNode schema cache. On a cluster with schema_replication_factor=3, that is a
Ratis read on the schema region per statement. In our deployment (3C3D, 2.0.8), a 40-device
SELECT LAST spent about 180 ms of its 575 ms average in schema fetch. With both fixes below,
schema fetch fell to 13 ms and the average to 192 ms.

Two bugs cause it.

1. ClusterSchemaFetcher.fetchSchema reads the cache with the wrong key. For template
devices it calls schemaCache.getMatchedTemplateSchema(pattern.getDevicePath()). pattern is a
PartialPath for a full series such as root.db.d1.s1, and PartialPath.getDevicePath()
returns the path itself, so the lookup is for device root.db.d1.s1. The cache stores the
device as root.db.d1, so every lookup misses and the statement goes to
fetchSchemaOfPreciseMatchOrPreciseDeviceUsingTemplate. The normal-series branch of the same
method already strips the measurement.

2. TableDeviceCacheEntry.setDeviceSchema cannot replace a normal schema. This shows only
once (1) is fixed, and only on clusters:

  1. DataNode A coordinates SELECT LAST on template device d, whose data region is on
    DataNode B.
  2. B runs the fragment. OperatorTreeGenerator calls declareLastCache, which reaches
    TableDeviceCacheEntry.setMeasurementSchema and creates the entry with a
    TreeDeviceNormalSchema.
  3. B later coordinates a query on d. getMatchedTemplateSchema misses because the entry holds
    a normal schema. B fetches the template schema and calls putDeviceSchema.
  4. setDeviceSchema uses compareAndSet(null, new TreeDeviceTemplateSchema(...)), which fails
    because the field is not null. Steps 3 and 4 repeat for every query B coordinates on d,
    until restart.

A multi-device statement does the remote fetch when any one device misses, so a few stuck
devices per node keep every multi-device SELECT LAST slow.

Fix

  1. Look up the template device by its device path: drop the last node of pattern.
  2. In setDeviceSchema, replace a missing schema or a TreeDeviceNormalSchema with the fetched
    TreeDeviceTemplateSchema (getAndUpdate), and return the size delta. A template schema or
    a table-model TableAttributeSchema is never replaced. The separate lastCache field is
    untouched, so values cached by the fragment are kept.

Alternative considered for (2): stop declareLastCache from creating a normal schema for
template devices. The fragment would then need a template check on the query execution path.
Replacing the normal schema at put time keeps the change inside the template branch of
setDeviceSchema.

Reproduction (1 ConfigNode and 3 DataNodes, local Docker, 40 template devices)

Steady Fetch Schema Cost from EXPLAIN ANALYZE. A device counts as "stuck" when it costs more
than 1 ms; a cache hit is about 0.07 ms.

Build Stuck devices on dn1 / dn2 / dn3 40-device SELECT LAST fetch cost
master 40 / 40 / 40 11.9 ms
fix 1 only 0 / 13 / 12 (the devices whose fragments ran on that node) 12.0 ms
both fixes 0 / 0 / 0 0.37 ms

Rows were identical across nodes and builds, and new writes were visible in SELECT LAST on
every node.


This PR has:

  • been self-reviewed.
    • concurrent read
    • concurrent write
  • added comments explaining the "why" and the intent of the code wherever would not be obvious
    for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths.
  • been tested in a test IoTDB cluster.

Key changed/added classes in this PR
  • ClusterSchemaFetcher
  • TableDeviceCacheEntry
  • ClusterSchemaFetcherTest (new)
  • TreeDeviceSchemaCacheManagerTest

Tree-model queries on schema-template devices fetched the schema
remotely on every run instead of reading the DataNode schema cache.
Two bugs caused it.

1. ClusterSchemaFetcher.fetchSchema looked up template devices with the
   full series path (root.db.d1.s1), because PartialPath.getDevicePath()
   returns the path itself. The cache stores them under the device path
   (root.db.d1), so every lookup missed. Strip the measurement before
   the lookup, as the normal-series branch already does.

2. On a cluster, a DataNode that runs the fragment of a SELECT LAST
   coordinated by another DataNode declares the last cache, which
   creates the device's cache entry with a TreeDeviceNormalSchema.
   TableDeviceCacheEntry.setDeviceSchema stored the fetched template
   schema with compareAndSet(null, ...), which never replaces it, so
   the device missed on that DataNode until restart. Replace a missing
   or normal schema with the template schema, keep the last-cache
   field, and adjust the memory estimate by the replaced size.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Tree-model queries on schema-template devices always miss the DataNode schema cache

1 participant