Skip to content

Batch scan TimeSeriesMetadata for wide-table queries - #19198

Open
ColinLeeo wants to merge 35 commits into
apache:masterfrom
ColinLeeo:colin/batch-scan-query-a2e8cc
Open

ColinLeeo wants to merge 35 commits into
apache:masterfrom
ColinLeeo:colin/batch-scan-query-a2e8cc

Conversation

@ColinLeeo

@ColinLeeo ColinLeeo commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

Description

A device's TimeseriesMetadata is stored in measurement leaf groups (up to max_degree_of_index_node series per group). Loading it column by column walks the device metadata index and deserializes the containing leaf group once per requested column, so a scan that selects almost all columns pays that traversal once per column, and every column absent from a file costs its own cache probe and BloomFilter check. In the motivating workload (one device with ~700 columns, ~20 written per day, 12 files) loading the metadata of one device across its files dominated the query.

The TsFile side of this change is apache/tsfile#1123; this PR pins the dated snapshot built from that branch.

What changed

TsFile side (apache/tsfile#1123)

  • TsFileSequenceReader.readTimeseriesMetadataForMeasurements(device, offset, measurements, returnedMeasurements, mayExist, ignoreNotExistDevice, ioSizeRecorder) locates the device's measurement metadata index node (or reads the supplied offset for external TsFiles), routes each name to its floor child with a binary search over the node's children, and reads each visited internal node and each visited leaf group once. Chunk metadata is decoded only for the measurements the call returns. returnedMeasurements may be a superset of measurements, which lets a single-column call warm its group siblings.
  • mayExist is optional: a name is tested before it selects a group, so a group that only absent names point to is not read. It must have no false negatives; a BloomFilter qualifies.
  • There is no global sort; the batch index search used by the earlier revision of this PR is gone.

DataNode

TimeSeriesMetadataCache.getTimeseriesMetadata is now:

  1. per-measurement cache pass;
  2. device-absence check on the time column via the BloomFilter;
  3. per-miss BloomFilter probe: a column the filter rules out is answered with null and not remembered, because probing the filter again is cheaper than one cache entry per absent column per file;
  4. the surviving columns go to the reader in one call under the per-(device, file) lock;
  5. every surviving column the reader does not return, or returns with count == 0, is answered with null and remembered with a negative cache entry, so a later query does not walk its leaf group again.

Offset-addressed external TsFiles skip BloomFilter filtering, because their BloomFilter may not cover the series. get() remains the single-column entry point used by tree-model scans. (Earlier revisions of this PR used a device-wide read gated by a 0.6 projection ratio; the selective reader replaces that design.)

tsfile.version moves to 2.5.1-261120-SNAPSHOT, cut from the merged develop (apache/tsfile branch snapshot/2.5.1-261120, squash commit 0d89263e) and published to the Apache snapshots repository. It must be replaced by a released TsFile version before merge.

Example

Table with 700 field columns; each of 12 TsFiles holds 20 of them (rotating); SELECT * for one device:

  • Before: 701 per-column cache lookups per file; every absent column took the per-device lock, a second cache check and a BloomFilter probe; the columns that got through the filter paid a leaf-group read and were remembered, the rest were re-probed on every query. 8,412 lookups and about 8,200 lock/probe cycles per query, and 420 leaf-group reads per cold query (35 per file).
  • After: one cache pass and one lock per file; about 700 BloomFilter probes per cold file (one per cache-missed column), one batch read per file (a single leaf group in this fixture), and about 34 negative entries per file (the columns that pass the filter). Warm queries do no leaf-group read; the BloomFilter-excluded columns are probed again.

The per-column probe is the price of keeping negative entries bounded by the BloomFilter false-positive rate instead of one entry per absent column per file. A dense 700-column device adds the same probe per column but keeps the reads and bytes unchanged.

Benchmark

Measured on the latest revision with the same machine and TsFile snapshot for both variants, cold metadata cache unless noted; bytes and read counts come from QueryStatistics. The base column was measured on master with the same harness. Each figure is the median of five timed rounds after warm-up; the dense cold case varies by up to about 20% between runs, so the table reports ranges.

Scenario base (master) this PR
700 columns, select all, cold / warm 12.2–12.4 / 1.35 ms 15.4–17.8 / 1.4–1.7 ms
700 columns, first 60%, cold / warm 2.9–4.1 / 0.6 ms 5.1–5.8 / 0.5–0.8 ms
Sparse: 20 of 700 columns per file, cold / warm 7.7–7.8 / 5.0–5.3 ms 2.5–3.0 / 1.5–2.0 ms
Repeated select-all, working set larger than the 61 MB cache (steady state) 6.0–13.0 ms (median 4.7–5.3) 6.3–6.4 ms (median 4.2–4.4)
Bytes per query (select-all / 60% / sparse) 2021.0 / 1475.6 / 64.1 KB 2019.9 / 1475.1 / 59.2 KB

The sparse wide-table case that motivated the change is the main gain: cold 7.7–7.8 ms to 2.5–3.0 ms, warm 5.0–5.3 ms to 1.5–2.0 ms. The dense cold case varies by 15–20% between rounds on this machine (repeated runs of the previous revision show the same range, including a 30 ms outlier), so the table reports the observed range rather than a single sample; the bytes and read counts are identical across runs. The latest revision checks the aligned time column before the per-column cache pass and filters BloomFilter-excluded measurements before taking the device lock, so a warm sparse query no longer enters the locked recheck for columns that are absent from the file. Sparse cold still pays one BloomFilter probe per cache-missed column, which is what keeps negative entries bounded by the false-positive rate instead of one per absent column. The single-series get() fast path answers a cache hit on the first LRU lookup, like master.

Merge checklist

  • apache/tsfile#1123 is merged on develop (squash commit 0d89263e); the dated snapshot is cut from that commit.
  • Replace the dated 2.5.1-261120-SNAPSHOT in pom.xml with a released TsFile version before merge.

Tests

  • TsFile (apache/tsfile#1123): TimeSeriesMetadataReadTest covers batch vs per-measurement reads, absent names and devices, the offset-addressed variant, sibling warming, exact node/group read counts on an internal measurement index, aligned (time column) and table-model fixtures, and what an absent name costs with and without mayExist.
  • DataNode: TimeSeriesMetadataCacheTest adds "reader-confirmed absence is remembered, so the second query does not call the reader" and "a BloomFilter-excluded column is probed again instead of being remembered". It also covers cache hits, placeholders, BloomFilter exclusion and external TsFiles. AlignedSeriesScanBatchTest (dense, sparse and all-null projections end to end; the dense case asserts one reader call per file through a spied reader) and ReadPointAlignedCrossCompactionTest cover the device-absent path.

🤖 Generated with Claude Code

@JackieTien97 JackieTien97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the optimization. I checked out 3fba980 and ran the PR's tests locally (FileLoaderUtilsTest, TimeSeriesMetadataCacheTest, AlignedSeriesScanBatchTest); all pass. I also added a throwaway probe test (not pushed) that writes a real aligned device with 700 columns (three measurement leaf groups) and compares getDeviceTimeSeriesMetadata against the per-column get(): all 701 series (time column plus 700 fields) matched, including statistics and chunk-metadata counts. So I found no wrong-result issue. The comments below are mostly about performance, diagnostics and metrics.

Main points:

  1. [P1] The batch path only counts as a cache hit when every selected sensor is in the LRU. If any selected column is absent from a file and that absence can't be cached as a placeholder (ignoreNotExists=true or may_cache_nonexist_series=false), every query re-reads the whole device. The per-column path doesn't (TimeSeriesMetadataCache L362).
  2. [P1] The missing-device error message describes the opposite inconsistency (L491).
  3. [P2] Metadata IO size and load counts are no longer comparable with the per-column path (L483, L370).
  4. [P2] The I/O saving only exists when the selected columns span more than one measurement leaf group (more than 256 series by default), and the gate doesn't reflect that (FileLoaderUtils L447). Could the same batching be done inside the existing get() miss path instead of a parallel path (L342)?
  5. The PR description is out of date. It still describes a configurable device_metadata_batch_scan_ratio (default 0.8), an IoTDBConfig change, an ignoreNotExists gate and readDeviceMetadata. The code hard-codes 0.6 with no switch, has no such gate, and uses getDeviceTimeseriesMetadata. Could you update it and add benchmark numbers for a wide device (e.g. 800 columns, cold and warm cache) compared with master? That would make it easier to judge the 0.6 threshold.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved batch-path issues can trigger unnecessary device reads and absent-device I/O errors, while regression tests leave sparse all-null rows and multi-leaf field values unverified.

5 open issues
What changed in this PR

This PR reduces repeated metadata-index traversal in DataNode wide-table queries by loading a device’s metadata in one batch when most field columns are selected.

Changes:

  • Adds cache- and bloom-filter-aware batch loading with per-series query statistics.
  • Gates batching by table width and a 60% projection threshold, excluding DEBUG and offset-addressed scans.
  • Shares aligned metadata assembly, adds localized missing-device errors, and introduces regression tests.
File Description
iotdb-core/​datanode/​src/​test/​java/​org/​apache/​iotdb/​db/​storageengine/​dataregion/​read/​reader/​series/​AlignedSeriesScanBatchTest.java Adds a real-file wide-device batch scan regression.
iotdb-core/​datanode/​src/​test/​java/​org/​apache/​iotdb/​db/​storageengine/​buffer/​TimeSeriesMetadataCacheTest.java Tests batch loading, caching, missing devices, and metadata equivalence.
iotdb-core/​datanode/​src/​test/​java/​org/​apache/​iotdb/​db/​queryengine/​execution/​operator/​source/​FileLoaderUtilsTest.java Tests batch eligibility boundaries and exclusions.
iotdb-core/​datanode/​src/​main/​java/​org/​apache/​iotdb/​db/​storageengine/​buffer/​TimeSeriesMetadataCache.java Implements cache-aware device metadata loading and statistics accounting.
iotdb-core/​datanode/​src/​main/​java/​org/​apache/​iotdb/​db/​queryengine/​plan/​planner/​plan/​parameter/​SeriesScanOptions.java Carries the table’s total field-column count.
iotdb-core/​datanode/​src/​main/​java/​org/​apache/​iotdb/​db/​queryengine/​plan/​planner/​DataNodeTableOperatorGenerator.java Supplies field counts for eligible table scans.
iotdb-core/​datanode/​src/​main/​java/​org/​apache/​iotdb/​db/​queryengine/​execution/​operator/​source/​relational/​ExternalTsFileSeriesScanUtil.java Keeps external offset-addressed scans on the per-column path.
iotdb-core/​datanode/​src/​main/​java/​org/​apache/​iotdb/​db/​queryengine/​execution/​operator/​source/​FileLoaderUtils.java Selects the batch path and shares aligned metadata assembly.
iotdb-core/​datanode/​src/​main/​java/​org/​apache/​iotdb/​db/​queryengine/​execution/​operator/​source/​AlignedSeriesScanUtil.java Passes the field count into metadata loading.
iotdb-core/​datanode/​src/​main/​i18n/​zh/​org/​apache/​iotdb/​db/​i18n/​StorageEngineMessages.java Adds the Chinese missing-device error message.
iotdb-core/​datanode/​src/​main/​i18n/​en/​org/​apache/​iotdb/​db/​i18n/​StorageEngineMessages.java Adds the English error naming the device and TsFile.

Review effort: Max

@ColinLeeo
ColinLeeo requested a balanced review from Copilot November 16, 2026 03:37
Copilot stopped reviewing on behalf of ColinLeeo due to an error November 16, 2026 03:38

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Aggregation scans can unintentionally take the whole-device batch path, while cold-cache fallback underreports metadata-load time by omitting initial cache probes and device-lock waits.

2 open issues

Review effort: Balanced

@JackieTien97 JackieTien97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the quick turnaround. Status of my earlier comments: 9 of 12 are fixed, including both P1s. Still open by design: the batch read's IO bytes are not reported, the 0.6 ratio has no off switch, and the selective reader is an upstream follow-up.

Since this PR is a performance change, I benchmarked it against its base. A JUnit harness (not pushed) drives FileLoaderUtils.loadAlignedTimeSeriesMetadata over real TsFiles written with TsFileWriter (12 files, aligned devices, 4 chunks per series). It counts the bytes and read calls that reach the file and runs each variant in its own JVM (surefire defaults with -Xmx1024m, which gives a ~61 MB TimeSeriesMetadataCache). I measured f8b9c83 against ca8074d. The only batch-path change since then, 469394f, is time accounting, so the numbers apply to 6ab4b8e. Loaded metadata was identical to base in every scenario (checksums), so there is no correctness issue here.

Two runs per variant; ms per query (one device across 12 files), cold / warm:

Scenario base PR PR + the two changes below
Dense, 700 fields, select all, cache fits 5.1–5.3 / 1.7–1.8 8.5 / 1.5–2.7 5.5–5.9 / 1.5–1.7
Dense, 700 fields, select first 60% 3.4–3.6 / 1.0–1.1 5.3–5.4 / 1.0 4.0–4.4 / 1.1
Repeated select-all, working set larger than the metadata cache (steady state) 4.5–9.9 ms, 1.7–4.1 MB per query 10.1–16.1 ms, 11–15 MB per query 4.4–5.4 ms, 1.0–2.0 MB per query
Sparse, ~20 of 700 fields per file (the motivating case) 4.8–4.9 / 4.7 2.8–2.9 / 2.1 1.9–2.0 / 2.1
100-field table (below the gate) 0.9 / 0.27 unchanged unchanged
Cold load, 96 devices x 300 fields, 8 / 32 threads (wall ms) 225–232 / 237–247 208–220 / 193–210 199–213 / 186–189

The sparse case gains about 2x as intended, and the lock stripes do not hurt concurrency. There are two regressions, though. Each has an inline comment with the change that removed it locally:

  1. [P1] Under metadata-cache pressure, the "cache holds most of the request, so delegate to per-column get()" fallback multiplies metadata reads 3–8x.
  2. [P1] For dense wide devices, a cold call costs 45–70% more time than base. It moves the same bytes in fewer reads, so the extra cost is CPU: every selected measurement goes through two cache passes and a BloomFilter probe before the device read.

Two smaller points:

  • With a 60% prefix projection, the device-wide read still loads a leaf group that no selected column needs, so it stays 10–30% slower even after both changes. That cost remains until the selective reader lands.
  • The description says getDeviceTimeseriesMetadata costs about 2x IO. In these runs the batch read moved the same bytes as the per-column path (2019.9 KB vs 2021.0 KB per query, in 48 vs 72 reads). Also, the gate in the code is >=, while the description says "more than".

I'm happy to share the harness if it helps.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved issues alter per-series counters, allow repeated metadata reads and unnecessary sibling loading, and leave shared test state dependent on execution order.

4 open issues
2 resolved since last review

Review effort: Balanced

@ColinLeeo
ColinLeeo requested a balanced review from Copilot November 18, 2026 02:34

@JackieTien97 JackieTien97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The set-based reader is a nice redesign, and it resolves two of the three regressions from my previous review:

  • Under cache pressure, steady-state metadata IO is now below base. Repeated select-all reads 1.0–1.3 MB per query against base's 1.7–2.0 MB, and takes 4.3–4.5 ms against 5.1–7.0 ms.
  • A 60% prefix projection now reads the same bytes as base.

I reran the same harness on aa52c51 and on its merge base d5c1534. It uses real TsFiles (12 files, aligned and non-aligned devices, 4 chunks per series), counts bytes and reads at the file, and runs each benchmark in its own JVM (two runs per build, -Xmx1024m). Loaded metadata matched base in every scenario. The PR's tests plus ReadPointAlignedCrossCompactionTest (37 cases) pass locally.

ms per query (one device across 12 files), cold / warm:

Scenario base PR
Dense, 700 fields, select all, cache fits 5.7 / 1.9–2.2 6.6–7.5 / 1.9–2.0
Dense, 700 fields, first 60% 3.5–3.8 / 1.0–1.4 4.0–4.2 / 1.0–1.2
Dense, 1 column 1.0 0.5–0.6
Re-query after evicting 1% / 30% 5.5–5.9 / 6.9–7.0 3.0–3.3 / 5.2–6.0
Repeated select-all, working set larger than the cache (steady state) 5.1–7.0, 1.7–2.0 MB per query 4.3–4.5, 1.0–1.3 MB per query
10 random fields absent per file, warm 2.1–2.5, 0 KB 3.0–3.2, 1.9 MB / 46 reads per query
10 fields added later (absent in every file), warm 2.1–2.2, 0 KB 2.4–2.5, 517 KB / 24 reads per query
Sparse, ~20 of 700 fields per file 5.0–5.2 / 4.9–5.0, warm 0 KB 2.1–2.3 / 2.2–2.3, warm 59 KB / 24 reads
100-field table 0.9–1.0 / 0.3 1.0 / 0.25
Tree model: 300 non-aligned series, loaded one by one 2.9–3.1 / 1.1 4.1–4.4 / 1.7–2.1
Tree model: 5 of 300 non-aligned series 0.53 / 0.01 0.29–0.40 / 0.03
Cold load, 96 devices x 300 fields, 8 / 32 threads (wall ms) 221–227 / 222–233 163–189 / 160–161

Still open (each has an inline comment):

  1. [P1] A selected column that is absent from a file is never cached as absent, so every warm query re-reads the leaf groups it maps into. Base does no metadata IO for such columns once warm. Adding columns to a wide table leaves every older file without them, so this will be common.
  2. [P1] The single-column get() is 33–53% slower cold and 48–96% slower warm. This is the tree-model path (300 non-aligned series per device). Bytes are identical; the cost is per-call allocation, even on cache hits.
  3. [P2] Dense cold loads are still 15–32% slower, and 7–18% for the 60% projection, in line with the numbers in the description.
  4. [P2] Offset-addressed (external) reads now consult the BloomFilter. The javadoc says this is skipped, and master did skip it.

Also, tsfile.version points at 2.5.1-261117-SNAPSHOT. That snapshot is built from apache/tsfile#1123, which is still open with no maintainer approval and was republished on 11-18. This PR can't merge until that change is reviewed, merged and released.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread pom.xml Resolved
A table scan that selects (nearly) all field columns of a device previously
deserialized the device's TimeSeriesMetadata column by column, re-walking the
device metadata index once per leaf node. Add a batch path that reads the whole
device's metadata in a single pass via TsFileSequenceReader.readDeviceMetadata,
populating the per-measurement cache and building the aligned metadata through
the existing modification/loader logic.

The batch path is taken when the selected field columns reach a configurable
fraction (device_metadata_batch_scan_ratio, default 0.8) of the table's total
field columns.
The batch path read a device's TimeseriesMetadata via
TsFileSequenceReader.readDeviceMetadata, which deserializes only the
header and statistics and leaves chunkMetadataList null. The downstream
DiskAlignedChunkMetadataLoader dereferences that list in
getCopiedChunkMetadataList, so a wide-table scan NPE'd at query time, and
the null-chunk entries also polluted the shared per-measurement LRU cache
(breaking later narrow queries on the same device).

Switch to getDeviceTimeseriesMetadata, which deserializes chunk metadata
too, and adapt the unit test to the list-based API, asserting that chunk
metadata flows through to the result.
Remove the unreachable time-column-only branch in the batch loader (the
batch gate already requires a non-empty selection), and validate
device_metadata_batch_scan_ratio so a non-positive value falls back to
the default instead of routing every table scan through the batch path.
Document the valid range in the properties template.
Remove the ignoreNotExists gate so the batch path is also taken for files
with a device-level time index, where previously only file-level-index
files (or recovery) used it. When the batch read finds the device is
missing from the file, replicate the per-column path: skip silently when
ignoreNotExists is set, otherwise throw CorruptedTsFileException to surface
the time-index/metadata inconsistency.
Skip the device read when the time column is known absent, so a bloom
filter false positive on a field cannot force an absent-device read.
Fall back to per-column lookups for a small remainder (one or two
measurements), which reads a single leaf group instead of the whole
device, and count the cold-call cache probe and lock wait in the disk
lookup time.

Strengthen the wide-scan tests: assert timestamps and every field value,
compare a sparse batch scan against the per-column path, cover an
all-null projection under ignoreAllNullRows, and add cache regressions
for the filtered time column and the single-miss fallback.
TsFileSequenceReader is an external apache/tsfile dependency and has no
batch API that accepts a measurement set, so the batch path still reads
the whole device and filters afterwards. Note the constraint and the
existing mitigations (bloom filter, per-column fallback) at the call
site; a truly selective reader needs an upstream change.
Load the remaining misses through the per-column get() when the cache
already holds most of the request: it touches only their leaf groups
instead of re-reading the whole device. Replace the GC-sensitive
WeakHashMap monitor with fixed lock stripes, and make the returned map
complete for allSensors on every early return.

Give aggregation table scans a totalFieldColumnCount so they can use the
batch path, and make the fan-out gate span two leaf groups (>=), fixing
its off-by-one and comment.

Tests: production ignoreAllNullRows=false coverage including all-null
projections, batch negative-caching parity, copy-on-return, a three-miss
eviction case, the multi-leaf premise, and per-chunk offset/statistics
comparison.
When a cold batch call delegates the remaining measurements to the
per-column get(), the cache probes and lock wait were assigned to cache
time while the cache hit count was zero, so neither branch reported
them. Attribute the phase to disk time when there are no cache hits and
allow a disk time with a zero count to be recorded.
Replace the device-wide batch read and its 0.6 projection gate with one
getTimeseriesMetadata(Set) entry point: resolve cached columns first,
then read each measurement leaf group that holds a selected column
once, bloom-probing only the group probe and answering every requested
column of the group from that read. get() becomes a single-column
wrapper over it.

This removes the static ratio, the parallel batch branch, the
totalFieldColumnCount planner plumbing and the now-unused i18n
exception constant. The cache and wide-device scan tests are rewritten
for the new semantics.
An aligned device always has its time column, so when the requested set
contains it and the cache or the bloom filter proves it absent, no
selected series of that device can be read from this file. The
per-column loader returned null at that point; the set-based loader kept
probing the remaining measurements, and a bloom-filter false positive on
any of them sent the device metadata index lookup into the reader, which
throws when the device is not in the file (e.g. a seq file that only
holds a subset of the devices read by cross compaction).

Probe the time column first and stop the load when it resolves to null.
Found by ReadPointAlignedCrossCompactionTest.test21.
The requested measurements were only added to the pending set by the
cache pass. The cache-disabled and external-TsFile branch skipped that
pass without seeding the set, so it stayed empty, no leaf group was
read and the call returned nothing: read_tsfile queries returned zero
rows on every external TsFile.

Seed pending from the requested set in that branch and add a regression
test that loads an external file through getTimeseriesMetadata with the
device metadata offset.
Count disk lookups per requested measurement, cache hits included in
the same total, as the per-column path did, so EXPLAIN ANALYZE and the
monitoring counters stay comparable; leaf-read behavior stays covered
by the cache tests rather than by the counter.

When the metadata cache is disabled, filter the group read by the
requested measurements instead of allSensors, so a non-aligned get()
does not deserialize and drop its siblings' chunk metadata. Restore the
shared test context to SERIES_SCAN after the external-file test, and
document that the once-per-group guarantee needs a bloom filter.
Keep only the helpers with several call sites or real logic: inline the
time-column and bloom predicates, rebuild the probe order where it is
used, and return the cache-hit count from loadLeafGroups instead of
wrapping it in a one-field class.
Replace the per-probe leaf-group loop with one call to
TsFileSequenceReader.readTimeseriesMetadata(device, offset,
measurements, returnedMeasurements, ...), which resolves the pending
measurements against the leaf groups that hold them, reads each visited
group once, and also returns the group siblings a single-column call
needs for cache warming.

The cache keeps the cache pass, the bloom-filter device gate, the
negative placeholder and the per-series counters; the probe loop,
probeFound, the group filter and the per-column bloom probes are gone.
The aligned loader now always requests the time column.

This needs the tsfile change that adds the set-based overload; the
tsfile dependency must be bumped to a version containing it before this
branch can be pushed.
The set-based readTimeseriesMetadata overload is not in the dated tsfile
snapshot the build used before. Point the dependency at the 2.5.1
snapshot, which will contain it once apache/tsfile#1123 is merged and
published.

The IoTDB change that uses the new API must not be pushed until that
snapshot exists in the configured repositories.
The build must not resolve the shared 2.5.1-SNAPSHOT, whose content can
change under it. Depend on the dated 2.5.1-261117-SNAPSHOT that carries
the selective metadata reader, and publish that snapshot before pushing
the branch so CI no longer resolves a shared moving artifact.
Cache the TimeSeriesMetadataCacheKey hash in the constructor instead of
allocating the Objects.hash argument array on every lookup and write,
and pass the pending set straight to the tsfile reader when the aligned
batch path requests exactly the set it caches.
Each cache-missed measurement is probed against the file's bloom filter:
a measurement the filter rules out is answered with null without a read
and without a negative cache entry, because probing the filter again is
cheaper than one entry per absent column per file. The measurements the
filter cannot rule out go to the reader in one call, and every one the
reader does not return is remembered as absent, so a later query does
not walk its leaf group again.

Offset-addressed external TsFiles may not have a bloom filter that
covers their series, so bloom filtering is skipped for them. The
single-column special case of the device gate is gone because the
per-column probe covers it.

The tsfile dependency moves to the dated snapshot that carries the
renamed set-based reader.
- Answer a single-series cache hit before building the batch request
  (singleton set, result map, pending set) or the device prefix.
- Time the BloomFilter load, so loadBloomFilterTime is no longer always
  zero in EXPLAIN ANALYZE and the monitoring counters.
- Re-check pending entries under the device lock: a thread that waited
  no longer re-reads what another thread has just cached.
- Skip BloomFilter loading in the cache branch for offset-addressed
  files, as the method javadoc promises, not only in the disabled-cache
  branch.
- Include the pending set in `returned`; a miss outside allSensors
  would otherwise be mistaken for an absent series and pollute the
  negative cache. Reuse allSensors without copying when nothing is
  cached, which is the single-column cold path.
- Pass the query debug flag to the BloomFilter cache again.
- Test hygiene: delete the external fixture, bound the hasFile retry,
  and assert one reader call per file in the wide aligned scan.
@ColinLeeo
ColinLeeo force-pushed the colin/batch-scan-query-a2e8cc branch from 7e24c11 to 37ca9c7 Compare November 19, 2026 06:29
- Drop the redundant time-column guard in the aligned loader:
  getAllSensors already adds the time column.
- Build the device string once per call and reuse it for the lock and the
  BloomFilter prefix.
- Recognize the aligned batch caller by set identity, so the equal
  contents of its two set arguments are known without comparing them.
- Document the roles of pending, requested and allSensors in
  readAndMerge.
- Remove every loaded measurement from pending and remember the
  remainder directly; the final fill pass is no longer needed.
- Drop the cached hash field of the cache key.
- Fold buildAlignedTimeSeriesMetadata back into its only caller: the
  device-wide batch path that shared it is gone.
- Rename FragmentInstanceContext.ignoreNotExistsDevice to
  isStorageEngineRecovering. It is set at query start from
  !StorageEngine.isReadyForNonReadWriteFunctions(), so it reports whether
  the data regions were still recovering, not whether a missing device
  can be ignored. The cache and reader parameters keep their policy
  names because they also cover the FILE_TIME_INDEX case.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

- Check the aligned time column before the per-column cache pass: a file
  without data for the device then costs one lookup and at most one
  BloomFilter probe instead of one lookup per selected measurement.
- Load the BloomFilter and reject excluded measurements before taking the
  device lock; the locked recheck now only sees the survivors, so a warm
  sparse scan does not recheck every excluded column under the lock.
- Derive diskLookupCount in finally, so the device-gate early returns and
  the partial-cache paths are counted and the cache/disk time split uses
  the real miss count.
- Hold an external reader reference in the external-TsFile test and
  release it before deleting the fixture; the internal close helper never
  saw the external registry entry.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

When only the time-column entry is evicted, BloomFilter loading is misattributed between cache and disk phases, potentially producing negative disk-time statistics and metadata timers.

1 open issue
3 resolved since last review

Review effort: Balanced

ColinLeeo and others added 3 commits November 19, 2026 17:33
The time-column check can load the BloomFilter before the cache phase
ends. Subtracting that load from the disk interval, which does not contain
it, could record negative disk time and metadata timers. Track the load
before and after the cache phase separately and subtract each from its own
interval.

Add a regression case that evicts only the time-column entry and makes
the BloomFilter load slow, so the accounting error would turn the
recorded disk time negative.
apache/tsfile#1123 is merged on develop (squash 0d89263e); the dated
snapshot 2.5.1-261120-SNAPSHOT is built from that commit and published to
Apache snapshots.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants