Repository navigation
HIVE-30264 - #6966
HIVE-30264#6966illiabarbashov-sketch wants to merge 9 commits into
Conversation
- Hive.getTableConstraints moved to the combined getAllTableConstraints call, which the query-level HMS cache does not cover (it caches the per-type constraint calls); every Table copy made during compilation fetched again - Impact (local, TPC-DS 30 TB perf driver, 201 statements): 1,866 -> 992 calls (the distinct tables), compile 350.3 -> 283.8 s (-19 %); one statement spent 2.0 s of 2.0 s Calcite plan generation in 26 calls at ~80 ms - cluster (TPC-DS 10TB Parquet, 24 x r6id, 198 query-iterations): warm compile 386.8 -> 383.6 s, no wall gain against the cluster HMS - the cache hands out a copy: Hive.getTableConstraints keeps only the RELY constraints in the object it gets - test: one call per query; the copy (mutation: the cached object itself) keeps the second result whole
Cut the HMS calls made while compiling a query: - Do not fetch constraints for materialized view scans in the registry; materialized views cannot have constraints. - Share the getTable entry with the write id list lookup: getValidWriteIdList sent a request without getColumnStats/engine, so the same table took a second HMS call. The TABLE cache key is now a copy of the caller's request. - Cache getCheckConstraints and getDefaultConstraints, fetched once per MERGE branch and per insert destination. - Cache getPartitionWithAuthInfo for the static partition of an insert target, handing out a copy as ql Partition changes the object it gets; a missing partition is not cached. - Remember tables missing from the metastore (CTE names, optional lookups) within a compilation; createTable drops the entry. - Pass the Table to getPartitionColumnStatistics instead of fetching it again. - HiveMetaStoreClientWithLocalCache takes the table for its watermark and transactional check from the query-level cache instead of two getTable RPCs per statistics/partition call. TPC-DS 30TB perf driver (201 statements), on top of the getAllTableConstraints cache: HMS RPCs 5956 -> 3148 (master: 6830), median summed compile time 180.2 s (master) -> 147.9 s. Co-Authored-By: Claude Opus 5.5 <[email protected]>
…e poll The poll thread asked Hive.get() for a client, which builds a default HiveConf; a metastore set in the HS2 configuration only (as in q-tests) was then not the one polled. Co-Authored-By: Claude Opus 5.5 <[email protected]>
…ache is created ExplainSemanticAnalyzer creates the query-level HMS cache without SemanticAnalyzer.startAnalysis, so EXPLAIN queries never used the cache by versions. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Key the HS2-level cache of aggregate column statistics by the partitions only and keep each column's aggregate in the entry, so that a request fetches only the columns not cached yet. partsFound is kept per column; the combined value is their minimum, which equals the number of partitions exactly when the value of the full request does. Co-Authored-By: Claude Opus 5.5 <[email protected]>
…ional tables without a writeIdList Hive.getAggrColStatsFor passes no writeIdList unless it checks the statistics against the snapshot, which left the watermark of a transactional table invalid, so these calls were never cached. Take the snapshot of the query instead, as getTableColumnStatistics does. Co-Authored-By: Claude Opus 5.5 <[email protected]>
|
| /** | ||
| * The watermark of a table: by its write ids if it has them, else by its version if the query may use those. | ||
| */ | ||
| private TableWatermark watermark(String validWriteIdList, String catName, String dbName, String tblName) |
There was a problem hiding this comment.
With no write-id list and no version cache for the query (hive.metadata.cache.nontransactional.mode=off, the default), this watermark can never be valid, but it still fetches the table first. Outside compilation, or with the query-level cache off, that's a metastore call per partition or statistics lookup of every non-transactional table. Could it return before the fetch?
if (validWriteIdList == null && getQuerySettings() == null) {
return new TableWatermark(null, -1); // no write ids and no versions: never valid
}
A unit test that counts the delegate's getTable calls for a non-transactional table with the mode off would keep it that way.
| "Comma-separated list of class names extending EventConsumer," + | ||
| "to handle the NotificationEvents retrieved by the notification event poll."), | ||
|
|
||
| HIVE_METADATA_CACHE_NONTRANSACTIONAL_MODE("hive.metadata.cache.nontransactional.mode", "off", |
There was a problem hiding this comment.
2 safety questions about the cache for non-transactional tables:
-
If the metastore has no DbNotificationListener, no events arrive, and strict would treat the cache as current for up to max.staleness. Can HiveServer2 detect that (for example, the event id never moves while the cache sees writes) and keep the cache off instead of trusting it?
-
Should
hive.disable.unsafe.external.table.operations=truealso keep the cache off for external tables? A user who sets it doesn't trust external tables to stay unchanged, so following that setting would be the conservative choice.



What changes were proposed in this pull request?
Why are the changes needed?
Does this PR introduce any user-facing change?
How was this patch tested?