Skip to content

HIVE-30264 - #6966

Open
illiabarbashov-sketch wants to merge 9 commits into
apache:masterfrom
illiabarbashov-sketch:HIVE-30264
Open

illiabarbashov-sketch wants to merge 9 commits into
apache:masterfrom
illiabarbashov-sketch:HIVE-30264

Conversation

@illiabarbashov-sketch

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

deniskuzZ and others added 3 commits November 12, 2026 11:20
- Hive.getTableConstraints moved to the combined getAllTableConstraints call, which the query-level HMS cache does not
  cover (it caches the per-type constraint calls); every Table copy made during compilation fetched again
- Impact (local, TPC-DS 30 TB perf driver, 201 statements): 1,866 -> 992 calls (the distinct tables), compile 350.3 ->
  283.8 s (-19 %); one statement spent 2.0 s of 2.0 s Calcite plan generation in 26 calls at ~80 ms
- cluster (TPC-DS 10TB Parquet, 24 x r6id, 198 query-iterations): warm compile 386.8 -> 383.6 s, no wall gain
  against the cluster HMS
- the cache hands out a copy: Hive.getTableConstraints keeps only the RELY constraints in the object it gets
- test: one call per query; the copy (mutation: the cached object itself) keeps the second result whole
Cut the HMS calls made while compiling a query:
- Do not fetch constraints for materialized view scans in the registry; materialized views
  cannot have constraints.
- Share the getTable entry with the write id list lookup: getValidWriteIdList sent a request
  without getColumnStats/engine, so the same table took a second HMS call. The TABLE cache key
  is now a copy of the caller's request.
- Cache getCheckConstraints and getDefaultConstraints, fetched once per MERGE branch and per
  insert destination.
- Cache getPartitionWithAuthInfo for the static partition of an insert target, handing out a
  copy as ql Partition changes the object it gets; a missing partition is not cached.
- Remember tables missing from the metastore (CTE names, optional lookups) within a
  compilation; createTable drops the entry.
- Pass the Table to getPartitionColumnStatistics instead of fetching it again.
- HiveMetaStoreClientWithLocalCache takes the table for its watermark and transactional check
  from the query-level cache instead of two getTable RPCs per statistics/partition call.

TPC-DS 30TB perf driver (201 statements), on top of the getAllTableConstraints cache:
HMS RPCs 5956 -> 3148 (master: 6830), median summed compile time 180.2 s (master) -> 147.9 s.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…e poll

The poll thread asked Hive.get() for a client, which builds a default HiveConf; a metastore set
in the HS2 configuration only (as in q-tests) was then not the one polled.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…ache is created

ExplainSemanticAnalyzer creates the query-level HMS cache without SemanticAnalyzer.startAnalysis,
so EXPLAIN queries never used the cache by versions.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Key the HS2-level cache of aggregate column statistics by the partitions only and keep
each column's aggregate in the entry, so that a request fetches only the columns not
cached yet. partsFound is kept per column; the combined value is their minimum, which
equals the number of partitions exactly when the value of the full request does.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…ional tables without a writeIdList

Hive.getAggrColStatsFor passes no writeIdList unless it checks the statistics against the
snapshot, which left the watermark of a transactional table invalid, so these calls were
never cached. Take the snapshot of the query instead, as getTableColumnStatistics does.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@sonarqubecloud

Copy link
Copy Markdown

/**
* The watermark of a table: by its write ids if it has them, else by its version if the query may use those.
*/
private TableWatermark watermark(String validWriteIdList, String catName, String dbName, String tblName)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With no write-id list and no version cache for the query (hive.metadata.cache.nontransactional.mode=off, the default), this watermark can never be valid, but it still fetches the table first. Outside compilation, or with the query-level cache off, that's a metastore call per partition or statistics lookup of every non-transactional table. Could it return before the fetch?

if (validWriteIdList == null && getQuerySettings() == null) {                                                                                                                                                
  return new TableWatermark(null, -1);  // no write ids and no versions: never valid                                                                                                                         
}   

A unit test that counts the delegate's getTable calls for a non-transactional table with the mode off would keep it that way.

"Comma-separated list of class names extending EventConsumer," +
"to handle the NotificationEvents retrieved by the notification event poll."),

HIVE_METADATA_CACHE_NONTRANSACTIONAL_MODE("hive.metadata.cache.nontransactional.mode", "off",

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2 safety questions about the cache for non-transactional tables:

  1. If the metastore has no DbNotificationListener, no events arrive, and strict would treat the cache as current for up to max.staleness. Can HiveServer2 detect that (for example, the event id never moves while the cache sees writes) and keep the cache off instead of trusting it?

  2. Should hive.disable.unsafe.external.table.operations=true also keep the cache off for external tables? A user who sets it doesn't trust external tables to stay unchanged, so following that setting would be the conservative choice.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants