Repository navigation
HIVE-29681: Introduce contextual table search - #6595
dengzhhu653 wants to merge 36 commits into
Conversation
There was a problem hiding this comment.
Pull request overview
This PR introduces a new metastore-search module to the standalone metastore, adding an embedded Lucene-based indexing/search subsystem with hybrid (keyword + semantic) retrieval, embedding-model integration, and backup/restore mechanics, and wires it into distribution packaging.
Changes:
- Adds a new
standalone-metastore/metastore-searchMaven module and includes it in standalone-metastore build + assembly packaging. - Implements Lucene-backed indexing/search components (mapping, query contract, hybrid fusion, embedding inference + caching, index state/backup utilities).
- Adds extensive unit/integration tests and supporting test utilities for indexing/search behavior and recovery scenarios.
Reviewed changes
Copilot reviewed 89 out of 89 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| standalone-metastore/pom.xml | Adds metastore-search module to standalone metastore reactor build. |
| standalone-metastore/packaging/src/assembly/src.xml | Includes metastore-search artifacts in standalone metastore assembly. |
| standalone-metastore/packaging/pom.xml | Adds packaging dependency on hive-standalone-metastore-search. |
| standalone-metastore/metastore-search/src/test/resources/log4j2.properties | Test logging config for the new module. |
| standalone-metastore/metastore-search/src/test/java/org/apache/hive/search/testutil/TestLeaderElection.java | Test leader-election stub used by indexer tests. |
| standalone-metastore/metastore-search/src/test/java/org/apache/hive/search/testutil/StubEmbedModel.java | Deterministic embedding model for tests (no ONNX runtime). |
| standalone-metastore/metastore-search/src/test/java/org/apache/hive/search/testutil/RealMetastoreServer.java | In-process HMS server fixture for integration tests. |
| standalone-metastore/metastore-search/src/test/java/org/apache/hive/search/testutil/MetastoreBootstrapMocks.java | Mockito helpers for bootstrap/indexer initialization tests. |
| standalone-metastore/metastore-search/src/test/java/org/apache/hive/search/testutil/InMemoryIndexStateClient.java | In-memory IndexStateClient used across backup/restore tests. |
| standalone-metastore/metastore-search/src/test/java/org/apache/hive/search/testutil/IndexMutationApplier.java | Applies metastore event tasks to the Indexer in tests. |
| standalone-metastore/metastore-search/src/test/java/org/apache/hive/search/search/TestSearchQuery.java | Tests SearchQuery defaults and validation. |
| standalone-metastore/metastore-search/src/test/java/org/apache/hive/search/search/TestHybridSearch.java | Tests hybrid query parsing and fusion-request generation. |
| standalone-metastore/metastore-search/src/test/java/org/apache/hive/search/search/InMemorySearchFixture.java | In-memory Lucene+stub-embedding search fixture for tests. |
| standalone-metastore/metastore-search/src/test/java/org/apache/hive/search/metastore/TestRealMetastoreSearchIntegration.java | Integration tests for bootstrap + incremental updates against real HMS. |
| standalone-metastore/metastore-search/src/test/java/org/apache/hive/search/metastore/TestRealMetastoreSearchHybrid.java | Integration tests for keyword/hybrid behavior and ranking. |
| standalone-metastore/metastore-search/src/test/java/org/apache/hive/search/metastore/TestRealMetastoreSearchFailureRecovery.java | Integration tests for restore and poison-event recovery behavior. |
| standalone-metastore/metastore-search/src/test/java/org/apache/hive/search/metastore/TestMetastoreIndexer.java | Tests leader/follower bootstrap and backup restore flows. |
| standalone-metastore/metastore-search/src/test/java/org/apache/hive/search/metastore/TestMetastoreEventHandlerIntegration.java | Tests event polling and mutation coalescing integration. |
| standalone-metastore/metastore-search/src/test/java/org/apache/hive/search/metastore/TestBootstrapIndexer.java | Tests bulk bootstrap indexing behavior and failures. |
| standalone-metastore/metastore-search/src/test/java/org/apache/hive/search/mapping/TestIndexMapping.java | Tests default schema/hybrid-field exposure and analyzers. |
| standalone-metastore/metastore-search/src/test/java/org/apache/hive/search/inference/TestEmbedModelRegistry.java | Tests model registry behavior and deterministic stub vectors. |
| standalone-metastore/metastore-search/src/test/java/org/apache/hive/search/inference/TestEmbeddingPrompt.java | Tests embedding prompt prefix logic. |
| standalone-metastore/metastore-search/src/test/java/org/apache/hive/search/inference/TestEmbeddingCache.java | Tests embedding cache scoping, copying, and hit/miss tracking. |
| standalone-metastore/metastore-search/src/test/java/org/apache/hive/search/index/TestIndexManagerFailureRecovery.java | Tests unhealthy index state gating. |
| standalone-metastore/metastore-search/src/test/java/org/apache/hive/search/index/TestIndexerSearchIntegration.java | Tests end-to-end indexing/search over in-memory fixture. |
| standalone-metastore/metastore-search/src/test/java/org/apache/hive/search/index/TestIndexerFlushCommit.java | Tests commit/flush thresholds and forced commits. |
| standalone-metastore/metastore-search/src/test/java/org/apache/hive/search/index/TestIndexerEmbeddingCache.java | Tests embedding cache dedupe behavior in indexing pipeline. |
| standalone-metastore/metastore-search/src/test/java/org/apache/hive/search/index/store/TestIndexBackupUtils.java | Tests backup sync/restore correctness and staging handling. |
| standalone-metastore/metastore-search/src/test/java/org/apache/hive/search/index/store/TestIndexBackupFailureRecovery.java | Tests interrupted/partial restore and stale-file cleanup behavior. |
| standalone-metastore/metastore-search/src/test/java/org/apache/hive/search/index/manifest/TestIndexManifest.java | Tests manifest JSON, diffing, and equality semantics. |
| standalone-metastore/metastore-search/src/test/java/org/apache/hive/search/config/TestSearchConfig.java | Tests SearchConfig defaults and overrides. |
| standalone-metastore/metastore-search/src/test/java/org/apache/hive/search/config/TestIndexStateConfig.java | Tests IndexStateConfig remote/memory flags and URI validation. |
| standalone-metastore/metastore-search/src/test/java/org/apache/hive/search/config/TestIndexConfig.java | Tests IndexConfig defaults and overrides. |
| standalone-metastore/metastore-search/src/main/java/org/apache/lucene/search/package-info.java | Documents copied Lucene-like classes placed under Lucene package namespace. |
| standalone-metastore/metastore-search/src/main/java/org/apache/lucene/search/BayesianScoreEstimator.java | Adds Bayesian score parameter estimation utility used by fusion scoring. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/search/TableSearchResult.java | Public-ish result wrapper for table search hits and totals. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/search/TableSearchHit.java | Search hit type with table identity, score, and returned fields. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/search/SearchReqResp.java | Internal request/response records and search mode enum. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/search/SearchQuery.java | Narrow public query contract and validation helpers. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/search/SearchBackend.java | Pluggable backend interface for search implementations. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/search/LuceneSearchBackend.java | Embedded Lucene backend implementation + async init readiness. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/search/HybridSearch.java | Hybrid query parsing and fusion request builder. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/metastore/MetastoreTableMapper.java | Maps HMS Table metadata into a Lucene document schema. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/metastore/MetastoreSchemas.java | Defines the default index mapping for Hive tables. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/metastore/MetastoreEventListener.java | Listener contract for coalesced indexing tasks and health signals. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/metastore/MetastoreCluster.java | Leader-election wiring for distributed index coordination. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/mapping/TableDocument.java | Builds Lucene Documents from mapped fields (lexical/filter/semantic). |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/mapping/SearchParams.java | Field search parameters (lexical/semantic/hybrid + distance). |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/mapping/IndexMapping.java | Index schema container with analyzers and hybrid-field helpers. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/mapping/FieldSchema.java | Schema types describing field storage/filter/search behavior. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/mapping/field/TextField.java | Field value + optional embedding. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/mapping/field/IdField.java | Identifier field wrapper. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/mapping/field/Field.java | Field interface and Javadoc. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/inference/LocalOnnxEmbeddingModel.java | ONNX embedding model adapter with single-thread inference executor. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/inference/EmbedModelRegistry.java | Model registry creation/warmup + optional embedding cache plumbing. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/inference/EmbedModel.java | Embedding model interface and task types. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/inference/EmbeddingPrompt.java | Task-based prefixing for model input strings. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/inference/EmbeddingCache.java | In-memory embedding cache keyed by model/task/text. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/index/store/RemoteStateClient.java | Hadoop FS-backed remote index state storage for backup/restore. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/index/store/LocalStateClient.java | Lucene Directory-backed local index state reader/writer + validation. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/index/store/IndexStateClient.java | Storage abstraction for manifests and index file operations. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/index/store/IndexBackupUtils.java | Backup sync/restore logic and staging manifest flow. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/index/manifest/IndexManifest.java | JSON manifest model + file diffing ops. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/index/IndexSession.java | Co-locates indexer + searcher manager + periodic refresh. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/index/IndexManager.java | Opens local/remote index state and manages health + backup flows. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/index/FlushTrackingWriter.java | Tracks Lucene flushes to gate commit frequency. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/exception/SearchException.java | Search-layer checked exception type. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/exception/InitializeException.java | Initialization exception type + wrapper helper. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/exception/IndexNotReadyException.java | Not-ready exception used by readiness checks. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/exception/IndexNotHealthyException.java | Health-gating exception surfaced when index is unhealthy. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/exception/IndexIOException.java | IO exception type for index state operations. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/exception/IndexException.java | Indexing exception type + wrapper helper. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/config/SearchConfig.java | Search tuning knobs (refresh interval, hybrid weights, init timeout, etc.). |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/config/InferenceConfig.java | Embedding model configuration and local/remote model file staging. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/config/IndexStateConfig.java | Local/remote index location configuration. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/config/IndexConfig.java | Indexing configuration knobs (flush/commit, bootstrap, poll, etc.). |
| packaging/pom.xml | Adds top-level packaging dependency on standalone-metastore-search artifact. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| Configuration conf = new Configuration(Objects.requireNonNull(configuration)); | ||
| this.indexConfig = new IndexOptions(conf); | ||
| this.clientProvider = Objects.requireNonNull(clientProvider); | ||
| this.deserializer = MessageFactory.getDefaultInstance(conf).getDeserializer(); |
There was a problem hiding this comment.
getDefaultInstance(conf).getDeserializer() assumes every event was written in the currently configured format. Replication code uses MessageFactory.getInstance(event.getMessageFormat()).getDeserializer() for each event. Could we do the same here, so events written before a format change still parse?
| final NotificationEventResponse resp; | ||
| try (MetastoreClientProvider.MetastoreClientBundle clientBundle = | ||
| clientProvider.getMetastoreClient(indexConfig.configuration())) { | ||
| resp = clientBundle.client().getNextNotification(request, true, null); |
There was a problem hiding this comment.
This passes allowGaps=true, while canCatchUp correctly uses false. If the poller is unhealthy or backed off for longer than metastore.event.db.listener.timetolive (1 day by default), the cleaned-up events are silently skipped and this instance keeps serving a stale index while reporting healthy.
Could we detect the gap (first returned id > lastEventId + 1) and throw IndexNotHealthyException, so the index is rebuilt?
| TableName tableName = new TableName(table.getCatName(), table.getDbName(), table.getTableName()); | ||
| task.tablesToAdd.put(tableName, table); | ||
| } | ||
| case MessageBuilder.DROP_TABLE_EVENT -> { |
There was a problem hiding this comment.
Merging assumes a table created earlier in the same batch was never indexed. That’s false when bootstrap overlaps: rebuildLeaderIndex captures the notification id before scanning, so a table created afterwards gets scanned in, and then its CREATE and DROP events replay in one batch and cancel out. The scanned document is never deleted. Deletes are idempotent, so could DROP always add to tablesToDrop (and still remove from tablesToAdd)? The same applies to the old name on a rename at line 362.
There was a problem hiding this comment.
its CREATE and DROP events replay in one batch and cancel out
Very good catch!!!
could DROP always add to tablesToDrop
No, in case if the table is re-created after the drop
We need special handling of the notifications within bootstrap overlaps.
| } else { | ||
| task.tablesToDrop.add(tblNameBefore); | ||
| } | ||
| task.tablesToAdd.put(tblNameAfter, tableAfter); |
There was a problem hiding this comment.
Every Iceberg commit and stats update triggers ALTER_TABLE (metadata_location, snapshot and stats properties). Each one rebuilds the document and re-runs ONNX on text that hasn’t changed, on the poller thread. On a busy catalog this alone can keep inference saturated.
Could we hash the indexed and semantic text and skip the rewrite, or at least reuse the existing vectors, when it’s unchanged? Also, for a non-rename alter, tblNameBefore goes into tablesToDrop and the same name into tablesToAdd, which is a redundant delete.
There was a problem hiding this comment.
Besides the vectors, we also store a copy of the table into the index, see MetastoreTableMapper.FIELD_TABLE_BLOB, it can answer the table object request without asking for ObjectStore.
Unfortunately the Lucene cannot update a field of the document, so here just update the whole document for simplicity.
| ids.add(doc.idField().value()); | ||
| luceneDocs.addAll(doc.toDocuments()); | ||
| } | ||
| delete(ids.toArray(new String[0])); |
There was a problem hiding this comment.
delete(ids) followed by writer.addDocuments(...) isn’t atomic. An NRT reader refresh or a commit between the two calls makes the table disappear from search. writer.updateDocument(new Term("_id_filter", id), doc) for each document (or updateDocuments with a block) replaces it atomically. Separately, the “Deleted N docs” log reads getDocStats().numDocs, which doesn’t reflect buffered deletes, so the count is wrong.
| if (clientProvider == null) { | ||
| IMetaStoreClient client = RetryingMetaStoreClient.getProxy(configuration, true); | ||
| Thread currentThread = Thread.currentThread(); | ||
| clientProvider = conf -> { |
There was a problem hiding this comment.
With this provider, the event poller isn’t the constructing thread, so every getNextMetastoreEvents call (once a second by default when idle) opens and closes a new RetryingMetaStoreClient. That’s a Thrift connection, plus a SASL handshake on Kerberos, every second.
Could the poller hold one client for its lifetime, and each bootstrap fetch worker one client rather than one per batch? That would also remove the need to decide by which thread is calling.
| MetastoreClientProvider clientProvider) throws Exception { | ||
| try { | ||
| this.clientProvider = clientProvider; | ||
| this.client = clientProvider.getMetastoreClient(configuration).client(); |
There was a problem hiding this comment.
If getMetastoreClient throws here, client stays null, and close() in the catch block throws an NPE at client.close(), which replaces the original exception. Also, releaseAfterUse is ignored, so an injected shared client always gets closed.
Could you keep the bundle itself and close it null-safely?
| this.handler = new MetastoreEventHandler(configuration, clientProvider); | ||
| this.handler.addListeners(flushIndexListener, indexManager); | ||
| this.cluster = new MetastoreCluster(configuration, flushIndexListener); | ||
| this.lastEventId = initialize(); |
There was a problem hiding this comment.
initialize() runs inside the constructor, so HMS startup waits for either a full bootstrap (2.5+ days for 2M tables, per the PR description) or waitForRemoteBackup(), which loops on sleep(3000) with no timeout until a remote manifest appears or this node becomes leader. Without a remote store configured, followers spin forever. The design doc (§4.3) says the instance should report not-ready instead.
Could initialization run on a background thread, with isReady() false until it completes and a bounded wait or a fallback to a local rebuild?
There was a problem hiding this comment.
The check happens in LuceneSearchBackend
| Query query = | ||
| switch (request.body()) { | ||
| case MatchQuery match -> compileMatchQuery(match.resolve(mapping), filter); | ||
| case SemanticQuery semantic -> compileSemanticQuery(semantic.resolve(mapping), size, filter); |
There was a problem hiding this comment.
Removing the semanticK multiplier makes k = limit. Moving the filter into the kNN query fixes scoping, but for hybrid queries LogOddsFusionQuery now sees only limit semantic candidates per field, which lowers fusion recall.
Was that intended? I’d keep oversampling (max(limit, limit * multiplier)) at least for the hybrid branch.
| // 1. Enable full ONNX graph optimizations (constant folding, node fusion, etc.) | ||
| opts.setOptimizationLevel(OrtSession.SessionOptions.OptLevel.ALL_OPT); | ||
| opts.setExecutionMode(OrtSession.SessionOptions.ExecutionMode.SEQUENTIAL); | ||
| int effectiveIntraOpThreads = Math.max(1, Runtime.getRuntime().availableProcessors()); |
There was a problem hiding this comment.
intraOpNumThreads = availableProcessors(), and the config parameter is ignored. Combined with the session lock in LocalOnnxEmbedder, each embedding batch can take every core on the HMS host and compete with the Thrift handler threads.
Could this be configurable, with a small default such as 2–4?
| import org.apache.hadoop.hive.metastore.leader.LeaderException; | ||
| import org.apache.hadoop.hive.metastore.leader.LeaseLeaderElection; | ||
|
|
||
| @SuppressWarnings("rawtypes, unchecked") |
There was a problem hiding this comment.
A few smaller cleanups:
-
IndexStateClient.readManifest(Optional): Optional as a parameter is an anti-pattern. Use an overload readManifest(IndexCommit) alongside the no-arg version. RemoteStateClient ignores the argument anyway.
-
LocalStateClient.getCommitUserData returns Map.Entry<Map<String,String>, Collection>. A small record CommitInfo(Map<String,String> userData, Collection files) would read better.
-
MetastoreIndexer.close(): IOUtils.closeQuietly(cluster, handler, flushIndexListener) discards close failures without logging them. The closeQuietly(Closeable, Consumer) form with a LOG.warn would keep them visible.
-
MetastoreClientProvider: the Javadoc should say “An IMetaStoreClient provider”. MetastoreClientBundle could have a clearer name, e.g. ClientLease.
-
IndexBackupUtils.syncToBackup: rename localM to localManifest, and the reflowed if fits on one line.
-
Searcher.readHit: binaryValue().bytes ignores the BytesRef offset and length. Use BytesRef.deepCopyOf(...) or pass offset and length explicitly.
-
SonarCloud reports 307 new issues and 0% coverage on new code. Worth a pass before merge.
|



What changes were proposed in this pull request?
Provide contextual search against the Metastore tables, support natural language, instead of just keyword as of now.
Why are the changes needed?
Make the Metastore more than a static meta repository, also a dynamic, context-aware metadata serving store.
More refer to the design document in the jira.
Does this PR introduce any user-facing change?
Yes, the user-facing API will be vested in HIVE-29721. This would involve lots of thrift changes, I would like to
separate the user-facing change from the core functionality, so it's easy for review and management.
How was this patch tested?
Tested on a Metastore with 2+ millions tables, model granite-embedding-30m-english (768 dimensions), other default settings: