Repository navigation
Conversation
deniskuzZ
marked this pull request as ready for review
November 6, 2026 18:16
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The hook fails before job submission for MapReduce V3 merge-on-read DELETE/UPDATE operations with existing position deletes, and LLAP’s soft-reference wrapper caching can cause duplicate delete-map deserialization under memory pressure.
2 open issues
What changed in this PR
This PR reduces duplicate deletion vectors and commit-time memory usage for Iceberg V3 merge-on-read DML by routing each data file’s deletes to one writer and sharing previously planned delete metadata.
Changes:
- Distributes V3 deletes by file path and combines MERGE update/delete branches while preserving affected-row counts.
- Plans previous deletes during job setup and loads their compressed metadata through the query cache.
- Adds regression coverage for DML distribution, existing deletion vectors, branches, and sorted tables.
| File | Description |
|---|---|
| ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java | Adds file-sink-aware storage-handler configuration. |
| ql/src/java/org/apache/hadoop/hive/ql/plan/BaseWork.java | Passes sink descriptors during job configuration. |
| ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java | Skips dynamic-partition context for merge-on-read delete sinks. |
| ql/src/java/org/apache/hadoop/hive/ql/parse/rewrite/sql/WhereClausePatcher.java | Places WHERE before distribution and sorting clauses. |
| ql/src/java/org/apache/hadoop/hive/ql/parse/rewrite/sql/NonNativeAcidMultiInsertSqlGenerator.java | Obtains distribution keys from storage handlers. |
| ql/src/java/org/apache/hadoop/hive/ql/parse/rewrite/sql/MultiInsertSqlGenerator.java | Emits DISTRIBUTE BY before SORT BY. |
| ql/src/java/org/apache/hadoop/hive/ql/parse/rewrite/SplitMergeRewriter.java | Combines delete branches and adjusts row counting. |
| ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveStorageHandler.java | Adds distribution and file-sink configuration hooks. |
| ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java | Supplies query scratch space before handler configuration. |
| ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java | Updates logging terminology for uncounted records. |
| iceberg/iceberg-handler/src/test/results/positive/llap/iceberg_row_lineage_compactions.q.out | Updates expected operator identifiers. |
| iceberg/iceberg-handler/src/test/results/positive/iceberg_v3_dml_distribution.q.out | Records expected DML plans and results. |
| iceberg/iceberg-handler/src/test/queries/positive/iceberg_v3_dml_distribution.q | Adds distribution and sorted-table regression queries. |
| iceberg/iceberg-handler/src/test/java/org/apache/iceberg/mr/hive/writer/TestRewritableDeletes.java | Tests metadata staging and daemon-cache reuse. |
| iceberg/iceberg-handler/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergDeletionVectors.java | Expands deletion-vector and DML integration coverage. |
| iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/writer/WriterBuilder.java | Accepts a supplier for previous-delete metadata. |
| iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/writer/RewritableDeletes.java | Plans, stages, and caches previous deletes. |
| iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/writer/HiveIcebergWriterBase.java | Loads previous deletes for deletion-vector writers. |
| iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/writer/HiveIcebergDeleteWriter.java | Passes the metadata supplier into writer creation. |
| iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java | Configures V3 distribution and job-level delete planning. |
| iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergOutputFormat.java | Connects writers to staged delete metadata. |
Review effort: Balanced
deniskuzZ
force-pushed
the
HIVE-30112
branch
from
November 19, 2026 16:40
00ea5b2 to
7d0e241
Compare
deniskuzZ
force-pushed
the
HIVE-30112
branch
from
November 19, 2026 18:49
1296d90 to
bfa6dac
Compare
… file - HiveStorageHandler#acidDistributeColumns: distribution of the deleted records in the rewritten query - V3: DISTRIBUTE BY FILE__PATH, no SORT BY, so all deletes of a data file reach one DV writer - MERGE with UPDATE and DELETE clauses on V3: one delete branch; the update's insert branch is not counted, rows affected unchanged - HiveStorageHandler#configureJobConf(FileSinkDesc, JobConf): called for every file sink while the job is built - Iceberg plans the previous deletes of a V3 merge-on-read delete sink there, once per job, into a gzipped file, instead of a full-table planFiles per task - the file path is unique per job build and recorded in iceberg.mr.rewritable.deletes.<table> - writers load it once per daemon through the query ObjectCache; a DV writer without it fails - commit and abort delete the recorded file through the table's FileIO - iceberg_v3_dml_distribution.q: plans of V3 DELETE, UPDATE, MERGE - iceberg_row_lineage_compactions.q.out: operator id in a cross-product warning - Impact (cluster, 1TRC V3 DML): 2,299 duplicate DVs and an HS2 OOM (heap 0.37 -> 11.7 GB in 40 s) before; 17,016 DVs for 17,016 files, HS2 peak 5.15 GB after
- the location of the planned deletes comes from the job build only: a value of the key that is not a location of this query under the table's temp dir is dropped while the job is built, and the committer removes no other - only DV writers read the planned deletes - test: a session value of the key is neither read nor removed, for V3 and V2 DML
- RewritableDeletes: plans, loads and cleans up the previous deletes of the DV writers - Temp file named after the query, the cache key adds the snapshot; no location key in the job conf - The delete writer gets the supplier; only the DV writer reads it - Materialized view rebuild distribution moves to HIVE-30227 - SplitMergeRewriter: updateInDeleteBranch, clausePredicate
- Rewritable deletes in the temp dir of the query, named in the job conf; no file without previous deletes - A merge-on-read delete sink gets no dynamic-partition context
- RewritableDeletes.addToJobConf keeps the key only when it holds the location this query plans: a value from the session is dropped, and the deletes are planned - testRewritableDeletesKeyOfTheSessionIsIgnored - iceberg_v3_dml_distribution.q: blank line between the explained MERGE and the MERGE
…e and delete clauses - SplitMergeRewriter: the update and delete clauses never match the same row, so every target (native ACID, Iceberg V2 and V3) gets one delete branch, ((update) OR (delete)), and writes one delete delta or set of delete files - Plans: one file sink and reducer fewer; result rows and rows affected unchanged
Collect the rewritable deletes from the files the DML's scans of the target table read, instead of planning the whole snapshot while the job is built. - The job names a temp dir per target table, under a key of the query id. - Every scan of the table collects the position deletes of the data files it plans during split generation, and writes them into a file of its own in that dir, also when empty. - A DV writer reads and merges all files of the dir on first use, when it writes its DVs, once per LLAP daemon, and fails if no scan wrote a file. The map the writers share is read-only. - Tests: partition- and stats-filtered DMLs collect only the files they scan; a MERGE whose subquery reads the target; a dir named in the session is ignored.
- A scan collects into the subdir of the snapshot it reads, a DV writer reads the one of the snapshot it writes - Tests: DELETE from a branch with a subquery over main, MERGE into main from a branch, MERGE as of an older snapshot - Tests read the scan files through RewritableDeletes
- A DV writer decodes a position delete file once per query (per LLAP daemon or Tez container), not once per data file - WriterBuilder takes the job configuration - The Tez object cache logs a hit at DEBUG, without its value: a cached position delete file logged all its entries at INFO - RewritableDeletes logs the read of the deletes at DEBUG - Test: a V2 position delete file of two data files, merged into their DVs after an upgrade to V3
deniskuzZ
force-pushed
the
HIVE-30112
branch
from
November 20, 2026 06:34
bfa6dac to
1977348
Compare
- RewritableDeletes lists the dir of a snapshot with Utilities.listStatusIfExists, instead of a nested try - RewritableDeletes.location without a nested ternary - scanFiles/isScanFile renamed to collectorFiles/isCollectorFile: the files a Collector wrote - SplitMergeRewriter: the ternary of the update in the delete branch without a negated condition - TestRewritableDeletes: the lambdas of assertThrows make a single call
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.





What changes were proposed in this pull request?
Why are the changes needed?
Bug-Fix: 1TRC V3 DML wrote 2,299 duplicate DVs and ran HS2 out of memory at commit (heap 0.37 -> 11.7 GB in 40 s). After: 17,016 DVs for 17,016 files, HS2 peak 5.15 GB
Does this PR introduce any user-facing change?
Plans of V3 merge-on-read DELETE, UPDATE and MERGE: the deleted records are shuffled by FILE__PATH to the DV writers (a reduce stage where a DELETE was map-only), and a MERGE with UPDATE and DELETE clauses has one delete branch. Plans of a merge-on-read UPDATE or MERGE on a table with write.delete.mode=copy-on-write (V2 too): the deleted records are no longer sorted by the partition transforms and write order. No change to results.
Known limitation: the delete branches of a materialized view rebuild in insert-delete mode (a V3 Iceberg MV) are not distributed by FILE__PATH yet, so Iceberg may merge their DVs at commit; follow-up HIVE-30227.
How was this patch tested?