Skip to content

HIVE-30112: Iceberg: V3 merge-on-read DML writes several DVs per data file - #6924

Open
deniskuzZ wants to merge 10 commits into
apache:masterfrom
deniskuzZ:HIVE-30112
Open

deniskuzZ wants to merge 10 commits into
apache:masterfrom
deniskuzZ:HIVE-30112

Conversation

@deniskuzZ

@deniskuzZ deniskuzZ commented Oct 28, 2026 •

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

  • HiveStorageHandler#acidDistributeColumns: how the deleted records of the rewritten query are distributed
  • V3: DISTRIBUTE BY FILE__PATH, no SORT BY, so all deletes of a data file reach one DV writer
  • A merge-on-read delete sink gets no dynamic-partition context: the rewritten statement distributes and sorts its records (a V3 UPDATE of a sorted table with write.delete.mode=copy-on-write failed to compile)
  • MERGE with UPDATE and DELETE clauses on V3: one delete branch (SplitMergeRewriter ORs the clauses' conditions); the update's insert branch is not counted, rows affected unchanged
  • WhereClausePatcher puts the WHERE right after the SELECT, before the optional DISTRIBUTE BY and SORT BY
  • HiveStorageHandler#configureJobConf(FileSinkDesc, JobConf): called for every file sink while the job is built, with a temp dir of the query
  • Iceberg collects the previous deletes of a V3 merge-on-read delete sink during split generation (RewritableDeletes): every scan of the target table writes the deletes of the data files it plans into the subdir of the snapshot it reads, under a temp dir of the query that the job conf names
  • DV writers load it once per LLAP daemon (per container in Tez container mode) through the query ObjectCache; Hive removes it with the query's scratch dir

Why are the changes needed?

Bug-Fix: 1TRC V3 DML wrote 2,299 duplicate DVs and ran HS2 out of memory at commit (heap 0.37 -> 11.7 GB in 40 s). After: 17,016 DVs for 17,016 files, HS2 peak 5.15 GB

Does this PR introduce any user-facing change?

Plans of V3 merge-on-read DELETE, UPDATE and MERGE: the deleted records are shuffled by FILE__PATH to the DV writers (a reduce stage where a DELETE was map-only), and a MERGE with UPDATE and DELETE clauses has one delete branch. Plans of a merge-on-read UPDATE or MERGE on a table with write.delete.mode=copy-on-write (V2 too): the deleted records are no longer sorted by the partition transforms and write order. No change to results.

Known limitation: the delete branches of a materialized view rebuild in insert-delete mode (a V3 Iceberg MV) are not distributed by FILE__PATH yet, so Iceberg may merge their DVs at commit; follow-up HIVE-30227.

How was this patch tested?

  • TestHiveIcebergDeletionVectors (split data files, partitioned tables, UPDATE, MERGE clause orders, DML into a branch, deletes collected by the scans, partition- and stats-filtered DMLs, DML across main, a branch and an older snapshot), TestRewritableDeletes
  • iceberg_v3_dml_distribution.q (plans of V3 DELETE, UPDATE, MERGE, and of an UPDATE of a sorted table that copies on DELETE)

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The hook fails before job submission for MapReduce V3 merge-on-read DELETE/UPDATE operations with existing position deletes, and LLAP’s soft-reference wrapper caching can cause duplicate delete-map deserialization under memory pressure.

2 open issues
What changed in this PR

This PR reduces duplicate deletion vectors and commit-time memory usage for Iceberg V3 merge-on-read DML by routing each data file’s deletes to one writer and sharing previously planned delete metadata.

Changes:

  • Distributes V3 deletes by file path and combines MERGE update/delete branches while preserving affected-row counts.
  • Plans previous deletes during job setup and loads their compressed metadata through the query cache.
  • Adds regression coverage for DML distribution, existing deletion vectors, branches, and sorted tables.
File Description
ql/​src/​java/​org/​apache/​hadoop/​hive/​ql/​plan/​PlanUtils.java Adds file-sink-aware storage-handler configuration.
ql/​src/​java/​org/​apache/​hadoop/​hive/​ql/​plan/​BaseWork.java Passes sink descriptors during job configuration.
ql/​src/​java/​org/​apache/​hadoop/​hive/​ql/​parse/​SemanticAnalyzer.java Skips dynamic-partition context for merge-on-read delete sinks.
ql/​src/​java/​org/​apache/​hadoop/​hive/​ql/​parse/​rewrite/​sql/​WhereClausePatcher.java Places WHERE before distribution and sorting clauses.
ql/​src/​java/​org/​apache/​hadoop/​hive/​ql/​parse/​rewrite/​sql/​NonNativeAcidMultiInsertSqlGenerator.java Obtains distribution keys from storage handlers.
ql/​src/​java/​org/​apache/​hadoop/​hive/​ql/​parse/​rewrite/​sql/​MultiInsertSqlGenerator.java Emits DISTRIBUTE BY before SORT BY.
ql/​src/​java/​org/​apache/​hadoop/​hive/​ql/​parse/​rewrite/​SplitMergeRewriter.java Combines delete branches and adjusts row counting.
ql/​src/​java/​org/​apache/​hadoop/​hive/​ql/​metadata/​HiveStorageHandler.java Adds distribution and file-sink configuration hooks.
ql/​src/​java/​org/​apache/​hadoop/​hive/​ql/​exec/​tez/​TezTask.java Supplies query scratch space before handler configuration.
ql/​src/​java/​org/​apache/​hadoop/​hive/​ql/​exec/​FileSinkOperator.java Updates logging terminology for uncounted records.
iceberg/​iceberg-handler/​src/​test/​results/​positive/​llap/​iceberg_row_lineage_compactions.q.out Updates expected operator identifiers.
iceberg/​iceberg-handler/​src/​test/​results/​positive/​iceberg_v3_dml_distribution.q.out Records expected DML plans and results.
iceberg/​iceberg-handler/​src/​test/​queries/​positive/​iceberg_v3_dml_distribution.q Adds distribution and sorted-table regression queries.
iceberg/​iceberg-handler/​src/​test/​java/​org/​apache/​iceberg/​mr/​hive/​writer/​TestRewritableDeletes.java Tests metadata staging and daemon-cache reuse.
iceberg/​iceberg-handler/​src/​test/​java/​org/​apache/​iceberg/​mr/​hive/​TestHiveIcebergDeletionVectors.java Expands deletion-vector and DML integration coverage.
iceberg/​iceberg-handler/​src/​main/​java/​org/​apache/​iceberg/​mr/​hive/​writer/​WriterBuilder.java Accepts a supplier for previous-delete metadata.
iceberg/​iceberg-handler/​src/​main/​java/​org/​apache/​iceberg/​mr/​hive/​writer/​RewritableDeletes.java Plans, stages, and caches previous deletes.
iceberg/​iceberg-handler/​src/​main/​java/​org/​apache/​iceberg/​mr/​hive/​writer/​HiveIcebergWriterBase.java Loads previous deletes for deletion-vector writers.
iceberg/​iceberg-handler/​src/​main/​java/​org/​apache/​iceberg/​mr/​hive/​writer/​HiveIcebergDeleteWriter.java Passes the metadata supplier into writer creation.
iceberg/​iceberg-handler/​src/​main/​java/​org/​apache/​iceberg/​mr/​hive/​HiveIcebergStorageHandler.java Configures V3 distribution and job-level delete planning.
iceberg/​iceberg-handler/​src/​main/​java/​org/​apache/​iceberg/​mr/​hive/​HiveIcebergOutputFormat.java Connects writers to staged delete metadata.

Review effort: Balanced

… file

- HiveStorageHandler#acidDistributeColumns: distribution of the deleted records in the rewritten query
- V3: DISTRIBUTE BY FILE__PATH, no SORT BY, so all deletes of a data file reach one DV writer
- MERGE with UPDATE and DELETE clauses on V3: one delete branch; the update's insert branch is not counted, rows affected unchanged
- HiveStorageHandler#configureJobConf(FileSinkDesc, JobConf): called for every file sink while the job is built
- Iceberg plans the previous deletes of a V3 merge-on-read delete sink there, once per job, into a gzipped file,
  instead of a full-table planFiles per task
- the file path is unique per job build and recorded in iceberg.mr.rewritable.deletes.<table>
- writers load it once per daemon through the query ObjectCache; a DV writer without it fails
- commit and abort delete the recorded file through the table's FileIO
- iceberg_v3_dml_distribution.q: plans of V3 DELETE, UPDATE, MERGE
- iceberg_row_lineage_compactions.q.out: operator id in a cross-product warning
- Impact (cluster, 1TRC V3 DML): 2,299 duplicate DVs and an HS2 OOM (heap 0.37 -> 11.7 GB in 40 s) before;
  17,016 DVs for 17,016 files, HS2 peak 5.15 GB after
- the location of the planned deletes comes from the job build only: a value of the key that is not a location
  of this query under the table's temp dir is dropped while the job is built, and the committer removes no other
- only DV writers read the planned deletes
- test: a session value of the key is neither read nor removed, for V3 and V2 DML
- RewritableDeletes: plans, loads and cleans up the previous deletes of the DV writers
- Temp file named after the query, the cache key adds the snapshot; no location key in the job conf
- The delete writer gets the supplier; only the DV writer reads it
- Materialized view rebuild distribution moves to HIVE-30227
- SplitMergeRewriter: updateInDeleteBranch, clausePredicate
- Rewritable deletes in the temp dir of the query, named in the job conf; no file without previous deletes  - A merge-on-read delete sink gets no dynamic-partition context
- RewritableDeletes.addToJobConf keeps the key only when it holds the location this query plans: a value from the
  session is dropped, and the deletes are planned
- testRewritableDeletesKeyOfTheSessionIsIgnored
- iceberg_v3_dml_distribution.q: blank line between the explained MERGE and the MERGE
…e and delete clauses

- SplitMergeRewriter: the update and delete clauses never match the same row, so every target (native ACID, Iceberg V2
  and V3) gets one delete branch, ((update) OR (delete)), and writes one delete delta or set of delete files
- Plans: one file sink and reducer fewer; result rows and rows affected unchanged
Collect the rewritable deletes from the files the DML's scans of the target table read, instead of planning the
whole snapshot while the job is built.

- The job names a temp dir per target table, under a key of the query id.
- Every scan of the table collects the position deletes of the data files it plans during split generation, and
  writes them into a file of its own in that dir, also when empty.
- A DV writer reads and merges all files of the dir on first use, when it writes its DVs, once per LLAP daemon, and
  fails if no scan wrote a file. The map the writers share is read-only.
- Tests: partition- and stats-filtered DMLs collect only the files they scan; a MERGE whose subquery reads the
  target; a dir named in the session is ignored.
- A scan collects into the subdir of the snapshot it reads, a DV writer reads the one of the snapshot it writes
- Tests: DELETE from a branch with a subquery over main, MERGE into main from a branch, MERGE as of an older snapshot
- Tests read the scan files through RewritableDeletes
- A DV writer decodes a position delete file once per query (per LLAP daemon or Tez container), not once per data file
- WriterBuilder takes the job configuration
- The Tez object cache logs a hit at DEBUG, without its value: a cached position delete file logged all its entries at INFO
- RewritableDeletes logs the read of the deletes at DEBUG
- Test: a V2 position delete file of two data files, merged into their DVs after an upgrade to V3
- RewritableDeletes lists the dir of a snapshot with Utilities.listStatusIfExists, instead of a nested try
- RewritableDeletes.location without a nested ternary
- scanFiles/isScanFile renamed to collectorFiles/isCollectorFile: the files a Collector wrote
- SplitMergeRewriter: the ternary of the update in the delete branch without a negated condition
- TestRewritableDeletes: the lambdas of assertThrows make a single call
@sonarqubecloud

Copy link
Copy Markdown

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants