Skip to content

HIVE-30113: Iceberg: V3 merge-on-read deletes ship every column to the DV writer - #6925

Open
deniskuzZ wants to merge 14 commits into
apache:masterfrom
deniskuzZ:HIVE-30113
Open

deniskuzZ wants to merge 14 commits into
apache:masterfrom
deniskuzZ:HIVE-30113

Conversation

@deniskuzZ

@deniskuzZ deniskuzZ commented Oct 28, 2026 •

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

  • Merge-on-read DELETE, UPDATE and MERGE ship only PARTITION__SPEC__ID, PARTITION__KEY, FILE__PATH and ROW__POSITION to the delete writer, for V3 deletion vectors and V2 position deletes alike, and the V2 merge task uses the same record layout
  • The new PARTITION__KEY virtual column (binary) holds a compact encoding of the data file's partition; readers fill it once per file, and the delete writer decodes each distinct partition once
  • Statistics: a projected PARTITION__KEY is sized by an 8-byte average length (VirtualColumn), not by hive.stats.max.variable.length
  • The file.sink.merge.task.enabled.* table property is no longer set, which changes EXPLAIN EXTENDED goldens of non-Iceberg tables
  • Deletes from files of a spec whose partition source column was dropped fail with a clear message instead of "Cannot build accessor"
  • Stacked on HIVE-30112

Why are the changes needed?

Optimization: a deleted record carried every column of the table (with a partition hash), and the merge task a Base64, Java-serialized PartitionKey of 1.5-3.4 KB per record. Now it carries 4 virtual columns; the partition encoding was 0-17 B in the tested specs and grows with the length of identity string values.

Does this PR introduce any user-facing change?

Yes:

  • the PARTITION__HASH virtual column is removed and PARTITION__KEY is added
  • a MERGE checks cardinality on FILE__PATH, ROW__POSITION
  • plans of merge-on-read DML shuffle fewer columns

How was this patch tested?

  • TestIcebergAcidUtil, TestHiveIcebergDeleteWriter, TestHiveIcebergDeletionVectors, TestHiveIcebergCRUD (including deletes from files of a spec with a dropped partition source), TestStatsUtils
  • iceberg_dml_deleted_record_columns.q, iceberg_merge_delete_files.q, query_iceberg_virtualcol.q
  • regenerated plan goldens; result rows are unchanged

… file

- HiveStorageHandler#acidDistributeColumns: distribution of the deleted records in the rewritten query
- V3: DISTRIBUTE BY FILE__PATH, no SORT BY, so all deletes of a data file reach one DV writer
- MERGE with UPDATE and DELETE clauses on V3: one delete branch; the update's insert branch is not counted, rows affected unchanged
- HiveStorageHandler#configureJobConf(FileSinkDesc, JobConf): called for every file sink while the job is built
- Iceberg plans the previous deletes of a V3 merge-on-read delete sink there, once per job, into a gzipped file,
  instead of a full-table planFiles per task
- the file path is unique per job build and recorded in iceberg.mr.rewritable.deletes.<table>
- writers load it once per daemon through the query ObjectCache; a DV writer without it fails
- commit and abort delete the recorded file through the table's FileIO
- iceberg_v3_dml_distribution.q: plans of V3 DELETE, UPDATE, MERGE
- iceberg_row_lineage_compactions.q.out: operator id in a cross-product warning
- Impact (cluster, 1TRC V3 DML): 2,299 duplicate DVs and an HS2 OOM (heap 0.37 -> 11.7 GB in 40 s) before;
  17,016 DVs for 17,016 files, HS2 peak 5.15 GB after
- the location of the planned deletes comes from the job build only: a value of the key that is not a location
  of this query under the table's temp dir is dropped while the job is built, and the committer removes no other
- only DV writers read the planned deletes
- test: a session value of the key is neither read nor removed, for V3 and V2 DML
- RewritableDeletes: plans, loads and cleans up the previous deletes of the DV writers
- Temp file named after the query, the cache key adds the snapshot; no location key in the job conf
- The delete writer gets the supplier; only the DV writer reads it
- Materialized view rebuild distribution moves to HIVE-30227
- SplitMergeRewriter: updateInDeleteBranch, clausePredicate
- Rewritable deletes in the temp dir of the query, named in the job conf; no file without previous deletes  - A merge-on-read delete sink gets no dynamic-partition context
- RewritableDeletes.addToJobConf keeps the key only when it holds the location this query plans: a value from the
  session is dropped, and the deletes are planned
- testRewritableDeletesKeyOfTheSessionIsIgnored
- iceberg_v3_dml_distribution.q: blank line between the explained MERGE and the MERGE
…e and delete clauses

- SplitMergeRewriter: the update and delete clauses never match the same row, so every target (native ACID, Iceberg V2
  and V3) gets one delete branch, ((update) OR (delete)), and writes one delete delta or set of delete files
- Plans: one file sink and reducer fewer; result rows and rows affected unchanged
Collect the rewritable deletes from the files the DML's scans of the target table read, instead of planning the
whole snapshot while the job is built.

- The job names a temp dir per target table, under a key of the query id.
- Every scan of the table collects the position deletes of the data files it plans during split generation, and
  writes them into a file of its own in that dir, also when empty.
- A DV writer reads and merges all files of the dir on first use, when it writes its DVs, once per LLAP daemon, and
  fails if no scan wrote a file. The map the writers share is read-only.
- Tests: partition- and stats-filtered DMLs collect only the files they scan; a MERGE whose subquery reads the
  target; a dir named in the session is ignored.
- A scan collects into the subdir of the snapshot it reads, a DV writer reads the one of the snapshot it writes
- Tests: DELETE from a branch with a subquery over main, MERGE into main from a branch, MERGE as of an older snapshot
- Tests read the scan files through RewritableDeletes
- A DV writer decodes a position delete file once per query (per LLAP daemon or Tez container), not once per data file
- WriterBuilder takes the job configuration
- The Tez object cache logs a hit at DEBUG, without its value: a cached position delete file logged all its entries at INFO
- RewritableDeletes logs the read of the deletes at DEBUG
- Test: a V2 position delete file of two data files, merged into their DVs after an upgrade to V3
- RewritableDeletes lists the dir of a snapshot with Utilities.listStatusIfExists, instead of a nested try
- RewritableDeletes.location without a nested ternary
- scanFiles/isScanFile renamed to collectorFiles/isCollectorFile: the files a Collector wrote
- SplitMergeRewriter: the ternary of the update in the delete branch without a negated condition
- TestRewritableDeletes: the lambdas of assertThrows make a single call
…e DV writer

- Merge-on-read deleted records carry only PARTITION__SPEC__ID, PARTITION__KEY, FILE__PATH, ROW__POSITION; readers fill PARTITION__KEY once per file with a compact encoding of the partition
- Ordinary deletes and the merge task share one record layout; the delete writer decodes each distinct partition once
- Statistics size PARTITION__KEY by a fixed 8-byte average length on its VirtualColumn, the width PARTITION__HASH had
- deletes from files of a spec whose partition source was dropped fail with a clear message (they failed with
  "Cannot build accessor"): a delete file holds its data files' partition, and a manifest stores the dropped
  source's field as null (Iceberg 1.12 rejects such a DV when reading)
- Release note: the PARTITION__HASH virtual column is removed; a MERGE checks cardinality on FILE__PATH, ROW__POSITION
- Impact (local): partition encoding per deleted record 1.5-3.4 KB -> 0-17 B; shuffle width from all table columns to
  4 virtual columns; not timed on the cluster
- statistics: a TableScan no longer gets PARTITION__KEY column stats after its data size is computed; a projected
  PARTITION__KEY takes the 8-byte hint where the virtual columns' stats are built, so filters no longer estimate more
  than their scans and the copy-on-write MERGE estimates are back to the parent's
- merge_iceberg_orc, merge_iceberg_partitioned_orc regenerated: they still showed PARTITION__HASH
- PARTITION__KEY in the predicate pushdown skip list, as PARTITION__HASH was
- comments; redundant asserts in TestIcebergAcidUtil
- parseSpecId reads the deleted record by its own layout (SERDE_META_COLS), like parseSerializedPartition
- MergeTaskVirtualColumnAwareIterator creates its record with createSerdeSchemaForDelete(): with the merge-task layout
  gone, its schema parameter had a single valid value
- javadoc: a merge-on-read DELETE selects acidVirtualColumns, not acidSelectColumns; the average length of a virtual
  column is an estimate; the two-record getPositionDelete and createSerdeSchemaForDelete(List) are copy-on-write
- NonNativeAcidMultiInsertSqlGenerator: a select expression helper lowers the cognitive complexity of
  appendAcidSelectColumns
- VirtualColumn: import order
- TestHiveIcebergDeleteWriter: no record variables in testDeleteWithNestedPartitionSource
@sonarqubecloud

Copy link
Copy Markdown

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants