Repository navigation
Conversation
deniskuzZ
marked this pull request as ready for review
November 6, 2026 18:16
deniskuzZ
force-pushed
the
HIVE-30113
branch
2 times, most recently
from
November 19, 2026 18:49
205aec8 to
9e1b25d
Compare
… file - HiveStorageHandler#acidDistributeColumns: distribution of the deleted records in the rewritten query - V3: DISTRIBUTE BY FILE__PATH, no SORT BY, so all deletes of a data file reach one DV writer - MERGE with UPDATE and DELETE clauses on V3: one delete branch; the update's insert branch is not counted, rows affected unchanged - HiveStorageHandler#configureJobConf(FileSinkDesc, JobConf): called for every file sink while the job is built - Iceberg plans the previous deletes of a V3 merge-on-read delete sink there, once per job, into a gzipped file, instead of a full-table planFiles per task - the file path is unique per job build and recorded in iceberg.mr.rewritable.deletes.<table> - writers load it once per daemon through the query ObjectCache; a DV writer without it fails - commit and abort delete the recorded file through the table's FileIO - iceberg_v3_dml_distribution.q: plans of V3 DELETE, UPDATE, MERGE - iceberg_row_lineage_compactions.q.out: operator id in a cross-product warning - Impact (cluster, 1TRC V3 DML): 2,299 duplicate DVs and an HS2 OOM (heap 0.37 -> 11.7 GB in 40 s) before; 17,016 DVs for 17,016 files, HS2 peak 5.15 GB after
- the location of the planned deletes comes from the job build only: a value of the key that is not a location of this query under the table's temp dir is dropped while the job is built, and the committer removes no other - only DV writers read the planned deletes - test: a session value of the key is neither read nor removed, for V3 and V2 DML
- RewritableDeletes: plans, loads and cleans up the previous deletes of the DV writers - Temp file named after the query, the cache key adds the snapshot; no location key in the job conf - The delete writer gets the supplier; only the DV writer reads it - Materialized view rebuild distribution moves to HIVE-30227 - SplitMergeRewriter: updateInDeleteBranch, clausePredicate
- Rewritable deletes in the temp dir of the query, named in the job conf; no file without previous deletes - A merge-on-read delete sink gets no dynamic-partition context
- RewritableDeletes.addToJobConf keeps the key only when it holds the location this query plans: a value from the session is dropped, and the deletes are planned - testRewritableDeletesKeyOfTheSessionIsIgnored - iceberg_v3_dml_distribution.q: blank line between the explained MERGE and the MERGE
…e and delete clauses - SplitMergeRewriter: the update and delete clauses never match the same row, so every target (native ACID, Iceberg V2 and V3) gets one delete branch, ((update) OR (delete)), and writes one delete delta or set of delete files - Plans: one file sink and reducer fewer; result rows and rows affected unchanged
Collect the rewritable deletes from the files the DML's scans of the target table read, instead of planning the whole snapshot while the job is built. - The job names a temp dir per target table, under a key of the query id. - Every scan of the table collects the position deletes of the data files it plans during split generation, and writes them into a file of its own in that dir, also when empty. - A DV writer reads and merges all files of the dir on first use, when it writes its DVs, once per LLAP daemon, and fails if no scan wrote a file. The map the writers share is read-only. - Tests: partition- and stats-filtered DMLs collect only the files they scan; a MERGE whose subquery reads the target; a dir named in the session is ignored.
- A scan collects into the subdir of the snapshot it reads, a DV writer reads the one of the snapshot it writes - Tests: DELETE from a branch with a subquery over main, MERGE into main from a branch, MERGE as of an older snapshot - Tests read the scan files through RewritableDeletes
- A DV writer decodes a position delete file once per query (per LLAP daemon or Tez container), not once per data file - WriterBuilder takes the job configuration - The Tez object cache logs a hit at DEBUG, without its value: a cached position delete file logged all its entries at INFO - RewritableDeletes logs the read of the deletes at DEBUG - Test: a V2 position delete file of two data files, merged into their DVs after an upgrade to V3
deniskuzZ
force-pushed
the
HIVE-30113
branch
from
November 20, 2026 06:34
9e1b25d to
0299a96
Compare
- RewritableDeletes lists the dir of a snapshot with Utilities.listStatusIfExists, instead of a nested try - RewritableDeletes.location without a nested ternary - scanFiles/isScanFile renamed to collectorFiles/isCollectorFile: the files a Collector wrote - SplitMergeRewriter: the ternary of the update in the delete branch without a negated condition - TestRewritableDeletes: the lambdas of assertThrows make a single call
…e DV writer - Merge-on-read deleted records carry only PARTITION__SPEC__ID, PARTITION__KEY, FILE__PATH, ROW__POSITION; readers fill PARTITION__KEY once per file with a compact encoding of the partition - Ordinary deletes and the merge task share one record layout; the delete writer decodes each distinct partition once - Statistics size PARTITION__KEY by a fixed 8-byte average length on its VirtualColumn, the width PARTITION__HASH had - deletes from files of a spec whose partition source was dropped fail with a clear message (they failed with "Cannot build accessor"): a delete file holds its data files' partition, and a manifest stores the dropped source's field as null (Iceberg 1.12 rejects such a DV when reading) - Release note: the PARTITION__HASH virtual column is removed; a MERGE checks cardinality on FILE__PATH, ROW__POSITION - Impact (local): partition encoding per deleted record 1.5-3.4 KB -> 0-17 B; shuffle width from all table columns to 4 virtual columns; not timed on the cluster
- statistics: a TableScan no longer gets PARTITION__KEY column stats after its data size is computed; a projected PARTITION__KEY takes the 8-byte hint where the virtual columns' stats are built, so filters no longer estimate more than their scans and the copy-on-write MERGE estimates are back to the parent's - merge_iceberg_orc, merge_iceberg_partitioned_orc regenerated: they still showed PARTITION__HASH - PARTITION__KEY in the predicate pushdown skip list, as PARTITION__HASH was - comments; redundant asserts in TestIcebergAcidUtil
- parseSpecId reads the deleted record by its own layout (SERDE_META_COLS), like parseSerializedPartition - MergeTaskVirtualColumnAwareIterator creates its record with createSerdeSchemaForDelete(): with the merge-task layout gone, its schema parameter had a single valid value - javadoc: a merge-on-read DELETE selects acidVirtualColumns, not acidSelectColumns; the average length of a virtual column is an estimate; the two-record getPositionDelete and createSerdeSchemaForDelete(List) are copy-on-write
- NonNativeAcidMultiInsertSqlGenerator: a select expression helper lowers the cognitive complexity of appendAcidSelectColumns - VirtualColumn: import order - TestHiveIcebergDeleteWriter: no record variables in testDeleteWithNestedPartitionSource
deniskuzZ
force-pushed
the
HIVE-30113
branch
from
November 20, 2026 15:02
0299a96 to
88b8ce8
Compare
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



What changes were proposed in this pull request?
Why are the changes needed?
Optimization: a deleted record carried every column of the table (with a partition hash), and the merge task a Base64, Java-serialized PartitionKey of 1.5-3.4 KB per record. Now it carries 4 virtual columns; the partition encoding was 0-17 B in the tested specs and grows with the length of identity string values.
Does this PR introduce any user-facing change?
Yes:
How was this patch tested?