Repository navigation
Conversation
0fd68ad to
766dac0
Compare
766dac0 to
7d9e984
Compare
7d9e984 to
6921dcc
Compare
|
| if (transaction != null) { | ||
| transaction.commitTransaction(); | ||
| } | ||
| commitUnknownTypePromotion(); |
There was a problem hiding this comment.
I can see its 2 commits operations, after first one success and second one fails. How to looks, is it tested?
There was a problem hiding this comment.
Good catch — promotion used to be two commits (transaction first, then promotion), so the statement could half-apply if the second failed. It’s now one commit: it stages rename/comment/reorder/defaults in a transaction, applies the UNKNOWN → type change on that staged metadata, and commits once; on conflict it refreshes and re-runs handleChangeColumn. Added tests for conflict retry and for commit failure leaving schema unchanged.
| case STRUCT -> { | ||
| List<Types.NestedField> fields = type.asStructType().fields(); | ||
| Preconditions.checkArgument(fields.stream().anyMatch(field -> !isUnknown(field.type())), | ||
| "Column '%s' of type %s is not supported: a struct must have at least one field that is not of type " + |
There was a problem hiding this comment.
CREATE TABLE t (u UNKNOWN) will this work
There was a problem hiding this comment.
Creating it works, but the table isn't usable on every format. UNKNOWN columns aren't written to data files, so a table whose columns are all UNKNOWN has an empty file schema. Parquet refuses to write it ("Cannot write a schema with an empty group"), and Hive's vectorized ORC reader can't open it. Avro and non-vectorized ORC work. This is the same reason structs whose fields are all UNKNOWN are already rejected.
I'll reject it at DDL time instead: CREATE TABLE, REPLACE COLUMNS and DROP COLUMN will fail with "a table must have at least one column that is not of type unknown". I'll add a negative q-test for it. A lone UNKNOWN column next to other columns stays supported, and positive/unknown_type.q covers that.
| ALTER TABLE unknown_test_middle CREATE TAG before_promotion; | ||
|
|
||
| ALTER TABLE unknown_test_middle CHANGE COLUMN b b INT; | ||
|
|
There was a problem hiding this comment.
after changing the column b INT how the view looks (SELECT * FROM unknown_test_view)
There was a problem hiding this comment.
Good point — I added coverage right after the promotion: SELECT * FROM unknown_test_view and DESCRIBE unknown_test_view. After CHANGE COLUMN b b INT, the view still works and b is int. Rows from before promotion still have b NULL;
| PREHOOK: type: CREATETABLE | ||
| PREHOOK: Output: database:default | ||
| PREHOOK: Output: default@unknown_test_struct | ||
| FAILED: Execution Error, return code 40000 from org.apache.hadoop.hive.ql.ddl.DDLTask. java.lang.IllegalArgumentException: Column 's' of type struct<3: u: optional unknown> is not supported: a struct must have at least one field that is not of type unknown |
There was a problem hiding this comment.
struct<3: u: optional unknown> this is struct<3: u: unknown> correct?
There was a problem hiding this comment.
Yes, it's the same type. The message printed Iceberg's internal type string, where 3 is the field ID and optional means nullable. I changed the struct, array and map error messages to print the Hive type name instead, so this one now reads Column 's' of type struct<u:unknown> is not supported: ..., and the array and map messages print array<unknown> and map<string,unknown>.
|
|
||
| job.set(ColumnProjectionUtils.ORC_SCHEMA_STRING, readOrcSchema.toString()); | ||
| boolean hasUnknownType = TypeUtil.indexById(readSchema.asStruct()).values().stream() | ||
| .anyMatch(field -> field.type().typeId() == Type.TypeID.UNKNOWN); |
There was a problem hiding this comment.
is this simple "TypeUtil.find(readSchema, t -> t.typeId() == UNKNOWN) != null"
There was a problem hiding this comment.
Done, thanks. Changed it to TypeUtil.find(readSchema, type -> type.typeId() == Type.TypeID.UNKNOWN) != null. It still finds UNKNOWN nested in structs and stops at the first match.
# Conflicts: # iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergMetaHook.java
7cefff0 to
0ead40b
Compare



What changes were proposed in this pull request?
Fixes for the Iceberg format-v3
UNKNOWNtype support. Before this change,INSERT INTO t (col list) ...failed on Iceberg tables that haveUNKNOWNcolumns, and several other read, write and DDL paths did not handle the type.UNKNOWNcolumns are always NULL and are never written to data files; this PR makes Hive follow those rules end to end.Serde / type system (
serde)TypeInfoUtils: implicit conversion of a NULL literal toUNKNOWNnow also works inside structs (e.g.named_struct('x', 5, 'u', NULL)intoSTRUCT<x:INT, u:UNKNOWN>), which is what column-list inserts andINSERT ... SELECTneed.ObjectInspectorUtils/SerDeUtils: added the missingUNKNOWNcases (copy, compare, hash, string conversion, constant object inspectors).UnknownConstantObjectInspectorfor NULL constants typed asUNKNOWN. Non-NULL values are rejected with "UNKNOWN type columns only accept NULL values".Query layer (
ql)VectorUDFStructField: extracting anUNKNOWNstruct field in vectorized mode returns a repeating NULL vector.Table: views can exposeUNKNOWNcolumns (e.g.CREATE VIEW AS SELECTover an Iceberg table). Non-Iceberg tables still reject the type.DDLPlanUtils:SHOW CREATE TABLEprintsUNKNOWNcolumns instead of failing with "Unknown type: UNKNOWN".Iceberg handler
BaseHiveIcebergMetaHook,IcebergTableUtil,HiveSchemaUtil). UNKNOWN values have no physical representation in data files, so the following are rejected with clear errors:UNKNOWNas an array element type, or as a map key or value type;UNKNOWN;UNKNOWNcolumn, both inPARTITIONED BYand inALTER TABLE ... SET PARTITION SPEC;DEFAULT NULLis allowed);UNKNOWNon format-version 2 tables (Iceberg's own check).HiveIcebergMetaHook): the v3 spec allowsALTER TABLE ... CHANGE COLUMNto promoteUNKNOWNto any primitive type, but Iceberg 1.x does not implement it yet (apache/iceberg#18305). Until it does, Hive commits the new schema directly throughTableOperations, retrying on commit conflicts. The field ID is kept, so existing files read the column as NULL and the metadata matches what a native promotion would produce. Rename, comment and reorder in the same statement still go throughUpdateSchema. This workaround can be replaced byUpdateSchema.updateColumnonce Iceberg supports the promotion.VectorizedReadUtils): Iceberg's ORC projection dropsUNKNOWNfields, but Hive's vectorized reader matches columns by position, so later columns were misaligned (AIOOBE / ClassCastException). The read schema now keeps a placeholder column for eachUNKNOWNfield, which ORC returns as NULL.Why are the changes needed?
This fixes INSERT with column list that was failing on Iceberg tables with UNKNOWN columns and additional gaps that were discovered with the UNKNOWN type implementation.
Does this PR introduce any user-facing change?
Yes: new error messages, promotion from UNKNOWN, and SHOW CREATE TABLE working on these tables.
How was this patch tested?
New unit and integration tests:
Tests
TestHiveIcebergUnknownType: read/write, DDL, column-list insert with UPDATE and DELETE, promotion (field ID kept, time travel to before promotion), rejection cases,SHOW CREATE TABLE,REPLACE COLUMNS. Runs on Parquet, ORC and Avro, vectorized and non-vectorized.positive/unknown_type.qcovers DML, MERGE, CTAS/CTLT, views, joins, aggregates, window functions, ANALYZE, promotion, tags, ORC, Avro andSHOW CREATE TABLE. Elevennegative/unknown_type_*.qfiles cover each rejection.VARIANTis out of scope for this PR.