Skip to content

HIVE-30219: INSERT with column list fails on Iceberg tables with UNKNOWN columns. - #6927

Open
difin wants to merge 2 commits into
apache:masterfrom
difin:unknown_type_bug
Open

difin wants to merge 2 commits into
apache:masterfrom
difin:unknown_type_bug

Conversation

@difin

@difin difin commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Fixes for the Iceberg format-v3 UNKNOWN type support. Before this change, INSERT INTO t (col list) ... failed on Iceberg tables that have UNKNOWN columns, and several other read, write and DDL paths did not handle the type. UNKNOWN columns are always NULL and are never written to data files; this PR makes Hive follow those rules end to end.

Serde / type system (serde)

  • TypeInfoUtils: implicit conversion of a NULL literal to UNKNOWN now also works inside structs (e.g. named_struct('x', 5, 'u', NULL) into STRUCT<x:INT, u:UNKNOWN>), which is what column-list inserts and INSERT ... SELECT need.
  • ObjectInspectorUtils / SerDeUtils: added the missing UNKNOWN cases (copy, compare, hash, string conversion, constant object inspectors).
  • New UnknownConstantObjectInspector for NULL constants typed as UNKNOWN. Non-NULL values are rejected with "UNKNOWN type columns only accept NULL values".

Query layer (ql)

  • VectorUDFStructField: extracting an UNKNOWN struct field in vectorized mode returns a repeating NULL vector.
  • Table: views can expose UNKNOWN columns (e.g. CREATE VIEW AS SELECT over an Iceberg table). Non-Iceberg tables still reject the type.
  • DDLPlanUtils: SHOW CREATE TABLE prints UNKNOWN columns instead of failing with "Unknown type: UNKNOWN".

Iceberg handler

  • DDL validation (BaseHiveIcebergMetaHook, IcebergTableUtil, HiveSchemaUtil). UNKNOWN values have no physical representation in data files, so the following are rejected with clear errors:
    • UNKNOWN as an array element type, or as a map key or value type;
    • structs whose fields are all UNKNOWN;
    • partitioning by an UNKNOWN column, both in PARTITIONED BY and in ALTER TABLE ... SET PARTITION SPEC;
    • non-NULL defaults (DEFAULT NULL is allowed);
    • UNKNOWN on format-version 2 tables (Iceberg's own check).
  • Type promotion (HiveIcebergMetaHook): the v3 spec allows ALTER TABLE ... CHANGE COLUMN to promote UNKNOWN to any primitive type, but Iceberg 1.x does not implement it yet (apache/iceberg#18305). Until it does, Hive commits the new schema directly through TableOperations, retrying on commit conflicts. The field ID is kept, so existing files read the column as NULL and the metadata matches what a native promotion would produce. Rename, comment and reorder in the same statement still go through UpdateSchema. This workaround can be replaced by UpdateSchema.updateColumn once Iceberg supports the promotion.
  • Vectorized ORC reads (VectorizedReadUtils): Iceberg's ORC projection drops UNKNOWN fields, but Hive's vectorized reader matches columns by position, so later columns were misaligned (AIOOBE / ClassCastException). The read schema now keeps a placeholder column for each UNKNOWN field, which ORC returns as NULL.

Why are the changes needed?

This fixes INSERT with column list that was failing on Iceberg tables with UNKNOWN columns and additional gaps that were discovered with the UNKNOWN type implementation.

Does this PR introduce any user-facing change?

Yes: new error messages, promotion from UNKNOWN, and SHOW CREATE TABLE working on these tables.

How was this patch tested?

New unit and integration tests:

Tests

  • TestHiveIcebergUnknownType: read/write, DDL, column-list insert with UPDATE and DELETE, promotion (field ID kept, time travel to before promotion), rejection cases, SHOW CREATE TABLE, REPLACE COLUMNS. Runs on Parquet, ORC and Avro, vectorized and non-vectorized.
  • q-tests: positive/unknown_type.q covers DML, MERGE, CTAS/CTLT, views, joins, aggregates, window functions, ANALYZE, promotion, tags, ORC, Avro and SHOW CREATE TABLE. Eleven negative/unknown_type_*.q files cover each rejection.
    VARIANT is out of scope for this PR.

@sonarqubecloud

Copy link
Copy Markdown

if (transaction != null) {
transaction.commitTransaction();
}
commitUnknownTypePromotion();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can see its 2 commits operations, after first one success and second one fails. How to looks, is it tested?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch — promotion used to be two commits (transaction first, then promotion), so the statement could half-apply if the second failed. It’s now one commit: it stages rename/comment/reorder/defaults in a transaction, applies the UNKNOWN → type change on that staged metadata, and commits once; on conflict it refreshes and re-runs handleChangeColumn. Added tests for conflict retry and for commit failure leaving schema unchanged.

case STRUCT -> {
List<Types.NestedField> fields = type.asStructType().fields();
Preconditions.checkArgument(fields.stream().anyMatch(field -> !isUnknown(field.type())),
"Column '%s' of type %s is not supported: a struct must have at least one field that is not of type " +

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CREATE TABLE t (u UNKNOWN) will this work

@difin difin Nov 19, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Creating it works, but the table isn't usable on every format. UNKNOWN columns aren't written to data files, so a table whose columns are all UNKNOWN has an empty file schema. Parquet refuses to write it ("Cannot write a schema with an empty group"), and Hive's vectorized ORC reader can't open it. Avro and non-vectorized ORC work. This is the same reason structs whose fields are all UNKNOWN are already rejected.

I'll reject it at DDL time instead: CREATE TABLE, REPLACE COLUMNS and DROP COLUMN will fail with "a table must have at least one column that is not of type unknown". I'll add a negative q-test for it. A lone UNKNOWN column next to other columns stays supported, and positive/unknown_type.q covers that.

ALTER TABLE unknown_test_middle CREATE TAG before_promotion;

ALTER TABLE unknown_test_middle CHANGE COLUMN b b INT;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

after changing the column b INT how the view looks (SELECT * FROM unknown_test_view)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point — I added coverage right after the promotion: SELECT * FROM unknown_test_view and DESCRIBE unknown_test_view. After CHANGE COLUMN b b INT, the view still works and b is int. Rows from before promotion still have b NULL;

PREHOOK: type: CREATETABLE
PREHOOK: Output: database:default
PREHOOK: Output: default@unknown_test_struct
FAILED: Execution Error, return code 40000 from org.apache.hadoop.hive.ql.ddl.DDLTask. java.lang.IllegalArgumentException: Column 's' of type struct<3: u: optional unknown> is not supported: a struct must have at least one field that is not of type unknown

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

struct<3: u: optional unknown> this is struct<3: u: unknown> correct?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it's the same type. The message printed Iceberg's internal type string, where 3 is the field ID and optional means nullable. I changed the struct, array and map error messages to print the Hive type name instead, so this one now reads Column 's' of type struct<u:unknown> is not supported: ..., and the array and map messages print array<unknown> and map<string,unknown>.


job.set(ColumnProjectionUtils.ORC_SCHEMA_STRING, readOrcSchema.toString());
boolean hasUnknownType = TypeUtil.indexById(readSchema.asStruct()).values().stream()
.anyMatch(field -> field.type().typeId() == Type.TypeID.UNKNOWN);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this simple "TypeUtil.find(readSchema, t -> t.typeId() == UNKNOWN) != null"

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done, thanks. Changed it to TypeUtil.find(readSchema, type -> type.typeId() == Type.TypeID.UNKNOWN) != null. It still finds UNKNOWN nested in structs and stops at the first match.

Dmitriy Fingerman added 2 commits November 20, 2026 14:11
# Conflicts:
#	iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergMetaHook.java

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants