Skip to content

fix: preserve CASE NULL types and metadata - #177

Merged
osipovartem merged 1 commit into
df55-physical-udf-instancefrom
fix/case-false-null-metadata-20261104
Nov 4, 2026
Merged

osipovartem merged 1 commit into
df55-physical-udf-instancefrom
fix/case-false-null-metadata-20261104

Conversation

@osipovartem

Copy link
Copy Markdown
Collaborator

Which issue does this PR close?

Related to apache#24860 and the existing apache#26951 by James Willis. This adapts that fix to the Rustice DF55 runtime branch; it is not a duplicate upstream submission.

Rationale for this change

An all-false CASE without ELSE becomes a NULL literal with no field metadata. This loses extension/type annotations. It can also choose an untyped NULL from the first branch instead of the type of the complete CASE.

Copying the complete CASE field fixes correctness, but our first local screen found a 24.52-31.54% planning-cost increase for unannotated column cases. That version was rejected, not merged or hidden.

What changes are included in this PR?

  • Add Case::result_type_and_metadata, sharing the existing result-type and metadata policy without computing an output name or outer-CASE nullability.
  • Keep the existing native to_field type -> nullability -> metadata order and predicate-aware nullability logic. The metadata rules are shared rather than duplicated in the optimizer.
  • Use the component inference API only for all-false, no-ELSE searched CASE folds. Preserve the correct NULL type and metadata, and avoid empty literal-metadata allocation.
  • Cover metadata conflicts and marked/unmarked contributors in both orders, initial untyped NULL branches, primitive/decimal/timestamp/struct output types, and the native API contract.
  • Add a reusable planning-only example with scalar, column, wide-prefix and direct native-field controls. No new dependency or execution operator.

Are these changes tested?

  • Executed both new regression tests against the original production logic: both fail for the expected type/metadata differences.
  • Owning library suites: 270 expression tests and 770 optimizer tests pass, zero failed or ignored. The counted-UDF test verifies one fewer field-inference callback while preserving native result annotations and nullability.
  • Both metadata.slt and cast_extension_type_metadata.slt pass on the final implementation.
  • cargo +1.97.0 fmt --all --check passes.
  • Full cargo +1.97.0 clippy --locked --offline --profile ci --all-targets --all-features -- -D warnings passes.
  • Extended workspace command: 10,761 Rust tests passed, zero failed, 10 ignored; the default SQLLogicTest runner completed 509/509 selected files. These are DataFusion tests, not a new Snowflake live audit or an increase in the Rustice fixture denominator.
  • Independent read-only review found no blocking issues and explicitly approved the source, test coverage, frozen evidence and bounded planning tradeoff for this fork branch. The review verified all final checks against the same source bytes.

EXPLAIN diagnostics show a typed NULL literal retaining its extension metadata after simplification. The physical plan remains a constant projection; metadata introspection folds to the correct constant extension name. The diagnostics were captured separately, not substituted for oracle expectations.

Local performance screen

Same driver, compiler, CI profile, features and dependencies; fresh expression/optimizer libraries and frozen ELFs; four interleaved AB/BA process pairs. All eight children pass their untimed checks, yielding 896 retained repetition records across 16 groups. Ratios are medians of matched process ratios, not pooled repetitions or individual-call percentiles.

Planning control Paired cost change Paired time change
All-false column, 1 branch +6.36% +262.75 ns
All-false column, 64 branches +0.40% +331.46 ns
All-false column, 256 branches -3.77% -13.88 us
Ordinary dynamic column CASE +0.27% +11.34 ns
Native field controls, unmarked/marked, 1/64/256 branches -0.31% to +0.82% All observations retained

No group crosses the predeclared AND gate of a matched median ratio above 1.10 and a matched median increase above 100 ns. This is not zero regression: the single-column fold costs about 0.263 us more in this CI-profile screen, adverse in all four pairs. That bounded planning cost is accepted for correct result annotations on the guarded fold; there is no new per-row execution work.

All 25 adverse pair observations, six positive-cost group summaries and the rejected first screen remain retained. The wider revised control corpus and separate campaigns do not establish a direct v1-to-v2 speedup. These are opt-level-0 planning measurements, not release/SQL/SPCS speedups, workload-frequency evidence, allocation profiling or statistical significance. Six receipt controls also pass normally and under python -O.

Are there any user-facing changes?

CASE-to-NULL simplification retains the expression's type and field metadata. The new component inference API is additive. No Snowflake comparison baseline, Rustice compatibility denominator or live-audit result was changed; Rustice dependency integration is a separate step.

Adapt apache#26951 for DF55, sharing native result type and metadata inference without redundant name/nullability work.

Co-authored-by: jameswillis <[email protected]>
@osipovartem
osipovartem merged commit 93324f7 into df55-physical-udf-instance Nov 4, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant