Repository navigation
refactor(optimizer): deduplicate boolean literal comparison rules - #26375
emanuele-em wants to merge 2 commits into
Conversation
alamb
left a comment
There was a problem hiding this comment.
Thansk for this PR @emanuele-em -- but to be honest I am not sure this is obviously more readable
Also I don't understand why we changed the tests
|
|
||
| // c2 != false -> c2 | ||
| assert_eq!(simplify(col("c2").not_eq(lit(false))), col("c2"),); | ||
| for name in ["c2", "c2_non_null"] { |
There was a problem hiding this comment.
I added c2_non_null to check that the refactor preserves the same simplifications for non-nullable columns as for nullable c2. This adds coverage and it does not change the expected behavior. I can keep the original assertions and add that coverage separately to make the refactor easier to review if you prefer
There was a problem hiding this comment.
nit: I agree with @alamb here. Could you keep the original simplify(col("c2").eq(lit(true))) style assertions unchanged and add the new simplify_no_canonicalize loops alongside them? The originals cover the default path with canonicalization on, and leaving them untouched makes it easy to see the refactor didn't change any expected output. I ran the new loops against the pre-PR code and they pass there too, so the extra coverage is a good addition.
There was a problem hiding this comment.
Restored the original simplify(...) assertions unchanged in c4f2692, and kept the new simplify_no_canonicalize loops alongside them, so the diff in the tests is now purely additive. Thanks @alamb and @jayzhan211.
4190295 to
46d882f
Compare
46d882f to
0842227
Compare
| other: Box<Expr>, | ||
| op: Operator, | ||
| ) -> Result<Expr> { | ||
| Ok(match (as_bool_lit(literal)?, op) { |
There was a problem hiding this comment.
nit: the (Some(_), _) arm is a catch-all, so if anyone ever calls this with another operator it silently becomes NOT other. Listing the four cases explicitly reads like the old rule table and might also help with the readability concern:
- Ok(match (as_bool_lit(literal)?, op) {
- (Some(true), Operator::Eq) | (Some(false), Operator::NotEq) => *other,
- (Some(_), _) => Expr::Not(other),
- (None, _) => lit_bool_null(),
+ Ok(match (op, as_bool_lit(literal)?) {
+ (Operator::Eq, Some(true)) | (Operator::NotEq, Some(false)) => *other,
+ (Operator::Eq, Some(false)) | (Operator::NotEq, Some(true)) => Expr::Not(other),
+ (Operator::Eq | Operator::NotEq, None) => lit_bool_null(),
+ _ => return internal_err!("unexpected operator {op} for boolean comparison"),
})There was a problem hiding this comment.
Done in c4f2692: the four cases are now listed explicitly and any other operator returns an internal error. Agreed it reads closer to the old rule table.
| } | ||
| } | ||
|
|
||
| /// Simplifies `expr {=, !=} literal` in either operand order after checking |
There was a problem hiding this comment.
nit: the helper doesn't check the operand types itself; the match guards at the call sites do. Maybe word it as "Callers must ensure both operands are boolean."
Share Eq/NotEq literal rewrites and cover both operand orders, nullable columns, and nonboolean type guards.
0842227 to
b4a29c9
Compare
- List the four Eq/NotEq cases explicitly and return an internal error for any other operator. - Clarify that callers must ensure both operands are boolean. - Restore the original simplify() assertions and keep the new no-canonicalize loops alongside them.
Which issue does this PR close?
Part of #26302. This PR covers the boolean-literal comparison family only and does not close the full issue.
Rationale for this change
Keep the existing boolean predicate simplifications consistent when maintaining equality and inequality rules. The issue requests small, behavior-preserving changes, one rule family at a time.
The issue discussion also contains another contributor's plan for the same Eq/NotEq family: #26302 (comment). No matching open PR was found when preparing this PR. Coordination on this overlap is pending; this PR makes the completed refactor available for that discussion.
What changes are included in this PR?
What is the testing strategy for this PR?
The existing Rust comparison tests disable canonicalization so both literal positions reach the simplifier directly. They also check the nonboolean type guards on raw expressions before type coercion. These are characterization tests for a refactor and passed before and after the production change.
Completed validation:
cargo test --profile ci -p datafusion-optimizer: 941 unit tests, 29 integration tests, and 5 doctests passed; 1 doctest ignored.simplifyfilter: all 4 matching files passed.cargo fmt --alland the CI Clippy configuration passed. TOML formatting, license headers, spelling, and Markdown formatting also passed.simplify_function_over_casesimplifier benchmark against unchanged main at cf4869c. The three cases showed no statistically significant regression. This benchmark exercises the general simplifier and does not directly measure the boolean comparison family; no speedup is claimed.Validation gaps:
cargo clippy --all-targets --all-features -- -D warningsreported 3mutable_key_typeerrors in unchangedpush_down_filter.rs. A focused all-features Clippy run reproduced the same errors on unchanged main../dev/rust_lint.shsuite did not finish; it was stopped at documentation generation after the checks listed above passed.cargo test -p datafusionandcargo test -p datafusion-clicommands have not been run separately. The extended workspace command excluded the CLI.@alamb, could you help trigger CI for this first contribution? The validation gaps above remain open.
Are there any user-facing changes?
No intended changes to SQL results or public Rust APIs. This is an internal refactor of existing simplification rules.