Repository navigation
Conversation
| return new CaseSplit(condition, value, elseBranch, conditionCols); | ||
| } | ||
|
|
||
| /** True unless the expression is built only from columns, constants, +, -, * and casts, which cannot fail. */ |
There was a problem hiding this comment.
Perhaps this could be a bit less strict? IIUC Hive division returns NULL instead of throwing, why not considering it?
How about other expressions like COALESCE, NVL, or nested IF/CASE (whose branches are safe) ?
There was a problem hiding this comment.
The / operator can be supported because as you described it returns null in case of division by 0. However the div operator throws exception so I will update the PR to allow only the first.
NVL and COALESCE has the same implementation in Hive and these are converted to case by CBO
nested IF/CASE (whose branches are safe)
Could you please share some examples?
There was a problem hiding this comment.
Maybe it's a bit of an edge case, I meant something like:
case when d.year=2024 then case when f.type='R' then -f.amt else f.amt end end
There was a problem hiding this comment.
It seems that the current implementation benefits from CBO again:
explain cbo
select d.g,
sum(case when d.year = 2024 then case when f.type = 'R' then -f.amt else f.amt end end),
count(case when d.year = 2024 then case when f.type = 'R' then -f.amt else f.amt end end)
from t_fact f
join t_dim d on f.k = d.k
group by d.g;
CBO PLAN:
HiveProject(d.g=[$0], _c1=[$1], _c2=[$2])
HiveAggregate(group=[{0}], agg#0=[sum($1)], agg#1=[count($1)])
HiveProject($f0=[$3], $f1=[CASE($4, $1, null:DECIMAL(7, 2))])
HiveJoin(condition=[=($0, $2)], joinType=[inner], algorithm=[none], cost=[not available])
HiveProject(k=[$1], EXPR$0=[CASE(=($2, 'R'), -($3), $3)])
HiveFilter(condition=[IS NOT NULL($1)])
HiveTableScan(table=[[default, t_fact]], table:alias=[f])
HiveProject(k=[$0], g=[$1], EXPR$0=[=($2, 2024)])
HiveFilter(condition=[IS NOT NULL($0)])
HiveTableScan(table=[[default, t_dim]], table:alias=[d])
The inner CASE expression is extracted and pushed below the join input where the columns it references are present.
In the physical plan the partial aggregate is pushed down.
Map 1
Map Operator Tree:
TableScan
alias: f
filterExpr: k is not null (type: boolean)
Statistics: Num rows: 10 Data size: 2010 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
predicate: k is not null (type: boolean)
Statistics: Num rows: 10 Data size: 2010 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: k (type: int), if((type = 'R'), (- amt), amt) (type: decimal(7,2))
outputColumnNames: _col0, _col1
Statistics: Num rows: 10 Data size: 1160 Basic stats: COMPLETE Column stats: COMPLETE
Group By Operator
aggregations: sum(_col1), count(_col1)
keys: _col0 (type: int)
minReductionHashAggr: 0.6
mode: partialhash
outputColumnNames: _col0, _col1, _col2
Statistics: Num rows: 10 Data size: 1240 Basic stats: COMPLETE Column stats: COMPLETE
Reduce Output Operator
key expressions: _col0 (type: int)
There was a problem hiding this comment.
I moved the case-split improvement to the jira HIVE-30274 and will submit another PR to implement it.
| } | ||
| for (ExprNodeDesc key : pushedKeys) { | ||
| ColStatistics keyStats = StatsUtils.getColStatisticsFromExpression(conf, inputStats, key); | ||
| if (keyStats == null || keyStats.getCountDistint() > maxKeyNdvRatio * inputRows) { |
There was a problem hiding this comment.
Shouldn't we check getColumnStatsState() != State.COMPLETE and bail out in that case too? (as it happens already in SetHashGroupByMinReduction)
There was a problem hiding this comment.
Currently, this is not as strict as the check in SetHashGroupByMinReduction.
In the case of NONE, we bail out.
I think the PARTIAL state shouldn't be a blocker in all cases—for example, when we have stats for the key columns, even if stats for some other columns are missing.
The question is whether we want to enable the push-down when a key column's stats are only estimated.
| } | ||
| // The push is skipped when the pushed keys (the original keys on this input, its | ||
| // join keys and its columns used by residual filters) are empty or outnumber the keys of the original GROUP BY | ||
| // plus the columns of the split IF conditions. |
There was a problem hiding this comment.
why? is this a perf-wise heuristic?
There was a problem hiding this comment.
The key-count check is a rough test to ensure that the pushed-down GBY groups no finer than the GBY above the join already does.
Please see this query:
select f.q, sum(case when d.c = 'x' then f.v end)
from t_fact f join t_dim d on f.k = d.k
group by f.q
The only declared key is f.q, so keys.size() is 1.
The pushed keys are f.q plus the join key f.k, so there are 2.
Without the condition columns, the check evaluates to 2 > 1, and the push-down is skipped.
If we rewrite the query like this:
select q, sum(case when c = 'x' then s end) from (
select f.q, d.c, sum(f.v) s
from t_fact f
join t_dim d on f.k = d.k
group by f.q, d.c) t
group by q
we can see that there are actually 2 GBY keys: d.c is the second one, and it comes from the CASE condition.
Please let me know if I should rewrite the comment to make this clearer.
There was a problem hiding this comment.
I moved the case-split improvement to the jira HIVE-30274 and will submit another PR to implement it.
| + "row the join discards. 0 disables the check."), | ||
| PARTIAL_AGGR_JOIN_TRANSPOSE_MAX_JOIN_EXPANSION("hive.optimize.partial.aggr.join.transpose.max.join.expansion", | ||
| 1.1f, | ||
| "A partial aggregate is not pushed below a join estimated to output more rows than this factor times the\n" |
There was a problem hiding this comment.
I'm not sure I understand this, why do we need a MAX_JOIN_EXPANSION? IIUC an expanding join is where aggregating early should help most, or am I missing something?
There was a problem hiding this comment.
I removed this setting and the corresponding code. After running the TPC-DS test suit again we found that it doesn't have any affect.
| if (maxJoinReduction > 0 && joinRows * maxJoinReduction < inputRows) { | ||
| return false; | ||
| } | ||
| for (ExprNodeDesc key : pushedKeys) { |
There was a problem hiding this comment.
Keys NDV are analyzed individually, but in case of correlation, two keys could each pass while together they could be nearly unique, wouldn't it be better bounding the combined NDV (e.g. using StatsUtils.computeNDVGroupingColumns) ?
There was a problem hiding this comment.
Your point makes sense, let me run some tests. In the meantime, I created a follow-up jira to track it: https://issues.apache.org/jira/browse/HIVE-30276
….expansion - based on running the TPCDS suite this gate doesn't prevent any push
c82516d to
fdc2b6e
Compare
|



What changes were proposed in this pull request?
Why are the changes needed?
To improve performance of queries with joins and aggregates.
Does this PR introduce any user-facing change?
No.
How was this patch tested?