Skip to content

HIVE-30261: Push a partial aggregation below an inner join - #6963

Draft
kasakrisz wants to merge 12 commits into
apache:masterfrom
kasakrisz:hive-perf-partial-aggr
Draft

kasakrisz wants to merge 12 commits into
apache:masterfrom
kasakrisz:hive-perf-partial-aggr

Conversation

@kasakrisz

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

  • Add a new Tez physical optimization: PartialGroupByJoinTransposeOptimizer. It pushes a copy of the map-side (hash) GROUP BY below an inner join. The copy goes to the join input that holds every aggregation argument.
TS - SEL - RS \
               JOIN - SEL - GBY(hash) - RS - GBY(mergepartial)
TS - SEL - RS /

=>

TS - SEL - GBY(partialhash) - RS \
                                  JOIN - GBY(hash) - RS - GBY(mergepartial)
TS - SEL -------------------- RS /
  • The pushed GROUP BY runs in a new mode: partialhash, in the vertex of the join input, before its existing shuffle.
  • The GROUP BY above the join keeps its mode and output, but it merges the partial results (PARTIAL2) instead of reading the rows. The reduce-side GROUP BY does not change.
  • When a pushed GROUP BY that is matched again can be pushed further down a chain of joins and a Select operator that projects its keys and partial results replaces it above the lower join.
  • The pushed GROUB BY keys are the original keys available on that input, the input's join keys, and its columns used by residual (non-equi) join filters.
  • The new optimization is run by TezCompiler after annotates the statistics. It is run before the TopNKey optimization and dynamic partition pruning. Otherwise, TopNKey operators and semijoin branches would block the push-down.
  • Synthetic join predicates of residual filters are re-pointed to the output of the pushed GROUP BY.
  • In StatsRulesProcFactory a GROUP BY with partial hash mode is estimated at its parent's row count. This is pessimistic, so map join conversion decisions don't change.
  • Synthetic join predicates doesn't change filter statistics. This was necessary because stats annotation run is moved before dynamic partition pruning which removes Synthetic join predicates.

Why are the changes needed?

To improve performance of queries with joins and aggregates.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

mvn test -Dtest.output.overwrite -Dtest=TestMiniLlapLocalCliDriver -Dqfile=partial_aggr_join_transpose.q -pl itests/qtest -Pitests

return new CaseSplit(condition, value, elseBranch, conditionCols);
}

/** True unless the expression is built only from columns, constants, +, -, * and casts, which cannot fail. */

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps this could be a bit less strict? IIUC Hive division returns NULL instead of throwing, why not considering it?
How about other expressions like COALESCE, NVL, or nested IF/CASE (whose branches are safe) ?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The / operator can be supported because as you described it returns null in case of division by 0. However the div operator throws exception so I will update the PR to allow only the first.

NVL and COALESCE has the same implementation in Hive and these are converted to case by CBO

nested IF/CASE (whose branches are safe)

Could you please share some examples?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe it's a bit of an edge case, I meant something like:
case when d.year=2024 then case when f.type='R' then -f.amt else f.amt end end

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems that the current implementation benefits from CBO again:

explain cbo
select d.g,
  sum(case when d.year = 2024 then case when f.type = 'R' then -f.amt else f.amt end end),
  count(case when d.year = 2024 then case when f.type = 'R' then -f.amt else f.amt end end)
from t_fact f
join t_dim d on f.k = d.k
group by d.g;
CBO PLAN:
HiveProject(d.g=[$0], _c1=[$1], _c2=[$2])
  HiveAggregate(group=[{0}], agg#0=[sum($1)], agg#1=[count($1)])
    HiveProject($f0=[$3], $f1=[CASE($4, $1, null:DECIMAL(7, 2))])
      HiveJoin(condition=[=($0, $2)], joinType=[inner], algorithm=[none], cost=[not available])
        HiveProject(k=[$1], EXPR$0=[CASE(=($2, 'R'), -($3), $3)])
          HiveFilter(condition=[IS NOT NULL($1)])
            HiveTableScan(table=[[default, t_fact]], table:alias=[f])
        HiveProject(k=[$0], g=[$1], EXPR$0=[=($2, 2024)])
          HiveFilter(condition=[IS NOT NULL($0)])
            HiveTableScan(table=[[default, t_dim]], table:alias=[d])

The inner CASE expression is extracted and pushed below the join input where the columns it references are present.

In the physical plan the partial aggregate is pushed down.

        Map 1 
            Map Operator Tree:
                TableScan
                  alias: f
                  filterExpr: k is not null (type: boolean)
                  Statistics: Num rows: 10 Data size: 2010 Basic stats: COMPLETE Column stats: COMPLETE
                  Filter Operator
                    predicate: k is not null (type: boolean)
                    Statistics: Num rows: 10 Data size: 2010 Basic stats: COMPLETE Column stats: COMPLETE
                    Select Operator
                      expressions: k (type: int), if((type = 'R'), (- amt), amt) (type: decimal(7,2))
                      outputColumnNames: _col0, _col1
                      Statistics: Num rows: 10 Data size: 1160 Basic stats: COMPLETE Column stats: COMPLETE
                      Group By Operator
                        aggregations: sum(_col1), count(_col1)
                        keys: _col0 (type: int)
                        minReductionHashAggr: 0.6
                        mode: partialhash
                        outputColumnNames: _col0, _col1, _col2
                        Statistics: Num rows: 10 Data size: 1240 Basic stats: COMPLETE Column stats: COMPLETE
                        Reduce Output Operator
                          key expressions: _col0 (type: int)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I moved the case-split improvement to the jira HIVE-30274 and will submit another PR to implement it.

}
for (ExprNodeDesc key : pushedKeys) {
ColStatistics keyStats = StatsUtils.getColStatisticsFromExpression(conf, inputStats, key);
if (keyStats == null || keyStats.getCountDistint() > maxKeyNdvRatio * inputRows) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't we check getColumnStatsState() != State.COMPLETE and bail out in that case too? (as it happens already in SetHashGroupByMinReduction)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently, this is not as strict as the check in SetHashGroupByMinReduction.
In the case of NONE, we bail out.
I think the PARTIAL state shouldn't be a blocker in all cases—for example, when we have stats for the key columns, even if stats for some other columns are missing.
The question is whether we want to enable the push-down when a key column's stats are only estimated.

}
// The push is skipped when the pushed keys (the original keys on this input, its
// join keys and its columns used by residual filters) are empty or outnumber the keys of the original GROUP BY
// plus the columns of the split IF conditions.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why? is this a perf-wise heuristic?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The key-count check is a rough test to ensure that the pushed-down GBY groups no finer than the GBY above the join already does.
Please see this query:

  select f.q, sum(case when d.c = 'x' then f.v end)
  from t_fact f join t_dim d on f.k = d.k
  group by f.q

The only declared key is f.q, so keys.size() is 1.
The pushed keys are f.q plus the join key f.k, so there are 2.
Without the condition columns, the check evaluates to 2 > 1, and the push-down is skipped.

If we rewrite the query like this:

select q, sum(case when c = 'x' then s end) from (
  select f.q, d.c, sum(f.v) s
  from t_fact f
  join t_dim d on f.k = d.k
  group by f.q, d.c) t
group by q

we can see that there are actually 2 GBY keys: d.c is the second one, and it comes from the CASE condition.

Please let me know if I should rewrite the comment to make this clearer.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I moved the case-split improvement to the jira HIVE-30274 and will submit another PR to implement it.

+ "row the join discards. 0 disables the check."),
PARTIAL_AGGR_JOIN_TRANSPOSE_MAX_JOIN_EXPANSION("hive.optimize.partial.aggr.join.transpose.max.join.expansion",
1.1f,
"A partial aggregate is not pushed below a join estimated to output more rows than this factor times the\n"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure I understand this, why do we need a MAX_JOIN_EXPANSION? IIUC an expanding join is where aggregating early should help most, or am I missing something?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I removed this setting and the corresponding code. After running the TPC-DS test suit again we found that it doesn't have any affect.

if (maxJoinReduction > 0 && joinRows * maxJoinReduction < inputRows) {
return false;
}
for (ExprNodeDesc key : pushedKeys) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keys NDV are analyzed individually, but in case of correlation, two keys could each pass while together they could be nearly unique, wouldn't it be better bounding the combined NDV (e.g. using StatsUtils.computeNDVGroupingColumns) ?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your point makes sense, let me run some tests. In the meantime, I created a follow-up jira to track it: https://issues.apache.org/jira/browse/HIVE-30276

@sonarqubecloud

Copy link
Copy Markdown

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants