Repository navigation
Push down sort order for ungrouped MIN/MAX aggregates and through FilterExec - #27450
Draft
galaxytemple wants to merge 4 commits into
Draft
galaxytemple wants to merge 4 commits into
galaxytemple wants to merge 4 commits into
Conversation
galaxytemple
force-pushed
the
feature-26809-filter-sort-pushdown
branch
from
November 15, 2026 03:28
25afd40 to
55d3b82
Compare
galaxytemple
force-pushed
the
feature-26809-filter-sort-pushdown
branch
from
November 15, 2026 03:35
55d3b82 to
0985821
Compare
galaxytemple
force-pushed
the
feature-26809-filter-sort-pushdown
branch
from
November 15, 2026 03:41
0985821 to
335a608
Compare
galaxytemple
force-pushed
the
feature-26809-filter-sort-pushdown
branch
from
November 15, 2026 04:48
a3249a2 to
00c1dde
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
Ungrouped
MIN(col)/MAX(col)queries with a filter (e.g.,SELECT MAX(ts) FROM t WHERE val > 0) and TopK queries over aFilterExeccurrently scan Parquet files and row groups in their original listing order rather than statistics order:FilterExecdid not implementtry_pushdown_sort: Even thoughFilterExecpreserves input ordering, it did not forward sort pushdown requests to its input, blocking sort pushdown whenever aFilterExecremained above the scan.PushdownSortonly matchedSortExec: UngroupedMIN/MAXAggregateExecnodes emit a dynamic filter (AggrDynFilter, introduced in optimizer: Support dynamic filter inMIN/MAXaggregates #18644), but without a scan-order hint fromPushdownSort, files and row groups were scanned in listing order rather than statistics order, preventing the dynamic filter bound from tightening early.Another possible approach was rewriting
SELECT MAX(col)intoORDER BY col DESC LIMIT 1in the logical optimizer. However,LIMIT 1returns 0 rows on empty input (whileMAXreturns a[NULL]row). And using the existingAggregateExecdynamic filter andtry_pushdown_sortfelt much simpler.What changes are included in this PR?
FilterExec::try_pushdown_sortindatafusion/physical-plan/src/filter.rs:FilterExecviaFilterExecBuilder::from(self).with_input(new_input).self.projectionusingFilterRemapper::with_column_mapping(same asFilterExec::handle_child_pushdown_result).Unsupportedifself.fetch.is_some().PushdownSort, push[col DESC NULLS LAST]for ungroupedMAX(col)and[col ASC NULLS LAST]forMIN(col)down to the aggregate input when dynamic filter pushdown is enabled and the input has nofetch.EXPLAINsnapshots whereWHERE+ORDER BYqueries now push sort order down to Parquet scans throughFilterExec(sort_order_for_reorder=[...]).What is the testing strategy for this PR?
pushdown_sort.rsusingTestScan:FilterExec(TopK).FilterExecwith column projection remapping.MAXandMINaggregates.tsorder) and runningEXPLAIN ANALYZE:SELECT MAX(ts) FROM t WHERE val > 0: 2.78s → 29ms (readsfile_050first and skips the remaining 49 files via dynamic filter).SELECT ts FROM t WHERE val > 0 ORDER BY ts DESC LIMIT 1: 2.75s → 34ms.Are there any user-facing changes?
No API changes.
Notes
Existing sqllogictests:
Several existing .slt tests (e.g., push_down_filter_regression.slt) had their EXPLAIN physical plans updated to include sort_order_for_reorder=[...].
Previously, when a FilterExec was placed above the scan, sort pushdown was blocked. But now that FilterExec::try_pushdown_sort is implemented, sort requirements are forwarded to the scan.
AggregateMode::Single
Initially, I also considered enabling dynamic filter for AggregateMode::Single as well (since single-partition plans can still scan across multiple Parquet files or row groups).
However, seeing that test_aggregate_dynamic_filter_not_created_for_single_mode explicitly asserts against creating dynamic filters in Single mode, and allowing Single is slightly out of scope for the original issue. I kept init_dynamic_filter restricted to AggregateMode::Partial to preserve existing design.