feat: aggregate a large final hash table as hash buckets (experimental, off by default) by jayzhan211 · Pull Request #25724 · apache/datafusion

jayzhan211 · 2026-09-25T01:27:56Z

Which issue does this PR close?

Related to Implement cache-efficient partial aggregation by Leis et al #20773. Not part of the [EPIC] Use blocked / chunked memory management in hash aggregation #24704 memory redesign: no API changes, and nothing changes unless the new option is set.

Rationale for this change

Once the table of a final hash aggregation outgrows the CPU caches, every probe and accumulator update is a cache miss, so the cost per row grows with the number of groups. With datafusion.execution.hash_aggregate_bucket_threshold = 262144, integer keys with 4M-60M groups run in 0.68x-0.76x of the time and ClickBench Q32 in 0.75x. Queries whose final table stays below the threshold run as before. The option is experimental and off by default.

When the final table of one partition reaches the threshold, the stream moves its state and all further input into 64 hash buckets, then aggregates the buckets one after another with one small, reused table. Each bucket is emitted, released, spilled or split again on its own.

Three constraints shaped the design:

Fixed-width keys and state only. Moving a row into its bucket costs 8-25 ns for fixed-width columns but 30-67 ns for strings, as much as the smaller table saves; string-keyed ClickBench queries were 2-10% slower on the benchmark runner before this restriction.
Buckets are compacted. When the partial aggregation was skipped, the final input repeats its groups, and buckets that only buffer rows grow with the input rather than with the groups. A bucket is aggregated in place once it has received at most twice its own rows; on input with 7.5 rows per group, peak memory stays at 1.13x of one table.
Spilling must be enabled. Buckets that cannot spill only hold more memory than one table.

M4 Pro, 12 partitions, same binary off and on, median of 5-7 interleaved rounds:

query	group key	on / off
60M rows, 4M / 20M / 60M groups	`BIGINT`	0.76x / 0.73x / 0.68x
ClickBench Q32 (half data)	`WatchID, ClientIP`	0.75x
ClickBench Q15	`UserID`	0.89x
ClickBench Q4, Q8, Q31, Q35	integer keys	0.89-0.97x

Controls and known costs

Controls (no bucketing): ClickBench Q5 1.00x, Q11 0.99x, Q13 0.98x, Q17 0.96x; 1M uniform keys (83k per final partition) 0.99x; near-unique integer keys ending at 104k-174k groups per partition 0.99-1.00x.

Known costs:

Aggregations that end at up to twice the threshold gain little: near-unique integer keys ending at 260k / 520k groups per partition run in 1.00x / 0.97x of the time.
When the partial aggregation was skipped and every group repeats, buckets hold rows until their next compaction: 60M rows in 8M groups (7.5 rows per group, the shape of TPC-H GROUP BY l_partkey, l_suppkey) run in 0.74x of the time with 1.13x peak RSS (622 to 702 MB).

What changes are included in this PR?

Start reading at BucketedAggregation in aggregates/bucketed_aggregation.rs: split, compact, reserve and output_stream are the whole life of the buckets. aggregates/final_buckets.rs stores them; FinalHashAggregateStream only decides when to switch.

The decision I most want reviewed is keeping compaction in this PR rather than adding it later: without it, buckets hold every row that arrives after the switch until the input ends, so memory grows with the input instead of with the groups.

Behavior of each piece

piece	behavior
switch point	the threshold, for every input
routing	64 buckets by hash bits, one gather per batch; a bucket that reaches the threshold again is split by the next bits, up to 4 levels
compaction	after a compaction that paid off, the next one waits for 1x-2x as many new rows as it kept; after one that kept more than 80% of its rows, 4x
memory	under pressure the largest bucket spills; a bucket's memory is released once its groups are emitted
eligibility	fixed-width keys and state, no group limit (`LIMIT` pushed into the aggregation), spilling enabled

What is the testing strategy for this PR?

hash_stream.rs unit tests: buckets return the same groups as one table, split again, spill under a memory limit, compact repeated groups, and stay off without a disk manager. final_buckets.rs and bucketed_aggregation.rs test routing and eligibility.
aggregate_bucketed.slt runs each query off and at 100 groups with identical results: integer, nullable, Utf8, Utf8View and mixed keys, AVG, DISTINCT, COUNT(DISTINCT), median, array_agg.
With the default set to 100 locally, the aggregate unit, fuzz and memory_limit tests and the whole sqllogictest suite pass, except information_schema.slt, which prints the default.

Are there any user-facing changes?

A new experimental option, off by default and documented in configs.md with its costs, and two metrics on AggregateExec: bucket_splits, bucket_compactions.

jayzhan211 · 2026-09-25T02:28:03Z

@Rachelint I couldn't get rid of all the regressions, but given the size of the win compared to the loss, do you think it's still worth adding this as opt-in? 🤔

codecov-commenter · 2026-09-25T02:33:44Z

Codecov Report

❌ Patch coverage is 88.74879% with 116 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.65%. Comparing base (22a0fdb) to head (88d36d4).
⚠️ Report is 333 commits behind head on main.

Files with missing lines	Patch %	Lines
...fusion/physical-plan/src/aggregates/hash_stream.rs	89.85%	9 Missing and 33 partials ⚠️
...sion/physical-plan/src/aggregates/final_buckets.rs	86.69%	8 Missing and 29 partials ⚠️
...ysical-plan/src/aggregates/bucketed_aggregation.rs	86.47%	10 Missing and 23 partials ⚠️
...src/aggregates/aggregate_hash_table/final_table.rs	90.00%	3 Missing ⚠️
...plan/src/aggregates/aggregate_hash_table/common.rs	98.33%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #25724      +/-   ##
==========================================
+ Coverage   82.64%   82.65%   +0.01%     
==========================================
  Files        1147     1149       +2     
  Lines      445440   446512    +1072     
  Branches   445440   446512    +1072     
==========================================
+ Hits       368123   369067     +944     
- Misses      54970    55002      +32     
- Partials    22347    22443      +96

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Rachelint · 2026-09-27T03:02:32Z

run benchmark clickbench_partitioned
changed:
env:
DATAFUSION_EXECUTION_HASH_AGGREGATE_BUCKET_THRESHOLD: "262144"

adriangbot · 2026-09-27T03:05:34Z

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c5852152570-2837-m9r88 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing agg-buckets-final-stage (f9df89e) to fb42cf5 (merge-base) diff

Run configuration

run benchmark clickbench_partitioned
changed:
  env:
    DATAFUSION_EXECUTION_HASH_AGGREGATE_BUCKET_THRESHOLD: "262144"

Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-09-27T03:19:20Z

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

Comparing agg-buckets-final-stage (f9df89e) to fb42cf5 (merge-base) diff

Run configuration

run benchmark clickbench_partitioned
changed:
  env:
    DATAFUSION_EXECUTION_HASH_AGGREGATE_BUCKET_THRESHOLD: "262144"

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

Comparing HEAD and agg-buckets-final-stage
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃       HEAD ┃ agg-buckets-final-stage ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0  │    1.28 ms │                 1.46 ms │  1.13x slower │
│ QQuery 1  │   12.06 ms │                12.44 ms │     no change │
│ QQuery 2  │   37.10 ms │                37.78 ms │     no change │
│ QQuery 3  │   31.70 ms │                33.35 ms │  1.05x slower │
│ QQuery 4  │  240.46 ms │               197.16 ms │ +1.22x faster │
│ QQuery 5  │  277.16 ms │               297.77 ms │  1.07x slower │
│ QQuery 6  │    1.33 ms │                 1.48 ms │  1.11x slower │
│ QQuery 7  │   13.33 ms │                13.98 ms │     no change │
│ QQuery 8  │  363.73 ms │               299.14 ms │ +1.22x faster │
│ QQuery 9  │  506.29 ms │               488.58 ms │     no change │
│ QQuery 10 │   64.74 ms │                71.14 ms │  1.10x slower │
│ QQuery 11 │   77.33 ms │                80.64 ms │     no change │
│ QQuery 12 │  269.03 ms │               283.70 ms │  1.05x slower │
│ QQuery 13 │  361.55 ms │               376.92 ms │     no change │
│ QQuery 14 │  304.80 ms │               318.67 ms │     no change │
│ QQuery 15 │  292.57 ms │               232.05 ms │ +1.26x faster │
│ QQuery 16 │  637.86 ms │               670.12 ms │  1.05x slower │
│ QQuery 17 │  682.91 ms │               615.08 ms │ +1.11x faster │
│ QQuery 18 │ 1320.41 ms │              1255.88 ms │     no change │
│ QQuery 19 │   28.74 ms │                29.54 ms │     no change │
│ QQuery 20 │  527.47 ms │               536.24 ms │     no change │
│ QQuery 21 │  515.78 ms │               547.87 ms │  1.06x slower │
│ QQuery 22 │ 1035.29 ms │              1112.90 ms │  1.07x slower │
│ QQuery 23 │ 3231.05 ms │              3332.93 ms │     no change │
│ QQuery 24 │   42.80 ms │                41.88 ms │     no change │
│ QQuery 25 │  111.22 ms │               108.35 ms │     no change │
│ QQuery 26 │   42.83 ms │                41.55 ms │     no change │
│ QQuery 27 │  517.82 ms │               512.48 ms │     no change │
│ QQuery 28 │ 2981.11 ms │              2996.24 ms │     no change │
│ QQuery 29 │   43.38 ms │                43.58 ms │     no change │
│ QQuery 30 │  332.42 ms │               320.25 ms │     no change │
│ QQuery 31 │  295.33 ms │               269.95 ms │ +1.09x faster │
│ QQuery 32 │ 1085.61 ms │               761.67 ms │ +1.43x faster │
│ QQuery 33 │ 1657.87 ms │              1523.38 ms │ +1.09x faster │
│ QQuery 34 │ 1654.20 ms │              1693.00 ms │     no change │
│ QQuery 35 │  357.73 ms │               248.53 ms │ +1.44x faster │
│ QQuery 36 │   71.68 ms │                69.75 ms │     no change │
│ QQuery 37 │   39.63 ms │                35.63 ms │ +1.11x faster │
│ QQuery 38 │   44.33 ms │                42.81 ms │     no change │
│ QQuery 39 │  149.75 ms │               146.74 ms │     no change │
│ QQuery 40 │   15.33 ms │                15.23 ms │     no change │
│ QQuery 41 │   14.85 ms │                14.42 ms │     no change │
│ QQuery 42 │   14.44 ms │                14.11 ms │     no change │
└───────────┴────────────┴─────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                      ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                      │ 20306.27ms │
│ Total Time (agg-buckets-final-stage)   │ 19746.36ms │
│ Average Time (HEAD)                    │   472.24ms │
│ Average Time (agg-buckets-final-stage) │   459.22ms │
│ Queries Faster                         │          9 │
│ Queries Slower                         │          9 │
│ Queries with No Change                 │         25 │
│ Queries with Failure                   │          0 │
└────────────────────────────────────────┴────────────┘

Distribution per query (min / mean ±stddev / max):

Comparing HEAD and agg-buckets-final-stage
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃                                   HEAD ┃                agg-buckets-final-stage ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0  │           1.28 / 4.29 ±5.76 / 15.80 ms │           1.46 / 4.53 ±5.97 / 16.46 ms │  1.06x slower │
│ QQuery 1  │         12.06 / 12.28 ±0.15 / 12.46 ms │         12.44 / 12.58 ±0.12 / 12.80 ms │     no change │
│ QQuery 2  │         37.10 / 37.28 ±0.13 / 37.50 ms │         37.78 / 38.11 ±0.37 / 38.66 ms │     no change │
│ QQuery 3  │         31.70 / 34.13 ±3.20 / 40.33 ms │         33.35 / 33.95 ±0.54 / 34.94 ms │     no change │
│ QQuery 4  │      240.46 / 244.85 ±4.38 / 252.09 ms │      197.16 / 200.24 ±2.37 / 204.09 ms │ +1.22x faster │
│ QQuery 5  │      277.16 / 282.21 ±3.03 / 286.17 ms │      297.77 / 305.40 ±6.49 / 312.94 ms │  1.08x slower │
│ QQuery 6  │            1.33 / 1.48 ±0.23 / 1.93 ms │            1.48 / 1.62 ±0.22 / 2.06 ms │  1.09x slower │
│ QQuery 7  │         13.33 / 13.56 ±0.17 / 13.76 ms │         13.98 / 14.08 ±0.12 / 14.31 ms │     no change │
│ QQuery 8  │      363.73 / 372.66 ±8.94 / 383.73 ms │      299.14 / 305.28 ±3.56 / 309.29 ms │ +1.22x faster │
│ QQuery 9  │     506.29 / 520.41 ±13.98 / 541.69 ms │     488.58 / 510.69 ±18.23 / 543.43 ms │     no change │
│ QQuery 10 │         64.74 / 68.50 ±5.61 / 79.64 ms │        71.14 / 79.71 ±10.84 / 99.18 ms │  1.16x slower │
│ QQuery 11 │         77.33 / 78.25 ±0.70 / 79.42 ms │         80.64 / 82.54 ±1.21 / 84.21 ms │  1.05x slower │
│ QQuery 12 │     269.03 / 284.00 ±12.57 / 298.76 ms │     283.70 / 301.06 ±10.05 / 315.05 ms │  1.06x slower │
│ QQuery 13 │     361.55 / 381.24 ±17.19 / 410.35 ms │     376.92 / 387.69 ±10.43 / 400.61 ms │     no change │
│ QQuery 14 │      304.80 / 312.19 ±5.02 / 319.43 ms │      318.67 / 328.78 ±5.98 / 334.46 ms │  1.05x slower │
│ QQuery 15 │      292.57 / 305.01 ±9.89 / 320.08 ms │     232.05 / 251.43 ±34.65 / 320.66 ms │ +1.21x faster │
│ QQuery 16 │     637.86 / 680.62 ±37.94 / 743.40 ms │     670.12 / 710.89 ±37.52 / 772.52 ms │     no change │
│ QQuery 17 │     682.91 / 715.67 ±33.82 / 775.03 ms │     615.08 / 636.81 ±17.19 / 658.07 ms │ +1.12x faster │
│ QQuery 18 │  1320.41 / 1418.94 ±58.46 / 1491.86 ms │ 1255.88 / 1351.85 ±123.60 / 1591.99 ms │     no change │
│ QQuery 19 │         28.74 / 29.45 ±0.91 / 31.23 ms │      29.54 / 89.18 ±111.85 / 312.59 ms │  3.03x slower │
│ QQuery 20 │     527.47 / 543.93 ±14.75 / 565.80 ms │      536.24 / 542.15 ±4.51 / 546.87 ms │     no change │
│ QQuery 21 │     515.78 / 534.39 ±16.28 / 563.95 ms │     547.87 / 583.43 ±39.12 / 654.30 ms │  1.09x slower │
│ QQuery 22 │   1035.29 / 1046.91 ±8.92 / 1061.00 ms │  1112.90 / 1173.32 ±62.55 / 1294.10 ms │  1.12x slower │
│ QQuery 23 │ 3231.05 / 3516.82 ±271.23 / 4026.96 ms │ 3332.93 / 3453.14 ±105.25 / 3623.59 ms │     no change │
│ QQuery 24 │       42.80 / 71.82 ±52.68 / 176.95 ms │     41.88 / 213.88 ±229.00 / 645.92 ms │  2.98x slower │
│ QQuery 25 │      111.22 / 113.04 ±1.89 / 116.58 ms │      108.35 / 109.86 ±1.62 / 112.43 ms │     no change │
│ QQuery 26 │        42.83 / 54.69 ±19.47 / 93.37 ms │         41.55 / 42.66 ±0.99 / 43.99 ms │ +1.28x faster │
│ QQuery 27 │      517.82 / 530.94 ±8.57 / 542.49 ms │     512.48 / 537.02 ±18.57 / 567.04 ms │     no change │
│ QQuery 28 │  2981.11 / 3012.29 ±25.35 / 3040.10 ms │  2996.24 / 3077.03 ±51.23 / 3154.21 ms │     no change │
│ QQuery 29 │       43.38 / 78.90 ±49.19 / 168.76 ms │       43.58 / 88.88 ±61.31 / 198.61 ms │  1.13x slower │
│ QQuery 30 │     332.42 / 350.66 ±10.94 / 360.73 ms │     320.25 / 346.57 ±38.12 / 420.40 ms │     no change │
│ QQuery 31 │      295.33 / 303.20 ±7.33 / 314.83 ms │     269.95 / 288.70 ±25.68 / 338.97 ms │     no change │
│ QQuery 32 │ 1085.61 / 1282.31 ±178.36 / 1534.97 ms │   761.67 / 915.04 ±166.83 / 1232.13 ms │ +1.40x faster │
│ QQuery 33 │  1657.87 / 1774.72 ±60.26 / 1823.48 ms │ 1523.38 / 1748.59 ±200.70 / 2087.13 ms │     no change │
│ QQuery 34 │  1654.20 / 1694.37 ±41.97 / 1749.35 ms │  1693.00 / 1781.32 ±91.54 / 1921.96 ms │  1.05x slower │
│ QQuery 35 │    357.73 / 429.60 ±107.53 / 639.33 ms │     248.53 / 294.79 ±71.16 / 436.35 ms │ +1.46x faster │
│ QQuery 36 │         71.68 / 76.21 ±4.82 / 83.38 ms │         69.75 / 71.15 ±1.34 / 73.35 ms │ +1.07x faster │
│ QQuery 37 │         39.63 / 45.73 ±8.91 / 63.34 ms │         35.63 / 37.63 ±2.77 / 43.01 ms │ +1.22x faster │
│ QQuery 38 │         44.33 / 48.14 ±4.62 / 57.15 ms │         42.81 / 47.63 ±6.33 / 59.97 ms │     no change │
│ QQuery 39 │      149.75 / 165.89 ±9.51 / 175.13 ms │      146.74 / 155.89 ±5.60 / 162.90 ms │ +1.06x faster │
│ QQuery 40 │         15.33 / 18.32 ±5.15 / 28.61 ms │         15.23 / 16.80 ±2.98 / 22.76 ms │ +1.09x faster │
│ QQuery 41 │         14.85 / 15.16 ±0.34 / 15.79 ms │         14.42 / 14.56 ±0.14 / 14.82 ms │     no change │
│ QQuery 42 │         14.44 / 17.02 ±4.68 / 26.37 ms │         14.11 / 17.22 ±4.14 / 24.52 ms │     no change │
└───────────┴────────────────────────────────────────┴────────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                      ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                      │ 21522.08ms │
│ Total Time (agg-buckets-final-stage)   │ 21203.73ms │
│ Average Time (HEAD)                    │   500.51ms │
│ Average Time (agg-buckets-final-stage) │   493.11ms │
│ Queries Faster                         │         11 │
│ Queries Slower                         │         13 │
│ Queries with No Change                 │         19 │
│ Queries with Failure                   │          0 │
└────────────────────────────────────────┴────────────┘

Resource Usage

clickbench_partitioned — base (merge-base)

Metric	Value
Wall time	110.0s
Peak memory	18.1 GiB
Avg memory	5.7 GiB
CPU user	1057.6s
CPU sys	106.8s
Peak spill	0 B

clickbench_partitioned — branch

Metric	Value
Wall time	110.0s
Peak memory	18.3 GiB
Avg memory	5.4 GiB
CPU user	1000.1s
CPU sys	105.9s
Peak spill	0 B

File an issue against this benchmark runner

Rachelint · 2026-09-28T17:43:25Z

run benchmark clickbench_partitioned
changed:
env:
DATAFUSION_EXECUTION_HASH_AGGREGATE_BUCKET_THRESHOLD: "262144"

adriangbot · 2026-09-28T17:43:28Z

Hi @Rachelint, your benchmark configuration could not be parsed (#25724 (comment)).

Error: invalid configuration: unknown field DATAFUSION_EXECUTION_HASH_AGGREGATE_BUCKET_THRESHOLD, expected one of env, baseline, changed, resources at line 3 column 1

Usage:

run benchmark <name>           # run specific benchmark(s)
run benchmarks                 # run default suite
run benchmarks <name1> <name2> # run specific benchmarks

Any benchmark name is accepted: bench.sh suite names (e.g. tpch, clickbench_partitioned, wide_schema) and Criterion bench targets (e.g. sql_planner) are resolved automatically. A name that matches neither fails on the runner.

Per-side configuration (run benchmark tpch followed by):

env:
# shared env is inherited by BOTH the build and the run, so build
# flags go here. Builds default to no debuginfo for speed; opt back
# in for hung-job gdb dumps and cap jobs to stay within memory:
CARGO_PROFILE_RELEASE_DEBUG: "1"
CARGO_BUILD_JOBS: "1"
baseline:
ref: v45.0.0
env:
# per-side env only reaches the benchmark run, not the build
DATAFUSION_RUNTIME_MEMORY_LIMIT: 1G
changed:
ref: v46.0.0
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: 2G

Pod size and CPU architecture (each key is optional and keeps the default when absent; requests and limits are set equal):

resources:
cpu: "16"
memory: "128Gi"
arch: arm64          # arm64 or amd64

File an issue against this benchmark runner

Rachelint · 2026-09-28T17:44:22Z

run benchmark clickbench_partitioned
changed:
env:
DATAFUSION_EXECUTION_HASH_AGGREGATE_BUCKET_THRESHOLD: "262144"

adriangbot · 2026-09-28T17:47:30Z

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c5875405044-2893-jnf6n 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing agg-buckets-final-stage (f9df89e) to fb42cf5 (merge-base) diff

Run configuration

run benchmark clickbench_partitioned
changed:
  env:
    DATAFUSION_EXECUTION_HASH_AGGREGATE_BUCKET_THRESHOLD: "262144"

Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-09-28T17:56:08Z

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

Comparing agg-buckets-final-stage (f9df89e) to fb42cf5 (merge-base) diff

Run configuration

run benchmark clickbench_partitioned
changed:
  env:
    DATAFUSION_EXECUTION_HASH_AGGREGATE_BUCKET_THRESHOLD: "262144"

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

Comparing HEAD and agg-buckets-final-stage
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃       HEAD ┃ agg-buckets-final-stage ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0  │    1.25 ms │                 1.25 ms │     no change │
│ QQuery 1  │   11.93 ms │                11.55 ms │     no change │
│ QQuery 2  │   36.64 ms │                36.74 ms │     no change │
│ QQuery 3  │   31.78 ms │                31.45 ms │     no change │
│ QQuery 4  │  236.45 ms │               190.34 ms │ +1.24x faster │
│ QQuery 5  │  271.14 ms │               282.52 ms │     no change │
│ QQuery 6  │    1.27 ms │                 1.28 ms │     no change │
│ QQuery 7  │   13.11 ms │                12.89 ms │     no change │
│ QQuery 8  │  331.99 ms │               296.68 ms │ +1.12x faster │
│ QQuery 9  │  466.82 ms │               454.35 ms │     no change │
│ QQuery 10 │   63.26 ms │                67.80 ms │  1.07x slower │
│ QQuery 11 │   74.20 ms │                78.98 ms │  1.06x slower │
│ QQuery 12 │  261.37 ms │               272.29 ms │     no change │
│ QQuery 13 │  356.30 ms │               362.00 ms │     no change │
│ QQuery 14 │  279.75 ms │               283.93 ms │     no change │
│ QQuery 15 │  278.27 ms │               217.48 ms │ +1.28x faster │
│ QQuery 16 │  616.75 ms │               622.61 ms │     no change │
│ QQuery 17 │  625.96 ms │               600.06 ms │     no change │
│ QQuery 18 │ 1267.68 ms │              1137.26 ms │ +1.11x faster │
│ QQuery 19 │   27.41 ms │                27.84 ms │     no change │
│ QQuery 20 │  521.60 ms │               520.63 ms │     no change │
│ QQuery 21 │  509.02 ms │               526.61 ms │     no change │
│ QQuery 22 │ 1009.58 ms │              1065.78 ms │  1.06x slower │
│ QQuery 23 │ 3133.84 ms │              3177.50 ms │     no change │
│ QQuery 24 │   40.93 ms │                40.53 ms │     no change │
│ QQuery 25 │  104.79 ms │               104.49 ms │     no change │
│ QQuery 26 │   42.01 ms │                41.78 ms │     no change │
│ QQuery 27 │  513.30 ms │               522.34 ms │     no change │
│ QQuery 28 │ 2880.26 ms │              2973.47 ms │     no change │
│ QQuery 29 │   41.20 ms │                41.68 ms │     no change │
│ QQuery 30 │  300.59 ms │               290.77 ms │     no change │
│ QQuery 31 │  279.87 ms │               251.27 ms │ +1.11x faster │
│ QQuery 32 │ 1033.40 ms │               716.41 ms │ +1.44x faster │
│ QQuery 33 │ 1587.55 ms │              1479.45 ms │ +1.07x faster │
│ QQuery 34 │ 1570.98 ms │              1493.84 ms │     no change │
│ QQuery 35 │  296.13 ms │               223.02 ms │ +1.33x faster │
│ QQuery 36 │   69.96 ms │                65.48 ms │ +1.07x faster │
│ QQuery 37 │   35.20 ms │                36.70 ms │     no change │
│ QQuery 38 │   41.71 ms │                42.92 ms │     no change │
│ QQuery 39 │  148.55 ms │               148.26 ms │     no change │
│ QQuery 40 │   14.45 ms │                14.30 ms │     no change │
│ QQuery 41 │   13.22 ms │                13.48 ms │     no change │
│ QQuery 42 │   13.01 ms │                13.05 ms │     no change │
└───────────┴────────────┴─────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                      ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                      │ 19454.47ms │
│ Total Time (agg-buckets-final-stage)   │ 18793.03ms │
│ Average Time (HEAD)                    │   452.43ms │
│ Average Time (agg-buckets-final-stage) │   437.05ms │
│ Queries Faster                         │          9 │
│ Queries Slower                         │          3 │
│ Queries with No Change                 │         31 │
│ Queries with Failure                   │          0 │
└────────────────────────────────────────┴────────────┘

Distribution per query (min / mean ±stddev / max):

Comparing HEAD and agg-buckets-final-stage
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃                                   HEAD ┃                agg-buckets-final-stage ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0  │           1.25 / 4.11 ±5.59 / 15.29 ms │           1.25 / 3.97 ±5.39 / 14.75 ms │     no change │
│ QQuery 1  │         11.93 / 12.04 ±0.07 / 12.15 ms │         11.55 / 11.82 ±0.17 / 12.03 ms │     no change │
│ QQuery 2  │         36.64 / 36.98 ±0.32 / 37.43 ms │         36.74 / 37.32 ±0.41 / 38.01 ms │     no change │
│ QQuery 3  │         31.78 / 32.72 ±0.93 / 34.27 ms │         31.45 / 31.92 ±0.30 / 32.27 ms │     no change │
│ QQuery 4  │      236.45 / 239.33 ±2.56 / 244.02 ms │      190.34 / 193.22 ±2.20 / 196.23 ms │ +1.24x faster │
│ QQuery 5  │      271.14 / 273.61 ±1.45 / 275.36 ms │      282.52 / 286.51 ±3.67 / 292.84 ms │     no change │
│ QQuery 6  │            1.27 / 1.43 ±0.24 / 1.89 ms │            1.28 / 1.44 ±0.23 / 1.90 ms │     no change │
│ QQuery 7  │         13.11 / 13.44 ±0.35 / 14.01 ms │         12.89 / 12.97 ±0.05 / 13.05 ms │     no change │
│ QQuery 8  │      331.99 / 332.48 ±0.44 / 333.05 ms │      296.68 / 298.78 ±2.59 / 303.43 ms │ +1.11x faster │
│ QQuery 9  │      466.82 / 475.43 ±9.19 / 489.77 ms │      454.35 / 467.37 ±6.77 / 474.23 ms │     no change │
│ QQuery 10 │         63.26 / 65.13 ±1.93 / 68.83 ms │         67.80 / 68.53 ±0.66 / 69.72 ms │  1.05x slower │
│ QQuery 11 │         74.20 / 75.30 ±1.08 / 77.16 ms │         78.98 / 81.95 ±4.94 / 91.82 ms │  1.09x slower │
│ QQuery 12 │      261.37 / 265.88 ±4.35 / 272.23 ms │      272.29 / 281.56 ±7.31 / 290.18 ms │  1.06x slower │
│ QQuery 13 │     356.30 / 368.73 ±15.34 / 398.45 ms │      362.00 / 365.58 ±2.77 / 370.44 ms │     no change │
│ QQuery 14 │      279.75 / 284.88 ±4.84 / 293.36 ms │      283.93 / 289.90 ±5.90 / 300.53 ms │     no change │
│ QQuery 15 │      278.27 / 290.82 ±9.62 / 304.65 ms │      217.48 / 223.94 ±5.39 / 233.84 ms │ +1.30x faster │
│ QQuery 16 │      616.75 / 622.20 ±5.52 / 632.79 ms │     622.61 / 645.20 ±12.72 / 657.14 ms │     no change │
│ QQuery 17 │     625.96 / 635.24 ±10.17 / 653.55 ms │      600.06 / 607.38 ±6.13 / 615.41 ms │     no change │
│ QQuery 18 │  1267.68 / 1291.05 ±17.77 / 1318.81 ms │  1137.26 / 1203.84 ±44.57 / 1268.27 ms │ +1.07x faster │
│ QQuery 19 │         27.41 / 27.74 ±0.27 / 28.20 ms │        27.84 / 36.53 ±17.10 / 70.72 ms │  1.32x slower │
│ QQuery 20 │     521.60 / 532.08 ±11.71 / 550.81 ms │     520.63 / 546.96 ±17.88 / 576.03 ms │     no change │
│ QQuery 21 │      509.02 / 517.95 ±5.44 / 524.99 ms │     526.61 / 543.45 ±14.22 / 561.74 ms │     no change │
│ QQuery 22 │  1009.58 / 1037.00 ±20.20 / 1061.82 ms │  1065.78 / 1110.67 ±29.94 / 1147.08 ms │  1.07x slower │
│ QQuery 23 │ 3133.84 / 3337.36 ±236.13 / 3799.40 ms │ 3177.50 / 3322.65 ±189.26 / 3689.78 ms │     no change │
│ QQuery 24 │       40.93 / 56.32 ±27.32 / 110.85 ms │         40.53 / 40.98 ±0.44 / 41.57 ms │ +1.37x faster │
│ QQuery 25 │      104.79 / 109.26 ±7.39 / 123.94 ms │    104.49 / 218.48 ±222.25 / 662.98 ms │  2.00x slower │
│ QQuery 26 │       42.01 / 59.59 ±33.32 / 126.20 ms │         41.78 / 45.28 ±2.47 / 47.89 ms │ +1.32x faster │
│ QQuery 27 │      513.30 / 519.79 ±3.60 / 523.25 ms │     522.34 / 531.02 ±10.75 / 552.12 ms │     no change │
│ QQuery 28 │  2880.26 / 2915.88 ±39.82 / 2981.99 ms │  2973.47 / 3009.99 ±33.26 / 3070.20 ms │     no change │
│ QQuery 29 │        41.20 / 48.77 ±14.29 / 77.34 ms │       41.68 / 76.04 ±56.08 / 186.95 ms │  1.56x slower │
│ QQuery 30 │      300.59 / 309.98 ±7.12 / 319.47 ms │      290.77 / 302.80 ±9.15 / 318.69 ms │     no change │
│ QQuery 31 │     279.87 / 301.54 ±25.23 / 344.65 ms │      251.27 / 256.75 ±6.73 / 269.80 ms │ +1.17x faster │
│ QQuery 32 │  1033.40 / 1176.64 ±90.27 / 1313.85 ms │     716.41 / 793.63 ±82.22 / 952.83 ms │ +1.48x faster │
│ QQuery 33 │  1587.55 / 1632.83 ±32.58 / 1689.29 ms │  1479.45 / 1575.22 ±70.80 / 1698.19 ms │     no change │
│ QQuery 34 │  1570.98 / 1634.45 ±70.43 / 1760.13 ms │ 1493.84 / 1692.08 ±126.33 / 1833.13 ms │     no change │
│ QQuery 35 │     296.13 / 372.92 ±88.63 / 513.56 ms │    223.02 / 293.87 ±104.13 / 499.91 ms │ +1.27x faster │
│ QQuery 36 │         69.96 / 76.65 ±9.53 / 95.37 ms │       65.48 / 76.51 ±12.69 / 100.65 ms │     no change │
│ QQuery 37 │         35.20 / 38.57 ±3.77 / 43.97 ms │         36.70 / 38.61 ±3.21 / 45.00 ms │     no change │
│ QQuery 38 │         41.71 / 51.83 ±8.66 / 63.41 ms │         42.92 / 46.58 ±2.94 / 51.76 ms │ +1.11x faster │
│ QQuery 39 │      148.55 / 154.12 ±3.73 / 158.80 ms │     148.26 / 157.82 ±11.53 / 179.09 ms │     no change │
│ QQuery 40 │         14.45 / 14.94 ±0.37 / 15.59 ms │         14.30 / 15.82 ±2.23 / 20.23 ms │  1.06x slower │
│ QQuery 41 │         13.22 / 16.67 ±4.23 / 24.24 ms │         13.48 / 15.00 ±1.82 / 18.01 ms │ +1.11x faster │
│ QQuery 42 │         13.01 / 15.76 ±5.00 / 25.74 ms │         13.05 / 13.17 ±0.12 / 13.38 ms │ +1.20x faster │
└───────────┴────────────────────────────────────────┴────────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                      ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                      │ 20279.44ms │
│ Total Time (agg-buckets-final-stage)   │ 19873.10ms │
│ Average Time (HEAD)                    │   471.61ms │
│ Average Time (agg-buckets-final-stage) │   462.17ms │
│ Queries Faster                         │         12 │
│ Queries Slower                         │          8 │
│ Queries with No Change                 │         23 │
│ Queries with Failure                   │          0 │
└────────────────────────────────────────┴────────────┘

Resource Usage

clickbench_partitioned — base (merge-base)

Metric	Value
Wall time	105.0s
Peak memory	17.6 GiB
Avg memory	5.8 GiB
CPU user	995.0s
CPU sys	97.8s
Peak spill	0 B

clickbench_partitioned — branch

Metric	Value
Wall time	100.0s
Peak memory	18.1 GiB
Avg memory	5.7 GiB
CPU user	946.0s
CPU sys	93.3s
Peak spill	0 B

File an issue against this benchmark runner

jayzhan211 · 2026-10-03T14:10:04Z

run benchmarks clickbench_partitioned tpch10
changed:
env:
DATAFUSION_EXECUTION_HASH_AGGREGATE_BUCKET_THRESHOLD: "262144"

adriangbot · 2026-10-03T14:12:31Z

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c5969954322-3048-glb46 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing agg-buckets-final-stage (7b7c506) to 22a0fdb (merge-base) diff

Run configuration

run benchmark tpch10
changed:
  env:
    DATAFUSION_EXECUTION_HASH_AGGREGATE_BUCKET_THRESHOLD: "262144"

Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-10-03T14:13:14Z

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c5969954322-3047-4wqgg 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing agg-buckets-final-stage (7b7c506) to 22a0fdb (merge-base) diff

Run configuration

run benchmark clickbench_partitioned
changed:
  env:
    DATAFUSION_EXECUTION_HASH_AGGREGATE_BUCKET_THRESHOLD: "262144"

Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-10-03T14:22:25Z

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

Comparing agg-buckets-final-stage (7b7c506) to 22a0fdb (merge-base) diff

Run configuration

run benchmark tpch10
changed:
  env:
    DATAFUSION_EXECUTION_HASH_AGGREGATE_BUCKET_THRESHOLD: "262144"

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

Comparing HEAD and agg-buckets-final-stage
--------------------
Benchmark tpch_sf10.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃      HEAD ┃ agg-buckets-final-stage ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1  │ 332.56 ms │               328.98 ms │     no change │
│ QQuery 2  │  91.24 ms │                92.03 ms │     no change │
│ QQuery 3  │ 217.42 ms │               218.29 ms │     no change │
│ QQuery 4  │ 118.56 ms │               117.54 ms │     no change │
│ QQuery 5  │ 349.10 ms │               344.63 ms │     no change │
│ QQuery 6  │ 100.56 ms │               101.46 ms │     no change │
│ QQuery 7  │ 445.97 ms │               451.66 ms │     no change │
│ QQuery 8  │ 359.59 ms │               359.08 ms │     no change │
│ QQuery 9  │ 516.93 ms │               511.76 ms │     no change │
│ QQuery 10 │ 287.14 ms │               293.06 ms │     no change │
│ QQuery 11 │  61.91 ms │                61.35 ms │     no change │
│ QQuery 12 │ 121.74 ms │               119.65 ms │     no change │
│ QQuery 13 │ 309.18 ms │               302.16 ms │     no change │
│ QQuery 14 │ 169.84 ms │               170.49 ms │     no change │
│ QQuery 15 │ 301.13 ms │               303.24 ms │     no change │
│ QQuery 16 │  65.75 ms │                64.02 ms │     no change │
│ QQuery 17 │ 586.83 ms │               421.03 ms │ +1.39x faster │
│ QQuery 18 │ 716.39 ms │               675.79 ms │ +1.06x faster │
│ QQuery 19 │ 235.42 ms │               233.03 ms │     no change │
│ QQuery 20 │ 267.80 ms │               237.57 ms │ +1.13x faster │
│ QQuery 21 │ 666.34 ms │               667.85 ms │     no change │
│ QQuery 22 │  59.68 ms │                62.37 ms │     no change │
└───────────┴───────────┴─────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary                      ┃           ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD)                      │ 6381.07ms │
│ Total Time (agg-buckets-final-stage)   │ 6137.02ms │
│ Average Time (HEAD)                    │  290.05ms │
│ Average Time (agg-buckets-final-stage) │  278.96ms │
│ Queries Faster                         │         3 │
│ Queries Slower                         │         0 │
│ Queries with No Change                 │        19 │
│ Queries with Failure                   │         0 │
└────────────────────────────────────────┴───────────┘

Distribution per query (min / mean ±stddev / max):

Comparing HEAD and agg-buckets-final-stage
--------------------
Benchmark tpch_sf10.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃                               HEAD ┃            agg-buckets-final-stage ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1  │  332.56 / 334.84 ±1.83 / 337.61 ms │  328.98 / 331.48 ±1.41 / 333.22 ms │     no change │
│ QQuery 2  │     91.24 / 93.34 ±2.03 / 96.87 ms │     92.03 / 93.92 ±1.02 / 94.92 ms │     no change │
│ QQuery 3  │  217.42 / 220.83 ±3.11 / 225.84 ms │  218.29 / 223.59 ±4.00 / 228.89 ms │     no change │
│ QQuery 4  │  118.56 / 120.13 ±0.94 / 121.31 ms │  117.54 / 118.87 ±1.32 / 121.25 ms │     no change │
│ QQuery 5  │  349.10 / 358.11 ±7.89 / 371.65 ms │  344.63 / 350.77 ±5.41 / 360.37 ms │     no change │
│ QQuery 6  │  100.56 / 101.84 ±1.31 / 104.13 ms │  101.46 / 104.34 ±2.95 / 109.90 ms │     no change │
│ QQuery 7  │  445.97 / 455.78 ±7.20 / 464.86 ms │  451.66 / 465.14 ±8.62 / 474.53 ms │     no change │
│ QQuery 8  │  359.59 / 364.72 ±4.40 / 372.43 ms │  359.08 / 362.87 ±4.04 / 369.84 ms │     no change │
│ QQuery 9  │  516.93 / 523.33 ±6.27 / 533.50 ms │  511.76 / 524.05 ±8.31 / 537.58 ms │     no change │
│ QQuery 10 │ 287.14 / 302.24 ±13.39 / 319.37 ms │  293.06 / 302.75 ±7.46 / 313.60 ms │     no change │
│ QQuery 11 │     61.91 / 62.89 ±0.66 / 63.91 ms │     61.35 / 63.62 ±1.85 / 66.94 ms │     no change │
│ QQuery 12 │  121.74 / 128.17 ±5.72 / 136.55 ms │  119.65 / 127.52 ±6.85 / 138.43 ms │     no change │
│ QQuery 13 │  309.18 / 317.38 ±6.89 / 329.77 ms │  302.16 / 310.84 ±6.50 / 321.10 ms │     no change │
│ QQuery 14 │  169.84 / 176.58 ±5.31 / 184.35 ms │  170.49 / 176.08 ±4.80 / 184.09 ms │     no change │
│ QQuery 15 │  301.13 / 304.13 ±2.79 / 307.83 ms │  303.24 / 307.77 ±2.78 / 311.83 ms │     no change │
│ QQuery 16 │     65.75 / 66.72 ±1.05 / 68.06 ms │     64.02 / 66.18 ±1.29 / 67.83 ms │     no change │
│ QQuery 17 │  586.83 / 601.67 ±9.32 / 610.81 ms │ 421.03 / 441.90 ±20.53 / 476.00 ms │ +1.36x faster │
│ QQuery 18 │ 716.39 / 741.09 ±16.14 / 765.49 ms │ 675.79 / 692.68 ±16.77 / 723.53 ms │ +1.07x faster │
│ QQuery 19 │ 235.42 / 251.22 ±14.95 / 272.82 ms │ 233.03 / 244.09 ±13.41 / 265.28 ms │     no change │
│ QQuery 20 │ 267.80 / 279.25 ±10.11 / 293.63 ms │  237.57 / 245.84 ±6.32 / 256.51 ms │ +1.14x faster │
│ QQuery 21 │  666.34 / 671.00 ±5.43 / 680.97 ms │  667.85 / 675.06 ±5.75 / 682.28 ms │     no change │
│ QQuery 22 │     59.68 / 61.40 ±1.06 / 62.75 ms │     62.37 / 66.84 ±5.97 / 78.47 ms │  1.09x slower │
└───────────┴────────────────────────────────────┴────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary                      ┃           ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD)                      │ 6536.66ms │
│ Total Time (agg-buckets-final-stage)   │ 6296.19ms │
│ Average Time (HEAD)                    │  297.12ms │
│ Average Time (agg-buckets-final-stage) │  286.19ms │
│ Queries Faster                         │         3 │
│ Queries Slower                         │         1 │
│ Queries with No Change                 │        18 │
│ Queries with Failure                   │         0 │
└────────────────────────────────────────┴───────────┘

Resource Usage

tpch10 — base (merge-base)

Metric	Value
Wall time	35.0s
Peak memory	4.9 GiB
Avg memory	1.5 GiB
CPU user	332.5s
CPU sys	19.7s
Peak spill	0 B

tpch10 — branch

Metric	Value
Wall time	35.0s
Peak memory	5.1 GiB
Avg memory	1.5 GiB
CPU user	318.4s
CPU sys	19.6s
Peak spill	0 B

File an issue against this benchmark runner

adriangbot · 2026-10-03T14:26:06Z

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

Comparing agg-buckets-final-stage (7b7c506) to 22a0fdb (merge-base) diff

Run configuration

run benchmark clickbench_partitioned
changed:
  env:
    DATAFUSION_EXECUTION_HASH_AGGREGATE_BUCKET_THRESHOLD: "262144"

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

Comparing HEAD and agg-buckets-final-stage
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃       HEAD ┃ agg-buckets-final-stage ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0  │    1.25 ms │                 1.25 ms │     no change │
│ QQuery 1  │   12.02 ms │                11.94 ms │     no change │
│ QQuery 2  │   36.91 ms │                36.61 ms │     no change │
│ QQuery 3  │   32.00 ms │                31.56 ms │     no change │
│ QQuery 4  │  240.79 ms │               188.94 ms │ +1.27x faster │
│ QQuery 5  │  277.63 ms │               277.07 ms │     no change │
│ QQuery 6  │    1.29 ms │                 1.28 ms │     no change │
│ QQuery 7  │   12.73 ms │                12.88 ms │     no change │
│ QQuery 8  │  339.92 ms │               295.97 ms │ +1.15x faster │
│ QQuery 9  │  474.51 ms │               479.08 ms │     no change │
│ QQuery 10 │   65.25 ms │                65.69 ms │     no change │
│ QQuery 11 │   76.84 ms │                76.19 ms │     no change │
│ QQuery 12 │  269.11 ms │               271.28 ms │     no change │
│ QQuery 13 │  360.14 ms │               365.59 ms │     no change │
│ QQuery 14 │  282.46 ms │               285.29 ms │     no change │
│ QQuery 15 │  289.62 ms │               215.93 ms │ +1.34x faster │
│ QQuery 16 │  633.80 ms │               634.11 ms │     no change │
│ QQuery 17 │  632.48 ms │               638.86 ms │     no change │
│ QQuery 18 │ 1290.93 ms │              1329.03 ms │     no change │
│ QQuery 19 │   27.96 ms │                27.54 ms │     no change │
│ QQuery 20 │  521.61 ms │               526.21 ms │     no change │
│ QQuery 21 │  519.17 ms │               516.81 ms │     no change │
│ QQuery 22 │ 1045.61 ms │              1026.03 ms │     no change │
│ QQuery 23 │ 3112.51 ms │              3090.61 ms │     no change │
│ QQuery 24 │   40.13 ms │                40.91 ms │     no change │
│ QQuery 25 │  106.12 ms │               106.56 ms │     no change │
│ QQuery 26 │   40.92 ms │                41.35 ms │     no change │
│ QQuery 27 │  511.55 ms │               519.03 ms │     no change │
│ QQuery 28 │ 2900.73 ms │              2898.56 ms │     no change │
│ QQuery 29 │   39.96 ms │                39.49 ms │     no change │
│ QQuery 30 │  309.29 ms │               295.78 ms │     no change │
│ QQuery 31 │  278.81 ms │               254.46 ms │ +1.10x faster │
│ QQuery 32 │ 1076.79 ms │               746.25 ms │ +1.44x faster │
│ QQuery 33 │ 1600.54 ms │              1569.52 ms │     no change │
│ QQuery 34 │ 1620.90 ms │              1601.89 ms │     no change │
│ QQuery 35 │  309.86 ms │               231.45 ms │ +1.34x faster │
│ QQuery 36 │   66.55 ms │                71.17 ms │  1.07x slower │
│ QQuery 37 │   35.65 ms │                35.29 ms │     no change │
│ QQuery 38 │   43.19 ms │                40.54 ms │ +1.07x faster │
│ QQuery 39 │  150.02 ms │               146.98 ms │     no change │
│ QQuery 40 │   14.42 ms │                14.51 ms │     no change │
│ QQuery 41 │   13.97 ms │                13.82 ms │     no change │
│ QQuery 42 │   13.73 ms │                13.44 ms │     no change │
└───────────┴────────────┴─────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                      ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                      │ 19729.69ms │
│ Total Time (agg-buckets-final-stage)   │ 19086.74ms │
│ Average Time (HEAD)                    │   458.83ms │
│ Average Time (agg-buckets-final-stage) │   443.88ms │
│ Queries Faster                         │          7 │
│ Queries Slower                         │          1 │
│ Queries with No Change                 │         35 │
│ Queries with Failure                   │          0 │
└────────────────────────────────────────┴────────────┘

Distribution per query (min / mean ±stddev / max):

Comparing HEAD and agg-buckets-final-stage
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃                                   HEAD ┃                agg-buckets-final-stage ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0  │           1.25 / 4.10 ±5.57 / 15.24 ms │           1.25 / 4.16 ±5.69 / 15.53 ms │     no change │
│ QQuery 1  │         12.02 / 12.15 ±0.08 / 12.25 ms │         11.94 / 12.20 ±0.27 / 12.68 ms │     no change │
│ QQuery 2  │         36.91 / 37.26 ±0.30 / 37.69 ms │         36.61 / 37.62 ±1.00 / 39.47 ms │     no change │
│ QQuery 3  │         32.00 / 33.01 ±0.76 / 34.06 ms │         31.56 / 31.68 ±0.14 / 31.94 ms │     no change │
│ QQuery 4  │      240.79 / 246.65 ±4.51 / 254.35 ms │      188.94 / 192.04 ±2.27 / 195.07 ms │ +1.28x faster │
│ QQuery 5  │      277.63 / 281.27 ±2.59 / 284.05 ms │      277.07 / 279.83 ±2.20 / 283.37 ms │     no change │
│ QQuery 6  │            1.29 / 1.44 ±0.23 / 1.88 ms │            1.28 / 1.43 ±0.25 / 1.92 ms │     no change │
│ QQuery 7  │         12.73 / 13.94 ±1.55 / 16.99 ms │         12.88 / 13.12 ±0.19 / 13.46 ms │ +1.06x faster │
│ QQuery 8  │      339.92 / 342.76 ±3.29 / 349.15 ms │      295.97 / 302.41 ±5.43 / 311.83 ms │ +1.13x faster │
│ QQuery 9  │      474.51 / 481.78 ±4.91 / 488.81 ms │     479.08 / 497.91 ±16.08 / 527.50 ms │     no change │
│ QQuery 10 │         65.25 / 70.49 ±7.01 / 84.34 ms │         65.69 / 66.68 ±1.45 / 69.54 ms │ +1.06x faster │
│ QQuery 11 │         76.84 / 77.80 ±0.80 / 78.99 ms │         76.19 / 77.51 ±0.93 / 79.00 ms │     no change │
│ QQuery 12 │      269.11 / 274.31 ±3.45 / 279.72 ms │      271.28 / 276.53 ±4.05 / 282.72 ms │     no change │
│ QQuery 13 │      360.14 / 371.97 ±8.29 / 383.37 ms │     365.59 / 374.94 ±11.34 / 397.19 ms │     no change │
│ QQuery 14 │      282.46 / 289.98 ±5.94 / 299.03 ms │      285.29 / 290.38 ±4.19 / 297.67 ms │     no change │
│ QQuery 15 │     289.62 / 305.20 ±20.85 / 346.08 ms │     215.93 / 227.72 ±13.11 / 253.25 ms │ +1.34x faster │
│ QQuery 16 │      633.80 / 643.59 ±7.97 / 653.26 ms │      634.11 / 642.66 ±7.00 / 653.36 ms │     no change │
│ QQuery 17 │     632.48 / 645.24 ±10.83 / 663.77 ms │     638.86 / 656.57 ±13.21 / 675.65 ms │     no change │
│ QQuery 18 │  1290.93 / 1314.29 ±25.61 / 1358.55 ms │  1329.03 / 1345.34 ±22.82 / 1389.26 ms │     no change │
│ QQuery 19 │         27.96 / 31.69 ±6.90 / 45.49 ms │         27.54 / 30.30 ±4.87 / 40.03 ms │     no change │
│ QQuery 20 │      521.61 / 530.76 ±9.00 / 544.59 ms │     526.21 / 539.34 ±15.28 / 567.63 ms │     no change │
│ QQuery 21 │      519.17 / 522.41 ±2.92 / 526.16 ms │     516.81 / 533.32 ±11.53 / 551.25 ms │     no change │
│ QQuery 22 │  1045.61 / 1064.52 ±23.64 / 1109.73 ms │  1026.03 / 1043.61 ±24.42 / 1091.85 ms │     no change │
│ QQuery 23 │ 3112.51 / 3337.75 ±129.82 / 3514.79 ms │ 3090.61 / 3301.25 ±164.52 / 3588.27 ms │     no change │
│ QQuery 24 │     40.13 / 191.29 ±299.00 / 789.29 ms │       40.91 / 66.98 ±32.00 / 114.78 ms │ +2.86x faster │
│ QQuery 25 │      106.12 / 108.02 ±1.48 / 110.63 ms │    106.56 / 220.05 ±153.49 / 498.52 ms │  2.04x slower │
│ QQuery 26 │         40.92 / 42.46 ±1.02 / 43.93 ms │         41.35 / 42.31 ±0.98 / 43.96 ms │     no change │
│ QQuery 27 │      511.55 / 527.04 ±9.95 / 541.71 ms │      519.03 / 526.54 ±6.61 / 534.81 ms │     no change │
│ QQuery 28 │  2900.73 / 2965.80 ±47.48 / 3036.68 ms │  2898.56 / 2954.65 ±35.57 / 3008.03 ms │     no change │
│ QQuery 29 │       39.96 / 62.20 ±34.28 / 129.15 ms │       39.49 / 65.24 ±31.99 / 116.07 ms │     no change │
│ QQuery 30 │      309.29 / 315.85 ±5.98 / 323.02 ms │     295.78 / 310.58 ±12.58 / 327.31 ms │     no change │
│ QQuery 31 │     278.81 / 310.03 ±25.43 / 341.20 ms │      254.46 / 264.88 ±5.75 / 271.30 ms │ +1.17x faster │
│ QQuery 32 │  1076.79 / 1144.97 ±67.85 / 1273.60 ms │     746.25 / 766.52 ±15.07 / 789.25 ms │ +1.49x faster │
│ QQuery 33 │  1600.54 / 1663.00 ±37.53 / 1716.17 ms │  1569.52 / 1635.60 ±43.77 / 1682.80 ms │     no change │
│ QQuery 34 │  1620.90 / 1663.80 ±25.06 / 1686.41 ms │  1601.89 / 1660.88 ±48.76 / 1729.06 ms │     no change │
│ QQuery 35 │     309.86 / 351.42 ±47.83 / 422.48 ms │    231.45 / 360.34 ±244.44 / 849.12 ms │     no change │
│ QQuery 36 │         66.55 / 81.14 ±8.25 / 91.59 ms │         71.17 / 74.94 ±4.20 / 81.64 ms │ +1.08x faster │
│ QQuery 37 │         35.65 / 38.65 ±4.08 / 46.55 ms │         35.29 / 36.06 ±0.61 / 36.70 ms │ +1.07x faster │
│ QQuery 38 │         43.19 / 48.42 ±5.10 / 56.32 ms │         40.54 / 41.84 ±1.28 / 43.91 ms │ +1.16x faster │
│ QQuery 39 │      150.02 / 154.33 ±4.73 / 163.15 ms │      146.98 / 153.69 ±6.17 / 163.09 ms │     no change │
│ QQuery 40 │         14.42 / 16.87 ±3.89 / 24.59 ms │         14.51 / 15.01 ±0.47 / 15.88 ms │ +1.12x faster │
│ QQuery 41 │         13.97 / 14.11 ±0.14 / 14.37 ms │         13.82 / 13.88 ±0.06 / 13.98 ms │     no change │
│ QQuery 42 │         13.73 / 16.10 ±3.22 / 22.23 ms │         13.44 / 13.63 ±0.15 / 13.86 ms │ +1.18x faster │
└───────────┴────────────────────────────────────────┴────────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                      ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                      │ 20649.87ms │
│ Total Time (agg-buckets-final-stage)   │ 20002.18ms │
│ Average Time (HEAD)                    │   480.23ms │
│ Average Time (agg-buckets-final-stage) │   465.17ms │
│ Queries Faster                         │         13 │
│ Queries Slower                         │          1 │
│ Queries with No Change                 │         29 │
│ Queries with Failure                   │          0 │
└────────────────────────────────────────┴────────────┘

Resource Usage

clickbench_partitioned — base (merge-base)

Metric	Value
Wall time	105.0s
Peak memory	17.5 GiB
Avg memory	6.3 GiB
CPU user	1012.8s
CPU sys	103.6s
Peak spill	0 B

clickbench_partitioned — branch

Metric	Value
Wall time	105.0s
Peak memory	17.1 GiB
Avg memory	5.6 GiB
CPU user	978.4s
CPU sys	95.1s
Peak spill	0 B

File an issue against this benchmark runner

jayzhan211 · 2026-10-03T14:40:35Z

On the Q25 "2.04x slower" in the latest run: I don't think it comes from this PR.

Q25 does not aggregate. It is SELECT "SearchPhrase" ... WHERE "SearchPhrase" <> '' ORDER BY "SearchPhrase" LIMIT 10, planned as DataSourceExec → FilterExec → SortExec TopK → SortPreservingMergeExec with or without the option. hash_aggregate_bucket_threshold is only read inside the hash aggregation streams, so Q25 runs the same code on both sides.
The fastest iteration is unchanged: 106.12 ms on main, 106.56 ms on the branch. The 2.04x compares means, and the branch mean is pulled up by slow iterations (max 498 ms).
Main shows the same stalls, in the same place. In all three runs, one of the cheap queries right after Q23 (SELECT *, the most memory-heavy query) has a slow iteration, on either side:

run	main (fastest → slowest of 5)	branch (fastest → slowest of 5)
1	Q24 43 → 177 ms	Q24 42 → 646 ms
2	Q24 41 → 111 ms, Q26 42 → 126 ms	Q25 104 → 663 ms
3	Q24 40 → 789 ms	Q25 107 → 499 ms

In this run, main's Q24 stall is larger than the branch's Q25 one. Which query absorbs the stall changes between runs; what stays the same is that it follows Q23. Q19 and Q29 get similar outliers on both sides.

Locally it does not reproduce. I ran the whole suite in one process, as the runner does, alternating the option off and on: 15 iterations of Q25 per side, no outlier (slowest 98 ms off, 82 ms on).

To separate the option from the binary, I can ask for a run where both sides use this commit and only the threshold differs (0 vs 262144).

adriangbot · 2026-10-07T16:31:11Z

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

Comparing agg-buckets-final-stage (88d36d4) to 7b1e55d diff

Run configuration

run benchmark clickbench_partitioned
env:
  DATAFUSION_EXECUTION_HASH_AGGREGATE_BUCKET_THRESHOLD: "262144"
baseline:
  ref: "7b1e55d"

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

Comparing HEAD and agg-buckets-final-stage
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃       HEAD ┃ agg-buckets-final-stage ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0  │    1.19 ms │                 1.22 ms │     no change │
│ QQuery 1  │   11.42 ms │                11.68 ms │     no change │
│ QQuery 2  │   36.81 ms │                37.30 ms │     no change │
│ QQuery 3  │   31.33 ms │                32.03 ms │     no change │
│ QQuery 4  │  235.69 ms │               191.30 ms │ +1.23x faster │
│ QQuery 5  │  270.52 ms │               274.13 ms │     no change │
│ QQuery 6  │    1.24 ms │                 1.28 ms │     no change │
│ QQuery 7  │   12.69 ms │                12.64 ms │     no change │
│ QQuery 8  │  333.92 ms │               293.27 ms │ +1.14x faster │
│ QQuery 9  │  467.02 ms │               472.59 ms │     no change │
│ QQuery 10 │   64.77 ms │                65.35 ms │     no change │
│ QQuery 11 │   75.34 ms │                75.56 ms │     no change │
│ QQuery 12 │  261.88 ms │               261.01 ms │     no change │
│ QQuery 13 │  359.41 ms │               358.58 ms │     no change │
│ QQuery 14 │  292.39 ms │               276.26 ms │ +1.06x faster │
│ QQuery 15 │  328.88 ms │               219.03 ms │ +1.50x faster │
│ QQuery 16 │  665.95 ms │               632.34 ms │ +1.05x faster │
│ QQuery 17 │  671.33 ms │               628.45 ms │ +1.07x faster │
│ QQuery 18 │ 1377.09 ms │              1314.04 ms │     no change │
│ QQuery 19 │   28.78 ms │                29.52 ms │     no change │
│ QQuery 20 │  519.93 ms │               534.96 ms │     no change │
│ QQuery 21 │  512.78 ms │               521.07 ms │     no change │
│ QQuery 22 │  994.11 ms │              1061.78 ms │  1.07x slower │
│ QQuery 23 │ 3116.39 ms │              3241.72 ms │     no change │
│ QQuery 24 │   40.19 ms │                40.14 ms │     no change │
│ QQuery 25 │  104.55 ms │               105.25 ms │     no change │
│ QQuery 26 │   40.95 ms │                40.67 ms │     no change │
│ QQuery 27 │  508.33 ms │               518.05 ms │     no change │
│ QQuery 28 │ 2891.96 ms │              2898.08 ms │     no change │
│ QQuery 29 │   39.95 ms │                39.07 ms │     no change │
│ QQuery 30 │  320.86 ms │               287.44 ms │ +1.12x faster │
│ QQuery 31 │  305.65 ms │               248.27 ms │ +1.23x faster │
│ QQuery 32 │ 1234.31 ms │               734.57 ms │ +1.68x faster │
│ QQuery 33 │ 1706.43 ms │              1722.76 ms │     no change │
│ QQuery 34 │ 1677.40 ms │              1740.13 ms │     no change │
│ QQuery 35 │  323.74 ms │               240.92 ms │ +1.34x faster │
│ QQuery 36 │   70.78 ms │                68.05 ms │     no change │
│ QQuery 37 │   36.36 ms │                37.62 ms │     no change │
│ QQuery 38 │   43.61 ms │                43.97 ms │     no change │
│ QQuery 39 │  137.40 ms │               161.01 ms │  1.17x slower │
│ QQuery 40 │   13.77 ms │                16.52 ms │  1.20x slower │
│ QQuery 41 │   13.65 ms │                15.18 ms │  1.11x slower │
│ QQuery 42 │   12.75 ms │                14.69 ms │  1.15x slower │
└───────────┴────────────┴─────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                      ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                      │ 20193.49ms │
│ Total Time (agg-buckets-final-stage)   │ 19519.48ms │
│ Average Time (HEAD)                    │   469.62ms │
│ Average Time (agg-buckets-final-stage) │   453.94ms │
│ Queries Faster                         │         10 │
│ Queries Slower                         │          5 │
│ Queries with No Change                 │         28 │
│ Queries with Failure                   │          0 │
└────────────────────────────────────────┴────────────┘

Distribution per query (min / mean ±stddev / max):

Comparing HEAD and agg-buckets-final-stage
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃                                   HEAD ┃                agg-buckets-final-stage ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0  │           1.19 / 3.99 ±5.50 / 14.99 ms │           1.22 / 4.01 ±5.49 / 14.98 ms │     no change │
│ QQuery 1  │         11.42 / 11.67 ±0.20 / 11.92 ms │         11.68 / 11.84 ±0.24 / 12.32 ms │     no change │
│ QQuery 2  │         36.81 / 37.20 ±0.34 / 37.71 ms │         37.30 / 37.62 ±0.18 / 37.87 ms │     no change │
│ QQuery 3  │         31.33 / 32.05 ±0.86 / 33.67 ms │         32.03 / 33.25 ±0.94 / 34.77 ms │     no change │
│ QQuery 4  │     235.69 / 248.16 ±14.75 / 268.12 ms │      191.30 / 196.17 ±4.43 / 202.11 ms │ +1.26x faster │
│ QQuery 5  │      270.52 / 275.42 ±4.15 / 282.60 ms │      274.13 / 285.06 ±6.75 / 293.92 ms │     no change │
│ QQuery 6  │            1.24 / 1.39 ±0.24 / 1.86 ms │            1.28 / 1.43 ±0.24 / 1.91 ms │     no change │
│ QQuery 7  │         12.69 / 12.76 ±0.07 / 12.90 ms │         12.64 / 12.86 ±0.14 / 13.05 ms │     no change │
│ QQuery 8  │      333.92 / 344.48 ±7.28 / 353.48 ms │      293.27 / 301.94 ±7.29 / 314.29 ms │ +1.14x faster │
│ QQuery 9  │     467.02 / 482.00 ±16.68 / 512.99 ms │     472.59 / 485.59 ±15.06 / 514.85 ms │     no change │
│ QQuery 10 │         64.77 / 67.23 ±2.61 / 71.30 ms │         65.35 / 66.20 ±0.82 / 67.56 ms │     no change │
│ QQuery 11 │         75.34 / 77.97 ±3.21 / 84.21 ms │         75.56 / 76.36 ±0.91 / 78.09 ms │     no change │
│ QQuery 12 │      261.88 / 270.25 ±7.02 / 280.92 ms │      261.01 / 272.68 ±9.21 / 284.44 ms │     no change │
│ QQuery 13 │     359.41 / 376.21 ±15.13 / 402.43 ms │     358.58 / 378.82 ±17.73 / 405.84 ms │     no change │
│ QQuery 14 │     292.39 / 309.54 ±14.12 / 333.46 ms │     276.26 / 291.73 ±10.73 / 307.66 ms │ +1.06x faster │
│ QQuery 15 │      328.88 / 335.82 ±6.62 / 346.05 ms │      219.03 / 224.59 ±8.57 / 241.57 ms │ +1.50x faster │
│ QQuery 16 │     665.95 / 683.04 ±14.29 / 703.30 ms │      632.34 / 637.34 ±5.15 / 646.68 ms │ +1.07x faster │
│ QQuery 17 │     671.33 / 701.49 ±17.93 / 717.96 ms │      628.45 / 638.29 ±6.61 / 646.38 ms │ +1.10x faster │
│ QQuery 18 │  1377.09 / 1424.18 ±33.72 / 1460.54 ms │  1314.04 / 1403.82 ±57.21 / 1477.32 ms │     no change │
│ QQuery 19 │        28.78 / 35.25 ±12.61 / 60.46 ms │        29.52 / 41.68 ±15.69 / 69.25 ms │  1.18x slower │
│ QQuery 20 │      519.93 / 530.68 ±9.00 / 542.89 ms │     534.96 / 552.84 ±21.19 / 592.59 ms │     no change │
│ QQuery 21 │     512.78 / 529.54 ±13.17 / 550.71 ms │      521.07 / 533.48 ±7.30 / 541.89 ms │     no change │
│ QQuery 22 │    994.11 / 1002.87 ±8.84 / 1019.51 ms │  1061.78 / 1107.63 ±38.69 / 1148.96 ms │  1.10x slower │
│ QQuery 23 │ 3116.39 / 3331.13 ±162.55 / 3618.74 ms │ 3241.72 / 3362.77 ±190.48 / 3739.20 ms │     no change │
│ QQuery 24 │         40.19 / 40.54 ±0.29 / 40.97 ms │     40.14 / 212.02 ±343.08 / 898.18 ms │  5.23x slower │
│ QQuery 25 │    104.55 / 234.95 ±250.84 / 736.59 ms │      105.25 / 107.37 ±2.14 / 111.47 ms │ +2.19x faster │
│ QQuery 26 │         40.95 / 41.44 ±0.39 / 42.08 ms │         40.67 / 42.00 ±1.46 / 44.63 ms │     no change │
│ QQuery 27 │     508.33 / 521.18 ±12.55 / 541.73 ms │      518.05 / 523.31 ±4.21 / 528.42 ms │     no change │
│ QQuery 28 │  2891.96 / 2963.72 ±56.00 / 3060.75 ms │  2898.08 / 2936.02 ±29.04 / 2974.67 ms │     no change │
│ QQuery 29 │       39.95 / 73.70 ±34.55 / 129.53 ms │       39.07 / 81.40 ±48.13 / 162.82 ms │  1.10x slower │
│ QQuery 30 │     320.86 / 350.82 ±16.18 / 366.44 ms │     287.44 / 308.82 ±25.35 / 357.74 ms │ +1.14x faster │
│ QQuery 31 │     305.65 / 321.70 ±21.53 / 364.08 ms │      248.27 / 256.53 ±6.62 / 263.03 ms │ +1.25x faster │
│ QQuery 32 │  1234.31 / 1325.52 ±60.20 / 1420.24 ms │     734.57 / 812.05 ±86.13 / 956.60 ms │ +1.63x faster │
│ QQuery 33 │  1706.43 / 1756.06 ±42.36 / 1833.10 ms │  1722.76 / 1779.07 ±47.72 / 1853.14 ms │     no change │
│ QQuery 34 │  1677.40 / 1700.94 ±25.03 / 1747.60 ms │  1740.13 / 1780.27 ±29.18 / 1812.71 ms │     no change │
│ QQuery 35 │     323.74 / 390.18 ±67.79 / 486.89 ms │     240.92 / 304.97 ±52.18 / 358.23 ms │ +1.28x faster │
│ QQuery 36 │         70.78 / 78.89 ±8.94 / 89.98 ms │         68.05 / 73.60 ±6.07 / 84.83 ms │ +1.07x faster │
│ QQuery 37 │         36.36 / 42.13 ±7.39 / 55.28 ms │         37.62 / 41.42 ±4.30 / 48.56 ms │     no change │
│ QQuery 38 │         43.61 / 48.60 ±3.96 / 54.36 ms │         43.97 / 47.86 ±3.11 / 53.32 ms │     no change │
│ QQuery 39 │     137.40 / 155.52 ±11.69 / 173.25 ms │      161.01 / 175.09 ±9.64 / 190.84 ms │  1.13x slower │
│ QQuery 40 │         13.77 / 14.13 ±0.25 / 14.42 ms │         16.52 / 19.31 ±5.19 / 29.68 ms │  1.37x slower │
│ QQuery 41 │         13.65 / 17.99 ±5.17 / 26.69 ms │         15.18 / 16.22 ±0.74 / 17.38 ms │ +1.11x faster │
│ QQuery 42 │         12.75 / 14.19 ±2.18 / 18.54 ms │         14.69 / 18.04 ±6.11 / 30.25 ms │  1.27x slower │
└───────────┴────────────────────────────────────────┴────────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                      ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                      │ 21216.53ms │
│ Total Time (agg-buckets-final-stage)   │ 20493.39ms │
│ Average Time (HEAD)                    │   493.41ms │
│ Average Time (agg-buckets-final-stage) │   476.59ms │
│ Queries Faster                         │         13 │
│ Queries Slower                         │          7 │
│ Queries with No Change                 │         23 │
│ Queries with Failure                   │          0 │
└────────────────────────────────────────┴────────────┘

Resource Usage

clickbench_partitioned — base (merge-base)

Metric	Value
Wall time	110.0s
Peak memory	17.1 GiB
Avg memory	6.1 GiB
CPU user	1044.3s
CPU sys	103.6s
Peak spill	0 B

clickbench_partitioned — branch

Metric	Value
Wall time	105.0s
Peak memory	18.2 GiB
Avg memory	6.1 GiB
CPU user	997.3s
CPU sys	97.1s
Peak spill	0 B

File an issue against this benchmark runner

Rachelint · 2026-10-07T19:00:48Z

@Rachelint,

only aggregations whose group keys and aggregate state are all fixed-width (primitive or boolean) use buckets or flush their partial table. String, binary and nested state keep a single table. BucketedAggregation::supports_state replaces the old check, which excluded only nested state.

what do you think on this direction. If looks good, I will start PRs for review and landing. If not, I will find other direction to improve aggregation

I agree that we should start by applying the bucket approach only to fixed-width types. Despite trying several ideas over the past few days, I haven’t yet found a way to reduce the routing overhead enough to make the approach broadly beneficial...

I also have a few thoughts we could explore as follow-ups:

Perhaps proactive Partial flushing could have its own issue. It looks promising enough based on my experiments in [#26116](perf: flush partial batches with fixed group capacity (experiment) #26116), and I’d be happy to help investigate further.
Would it make sense to leave compaction for a follow-up? It introduces some complexity, and I’ve seen cases where it hurts performance while offering limited benefit for very high-cardinality workloads like Q32. Starting just with the bucket approach only might make the initial implementation easier to evaluate.
I’m also curious how this would interact with [#24704]([EPIC] Use blocked / chunked memory management in hash aggregation #24704). If proactive flushing keeps Partial output batches small, and Final aggregation materializes its output in 64 or more buckets, could that address much of the peak-memory concern? I may be missing something here—what remaining scenarios would [EPIC] Use blocked / chunked memory management in hash aggregation #24704 help with?

jayzhan211 · 2026-10-24T06:59:52Z

@Rachelint I'll take the Partial flush out of this PR and leave it to #26124 / #26116, so this PR only buckets the Final stage.

On compaction, I'd rather keep it. Without it, a bucket holds every row it receives until the end. When the Partial stage passes rows through mostly unreduced (skipped, or few rows per group), Final memory then grows with the input rows instead of the groups. Before I added compaction, peak RSS with bucketing on was 2.3x of bucketing off on 20M random integer keys and 2.26x on TPC-H SF10 GROUP BY l_partkey, l_suppkey. With compaction it was 1.13x and 1.40x. It does cost something on Q32 (0.34x → 0.42x of the time with bucketing off, still faster). But when the table being split kept more than 80% of its rows as groups, as in Q32, buckets already wait four times longer before their first compaction. Do your measurements on #25959, which turns compaction off, show which queries it slows down? I'd rather tune the wait than drop it.

I answered the question about #24704 on the issue.

The heavy ClickBench GROUP BY statements run one plan shape: a partial aggregate that skips, a 32-way hash repartition, and a FinalPartitioned aggregate whose group tables are far larger than any cache. On q33 the final never hits (99,997,493 groups over as many rows), 35.5% of server cycles are stalled on memory, and the cache-miss profile puts 0.40 of the misses in the final's group-table probe and 0.36 in the exchange's scatter. DataFusion 55.2 has no hook to give the final a different group table, and the upstream bucketed aggregate (apache/datafusion#25724) is a draft against a later release. Record the decision to add a Ravel ExecutionPlan, behind a SqlConfig field that is off by default, that aggregates one hash bucket at a time in a cache-sized table once a partition's final passes 65,536 groups, and cuts each completed bucket to its exact top k when a SortExec with a fetch sits above it. The exec charges its retained input to its own consumer before retention, never spills, and refuses string, float, boolean, Decimal128 and Duration keys. It is enabled only after a pre-registered paired A/B on the reference machine, with the bands in the ADR. Fused routing, which would take back the router copy the exec adds, is a later amendment measured against this decision. The memory gate rests on a new factor-free peakAskedIntermediateBytes query stat, because the ledger's peak_intermediate_bytes charges the stock consumers at the reserve-ahead factor and the new one at 1.0, and process VmHWM is a lifetime figure once the allocator keeps its arenas. Refs: NOFireAI#3565 Signed-off-by: Panagiotis Moustafellos <[email protected]>

rluvaton · 2026-11-08T13:27:05Z

Just a suggestion since I fell for it. don't benchmark on macbooks since they have very different mem and cpu behavior than linux machines (especially in memory performance once it pass L1 or something)

comphead · 2026-11-12T01:55:19Z

Thanks @jayzhan211 I was actually about experimenting the same direction. I can benchmark it on 1TB if you resolve conflicts

…, off by default)

…ting buckets of unique groups

…idth

jayzhan211 · 2026-11-15T01:18:53Z

run benchmarks clickbench_partitioned clickbench_extended tpch10
baseline:
ref: ee5866b
env:
DATAFUSION_EXECUTION_HASH_AGGREGATE_BUCKET_THRESHOLD: "0"
changed:
env:
DATAFUSION_EXECUTION_HASH_AGGREGATE_BUCKET_THRESHOLD: "262144"

jayzhan211 · 2026-11-15T01:20:54Z

@comphead the conflicts are resolved. Could you run your 1TB benchmark with datafusion.execution.hash_aggregate_bucket_threshold set to 262144 and to 0? It is a session setting, so both runs can use the same build.

What to expect:

Only aggregations whose group keys and state are fixed-width use buckets, and only with spilling enabled (a disk manager). String-keyed aggregations should be unchanged.
The gain grows with the groups per final partition. Below the threshold nothing changes; up to about twice the threshold the gain is small.
Peak memory is worth recording: when the partial aggregation is skipped and groups repeat, rows wait in buckets until they are compacted. On input with 7.5 rows per group that was 1.13x of one table.

adriangbot · 2026-11-15T01:21:22Z

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c6887201304-5116-dv5nx 6.12.105+ #1 SMP Sat Sep 19 07:55:25 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing agg-buckets-final-stage (ee5866b) to ee5866b diff

Run configuration

run benchmark clickbench_extended
baseline:
  ref: "ee5866bc6fd319baf7bde1085ff9de0dc9095f30"
  env:
    DATAFUSION_EXECUTION_HASH_AGGREGATE_BUCKET_THRESHOLD: "0"
changed:
  env:
    DATAFUSION_EXECUTION_HASH_AGGREGATE_BUCKET_THRESHOLD: "262144"

Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-11-15T01:21:28Z

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c6887201304-5117-jqngw 6.12.105+ #1 SMP Sat Sep 19 07:55:25 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing agg-buckets-final-stage (ee5866b) to ee5866b diff

Run configuration

run benchmark tpch10
baseline:
  ref: "ee5866bc6fd319baf7bde1085ff9de0dc9095f30"
  env:
    DATAFUSION_EXECUTION_HASH_AGGREGATE_BUCKET_THRESHOLD: "0"
changed:
  env:
    DATAFUSION_EXECUTION_HASH_AGGREGATE_BUCKET_THRESHOLD: "262144"

Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-11-15T01:22:08Z

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c6887201304-5115-rzkzr 6.12.105+ #1 SMP Sat Sep 19 07:55:25 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing agg-buckets-final-stage (ee5866b) to ee5866b diff

Run configuration

run benchmark clickbench_partitioned
baseline:
  ref: "ee5866bc6fd319baf7bde1085ff9de0dc9095f30"
  env:
    DATAFUSION_EXECUTION_HASH_AGGREGATE_BUCKET_THRESHOLD: "0"
changed:
  env:
    DATAFUSION_EXECUTION_HASH_AGGREGATE_BUCKET_THRESHOLD: "262144"

Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-11-15T01:32:02Z

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

Comparing agg-buckets-final-stage (ee5866b) to ee5866b diff

Run configuration

run benchmark tpch10
baseline:
  ref: "ee5866bc6fd319baf7bde1085ff9de0dc9095f30"
  env:
    DATAFUSION_EXECUTION_HASH_AGGREGATE_BUCKET_THRESHOLD: "0"
changed:
  env:
    DATAFUSION_EXECUTION_HASH_AGGREGATE_BUCKET_THRESHOLD: "262144"

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

Comparing HEAD and agg-buckets-final-stage
--------------------
Benchmark tpch_sf10.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃      HEAD ┃ agg-buckets-final-stage ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1  │ 330.07 ms │               329.71 ms │     no change │
│ QQuery 2  │  89.30 ms │                90.47 ms │     no change │
│ QQuery 3  │ 220.41 ms │               218.83 ms │     no change │
│ QQuery 4  │ 115.15 ms │               115.47 ms │     no change │
│ QQuery 5  │ 346.18 ms │               349.09 ms │     no change │
│ QQuery 6  │ 101.49 ms │               101.69 ms │     no change │
│ QQuery 7  │ 445.13 ms │               452.59 ms │     no change │
│ QQuery 8  │ 357.73 ms │               359.64 ms │     no change │
│ QQuery 9  │ 508.99 ms │               515.79 ms │     no change │
│ QQuery 10 │ 290.13 ms │               285.84 ms │     no change │
│ QQuery 11 │  63.54 ms │                62.67 ms │     no change │
│ QQuery 12 │ 118.46 ms │               120.62 ms │     no change │
│ QQuery 13 │ 305.24 ms │               305.02 ms │     no change │
│ QQuery 14 │ 172.36 ms │               171.46 ms │     no change │
│ QQuery 15 │ 305.46 ms │               303.87 ms │     no change │
│ QQuery 16 │  64.17 ms │                64.64 ms │     no change │
│ QQuery 17 │ 601.02 ms │               591.58 ms │     no change │
│ QQuery 18 │ 431.03 ms │               395.95 ms │ +1.09x faster │
│ QQuery 19 │ 234.59 ms │               231.68 ms │     no change │
│ QQuery 20 │ 265.37 ms │               275.91 ms │     no change │
│ QQuery 21 │ 655.90 ms │               664.52 ms │     no change │
│ QQuery 22 │  63.32 ms │                60.91 ms │     no change │
└───────────┴───────────┴─────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary                      ┃           ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD)                      │ 6085.04ms │
│ Total Time (agg-buckets-final-stage)   │ 6067.95ms │
│ Average Time (HEAD)                    │  276.59ms │
│ Average Time (agg-buckets-final-stage) │  275.82ms │
│ Queries Faster                         │         1 │
│ Queries Slower                         │         0 │
│ Queries with No Change                 │        21 │
│ Queries with Failure                   │         0 │
└────────────────────────────────────────┴───────────┘

Distribution per query (min / mean ±stddev / max):

Comparing HEAD and agg-buckets-final-stage
--------------------
Benchmark tpch_sf10.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃                               HEAD ┃            agg-buckets-final-stage ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1  │  330.07 / 332.51 ±1.45 / 333.98 ms │  329.71 / 332.83 ±2.14 / 335.61 ms │     no change │
│ QQuery 2  │     89.30 / 90.22 ±0.58 / 91.09 ms │     90.47 / 93.70 ±3.47 / 99.17 ms │     no change │
│ QQuery 3  │  220.41 / 221.99 ±1.36 / 224.41 ms │  218.83 / 221.57 ±2.07 / 224.77 ms │     no change │
│ QQuery 4  │  115.15 / 116.63 ±0.82 / 117.54 ms │  115.47 / 118.13 ±1.57 / 120.23 ms │     no change │
│ QQuery 5  │  346.18 / 351.16 ±4.55 / 358.99 ms │  349.09 / 353.22 ±4.01 / 360.83 ms │     no change │
│ QQuery 6  │  101.49 / 103.36 ±1.45 / 105.58 ms │  101.69 / 104.25 ±2.52 / 109.09 ms │     no change │
│ QQuery 7  │ 445.13 / 458.97 ±10.36 / 471.06 ms │  452.59 / 461.35 ±8.23 / 474.79 ms │     no change │
│ QQuery 8  │  357.73 / 360.99 ±2.88 / 364.44 ms │  359.64 / 362.39 ±2.12 / 365.71 ms │     no change │
│ QQuery 9  │ 508.99 / 519.09 ±10.92 / 539.42 ms │  515.79 / 520.80 ±4.05 / 526.69 ms │     no change │
│ QQuery 10 │  290.13 / 297.36 ±3.92 / 300.84 ms │  285.84 / 296.69 ±9.50 / 314.18 ms │     no change │
│ QQuery 11 │     63.54 / 64.38 ±0.89 / 65.48 ms │     62.67 / 67.46 ±5.17 / 77.44 ms │     no change │
│ QQuery 12 │  118.46 / 123.34 ±8.32 / 139.94 ms │  120.62 / 121.50 ±0.96 / 123.07 ms │     no change │
│ QQuery 13 │  305.24 / 315.82 ±5.91 / 323.25 ms │  305.02 / 313.25 ±7.73 / 325.82 ms │     no change │
│ QQuery 14 │  172.36 / 178.00 ±5.69 / 186.47 ms │  171.46 / 176.41 ±6.07 / 187.45 ms │     no change │
│ QQuery 15 │  305.46 / 307.05 ±1.48 / 309.78 ms │  303.87 / 307.36 ±2.66 / 311.05 ms │     no change │
│ QQuery 16 │     64.17 / 68.11 ±4.39 / 76.35 ms │     64.64 / 66.36 ±1.37 / 68.71 ms │     no change │
│ QQuery 17 │  601.02 / 615.91 ±8.86 / 627.66 ms │ 591.58 / 605.74 ±11.57 / 617.44 ms │     no change │
│ QQuery 18 │  431.03 / 438.13 ±6.90 / 446.91 ms │  395.95 / 414.78 ±9.98 / 424.97 ms │ +1.06x faster │
│ QQuery 19 │ 234.59 / 247.88 ±14.37 / 270.70 ms │  231.68 / 242.48 ±8.64 / 255.68 ms │     no change │
│ QQuery 20 │  265.37 / 275.07 ±9.06 / 287.85 ms │  275.91 / 283.66 ±5.20 / 289.14 ms │     no change │
│ QQuery 21 │  655.90 / 662.19 ±4.43 / 667.72 ms │  664.52 / 670.12 ±5.35 / 679.78 ms │     no change │
│ QQuery 22 │     63.32 / 67.65 ±3.93 / 72.94 ms │     60.91 / 65.17 ±3.81 / 71.54 ms │     no change │
└───────────┴────────────────────────────────────┴────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary                      ┃           ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD)                      │ 6215.81ms │
│ Total Time (agg-buckets-final-stage)   │ 6199.25ms │
│ Average Time (HEAD)                    │  282.54ms │
│ Average Time (agg-buckets-final-stage) │  281.78ms │
│ Queries Faster                         │         1 │
│ Queries Slower                         │         0 │
│ Queries with No Change                 │        21 │
│ Queries with Failure                   │         0 │
└────────────────────────────────────────┴───────────┘

Resource Usage

tpch10 — base (merge-base)

Metric	Value
Wall time	35.0s
Peak memory	3.7 GiB
Avg memory	1.3 GiB
CPU user	315.1s
CPU sys	19.4s
Peak spill	0 B

tpch10 — branch

Metric	Value
Wall time	35.0s
Peak memory	3.7 GiB
Avg memory	1.2 GiB
CPU user	314.0s
CPU sys	18.9s
Peak spill	0 B

File an issue against this benchmark runner

adriangbot · 2026-11-15T01:35:08Z

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

Comparing agg-buckets-final-stage (ee5866b) to ee5866b diff

Run configuration

run benchmark clickbench_partitioned
baseline:
  ref: "ee5866bc6fd319baf7bde1085ff9de0dc9095f30"
  env:
    DATAFUSION_EXECUTION_HASH_AGGREGATE_BUCKET_THRESHOLD: "0"
changed:
  env:
    DATAFUSION_EXECUTION_HASH_AGGREGATE_BUCKET_THRESHOLD: "262144"

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

Comparing HEAD and agg-buckets-final-stage
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃       HEAD ┃ agg-buckets-final-stage ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0  │    1.16 ms │                 1.18 ms │     no change │
│ QQuery 1  │   11.77 ms │                11.78 ms │     no change │
│ QQuery 2  │   34.95 ms │                34.53 ms │     no change │
│ QQuery 3  │   30.89 ms │                30.53 ms │     no change │
│ QQuery 4  │  233.78 ms │               199.47 ms │ +1.17x faster │
│ QQuery 5  │  274.91 ms │               272.15 ms │     no change │
│ QQuery 6  │    1.20 ms │                 1.19 ms │     no change │
│ QQuery 7  │   13.00 ms │                13.24 ms │     no change │
│ QQuery 8  │  324.23 ms │               290.31 ms │ +1.12x faster │
│ QQuery 9  │  468.43 ms │               466.55 ms │     no change │
│ QQuery 10 │   64.24 ms │                65.12 ms │     no change │
│ QQuery 11 │   75.35 ms │                75.61 ms │     no change │
│ QQuery 12 │  263.05 ms │               263.69 ms │     no change │
│ QQuery 13 │  357.52 ms │               356.59 ms │     no change │
│ QQuery 14 │  278.59 ms │               278.34 ms │     no change │
│ QQuery 15 │  285.02 ms │               232.66 ms │ +1.23x faster │
│ QQuery 16 │  613.17 ms │               618.28 ms │     no change │
│ QQuery 17 │  620.05 ms │               634.49 ms │     no change │
│ QQuery 18 │ 1217.51 ms │              1215.49 ms │     no change │
│ QQuery 19 │   27.13 ms │                27.39 ms │     no change │
│ QQuery 20 │  518.36 ms │               518.35 ms │     no change │
│ QQuery 21 │  508.75 ms │               516.74 ms │     no change │
│ QQuery 22 │  992.35 ms │               976.91 ms │     no change │
│ QQuery 23 │ 3051.91 ms │              3064.46 ms │     no change │
│ QQuery 24 │   42.21 ms │                40.52 ms │     no change │
│ QQuery 25 │  105.79 ms │               104.80 ms │     no change │
│ QQuery 26 │   41.32 ms │                41.30 ms │     no change │
│ QQuery 27 │  511.93 ms │               510.55 ms │     no change │
│ QQuery 28 │ 2886.35 ms │              2866.44 ms │     no change │
│ QQuery 29 │   38.26 ms │                37.80 ms │     no change │
│ QQuery 30 │  303.10 ms │               308.60 ms │     no change │
│ QQuery 31 │  284.93 ms │               263.48 ms │ +1.08x faster │
│ QQuery 32 │  951.42 ms │               700.37 ms │ +1.36x faster │
│ QQuery 33 │ 1414.31 ms │              1467.44 ms │     no change │
│ QQuery 34 │ 1451.17 ms │              1482.01 ms │     no change │
│ QQuery 35 │  287.93 ms │               268.90 ms │ +1.07x faster │
│ QQuery 36 │   64.22 ms │                63.70 ms │     no change │
│ QQuery 37 │   33.92 ms │                34.26 ms │     no change │
│ QQuery 38 │   38.61 ms │                39.26 ms │     no change │
│ QQuery 39 │  133.25 ms │               135.69 ms │     no change │
│ QQuery 40 │   13.07 ms │                13.33 ms │     no change │
│ QQuery 41 │   12.51 ms │                12.66 ms │     no change │
│ QQuery 42 │   11.87 ms │                12.01 ms │     no change │
└───────────┴────────────┴─────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                      ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                      │ 18893.47ms │
│ Total Time (agg-buckets-final-stage)   │ 18568.15ms │
│ Average Time (HEAD)                    │   439.38ms │
│ Average Time (agg-buckets-final-stage) │   431.82ms │
│ Queries Faster                         │          6 │
│ Queries Slower                         │          0 │
│ Queries with No Change                 │         37 │
│ Queries with Failure                   │          0 │
└────────────────────────────────────────┴────────────┘

Distribution per query (min / mean ±stddev / max):

Comparing HEAD and agg-buckets-final-stage
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃                                  HEAD ┃               agg-buckets-final-stage ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0  │           1.16 / 1.76 ±1.11 / 3.97 ms │           1.18 / 1.78 ±1.07 / 3.92 ms │     no change │
│ QQuery 1  │        11.77 / 11.95 ±0.24 / 12.41 ms │        11.78 / 11.81 ±0.03 / 11.87 ms │     no change │
│ QQuery 2  │        34.95 / 35.32 ±0.38 / 36.03 ms │        34.53 / 34.86 ±0.25 / 35.30 ms │     no change │
│ QQuery 3  │        30.89 / 31.76 ±0.63 / 32.78 ms │        30.53 / 31.07 ±0.30 / 31.44 ms │     no change │
│ QQuery 4  │     233.78 / 241.26 ±4.62 / 246.27 ms │     199.47 / 201.00 ±1.77 / 204.34 ms │ +1.20x faster │
│ QQuery 5  │     274.91 / 277.01 ±1.50 / 279.40 ms │     272.15 / 276.74 ±2.78 / 279.74 ms │     no change │
│ QQuery 6  │           1.20 / 1.37 ±0.26 / 1.89 ms │           1.19 / 1.34 ±0.25 / 1.83 ms │     no change │
│ QQuery 7  │        13.00 / 13.10 ±0.08 / 13.22 ms │        13.24 / 13.36 ±0.10 / 13.54 ms │     no change │
│ QQuery 8  │     324.23 / 330.71 ±4.26 / 335.72 ms │     290.31 / 295.31 ±3.13 / 300.09 ms │ +1.12x faster │
│ QQuery 9  │     468.43 / 472.91 ±3.65 / 479.07 ms │     466.55 / 475.88 ±9.35 / 492.26 ms │     no change │
│ QQuery 10 │        64.24 / 64.93 ±0.69 / 66.20 ms │        65.12 / 65.41 ±0.39 / 66.15 ms │     no change │
│ QQuery 11 │        75.35 / 76.26 ±0.91 / 77.73 ms │        75.61 / 76.45 ±0.65 / 77.50 ms │     no change │
│ QQuery 12 │     263.05 / 267.48 ±3.90 / 274.74 ms │     263.69 / 269.95 ±6.24 / 280.99 ms │     no change │
│ QQuery 13 │     357.52 / 365.56 ±5.22 / 370.98 ms │    356.59 / 377.23 ±16.38 / 395.69 ms │     no change │
│ QQuery 14 │     278.59 / 283.35 ±2.53 / 286.07 ms │     278.34 / 283.13 ±3.26 / 288.15 ms │     no change │
│ QQuery 15 │    285.02 / 295.19 ±11.44 / 317.39 ms │     232.66 / 238.24 ±5.70 / 245.29 ms │ +1.24x faster │
│ QQuery 16 │     613.17 / 623.92 ±8.40 / 637.27 ms │     618.28 / 627.63 ±6.52 / 635.99 ms │     no change │
│ QQuery 17 │    620.05 / 631.91 ±10.12 / 649.94 ms │     634.49 / 645.34 ±9.26 / 657.48 ms │     no change │
│ QQuery 18 │ 1217.51 / 1237.77 ±18.19 / 1265.51 ms │ 1215.49 / 1232.61 ±24.26 / 1279.06 ms │     no change │
│ QQuery 19 │        27.13 / 27.47 ±0.20 / 27.65 ms │       27.39 / 40.46 ±18.09 / 73.77 ms │  1.47x slower │
│ QQuery 20 │    518.36 / 531.42 ±13.83 / 553.82 ms │    518.35 / 530.58 ±16.73 / 563.78 ms │     no change │
│ QQuery 21 │     508.75 / 517.79 ±7.66 / 531.82 ms │     516.74 / 522.65 ±7.42 / 536.76 ms │     no change │
│ QQuery 22 │     992.35 / 995.72 ±2.41 / 999.50 ms │  976.91 / 1000.07 ±17.21 / 1029.84 ms │     no change │
│ QQuery 23 │ 3051.91 / 3099.12 ±36.12 / 3154.74 ms │ 3064.46 / 3105.28 ±36.76 / 3161.04 ms │     no change │
│ QQuery 24 │        42.21 / 46.98 ±6.18 / 58.98 ms │       40.52 / 50.26 ±11.31 / 68.85 ms │  1.07x slower │
│ QQuery 25 │     105.79 / 109.20 ±4.20 / 115.84 ms │     104.80 / 105.22 ±0.32 / 105.75 ms │     no change │
│ QQuery 26 │        41.32 / 41.71 ±0.42 / 42.46 ms │        41.30 / 46.59 ±5.91 / 54.70 ms │  1.12x slower │
│ QQuery 27 │     511.93 / 518.04 ±4.91 / 524.57 ms │     510.55 / 519.68 ±7.05 / 531.13 ms │     no change │
│ QQuery 28 │ 2886.35 / 2906.90 ±13.43 / 2923.16 ms │ 2866.44 / 2911.30 ±28.76 / 2955.15 ms │     no change │
│ QQuery 29 │       38.26 / 47.19 ±10.35 / 66.16 ms │       37.80 / 51.79 ±22.35 / 95.58 ms │  1.10x slower │
│ QQuery 30 │     303.10 / 310.44 ±8.03 / 325.84 ms │     308.60 / 318.82 ±5.24 / 323.19 ms │     no change │
│ QQuery 31 │     284.93 / 290.47 ±5.63 / 301.21 ms │     263.48 / 272.08 ±9.37 / 288.22 ms │ +1.07x faster │
│ QQuery 32 │    951.42 / 974.99 ±16.48 / 994.51 ms │    700.37 / 747.54 ±57.65 / 857.31 ms │ +1.30x faster │
│ QQuery 33 │ 1414.31 / 1466.32 ±31.23 / 1510.91 ms │ 1467.44 / 1488.49 ±17.15 / 1510.84 ms │     no change │
│ QQuery 34 │ 1451.17 / 1483.20 ±21.92 / 1515.40 ms │ 1482.01 / 1504.99 ±17.82 / 1528.25 ms │     no change │
│ QQuery 35 │    287.93 / 340.56 ±86.45 / 512.78 ms │     268.90 / 276.91 ±5.31 / 282.98 ms │ +1.23x faster │
│ QQuery 36 │        64.22 / 68.16 ±2.61 / 71.91 ms │        63.70 / 76.51 ±8.45 / 89.47 ms │  1.12x slower │
│ QQuery 37 │        33.92 / 34.38 ±0.58 / 35.51 ms │        34.26 / 37.99 ±5.96 / 49.82 ms │  1.10x slower │
│ QQuery 38 │        38.61 / 44.51 ±4.98 / 53.70 ms │        39.26 / 44.63 ±4.64 / 52.98 ms │     no change │
│ QQuery 39 │    133.25 / 151.10 ±14.07 / 176.36 ms │    135.69 / 146.74 ±10.83 / 167.20 ms │     no change │
│ QQuery 40 │        13.07 / 13.45 ±0.48 / 14.31 ms │        13.33 / 15.96 ±3.79 / 23.49 ms │  1.19x slower │
│ QQuery 41 │        12.51 / 12.83 ±0.32 / 13.25 ms │        12.66 / 14.30 ±2.60 / 19.49 ms │  1.11x slower │
│ QQuery 42 │        11.87 / 12.66 ±1.20 / 15.04 ms │        12.01 / 12.27 ±0.29 / 12.78 ms │     no change │
└───────────┴───────────────────────────────────────┴───────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                      ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                      │ 19308.12ms │
│ Total Time (agg-buckets-final-stage)   │ 19000.24ms │
│ Average Time (HEAD)                    │   449.03ms │
│ Average Time (agg-buckets-final-stage) │   441.87ms │
│ Queries Faster                         │          6 │
│ Queries Slower                         │          8 │
│ Queries with No Change                 │         29 │
│ Queries with Failure                   │          0 │
└────────────────────────────────────────┴────────────┘

Resource Usage

clickbench_partitioned — base (merge-base)

Metric	Value
Wall time	100.0s
Peak memory	10.4 GiB
Avg memory	4.3 GiB
CPU user	986.8s
CPU sys	71.3s
Peak spill	0 B

clickbench_partitioned — branch

Metric	Value
Wall time	100.0s
Peak memory	9.8 GiB
Avg memory	4.2 GiB
CPU user	965.5s
CPU sys	72.5s
Peak spill	0 B

File an issue against this benchmark runner

adriangbot · 2026-11-15T01:39:37Z

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

Comparing agg-buckets-final-stage (ee5866b) to ee5866b diff

Run configuration

run benchmark clickbench_extended
baseline:
  ref: "ee5866bc6fd319baf7bde1085ff9de0dc9095f30"
  env:
    DATAFUSION_EXECUTION_HASH_AGGREGATE_BUCKET_THRESHOLD: "0"
changed:
  env:
    DATAFUSION_EXECUTION_HASH_AGGREGATE_BUCKET_THRESHOLD: "262144"

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

Comparing HEAD and agg-buckets-final-stage
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃        HEAD ┃ agg-buckets-final-stage ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0  │   785.76 ms │               774.13 ms │     no change │
│ QQuery 1  │   174.52 ms │               173.70 ms │     no change │
│ QQuery 2  │   432.44 ms │               436.46 ms │     no change │
│ QQuery 3  │   320.27 ms │               319.08 ms │     no change │
│ QQuery 4  │  1178.59 ms │              1167.02 ms │     no change │
│ QQuery 5  │ 10616.88 ms │             10909.52 ms │     no change │
│ QQuery 6  │     2.59 ms │                 2.60 ms │     no change │
│ QQuery 7  │   735.78 ms │               429.98 ms │ +1.71x faster │
│ QQuery 8  │   426.24 ms │               428.24 ms │     no change │
│ QQuery 9  │  1319.61 ms │              1340.96 ms │     no change │
│ QQuery 10 │   615.54 ms │               621.44 ms │     no change │
│ QQuery 11 │   490.85 ms │               491.81 ms │     no change │
│ QQuery 12 │   178.45 ms │               175.70 ms │     no change │
│ QQuery 13 │   547.39 ms │               546.46 ms │     no change │
│ QQuery 14 │   586.21 ms │               566.84 ms │     no change │
│ QQuery 15 │  2559.27 ms │              2844.14 ms │  1.11x slower │
│ QQuery 16 │   278.87 ms │               279.91 ms │     no change │
└───────────┴─────────────┴─────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                      ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                      │ 21249.26ms │
│ Total Time (agg-buckets-final-stage)   │ 21508.00ms │
│ Average Time (HEAD)                    │  1249.96ms │
│ Average Time (agg-buckets-final-stage) │  1265.18ms │
│ Queries Faster                         │          1 │
│ Queries Slower                         │          1 │
│ Queries with No Change                 │         15 │
│ Queries with Failure                   │          0 │
└────────────────────────────────────────┴────────────┘

Distribution per query (min / mean ±stddev / max):

Comparing HEAD and agg-buckets-final-stage
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃                                      HEAD ┃                   agg-buckets-final-stage ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0  │        785.76 / 799.55 ±12.24 / 820.03 ms │        774.13 / 788.28 ±12.40 / 810.80 ms │     no change │
│ QQuery 1  │         174.52 / 175.57 ±1.14 / 177.00 ms │         173.70 / 174.89 ±1.15 / 177.02 ms │     no change │
│ QQuery 2  │         432.44 / 437.80 ±4.52 / 444.10 ms │         436.46 / 440.35 ±4.08 / 447.88 ms │     no change │
│ QQuery 3  │         320.27 / 321.57 ±1.02 / 323.20 ms │         319.08 / 321.21 ±1.70 / 323.86 ms │     no change │
│ QQuery 4  │      1178.59 / 1190.26 ±6.21 / 1196.86 ms │     1167.02 / 1181.68 ±11.84 / 1198.18 ms │     no change │
│ QQuery 5  │ 10616.88 / 11162.62 ±409.80 / 11623.30 ms │ 10909.52 / 11160.42 ±202.11 / 11402.17 ms │     no change │
│ QQuery 6  │               2.59 / 2.82 ±0.32 / 3.45 ms │               2.60 / 2.78 ±0.33 / 3.43 ms │     no change │
│ QQuery 7  │        735.78 / 794.69 ±38.21 / 843.14 ms │        429.98 / 466.94 ±48.97 / 563.07 ms │ +1.70x faster │
│ QQuery 8  │        426.24 / 444.83 ±19.69 / 473.21 ms │        428.24 / 445.41 ±29.17 / 503.53 ms │     no change │
│ QQuery 9  │     1319.61 / 1374.04 ±55.28 / 1471.95 ms │     1340.96 / 1387.00 ±45.55 / 1449.02 ms │     no change │
│ QQuery 10 │         615.54 / 633.49 ±9.33 / 640.90 ms │        621.44 / 633.32 ±14.89 / 662.54 ms │     no change │
│ QQuery 11 │        490.85 / 523.87 ±19.80 / 543.05 ms │        491.81 / 550.17 ±34.28 / 594.43 ms │  1.05x slower │
│ QQuery 12 │         178.45 / 183.20 ±5.31 / 191.95 ms │         175.70 / 176.45 ±0.62 / 177.31 ms │     no change │
│ QQuery 13 │         547.39 / 560.79 ±8.01 / 572.06 ms │        546.46 / 565.14 ±16.10 / 591.58 ms │     no change │
│ QQuery 14 │        586.21 / 609.99 ±21.37 / 643.83 ms │        566.84 / 617.07 ±31.63 / 648.15 ms │     no change │
│ QQuery 15 │     2559.27 / 2627.66 ±41.58 / 2684.42 ms │     2844.14 / 2974.43 ±85.66 / 3083.36 ms │  1.13x slower │
│ QQuery 16 │        278.87 / 292.17 ±17.17 / 326.13 ms │         279.91 / 285.80 ±7.86 / 301.35 ms │     no change │
└───────────┴───────────────────────────────────────────┴───────────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                      ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                      │ 22134.90ms │
│ Total Time (agg-buckets-final-stage)   │ 22171.34ms │
│ Average Time (HEAD)                    │  1302.05ms │
│ Average Time (agg-buckets-final-stage) │  1304.20ms │
│ Queries Faster                         │          1 │
│ Queries Slower                         │          2 │
│ Queries with No Change                 │         14 │
│ Queries with Failure                   │          0 │
└────────────────────────────────────────┴────────────┘

Resource Usage

clickbench_extended — base (merge-base)

Metric	Value
Wall time	115.0s
Peak memory	12.8 GiB
Avg memory	5.5 GiB
CPU user	1063.7s
CPU sys	53.1s
Peak spill	0 B

clickbench_extended — branch

Metric	Value
Wall time	115.0s
Peak memory	14.1 GiB
Avg memory	5.3 GiB
CPU user	1039.7s
CPU sys	48.0s
Peak spill	0 B

File an issue against this benchmark runner

jayzhan211 · 2026-11-15T01:56:48Z

I'm turning this PR back to draft: with buckets on, it makes some aggregations faster and others slower, and I don't see a fix yet that doesn't add more tuned limits.

On the benchmark runner (same commit, threshold 0 vs 262144):

Faster: ClickBench Q32 1.36x, Q15 1.23x, Q4 1.17x (results); suite total 1.7%. TPC-H SF10 unchanged.
Slower: clickbench_extended Q15 (covar_samp grouped by UserID) 1.11-1.13x, and that suite's peak memory goes from 12.8 to 14.1 GiB (results). My laptop reproduces it at 1.22x.

My laptop also shows costs the description understates:

Peak memory up to 1.9x, not 1.13x, when the partial aggregation is skipped and groups repeat (4M groups, 8 aggregates): rows wait in buckets until they are compacted.
A query that ends just above the threshold pays for the move and gains nothing: ClickBench Q30 on half the data ends at about 264k groups per partition and runs 1.09x slower.
Under a 2 GB memory limit, the 8-aggregate query above runs 2.2x slower.

github-actions Bot added documentation Improvements or additions to documentation sqllogictest SQL Logic Tests (.slt) common Related to common crate physical-plan Changes to the physical-plan crate labels Sep 25, 2026

jayzhan211 mentioned this pull request Sep 25, 2026

feat: bucket the hash table of single stage aggregations that run on every partition #25725

Draft

jayzhan211 force-pushed the agg-buckets-final-stage branch from bdd382c to f9df89e Compare September 25, 2026 02:03

github-actions Bot added the auto detected api change Auto detected API change label Sep 25, 2026

jayzhan211 mentioned this pull request Sep 25, 2026

feat: aggregate large hash tables as hash buckets (experimental, off by default) #25567

Draft

This was referenced Sep 28, 2026

feat: prototype frozen final aggregation buckets jayzhan211/datafusion#8

Closed

feat: prototype frozen final aggregation buckets #25846

Closed

perf: compare bucket group indexes #25852

Closed

perf: route final aggregate buckets by column #25959

Draft

jayzhan211 force-pushed the agg-buckets-final-stage branch 2 times, most recently from 52a76ba to 7b7c506 Compare October 3, 2026 14:03

Rachelint mentioned this pull request Oct 7, 2026

[EPIC] Use blocked / chunked memory management in hash aggregation #24704

Open

1 of 11 tasks

jayzhan211 mentioned this pull request Oct 24, 2026

perf: flush partial batches with fixed group capacity (experiment) #26116

Draft

jayzhan211 force-pushed the agg-buckets-final-stage branch from 88d36d4 to 8f2fb69 Compare November 14, 2026 04:13

github-actions Bot removed the auto detected api change Auto detected API change label Nov 14, 2026

jayzhan211 added 5 commits November 15, 2026 09:13

feat: aggregate large final hash tables as hash buckets (experimental…

91f97d4

…, off by default)

perf: reuse one hash table across aggregation buckets and skip compac…

1f8324a

…ting buckets of unique groups

refactor: move the aggregation bucket driver into its own module

4902650

perf: bucket only aggregations whose group keys and state are fixed-w…

59edc83

…idth

fix: keep one final aggregation table when spilling is disabled

ee5866b

jayzhan211 force-pushed the agg-buckets-final-stage branch from 8f2fb69 to ee5866b Compare November 15, 2026 01:17

jayzhan211 marked this pull request as ready for review November 15, 2026 01:17

jayzhan211 marked this pull request as draft November 15, 2026 01:56

Conversation

jayzhan211 commented Sep 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

What is the testing strategy for this PR?

Are there any user-facing changes?

Uh oh!

jayzhan211 commented Sep 25, 2026

Uh oh!

codecov-commenter commented Sep 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Rachelint commented Sep 27, 2026

Uh oh!

adriangbot commented Sep 27, 2026

Uh oh!

adriangbot commented Sep 27, 2026

Uh oh!

Rachelint commented Sep 28, 2026

Uh oh!

adriangbot commented Sep 28, 2026

Uh oh!

Rachelint commented Sep 28, 2026

Uh oh!

adriangbot commented Sep 28, 2026

Uh oh!

adriangbot commented Sep 28, 2026

Uh oh!

jayzhan211 commented Oct 3, 2026

Uh oh!

adriangbot commented Oct 3, 2026

Uh oh!

adriangbot commented Oct 3, 2026

Uh oh!

adriangbot commented Oct 3, 2026

Uh oh!

adriangbot commented Oct 3, 2026

Uh oh!

jayzhan211 commented Oct 3, 2026

Uh oh!

adriangbot commented Oct 7, 2026

Uh oh!

Rachelint commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jayzhan211 commented Oct 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rluvaton commented Nov 8, 2026

Uh oh!

comphead commented Nov 12, 2026

Uh oh!

jayzhan211 commented Nov 15, 2026

Uh oh!

jayzhan211 commented Nov 15, 2026

Uh oh!

adriangbot commented Nov 15, 2026

Uh oh!

adriangbot commented Nov 15, 2026

Uh oh!

adriangbot commented Nov 15, 2026

Uh oh!

adriangbot commented Nov 15, 2026

Uh oh!

adriangbot commented Nov 15, 2026

Uh oh!

adriangbot commented Nov 15, 2026

Uh oh!

jayzhan211 commented Nov 15, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

jayzhan211 commented Sep 25, 2026 •

edited

Loading

codecov-commenter commented Sep 25, 2026 •

edited

Loading

Rachelint commented Oct 7, 2026 •

edited

Loading

jayzhan211 commented Oct 24, 2026 •

edited

Loading