Repository navigation
Optimize single-column dictionary grouping - #27272
Rich-T-kid wants to merge 3 commits into
Conversation
|
run benchmark tpch tpcds clickbench_partitioned |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing rich-t-kid/single-dictionary-group-values (0d75b53) to 14b3edd (merge-base) diff Run configurationrun benchmark tpchResults will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing rich-t-kid/single-dictionary-group-values (0d75b53) to 14b3edd (merge-base) diff Run configurationrun benchmark tpcdsResults will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing rich-t-kid/single-dictionary-group-values (0d75b53) to 14b3edd (merge-base) diff Run configurationrun benchmark clickbench_partitionedResults will be posted here when complete File an issue against this benchmark runner |
|
run benchmark dictionary_group_values |
|
Benchmark for this request failed before finishing (Kubernetes reason: Benchmarks requested: Runner log (last 40 lines)Kubernetes messageFile an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing rich-t-kid/single-dictionary-group-values (0d75b53) to 14b3edd (merge-base) diff Run configurationrun benchmark dictionary_group_valuesResults will be posted here when complete File an issue against this benchmark runner |
|
run benchmark tpch tpcds clickbench_partitioned env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true" |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing rich-t-kid/single-dictionary-group-values (0d75b53) to 14b3edd (merge-base) diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing rich-t-kid/single-dictionary-group-values (0d75b53) to 14b3edd (merge-base) diff Run configurationrun benchmark tpcds
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing rich-t-kid/single-dictionary-group-values (0d75b53) to 14b3edd (merge-base) diff Run configurationrun benchmark tpch
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"Results will be posted here when complete File an issue against this benchmark runner |
|
Ignore the earlier three-suite run; it did not enable RLE dictionary reads. The rerun above does. |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing rich-t-kid/single-dictionary-group-values (0d75b53) to 14b3edd (merge-base) diff Run configurationrun benchmark tpchCPU Details (lscpu)Details
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing rich-t-kid/single-dictionary-group-values (0d75b53) to 14b3edd (merge-base) diff Run configurationrun benchmark tpcdsCPU Details (lscpu)Details
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing rich-t-kid/single-dictionary-group-values (0d75b53) to 14b3edd (merge-base) diff Run configurationrun benchmark tpch
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"CPU Details (lscpu)Details
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing rich-t-kid/single-dictionary-group-values (0d75b53) to 14b3edd (merge-base) diff Run configurationrun benchmark tpcds
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"CPU Details (lscpu)Details
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing rich-t-kid/single-dictionary-group-values (0d75b53) to 14b3edd (merge-base) diff Run configurationrun benchmark clickbench_partitionedCPU Details (lscpu)Details
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing rich-t-kid/single-dictionary-group-values (0d75b53) to 14b3edd (merge-base) diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"CPU Details (lscpu)Details
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing rich-t-kid/single-dictionary-group-values (0d75b53) to 14b3edd (merge-base) diff Run configurationrun benchmark dictionary_group_valuesCPU Details (lscpu)Details
Resource Usagedictionary_group_values — base (merge-base)
dictionary_group_values — branch
File an issue against this benchmark runner |
0d75b53 to
8c55192
Compare
|
run benchmark dictionary_group_values |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing rich-t-kid/single-dictionary-group-values (8c55192) to 14b3edd (merge-base) diff Run configurationrun benchmark dictionary_group_valuesResults will be posted here when complete File an issue against this benchmark runner |
|
run benchmark tpch tpcds clickbench_partitioned env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true" |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing rich-t-kid/single-dictionary-group-values (8c55192) to 14b3edd (merge-base) diff Run configurationrun benchmark tpcds
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing rich-t-kid/single-dictionary-group-values (8c55192) to 14b3edd (merge-base) diff Run configurationrun benchmark tpch
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing rich-t-kid/single-dictionary-group-values (8c55192) to 14b3edd (merge-base) diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing rich-t-kid/single-dictionary-group-values (8c55192) to 14b3edd (merge-base) diff Run configurationrun benchmark tpch
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"CPU Details (lscpu)Details
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
f34451c to
f6ce571
Compare
|
run benchmark dictionary_group_values |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing rich-t-kid/single-dictionary-group-values (f6ce571) to c8265b6 (merge-base) diff Run configurationrun benchmark dictionary_group_valuesResults will be posted here when complete File an issue against this benchmark runner |
|
run benchmark tpch tpcds clickbench_partitioned env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true" |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing rich-t-kid/single-dictionary-group-values (f6ce571) to c8265b6 (merge-base) diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing rich-t-kid/single-dictionary-group-values (f6ce571) to c8265b6 (merge-base) diff Run configurationrun benchmark tpch
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing rich-t-kid/single-dictionary-group-values (f6ce571) to c8265b6 (merge-base) diff Run configurationrun benchmark tpcds
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing rich-t-kid/single-dictionary-group-values (f6ce571) to c8265b6 (merge-base) diff Run configurationrun benchmark tpch
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"CPU Details (lscpu)Details
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing rich-t-kid/single-dictionary-group-values (f6ce571) to c8265b6 (merge-base) diff Run configurationrun benchmark tpcds
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"CPU Details (lscpu)Details
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
|
run benchmark tpcds env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true" |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing rich-t-kid/single-dictionary-group-values (f6ce571) to c8265b6 (merge-base) diff Run configurationrun benchmark tpcds
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing rich-t-kid/single-dictionary-group-values (f6ce571) to c8265b6 (merge-base) diff Run configurationrun benchmark dictionary_group_valuesCPU Details (lscpu)Details
Resource Usagedictionary_group_values — base (merge-base)
dictionary_group_values — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing rich-t-kid/single-dictionary-group-values (f6ce571) to c8265b6 (merge-base) diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"CPU Details (lscpu)Details
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing rich-t-kid/single-dictionary-group-values (f6ce571) to c8265b6 (merge-base) diff Run configurationrun benchmark tpcds
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"CPU Details (lscpu)Details
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
|
@kumarUjjawal could you take a look |
kumarUjjawal
left a comment
There was a problem hiding this comment.
Thank you @Rich-T-kid
Left few comments please take a look
|
run benchmark dictionary_group_values tpch tpcds clickbench_partitioned env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true" |
|
@kumarUjjawal PTAL |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing rich-t-kid/single-dictionary-group-values (3f7f357) to c8265b6 (merge-base) diff Run configurationrun benchmark tpcds
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing rich-t-kid/single-dictionary-group-values (3f7f357) to c8265b6 (merge-base) diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing rich-t-kid/single-dictionary-group-values (3f7f357) to c8265b6 (merge-base) diff Run configurationrun benchmark tpch
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing rich-t-kid/single-dictionary-group-values (3f7f357) to c8265b6 (merge-base) diff Run configurationrun benchmark dictionary_group_values
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing rich-t-kid/single-dictionary-group-values (3f7f357) to c8265b6 (merge-base) diff Run configurationrun benchmark tpch
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"CPU Details (lscpu)Details
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing rich-t-kid/single-dictionary-group-values (3f7f357) to c8265b6 (merge-base) diff Run configurationrun benchmark tpcds
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"CPU Details (lscpu)Details
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing rich-t-kid/single-dictionary-group-values (3f7f357) to c8265b6 (merge-base) diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"CPU Details (lscpu)Details
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing rich-t-kid/single-dictionary-group-values (3f7f357) to c8265b6 (merge-base) diff Run configurationrun benchmark dictionary_group_values
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"CPU Details (lscpu)Details
Resource Usagedictionary_group_values — base (merge-base)
dictionary_group_values — branch
File an issue against this benchmark runner |
What does this PR do?
Adds a dedicated single-column dictionary grouping implementation. It resolves referenced dictionary values through the existing single-column value implementation, caches their group ids, and maps raw dictionary keys directly to those groups. Multi-column grouping is unchanged.
Related to #24822.
Target workload
This targets aggregates with one dictionary-encoded grouping expression, such as
GROUP BY dictionary_columnandCOUNT(DISTINCT dictionary_column). It is most useful for low- and medium-cardinality strings where many rows reuse dictionary values. It does not affect scans, filters, or multi-column grouping.Examples in ClickBench are Q5 (
COUNT(DISTINCT SearchPhrase)), Q33 (GROUP BY URL), and Q34, where the constant grouping expression leavesURLas the only data grouping key.Benchmarks
Bot results on
f6ce57135, with RLE dictionary reads enabled for the query suites:dictionary_group_values: all 35 cases improved by 1.27-4.10x.