Skip to content

Use the full hash when deduplicating and filtering bench cases - #229

Draft
0z5a wants to merge 1 commit into
IntelPython:mainfrom
0z5a:full-hash-dedup
Draft

0z5a wants to merge 1 commit into
IntelPython:mainfrom
0z5a:full-hash-dedup

Conversation

@0z5a

@0z5a 0z5a commented Nov 18, 2026

Copy link
Copy Markdown

Description

Fixes #228.

remove_duplicated_bench_cases keyed bench cases by hash_from_json_repr, which keeps only 5 hex digits of the sha256. Two different cases with the same prefix replaced each other without any message. With configs/regular configs/weekly configs/experiments together, 1679 distinct cases went in and 1678 came out: ExtraTreesClassifier/mnist from regular/ensemble.json and a raft NearestNeighbors/fashion_mnist case from experiments/nearest_neighbors.json both hash to a9f34, and the ExtraTrees case was dropped.

This change passes hash_limit=None (full digest) in remove_duplicated_bench_cases and in bench_case_filter, and makes hash_limit an optional int in the signature. The short hash is still used for the environment name and the vtune result path, where it is only part of a name.

How I found it

Reading the config expansion code, I saw the dedup key was the same shortened hash used in display names. I expanded the shipped configs with a small script (dataset loading stubbed) and compared the number of cases after dedup with a count by full sha256 of the same JSON repr. Before: 1679 → 1678, one colliding prefix. After: 1679 → 1679. Each config directory alone, and regular+weekly together, has no collision, so a normal single-directory run is not affected today.

Where this can show up in practice

This is my estimate; I have not seen a field report.

  • A combined run over several config directories (like the one above) reports results with one case quietly missing, so a regression on that case would not be seen.
  • As more configs are added, the number of cases in one run goes up and the chance of a collision grows roughly with the square of the case count.
  • Comparing two reports where a different case got dropped in each would make it look like coverage changed when it did not.

Testing

  • The counting script above, before and after the change.
  • python -m sklbench -c configs/sklearn_example.json goes through all 22 cases locally (macOS arm64). The 11 sklearn cases run; the 11 sklearnex ones fail only because sklearnex is not installable there. (py-cpuinfo returns no flags key on Apple silicon, so I stubbed that key for this run; unrelated to this change.)
  • black 25.1.0 / isort 5.13.2 (versions from .pre-commit-config.yaml) are clean.

Checklist:

Completeness and readability

  • I have commented my code, particularly in hard-to-understand areas.
  • I have updated the documentation to reflect the changes or created a separate PR with updates and provided its number in the description, if necessary. (no doc change needed)
  • Git commit message contains an appropriate signed-off-by string (see CONTRIBUTING.md for details).
  • I have resolved any merge conflicts that might occur with the base branch.

Testing

  • I have run it locally and tested the changes extensively.
  • All CI jobs are green or I have provided justification why they aren't.
  • I have extended testing suite if new functionality was introduced in this PR. (no new functionality; the repo has no unit tests for this module)

hash_from_json_repr keeps only 5 hex digits (20 bits). remove_duplicated_bench_cases
keys cases by that short hash, so two different cases with the same prefix
silently replace each other. Running configs/regular, configs/weekly and
configs/experiments together has 1679 distinct cases but kept 1678:
ExtraTreesClassifier/mnist and a raft NearestNeighbors/fashion_mnist case
share a9f34.

Use the full digest for deduplication and for bench_case_filter. The short
hash is still used where it is only part of a display name.

Fixes IntelPython#228

Signed-off-by: Dezhen Lu <[email protected]>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bench case deduplication drops distinct cases when their 5-digit hashes collide

1 participant