Skip to content

Remove unmaintained RustSec dependencies from benchmarks - #187

Merged
gorzell merged 1 commit into
mainfrom
gorzell/shiny-fortnight
Nov 18, 2026
Merged

gorzell merged 1 commit into
mainfrom
gorzell/shiny-fortnight

Conversation

@gorzell

@gorzell gorzell commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

The benchmark's tokenizer HTTP feature pulled in hf-hub 1.0 and its Xet dependency chain, which introduced the unmaintained chrono and humantime crates. Keep the existing tokenizer API and cache behavior, but use hf-hub 0.4 directly for model downloads and disable the tokenizer crate's HTTP feature.

Validation: benchmark unit tests pass and equivalence tests compile. The live model comparison could not run because this environment cannot resolve huggingface.co.

Closes #185
Closes #186

Co-authored-by: Copilot <[email protected]>

Co-authored-by: copilot-swe-agent[bot] <[email protected]>
@gorzell
gorzell requested a review from a team as a code owner November 18, 2026 09:05
Copilot AI adaptive review requested due to automatic review settings November 18, 2026 09:05

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The hf-hub downgrade changes download, cache, and TLS behavior, requiring compatibility validation for existing caches and live model retrieval beyond the available source review and offline checks.

No open issues 🎉

What changed in this PR

This PR removes the Xet dependency chain from the BPE benchmarks by downloading tokenizer models directly through hf-hub 0.4, while leaving tokenizer parsing unchanged.

Changes:

  • Replace the tokenizer crate’s HTTP helper with the synchronous hf-hub download API.
  • Disable the tokenizer HTTP feature and add hf-hub 0.4.3 directly.
  • Update the lockfile to remove Xet, chrono, humantime, and related dependencies.
File Description
crates/​bpe/​benchmarks/​lib.rs Download tokenizer.json directly through hf-hub.
crates/​bpe/​benchmarks/​Cargo.toml Add the direct download dependency and disable tokenizer HTTP support.
Cargo.lock Record the revised dependency graph and remove the Xet chain.

Review effort: Balanced (adaptive)

@tclem
tclem requested a review from itsibitzi November 18, 2026 09:22
@gorzell
gorzell merged commit 6c8db0b into main Nov 18, 2026
10 checks passed
@gorzell
gorzell deleted the gorzell/shiny-fortnight branch November 18, 2026 09:59
@tclem
tclem removed the request for review from itsibitzi November 18, 2026 10:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

RUSTSEC-2026-0542: humantime is unmaintained RUSTSEC-2026-0543: chrono is unmaintained

4 participants