Repository navigation
Remove unmaintained RustSec dependencies from benchmarks - #187
Merged
Merged
Conversation
Co-authored-by: Copilot <[email protected]> Co-authored-by: copilot-swe-agent[bot] <[email protected]>
Contributor
There was a problem hiding this comment.
🔵 Needs a closer look
The hf-hub downgrade changes download, cache, and TLS behavior, requiring compatibility validation for existing caches and live model retrieval beyond the available source review and offline checks.
No open issues 🎉
What changed in this PR
This PR removes the Xet dependency chain from the BPE benchmarks by downloading tokenizer models directly through hf-hub 0.4, while leaving tokenizer parsing unchanged.
Changes:
- Replace the tokenizer crate’s HTTP helper with the synchronous
hf-hubdownload API. - Disable the tokenizer HTTP feature and add
hf-hub0.4.3 directly. - Update the lockfile to remove Xet,
chrono,humantime, and related dependencies.
| File | Description |
|---|---|
crates/bpe/benchmarks/lib.rs |
Download tokenizer.json directly through hf-hub. |
crates/bpe/benchmarks/Cargo.toml |
Add the direct download dependency and disable tokenizer HTTP support. |
Cargo.lock |
Record the revised dependency graph and remove the Xet chain. |
Review effort: Balanced (adaptive)
mikerhodes
approved these changes
Nov 18, 2026
mikerhodes
approved these changes
Nov 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The benchmark's tokenizer HTTP feature pulled in
hf-hub1.0 and its Xet dependency chain, which introduced the unmaintainedchronoandhumantimecrates. Keep the existing tokenizer API and cache behavior, but usehf-hub0.4 directly for model downloads and disable the tokenizer crate's HTTP feature.Validation: benchmark unit tests pass and equivalence tests compile. The live model comparison could not run because this environment cannot resolve
huggingface.co.Closes #185
Closes #186