Skip to content

feat(benchmark): add raw plot mode and refresh figures with the 10-rep sweep - #114

Merged
prdai merged 1 commit into
mainfrom
feat/benchmark-raw-plots-reps10
Nov 15, 2026
Merged

prdai merged 1 commit into
mainfrom
feat/benchmark-raw-plots-reps10

Conversation

@prdai

@prdai prdai commented Nov 15, 2026 •

Copy link
Copy Markdown
Member

what changed

  • Add --no-smooth to benchmark/src/plot.py, which skips the LOWESS overlay and plots the raw runs.
  • Replace the committed sweep results with a 10-rep run: 15 sizes from 10K to 150K, one CE and one SCE run per rep, both launched concurrently.
  • Regenerate phase-timings.csv and all five figures without smoothing.
  • Move older sweep JSON files to results/raw/archive/ so export only merges the current run.

why

The previous figures mixed runs from multiple sweeps, and the scaling curve needed a raw view with real repetition spread. The 10-rep data shows SCE overhead is stable at about 9.9% across sizes instead of noisy point-to-point swings.

verification

  • Smoke run at 10K: CE 94.1s, SCE 106.0s, both succeeded.
  • Full run: 298/300 rows succeeded; the two failures are 80K rep 7 for both trees (ConnectError, transient API error).
  • Spot values: 150K CE 122.9s +/- 0.9s, SCE 134.9s +/- 1.9s; mean overhead +9.9%.
  • make export then PYTHONPATH=src uv run python -m plot --no-smooth.

known gaps

  • 80K rep 7 missing for both trees, 9 reps instead of 10 at that size.
  • Logs (results/reps10.log, results/smoke.log) are untracked.

Autofix with [code]smith
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.

@prdai
prdai marked this pull request as ready for review November 15, 2026 18:44
@prdai
prdai merged commit d0bf09e into main Nov 15, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant