Skip to content

fix: straight-line calibration fallback for references with few distinct peptidoforms - #138

Merged
RobbinBouwmeester merged 1 commit into
mainfrom
fix/calibration-min-reference
Nov 17, 2026
Merged

RobbinBouwmeester merged 1 commit into
mainfrom
fix/calibration-min-reference

Conversation

@RobbinBouwmeester

Copy link
Copy Markdown
Member

Summary

Fixes the thread-dependent predictions of ±1e7 s reported in #136.

MultiHeadRidgeCalibration combined up to 80 heads with per-head splines and a RidgeCV even when the reference held only a few distinct peptidoforms. Repeated PSMs of one peptidoform counted as separate points. A reference of 1,125 PSMs from 15 peptidoforms, or 169 PSMs from 2, therefore passed every size check. With so few distinct points, the head ranking, the splines and the ridge are underdetermined. Rounding differences in the predictions then decide the fit. These differences come from the torch thread count (about 2e-3 min) and from slicing the FactoredPredictionMatrix (about 1.5e-5 min).

Change

  • New parameter min_distinct=200. Distinct peptidoforms are counted from the source matrix itself, by tolerance grouping (1e-3 min) over the five best-correlating heads, so callers that pass duplicated PSMs (such as MS2Rescore, which calls fit directly) are covered.
  • Below min_distinct: a straight line on the best-correlating head, with a warning (marked unreliable below 10).
  • Below 3 distinct peptidoforms: fit raises CalibrationError.
  • The head cap uses the distinct count instead of the row count.
  • New properties uses_fallback and n_distinct; disagreement returns None in fallback mode.

Why 200

Sweep over 19 held-out setups (holdout_v5, never in training), k = 5 to 2,000 distinct reference peptidoforms, 10 draws per k, test error relative to the setup's RT span:

k ridge fits off by more than the gradient ridge beats best-head straight line
10 32 %
15–20 5–10 %
30 1.6 % 69 %
50–150 0.5 % 78 % (k=50)
≥ 200 0 % ≥ 91 %

From 200 the ridge has no failures and is better than the straight line in at least 91 % of paired draws (median MAE ratio 0.77).

Effect on the #136 PSM list

before after
MS2Rescore default reference (15 peptidoforms): spread across thread counts up to 376 s 0.016 s
same reference: MAE on reference PSMs 208 s 234 s
calibration_set_size≈0.15 reference (2 peptidoforms) ±1e7 s CalibrationError

Tests

5 new tests: duplicated PSMs count once and trigger the fallback; the fallback is insensitive to thread-sized noise; large references still combine heads; refusal below 3; validation of min_distinct. Three existing tests that exercise head combination on 150-row synthetic references now pass min_distinct=3. Full suite: 255 passed.

Follow-up (not in this PR)

MS2Rescore could densify the prediction matrix once and deduplicate the reference before calling fit.

🤖 Generated with Claude Code

…tidoforms

MultiHeadRidgeCalibration combined 80 heads with per-head splines and a ridge
even when the reference held only a handful of distinct peptidoforms. Repeated
PSMs of one peptidoform were counted as separate points, so a reference of
1,125 PSMs from 15 peptidoforms (or 169 PSMs from 2) passed every size check.
With so few distinct points the head ranking, splines and ridge are
underdetermined and rounding differences in the predictions decide the fit,
giving predictions of +-1e7 s that change with the thread count (#136).

Distinct peptidoforms are now counted from the predictions themselves
(tolerance grouping over the five best heads). Below min_distinct=200 the
calibration fits a straight line on the best-correlating head and logs a
warning; below three distinct peptidoforms fit raises CalibrationError. The
head cap now uses the distinct count. Threshold from a sweep over 19 held-out
setups (k = 5..2,000 distinct references, 10 draws each).

Tests that exercise the head combination on small synthetic references opt in
with min_distinct=3.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@RobbinBouwmeester
RobbinBouwmeester merged commit 35025c3 into main Nov 17, 2026
5 checks passed
@RobbinBouwmeester
RobbinBouwmeester deleted the fix/calibration-min-reference branch November 17, 2026 12:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant