From ceaf0a3464a87b638552ddd959a8b76dd2b07e27 Mon Sep 17 00:00:00 2001 From: tannevaled Date: Sat, 31 Oct 2026 21:19:35 +0100 Subject: [PATCH 1/2] conformancefigure: guard the version the total was counted at, not only the total MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `23,511 examples pass` is a count over a POPULATION, and nothing on these pages said which population. The shim resolves ruby/spec's own `ruby_version_is` guards at Ruby 3.4.1, so every example behind a `ruby_version_is "4.0"` guard is removed from the suite before anything is counted. The figure describes 3.4-shaped behaviour while the pages say the project targets Ruby 4.0 semantics and names an MRI 4.0.5/4.0.7 oracle -- all true, and together they read as a 4.0 result. So the version is a claim of the same kind as the number, and it is now checked the same way. Measured here with the published command: $ go run ./tools/conformancefigure --root content/en 4 total claim(s) and 3 version claim(s) checked against the gate's 23511 at Ruby 3.4.1 ## Read from BOTH copies, refusing when they disagree ruby#799 records that the version is spelled twice -- `rubyVersion` in internal/vm/kernel_introspect.go, `CUR_VERSION` in the conformance shim -- and that "today they happen to agree" is the defect waiting to happen. A parity change that moves the runtime and forgets the shim leaves the gate measuring the old population while every page claims the new version, and each half looks fine alone. This reads both and exits 2 naming both values and the issue. That is a refusal, not a verdict on the pages: with the copies apart there is no single version to judge them against, so calling a page wrong would be picking one arbitrarily. ## ⛔ Two defects this change introduced, found by its own tests **The new check swallowed the old one's finding.** Written with the version-absence refusal returning first, the tool answered "found no statement of the version" on a page whose total was wrong by 1,023 -- and the stale figure it exists to catch never reached the output. Disagreements are now printed before absences, with a test for the page that has both. **A claim a line break cut in half was invisible.** On the first real run the tool reported 2 version claims where 3 had been written. The miss was `_index.md`'s sentence, wrapped as … version guards set to resolve at Ruby **3.4.1**, the version this runtime reports … and the scan reads one line at a time. 2 looks exactly like a clean answer; it was found only by insisting on explaining why it was not 3. Markdown prose is re-wrapped by every editor that touches it, so the scan now reads across the break -- attempted only when neither line claims anything alone, so nothing is counted twice. The verbatim sentence is a test. ## The oracle is a different claim and must not match These pages say "differential-tested against MRI Ruby 4.0.5" and "against MRI 4.0.7 on the same host". That is the interpreter rbgo was COMPARED WITH, free to be newer than the version guards resolve at -- and deliberately is. A pattern that read it as the conformance version would report a disagreement that is not one, and once both were wrong in the same direction would report agreement. "against" is absent from every alternative, and three real oracle sentences are negative cases in the vocabulary's own control. ## Also - The known-limitations row quoted **167 examples** behind the 4.0 boundary. ruby#836's census supersedes it: **61 block guards and 5 inline sites** (not 56), and **41** guards asserting pre-4.0 behaviour (not 40). The guard counts are exact; the example count is no longer stated, because a static scan cannot expand `it_behaves_like` and 167 does not reproduce -- it needs the CI sweep ruby#799 asks for. A figure that cannot be reproduced should not be published as if it had been. - The row also now names MRI's latest release, 4.0.9 (2026-10-23), read from ruby/ruby's tag `v4.0.9` -- note the tag naming changed at 4.0.0, from `v3_4_1` to `v4.0.9`. - Security: the two new fetches, and the pre-existing BASELINE fetch, now go through one bounded reader (8 MiB, error rather than truncation -- a truncated file is one whose constants are merely missing, which would be reported as "no version found" and point at the wrong thing). Go's RE2 has no backtracking, so the added patterns cannot be made to run long. - Performance: the join path reuses what each line already yielded instead of running three patterns over every line twice. gofmt clean, `go vet` clean, tests green. `upstreamlinks` re-run: 12 links over 11 module sections, 0 disagree. `numericsurface` could not be re-run here -- the local rbgo is from 2026-08-30 and refuses on a constant it does not have, which is the tool declining to answer rather than answering wrongly; CI builds rbgo@main. The edits are in `_index.md`'s intro and `conformance.md`, neither of which is one of the eleven module sections those two tools walk. Co-Authored-By: Claude Opus 5 --- README.md | 2 +- content/en/_index.md | 8 +- content/en/conformance.md | 7 +- tools/conformancefigure/main.go | 338 ++++++++++++++++++++++++--- tools/conformancefigure/main_test.go | 197 +++++++++++++++- 5 files changed, 506 insertions(+), 46 deletions(-) diff --git a/README.md b/README.md index d112324..092b633 100644 --- a/README.md +++ b/README.md @@ -25,7 +25,7 @@ after publication. | tool | what it judges | its authority | |---|---|---| -| `conformancefigure` | every **present-tense** claim about the ruby/spec total. A *dated* sweep going stale is not an error, and is exempt by its date. | the gate's own `BASELINE` | +| `conformancefigure` | every **present-tense** claim about the ruby/spec total, **and the Ruby version that total was counted at** — the version selects the population before anything is counted, so a total stated without it is half a measurement. A *dated* sweep going stale is not an error, and is exempt by its date. The differential **oracle** named on these pages is a different claim, free to be newer, and is deliberately not matched. | the gate's own `BASELINE`, and the version read from **both** files that spell it — the runtime's `rubyVersion` and the shim's `CUR_VERSION` — refusing if they disagree ([ruby#799](https://github.com/go-embedded-ruby/ruby/issues/799)) | | `docexample` | every fenced `ruby` block that claims an answer in a `# =>` comment is **run**, and the lines it prints compared in order. A block that claims nothing is skipped. | a real interpreter | | `numericsurface` | every method name the prose **prints** in a module's section, and the counts the pages state. | a real interpreter | | `upstreamlinks` | every **library a module section links to** is one rbgo still depends on. | rbgo's `go.mod` | diff --git a/content/en/_index.md b/content/en/_index.md index 2cd0137..8449d16 100644 --- a/content/en/_index.md +++ b/content/en/_index.md @@ -12,7 +12,13 @@ C toolchain and no libruby. **How complete is it?** It passes **23,511** of ruby/spec's `language/` + `core/` examples — a large and growing subset of Ruby, and **not** a drop-in replacement for -CRuby. [Conformance]({{< relref "conformance.md#where-it-stands-measured-2026-09-26" >}}) states +CRuby. That count is taken with ruby/spec's own version guards set to resolve at +Ruby **3.4.1**, the version this runtime reports: every example behind a +`ruby_version_is "4.0"` guard is removed from the suite before anything is +counted, so the figure describes 3.4-shaped behaviour even though the project +targets 4.0 semantics — 61 block guards and 5 inline sites sit at that boundary +([ruby#799](https://github.com/go-embedded-ruby/ruby/issues/799)). +[Conformance]({{< relref "conformance.md#where-it-stands-measured-2026-09-26" >}}) states exactly what that counts and what it leaves out, and [Known limitations]({{< relref "conformance.md#known-limitations" >}}) is worth reading before you depend on it. diff --git a/content/en/conformance.md b/content/en/conformance.md index ee5461f..6c52adc 100644 --- a/content/en/conformance.md +++ b/content/en/conformance.md @@ -147,6 +147,11 @@ workflow enforces it on every push and pull request. {{< callout type="info" >}} **The gate's current total is **23 511**** +Counted with ruby/spec's version guards set to resolve at Ruby **3.4.1** — the +version `RUBY_VERSION` reports and the version the shim measures at. That +selects the population before any counting happens, so the total is not a +percentage of Ruby 4.0; see [Known limitations](#known-limitations). + The table below is one **dated full sweep**, and a dated measurement going stale is not an error: it is what it says it is. The number that moves with every merge is the per-file baseline's sum, which the badge and the README @@ -286,7 +291,7 @@ the call returns successfully and nothing says the guarantee is missing: | | | | --- | --- | -| `RUBY_VERSION` reports `"3.4.1"`, though the differential oracle is MRI 4.0.7 and the pinned corpus is 4.0.x. That is not only cosmetic: the shim's version guard is pinned to the same `3.4.1`, so **167 examples behind a `ruby_version_is "4.0"` guard are never reached**, while 40 guards that end *before* 4.0 are being asserted | [#799](https://github.com/go-embedded-ruby/ruby/issues/799) | +| `RUBY_VERSION` reports `"3.4.1"`, though the differential oracle is MRI 4.0.7, the pinned corpus is 4.0.x, and the latest MRI release is 4.0.9 (2026-10-23). That is not only cosmetic: the shim's `CUR_VERSION` is pinned to the same `3.4.1`, so **61 block guards and 5 inline sites at the `ruby_version_is "4.0"` boundary never run**, while **41 guards that end *before* 4.0 are being asserted**. Guard counts are exact, censused by `scripts/conformance/rubyspec/cmd/guards`; the example count behind them is not stated here because a static scan cannot expand `it_behaves_like`, and the figure of 167 quoted when this was first reported does not reproduce — it needs a CI sweep that moves `CUR_VERSION` without regenerating the baseline, so the ratchet names every file that moves in both directions | [#799](https://github.com/go-embedded-ruby/ruby/issues/799) | | `Errno` carries all **158** of MRI's constant names, but **32** report errno `0` where MRI has a real number (`EAUTH`, `EBADRPC`, `EDEVERR`, …). By set difference on darwin: 81 names are zero-valued under rbgo, 49 under MRI, and every MRI zero is also zero under rbgo | [#633](https://github.com/go-embedded-ruby/ruby/issues/633) | | `Thread#backtrace` answers for the **current** thread only; another thread raises `NotImplementedError` | — | | `Process.fork` **does not exist** — Go's runtime cannot be forked safely | — | diff --git a/tools/conformancefigure/main.go b/tools/conformancefigure/main.go index 4fa8f8c..e52c6b7 100644 --- a/tools/conformancefigure/main.go +++ b/tools/conformancefigure/main.go @@ -1,5 +1,6 @@ // conformancefigure checks that every CURRENT claim about rbgo's ruby/spec -// total agrees with the number the gate actually records. +// total — and about the Ruby version that total was counted at — agrees with +// what the gate actually records. // // Why it exists: on 2026-10-11 this site said rbgo passed **22 488** examples on // its front page and **23 481** "today" in two other places, while the gate's @@ -23,6 +24,7 @@ package main import ( "bufio" + "bytes" "errors" "fmt" "io" @@ -43,6 +45,41 @@ import ( // are themselves downstream of it. const baselineURL = "https://raw.githubusercontent.com/go-embedded-ruby/ruby/main/scripts/conformance/rubyspec/BASELINE" +// The total is only half of what a reader needs. `23,511 examples pass` is a +// count over a POPULATION, and the population is selected by the Ruby version +// the shim resolves `ruby_version_is` guards at: at 3.4.1 every spec behind +// `ruby_version_is "4.0"` is removed before anything is counted, so the figure +// is a 3.4-shaped result whichever MRI the project says it targets. Measured +// 2026-10-27 (ruby#799, ruby#836): 61 block guards and 5 inline sites sit at +// the "4.0" boundary, and 41 guards assert what 4.0 changed away from. +// +// So the version is a claim of the same kind as the number, and it is read from +// BOTH places that spell it: +// +// internal/vm/kernel_introspect.go rubyVersion -- what the runtime REPORTS +// scripts/.../spec_helper.rb CUR_VERSION -- what the gate MEASURES AT +// +// ⛔ These are two hardcoded copies of one fact. ruby#799 asks for the second +// to be derived from the first precisely because "today they happen to agree" +// is the defect waiting to happen: a parity change that moves the runtime and +// forgets the shim would leave the gate measuring the old population while +// every page claimed the new version, and nothing would say so. Reading both +// and refusing when they differ is why this tool exists at all -- two readers +// of one fact cannot drift, where two transcriptions already had. +const ( + runtimeURL = "https://raw.githubusercontent.com/go-embedded-ruby/ruby/main/internal/vm/kernel_introspect.go" + shimURL = "https://raw.githubusercontent.com/go-embedded-ruby/ruby/main/scripts/conformance/rubyspec/spec_helper.rb" +) + +// Anchored at the start of a line so a mention inside a comment or a string -- +// both files discuss these constants at length -- cannot be read as the +// definition. The runtime's is indented inside a const block; the shim's is at +// column zero. +var ( + runtimeVersionRe = regexp.MustCompile(`(?m)^\trubyVersion\s*=\s*"(\d+\.\d+\.\d+)"`) + shimVersionRe = regexp.MustCompile(`(?m)^CUR_VERSION\s*=\s*"(\d+\.\d+\.\d+)"`) +) + // Two shapes of CURRENT claim, because two sites phrase it two ways round. // // claimBefore: the words come first -- "It passes 23,511", "whose sum is @@ -75,6 +112,52 @@ var ( // is a known limit and not a surprise. var datedLine = regexp.MustCompile(`(?i)measured\s+\d{4}-\d{2}-\d{2}`) +// The version claim, in the three shapes these sites use for it. +// +// ⛔ THE ORACLE IS A DIFFERENT CLAIM AND MUST NOT MATCH. These pages say +// "differential-tested against MRI Ruby 4.0.5" and "against MRI 4.0.7 on the +// same host": that is the interpreter rbgo's behaviour was COMPARED WITH, which +// is free to be newer than the version the shim resolves guards at -- and is, +// deliberately. A pattern that read "against MRI 4.0.5" as the conformance +// version would compare the oracle against CUR_VERSION and report a +// disagreement that is not one; worse, once both were wrong in the same +// direction it would report agreement. So "against" is deliberately absent from +// every alternative below, and three oracle sentences are negative cases. +var versionClaim = []*regexp.Regexp{ + // "those guards resolve at Ruby 3.4.1" + regexp.MustCompile(`(?i)\bresolve[sd]?\s+at\b[^0-9]{0,24}?\**v?(\d+\.\d+\.\d+)\**`), + // "the suite runs as Ruby 3.4.1" + regexp.MustCompile(`(?i)\bruns?\s+as\b[^0-9]{0,24}?\**v?(\d+\.\d+\.\d+)\**`), + // "`RUBY_VERSION` reports `\"3.4.1\"`" and "`CUR_VERSION` is 3.4.1" -- the + // constants named outright, which is how the known-defect tables phrase it. + regexp.MustCompile("(?i)`?(?:RUBY_VERSION|CUR_VERSION)`?\\s*(?:reports|is|=)\\s*\\**`?\"?(\\d+\\.\\d+\\.\\d+)"), +} + +// versionCases is the version vocabulary's own control, in the same spirit as +// vocabularyCases below and run on every invocation for the same reason. +var versionCases = []struct { + line string + want bool + v string +}{ + // The three shapes, as the pages will carry them. + {"and those guards **resolve at Ruby 3.4.1**, not at the version the oracle runs.", true, "3.4.1"}, + {"the suite runs as Ruby **3.4.1** while the oracle is newer", true, "3.4.1"}, + {"| `RUBY_VERSION` reports `\"3.4.1\"`, though the differential oracle is MRI 4.0.7 |", true, "3.4.1"}, + {"`CUR_VERSION` is 3.4.1, which selects the population before anything is counted", true, "3.4.1"}, + // ⛔ The oracle. Three real sentences from these pages, all of which name a + // DIFFERENT and newer MRI on purpose. + {"All behaviour is differential-tested against MRI Ruby 4.0.5 (where MRI has an", false, ""}, + {"Each entry below was re-run on `4872e01` against **MRI 4.0.7 on the same host**", false, ""}, + {"- with **every feature differential-tested against MRI 4.0.5**.", false, ""}, + // A target stated without a patch level is a statement of intent about + // semantics, not of the version a guard resolves at. + {"go-embedded-ruby targets **Ruby 4.0** semantics. Correctness is not asserted by", false, ""}, + // The total, and a benchmark table: numbers that are not versions. + {`

23,511 ruby/spec language/ + core/ examples pass`, false, ""}, + {"| module.op | rbgo | MRI 4.0.5 | MRI+YJIT | JRuby 10.1 | TruffleRuby 34.0.1 | rbgo/MRI |", false, ""}, +} + // vocabularyCases is the guard's own control, run before it judges anything. // Each is a line this tool will meet; `want` is whether a current claim should // be recognised in it. @@ -125,6 +208,61 @@ func match(line string) (string, bool) { return "", false } +// matchVersion returns the version a line claims the gate resolves guards at. +// A dated line is exempt for the same reason a dated total is. +func matchVersion(line string) (string, bool) { + if datedLine.MatchString(line) { + return "", false + } + for _, re := range versionClaim { + if m := re.FindStringSubmatch(line); m != nil { + return m[1], true + } + } + return "", false +} + +// allVersions returns every version claim on a line. +func allVersions(line string) []string { + if datedLine.MatchString(line) { + return nil + } + var out []string + for _, re := range versionClaim { + for _, m := range re.FindAllStringSubmatch(line, -1) { + out = append(out, m[1]) + } + } + return out +} + +// versionsAcross reads a claim that a line break cut in half. +// +// ⛔ Measured 2026-10-31, on this tool's own first run over the real pages: a +// sentence was written as +// +// … version guards set to resolve at +// Ruby **3.4.1**, the version this runtime reports … +// +// and the scan, which reads one line at a time, saw neither half. The tool +// reported "2 version claim(s)" where three had been written, and 2 is a +// perfectly clean-looking number — the miss was found only by insisting on +// explaining why it was not 3. Markdown prose is re-wrapped by every editor +// that touches it, so this is not a one-off typo; it is the normal state of a +// paragraph. +// +// It is attempted only when NEITHER line claims anything on its own, so a claim +// is never counted twice, and the join is attributed to the second line because +// that is where the version itself is written. The caller passes what each line +// already yielded rather than having this recompute it: three patterns over +// every line of every page, twice, for a result already in hand. +func versionsAcross(prev string, prevHad bool, line string, lineHad bool) []string { + if prev == "" || prevHad || lineHad { + return nil + } + return allVersions(prev + " " + line) +} + // allMatches returns every current-total figure on a line, so a line carrying // two claims is not judged on the first alone. func allMatches(line string) []string { @@ -169,6 +307,15 @@ func checkVocabulary() error { return &exitError{2, fmt.Sprintf("the vocabulary read %q as %q, want %q", c.line, n, c.n)} } } + for _, c := range versionCases { + v, got := matchVersion(c.line) + if got != c.want { + return &exitError{2, fmt.Sprintf("the version vocabulary is wrong: %q matched=%v, want %v", c.line, got, c.want)} + } + if c.want && v != c.v { + return &exitError{2, fmt.Sprintf("the version vocabulary read %q as %q, want %q", c.line, v, c.v)} + } + } return nil } @@ -176,22 +323,31 @@ func checkVocabulary() error { // SetOut and SetErr instead of a subprocess. func RootCmd() *cobra.Command { var ( - root string - baseline string - timeout time.Duration + root string + baseline string + runtimePath string + shimPath string + timeout time.Duration ) cmd := &cobra.Command{ Use: "conformancefigure", - Short: "Check that every stated ruby/spec total agrees with the gate's BASELINE", + Short: "Check the stated ruby/spec total, and the version it was counted at, against the gate", Long: "conformancefigure reads the present-tense claims this site makes about\n" + "rbgo's ruby/spec total and compares each with the sum of the gate's own\n" + "per-file baseline. A dated sweep going stale is not an error and is\n" + - "exempt by its date.", + "exempt by its date.\n\n" + + "It also checks the Ruby version the gate resolves `ruby_version_is`\n" + + "guards at, because that version selects the population the total counts\n" + + "over. The version is read from both files that spell it — the runtime's\n" + + "`rubyVersion` and the shim's `CUR_VERSION` — and a disagreement between\n" + + "them is a refusal, not a verdict on these pages.\n\n" + + "The differential oracle named on these pages is a DIFFERENT claim, free\n" + + "to be newer, and is deliberately not matched.", SilenceUsage: true, SilenceErrors: true, Args: cobra.NoArgs, RunE: func(cmd *cobra.Command, _ []string) error { - return check(cmd.OutOrStdout(), cmd.ErrOrStderr(), root, baseline, timeout) + return check(cmd.OutOrStdout(), cmd.ErrOrStderr(), root, baseline, runtimePath, shimPath, timeout) }, } // content/en since the move to Hugo (#28); it was docs/ under MkDocs, and a @@ -199,11 +355,13 @@ func RootCmd() *cobra.Command { // this tool refused with "lstat docs: no such file". cmd.Flags().StringVar(&root, "root", "content/en", "the directory of Markdown and HTML to check") cmd.Flags().StringVar(&baseline, "baseline", "", "read the authoritative BASELINE from this path instead of fetching it") + cmd.Flags().StringVar(&runtimePath, "runtime", "", "read kernel_introspect.go from this path instead of fetching it") + cmd.Flags().StringVar(&shimPath, "shim", "", "read spec_helper.rb from this path instead of fetching it") cmd.Flags().DurationVar(&timeout, "timeout", 30*time.Second, "how long to wait for the fetch") return cmd } -func check(stdout, stderr io.Writer, root, baseline string, timeout time.Duration) error { +func check(stdout, stderr io.Writer, root, baseline, runtimePath, shimPath string, timeout time.Duration) error { // The control first: if the vocabulary is wrong, every verdict after it is // worthless. if err := checkVocabulary(); err != nil { @@ -217,13 +375,23 @@ func check(stdout, stderr io.Writer, root, baseline string, timeout time.Duratio return &exitError{2, fmt.Sprintf("cannot read the gate's total: %v", err)} } + wantVersion, err := authoritativeVersion(runtimePath, shimPath, timeout) + if err != nil { + return &exitError{2, fmt.Sprintf("cannot read the version the gate measures at: %v", err)} + } + type finding struct { where string got int text string } + type versionFinding struct { + where string + got string + } var wrong []finding - checked := 0 + var wrongVersion []versionFinding + checked, versionsChecked := 0, 0 err = filepath.WalkDir(root, func(path string, d fs.DirEntry, err error) error { if err != nil { @@ -253,6 +421,9 @@ func check(stdout, stderr io.Writer, root, baseline string, timeout time.Duratio defer f.Close() sc := bufio.NewScanner(f) sc.Buffer(make([]byte, 1<<20), 1<<20) // roadmap.md has very long lines + // Carried across lines so a claim a line break cut in half is still read; + // prevHadVersion keeps that join from counting a claim twice. + prev, prevHadVersion := "", false for line := 1; sc.Scan(); line++ { text := sc.Text() for _, fig := range allMatches(text) { @@ -269,6 +440,19 @@ func check(stdout, stderr io.Writer, root, baseline string, timeout time.Duratio }) } } + own := allVersions(text) + ownHad := len(own) > 0 + versions := append(own, versionsAcross(prev, prevHadVersion, text, ownHad)...) + prev, prevHadVersion = text, ownHad + for _, v := range versions { + versionsChecked++ + if v != wantVersion { + wrongVersion = append(wrongVersion, versionFinding{ + where: fmt.Sprintf("%s:%d", path, line), + got: v, + }) + } + } } return sc.Err() }) @@ -278,16 +462,35 @@ func check(stdout, stderr io.Writer, root, baseline string, timeout time.Duratio // "N of M, never N alone": a walk that matched nothing reports no // disagreement, which is the same output as a correct site. - fmt.Fprintf(stdout, "%d current claim(s) checked against the gate's %d\n", checked, want) + fmt.Fprintf(stdout, "%d total claim(s) and %d version claim(s) checked against the gate's %d at Ruby %s\n", + checked, versionsChecked, want, wantVersion) + // ⛔ THE DISAGREEMENTS ARE REPORTED BEFORE THE ABSENCES, and that order is + // the whole of it. Written the other way round -- the version-absence + // refusal returning first -- this tool answered "found no statement of the + // version" on a page whose TOTAL was wrong by 1,023, and the stale figure it + // exists to catch never reached the output. A new check must not be able to + // swallow the finding of the old one: a concrete disagreement outranks a + // structural complaint about what the page does not say. + for _, w := range wrong { + fmt.Fprintf(stderr, "%s: claims %d, the gate records %d\n %s\n", w.where, w.got, want, w.text) + } + for _, w := range wrongVersion { + fmt.Fprintf(stderr, "%s: says the gate measures at %s, it measures at %s\n", w.where, w.got, wantVersion) + } + if n := len(wrong) + len(wrongVersion); n > 0 { + return &exitError{1, fmt.Sprintf("%d claim(s) disagree with the ruby repository", n)} + } if checked == 0 { return &exitError{1, fmt.Sprintf("found no statement of the total in %s; "+ "either the vocabulary no longer matches how the docs phrase it, or the figure was dropped", root)} } - for _, w := range wrong { - fmt.Fprintf(stderr, "%s: claims %d, the gate records %d\n %s\n", w.where, w.got, want, w.text) - } - if len(wrong) > 0 { - return &exitError{1, fmt.Sprintf("%d claim(s) disagree with %s", len(wrong), baselineURL)} + // The same refusal, for the same reason: a site that states a total and + // never says which version selected the population is stating half a + // measurement, and a version vocabulary that matches nothing is + // indistinguishable from a site that states it correctly. + if versionsChecked == 0 { + return &exitError{1, fmt.Sprintf("found no statement of the version the gate measures at in %s; "+ + "a total without it is a count over an unnamed population (Ruby %s today)", root, wantVersion)} } return nil } @@ -318,31 +521,16 @@ func main() { // does, deliberately — two readers of one file cannot drift, where two // transcriptions of one number already have. func authoritative(path string, timeout time.Duration) (int, error) { - var r io.Reader - if path != "" { - f, err := os.Open(path) - if err != nil { - return 0, err - } - defer f.Close() - r = f - } else { - cl := &http.Client{Timeout: timeout} - resp, err := cl.Get(baselineURL) - if err != nil { - return 0, err - } - defer resp.Body.Close() - if resp.StatusCode != http.StatusOK { - // A 404 body is still a body: without this the scan below would read - // GitHub's error page, find no OK rows, and report a total of 0. - return 0, fmt.Errorf("%s: HTTP %d", baselineURL, resp.StatusCode) - } - r = resp.Body + // Through the same bounded fetch as the version files: this path used to + // stream the response straight into a Scanner, which is an unbounded read of + // a remote answer. The baseline is ~2,200 short rows. + b, err := fetch(path, baselineURL, timeout) + if err != nil { + return 0, err } total, rows := 0, 0 - sc := bufio.NewScanner(r) + sc := bufio.NewScanner(bytes.NewReader(b)) for sc.Scan() { parts := strings.Split(sc.Text(), "\t") if len(parts) != 3 || parts[0] != "OK" { @@ -366,6 +554,82 @@ func authoritative(path string, timeout time.Duration) (int, error) { return total, nil } +// fetch returns the bytes of a local path when one is given, and otherwise of +// the URL. The 404 guard is the same one authoritative() needed and for the +// same reason: GitHub answers a missing file with a page, and a page a pattern +// does not match reads as "the constant is not there" rather than as "the +// request was wrong". +// maxFetch bounds what a remote answer may cost us. The three files read here +// are tens of kilobytes; 8 MiB is far above any of them and far below what an +// unbounded io.ReadAll would let a redirected or replaced URL spend. Reading +// one byte past the limit is an error rather than a truncation, because a +// truncated file is a file whose constants may simply be missing — which this +// tool would then report as "no version found", a refusal that points at the +// wrong thing. +const maxFetch = 8 << 20 + +func fetch(path, url string, timeout time.Duration) ([]byte, error) { + if path != "" { + return os.ReadFile(path) + } + cl := &http.Client{Timeout: timeout} + resp, err := cl.Get(url) + if err != nil { + return nil, err + } + defer resp.Body.Close() + if resp.StatusCode != http.StatusOK { + // A 404 body is still a body: without this the callers would read + // GitHub's error page and find no rows and no constants in it, which + // reads as an answer rather than as a failed request. + return nil, fmt.Errorf("%s: HTTP %d", url, resp.StatusCode) + } + b, err := io.ReadAll(io.LimitReader(resp.Body, maxFetch+1)) + if err != nil { + return nil, err + } + if len(b) > maxFetch { + return nil, fmt.Errorf("%s: larger than %d bytes", url, maxFetch) + } + return b, nil +} + +// authoritativeVersion reports the Ruby version the conformance population is +// selected by, read from both files that spell it and refusing when they +// disagree. +// +// It returns an error rather than a best guess in three cases, and each is a +// refusal to answer rather than a verdict: the file could not be read, the +// constant was not found in it (a rename must not read as "no version"), or the +// two copies differ. The third is the one worth having: it is the exact state +// ruby#799 describes as waiting to happen, and on the day half a parity change +// lands this is what says so. +func authoritativeVersion(runtimePath, shimPath string, timeout time.Duration) (string, error) { + rt, err := fetch(runtimePath, runtimeURL, timeout) + if err != nil { + return "", fmt.Errorf("the runtime's version: %w", err) + } + sh, err := fetch(shimPath, shimURL, timeout) + if err != nil { + return "", fmt.Errorf("the shim's version: %w", err) + } + rm := runtimeVersionRe.FindSubmatch(rt) + if rm == nil { + return "", fmt.Errorf("no `rubyVersion = \"x.y.z\"` in %s", runtimeURL) + } + sm := shimVersionRe.FindSubmatch(sh) + if sm == nil { + return "", fmt.Errorf("no `CUR_VERSION = \"x.y.z\"` in %s", shimURL) + } + runtime, shim := string(rm[1]), string(sm[1]) + if runtime != shim { + return "", fmt.Errorf("the two copies of the version disagree: the runtime reports %s "+ + "(kernel_introspect.go) while the gate measures at %s (spec_helper.rb) — "+ + "half a version change has landed; see ruby#799", runtime, shim) + } + return runtime, nil +} + // hasAnySuffix is spelled out so the file types this walks are a list in one // place rather than a chain of && that grows a bug each time it is extended. func hasAnySuffix(path string, suffixes ...string) bool { diff --git a/tools/conformancefigure/main_test.go b/tools/conformancefigure/main_test.go index 443b54c..b9fddf9 100644 --- a/tools/conformancefigure/main_test.go +++ b/tools/conformancefigure/main_test.go @@ -51,6 +51,32 @@ func TestTheVocabularyControlBites(t *testing.T) { }) } +// versionFixtures writes the two files the version check reads, in the shape +// the ruby repository spells them. +// +// ⛔ Every test that gets as far as the version check must pass these. Without +// them the check fetches raw.githubusercontent.com, and a test that reaches the +// network is a test whose verdict depends on someone else's main branch and on +// having a connection — it read "at Ruby 3.4.1" from the live repository when +// this was first written, which is the right answer arrived at the wrong way. +func versionFixtures(t *testing.T, runtimeV, shimV string) (string, string) { + t.Helper() + dir := t.TempDir() + rp := filepath.Join(dir, "kernel_introspect.go") + // A tab, because the pattern is anchored to the indented const block: the + // file discusses `rubyVersion` in prose above it. + body := "// the comment above mentions rubyVersion = \"9.9.9\" in prose\nconst (\n" + + "\trubyVersion = \"" + runtimeV + "\"\n\treleaseDate = \"2024-12-25\"\n)\n" + if err := os.WriteFile(rp, []byte(body), 0o644); err != nil { + t.Fatal(err) + } + sp := filepath.Join(dir, "spec_helper.rb") + if err := os.WriteFile(sp, []byte("CUR_VERSION = \""+shimV+"\"\nPLATFORMS = [:darwin]\n"), 0o644); err != nil { + t.Fatal(err) + } + return rp, sp +} + // ⛔ An unreadable baseline is a refusal (2), never a pass: it finds no // disagreement either, and that silence reads exactly like agreement. func TestAnUnreadableBaselineRefuses(t *testing.T) { @@ -104,11 +130,11 @@ func TestItBites(t *testing.T) { }{ { name: "a page stating the baseline's own total", - md: "It passes **23,511** of ruby/spec's examples.\n", + md: "It passes **23,511** of ruby/spec's examples, and those guards resolve at Ruby 3.4.1.\n", }, { name: "a page stating a different total", - md: "It passes **22,488** of ruby/spec's examples.\n", + md: "It passes **22,488** of ruby/spec's examples, and those guards resolve at Ruby 3.4.1.\n", wantCode: 1, wantMsg: "claims 22488, the gate records 23511", }, @@ -120,7 +146,37 @@ func TestItBites(t *testing.T) { }, { name: "a DATED sweep going stale is not an error", - md: "It passes **23,511** today.\n\n**22,488 examples pass**, measured 2026-09-26 on main\n", + md: "It passes **23,511** today, and those guards resolve at Ruby 3.4.1.\n\n" + + "**22,488 examples pass**, measured 2026-09-26 on main\n", + }, + // ⛔ The reason the disagreements are printed before the absences. Written + // the other way round this case answered "found no statement of the + // version" and never mentioned that the total was wrong by 1,023 — a new + // check swallowing the finding of the old one. + { + name: "a wrong total is named even when the version is not stated", + md: "It passes **22,488** of ruby/spec's examples.\n", + wantCode: 1, + wantMsg: "claims 22488, the gate records 23511", + }, + { + name: "a page stating the total but not the version it was measured at", + md: "It passes **23,511** of ruby/spec's examples.\n", + wantCode: 1, + wantMsg: "found no statement of the version the gate measures at", + }, + { + name: "a page stating a stale version", + md: "It passes **23,511** of ruby/spec's examples, and those guards resolve at Ruby 4.0.9.\n", + wantCode: 1, + wantMsg: "says the gate measures at 4.0.9, it measures at 3.4.1", + }, + // The oracle is a different claim and is named on these pages on purpose. + // It must not be read as the version the gate measures at. + { + name: "the differential oracle may be newer than the gate's version", + md: "It passes **23,511** of ruby/spec's examples, and those guards resolve at Ruby 3.4.1.\n\n" + + "All behaviour is differential-tested against MRI Ruby 4.0.5.\n", }, } @@ -130,9 +186,11 @@ func TestItBites(t *testing.T) { if err := os.WriteFile(filepath.Join(root, "page.md"), []byte(c.md), 0o644); err != nil { t.Fatal(err) } + rp, sp := versionFixtures(t, "3.4.1", "3.4.1") var out, errb bytes.Buffer cmd := RootCmd() - cmd.SetArgs([]string{"--root", root, "--baseline", base, "--timeout", "5s"}) + cmd.SetArgs([]string{"--root", root, "--baseline", base, + "--runtime", rp, "--shim", sp, "--timeout", "5s"}) cmd.SetOut(&out) cmd.SetErr(&errb) err := cmd.Execute() @@ -157,6 +215,130 @@ func TestItBites(t *testing.T) { } } +// ⛔ THE CASE THIS CHECK EXISTS FOR. ruby#799: the version is spelled twice, +// once in the runtime and once in the conformance shim, and "today they happen +// to agree" is the defect waiting to happen. A parity change that moves +// `rubyVersion` to 4.0.9 and forgets `CUR_VERSION` leaves the gate measuring the +// 3.4 population while every page claims 4.0.9 — and both halves look fine on +// their own. Reading both and REFUSING (2, not 1) is the only moment anything +// says so. +func TestTheTwoCopiesOfTheVersionMustAgree(t *testing.T) { + dir := t.TempDir() + base := filepath.Join(dir, "BASELINE") + if err := os.WriteFile(base, []byte("OK\ta_spec.rb\t23511\n"), 0o644); err != nil { + t.Fatal(err) + } + root := t.TempDir() + if err := os.WriteFile(filepath.Join(root, "page.md"), + []byte("It passes **23,511**, and those guards resolve at Ruby 4.0.9.\n"), 0o644); err != nil { + t.Fatal(err) + } + + // The runtime moved, the shim did not. + rp, sp := versionFixtures(t, "4.0.9", "3.4.1") + var out, errb bytes.Buffer + cmd := RootCmd() + cmd.SetArgs([]string{"--root", root, "--baseline", base, + "--runtime", rp, "--shim", sp, "--timeout", "5s"}) + cmd.SetOut(&out) + cmd.SetErr(&errb) + err := cmd.Execute() + + var ee *exitError + if !errors.As(err, &ee) { + t.Fatalf("two disagreeing copies were accepted: %v\nstdout: %s", err, out.String()) + } + // 2, not 1: it is a refusal to answer. With the two copies apart there is no + // single version for the pages to be judged against, so calling the page + // wrong would be picking one arbitrarily. + if ee.code != 2 { + t.Errorf("status %d, want 2", ee.code) + } + for _, want := range []string{"4.0.9", "3.4.1", "ruby#799"} { + if !bytes.Contains([]byte(ee.msg), []byte(want)) { + t.Errorf("the refusal %q does not name %q", ee.msg, want) + } + } +} + +// ⛔ A claim a line break cut in half. This is the sentence this tool missed on +// its own first run over the real pages, verbatim: it reported two version +// claims where three had been written, and two looks exactly like a clean +// answer. Markdown prose is re-wrapped by every editor that touches it, so the +// scan must read across the break. +func TestAVersionClaimSplitAcrossLinesIsSeen(t *testing.T) { + dir := t.TempDir() + base := filepath.Join(dir, "BASELINE") + if err := os.WriteFile(base, []byte("OK\ta_spec.rb\t23511\n"), 0o644); err != nil { + t.Fatal(err) + } + root := t.TempDir() + md := "It passes **23,511** of ruby/spec's examples. That count is taken\n" + + "with ruby/spec's own version guards set to resolve at\n" + + "Ruby **4.0.9**, the version this runtime reports.\n" + if err := os.WriteFile(filepath.Join(root, "page.md"), []byte(md), 0o644); err != nil { + t.Fatal(err) + } + + // The fixtures say 3.4.1, the split sentence says 4.0.9: seeing the claim at + // all is what makes this fail, which is the point. A tool blind to the break + // would exit 1 saying no version was stated — a different message, so the + // assertion below cannot pass by accident. + rp, sp := versionFixtures(t, "3.4.1", "3.4.1") + var out, errb bytes.Buffer + cmd := RootCmd() + cmd.SetArgs([]string{"--root", root, "--baseline", base, + "--runtime", rp, "--shim", sp, "--timeout", "5s"}) + cmd.SetOut(&out) + cmd.SetErr(&errb) + err := cmd.Execute() + + var ee *exitError + if !errors.As(err, &ee) { + t.Fatalf("a split version claim was not seen: %v\nstdout: %s", err, out.String()) + } + if said := errb.String(); !bytes.Contains([]byte(said), []byte("says the gate measures at 4.0.9")) { + t.Errorf("it said %q, want it to name the split claim", said) + } + // And it is counted once, not once per line of the join. + if got := out.String(); !bytes.Contains([]byte(got), []byte("1 version claim(s)")) { + t.Errorf("counted %q, want exactly one version claim", got) + } +} + +// A renamed constant must read as "I cannot tell", not as a version. A pattern +// that silently matches nothing is how a guard stops guarding. +func TestAMissingVersionConstantRefuses(t *testing.T) { + dir := t.TempDir() + base := filepath.Join(dir, "BASELINE") + if err := os.WriteFile(base, []byte("OK\ta_spec.rb\t23511\n"), 0o644); err != nil { + t.Fatal(err) + } + fix := t.TempDir() + rp := filepath.Join(fix, "kernel_introspect.go") + if err := os.WriteFile(rp, []byte("const (\n\tmriVersion = \"4.0.9\"\n)\n"), 0o644); err != nil { + t.Fatal(err) + } + sp := filepath.Join(fix, "spec_helper.rb") + if err := os.WriteFile(sp, []byte("CUR_VERSION = \"4.0.9\"\n"), 0o644); err != nil { + t.Fatal(err) + } + + cmd := RootCmd() + cmd.SetArgs([]string{"--root", "testdata", "--baseline", base, + "--runtime", rp, "--shim", sp, "--timeout", "5s"}) + cmd.SetOut(&bytes.Buffer{}) + cmd.SetErr(&bytes.Buffer{}) + err := cmd.Execute() + var ee *exitError + if !errors.As(err, &ee) || ee.code != 2 { + t.Fatalf("error %v, want status 2", err) + } + if !bytes.Contains([]byte(ee.msg), []byte("rubyVersion")) { + t.Errorf("the refusal %q does not name the constant it looked for", ee.msg) + } +} + // A positional argument is a mistake, not an extra root: the flag is --root. func TestAnArgumentIsRefused(t *testing.T) { cmd := RootCmd() @@ -201,14 +383,17 @@ func TestATomlFileIsWalked(t *testing.T) { root := t.TempDir() // A params.phases blurb, as the landing writes it. toml := "[[params.phases]]\n name = \"Phase 7\"\n" + - " blurb = \"measured against ruby/spec: **23,481 examples pass** — the total that BASELINE sums to.\"\n" + " blurb = \"measured against ruby/spec: **23,481 examples pass** — the total that BASELINE sums to, " + + "and those guards resolve at Ruby 3.4.1.\"\n" if err := os.WriteFile(filepath.Join(root, "hugo.toml"), []byte(toml), 0o644); err != nil { t.Fatal(err) } + rp, sp := versionFixtures(t, "3.4.1", "3.4.1") var out, errb bytes.Buffer cmd := RootCmd() - cmd.SetArgs([]string{"--root", root, "--baseline", base, "--timeout", "5s"}) + cmd.SetArgs([]string{"--root", root, "--baseline", base, + "--runtime", rp, "--shim", sp, "--timeout", "5s"}) cmd.SetOut(&out) cmd.SetErr(&errb) err := cmd.Execute() From 8b409701e382155ec4e90a0b28ef3b4c344af625 Mon Sep 17 00:00:00 2001 From: tannevaled Date: Sat, 31 Oct 2026 21:42:37 +0100 Subject: [PATCH 2/2] conformancefigure: recognise the shape this check will meet when its defect is fixed MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The check added in the previous commit reads the version from both places that spell it and refuses when they differ. ruby#799's conclusion is that the second copy should stop existing: CUR_VERSION = RUBY_VERSION # one fact, one place On that day `^CUR_VERSION\s*=\s*"x.y.z"` stops matching and this tool refuses with "no CUR_VERSION = \"x.y.z\"" — **failing because the drift it watches for became impossible**, in a different repository, with nothing in that PR to suggest it would. I was two minutes from opening that PR against rbgo when I noticed; this repository has already paid once for a correction that did not cross a repository boundary. So the derived form is recognised now, before the other half moves. A drift guard has to be able to see that there is only one copy left; the absence of the second is not an unreadable answer. It also says WHICH state it found, because "the two agree" and "there is only one" are different states of rbgo and the second is the one worth seeing on the day it arrives: $ go run ./tools/conformancefigure --root content/en 4 total claim(s) and 3 version claim(s) checked against the gate's 23511 at Ruby 3.4.1 (two copies, agreeing) and, with a derived shim (the new test's fixture): … at Ruby 4.0.9 (derived from RUBY_VERSION) The pattern tolerates a trailing comment, because that line will carry one. Tests green, gofmt clean. Co-Authored-By: Claude Opus 5 --- README.md | 2 +- tools/conformancefigure/main.go | 45 +++++++++++++++++------- tools/conformancefigure/main_test.go | 52 ++++++++++++++++++++++++++++ 3 files changed, 86 insertions(+), 13 deletions(-) diff --git a/README.md b/README.md index 092b633..8c421c8 100644 --- a/README.md +++ b/README.md @@ -25,7 +25,7 @@ after publication. | tool | what it judges | its authority | |---|---|---| -| `conformancefigure` | every **present-tense** claim about the ruby/spec total, **and the Ruby version that total was counted at** — the version selects the population before anything is counted, so a total stated without it is half a measurement. A *dated* sweep going stale is not an error, and is exempt by its date. The differential **oracle** named on these pages is a different claim, free to be newer, and is deliberately not matched. | the gate's own `BASELINE`, and the version read from **both** files that spell it — the runtime's `rubyVersion` and the shim's `CUR_VERSION` — refusing if they disagree ([ruby#799](https://github.com/go-embedded-ruby/ruby/issues/799)) | +| `conformancefigure` | every **present-tense** claim about the ruby/spec total, **and the Ruby version that total was counted at** — the version selects the population before anything is counted, so a total stated without it is half a measurement. A *dated* sweep going stale is not an error, and is exempt by its date. The differential **oracle** named on these pages is a different claim, free to be newer, and is deliberately not matched. | the gate's own `BASELINE`, and the version read from **both** files that spell it — the runtime's `rubyVersion` and the shim's `CUR_VERSION` — refusing if they disagree, and reporting which state it found (`two copies, agreeing`, or `derived from RUBY_VERSION` once [ruby#799](https://github.com/go-embedded-ruby/ruby/issues/799) lands and there is only one) | | `docexample` | every fenced `ruby` block that claims an answer in a `# =>` comment is **run**, and the lines it prints compared in order. A block that claims nothing is skipped. | a real interpreter | | `numericsurface` | every method name the prose **prints** in a module's section, and the counts the pages state. | a real interpreter | | `upstreamlinks` | every **library a module section links to** is one rbgo still depends on. | rbgo's `go.mod` | diff --git a/tools/conformancefigure/main.go b/tools/conformancefigure/main.go index e52c6b7..e748737 100644 --- a/tools/conformancefigure/main.go +++ b/tools/conformancefigure/main.go @@ -78,6 +78,18 @@ const ( var ( runtimeVersionRe = regexp.MustCompile(`(?m)^\trubyVersion\s*=\s*"(\d+\.\d+\.\d+)"`) shimVersionRe = regexp.MustCompile(`(?m)^CUR_VERSION\s*=\s*"(\d+\.\d+\.\d+)"`) + // ⛔ AND THE SHAPE THAT APPEARS WHEN THE DEFECT IS FIXED. ruby#799 asks for + // the shim to stop typing the version a second time and derive it: + // + // CUR_VERSION = RUBY_VERSION + // + // On that day `shimVersionRe` stops matching, and a guard that only knew the + // literal form would refuse with "no CUR_VERSION = \"x.y.z\"" — this check + // failing *because* the thing it checks for was repaired, in a different + // repository, with nothing in that PR to suggest it. A drift guard has to + // recognise when there is only one copy left and say so, not treat the + // absence of the second copy as an unreadable answer. + shimDerivedRe = regexp.MustCompile(`(?m)^CUR_VERSION\s*=\s*RUBY_VERSION\s*(?:#.*)?$`) ) // Two shapes of CURRENT claim, because two sites phrase it two ways round. @@ -375,7 +387,7 @@ func check(stdout, stderr io.Writer, root, baseline, runtimePath, shimPath strin return &exitError{2, fmt.Sprintf("cannot read the gate's total: %v", err)} } - wantVersion, err := authoritativeVersion(runtimePath, shimPath, timeout) + wantVersion, howSpelled, err := authoritativeVersion(runtimePath, shimPath, timeout) if err != nil { return &exitError{2, fmt.Sprintf("cannot read the version the gate measures at: %v", err)} } @@ -462,8 +474,8 @@ func check(stdout, stderr io.Writer, root, baseline, runtimePath, shimPath strin // "N of M, never N alone": a walk that matched nothing reports no // disagreement, which is the same output as a correct site. - fmt.Fprintf(stdout, "%d total claim(s) and %d version claim(s) checked against the gate's %d at Ruby %s\n", - checked, versionsChecked, want, wantVersion) + fmt.Fprintf(stdout, "%d total claim(s) and %d version claim(s) checked against the gate's %d at Ruby %s (%s)\n", + checked, versionsChecked, want, wantVersion, howSpelled) // ⛔ THE DISAGREEMENTS ARE REPORTED BEFORE THE ABSENCES, and that order is // the whole of it. Written the other way round -- the version-absence // refusal returning first -- this tool answered "found no statement of the @@ -604,30 +616,39 @@ func fetch(path, url string, timeout time.Duration) ([]byte, error) { // two copies differ. The third is the one worth having: it is the exact state // ruby#799 describes as waiting to happen, and on the day half a parity change // lands this is what says so. -func authoritativeVersion(runtimePath, shimPath string, timeout time.Duration) (string, error) { +func authoritativeVersion(runtimePath, shimPath string, timeout time.Duration) (string, string, error) { rt, err := fetch(runtimePath, runtimeURL, timeout) if err != nil { - return "", fmt.Errorf("the runtime's version: %w", err) + return "", "", fmt.Errorf("the runtime's version: %w", err) } sh, err := fetch(shimPath, shimURL, timeout) if err != nil { - return "", fmt.Errorf("the shim's version: %w", err) + return "", "", fmt.Errorf("the shim's version: %w", err) } rm := runtimeVersionRe.FindSubmatch(rt) if rm == nil { - return "", fmt.Errorf("no `rubyVersion = \"x.y.z\"` in %s", runtimeURL) + return "", "", fmt.Errorf("no `rubyVersion = \"x.y.z\"` in %s", runtimeURL) } + runtime := string(rm[1]) + + // One copy left: the shim reads RUBY_VERSION at run time, so there is nothing + // that can drift and the runtime's value IS the gate's. Reported as "derived" + // rather than silently, because "the two agree" and "there is only one" are + // different states of this repository and the second is the one worth seeing. + if shimDerivedRe.Match(sh) { + return runtime, "derived from RUBY_VERSION", nil + } + sm := shimVersionRe.FindSubmatch(sh) if sm == nil { - return "", fmt.Errorf("no `CUR_VERSION = \"x.y.z\"` in %s", shimURL) + return "", "", fmt.Errorf("no `CUR_VERSION = \"x.y.z\"` and no `CUR_VERSION = RUBY_VERSION` in %s", shimURL) } - runtime, shim := string(rm[1]), string(sm[1]) - if runtime != shim { - return "", fmt.Errorf("the two copies of the version disagree: the runtime reports %s "+ + if shim := string(sm[1]); runtime != shim { + return "", "", fmt.Errorf("the two copies of the version disagree: the runtime reports %s "+ "(kernel_introspect.go) while the gate measures at %s (spec_helper.rb) — "+ "half a version change has landed; see ruby#799", runtime, shim) } - return runtime, nil + return runtime, "two copies, agreeing", nil } // hasAnySuffix is spelled out so the file types this walks are a list in one diff --git a/tools/conformancefigure/main_test.go b/tools/conformancefigure/main_test.go index b9fddf9..7ae7a94 100644 --- a/tools/conformancefigure/main_test.go +++ b/tools/conformancefigure/main_test.go @@ -306,6 +306,58 @@ func TestAVersionClaimSplitAcrossLinesIsSeen(t *testing.T) { } } +// ⛔ The shape this check will meet when the defect it guards is REPAIRED. +// ruby#799 asks the shim to derive the version instead of typing it again: +// +// CUR_VERSION = RUBY_VERSION +// +// A guard that only knew `CUR_VERSION = "x.y.z"` would then refuse — failing +// because the drift it watches for became impossible, in another repository, +// with nothing in that PR to predict it. That is the cross-repository breakage +// this project has already paid for once. +func TestADerivedShimVersionIsNotADisagreement(t *testing.T) { + dir := t.TempDir() + base := filepath.Join(dir, "BASELINE") + if err := os.WriteFile(base, []byte("OK\ta_spec.rb\t23511\n"), 0o644); err != nil { + t.Fatal(err) + } + root := t.TempDir() + if err := os.WriteFile(filepath.Join(root, "page.md"), + []byte("It passes **23,511**, and those guards resolve at Ruby 4.0.9.\n"), 0o644); err != nil { + t.Fatal(err) + } + + fix := t.TempDir() + rp := filepath.Join(fix, "kernel_introspect.go") + if err := os.WriteFile(rp, []byte("const (\n\trubyVersion = \"4.0.9\"\n)\n"), 0o644); err != nil { + t.Fatal(err) + } + sp := filepath.Join(fix, "spec_helper.rb") + // With a trailing comment, because that is how the line will actually be + // written: the change wants explaining at the site. + if err := os.WriteFile(sp, + []byte("CUR_VERSION = RUBY_VERSION # one fact, one place (#799)\n"), 0o644); err != nil { + t.Fatal(err) + } + + var out, errb bytes.Buffer + cmd := RootCmd() + cmd.SetArgs([]string{"--root", root, "--baseline", base, + "--runtime", rp, "--shim", sp, "--timeout", "5s"}) + cmd.SetOut(&out) + cmd.SetErr(&errb) + if err := cmd.Execute(); err != nil { + t.Fatalf("a derived shim version was treated as a failure: %v\nstdout: %s\nstderr: %s", + err, out.String(), errb.String()) + } + // And it SAYS there is only one copy: "the two agree" and "there is only one" + // are different states of that repository, and the second is the one worth + // seeing on the day it arrives. + if got := out.String(); !bytes.Contains([]byte(got), []byte("derived from RUBY_VERSION")) { + t.Errorf("it said %q, want it to report the version as derived", got) + } +} + // A renamed constant must read as "I cannot tell", not as a version. A pattern // that silently matches nothing is how a guard stops guarding. func TestAMissingVersionConstantRefuses(t *testing.T) {