diff --git a/README.md b/README.md index d112324..8c421c8 100644 --- a/README.md +++ b/README.md @@ -25,7 +25,7 @@ after publication. | tool | what it judges | its authority | |---|---|---| -| `conformancefigure` | every **present-tense** claim about the ruby/spec total. A *dated* sweep going stale is not an error, and is exempt by its date. | the gate's own `BASELINE` | +| `conformancefigure` | every **present-tense** claim about the ruby/spec total, **and the Ruby version that total was counted at** — the version selects the population before anything is counted, so a total stated without it is half a measurement. A *dated* sweep going stale is not an error, and is exempt by its date. The differential **oracle** named on these pages is a different claim, free to be newer, and is deliberately not matched. | the gate's own `BASELINE`, and the version read from **both** files that spell it — the runtime's `rubyVersion` and the shim's `CUR_VERSION` — refusing if they disagree, and reporting which state it found (`two copies, agreeing`, or `derived from RUBY_VERSION` once [ruby#799](https://github.com/go-embedded-ruby/ruby/issues/799) lands and there is only one) | | `docexample` | every fenced `ruby` block that claims an answer in a `# =>` comment is **run**, and the lines it prints compared in order. A block that claims nothing is skipped. | a real interpreter | | `numericsurface` | every method name the prose **prints** in a module's section, and the counts the pages state. | a real interpreter | | `upstreamlinks` | every **library a module section links to** is one rbgo still depends on. | rbgo's `go.mod` | diff --git a/content/en/_index.md b/content/en/_index.md index 2cd0137..8449d16 100644 --- a/content/en/_index.md +++ b/content/en/_index.md @@ -12,7 +12,13 @@ C toolchain and no libruby. **How complete is it?** It passes **23,511** of ruby/spec's `language/` + `core/` examples — a large and growing subset of Ruby, and **not** a drop-in replacement for -CRuby. [Conformance]({{< relref "conformance.md#where-it-stands-measured-2026-09-26" >}}) states +CRuby. That count is taken with ruby/spec's own version guards set to resolve at +Ruby **3.4.1**, the version this runtime reports: every example behind a +`ruby_version_is "4.0"` guard is removed from the suite before anything is +counted, so the figure describes 3.4-shaped behaviour even though the project +targets 4.0 semantics — 61 block guards and 5 inline sites sit at that boundary +([ruby#799](https://github.com/go-embedded-ruby/ruby/issues/799)). +[Conformance]({{< relref "conformance.md#where-it-stands-measured-2026-09-26" >}}) states exactly what that counts and what it leaves out, and [Known limitations]({{< relref "conformance.md#known-limitations" >}}) is worth reading before you depend on it. diff --git a/content/en/conformance.md b/content/en/conformance.md index ee5461f..6c52adc 100644 --- a/content/en/conformance.md +++ b/content/en/conformance.md @@ -147,6 +147,11 @@ workflow enforces it on every push and pull request. {{< callout type="info" >}} **The gate's current total is **23 511**** +Counted with ruby/spec's version guards set to resolve at Ruby **3.4.1** — the +version `RUBY_VERSION` reports and the version the shim measures at. That +selects the population before any counting happens, so the total is not a +percentage of Ruby 4.0; see [Known limitations](#known-limitations). + The table below is one **dated full sweep**, and a dated measurement going stale is not an error: it is what it says it is. The number that moves with every merge is the per-file baseline's sum, which the badge and the README @@ -286,7 +291,7 @@ the call returns successfully and nothing says the guarantee is missing: | | | | --- | --- | -| `RUBY_VERSION` reports `"3.4.1"`, though the differential oracle is MRI 4.0.7 and the pinned corpus is 4.0.x. That is not only cosmetic: the shim's version guard is pinned to the same `3.4.1`, so **167 examples behind a `ruby_version_is "4.0"` guard are never reached**, while 40 guards that end *before* 4.0 are being asserted | [#799](https://github.com/go-embedded-ruby/ruby/issues/799) | +| `RUBY_VERSION` reports `"3.4.1"`, though the differential oracle is MRI 4.0.7, the pinned corpus is 4.0.x, and the latest MRI release is 4.0.9 (2026-10-23). That is not only cosmetic: the shim's `CUR_VERSION` is pinned to the same `3.4.1`, so **61 block guards and 5 inline sites at the `ruby_version_is "4.0"` boundary never run**, while **41 guards that end *before* 4.0 are being asserted**. Guard counts are exact, censused by `scripts/conformance/rubyspec/cmd/guards`; the example count behind them is not stated here because a static scan cannot expand `it_behaves_like`, and the figure of 167 quoted when this was first reported does not reproduce — it needs a CI sweep that moves `CUR_VERSION` without regenerating the baseline, so the ratchet names every file that moves in both directions | [#799](https://github.com/go-embedded-ruby/ruby/issues/799) | | `Errno` carries all **158** of MRI's constant names, but **32** report errno `0` where MRI has a real number (`EAUTH`, `EBADRPC`, `EDEVERR`, …). By set difference on darwin: 81 names are zero-valued under rbgo, 49 under MRI, and every MRI zero is also zero under rbgo | [#633](https://github.com/go-embedded-ruby/ruby/issues/633) | | `Thread#backtrace` answers for the **current** thread only; another thread raises `NotImplementedError` | — | | `Process.fork` **does not exist** — Go's runtime cannot be forked safely | — | diff --git a/tools/conformancefigure/main.go b/tools/conformancefigure/main.go index 4fa8f8c..e748737 100644 --- a/tools/conformancefigure/main.go +++ b/tools/conformancefigure/main.go @@ -1,5 +1,6 @@ // conformancefigure checks that every CURRENT claim about rbgo's ruby/spec -// total agrees with the number the gate actually records. +// total — and about the Ruby version that total was counted at — agrees with +// what the gate actually records. // // Why it exists: on 2026-10-11 this site said rbgo passed **22 488** examples on // its front page and **23 481** "today" in two other places, while the gate's @@ -23,6 +24,7 @@ package main import ( "bufio" + "bytes" "errors" "fmt" "io" @@ -43,6 +45,53 @@ import ( // are themselves downstream of it. const baselineURL = "https://raw.githubusercontent.com/go-embedded-ruby/ruby/main/scripts/conformance/rubyspec/BASELINE" +// The total is only half of what a reader needs. `23,511 examples pass` is a +// count over a POPULATION, and the population is selected by the Ruby version +// the shim resolves `ruby_version_is` guards at: at 3.4.1 every spec behind +// `ruby_version_is "4.0"` is removed before anything is counted, so the figure +// is a 3.4-shaped result whichever MRI the project says it targets. Measured +// 2026-10-27 (ruby#799, ruby#836): 61 block guards and 5 inline sites sit at +// the "4.0" boundary, and 41 guards assert what 4.0 changed away from. +// +// So the version is a claim of the same kind as the number, and it is read from +// BOTH places that spell it: +// +// internal/vm/kernel_introspect.go rubyVersion -- what the runtime REPORTS +// scripts/.../spec_helper.rb CUR_VERSION -- what the gate MEASURES AT +// +// ⛔ These are two hardcoded copies of one fact. ruby#799 asks for the second +// to be derived from the first precisely because "today they happen to agree" +// is the defect waiting to happen: a parity change that moves the runtime and +// forgets the shim would leave the gate measuring the old population while +// every page claimed the new version, and nothing would say so. Reading both +// and refusing when they differ is why this tool exists at all -- two readers +// of one fact cannot drift, where two transcriptions already had. +const ( + runtimeURL = "https://raw.githubusercontent.com/go-embedded-ruby/ruby/main/internal/vm/kernel_introspect.go" + shimURL = "https://raw.githubusercontent.com/go-embedded-ruby/ruby/main/scripts/conformance/rubyspec/spec_helper.rb" +) + +// Anchored at the start of a line so a mention inside a comment or a string -- +// both files discuss these constants at length -- cannot be read as the +// definition. The runtime's is indented inside a const block; the shim's is at +// column zero. +var ( + runtimeVersionRe = regexp.MustCompile(`(?m)^\trubyVersion\s*=\s*"(\d+\.\d+\.\d+)"`) + shimVersionRe = regexp.MustCompile(`(?m)^CUR_VERSION\s*=\s*"(\d+\.\d+\.\d+)"`) + // ⛔ AND THE SHAPE THAT APPEARS WHEN THE DEFECT IS FIXED. ruby#799 asks for + // the shim to stop typing the version a second time and derive it: + // + // CUR_VERSION = RUBY_VERSION + // + // On that day `shimVersionRe` stops matching, and a guard that only knew the + // literal form would refuse with "no CUR_VERSION = \"x.y.z\"" — this check + // failing *because* the thing it checks for was repaired, in a different + // repository, with nothing in that PR to suggest it. A drift guard has to + // recognise when there is only one copy left and say so, not treat the + // absence of the second copy as an unreadable answer. + shimDerivedRe = regexp.MustCompile(`(?m)^CUR_VERSION\s*=\s*RUBY_VERSION\s*(?:#.*)?$`) +) + // Two shapes of CURRENT claim, because two sites phrase it two ways round. // // claimBefore: the words come first -- "It passes 23,511", "whose sum is @@ -75,6 +124,52 @@ var ( // is a known limit and not a surprise. var datedLine = regexp.MustCompile(`(?i)measured\s+\d{4}-\d{2}-\d{2}`) +// The version claim, in the three shapes these sites use for it. +// +// ⛔ THE ORACLE IS A DIFFERENT CLAIM AND MUST NOT MATCH. These pages say +// "differential-tested against MRI Ruby 4.0.5" and "against MRI 4.0.7 on the +// same host": that is the interpreter rbgo's behaviour was COMPARED WITH, which +// is free to be newer than the version the shim resolves guards at -- and is, +// deliberately. A pattern that read "against MRI 4.0.5" as the conformance +// version would compare the oracle against CUR_VERSION and report a +// disagreement that is not one; worse, once both were wrong in the same +// direction it would report agreement. So "against" is deliberately absent from +// every alternative below, and three oracle sentences are negative cases. +var versionClaim = []*regexp.Regexp{ + // "those guards resolve at Ruby 3.4.1" + regexp.MustCompile(`(?i)\bresolve[sd]?\s+at\b[^0-9]{0,24}?\**v?(\d+\.\d+\.\d+)\**`), + // "the suite runs as Ruby 3.4.1" + regexp.MustCompile(`(?i)\bruns?\s+as\b[^0-9]{0,24}?\**v?(\d+\.\d+\.\d+)\**`), + // "`RUBY_VERSION` reports `\"3.4.1\"`" and "`CUR_VERSION` is 3.4.1" -- the + // constants named outright, which is how the known-defect tables phrase it. + regexp.MustCompile("(?i)`?(?:RUBY_VERSION|CUR_VERSION)`?\\s*(?:reports|is|=)\\s*\\**`?\"?(\\d+\\.\\d+\\.\\d+)"), +} + +// versionCases is the version vocabulary's own control, in the same spirit as +// vocabularyCases below and run on every invocation for the same reason. +var versionCases = []struct { + line string + want bool + v string +}{ + // The three shapes, as the pages will carry them. + {"and those guards **resolve at Ruby 3.4.1**, not at the version the oracle runs.", true, "3.4.1"}, + {"the suite runs as Ruby **3.4.1** while the oracle is newer", true, "3.4.1"}, + {"| `RUBY_VERSION` reports `\"3.4.1\"`, though the differential oracle is MRI 4.0.7 |", true, "3.4.1"}, + {"`CUR_VERSION` is 3.4.1, which selects the population before anything is counted", true, "3.4.1"}, + // ⛔ The oracle. Three real sentences from these pages, all of which name a + // DIFFERENT and newer MRI on purpose. + {"All behaviour is differential-tested against MRI Ruby 4.0.5 (where MRI has an", false, ""}, + {"Each entry below was re-run on `4872e01` against **MRI 4.0.7 on the same host**", false, ""}, + {"- with **every feature differential-tested against MRI 4.0.5**.", false, ""}, + // A target stated without a patch level is a statement of intent about + // semantics, not of the version a guard resolves at. + {"go-embedded-ruby targets **Ruby 4.0** semantics. Correctness is not asserted by", false, ""}, + // The total, and a benchmark table: numbers that are not versions. + {`

23,511 ruby/spec language/ + core/ examples pass`, false, ""}, + {"| module.op | rbgo | MRI 4.0.5 | MRI+YJIT | JRuby 10.1 | TruffleRuby 34.0.1 | rbgo/MRI |", false, ""}, +} + // vocabularyCases is the guard's own control, run before it judges anything. // Each is a line this tool will meet; `want` is whether a current claim should // be recognised in it. @@ -125,6 +220,61 @@ func match(line string) (string, bool) { return "", false } +// matchVersion returns the version a line claims the gate resolves guards at. +// A dated line is exempt for the same reason a dated total is. +func matchVersion(line string) (string, bool) { + if datedLine.MatchString(line) { + return "", false + } + for _, re := range versionClaim { + if m := re.FindStringSubmatch(line); m != nil { + return m[1], true + } + } + return "", false +} + +// allVersions returns every version claim on a line. +func allVersions(line string) []string { + if datedLine.MatchString(line) { + return nil + } + var out []string + for _, re := range versionClaim { + for _, m := range re.FindAllStringSubmatch(line, -1) { + out = append(out, m[1]) + } + } + return out +} + +// versionsAcross reads a claim that a line break cut in half. +// +// ⛔ Measured 2026-10-31, on this tool's own first run over the real pages: a +// sentence was written as +// +// … version guards set to resolve at +// Ruby **3.4.1**, the version this runtime reports … +// +// and the scan, which reads one line at a time, saw neither half. The tool +// reported "2 version claim(s)" where three had been written, and 2 is a +// perfectly clean-looking number — the miss was found only by insisting on +// explaining why it was not 3. Markdown prose is re-wrapped by every editor +// that touches it, so this is not a one-off typo; it is the normal state of a +// paragraph. +// +// It is attempted only when NEITHER line claims anything on its own, so a claim +// is never counted twice, and the join is attributed to the second line because +// that is where the version itself is written. The caller passes what each line +// already yielded rather than having this recompute it: three patterns over +// every line of every page, twice, for a result already in hand. +func versionsAcross(prev string, prevHad bool, line string, lineHad bool) []string { + if prev == "" || prevHad || lineHad { + return nil + } + return allVersions(prev + " " + line) +} + // allMatches returns every current-total figure on a line, so a line carrying // two claims is not judged on the first alone. func allMatches(line string) []string { @@ -169,6 +319,15 @@ func checkVocabulary() error { return &exitError{2, fmt.Sprintf("the vocabulary read %q as %q, want %q", c.line, n, c.n)} } } + for _, c := range versionCases { + v, got := matchVersion(c.line) + if got != c.want { + return &exitError{2, fmt.Sprintf("the version vocabulary is wrong: %q matched=%v, want %v", c.line, got, c.want)} + } + if c.want && v != c.v { + return &exitError{2, fmt.Sprintf("the version vocabulary read %q as %q, want %q", c.line, v, c.v)} + } + } return nil } @@ -176,22 +335,31 @@ func checkVocabulary() error { // SetOut and SetErr instead of a subprocess. func RootCmd() *cobra.Command { var ( - root string - baseline string - timeout time.Duration + root string + baseline string + runtimePath string + shimPath string + timeout time.Duration ) cmd := &cobra.Command{ Use: "conformancefigure", - Short: "Check that every stated ruby/spec total agrees with the gate's BASELINE", + Short: "Check the stated ruby/spec total, and the version it was counted at, against the gate", Long: "conformancefigure reads the present-tense claims this site makes about\n" + "rbgo's ruby/spec total and compares each with the sum of the gate's own\n" + "per-file baseline. A dated sweep going stale is not an error and is\n" + - "exempt by its date.", + "exempt by its date.\n\n" + + "It also checks the Ruby version the gate resolves `ruby_version_is`\n" + + "guards at, because that version selects the population the total counts\n" + + "over. The version is read from both files that spell it — the runtime's\n" + + "`rubyVersion` and the shim's `CUR_VERSION` — and a disagreement between\n" + + "them is a refusal, not a verdict on these pages.\n\n" + + "The differential oracle named on these pages is a DIFFERENT claim, free\n" + + "to be newer, and is deliberately not matched.", SilenceUsage: true, SilenceErrors: true, Args: cobra.NoArgs, RunE: func(cmd *cobra.Command, _ []string) error { - return check(cmd.OutOrStdout(), cmd.ErrOrStderr(), root, baseline, timeout) + return check(cmd.OutOrStdout(), cmd.ErrOrStderr(), root, baseline, runtimePath, shimPath, timeout) }, } // content/en since the move to Hugo (#28); it was docs/ under MkDocs, and a @@ -199,11 +367,13 @@ func RootCmd() *cobra.Command { // this tool refused with "lstat docs: no such file". cmd.Flags().StringVar(&root, "root", "content/en", "the directory of Markdown and HTML to check") cmd.Flags().StringVar(&baseline, "baseline", "", "read the authoritative BASELINE from this path instead of fetching it") + cmd.Flags().StringVar(&runtimePath, "runtime", "", "read kernel_introspect.go from this path instead of fetching it") + cmd.Flags().StringVar(&shimPath, "shim", "", "read spec_helper.rb from this path instead of fetching it") cmd.Flags().DurationVar(&timeout, "timeout", 30*time.Second, "how long to wait for the fetch") return cmd } -func check(stdout, stderr io.Writer, root, baseline string, timeout time.Duration) error { +func check(stdout, stderr io.Writer, root, baseline, runtimePath, shimPath string, timeout time.Duration) error { // The control first: if the vocabulary is wrong, every verdict after it is // worthless. if err := checkVocabulary(); err != nil { @@ -217,13 +387,23 @@ func check(stdout, stderr io.Writer, root, baseline string, timeout time.Duratio return &exitError{2, fmt.Sprintf("cannot read the gate's total: %v", err)} } + wantVersion, howSpelled, err := authoritativeVersion(runtimePath, shimPath, timeout) + if err != nil { + return &exitError{2, fmt.Sprintf("cannot read the version the gate measures at: %v", err)} + } + type finding struct { where string got int text string } + type versionFinding struct { + where string + got string + } var wrong []finding - checked := 0 + var wrongVersion []versionFinding + checked, versionsChecked := 0, 0 err = filepath.WalkDir(root, func(path string, d fs.DirEntry, err error) error { if err != nil { @@ -253,6 +433,9 @@ func check(stdout, stderr io.Writer, root, baseline string, timeout time.Duratio defer f.Close() sc := bufio.NewScanner(f) sc.Buffer(make([]byte, 1<<20), 1<<20) // roadmap.md has very long lines + // Carried across lines so a claim a line break cut in half is still read; + // prevHadVersion keeps that join from counting a claim twice. + prev, prevHadVersion := "", false for line := 1; sc.Scan(); line++ { text := sc.Text() for _, fig := range allMatches(text) { @@ -269,6 +452,19 @@ func check(stdout, stderr io.Writer, root, baseline string, timeout time.Duratio }) } } + own := allVersions(text) + ownHad := len(own) > 0 + versions := append(own, versionsAcross(prev, prevHadVersion, text, ownHad)...) + prev, prevHadVersion = text, ownHad + for _, v := range versions { + versionsChecked++ + if v != wantVersion { + wrongVersion = append(wrongVersion, versionFinding{ + where: fmt.Sprintf("%s:%d", path, line), + got: v, + }) + } + } } return sc.Err() }) @@ -278,16 +474,35 @@ func check(stdout, stderr io.Writer, root, baseline string, timeout time.Duratio // "N of M, never N alone": a walk that matched nothing reports no // disagreement, which is the same output as a correct site. - fmt.Fprintf(stdout, "%d current claim(s) checked against the gate's %d\n", checked, want) + fmt.Fprintf(stdout, "%d total claim(s) and %d version claim(s) checked against the gate's %d at Ruby %s (%s)\n", + checked, versionsChecked, want, wantVersion, howSpelled) + // ⛔ THE DISAGREEMENTS ARE REPORTED BEFORE THE ABSENCES, and that order is + // the whole of it. Written the other way round -- the version-absence + // refusal returning first -- this tool answered "found no statement of the + // version" on a page whose TOTAL was wrong by 1,023, and the stale figure it + // exists to catch never reached the output. A new check must not be able to + // swallow the finding of the old one: a concrete disagreement outranks a + // structural complaint about what the page does not say. + for _, w := range wrong { + fmt.Fprintf(stderr, "%s: claims %d, the gate records %d\n %s\n", w.where, w.got, want, w.text) + } + for _, w := range wrongVersion { + fmt.Fprintf(stderr, "%s: says the gate measures at %s, it measures at %s\n", w.where, w.got, wantVersion) + } + if n := len(wrong) + len(wrongVersion); n > 0 { + return &exitError{1, fmt.Sprintf("%d claim(s) disagree with the ruby repository", n)} + } if checked == 0 { return &exitError{1, fmt.Sprintf("found no statement of the total in %s; "+ "either the vocabulary no longer matches how the docs phrase it, or the figure was dropped", root)} } - for _, w := range wrong { - fmt.Fprintf(stderr, "%s: claims %d, the gate records %d\n %s\n", w.where, w.got, want, w.text) - } - if len(wrong) > 0 { - return &exitError{1, fmt.Sprintf("%d claim(s) disagree with %s", len(wrong), baselineURL)} + // The same refusal, for the same reason: a site that states a total and + // never says which version selected the population is stating half a + // measurement, and a version vocabulary that matches nothing is + // indistinguishable from a site that states it correctly. + if versionsChecked == 0 { + return &exitError{1, fmt.Sprintf("found no statement of the version the gate measures at in %s; "+ + "a total without it is a count over an unnamed population (Ruby %s today)", root, wantVersion)} } return nil } @@ -318,31 +533,16 @@ func main() { // does, deliberately — two readers of one file cannot drift, where two // transcriptions of one number already have. func authoritative(path string, timeout time.Duration) (int, error) { - var r io.Reader - if path != "" { - f, err := os.Open(path) - if err != nil { - return 0, err - } - defer f.Close() - r = f - } else { - cl := &http.Client{Timeout: timeout} - resp, err := cl.Get(baselineURL) - if err != nil { - return 0, err - } - defer resp.Body.Close() - if resp.StatusCode != http.StatusOK { - // A 404 body is still a body: without this the scan below would read - // GitHub's error page, find no OK rows, and report a total of 0. - return 0, fmt.Errorf("%s: HTTP %d", baselineURL, resp.StatusCode) - } - r = resp.Body + // Through the same bounded fetch as the version files: this path used to + // stream the response straight into a Scanner, which is an unbounded read of + // a remote answer. The baseline is ~2,200 short rows. + b, err := fetch(path, baselineURL, timeout) + if err != nil { + return 0, err } total, rows := 0, 0 - sc := bufio.NewScanner(r) + sc := bufio.NewScanner(bytes.NewReader(b)) for sc.Scan() { parts := strings.Split(sc.Text(), "\t") if len(parts) != 3 || parts[0] != "OK" { @@ -366,6 +566,91 @@ func authoritative(path string, timeout time.Duration) (int, error) { return total, nil } +// fetch returns the bytes of a local path when one is given, and otherwise of +// the URL. The 404 guard is the same one authoritative() needed and for the +// same reason: GitHub answers a missing file with a page, and a page a pattern +// does not match reads as "the constant is not there" rather than as "the +// request was wrong". +// maxFetch bounds what a remote answer may cost us. The three files read here +// are tens of kilobytes; 8 MiB is far above any of them and far below what an +// unbounded io.ReadAll would let a redirected or replaced URL spend. Reading +// one byte past the limit is an error rather than a truncation, because a +// truncated file is a file whose constants may simply be missing — which this +// tool would then report as "no version found", a refusal that points at the +// wrong thing. +const maxFetch = 8 << 20 + +func fetch(path, url string, timeout time.Duration) ([]byte, error) { + if path != "" { + return os.ReadFile(path) + } + cl := &http.Client{Timeout: timeout} + resp, err := cl.Get(url) + if err != nil { + return nil, err + } + defer resp.Body.Close() + if resp.StatusCode != http.StatusOK { + // A 404 body is still a body: without this the callers would read + // GitHub's error page and find no rows and no constants in it, which + // reads as an answer rather than as a failed request. + return nil, fmt.Errorf("%s: HTTP %d", url, resp.StatusCode) + } + b, err := io.ReadAll(io.LimitReader(resp.Body, maxFetch+1)) + if err != nil { + return nil, err + } + if len(b) > maxFetch { + return nil, fmt.Errorf("%s: larger than %d bytes", url, maxFetch) + } + return b, nil +} + +// authoritativeVersion reports the Ruby version the conformance population is +// selected by, read from both files that spell it and refusing when they +// disagree. +// +// It returns an error rather than a best guess in three cases, and each is a +// refusal to answer rather than a verdict: the file could not be read, the +// constant was not found in it (a rename must not read as "no version"), or the +// two copies differ. The third is the one worth having: it is the exact state +// ruby#799 describes as waiting to happen, and on the day half a parity change +// lands this is what says so. +func authoritativeVersion(runtimePath, shimPath string, timeout time.Duration) (string, string, error) { + rt, err := fetch(runtimePath, runtimeURL, timeout) + if err != nil { + return "", "", fmt.Errorf("the runtime's version: %w", err) + } + sh, err := fetch(shimPath, shimURL, timeout) + if err != nil { + return "", "", fmt.Errorf("the shim's version: %w", err) + } + rm := runtimeVersionRe.FindSubmatch(rt) + if rm == nil { + return "", "", fmt.Errorf("no `rubyVersion = \"x.y.z\"` in %s", runtimeURL) + } + runtime := string(rm[1]) + + // One copy left: the shim reads RUBY_VERSION at run time, so there is nothing + // that can drift and the runtime's value IS the gate's. Reported as "derived" + // rather than silently, because "the two agree" and "there is only one" are + // different states of this repository and the second is the one worth seeing. + if shimDerivedRe.Match(sh) { + return runtime, "derived from RUBY_VERSION", nil + } + + sm := shimVersionRe.FindSubmatch(sh) + if sm == nil { + return "", "", fmt.Errorf("no `CUR_VERSION = \"x.y.z\"` and no `CUR_VERSION = RUBY_VERSION` in %s", shimURL) + } + if shim := string(sm[1]); runtime != shim { + return "", "", fmt.Errorf("the two copies of the version disagree: the runtime reports %s "+ + "(kernel_introspect.go) while the gate measures at %s (spec_helper.rb) — "+ + "half a version change has landed; see ruby#799", runtime, shim) + } + return runtime, "two copies, agreeing", nil +} + // hasAnySuffix is spelled out so the file types this walks are a list in one // place rather than a chain of && that grows a bug each time it is extended. func hasAnySuffix(path string, suffixes ...string) bool { diff --git a/tools/conformancefigure/main_test.go b/tools/conformancefigure/main_test.go index 443b54c..7ae7a94 100644 --- a/tools/conformancefigure/main_test.go +++ b/tools/conformancefigure/main_test.go @@ -51,6 +51,32 @@ func TestTheVocabularyControlBites(t *testing.T) { }) } +// versionFixtures writes the two files the version check reads, in the shape +// the ruby repository spells them. +// +// ⛔ Every test that gets as far as the version check must pass these. Without +// them the check fetches raw.githubusercontent.com, and a test that reaches the +// network is a test whose verdict depends on someone else's main branch and on +// having a connection — it read "at Ruby 3.4.1" from the live repository when +// this was first written, which is the right answer arrived at the wrong way. +func versionFixtures(t *testing.T, runtimeV, shimV string) (string, string) { + t.Helper() + dir := t.TempDir() + rp := filepath.Join(dir, "kernel_introspect.go") + // A tab, because the pattern is anchored to the indented const block: the + // file discusses `rubyVersion` in prose above it. + body := "// the comment above mentions rubyVersion = \"9.9.9\" in prose\nconst (\n" + + "\trubyVersion = \"" + runtimeV + "\"\n\treleaseDate = \"2024-12-25\"\n)\n" + if err := os.WriteFile(rp, []byte(body), 0o644); err != nil { + t.Fatal(err) + } + sp := filepath.Join(dir, "spec_helper.rb") + if err := os.WriteFile(sp, []byte("CUR_VERSION = \""+shimV+"\"\nPLATFORMS = [:darwin]\n"), 0o644); err != nil { + t.Fatal(err) + } + return rp, sp +} + // ⛔ An unreadable baseline is a refusal (2), never a pass: it finds no // disagreement either, and that silence reads exactly like agreement. func TestAnUnreadableBaselineRefuses(t *testing.T) { @@ -104,11 +130,11 @@ func TestItBites(t *testing.T) { }{ { name: "a page stating the baseline's own total", - md: "It passes **23,511** of ruby/spec's examples.\n", + md: "It passes **23,511** of ruby/spec's examples, and those guards resolve at Ruby 3.4.1.\n", }, { name: "a page stating a different total", - md: "It passes **22,488** of ruby/spec's examples.\n", + md: "It passes **22,488** of ruby/spec's examples, and those guards resolve at Ruby 3.4.1.\n", wantCode: 1, wantMsg: "claims 22488, the gate records 23511", }, @@ -120,7 +146,37 @@ func TestItBites(t *testing.T) { }, { name: "a DATED sweep going stale is not an error", - md: "It passes **23,511** today.\n\n**22,488 examples pass**, measured 2026-09-26 on main\n", + md: "It passes **23,511** today, and those guards resolve at Ruby 3.4.1.\n\n" + + "**22,488 examples pass**, measured 2026-09-26 on main\n", + }, + // ⛔ The reason the disagreements are printed before the absences. Written + // the other way round this case answered "found no statement of the + // version" and never mentioned that the total was wrong by 1,023 — a new + // check swallowing the finding of the old one. + { + name: "a wrong total is named even when the version is not stated", + md: "It passes **22,488** of ruby/spec's examples.\n", + wantCode: 1, + wantMsg: "claims 22488, the gate records 23511", + }, + { + name: "a page stating the total but not the version it was measured at", + md: "It passes **23,511** of ruby/spec's examples.\n", + wantCode: 1, + wantMsg: "found no statement of the version the gate measures at", + }, + { + name: "a page stating a stale version", + md: "It passes **23,511** of ruby/spec's examples, and those guards resolve at Ruby 4.0.9.\n", + wantCode: 1, + wantMsg: "says the gate measures at 4.0.9, it measures at 3.4.1", + }, + // The oracle is a different claim and is named on these pages on purpose. + // It must not be read as the version the gate measures at. + { + name: "the differential oracle may be newer than the gate's version", + md: "It passes **23,511** of ruby/spec's examples, and those guards resolve at Ruby 3.4.1.\n\n" + + "All behaviour is differential-tested against MRI Ruby 4.0.5.\n", }, } @@ -130,9 +186,11 @@ func TestItBites(t *testing.T) { if err := os.WriteFile(filepath.Join(root, "page.md"), []byte(c.md), 0o644); err != nil { t.Fatal(err) } + rp, sp := versionFixtures(t, "3.4.1", "3.4.1") var out, errb bytes.Buffer cmd := RootCmd() - cmd.SetArgs([]string{"--root", root, "--baseline", base, "--timeout", "5s"}) + cmd.SetArgs([]string{"--root", root, "--baseline", base, + "--runtime", rp, "--shim", sp, "--timeout", "5s"}) cmd.SetOut(&out) cmd.SetErr(&errb) err := cmd.Execute() @@ -157,6 +215,182 @@ func TestItBites(t *testing.T) { } } +// ⛔ THE CASE THIS CHECK EXISTS FOR. ruby#799: the version is spelled twice, +// once in the runtime and once in the conformance shim, and "today they happen +// to agree" is the defect waiting to happen. A parity change that moves +// `rubyVersion` to 4.0.9 and forgets `CUR_VERSION` leaves the gate measuring the +// 3.4 population while every page claims 4.0.9 — and both halves look fine on +// their own. Reading both and REFUSING (2, not 1) is the only moment anything +// says so. +func TestTheTwoCopiesOfTheVersionMustAgree(t *testing.T) { + dir := t.TempDir() + base := filepath.Join(dir, "BASELINE") + if err := os.WriteFile(base, []byte("OK\ta_spec.rb\t23511\n"), 0o644); err != nil { + t.Fatal(err) + } + root := t.TempDir() + if err := os.WriteFile(filepath.Join(root, "page.md"), + []byte("It passes **23,511**, and those guards resolve at Ruby 4.0.9.\n"), 0o644); err != nil { + t.Fatal(err) + } + + // The runtime moved, the shim did not. + rp, sp := versionFixtures(t, "4.0.9", "3.4.1") + var out, errb bytes.Buffer + cmd := RootCmd() + cmd.SetArgs([]string{"--root", root, "--baseline", base, + "--runtime", rp, "--shim", sp, "--timeout", "5s"}) + cmd.SetOut(&out) + cmd.SetErr(&errb) + err := cmd.Execute() + + var ee *exitError + if !errors.As(err, &ee) { + t.Fatalf("two disagreeing copies were accepted: %v\nstdout: %s", err, out.String()) + } + // 2, not 1: it is a refusal to answer. With the two copies apart there is no + // single version for the pages to be judged against, so calling the page + // wrong would be picking one arbitrarily. + if ee.code != 2 { + t.Errorf("status %d, want 2", ee.code) + } + for _, want := range []string{"4.0.9", "3.4.1", "ruby#799"} { + if !bytes.Contains([]byte(ee.msg), []byte(want)) { + t.Errorf("the refusal %q does not name %q", ee.msg, want) + } + } +} + +// ⛔ A claim a line break cut in half. This is the sentence this tool missed on +// its own first run over the real pages, verbatim: it reported two version +// claims where three had been written, and two looks exactly like a clean +// answer. Markdown prose is re-wrapped by every editor that touches it, so the +// scan must read across the break. +func TestAVersionClaimSplitAcrossLinesIsSeen(t *testing.T) { + dir := t.TempDir() + base := filepath.Join(dir, "BASELINE") + if err := os.WriteFile(base, []byte("OK\ta_spec.rb\t23511\n"), 0o644); err != nil { + t.Fatal(err) + } + root := t.TempDir() + md := "It passes **23,511** of ruby/spec's examples. That count is taken\n" + + "with ruby/spec's own version guards set to resolve at\n" + + "Ruby **4.0.9**, the version this runtime reports.\n" + if err := os.WriteFile(filepath.Join(root, "page.md"), []byte(md), 0o644); err != nil { + t.Fatal(err) + } + + // The fixtures say 3.4.1, the split sentence says 4.0.9: seeing the claim at + // all is what makes this fail, which is the point. A tool blind to the break + // would exit 1 saying no version was stated — a different message, so the + // assertion below cannot pass by accident. + rp, sp := versionFixtures(t, "3.4.1", "3.4.1") + var out, errb bytes.Buffer + cmd := RootCmd() + cmd.SetArgs([]string{"--root", root, "--baseline", base, + "--runtime", rp, "--shim", sp, "--timeout", "5s"}) + cmd.SetOut(&out) + cmd.SetErr(&errb) + err := cmd.Execute() + + var ee *exitError + if !errors.As(err, &ee) { + t.Fatalf("a split version claim was not seen: %v\nstdout: %s", err, out.String()) + } + if said := errb.String(); !bytes.Contains([]byte(said), []byte("says the gate measures at 4.0.9")) { + t.Errorf("it said %q, want it to name the split claim", said) + } + // And it is counted once, not once per line of the join. + if got := out.String(); !bytes.Contains([]byte(got), []byte("1 version claim(s)")) { + t.Errorf("counted %q, want exactly one version claim", got) + } +} + +// ⛔ The shape this check will meet when the defect it guards is REPAIRED. +// ruby#799 asks the shim to derive the version instead of typing it again: +// +// CUR_VERSION = RUBY_VERSION +// +// A guard that only knew `CUR_VERSION = "x.y.z"` would then refuse — failing +// because the drift it watches for became impossible, in another repository, +// with nothing in that PR to predict it. That is the cross-repository breakage +// this project has already paid for once. +func TestADerivedShimVersionIsNotADisagreement(t *testing.T) { + dir := t.TempDir() + base := filepath.Join(dir, "BASELINE") + if err := os.WriteFile(base, []byte("OK\ta_spec.rb\t23511\n"), 0o644); err != nil { + t.Fatal(err) + } + root := t.TempDir() + if err := os.WriteFile(filepath.Join(root, "page.md"), + []byte("It passes **23,511**, and those guards resolve at Ruby 4.0.9.\n"), 0o644); err != nil { + t.Fatal(err) + } + + fix := t.TempDir() + rp := filepath.Join(fix, "kernel_introspect.go") + if err := os.WriteFile(rp, []byte("const (\n\trubyVersion = \"4.0.9\"\n)\n"), 0o644); err != nil { + t.Fatal(err) + } + sp := filepath.Join(fix, "spec_helper.rb") + // With a trailing comment, because that is how the line will actually be + // written: the change wants explaining at the site. + if err := os.WriteFile(sp, + []byte("CUR_VERSION = RUBY_VERSION # one fact, one place (#799)\n"), 0o644); err != nil { + t.Fatal(err) + } + + var out, errb bytes.Buffer + cmd := RootCmd() + cmd.SetArgs([]string{"--root", root, "--baseline", base, + "--runtime", rp, "--shim", sp, "--timeout", "5s"}) + cmd.SetOut(&out) + cmd.SetErr(&errb) + if err := cmd.Execute(); err != nil { + t.Fatalf("a derived shim version was treated as a failure: %v\nstdout: %s\nstderr: %s", + err, out.String(), errb.String()) + } + // And it SAYS there is only one copy: "the two agree" and "there is only one" + // are different states of that repository, and the second is the one worth + // seeing on the day it arrives. + if got := out.String(); !bytes.Contains([]byte(got), []byte("derived from RUBY_VERSION")) { + t.Errorf("it said %q, want it to report the version as derived", got) + } +} + +// A renamed constant must read as "I cannot tell", not as a version. A pattern +// that silently matches nothing is how a guard stops guarding. +func TestAMissingVersionConstantRefuses(t *testing.T) { + dir := t.TempDir() + base := filepath.Join(dir, "BASELINE") + if err := os.WriteFile(base, []byte("OK\ta_spec.rb\t23511\n"), 0o644); err != nil { + t.Fatal(err) + } + fix := t.TempDir() + rp := filepath.Join(fix, "kernel_introspect.go") + if err := os.WriteFile(rp, []byte("const (\n\tmriVersion = \"4.0.9\"\n)\n"), 0o644); err != nil { + t.Fatal(err) + } + sp := filepath.Join(fix, "spec_helper.rb") + if err := os.WriteFile(sp, []byte("CUR_VERSION = \"4.0.9\"\n"), 0o644); err != nil { + t.Fatal(err) + } + + cmd := RootCmd() + cmd.SetArgs([]string{"--root", "testdata", "--baseline", base, + "--runtime", rp, "--shim", sp, "--timeout", "5s"}) + cmd.SetOut(&bytes.Buffer{}) + cmd.SetErr(&bytes.Buffer{}) + err := cmd.Execute() + var ee *exitError + if !errors.As(err, &ee) || ee.code != 2 { + t.Fatalf("error %v, want status 2", err) + } + if !bytes.Contains([]byte(ee.msg), []byte("rubyVersion")) { + t.Errorf("the refusal %q does not name the constant it looked for", ee.msg) + } +} + // A positional argument is a mistake, not an extra root: the flag is --root. func TestAnArgumentIsRefused(t *testing.T) { cmd := RootCmd() @@ -201,14 +435,17 @@ func TestATomlFileIsWalked(t *testing.T) { root := t.TempDir() // A params.phases blurb, as the landing writes it. toml := "[[params.phases]]\n name = \"Phase 7\"\n" + - " blurb = \"measured against ruby/spec: **23,481 examples pass** — the total that BASELINE sums to.\"\n" + " blurb = \"measured against ruby/spec: **23,481 examples pass** — the total that BASELINE sums to, " + + "and those guards resolve at Ruby 3.4.1.\"\n" if err := os.WriteFile(filepath.Join(root, "hugo.toml"), []byte(toml), 0o644); err != nil { t.Fatal(err) } + rp, sp := versionFixtures(t, "3.4.1", "3.4.1") var out, errb bytes.Buffer cmd := RootCmd() - cmd.SetArgs([]string{"--root", root, "--baseline", base, "--timeout", "5s"}) + cmd.SetArgs([]string{"--root", root, "--baseline", base, + "--runtime", rp, "--shim", sp, "--timeout", "5s"}) cmd.SetOut(&out) cmd.SetErr(&errb) err := cmd.Execute()