conformance: an oracle that cannot load its gems scores zero, and a sweep calls that a result

scripts/conformance/rubyspec/README.md

@@ -295,6 +295,54 @@ Those 50 files are the surface any claim about the move has to cover.
 It remains the right instrument for the other half too: a `UPDATE_BASELINE=1`
 that absorbs a loss silently.
 
+## Making the reference interpreter able to answer — `cmd/mriref`
+
+⛔ **An MRI that cannot load its bundled gems scores a clean ZERO on every file,
+and a sweep reports that as a result.**
+
+A pkgx-installed MRI 4.0.6 on a developer machine starts with RubyGems absent
+from its load path:
+
+```text
+`RubyGems' were not loaded.
+cannot load such file -- rubygems (LoadError)
+```
+
+`net/ftp`, `net/http`, `stringio`, `bigdecimal` and most of the rest have been
+**bundled gems** since Ruby 3.1, so on such an interpreter every spec file that
+needs one raises `LoadError` before a single example runs. It does not error
+out of the sweep — it contributes 0, 56 times, and `cmd/specdelta` prints:
+
+```text
+56 file(s): control 0 passing, treatment 331 passing, +331
+```
+
+which reads exactly like rbgo beating the reference. The whole of
+`library/net-ftp` was compared that way once before anyone read the
+interpreter's stderr: **a per-file score cannot tell "failed every example"
+from "never started".**
+
+`cmd/mriref` puts every gem's `lib` on the load path — what RubyGems would have
+resolved anyway — and refuses to run if the gem root yields none, since an
+empty `-I` list reproduces the exact failure it exists to prevent:
+
+```sh
+MRIREF_RUBY=~/.pkgx/ruby-lang.org/v4.0.6/bin/ruby \
+MRIREF_GEMS=~/.pkgx/ruby-lang.org/v4.0.6/lib/ruby/gems/4.0.0/gems \
+go run ./scripts/conformance/rubyspec/cmd/specdelta \
+  -root ~/ruby-spec-corpus -dir library/net-ftp -move 4.0.6:4.0.9 \
+  $(go env GOPATH)/bin/mriref ./rbgo
+```
+
+It takes one script, so it drops into `cmd/specdelta` and `run.sh` wherever an
+interpreter is named.
+
+**The check that it is working:** with the gems on the path that interpreter
+scores **459** on `library/net-ftp`'s 56 files — the same figure this
+repository's own dated sweep recorded for MRI. An oracle that reproduces a
+number measured elsewhere is one you can lean on; one that scores zero is not
+an oracle at all.
+
 ## Asking the reference interpreter — `cmd/oracle`
 
 Every parity fix starts with *"what does MRI actually do here?"*, and the obvious

scripts/conformance/rubyspec/cmd/mriref/main.go

@@ -0,0 +1,117 @@
+// Copyright (c) the go-embedded-ruby/ruby authors
+//
+// SPDX-License-Identifier: BSD-3-Clause
+
+// Command mriref runs a reference MRI with every bundled gem on the load path,
+// so it can serve as the conformance oracle.
+//
+// ⛔ WHY IT EXISTS, AND WHAT IT COST TO FIND. A pkgx-installed MRI 4.0.6 on a
+// developer machine starts with RubyGems absent from its load path:
+//
+//	`RubyGems' were not loaded.
+//	cannot load such file -- rubygems (LoadError)
+//
+// `net/ftp`, `net/http`, `stringio`, `bigdecimal` and most of the rest have
+// been BUNDLED GEMS since Ruby 3.1, so on such an interpreter every ruby/spec
+// file that needs one raises LoadError before a single example runs.
+//
+// Used as an oracle that way it does not error out: it scores a clean ZERO on
+// every file. A sweep then reports `control 0 passing, treatment 331 passing,
+// +331`, which reads exactly like the implementation under test beating the
+// reference. It is the silence of a broken instrument wearing the shape of a
+// measurement. The whole of library/net-ftp was compared that way once before
+// anyone looked at the interpreter's stderr -- a per-file score cannot tell
+// "failed every example" from "never started".
+//
+// With the gem lib directories on the path the same interpreter scores 459 on
+// those 56 files, which is the figure the repository's own dated sweep
+// recorded, so the oracle is checkable against something.
+//
+// Rather than resolve net/ftp's dependency closure by hand (net-protocol,
+// timeout, resolv, io-wait, stringio, …) it adds EVERY gem's lib directory,
+// which is what RubyGems would have resolved anyway.
+//
+// Usage, as an interpreter that takes one script -- the shape cmd/specdelta
+// and run.sh expect:
+//
+//	MRIREF_RUBY=/path/to/bin/ruby \
+//	MRIREF_GEMS=/path/to/lib/ruby/gems/<abi>/gems \
+//	mriref library/net-ftp/acct_spec.rb
+package main
+
+import (
+	"errors"
+	"fmt"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"sort"
+)
+
+func main() {
+	args, err := buildArgs(os.Getenv("MRIREF_RUBY"), os.Getenv("MRIREF_GEMS"), os.Args[1:])
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "mriref: %v\n", err)
+		os.Exit(2)
+	}
+	cmd := exec.Command(os.Getenv("MRIREF_RUBY"), args...)
+	cmd.Stdin, cmd.Stdout, cmd.Stderr = os.Stdin, os.Stdout, os.Stderr
+	if err := cmd.Run(); err != nil {
+		var ee *exec.ExitError
+		if errors.As(err, &ee) {
+			os.Exit(ee.ExitCode())
+		}
+		fmt.Fprintf(os.Stderr, "mriref: %v\n", err)
+		os.Exit(2)
+	}
+}
+
+// errNoGems is returned when the gem root holds no lib directory at all.
+var errNoGems = errors.New("no gem lib directory")
+
+// buildArgs turns a gem root into the -I list, then appends the caller's own
+// arguments.
+//
+// ⛔ IT REFUSES AN EMPTY RESULT. A gem root that yields nothing is not a
+// gem-free Ruby, it is the wrong path -- and running anyway would reproduce
+// precisely the failure this command exists to prevent: an interpreter that
+// cannot load its bundled gems, scoring zero on every file while looking like
+// it ran.
+func buildArgs(ruby, gems string, rest []string) ([]string, error) {
+	if ruby == "" || gems == "" {
+		return nil, errors.New("MRIREF_RUBY and MRIREF_GEMS must both be set")
+	}
+	libs, err := gemLibs(gems)
+	if err != nil {
+		return nil, err
+	}
+	if len(libs) == 0 {
+		return nil, fmt.Errorf("%w under %s -- refusing to run as if there were none", errNoGems, gems)
+	}
+	out := make([]string, 0, 2*len(libs)+len(rest))
+	for _, l := range libs {
+		out = append(out, "-I", l)
+	}
+	return append(out, rest...), nil
+}
+
+// gemLibs lists the lib directory of every gem under root, sorted so a run is
+// reproducible.
+func gemLibs(root string) ([]string, error) {
+	entries, err := os.ReadDir(root)
+	if err != nil {
+		return nil, err
+	}
+	var libs []string
+	for _, e := range entries {
+		if !e.IsDir() {
+			continue
+		}
+		lib := filepath.Join(root, e.Name(), "lib")
+		if st, err := os.Stat(lib); err == nil && st.IsDir() {
+			libs = append(libs, lib)
+		}
+	}
+	sort.Strings(libs)
+	return libs, nil
+}

scripts/conformance/rubyspec/cmd/mriref/main_test.go

@@ -0,0 +1,109 @@
+// Copyright (c) the go-embedded-ruby/ruby authors
+//
+// SPDX-License-Identifier: BSD-3-Clause
+
+package main
+
+import (
+	"errors"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+)
+
+// gemRoot builds a gem tree: each name becomes <root>/<name>/lib, except names
+// ending in "!" which get the gem directory and NO lib (a gem with nothing to
+// add to the load path, which must be skipped rather than counted).
+func gemRoot(t *testing.T, names ...string) string {
+	t.Helper()
+	root := t.TempDir()
+	for _, n := range names {
+		if strings.HasSuffix(n, "!") {
+			if err := os.MkdirAll(filepath.Join(root, strings.TrimSuffix(n, "!")), 0o755); err != nil {
+				t.Fatal(err)
+			}
+			continue
+		}
+		if err := os.MkdirAll(filepath.Join(root, n, "lib"), 0o755); err != nil {
+			t.Fatal(err)
+		}
+	}
+	return root
+}
+
+// ⛔ THE REFUSAL IS THE POINT OF THIS COMMAND. An interpreter that cannot load
+// its bundled gems does not fail loudly -- it scores ZERO on every spec file,
+// which a sweep reports as a clean result for the control and reads as the
+// implementation under test beating the reference. So a gem root that yields
+// no lib directory must stop the run, not start it.
+func TestBuildArgsRefusesAGemRootThatYieldsNothing(t *testing.T) {
+	for _, tc := range []struct {
+		name string
+		root string
+	}{
+		{"an empty directory", gemRoot(t)},
+		{"only gems with no lib", gemRoot(t, "a!", "b!")},
+		{"a path that does not exist", filepath.Join(t.TempDir(), "absent")},
+	} {
+		t.Run(tc.name, func(t *testing.T) {
+			if _, err := buildArgs("/bin/ruby", tc.root, []string{"x.rb"}); err == nil {
+				t.Fatal("it must refuse, not run with an empty -I list")
+			}
+		})
+	}
+	// The empty-but-readable case names its own reason, so the message says what
+	// to fix rather than only that something failed.
+	_, err := buildArgs("/bin/ruby", gemRoot(t), []string{"x.rb"})
+	if !errors.Is(err, errNoGems) {
+		t.Errorf("err = %v, want it to wrap errNoGems", err)
+	}
+}
+
+// Both environment variables are required: a half-configured oracle would run
+// the wrong interpreter, or the right one with no gems, and both score zero.
+func TestBuildArgsRequiresBothSettings(t *testing.T) {
+	root := gemRoot(t, "net-ftp-0.3.9")
+	if _, err := buildArgs("", root, nil); err == nil {
+		t.Error("an unset MRIREF_RUBY must be refused")
+	}
+	if _, err := buildArgs("/bin/ruby", "", nil); err == nil {
+		t.Error("an unset MRIREF_GEMS must be refused")
+	}
+}
+
+// The -I list carries every gem that has a lib, in a stable order, and the
+// caller's own arguments come last so the script is still the script.
+func TestBuildArgsPutsEveryGemLibBeforeTheScript(t *testing.T) {
+	root := gemRoot(t, "timeout-0.6.0", "net-ftp-0.3.9", "nolib!", "net-protocol-0.2.2")
+	got, err := buildArgs("/bin/ruby", root, []string{"library/net-ftp/acct_spec.rb", "--extra"})
+	if err != nil {
+		t.Fatal(err)
+	}
+	// 3 gems with a lib -> 6 elements, then the 2 caller arguments.
+	if len(got) != 8 {
+		t.Fatalf("got %d argument(s) %v, want 8 (3 × -I plus 2 of the caller's)", len(got), got)
+	}
+	for i := 0; i < 6; i += 2 {
+		if got[i] != "-I" {
+			t.Errorf("argument %d = %q, want -I", i, got[i])
+		}
+	}
+	// ⛔ Sorted, because a load path that changes order between runs can change
+	// which gem wins a name collision, and an oracle has to be reproducible.
+	//
+	// Compared by the GEM DIRECTORY'S NAME, not by a "<gem>/lib" suffix: the
+	// first version of this test used the suffix and passed on macOS and Linux
+	// while failing on Windows, where the separator is `\`. The order was right
+	// all along; the assertion was not portable. Naming the directory is also
+	// what the check actually means.
+	want := []string{"net-ftp-0.3.9", "net-protocol-0.2.2", "timeout-0.6.0"}
+	for i, w := range want {
+		if gem := filepath.Base(filepath.Dir(got[2*i+1])); gem != w {
+			t.Errorf("-I entry %d is %q, want the gem %q (list: %v)", i, gem, w, got[:6])
+		}
+	}
+	if got[6] != "library/net-ftp/acct_spec.rb" || got[7] != "--extra" {
+		t.Errorf("the caller's arguments must come last, got %v", got[6:])
+	}
+}