Repository navigation
conformance: an oracle that cannot load its gems scores zero, and a sweep calls that a result - #930
Merged
Merged
Conversation
…weep calls that a result
⛔ WHAT HAPPENED. The pkgx MRI 4.0.6 on this machine starts with RubyGems
absent from its load path:
`RubyGems' were not loaded.
cannot load such file -- rubygems (LoadError)
net/ftp, net/http, stringio and most of the rest have been BUNDLED GEMS since
Ruby 3.1, so every library/net-ftp spec raised LoadError before running a
single example. That is not an error the sweep notices: the interpreter exits,
contributes 0, and cmd/specdelta prints
56 file(s): control 0 passing, treatment 331 passing, +331
which reads exactly like rbgo beating the reference. I compared the whole
module that way before reading the interpreter's stderr. A per-file score
cannot tell "failed every example" from "never started", and the silence of a
broken instrument wears the shape of a measurement.
cmd/mriref puts every gem's lib directory on the load path -- what RubyGems
would have resolved anyway, rather than hand-resolving net/ftp's closure
(net-protocol, timeout, resolv, io-wait, stringio, ...). It takes one script,
so it drops into cmd/specdelta and run.sh wherever an interpreter is named.
⛔ IT REFUSES AN EMPTY GEM ROOT, which is the whole point and is what the tests
assert first: a gem root yielding no lib directory is not a gem-free Ruby, it
is the wrong path, and running anyway reproduces precisely the failure above.
Three cases are covered (empty directory, gems that have no lib, a path that
does not exist), plus both settings being required and the -I list being
SORTED -- a load path whose order moves between runs can change which gem wins
a name collision, and an oracle has to be reproducible.
THE CHECK THAT IT WORKS, and the reason this is worth having rather than a
note in someone's shell history: with the gems on the path that interpreter
scores 459 on library/net-ftp's 56 files -- the same figure this repository's
own dated sweep recorded for MRI. An oracle that reproduces a number measured
independently is one you can lean on.
Co-Authored-By: Claude Opus 5 <[email protected]>
… the separator
The Windows lane failed, and it was the TEST, not the command:
main_test.go:97: the -I list is not in sorted order:
[-I …\net-ftp-0.3.9\lib -I …\net-protocol-0.2.2\lib -I …\timeout-0.6.0\lib]
The list printed in the error is in exactly the order the test wanted. The
assertion compared `strings.HasSuffix(got[1], "net-ftp-0.3.9/lib")` — a
forward slash — so it passed on macOS and Linux and could not pass on Windows,
where filepath.Join builds `\`.
Now compared by the GEM DIRECTORY'S NAME via filepath.Base(filepath.Dir(...)),
which is separator-independent by construction and is also what the check
actually means: entry i must be the i-th gem in sorted order. The error message
names the gem it found instead of printing three long paths.
⛔ I cannot run the Windows lane here, so the claim is that the new assertion
contains no separator at all, not that I have watched it pass on Windows.
Co-Authored-By: Claude Opus 5 <[email protected]>
tannevaled
added a commit
that referenced
this pull request
Nov 14, 2026
#934) * README: the Net::FTP hang figures describe code that has since changed The map of where the work is said: The largest single debt is a hang, not a missing feature. Net::FTP alone is 442 examples behind, and 39 of its 56 spec files have to be killed by the clock on rbgo and none on MRI (#916). Re-measured today against MRI 4.0.6, same corpus, 30 s per file, on main at ba89ba6: Net::FTP is 128 behind (459 -> 331) and NO file is killed at all. Only the figures this measurement actually covers are touched; the whole-corpus numbers above that bullet are left exactly as they were, because I have not re-swept the corpus. The replacement states the thing that took a day to establish and that a single number hides: ONE SYMPTOM, TWO UNRELATED CAUSES. Half the files hung because a blocking socket call held the emulated GVL (#924); the other half because TCPSocket.open with a block never closed the socket, so the peer waited for an EOF that never came (#932, open). Neither fix moves the other's files, which is why the first one looked like it had done nothing when it was measured against the second one's specs. The table also carries what is NOT fixed, with its cause named rather than left as a count: library/socket 8 and library/timeout 1, both because Thread#kill cannot interrupt a thread blocked in a native call (#933). A reader should be able to tell the fixed from the merely counted. ⛔ And the note that outlives all of these numbers: none of it was measurable until the reference interpreter was repaired. It had been scoring a clean 0 on every file because its bundled gems were off the load path, and a sweep reports that as a RESULT -- 56 files of "control 0 passing" reads as the implementation under test beating the reference. cmd/mriref (#930) is named so the next person measuring starts from a working oracle. Co-Authored-By: Claude Opus 5 <[email protected]> * README: net-http is now 0 too, measured on the merged main #932 merged while this was in review, so the table's net-http row said 27 as the CURRENT figure for a module that no longer hangs at all. Publishing that would have been the same defect this pull request exists to correct. Re-measured on main at 6b51deb, 321 files across library/net-http + library/socket + library/timeout, MRI 4.0.6 as the control, 30 s per file: 275 passing against MRI's 371, NINE killed under rbgo and none under MRI. So the tally is 75 -> 9, and the nine are the last two rows: Thread#kill cannot interrupt a thread blocked in a native call (#933). Co-Authored-By: Claude Opus 5 <[email protected]> --------- Co-authored-by: Claude Opus 5 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
conformance: an oracle that cannot load its gems scores zero, and a sweep calls that a result
What happened
The pkgx MRI 4.0.6 on this machine starts with RubyGems absent from its load
path:
net/ftp,net/http,stringioand most of the rest have been bundledgems since Ruby 3.1, so every
library/net-ftpspec raisedLoadErrorbefore running a single example.
⛔ That is not an error the sweep notices. The interpreter exits, each file
contributes 0, and
cmd/specdeltaprints:which reads exactly like rbgo beating the reference. I compared the whole
module that way before reading the interpreter's stderr. A per-file score
cannot tell "failed every example" from "never started" — the silence of a
broken instrument wears the shape of a measurement.
What this adds
cmd/mrirefputs every gem'slibdirectory on the load path — what RubyGemswould have resolved anyway, rather than hand-resolving
net/ftp's closure(
net-protocol,timeout,resolv,io-wait,stringio, …). It takes onescript, so it drops into
cmd/specdeltaandrun.shwherever an interpreter isnamed:
MRIREF_RUBY=…/bin/ruby MRIREF_GEMS=…/gems/4.0.0/gems \ go run ./scripts/conformance/rubyspec/cmd/specdelta \ -root ~/ruby-spec-corpus -dir library/net-ftp -move 4.0.6:4.0.9 \ mriref ./rbgo⛔ It refuses an empty gem root
Which is the point, and what the tests assert first. A gem root that yields no
libdirectory is not a gem-free Ruby — it is the wrong path, and runninganyway reproduces precisely the failure above. Three cases are covered (an
empty directory, gems that have no
lib, a path that does not exist), plusboth settings being required, and the
-Ilist being sorted: a load pathwhose order moves between runs can change which gem wins a name collision, and
an oracle has to be reproducible.
The check that it works
With the gems on the path, that interpreter scores 459 on
library/net-ftp's 56 files — the same figure this repository's own datedsweep recorded for MRI, and the number the README still quotes. An oracle that
reproduces a measurement taken independently is one you can lean on; one that
scores zero is not an oracle at all.
It also makes the gap measurable again: MRI 459 against rbgo's 331 on that
module today, with no file killed by the clock under either engine at a 30 s
limit.
Refs #916, #929
(Both stay open: #916 is the hang symptom, which this does not address and
which the sweep above does not reproduce on either engine; #929 is the largest
single item the restored oracle revealed.)
🤖 Generated with Claude Code