Repository navigation
Net::FTP: the data connection releases the GVL too, and the backlog shrinks - #927
Merged
Merged
Conversation
…hrinks #921 bounded the control connection and #924 released the GVL around it. Both left the DATA connection holding it, and the gate #924 added said so in its seeded backlog -- four entries I wrote down rather than fixed, so the gate could land without a sprawling change. This is the change. dialData, acceptData, sendData, retrlines and retrbinary now run their blocking call inside ftpIO, and the four entries are DELETED from testdata/gvl-blocking-backlog.txt. That list may only shrink, and an entry that stops reporting fails the gate too, so a fix cannot leave its allowance behind. ⛔ acceptData is the one #916 is actually about. In ACTIVE mode the client listens and the server dials back, so a Ruby program that serves itself has both ends in one process -- and holding the GVL in that Accept parks every other Ruby thread, INCLUDING the one that would make the call it is waiting for. ruby/spec's net-ftp fixtures are exactly that shape, and their spec_helper.rb puts Net::FTP into active mode (see #916's comment: rbgo has no Net::FTP.default_passive, so today that helper is skipped in silence and the suite never reaches this path at all). Only the blocking call goes inside each closure. ftpConnErr raises, and a panic crossing a released GVL would unwind without re-acquiring it, so every raise stays outside. ⛔ NO BEHAVIOURAL TEST, AND THE REASON IS NOT LAZINESS. The in-repo FTP test server is Go-side, so it cannot reproduce this deadlock at all: nothing about rbgo's emulated GVL blocks a Go goroutine, and a test built on it would pass whether or not the fix is present. A real regression test needs BOTH ends in Ruby -- a dummy FTP server in a Ruby Thread, which is what ruby/spec's fixture is -- and that is worth doing properly rather than approximating with something that looks like proof and is not. What does guard the property is the gate: TestEveryBlockingSocketCallReleasesTheGVL fails in about a second if any of these five is unwrapped again, and the backlog deletion means it also fails if one silently stops being reported. The measurement that would show this moving -- the library/net-ftp sweep -- is deferred: this machine has been carrying a 1-minute load average between 23 and 86, and a per-file timeout measured under that cannot tell a hang from a slow file. Refs #916 Co-Authored-By: Claude Opus 5 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Net::FTP: the data connection releases the GVL too, and the backlog shrinks
#921 bounded the control connection; #924 released the GVL around it and added a
gate. Both left the data connection holding it — and the gate said so, in
four seeded backlog entries I wrote down rather than fixed, so that the gate
could land without a sprawling change nobody would review.
This is that change.
dialData,acceptData,sendData,retrlinesandretrbinarynow run their blocking call insideftpIO, and the four entries aredeleted from
testdata/gvl-blocking-backlog.txt. That list may only shrink,and an entry that stops reporting fails the gate too, so a fix cannot leave its
allowance behind for the next regression to hide in.
⛔
acceptDatais the one issue 916 is actually aboutIn active mode the client listens and the server dials back, so a Ruby
program that serves itself has both ends in one process. Holding the GVL in that
Acceptparks every other Ruby thread — including the one that would make thecall it is waiting for.
ruby/spec's net-ftp fixtures are exactly that shape. Their
spec_helper.rbputsNet::FTPinto active mode, and rbgo has noNet::FTP.default_passive, so todaythat helper is skipped in silence and the suite never reaches this path at all
(written up in the issue's latest comment). The order matters and it is the
opposite of the obvious one: release the GVL here first, add
default_passivesecond, re-measure third — otherwise today's fast errors turninto fresh hangs.
Only the blocking call goes inside each closure:
ftpConnErrraises, and a paniccrossing a released GVL would unwind without re-acquiring it, so every raise
stays outside.
⛔ No behavioural test, and the reason is not laziness
The in-repo FTP test server is Go-side. Nothing about rbgo's emulated GVL
blocks a Go goroutine, so that server cannot reproduce this deadlock at all: a
test built on it would pass whether or not this change is present. A real
regression test needs both ends in Ruby — a dummy FTP server in a Ruby
Thread, which is what ruby/spec's fixture is — and that is worth doing properlyrather than approximating with something that looks like proof and is not.
What does guard the property is the gate itself:
TestEveryBlockingSocketCallReleasesTheGVLfails in about a second if any ofthese five is unwrapped again, and the backlog deletion means it also fails if
one silently stops being reported.
The measurement that would show this moving — the
library/net-ftpsweep — isdeferred: this machine has been carrying a 1-minute load average between 23
and 86, and a per-file timeout measured under that cannot tell a hang from a slow
file. Reporting a number taken there would be worse than reporting none.
Refs #916, #921, #924
(#916 stays open on purpose — the measurement has not moved. #921 and #924 are
the merged pull requests this one continues.)
🤖 Generated with Claude Code