Repository navigation
Conversation
|
Had a look. The argument side looks right to me, the tuple side I would drop, and there is a third window neither PR touches that I think is the better target. Everything below is against #399 head ( The argument side is right. Hoisting the And to confirm what you found: reverting d41db33 on top of this cannot work. One nit. The premarshal variable is declared in a new loop over The tuple side: I would drop it. _gstate := C.PyGILState_Ensure() // 1086
...
<buildTuple output spliced here> // 1110
C.PyObject_CallObject(_fun_arg, _fcargs) // 1111
C.gopy_decref(_fcargs) // 1112
C.gopy_err_handle() // 1117
C.PyGILState_Release(_gstate) // 1118So the new inner As future-proofing it also does not do what the comment says. The tuple is created inside the new bracket and used outside it:
If you want that contract enforced rather than conventional, the way to get it is to emit the whole Minor, separate from the GIL question: wrapping the body in Where the live GIL-less window actually isStill in C.PyGILState_Release(_gstate) // 1118
...
return C.GoString(C.PyBytes_AsString(_fcret)) // via pyObjectToGo, 1120-1124so While in there: The stress testWell aimed. The churn thread can only run while the main thread has released the GIL, which is exactly gopy's |
Convert Python-side arguments whose marshalling touches a raw *C.PyObject (complex64/complex128, via needsGILForArgMarshal) before releasing the GIL for the wrapped Go call, instead of inline afterward once it has already been released. Self-bracket buildTuple's PyTuple_New/PyTuple_SetItem output with PyGILState_Ensure/Release so it does not rely on its caller already holding the GIL. Adds a complex64/128 round-trip stress test that hammers the fix from a churn thread, and TestBuildTupleHoldsGIL.
addSignatureType released the GIL before converting the *C.PyObject return value from an invoked Python callback back to Go (e.g. via PyLong_AsLongLong/PyBytes_AsString) and never decref'd it. Both now happen before PyGILState_Release, closing a live GIL-less window (hit by any callback with a return value, e.g. _examples/funcs' CallBackRval) and a one-object-per-call leak. Drop buildTuple's self-bracketing GIL, added in the previous commit: its only caller (addSignatureType) already holds the GIL across the whole build/call/decref region, so the extra Ensure/Release was redundant. Replace TestBuildTupleHoldsGIL, which only checked the tuple-build fragment and could pass in the broken case, with a test on addSignatureType's full output. Extract argCallExpr as the single function deciding both an argument's call-site expression and whether it needs GIL premarshalling, so a premarshalled variable can no longer be declared without also being the expression consumed at the call site. Trim the complex stress test from 20000 to 500 iterations; the pre-fix crash is deterministic on the first call.
f8d6795 to
e24deba
Compare
|
Thank you again for the review and collaboration, @satarsa! 🙇 Good catch on the tuple side — you were totally right. That GIL-less window you flagged in For the gen_func.go nit, instead of just reordering the switch, I moved the logic for picking the expression (and whether it needs the GIL) into Also trimmed the stress test down from 20,000 iterations to 500 — good point that the crash is deterministic on the first call anyway. Everything's updated in e24deba. I squashed the branch down to two commits: ec3b50c is the state from your original review, and e24deba includes all the updates above (so the old comment thread will show as outdated against the new diff). |
Resolve the conflict in bind/gen_func.go's genFuncBody against master's shimABI refactor (feat: allow opt-in for new backends, #409): - Release the GIL via g.shim.saveThread instead of the inline PyEval_SaveThread/RestoreThread the branch emitted, so the plain-C backends (cffi, pybind11, nanobind), whose shim must not call the CPython API, still emit nothing there. The premarshal pass still runs ahead of it. - Make argCallExpr a pyGen method and have its isSignature case return g.shim.callbackArg(...) rather than sym.py2go. The two agree for the pybindgen and capi backends, but under the plain-C backends sym.py2go is the CPython-API closure from addSignatureType, which wouldn't compile against their preamble.
genFuncBody called argCallExpr twice per argument, once in the premarshal pass and again at the call site, and relied on both calls agreeing. Since the merge with master, argCallExpr's callback case goes through shimABI.callbackArg, which for the plain-C backends also registers the callback's trampoline (g.trampoline). That registration is deduplicated, so a second call was harmless, but correctness should not depend on it. The premarshal pass now records each argument's expression (or the _premarshalN variable that holds it) in argExprs, and the call site uses that directly. A premarshalled variable is declared exactly when it is the expression passed to the call, by construction rather than because two calls happen to agree.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #401 +/- ##
==========================================
+ Coverage 89.34% 89.38% +0.03%
==========================================
Files 33 33
Lines 4929 4947 +18
==========================================
+ Hits 4404 4422 +18
Misses 525 525 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Master's main_test.go now builds the gopy binary once in TestMain and has tests run it through runGopy, so they can call t.Parallel() (#416). Running gopy in-process is unsafe there, because the generator keeps its state in package variables. Move TestComplexGILStress to runGopy and t.Parallel(), like TestGilString, and drop its bind.ResetPackages(), which only cleaned up the in-process run.
Files added on master since #409 carry "Copyright 2015-present, The go-python Authors."; match it in the three files this branch adds.
Motivated by feedback from @satarsa (see PR 399 feedback).
Relates-to: #400