Repository navigation
Open
Labels
bugSomething isn't workingSomething isn't working
Description
Affects: JuliaCall
Describe the bug
I am seeing Julia tasks launched from parallel python threads lead to sporadic hangs. Short-running tasks suffer less from this. I had no luck using the tricks from #539 or here. I think the hangs are triggered by Julia GC - disabling GC with Base.GC.enable(false) beforehand seems to fix the hangs - and triggering Julia GC manually from a python thread immediately causes a hang.
This is all with PYTHON_JULIACALL_HANDLE_SIGNALS set to yes, in case that matters.
from juliacall import Main as jl
jl_gc = jl.Base.GC.gc
import threading
t = threading.Thread(target=jl_gc)
t.start() # hangs every time (100% CPU)Terminating python gives me
[89363] signal (15): Terminated: 15
in expression starting at none:0
jl_gc_wait_for_the_world at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-R17H3W25T9.0/build/default-honeycrisp-R17H3W25T9-0/julialang/julia-release-1-dot-10/src/gc.c:241 [inlined]
ijl_gc_collect at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-R17H3W25T9.0/build/default-honeycrisp-R17H3W25T9-0/julialang/julia-release-1-dot-10/src/gc.c:3515
gc at ./gcutils.jl:129 [inlined]
gc at ./gcutils.jl:129 [inlined]
pyjlany_call at /Users/username/.julia/packages/PythonCall/Nr75f/src/JlWrap/any.jl:45
_jl_invoke at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-R17H3W25T9.0/build/default-honeycrisp-R17H3W25T9-0/julialang/julia-release-1-dot-10/src/gf.c:0 [inlined]
ijl_apply_generic at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-R17H3W25T9.0/build/default-honeycrisp-R17H3W25T9-0/julialang/julia-release-1-dot-10/src/gf.c:3077
(pyproj) c889f3b4c8d7:pyproj username$ _pyjl_callmethod at /Users/username/.julia/packages/PythonCall/Nr75f/src/JlWrap/base.jl:73
_pyjl_callmethod at /Users/username/.julia/packages/PythonCall/Nr75f/src/JlWrap/C.jl:63
jfptr__pyjl_callmethod_9032 at /Users/username/.julia/compiled/v1.10/PythonCall/WdXsa_v1O0R.dylib (unknown line)
_jl_invoke at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-R17H3W25T9.0/build/default-honeycrisp-R17H3W25T9-0/julialang/julia-release-1-dot-10/src/gf.c:0 [inlined]
ijl_apply_generic at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-R17H3W25T9.0/build/default-honeycrisp-R17H3W25T9-0/julialang/julia-release-1-dot-10/src/gf.c:3077
jlcapi__pyjl_callmethod_9116 at /Users/username/.julia/compiled/v1.10/PythonCall/WdXsa_v1O0R.dylib (unknown line)
method_vectorcall_VARARGS at /Users/username/miniforge3/envs/pyproj/bin/python3.12 (unknown line)
PyObject_Vectorcall at /Users/username/miniforge3/envs/pyproj/bin/python3.12 (unknown line)
_PyEval_EvalFrameDefault at /Users/username/miniforge3/envs/pyproj/bin/python3.12 (unknown line)
_PyObject_FastCallDictTstate at /Users/username/miniforge3/envs/pyproj/bin/python3.12 (unknown line)
slot_tp_call at /Users/username/miniforge3/envs/pyproj/bin/python3.12 (unknown line)
_PyObject_Call at /Users/username/miniforge3/envs/pyproj/bin/python3.12 (unknown line)
_PyEval_EvalFrameDefault at /Users/username/miniforge3/envs/pyproj/bin/python3.12 (unknown line)
method_vectorcall at /Users/username/miniforge3/envs/pyproj/bin/python3.12 (unknown line)
thread_run at /Users/username/miniforge3/envs/pyproj/bin/python3.12 (unknown line)
pythread_wrapper at /Users/username/miniforge3/envs/pyproj/bin/python3.12 (unknown line)
_pthread_start at /usr/lib/system/libsystem_pthread.dylib (unknown line)
unknown function (ip: 0x0)
__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
unknown function (ip: 0x0)
__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
unknown function (ip: 0x0)
__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
unknown function (ip: 0x0)
__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
unknown function (ip: 0x0)
__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
unknown function (ip: 0x0)
__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
unknown function (ip: 0x0)
__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
unknown function (ip: 0x0)
__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
unknown function (ip: 0x0)
__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
unknown function (ip: 0x0)
__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
unknown function (ip: 0x0)
__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
unknown function (ip: 0x0)
__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
unknown function (ip: 0x0)
__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
unknown function (ip: 0x0)
__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
unknown function (ip: 0x0)
Allocations: 3689147 (Pool: 3684864; Big: 4283); GC: 6
Your system
- MacOS, Apple Silicon, Julia 1.10.5
- Linux, x86_64, Julia 1.10.6
- PythonCall/juliacall 0.9.23
Reacted by Carroll Vance
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't working
Activity
csvance commentedon Jul 29, 2025
While it may be suboptimal for many workloads, I ended up working around this by having the main Python thread do all Julia calls. After any thread calls .map on the thread pool executor, the main thread waits for Julia function closures from the worker threads, executes them, and sends the results back. Since I'm making extensive use of scheduling sensitive threading on the Julia side, it was fine to only allow one call into Julia from Python for optimal cache behavior anyway. I'm calling Julia for performance reasons, so I've grouped as much work into a single call as I possibly can, two extra thread context switches per call is a small price to pay compared to how long the Julia routines take to execute.
It would be great if we could get the GC to play nice here. Maybe a safe abstraction could be created where we can have multiple Python threads call into Julia while still providing a guarantee that the GC will actually be able to run at some point before memory becomes a problem. My solution to the problem is much more straightforward than needing to micromanage Julia's GC, but of course it's hardly optimal either.
cjdoris commentedon Jul 29, 2025
Yep I've seen what I think is the same thing before.
AFAIU it's not really supported to call into Julia from a thread that wasn't launched by Julia (e.g. one created by Python). In experiments, what actually happens is that you CAN call into Julia ok, but when it yields, it will go to some other task and never return to this thread. And therefore the thread hangs.
There's a function to get Julia to "adopt" an external thread but when I tried it it caused segfaults or something.
So yeah for now just multi thread on the Julia side.
csvance commentedon Jul 30, 2025
Sounds like that might be why the
jl_yield()workaround didn't work for me. It might become a bit more of an issue down the road when GIL removal is no longer experimental, but for now its more of a minor annoyance. Outside of this JuliaCall feels rock solid, it was the deciding factor to go ahead with using Julia over C++ and Fortran for the Python SaMD project I'm working on. If I get some spare time I will take a look at messing around withjl_adopt_threadamong other things.raphaelchinchilla commentedon Nov 20, 2026
Still reproducible on Julia 1.13.0 + PythonCall/JuliaCall 0.9.37; PyTorch's CUDA autograd hits it with no user threads
Minimal example (pure JuliaCall: no torch, no CUDA):
Versions: Linux x86_64, Python 3.13.15, Julia 1.13.0, JuliaCall/PythonCall 0.9.36 and 0.9.37 (12 Julia threads, auto,1). I have not tested with a single Julia thread. main at 4ed01ff has no threading changes since 0.9.37, so I didn't run it.
Why it matters: PyTorch triggers it without the user creating any threads. PyTorch's autograd engine runs the backward of CUDA nodes on its own per-device worker threads (pt_autograd_0, …), not on the thread that called loss.backward(). So any torch.autograd.Function whose backward calls Julia hangs on CUDA as soon as Julia collects garbage. On CPU, torch runs the backward on the calling thread, so the same code works there. The faulthandler dump from the hang:
Thread pt_autograd_0:
File ".../PythonCall/src/JlWrap/any.jl", line 263 in call
... (our backward -> the Julia call)
File ".../torch/autograd/function.py", line 452 in apply
Main thread:
File ".../torch/autograd/graph.py", line 1113 in _engine_run_backward
File ".../torch/_tensor.py", line 623 in backward
A guess at the mechanism (not verified): the trace in the original report ends in jl_gc_wait_for_the_world. That fits this picture: the thread that initialised Julia (Python's main thread) seems to remain in a GC-unsafe state while it runs ordinary Python code. If it then blocks in C (Thread.join, torch's engine wait) with the GIL released, it never reaches a Julia safepoint, so a stop-the-world GC started on another thread waits for it forever. If that's right, one fix would be for JuliaCall to put the calling thread into a GC-safe state whenever control returns from Julia to Python, and take it back out on the next call in. Julia already does this for adopted foreign threads around @cfunction calls. Does PR #824 (thread-state rework, on v1) cover this Python→Julia direction, or only Julia tasks calling Python?
cjdoris commentedon Nov 20, 2026
Can you try the v1 branch please? Does the issue occur there?
raphaelchinchilla commentedon Nov 20, 2026
Thanks, I tested it: v1 does not fix this. I found what does, and there are two separate causes.
Testing v1 (25f82cc, which includes #824), Julia 1.13.0,
PYTHON_JULIACALL_THREADS=4,1,HANDLE_SIGNALS=yes: the minimal example still hangs in the first round (withjl.sevalrenamed tojl.jl_eval). On 0.9.36 the whole process froze, so event.join(timeout=30)never returned. On v1 the join timeout fires, but the worker's Julia call never returns. That fits #824 managing CPython thread states (the GIL) but not Julia's own per-thread state.Both causes come from the same thing: the thread that ran
jl_init(Python's main thread) spends its time in Python, not in Julia.1. GC state. Your guess is right. After
jl_init, the main thread stays GC-unsafe (gc_state == 0) while it runs Python. A stop-the-world collection started on another thread waits for it forever. Splitting the worker's call (each function compiled on the main thread first):Threads.@threadsloop with atomicsGC.gc()Fix: put the starting thread into the GC-safe state once, right after Julia starts:
This works because every later Python→Julia call goes through a
@cfunction, whose wrapper switches the thread to GC-unsafe on entry and restores the saved state (now GC-safe) on exit. So Julia code never runs in the wrong state, and collections proceed while the main thread runs Python. It has to be done from Python: done inside a Julia call, the wrapper would undo it on return.jl_atexit_hookis called through ctypes rather than a@cfunction, so we restore the previous state withjl_gc_safe_leavein anatexithandler registered after JuliaCall's. With that, Julia's ownatexithooks still run.2. The I/O event loop. With the GC state fixed, the minimal example passes on 0.9.36 and v1, but PyTorch's CUDA backward still hung. gdb showed torch's
pt_autograd_0thread in CUDA.jl's first kernel compile, which pipes through the PTX assembler (run_and_collect→close(stream)→wait_close→jl_task_get_next), with every Julia thread idle in the scheduler. Outside a threaded region, Julia runs the libuv loop only on its I/O-loop thread, by default the thread that started Julia, and only while that thread waits in Julia's scheduler. Here that thread is waiting in torch's engine, so no thread processes the pipe event. Any Julia I/O (subprocesses, pipes,sleep, timers) from a Python-created thread hangs the same way while the main thread waits for it.Fix: Julia already has an API for this:
We call it once at start-up and wait until
Threads.maxthreadid()grows (the new thread has been adopted). With both changes, our PyTorch CUDA autograd tests pass: a torchautograd.Functionwhose backward calls Julia, run on torch's device thread. That's 61/61 CUDA tests and 658/658 CPU tests, on Julia 1.13.0 and JuliaCall 0.9.36. We also have regression tests where a Python thread collects garbage, and waits on a subprocess and a timer, in Julia while the main thread joins it. Without the corresponding fix, each one freezes the process.Suggested fix in JuliaCall: at the end of initialization, on the thread that ran
jl_init, (1) enter the GC-safe state and (2) move the I/O loop off that thread (make_io_thread()or equivalent). Anything that later calls the libjulia C API directly from Python, such as the exit hook, needs to switch back to GC-unsafe first.