Skip to content

juliacall: Julia GC triggered from a python thread causes hangs #578

Open

Description

@amilsted

Affects: JuliaCall

Describe the bug
I am seeing Julia tasks launched from parallel python threads lead to sporadic hangs. Short-running tasks suffer less from this. I had no luck using the tricks from #539 or here. I think the hangs are triggered by Julia GC - disabling GC with Base.GC.enable(false) beforehand seems to fix the hangs - and triggering Julia GC manually from a python thread immediately causes a hang.

This is all with PYTHON_JULIACALL_HANDLE_SIGNALS set to yes, in case that matters.

from juliacall import Main as jl
jl_gc = jl.Base.GC.gc

import threading
t = threading.Thread(target=jl_gc)
t.start()  # hangs every time (100% CPU)

Terminating python gives me

[89363] signal (15): Terminated: 15
in expression starting at none:0
jl_gc_wait_for_the_world at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-R17H3W25T9.0/build/default-honeycrisp-R17H3W25T9-0/julialang/julia-release-1-dot-10/src/gc.c:241 [inlined]
ijl_gc_collect at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-R17H3W25T9.0/build/default-honeycrisp-R17H3W25T9-0/julialang/julia-release-1-dot-10/src/gc.c:3515
gc at ./gcutils.jl:129 [inlined]
gc at ./gcutils.jl:129 [inlined]
pyjlany_call at /Users/username/.julia/packages/PythonCall/Nr75f/src/JlWrap/any.jl:45
_jl_invoke at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-R17H3W25T9.0/build/default-honeycrisp-R17H3W25T9-0/julialang/julia-release-1-dot-10/src/gf.c:0 [inlined]
ijl_apply_generic at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-R17H3W25T9.0/build/default-honeycrisp-R17H3W25T9-0/julialang/julia-release-1-dot-10/src/gf.c:3077
(pyproj) c889f3b4c8d7:pyproj username$ _pyjl_callmethod at /Users/username/.julia/packages/PythonCall/Nr75f/src/JlWrap/base.jl:73
_pyjl_callmethod at /Users/username/.julia/packages/PythonCall/Nr75f/src/JlWrap/C.jl:63
jfptr__pyjl_callmethod_9032 at /Users/username/.julia/compiled/v1.10/PythonCall/WdXsa_v1O0R.dylib (unknown line)
_jl_invoke at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-R17H3W25T9.0/build/default-honeycrisp-R17H3W25T9-0/julialang/julia-release-1-dot-10/src/gf.c:0 [inlined]
ijl_apply_generic at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-R17H3W25T9.0/build/default-honeycrisp-R17H3W25T9-0/julialang/julia-release-1-dot-10/src/gf.c:3077
jlcapi__pyjl_callmethod_9116 at /Users/username/.julia/compiled/v1.10/PythonCall/WdXsa_v1O0R.dylib (unknown line)
method_vectorcall_VARARGS at /Users/username/miniforge3/envs/pyproj/bin/python3.12 (unknown line)
PyObject_Vectorcall at /Users/username/miniforge3/envs/pyproj/bin/python3.12 (unknown line)
_PyEval_EvalFrameDefault at /Users/username/miniforge3/envs/pyproj/bin/python3.12 (unknown line)
_PyObject_FastCallDictTstate at /Users/username/miniforge3/envs/pyproj/bin/python3.12 (unknown line)
slot_tp_call at /Users/username/miniforge3/envs/pyproj/bin/python3.12 (unknown line)
_PyObject_Call at /Users/username/miniforge3/envs/pyproj/bin/python3.12 (unknown line)
_PyEval_EvalFrameDefault at /Users/username/miniforge3/envs/pyproj/bin/python3.12 (unknown line)
method_vectorcall at /Users/username/miniforge3/envs/pyproj/bin/python3.12 (unknown line)
thread_run at /Users/username/miniforge3/envs/pyproj/bin/python3.12 (unknown line)
pythread_wrapper at /Users/username/miniforge3/envs/pyproj/bin/python3.12 (unknown line)
_pthread_start at /usr/lib/system/libsystem_pthread.dylib (unknown line)
unknown function (ip: 0x0)
__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
unknown function (ip: 0x0)
__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
unknown function (ip: 0x0)
__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
unknown function (ip: 0x0)
__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
unknown function (ip: 0x0)
__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
unknown function (ip: 0x0)
__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
unknown function (ip: 0x0)
__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
unknown function (ip: 0x0)
__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
unknown function (ip: 0x0)
__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
unknown function (ip: 0x0)
__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
unknown function (ip: 0x0)
__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
unknown function (ip: 0x0)
__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
unknown function (ip: 0x0)
__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
unknown function (ip: 0x0)
__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
unknown function (ip: 0x0)
Allocations: 3689147 (Pool: 3684864; Big: 4283); GC: 6

Your system

  • MacOS, Apple Silicon, Julia 1.10.5
  • Linux, x86_64, Julia 1.10.6
  • PythonCall/juliacall 0.9.23

Activity

  1. csvance commented on Jul 29, 2025

    @csvance

    While it may be suboptimal for many workloads, I ended up working around this by having the main Python thread do all Julia calls. After any thread calls .map on the thread pool executor, the main thread waits for Julia function closures from the worker threads, executes them, and sends the results back. Since I'm making extensive use of scheduling sensitive threading on the Julia side, it was fine to only allow one call into Julia from Python for optimal cache behavior anyway. I'm calling Julia for performance reasons, so I've grouped as much work into a single call as I possibly can, two extra thread context switches per call is a small price to pay compared to how long the Julia routines take to execute.

    It would be great if we could get the GC to play nice here. Maybe a safe abstraction could be created where we can have multiple Python threads call into Julia while still providing a guarantee that the GC will actually be able to run at some point before memory becomes a problem. My solution to the problem is much more straightforward than needing to micromanage Julia's GC, but of course it's hardly optimal either.

  2. cjdoris commented on Jul 29, 2025

    @cjdoris
    Member

    Yep I've seen what I think is the same thing before.

    AFAIU it's not really supported to call into Julia from a thread that wasn't launched by Julia (e.g. one created by Python). In experiments, what actually happens is that you CAN call into Julia ok, but when it yields, it will go to some other task and never return to this thread. And therefore the thread hangs.

    There's a function to get Julia to "adopt" an external thread but when I tried it it caused segfaults or something.

    So yeah for now just multi thread on the Julia side.

  3. csvance commented on Jul 30, 2025

    @csvance

    Yep I've seen what I think is the same thing before.

    AFAIU it's not really supported to call into Julia from a thread that wasn't launched by Julia (e.g. one created by Python). In experiments, what actually happens is that you CAN call into Julia ok, but when it yields, it will go to some other task and never return to this thread. And therefore the thread hangs.

    There's a function to get Julia to "adopt" an external thread but when I tried it it caused segfaults or something.

    So yeah for now just multi thread on the Julia side.

    Sounds like that might be why the jl_yield() workaround didn't work for me. It might become a bit more of an issue down the road when GIL removal is no longer experimental, but for now its more of a minor annoyance. Outside of this JuliaCall feels rock solid, it was the deciding factor to go ahead with using Julia over C++ and Fortran for the Python SaMD project I'm working on. If I get some spare time I will take a look at messing around with jl_adopt_thread among other things.

  4. raphaelchinchilla commented on Nov 20, 2026

    @raphaelchinchilla

    Still reproducible on Julia 1.13.0 + PythonCall/JuliaCall 0.9.37; PyTorch's CUDA autograd hits it with no user threads

    Minimal example (pure JuliaCall: no torch, no CUDA):

    import os, threading
    os.environ["PYTHON_JULIACALL_HANDLE_SIGNALS"] = "yes"
    os.environ["PYTHON_JULIACALL_THREADS"] = "auto,1"
    from juliacall import Main as jl
    
    f = jl.seval("() -> (GC.gc(); sum(rand(1000)) > 0)")
    t = threading.Thread(target=f)
    t.start()
    t.join()           # never returns
    print("returned")
    • Removing GC.gc() from f makes it return on 0.9.37. On 0.9.36 it hangs even without GC.gc(), presumably because the first call compiles and that triggers a GC.
    • t.join(timeout=60) doesn't help: the main thread can't print afterwards, because the worker holds the GIL while it is inside Julia.
    • During the hang, every thread is in futex_wait, with no CPU use and no memory pressure.
    • With PYTHON_JULIACALL_HANDLE_SIGNALS=yes, SIGTERM is ignored and only SIGKILL ends the process.

    Versions: Linux x86_64, Python 3.13.15, Julia 1.13.0, JuliaCall/PythonCall 0.9.36 and 0.9.37 (12 Julia threads, auto,1). I have not tested with a single Julia thread. main at 4ed01ff has no threading changes since 0.9.37, so I didn't run it.

    Why it matters: PyTorch triggers it without the user creating any threads. PyTorch's autograd engine runs the backward of CUDA nodes on its own per-device worker threads (pt_autograd_0, …), not on the thread that called loss.backward(). So any torch.autograd.Function whose backward calls Julia hangs on CUDA as soon as Julia collects garbage. On CPU, torch runs the backward on the calling thread, so the same code works there. The faulthandler dump from the hang:

    Thread pt_autograd_0:
    File ".../PythonCall/src/JlWrap/any.jl", line 263 in call
    ... (our backward -> the Julia call)
    File ".../torch/autograd/function.py", line 452 in apply
    Main thread:
    File ".../torch/autograd/graph.py", line 1113 in _engine_run_backward
    File ".../torch/_tensor.py", line 623 in backward

    A guess at the mechanism (not verified): the trace in the original report ends in jl_gc_wait_for_the_world. That fits this picture: the thread that initialised Julia (Python's main thread) seems to remain in a GC-unsafe state while it runs ordinary Python code. If it then blocks in C (Thread.join, torch's engine wait) with the GIL released, it never reaches a Julia safepoint, so a stop-the-world GC started on another thread waits for it forever. If that's right, one fix would be for JuliaCall to put the calling thread into a GC-safe state whenever control returns from Julia to Python, and take it back out on the next call in. Julia already does this for adopted foreign threads around @cfunction calls. Does PR #824 (thread-state rework, on v1) cover this Python→Julia direction, or only Julia tasks calling Python?

  5. cjdoris commented on Nov 20, 2026

    @cjdoris
    Member

    Can you try the v1 branch please? Does the issue occur there?

  6. raphaelchinchilla commented on Nov 20, 2026

    @raphaelchinchilla

    Thanks, I tested it: v1 does not fix this. I found what does, and there are two separate causes.

    Testing v1 (25f82cc, which includes #824), Julia 1.13.0, PYTHON_JULIACALL_THREADS=4,1, HANDLE_SIGNALS=yes: the minimal example still hangs in the first round (with jl.seval renamed to jl.jl_eval). On 0.9.36 the whole process froze, so even t.join(timeout=30) never returned. On v1 the join timeout fires, but the worker's Julia call never returns. That fits #824 managing CPython thread states (the GIL) but not Julia's own per-thread state.

    Both causes come from the same thing: the thread that ran jl_init (Python's main thread) spends its time in Python, not in Julia.

    1. GC state. Your guess is right. After jl_init, the main thread stays GC-unsafe (gc_state == 0) while it runs Python. A stop-the-world collection started on another thread waits for it forever. Splitting the worker's call (each function compiled on the main thread first):

    worker's Julia call result
    arithmetic, no allocation returns
    Threads.@threads loop with atomics returns
    allocation-heavy (implicit GC) hangs; SIGKILL only
    GC.gc() hangs; SIGKILL only

    Fix: put the starting thread into the GC-safe state once, right after Julia starts:

    import ctypes
    from juliacall import Main as jl   # v1: jl.jl_eval instead of jl.seval
    lib = ctypes.CDLL(str(jl.seval('Base.Libc.Libdl.dlpath("libjulia-internal")')))
    lib.jl_get_ptls_states.restype = ctypes.c_void_p
    lib.jl_gc_safe_enter.argtypes = [ctypes.c_void_p]
    lib.jl_gc_safe_enter.restype = ctypes.c_int8
    previous = lib.jl_gc_safe_enter(lib.jl_get_ptls_states())   # returns 0: it was GC-unsafe
    

    This works because every later Python→Julia call goes through a @cfunction, whose wrapper switches the thread to GC-unsafe on entry and restores the saved state (now GC-safe) on exit. So Julia code never runs in the wrong state, and collections proceed while the main thread runs Python. It has to be done from Python: done inside a Julia call, the wrapper would undo it on return. jl_atexit_hook is called through ctypes rather than a @cfunction, so we restore the previous state with jl_gc_safe_leave in an atexit handler registered after JuliaCall's. With that, Julia's own atexit hooks still run.

    2. The I/O event loop. With the GC state fixed, the minimal example passes on 0.9.36 and v1, but PyTorch's CUDA backward still hung. gdb showed torch's pt_autograd_0 thread in CUDA.jl's first kernel compile, which pipes through the PTX assembler (run_and_collect → close(stream) → wait_close → jl_task_get_next), with every Julia thread idle in the scheduler. Outside a threaded region, Julia runs the libuv loop only on its I/O-loop thread, by default the thread that started Julia, and only while that thread waits in Julia's scheduler. Here that thread is waiting in torch's engine, so no thread processes the pipe event. Any Julia I/O (subprocesses, pipes, sleep, timers) from a Python-created thread hangs the same way while the main thread waits for it.

    Fix: Julia already has an API for this:

    jl.seval("Base.Experimental.make_io_thread()")
    

    We call it once at start-up and wait until Threads.maxthreadid() grows (the new thread has been adopted). With both changes, our PyTorch CUDA autograd tests pass: a torch autograd.Function whose backward calls Julia, run on torch's device thread. That's 61/61 CUDA tests and 658/658 CPU tests, on Julia 1.13.0 and JuliaCall 0.9.36. We also have regression tests where a Python thread collects garbage, and waits on a subprocess and a timer, in Julia while the main thread joins it. Without the corresponding fix, each one freezes the process.

    Suggested fix in JuliaCall: at the end of initialization, on the thread that ran jl_init, (1) enter the GC-safe state and (2) move the I/O loop off that thread (make_io_thread() or equivalent). Anything that later calls the libjulia C API directly from Python, such as the exit hook, needs to switch back to GC-unsafe first.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions