You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Windows 3.14 native parity: test_vector.py process died silently once on main #295
The native parity job on Windows with Python 3.14 failed once on main: the pytest process for tests/test_vector.py died without a traceback, 17 of 46 tests into the file.
The per-file log in the uploaded native-parity-windows-x64-py3.14 artifact shows collected 46 items and then 17 passing dots and nothing else: no failure, no Python traceback, no Windows fatal exception text. No XML was written, so the runner reported test_vector.py(CRASH) with 0 tests, while the other files of that job passed (2432 passed, 0 failed).
Counting the tests in file order, the 18th is TestLSMVectorIndex::test_lsm_vector_search_approximate_returns_k (a count of def test_ in the file at that commit, not a confirmed trace).
native/wheel/parity_wheel.py already carries a comment that whole-file runs of test_vector.py on Windows failed in the past, when the engine logged to the console under pytest's fd capture. This may be the same family, but nothing in this log shows it.
What is missing is the exit code of the dead process. An access violation, a kill by the runner and a pytest-internal abort all look the same in the current log. I am sending a one-line change so the next occurrence records it (CRASH(rc=...) in the status and the log). I will not retry crashed files automatically: one occurrence does not justify hiding a native crash.
If it happens again on Windows, this issue collects the occurrences.
Two more occurrences, both in the release run of 26.11.1.post1 (run 42744219731, tag commit dbe16709dd), job Parity against the installed wheel (windows-x64, ...):
Python
file that died
what the per-file log shows
3.11
tests/test_free_threading.py
collected 4 items, then nothing: no dot, no traceback, no XML
3.14
tests/test_find_nearest_query_forms.py
the process died the same way; the other 2,903 tests of that leg passed
Python 3.12 and 3.13 on Windows passed in the same run. The gate reports CRASH without a return code: the one-line change that records CRASH(rc=...) is not in release/26.11.1, so the exit code of the dead process is still unknown. Counting Windows parity legs of this code: the two full-matrix dispatches on the post1 candidate (3.12 and 3.14 each, 4 legs) had no crash; this release run has 2 crashes in 4 legs. The first occurrence (this issue) was on main before any post1 change, so the crash is not new in post1, but I cannot say yet whether its rate changed.
Next: re-run the failed legs of the release run; if the native wheel is published I will add a Windows-only loop (the two files, 30 repetitions per Python version, the 26.11.1 wheel against the post1 wheel) to estimate the rates, with the return code recorded.
The control run of the release configuration finished (run 42758616458): tag checkout (dbe16709dd), the stock harness (faulthandler disabled as in the release, no diagnostics), the wheel of the release run, Python 3.11 and 3.14, 5 repetitions each. All 10 legs ran all 148 test files to the end: 2,911 passed, 0 failed, 0 errors, 105 not applicable on the native wheel, no crash and no timeout. The jobs show red only because the throwaway workflow's own summary step exited 2; the parity step passed in each.
So the stock harness configuration does not by itself explain the crash: it did not happen in these 10 legs, and it did not happen in the 18 legs with diagnostics or in the 360 isolated runs of the two files either, while the release run lost a file in its Python 3.11 and 3.14 legs in both attempts. I do not know the cause. Something differs between the release run and every run I can start, and the next thing I can do is make the gate record it.
For 26.11.1.post2 I am changing the Windows parity harness only (no wheel code): record CRASH(rc=0x...), keep faulthandler output in a file for the native run and upload it with the parity artifact, retry a crashed file once in a fresh process on Windows only, mark a passing retry FLAKY-CRASH in the job summary and comment it here, and fail the leg and block the release when a file crashes twice. Test failures stay strict, and so do Linux and macOS. Tracking is in #347. If you have a Windows machine with Python 3.11 or 3.14 and want to help, I can share the wheel of the release run and a loop script that runs the two files with python -X faulthandler.
The Windows parity harness of the native release now handles the crashes this issue collects (in the tag 26.11.1.post2, CI and tests only):
a test file whose process dies without a result is reported as CRASH(rc=0x...), with the exit code of the process;
faulthandler writes to a file, and the file is uploaded with the parity artifact, so the crash site is kept;
on Windows only, a crashed file is run once more in a fresh process; a passing retry is FLAKY-CRASH in the job summary, and the crash-report job of test-native.yml posts it here; a file that dies twice fails the leg and blocks the release;
test failures stay strict, and Linux and macOS never retry.
The full-matrix dispatch of the native workflow on the post2 commit (run 42789425439) had no crash event: the Report parity crashes job printed no crash event in this run. The same holds for the release run of the tag (run 42804220685): the four Windows parity legs (Python 3.11 to 3.14) passed, and Report parity crashes printed no crash event in this run. The harness has not yet met a real crash.
Still unknown: why the processes of the 26.11.1.post1 release run died, in two files on two Python versions in both attempts, while 10 control legs, 18 diagnostic legs and the isolated runs did not reproduce it. The exit code and the fault trace are the missing facts, and the next occurrence records them.
I keep this issue open and append every CRASH and FLAKY-CRASH with its exit code and trace. A clean release run does not close it: the first occurrence was a single one as well.
Another occurrence on Windows, and this time the crash has a frame. In the examples job of the run for #374, 11_vector_index_build.py (vector build, arcadedb_sql backend, minimal) died 7 s in with EXCEPTION_ACCESS_VIOLATION (0xc0000005) in a C2-compiled jvector method, io.github.jbellis.jvector.graph.diversity.VamanaDiversityProvider.isDiverse. The JVM was Corretto 25.0.5+10 on windows-amd64 (G1, compact object headers). The same job passed in the three latest runs on main and in a rerun of the failed jobs, so it is intermittent.
The hs_err log and the minidump are kept as the artifacts jvm-crash-logdir-windows-amd64-py3.14 and jvm-crash-cwd-windows-amd64-py3.14 of https://github.com/ArcadeData/arcadedb-python/actions/runs/43137532222. I have not checked yet whether jvector's native or Vector API path is involved on Windows. If it happens again I will file it with the jvector project with the hs_err attached.
Activity
tae898 commentedon Nov 10, 2026
Two more occurrences, both in the release run of
26.11.1.post1(run 42744219731, tag commitdbe16709dd), jobParity against the installed wheel (windows-x64, ...):tests/test_free_threading.pycollected 4 items, then nothing: no dot, no traceback, no XMLtests/test_find_nearest_query_forms.pyPython 3.12 and 3.13 on Windows passed in the same run. The gate reports
CRASHwithout a return code: the one-line change that recordsCRASH(rc=...)is not inrelease/26.11.1, so the exit code of the dead process is still unknown. Counting Windows parity legs of this code: the two full-matrix dispatches on the post1 candidate (3.12 and 3.14 each, 4 legs) had no crash; this release run has 2 crashes in 4 legs. The first occurrence (this issue) was onmainbefore any post1 change, so the crash is not new in post1, but I cannot say yet whether its rate changed.Next: re-run the failed legs of the release run; if the native wheel is published I will add a Windows-only loop (the two files, 30 repetitions per Python version, the 26.11.1 wheel against the post1 wheel) to estimate the rates, with the return code recorded.
tae898 commentedon Nov 10, 2026
The control run of the release configuration finished (run 42758616458): tag checkout (
dbe16709dd), the stock harness (faulthandler disabled as in the release, no diagnostics), the wheel of the release run, Python 3.11 and 3.14, 5 repetitions each. All 10 legs ran all 148 test files to the end: 2,911 passed, 0 failed, 0 errors, 105 not applicable on the native wheel, no crash and no timeout. The jobs show red only because the throwaway workflow's own summary step exited 2; the parity step passed in each.So the stock harness configuration does not by itself explain the crash: it did not happen in these 10 legs, and it did not happen in the 18 legs with diagnostics or in the 360 isolated runs of the two files either, while the release run lost a file in its Python 3.11 and 3.14 legs in both attempts. I do not know the cause. Something differs between the release run and every run I can start, and the next thing I can do is make the gate record it.
For
26.11.1.post2I am changing the Windows parity harness only (no wheel code): recordCRASH(rc=0x...), keep faulthandler output in a file for the native run and upload it with the parity artifact, retry a crashed file once in a fresh process on Windows only, mark a passing retryFLAKY-CRASHin the job summary and comment it here, and fail the leg and block the release when a file crashes twice. Test failures stay strict, and so do Linux and macOS. Tracking is in #347. If you have a Windows machine with Python 3.11 or 3.14 and want to help, I can share the wheel of the release run and a loop script that runs the two files withpython -X faulthandler.tae898 commentedon Nov 10, 2026
The Windows parity harness of the native release now handles the crashes this issue collects (in the tag
26.11.1.post2, CI and tests only):CRASH(rc=0x...), with the exit code of the process;faulthandlerwrites to a file, and the file is uploaded with the parity artifact, so the crash site is kept;FLAKY-CRASHin the job summary, and thecrash-reportjob oftest-native.ymlposts it here; a file that dies twice fails the leg and blocks the release;The full-matrix dispatch of the native workflow on the post2 commit (run 42789425439) had no crash event: the
Report parity crashesjob printedno crash event in this run. The same holds for the release run of the tag (run 42804220685): the four Windows parity legs (Python 3.11 to 3.14) passed, andReport parity crashesprintedno crash event in this run. The harness has not yet met a real crash.Still unknown: why the processes of the
26.11.1.post1release run died, in two files on two Python versions in both attempts, while 10 control legs, 18 diagnostic legs and the isolated runs did not reproduce it. The exit code and the fault trace are the missing facts, and the next occurrence records them.I keep this issue open and append every
CRASHandFLAKY-CRASHwith its exit code and trace. A clean release run does not close it: the first occurrence was a single one as well.CI: Windows parity harness records crash exit codes, keeps a trace an…
integration: merge ci/post2-windows-crash-retry (aa223d0: Windows-onl…
integration: merge ci/post2-windows-crash-retry again (fc7b5e2: DIRTY…
Merge the 26.11.1.post2 release line into main
tae898 commentedon Nov 12, 2026
Another occurrence on Windows, and this time the crash has a frame. In the examples job of the run for #374,
11_vector_index_build.py(vector build,arcadedb_sqlbackend, minimal) died 7 s in withEXCEPTION_ACCESS_VIOLATION (0xc0000005)in a C2-compiled jvector method,io.github.jbellis.jvector.graph.diversity.VamanaDiversityProvider.isDiverse. The JVM was Corretto 25.0.5+10 on windows-amd64 (G1, compact object headers). The same job passed in the three latest runs on main and in a rerun of the failed jobs, so it is intermittent.The hs_err log and the minidump are kept as the artifacts
jvm-crash-logdir-windows-amd64-py3.14andjvm-crash-cwd-windows-amd64-py3.14of https://github.com/ArcadeData/arcadedb-python/actions/runs/43137532222. I have not checked yet whether jvector's native or Vector API path is involved on Windows. If it happens again I will file it with the jvector project with the hs_err attached.