Skip to content

Fix the OpenCL CPU compiler crash in the one-work-group radix sort - #3191

Open
ndgrigorian wants to merge 2 commits into
masterfrom
fix-radix-sort-cpu-crash
Open

ndgrigorian wants to merge 2 commits into
masterfrom
fix-radix-sort-cpu-crash

Conversation

@ndgrigorian

Copy link
Copy Markdown
Collaborator

This PR fixes a crash of the OpenCL CPU compiler in the one-work-group radix sort, which dpnp.sort, dpnp.argsort and their dpnp.tensor counterparts take for rows of 2049 to 16384 elements. Radix sort is the default for bool, int8, uint8, int16 and uint16, so a plain dpnp.sort of such rows can segfault on a CPU device, for example dpt.sort(dpt.zeros((1, 4096), dtype="i1", device="opencl:cpu")).

On a CPU device whose kernel gets sub-groups shorter than 16 (8 for the int8 kernels on an AVX2 CPU), the kernel puts a group_barrier after every unrolled counter update. That is the work-around for CMPLRLLVM-64592, added in IntelPython/dpctl@5125e11eb4. With 16 or more elements per work-item, those barriers crash the compiler in its sycl-kernel-reduce-cross-barrier-values pass (copyAndReplaceUses). Which data types crash depends on the compiler's heap state; once, int64 hung instead. The crash shows up with every OpenCL CPU runtime from 2024.18 (intel-opencl-rt 2025.0.4) to 2026.22 (2026.2.0).

The work-around is still needed: without it, the int8 radix argsort is unstable on the same runtimes. This PR replaces the barrier with a work-group atomic_fence. That keeps the int8 argsort stable, adds no barrier region, and so no longer crashes the compiler.

With a standalone SYCL reproducer, sort and stable argsort of bool, int8, uint8, int16, uint16, int32, int64, float32 and float64 match std::sort/std::stable_sort in both orders. That covers 1, 37 and 300 rows of every one-work-group size class, with full-range and low-cardinality data, on seven OpenCL CPU runtimes from 2025.0.4 to 2026.2.0. test_radix_sort_one_work_group covers rows of 257 to 16384 elements. Without the fix it segfaults on opencl:cpu for int8, uint8 and int32, and int64 times out; with it, it passes on opencl:cpu and level_zero:gpu, as do the sorting, top_k and unique tests.

GPUs don't take the work-around. Sort and argsort timings alternated between the two builds show no difference beyond noise on the iGPU (level_zero:gpu) or on opencl:cpu, at sizes the old build compiles.

  • Have you provided a meaningful PR description?
  • Have you added a test, reproducer or referred to an issue with a reproducer?
  • Have you tested your changes locally for CPU and GPU devices?
  • Have you made sure that new changes do not introduce compiler warnings?
  • Have you checked performance impact of proposed changes?
  • Have you added documentation for your changes, if necessary?
  • Have you added your changes to the changelog?

…ices rather than synchronize the work-group

On a CPU device whose kernel takes sub-groups shorter than 16, each counter
update was followed by a group_barrier, the work-around of CMPLRLLVM-64592
(int8 radix argsort comes out unstable). With 16 or more elements per
work-item, from rows of 2049 elements, these barriers crash the OpenCL CPU
compiler in its sycl-kernel-reduce-cross-barrier-values pass, so dpnp.sort and
dpnp.argsort of such rows segfault; bool, int8, uint8, int16 and uint16 take
the radix sort by default. Seen with every OpenCL CPU runtime from 2024.18 to
2026.22.

A work-group fence in place of the barrier still keeps int8 argsort stable and
adds no barrier region, so the compiler no longer crashes. Sort and argsort of
all integer and floating data types are correct on the same runtimes, and GPU
timings are unchanged since GPUs don't take the work-around.
// due to a bug in CPU device implementation, an additional
// synchronization is necessary for short sub-group sizes
// the CPU device compiler miscompiles the counting for short
// sub-groups unless each update is fenced (CMPLRLLVM-64592); a

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it the same as CMPLRLLVM-79305?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it's the same bug. CMPLRLLVM-79305 was filed from this kernel (int8 argsort, n=12320, so wg 512 and block 32, sub-group 8 on PTL-H), with the crash in sycl-kernel-reduce-cross-barrier-values (copyAndReplaceUses), and its reproducer has these two per-update barriers. Could you name CMPLRLLVM-79305 in this comment too, so the work-around can be reverted once it is fixed?

@github-actions

Copy link
Copy Markdown
Contributor

View rendered docs @ https://intelpython.github.io/dpnp/pull/3191/index.html

@github-actions

Copy link
Copy Markdown
Contributor

Array API standard conformance tests for dpnp=0.22.0dev2=np2py314h8d9cdd5_8 ran successfully.
Passed: 1491
Failed: 0
Skipped: 6

@coveralls

Copy link
Copy Markdown
Collaborator

Coverage Status

coverage: 84.269%. remained the same — fix-radix-sort-cpu-crash into master

Comment on lines +420 to +421
@pytest.mark.parametrize("n", [257, 1025, 2049, 4096, 8193, 16384])
@pytest.mark.parametrize("dtype", ["?", "i1", "u1", "i2", "i4", "i8"])

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This misses two size classes: 513..1024 and 4097..8192 (2049 and 4096 share one, 8193 and 16384 another). On master, int8 rows of 8192 crash the CPU compiler too. u2 takes the radix sort by default but is not here; on master a (2, 3000) uint16 sort spends over 10 minutes in the CPU compiler. Maybe [257, 513, 1025, 2049, 4097, 8193] and "u2"?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both additions catch crashes on master that the current sizes miss. I extended this test locally to 513, 1024, 4097 and 8192 for every dtype here, plus u2 at all sizes, and ran one node per fresh process on the CPU device:

  • Master crashed or hung in the 4097 nodes in 6-14 of 14 per CPU, and in the 8192 nodes in 4-14 of 14 (PTL-H, LNL, PVC's Xeon).
  • The u2 nodes failed in 8/20 (PTL-H), 5/20 (LNL) and 12/20 and 12/20 (Xeon, default and CL_CONFIG_CPU_VECTORIZER_MODE=8).
  • This PR passed all 68 nodes, on the CPU device and on level_zero:gpu.
  • 513 and 1024 never failed on master, since 8 elements per work-item don't crash the compiler. Still, that is the size class of the CMPLRLLVM-64592 report (1022 elements).

The stability check could also use data with few distinct values. In a standalone copy of the kernel, int8 argsort of 3-value data without the work-around was wrong in 34-52 % of runs at 1024-16384 on AVX2 CPUs. I haven't checked whether the random data here catches that.

@antonwolfy antonwolfy left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tested this head against master at the merge base, using the CI packages of both, on the CPU device of three AVX2 CPUs (PTL-H Core Ultra X7 368H, LNL, ARL-S Core Ultra 5 245K) and of PVC's AVX-512 Xeon 8480+. Three OpenCL CPU runtimes: intel-opencl-rt 2025.0.4 (2024.18.12.0.05), 2026.2.0 (2026.22.10.0.01) and the intel/llvm 2026-WW42 oclcpuexp (2026.22.10.0.02). Each sort or argsort ran in a fresh process, so each run built the kernel.

The fix works. At 4096-16384 elements, master failed in 4447 of 10440 runs, by a crash or a hang in sycl-kernel-reduce-cross-barrier-values. This PR failed in none of 15320:

CPU, cells master, failed/runs this PR
PTL-H, all 8 dtypes × 4096/8192/16384 × sort/argsort 491/1440 0/2400
LNL, 10 of those cells 100/300 0/500
ARL-S, the same 10 cells 91/300 0/500
Xeon 8480+, default 717/1440 0/2400
Xeon 8480+, CL_CONFIG_CPU_VECTORIZER_MODE=8 709/1440 0/2400
the cells that failed above, rerun on each of the three runtimes (PTL-H, LNL, Xeon mode 8) 2339/5520 0/5520

Rows of 1024 and 2048 never failed on either build. On the CPU device, test_radix_sort_one_work_group fails on master in 37-46 of its 72 nodes, all at 2049 and up, and passes on this PR. On level_zero:gpu it passes on both.

The work-around is still needed, and the fence keeps it correct. I used a standalone copy of the kernel, with int8 argsort of 3-value data. Without the work-around, the argsort was wrong in every process at sub-group 8 (what the kernels get on these AVX2 CPUs), in 33-52 % of reps at 1024, 4096 and 16384. With the fence, 0 of 45000 reps were wrong. With the barrier, none were wrong either, but the build crashed in up to 8 of 10 processes.

Some points in the description don't match what I saw:

  • Not only CPUs with sub-groups shorter than 16 crash. On the Xeon in the default mode the kernel has sub-group 16 and skips the work-around, yet master crashed in 717 of 1440 runs. work_around_needed is computed after the build, so the barrier is compiled in on every CPU device.
  • Every radix kernel gets sub-group 8 on the AVX2 CPUs, not only int8's, and every dtype crashes. All of bool, int8, uint8, int16, uint16, int32, int64 and float32 crash, in 22-99 of 180 runs each on PTL-H and the Xeon.
  • The int64 hang is the same bug. All 222 hangs dumped in that same pass (copyAndReplaceUses), int64 and uint16 among them.
  • Master can also return a wrong result without crashing. It did twice: a uint16 sort of 4096 elements (8121 of 8192 wrong) and an int64 argsort of 16384 (32756 of 32768 indices wrong).

The two open threads still apply: the comment could name CMPLRLLVM-79305, and the test could add the missing size class and u2 (my data is in that thread). The perf numbers are in a separate comment.

@antonwolfy

Copy link
Copy Markdown
Contributor

I timed sort and argsort against master at the merge base, with warm kernels. The cells were bool, int8, uint8, int16 and uint16 × 64-16384 elements × 1, 64 and 4096 rows (270 cells), over 3 rounds with the two builds alternating. Each process gives a median per cell.

Speedup = master/PR time (old/new, > 1 = faster now), geomean over cells:

Device n ≤ 2048 n ≥ 4096 cells of 1 ms or more
CPU, PTL-H Core Ultra X7 368H 0.98 1.08 1.06
CPU, LNL 1.03 1.11 1.08
CPU, PVC's Xeon 8480+ 1.00 1.01 1.00
Arc B390 (PTL-H), level_zero:gpu 0.95 0.99 1.00
LNL, level_zero:gpu 0.98 0.99 1.00
Arc Pro B70, level_zero:gpu 1.10 1.06 1.01
PVC, level_zero:gpu 0.99 0.99 1.01
  • AVX2 CPUs: faster from 4096 elements, where the fence replaces a barrier that runs. In a standalone copy of the kernel, the fence costs 0-1 % over no work-around and the barrier 15 %. On the Xeon the work-around doesn't run, and the change is neutral.
  • One small loss shows on all three CPUs: int8 sort of 128 elements × 4096 rows, at 0.86, 0.89 and 0.92 (PTL-H 1.76 → 1.93 ms). int8 sort of 64-128 elements × 64 rows is 0.82-0.89 on LNL and the Xeon. Master doesn't crash at these sizes. I didn't check whether those kernels take the work-around. The loss is small in absolute terms, so I don't think it needs a change.
  • GPUs: cells of 1 ms and more are within 1.5 % on all four. Below that, host-timed medians move by 15-30 % between processes of the same build on the iGPUs and the B70, more than any effect here, and which build looks faster flips between GPUs. On PVC the small cells are 1 % slower in geomean, and no cell is slower in every round. The GPU code of the one-work-group kernels does change (the fence is compiled into the branch that isn't taken), so not taking the work-around alone doesn't mean no GPU change.
  • The first call (kernel build + first sort) is 29-35 % faster at 4096-16384 elements on all four CPUs.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants