Repository navigation
Implement kernels for median and paritition functions - #3138
ndgrigorian wants to merge 42 commits into
Conversation
|
View rendered docs @ https://intelpython.github.io/dpnp/pull/3138/index.html |
|
Array API standard conformance tests for dpnp=0.22.0dev2=np2py314h8d9cdd5_48 ran successfully. |
0f57805 to
8667696
Compare
e24ca7b to
2df1e76
Compare
8667696 to
58ce933
Compare
236c2e1 to
43ca63d
Compare
58ce933 to
89cc1d4
Compare
74de517 to
c7864b9
Compare
759b1f2 to
9ab3e53
Compare
There was a problem hiding this comment.
🟡 Changes recommended
Active compatibility issues remain: even-count median and nanmedian return a finite value where NumPy overflows to infinity, and argpartition rejects valid NumPy integer indices.
2 open issues
What changed in this PR
This PR extends dpnp’s SYCL radix-selection kernels to support median and partition operations, replacing the legacy partition backend and adding public argpartition APIs.
Changes:
- Generalizes radix selection for
median,nanmedian,partition, andargpartition, retaining sorting fallbacks. - Adds
argpartitionfunction and array-method APIs, and fixes median output handling withkeepdims=True. - Expands correctness and device-placement tests, removes legacy partition code, and updates build and coverage configuration.
| File | Description |
|---|---|
scripts/gen_coverage.py |
Includes the sorting extension in dpnp coverage. |
dpnp/tests/third_party/cupy/sorting_tests/test_sort.py |
Enables upstream argpartition tests. |
dpnp/tests/test_usm_type.py |
Tests partition result allocation types. |
dpnp/tests/test_sycl_queue.py |
Tests partition queue placement. |
dpnp/tests/test_statistics.py |
Tests median selection and output handling. |
dpnp/tests/test_sort.py |
Adds partition and argpartition coverage. |
dpnp/tests/test_nanfunctions.py |
Tests NaN-aware median selection. |
dpnp/tests/test_indexing.py |
Enables partition/index equivalence checks. |
dpnp/libtensor/source/tensor_sorting.cpp |
Registers partition bindings. |
dpnp/libtensor/source/statistics/statistics_py.cpp |
Registers median bindings. |
dpnp/libtensor/source/statistics/median.hpp |
Declares median binding initialization. |
dpnp/libtensor/source/statistics/median.cpp |
Implements median validation and dispatch. |
dpnp/libtensor/source/sorting/partition.hpp |
Declares partition binding initialization. |
dpnp/libtensor/source/sorting/partition.cpp |
Implements partition validation and dispatch. |
dpnp/libtensor/source/CMakeLists.txt |
Builds new bindings and adjusts coverage instrumentation. |
dpnp/libtensor/include/kernels/sorting/radix_utils.hpp |
Adds inverse radix keys and revises zero normalization. |
dpnp/libtensor/include/kernels/sorting/radix_select.hpp |
Generalizes selection kernels for partition and median. |
dpnp/dpnp_utils/dpnp_utils_statistics.py |
Routes median operations and handles output views. |
dpnp/dpnp_utils/dpnp_algo_utils.pyx |
Removes inline from _object_to_tuple. |
dpnp/dpnp_iface_sorting.py |
Adds argpartition and shared partition dispatch. |
dpnp/dpnp_iface_nanfunctions.py |
Routes floating-point nanmedian through selection. |
dpnp/dpnp_array.py |
Adds the array argpartition method. |
dpnp/dpnp_algo/dpnp_algo.pyx |
Removes the legacy sorting wrapper include. |
dpnp/dpnp_algo/dpnp_algo.pxd |
Removes the legacy partition enum declaration. |
dpnp/dpnp_algo/dpnp_algo_sorting.pxi |
Deletes the legacy Cython partition wrapper. |
dpnp/dpnp_algo/CMakeLists.txt |
Removes obsolete sorting wrapper dependencies. |
dpnp/backend/src/dpnp_iface_fptr.cpp |
Removes legacy sorting registration. |
dpnp/backend/src/dpnp_fptr.hpp |
Removes the sorting registration declaration. |
dpnp/backend/kernels/dpnp_krnl_sorting.cpp |
Deletes legacy partition kernels. |
dpnp/backend/include/dpnp_iface.hpp |
Removes legacy partition API declarations. |
dpnp/backend/include/dpnp_iface_fptr.hpp |
Removes legacy partition function identifiers. |
dpnp/backend/CMakeLists.txt |
Removes the legacy sorting build source. |
dpnp/__init__.py |
Exports argpartition. |
CHANGELOG.md |
Records new APIs, migration, and output-shape fixes. |
Review effort: Balanced
antonwolfy
left a comment
There was a problem hiding this comment.
Tested on a PTL-H iGPU (level_zero:gpu) and on opencl:cpu. The selection itself looks right: the changed test files, the CuPy sorting and statistics tests and the tensor sort and top_k tests pass on both devices with all integer types, and about 300k fuzz cases against NumPy found no wrong partition, argpartition or top_k result, also for inputs still being written by earlier device work. No new compiler or pytest warnings.
Open bot points I confirm:
- Copilot, median of two equal large middle values (
radix_select.hpp:332): reproduced, and master givesinfthere; details in that thread. - Copilot, NumPy integer
kth(dpnp_iface_sorting.py:73): reproduced fornumpy.int64(1),numpy.array(1),numpy.array([1, 3])and lists of NumPy integers. Master'spartitionrejects them too, butargpartitionis new.
I'd fix the median overflow and the overlap check before merge; the inline points below are smaller.
The internal Jenkins Windows build jobs of this head failed on 11-05 and their logs are gone. Other PRs failed in the same jobs at that time for infrastructure reasons, and the Windows 3.13 build and the GitHub Windows jobs passed, so a rerun should tell.
9ab3e53 to
8616f96
Compare
There was a problem hiding this comment.
🟡 Changes recommended
An exception while obtaining the kernel bundle or allocating temporaries can leave the submitted fill without a keep-alive event, allowing it to write to freed USM memory during cleanup.
1 open issue
2 resolved since last review
Review effort: Balanced
There was a problem hiding this comment.
Re-tested the current head on a PTL-H iGPU (Arc B390, level_zero:gpu) and on opencl:cpu, with master at the merge base built the same way next to it.
The points from the last round are fixed, and I checked each on both devices: even rows average their two middle values for every float type on the sub-group, one-group and multi-group paths, _partition and _argpartition reject src sharing memory with dst, kth behaves like NumPy's for NumPy integers, 0-d and 1-D integer arrays, booleans, floats, and empty and 2-D kth (only numpy.uint64(2**64 - 1) differs: NumPy wraps it to -1), and the docstrings and backend leftovers are gone.
The full dpnp/tests (79.5k passed) and dpnp/tests/tensor (15.5k passed) suites pass on both devices, and so do the changed test files with DPNP_TEST_ALL_INT_TYPES=1. The only pytest warning is in test_sign_nan, which master has too. The build has no compiler warnings, the legacy backend included, and neither do the GitHub Linux and Windows build logs.
Performance, geomean of master time / PR time, alternating builds (master, PR, PR, master), every result checked against NumPy:
| level_zero:gpu | opencl:cpu | |
|---|---|---|
median |
6.1x | 13.6x |
nanmedian |
12.0x | 30.8x |
partition, one kth |
5.1x | 10.4x |
argpartition vs master's argsort |
9.2x | 12.6x |
partition, several kth (still a sort) |
0.99x | 0.99x |
complex median (still a sort) |
0.94x to 1.07x | 0.94x to 1.31x |
top_k |
0.98x | 1.00x |
Shapes are 1-D 106, 106 + 1, 107 and 107 + 1, 1000x1000 along each axis and flattened, 105x16 and 16x106, for float32, int32 and int64, plus float64 and float16 for median. Small inputs gain the most: median of 10 float32 elements takes 31 us instead of 173 us on the GPU. The only thing that got slower is a first call: on opencl:cpu the first 1-D partition JIT-compiles the radix select in 468 ms, where master's backend kernel took 92 ms, and each later call saves about 16 ms. The top_k slowdown is small but consistent; details inline.
Build and footprint (clean in-place build, 8 jobs, 16 cores):
| master | PR | |
|---|---|---|
| wall time | 15:35 | 15:39 |
| sum of all build steps | 4910 s | 5005 s |
largest ld.lld (_tensor_impl) |
3.41 GiB | 3.40 GiB |
ld.lld of _tensor_sorting_impl |
2.73 GiB | 2.90 GiB |
| peak of all build processes | 5.57 GiB | 5.56 GiB |
installed .so files |
183.2 MiB | 187.8 MiB |
The extra work is off the critical path, which is the _tensor_impl link: the two new sources take 17 to 18 s each, and the _tensor_sorting_impl and _statistics_impl links take about 25 s longer each. _tensor_sorting_impl and _statistics_impl grow by 2.7 MiB each, while libdpnp_backend_c shrinks by 0.7 MiB. RSS after import dpnp is 3 MB higher.
Coverage goes from 84.40% to 84.52%. partition.cpp has 99 of 106 lines covered and median.cpp 91 of 117; the uncovered lines are binding error paths, which you left for the binding-test PR, one early return, and a constexpr helper that llvm-cov counts as code. radix_select.hpp isn't reported at all, since llvm-cov ignores the headers under libtensor/include/kernels.
…han in an allocation of their own
… of a lower data type as NumPy does
…too, which takes the sort path
…has host USM atomics
6d970d3 to
066d95f
Compare
|
Measured the performance of the current head against master at the merge base on six Intel GPU hosts, on the GPU (
There are 515 cases per device:
Each case gets 3 warm-up calls, then the median of per-call times with a queue wait after each call. Master and the PR run alternately (master, PR, PR, master) in 2 rounds, 3 on LNL. NumPy runs in the same harness on the host CPU (single-threaded). Every dpnp result is checked against NumPy. Geomean of master time / PR time:
* master's runs on that CPU device kept crashing in the OpenCL CPU runtime's JIT, which happens with both builds, so only On the Arc B390 (PTL-H), Device memory and launches per call, from a UR trace on the Arc B390 (PTL-H):
Geomean of NumPy time / PR time:
NumPy wins below about 103 to 104 elements on launch latency, for example The numbers from the last round hold on the same GPU model, Arc B390 (PTL-H), with the same benchmark rerun on another PTL-H host and a newer driver (the CPU columns are different CPUs):
|
… done, waiting for that copy only
… quarter of its compute units
…e split over work-groups with keys of 4 bytes or more
…n one per kind When the elements ordering before, with and after the selected one go to sinks of one type, which differ only in their offsets, gather_selected now picks the sink and the rank per element and stores once. The one-work-group path and the final gather of the multi-group path now pass sinks of one type, since a shifted sink shifted again keeps its type. A CPU device vectorizes each of the three masked stores per work-item for 1-byte values: the 8-wide AVX2 body of the uint8 partition kernel had 96 vpextrb stores to memory, each behind a branch, and now has 32. On opencl:cpu uint8 partition gets 1.4-1.9x faster, level with or faster than sort, and uint32 partition and argpartition 15-30% faster; GPU timings don't change. The output is bit-identical to before.
|
|
||
| // from about a row per four compute units of a GPU, a work-group per row | ||
| // already occupies it and the extra passes cost more than they save | ||
| const std::size_t max_multi_group_iters = |
There was a problem hiding this comment.
The new cut doesn't depend on the row length. For rows of 2^18 elements or more, the row counts just above it are now 2-5x slower, top_k included. My cut timings in round 3 were all at n = 65536, and there n_cus / 4 works. But the extra passes of the multi-group path cost about the same at any n, while its gain over one work-group per row grows with n. So for longer rows the break-even moves up to n_cus rows and beyond.
Speedup over the round-3 head (old/new, < 1 = slower now), float32, level_zero:gpu unless noted:
| GPU (CUs, new cut) | rows | n | median |
partition |
top_k, vs master |
|---|---|---|---|---|---|
| PVC (448, 112) | 112 | 2^18 | 0.49 | 0.40 | 0.38 |
| 448 | 2^18 | 0.88 | 0.81 | 0.82 | |
| 112 | 10^6 | 0.27 | 0.25 | 0.24 | |
| 448 | 10^6 | 0.73 | 0.66 | 0.64 | |
| LNL (64, 16) | 16 | 2^18 | 0.42 | 0.45 | 0.53 |
| 16 | 10^6 | 0.30 | 0.35 | 0.56 | |
| 32 | 10^6 | 0.59 | 0.64 | 0.73 | |
LNL, opencl:gpu (56, 14) |
14 | 10^6 | 0.32 | 0.41 | 0.60 |
| PTL-H (4 Xe3 cores) (32, 8) | 8 | 2^18 | 0.29 | 0.26 | 0.36 (vs before) |
| 8 | 2^20 | 0.20 | 0.19 | 0.31 (vs before) |
int64 behaves the same: at (112, 10^6) on PVC, median is 0.35 and top_k 0.41 of master. top_k now takes this cut too, while master's top_k took the multi-group path up to 4·n_cus rows. So at these shapes it's up to 4x slower than on master: PVC float32 at (112, 10^6) takes 16.7 ms against 3.96 ms. Where I timed both, top_k before this change and on master were within 2% of each other.
On PVC the cut is low even at n = 65536. At 112-113 rows, median, partition and top_k are 0.62-0.77 of before for float32 and int64, and top_k is 0.62-0.65 of master. One work-group per row only wins from 448 rows there (1.1-1.6x). I have no rows between 113 and 448, so I can't say where it flips.
Suggestion: let the cut grow with the row length and keep n_cus / 4 at n = 65536, for example:
const std::size_t max_multi_group_iters =
(dev.is_gpu())
? std::min(target_groups, n_cus * n_values / (4 * multi_group_min_n))
: target_groups;That gives n_cus rows at 2^18 and the old 4·n_cus from about 10^6. On the numbers above it would remove every long-row loss except 12-19% at 448 rows of 2^18 on PVC and up to 20% at 64-128 rows of 2^18 on LNL. PVC's loss at 112-113 rows of 65536 would stay; a larger factor than / 4 for large GPUs may be worth a try. I haven't built it; I can time a candidate on PVC, an Arc Pro B70 and LNL.
This also changes dpnp.top_k and dpnp.tensor.top_k on GPUs: from n_cus rows of 65536 up they are up to 1.5x faster than on master. So the changelog could name them too.
| // sinks of one type differ only in their offsets, so an element is written | ||
| // with one store: a CPU device makes a masked store of each sink per | ||
| // work-item for 1-byte values | ||
| static constexpr bool one_sink = keeps_less && keeps_greater && |
There was a problem hiding this comment.
The single store makes 1-byte partition 4-17% slower on the Xe2 and Xe3 GPUs here, and 4-byte 4-7% on these and on PVC. On the AVX2 CPU devices it's 2-2.7x faster, more than you measured, but on PVC's AVX-512 Xeon only 1.0-1.3x. The commit message says GPU timings don't change. That holds on the two Xe-LP and Xe-LPG GPUs I have: Iris Xe (ADL-P), and the Arrow Lake-S iGPU, which is the generation of your 0x7d41. It doesn't hold on PVC, the Arc Pro B70, Arc B390 (PTL-H), PTL-H (4 Xe3 cores) or LNL.
Speedup over the round-3 head (old/new, < 1 = slower now), partition of 10^4 rows of 2048 elements, middle kth, builds alternating in the same harness:
| uint8 | int8 | bool | uint32 | float32 | |
|---|---|---|---|---|---|
| PVC | 0.97 | 0.97 | 0.99 | 0.94 | 0.93 |
| Arc Pro B70 | 0.92 | 0.92 | 0.99 | 0.95 | 0.96 |
| Arc B390 (PTL-H) | 0.83 | 0.85 | 0.99 | 0.95 | 0.97 |
| PTL-H (4 Xe3 cores) | 0.88 | 0.88 | 0.98 | 0.95 | 0.96 |
| LNL | 0.94 | 0.96 | 0.99 | 0.93 | 0.95 |
| Iris Xe (ADL-P) | 1.01 | 1.01 | 1.01 | 0.99 | 0.99 |
| 6 AVX2 CPU devices | 1.96-2.49 | 1.98-2.47 | 2.14-2.71 | 1.34-1.75 | 1.35-1.65 |
| Xeon Platinum 8480+ (AVX-512) | 1.07 | 1.10 | 1.05 | 1.06 | 1.02 |
The B390 numbers held in a separate run of three processes per build, alternating: uint8 took 1.9-2.2 ms per call before and 2.6 ms now. On the B70, a sweep over row lengths gives 0.90-0.93 for int8 and uint8 from 2048 to 65535 elements, and 0.88-0.96 for int16. On the PTL-H (4 Xe3 cores), a single uint8 row of 10^7 is 0.79. On the Xe2 and Xe3 GPUs argpartition loses less (0.82-1.0 for 1-byte types). On PVC the 2- to 8-byte keys lose in every round: partition and argpartition of 10^4 rows of 129-384 elements are 0.93-0.96. median and top_k don't change anywhere. The Xeon has masked byte stores, which fits your explanation of the AVX2 gain. The output is bit-identical everywhere: on seven CPU devices, all 24346 hashed outputs are equal raw.
If you'd rather keep the GPUs as they were, the single store could be used on CPU devices only, for example through a template flag the dispatch sets from dev.is_cpu(). That adds a second set of whole-row kernels. Keeping it for every device is fine with me too, since the CPU gain is much larger than the GPU loss.
| ht_ev.wait() | ||
| ev.wait() | ||
|
|
||
| expected = numpy.median(a, axis=-1).astype(dst.dtype) |
There was a problem hiding this comment.
On a device without fp64, the median of two 64-bit integers can now differ from NumPy's, and only the review thread says so. integer_midpoint rounds the exact mean once to float32, while NumPy first adds the two values in float64. On Iris Xe (ADL-P), where dpnp.median of integers returns float32:
| int64 row | dpnp.median on Iris Xe |
numpy.median cast to float32 |
exact mean |
|---|---|---|---|
[-2**63, 2**63 - 1] |
-0.5 | 0.0 | -0.5 |
[2**62 + 2**38, 2**62 + 2**38 + 1] |
262 + 239 | 2**62 | 262 + 238 + 0.5 |
The kernel's value is the float32 nearest to the exact mean, so it is the better one. But it isn't NumPy's value cast to float32, and on a device with fp64 the same rows give NumPy's. It only happens for values of about 2**52 or more in magnitude, where float64 can't hold the sum. The first row is the one new mismatch that fuzzing against NumPy found on that GPU.
AGENTS.md asks for such a difference to be in the docstring and in a test with a comment. A sentence in the Notes of dpnp.median and dpnp.nanmedian would cover it, plus these two rows here, expecting the exact value, with a comment that NumPy differs.
|
Re-measured the current head against the round-3 head and against master at the merge base, on the GPU ( Geomean speedup over master (old/new) on the main set,
On the CPU devices, over the round-3 head: What moved since round 3:
The test suites pass on every host where they ran. Fuzzing against NumPy, and comparing every output hash with the round-3 head, found nothing beyond what's in the comments. |
| if dtype.kind in "iu": | ||
| if dtype.itemsize < 4: | ||
| return False | ||
| elif dtype != dpnp.float32: | ||
| return False |
There was a problem hiding this comment.
The sort pays for 64-bit integers, but on most GPUs it loses for int32 and float32 random rows; I'd route int64 and uint64 only. These are the numbers from my reply in the resolved thread above. Routed rows (10^4 rows of 65-128 elements), sort now against select before (old/new, > 1 = faster now), level_zero:gpu, for four data distributions: full (the whole range of the dtype), rand (±10^9), d100 (0..99) and d4 (0..3):
| GPU | int64, uint64 d4/d100 |
int64, uint64 rand |
int64, uint64 full |
int32 d4/d100 |
int32 rand |
int32 full |
float32 rand |
|---|---|---|---|---|---|---|---|
| PVC | 2.65-3.99 | 1.84-2.81 | 0.60-0.79 | 1.44-2.18 | 0.71-0.90 | 0.65-0.82 | 0.74-1.03 |
| Arc Pro B70 | 2.25-3.61 | 1.64-2.54 | 0.54-0.74 | 1.28-1.86 | 0.70-0.84 | 0.61-0.75 | 0.72-0.93 |
| Arc B390 (PTL-H) | 1.52-2.45 | 1.13-1.63 | 0.49-0.52 | 0.94-1.36 | 0.56-0.59 | 0.52-0.56 | 0.59-0.84 |
| PTL-H (4 Xe3 cores) | 1.67-2.52 | 1.19-1.75 | 0.40-0.53 | 0.96-1.41 | 0.49-0.62 | 0.44-0.57 | 0.55-0.73 |
| LNL | 2.51-3.43 | 1.81-2.43 | 0.51-0.62 | 1.41-2.13 | 0.67-0.74 | 0.58-0.68 | 0.77-1.03 |
| Iris Xe (ADL-P) | 2.54-3.35 | 1.87-2.46 | 0.83-0.92 | 1.49-2.16 | 0.87-0.93 | 0.81-0.85 | 0.89-1.07 |
int32 gains only with a few distinct values. float32 d4/d100 is 0.70-1.04 on the B390 and PTL-H too. Single rows go the same way: float32 and int32 (100,) take 0.71-0.89 of the time with select on PVC, the B70 and LNL, while int64 (100,) is 1.1-1.5x faster by sort on every GPU. Full-range int64 loses too, but I'd take ±10^9 and duplicates as the common int64 data.
| if dtype.kind in "iu": | |
| if dtype.itemsize < 4: | |
| return False | |
| elif dtype != dpnp.float32: | |
| return False | |
| if dtype.kind not in "iu" or dtype.itemsize < 8: | |
| return False |
TestPartitionBySort.test_choice would then expect False for ("i4", 100, "gpu") and ("f4", 100, "gpu").
| * Reduced the overhead of `dpnp.ndarray.flags` and `dpnp.tensor.usm_ndarray.flags` [#3168](https://github.com/IntelPython/dpnp/pull/3168) | ||
| * Changed `dpnp.linalg.LinAlgError` to derive from `numpy.linalg.LinAlgError`, so `except numpy.linalg.LinAlgError` also catches errors raised by dpnp linear algebra functions [#3184](https://github.com/IntelPython/dpnp/pull/3184) | ||
| * Bumped the minimum required `dpctl` version to `0.24.0` [#3169](https://github.com/IntelPython/dpnp/pull/3169) | ||
| * Changed `dpnp.median`, `dpnp.nanmedian` and `dpnp.partition` with a single `kth` to use a radix select instead of a full sort for all but complex data types and the row lengths a sort partitions faster [#3138](https://github.com/IntelPython/dpnp/pull/3138) |
There was a problem hiding this comment.
"and the row lengths a sort partitions faster" reads as if dpnp.median and dpnp.nanmedian also took a sort for some row lengths, and as if the sort were always faster there. Only dpnp.partition on a GPU routes, and it depends on the data. Maybe: "... for all but complex data types and, on a GPU, dpnp.partition of rows of 65 to 128 elements of 8-byte integers", or whatever the route ends up covering. (dpnp.top_k changes with the new cut too; see the comment on the cut.)


This PR proposes the addition of updates to radix selection algorithms to generalize for cases of
medianandpartitionrelated functions, which allows the removal ofdpnp_partition_cfrom the DPNP backendThe median and partition functions can pass through either a sorting or a radix selection path, based on input types (and in the case of partition, always when
kthis multiple elements)Supersedes #2423