Repository navigation
Closed
Labels
Description
I'm starting to see multiple Array API compliant libraries that are just thin wrappers around another arbitrary Array API compliant library:
- @mdhaber's marray
- @lucascolley's quantity-array
- I'm deliberately keeping Dask out of scope for this issue, as there is a lot of nuance involved in it.
- xarray's wrapping design also predates the Array API and xarray is not (today) Array API compatible, but I can see it joining the fray soon.
These libraries currently fail to work with array-api-compat, array-api-extra, or anything that relies on them (like scipy) except in the most basic use cases, e.g. wrapping around a writeable numpy or cupy array.
These are the reasons I could find so far:
array_api_compat.is_writeable_arrayblindly returns True for unknown namespaces. This is problematic also when wrapping around read-only numpy arrays.array_api_compat.is_lazy_arrayincorrectly returns False for wrapped Dask. This will lead to major performance issues.xpx.at,xpx.apply_where, andxpx.lazy_applyhave special casing for Dask which won't be triggered when Dask is wrapped. You can't just changearray_api_compat.is_dask_arrayto return True for wrapped dask objects, because there are a wealth of dask-specific calls (da.map_blocks, passing the arrays to a@delayedfunction) that won't work out of the box.- The same functions, plus
nunique, have special casing for JAX. Again, there are jax-specific function calls e.g.jax.pure_callbackthat won't accept wrapped arrays. - Delegated functions (
xpx.pad) should work thanks to their generic code path, but they are untested; so for examplepadon PyTorch works due to special casing, but there are no tests that verify that the generic implementation ofpadapplied to a wrapper around PyTorch will work.
Activity
crusaderky commentedon Feb 19, 2025
jax.jitwill fail with these libraries too. The reason is that jitted functions require inputs that are jax arrays. This could be fixed by an ad-hoc wrapper around it, that unwraps the arrays just before entering the jitted function and then re-wraps it immediately afterwards.mdhaber commentedon Feb 20, 2025
I thought the
MArray__array_namespace__info might be relevant here:So if the base library underlying the
MArrayis important for special-casing, it's not hard to get at that information. Perhaps there can be an informal (or formal?) standard for this sort of thing, and standard recipe for parsing that information?As for library-specific calls, I've thought about having separately-installable packages (or maybe just separate repos that get vendored) that could be maintained by the developers of the underlying libraries. This would allow Dask developers to add additional features to the
marray.dasknamespace or to the array objects themselves.crusaderky commentedon Feb 20, 2025
I don't think that's realistic, particularly for lesser-known wrappers.
Maybe a more sustainable approach could be for all wrapper namespaces to offer a coherent API that lets you unwrap and rewrap the inner objects:
so this (mdhaber/marray#91 (comment)):
would become:
crusaderky commentedon Feb 20, 2025
Functions like
atthat deal in special cases would need to add two lines:[-]Wrapped namespaces support[/-][+]ENH: Wrapped namespaces support[/+]lucascolley commentedon Aug 12, 2026
it's not clear that there is anything actionable here at the moment, so I'm going to convert this to a discussion. The discussion here will be helpful, but it probably needs working through via a real use-case.