Repository navigation
Build and test conda packages and wheels - #294
jharlow-intel wants to merge 1 commit into
Conversation
02315bd to
f53b1ce
Compare
d7e1dc3 to
96e092f
Compare
96e092f to
95766aa
Compare
| # What the wheels need at run time, which conda packages get from the recipe: | ||
| # mkl >= the mkl-devel version the build used, < its next major | ||
| pin-compatible = { mkl = "mkl-devel" } | ||
| wheel-requires = ["intel-openmp >=2026.2,<2027", "intel-cmplr-lib-rt >=2026.2,<2027"] |
There was a problem hiding this comment.
On Linux icx isn't pinned (dnf install intel-oneapi-compiler-dpcpp-cpp, compiler/latest), but on Windows it's 2026.2.0, and the runtime requirements here are hardcoded to >=2026.2,<2027. With the next oneAPI release the Linux wheels will build with a newer icx and still allow the old runtime. dpnp#3151 pins icx to 2026.2 and gets these from the compiler version through pin-compatible. Could we do the same here?
There was a problem hiding this comment.
Done as in dpnp: Linux installs intel-oneapi-compiler-dpcpp-cpp-2026.2. The before-all and ci/install_icx.py print the compiler's version under Installed SDK versions:, and [tool.gh-ci.pin-compatible] maps intel-openmp and intel-cmplr-lib-rt to it, so the floors follow the icx the wheels were built with.
| @@ -1,18 +1,22 @@ | |||
| {% set version = "0.6.0" %} | |||
There was a problem hiding this comment.
The other repos take the version from GIT_DESCRIBE_TAG/GIT_DESCRIBE_NUMBER, and so did the old recipe. With 0.6.0 and build 0 hardcoded, every build from main looks the same as the 0.6.0 release. The "takes its version from git describe" comment in intel-ci.yml is also wrong now.
There was a problem hiding this comment.
Right, fixed: the version and build number come from GIT_DESCRIBE_TAG and GIT_DESCRIBE_NUMBER again, so the intel-ci.yml comment is true again.
| --no-index \ | ||
| --prefix "${PREFIX}" \ | ||
| CC=icx CXX=icpx $PYTHON -m pip install --no-build-isolation --no-deps \ | ||
| -Csetup-args="-Dmkl_threading=gnu_thread" \ |
There was a problem hiding this comment.
The Linux conda package now uses gnu_thread with llvm-openmp, while the old recipe, Windows and the wheels use intel_thread with libiomp5. Is that intended? If these packages go to the Intel channel, it might be worth mentioning in the description.
There was a problem hiding this comment.
Intended: it's conda-forge's recipe, and mkl_umath-feedstock's build.sh passes the same -Dmkl_threading=gnu_thread with llvm-openmp in host. Windows and the wheels keep intel_thread. The description now says so.
| [tool.cython-lint] | ||
| ignore = ['E722'] # do not use bare 'except' | ||
| max-line-length = 80 | ||
| [tool.cibuildwheel] |
There was a problem hiding this comment.
[tool.cython-lint] got removed here. Was that on purpose? cython-lint still runs in pre-commit, now with its default settings.
There was a problem hiding this comment.
Not on purpose; it got lost in a rebase. Restored.
| conda-remove-defaults: "true" | ||
|
|
||
| - name: Install conda-build | ||
| run: conda install -y -q -n base --override-channels -c conda-forge conda-build |
There was a problem hiding this comment.
mkl-service pins conda-build=26.9.1. Should we pin it here too?
There was a problem hiding this comment.
Pinned to 26.9.1, as in mkl-service.
| - {{ pin_compatible('mkl') }} | ||
| - {{ pin_compatible('intel-openmp') }} | ||
| - {{ pin_compatible('intel-cmplr-lib-rt') }} | ||
| # NumPy 1 vs 2 loops are chosen at compile time (USE_NUMPY_2), so a build |
There was a problem hiding this comment.
USE_NUMPY_2 was removed in gh-289, so this comment is out of date. Same for "CI overrides it with --numpy" in conda_build_config.yaml, since nothing passes --numpy now.
| wheel: "mkl_umath {platform} Wheels Python {python}" | ||
| test: | ||
| requires: [pytest] | ||
| benchmark: |
There was a problem hiding this comment.
intel-ci.yml doesn't run benchmark.py or read this section. Could ci/README.md mention that, like in dpnp#3151? The description also makes it sound like PRs run the benchmarks.
There was a problem hiding this comment.
ci/README.md now says so, in dpnp#3151's words, and the description no longer reads as if pull requests run the benchmarks.
| - **Workflows:** `.github/workflows/` | ||
| - `conda-package.yml` — main conda build/test pipeline | ||
| - `conda-package-cf.yml` — conda-forge-oriented build/test pipeline | ||
| - `intel-ci.yml` — Intel CI: builds and tests the conda packages and the wheels; see `ci/README.md` |
There was a problem hiding this comment.
Nothing tests against the Intel channel anymore, but line 80 still says Intel's NumPy is required. Maybe update it and mention in the description that the Intel channel build is gone?
There was a problem hiding this comment.
Updated line 80 and the matching line under Distribution: NumPy 2 from conda-forge or PyPI, which the conda packages and the wheels are tested with. The description already says conda-recipe/ is now the conda-forge recipe only.
Adds a ci/ directory and .github/workflows/intel-ci.yml. The workflow builds conda-recipe/ with conda-build 26.9.1 and the wheels with cibuildwheel, for every platform and Python version the recipe and pyproject.toml declare, and tests each package in a clean environment with ci/test.py. conda-recipe/ is now the only recipe, the conda-forge one that was conda-recipe-cf/; the Intel-channel recipe, conda-package.yml, which copied wheels out through WHEELS_OUTPUT_FOLDER, and conda-package-cf.yml are removed, as intel-ci.yml builds and tests the same recipe, running the test jobs for each package that built even when another build failed. conda_build_config.yaml now lists the Python versions. The recipe takes its version and build number from git describe, as the recipe it replaces did. Its Linux build threads MKL with gnu_thread and llvm-openmp, as conda-forge's packages do; Windows and the wheels use intel_thread. The wheels link MKL's layers, the Intel OpenMP runtime and icx's runtime. [build-system] requires mkl-devel, and [tool.gh-ci.pin-compatible] maps mkl to it and intel-openmp and intel-cmplr-lib-rt to icx: ci/build.py adds Requires-Dist >= the version the build installed, < its next major, the way conda's run_exports pin it, and fails a build that logs no version. It reads mkl-devel's from python -m build's list, and icx's from the "Installed SDK versions:" list the Linux before-all and ci/install_icx.py print; icx is 2026.2 on both. In [project] the runtime wheels would fail the recipe's pip check, as conda-forge's packages have no pip metadata. ci/build.py takes METADATA's line ending from the headers. The repair commands leave MKL's and the compiler runtime's libraries out, and the Linux wheels are tagged manylinux_2_28 only, like the mkl wheel. icx already links the Linux libraries with full RELRO and a stack protector, so the wheels need no extra flags. ci/build.py refuses a wheel whose ELF files lack full RELRO. It keeps an RPATH or RUNPATH entry only when it is $ORIGIN or a directory under it, as meson-python keeps the isolated build env's absolute lib directory, and refuses a file left with none. It rewrites the wheels in scratch, so a refused wheel never reaches --output, and refuses two built packages with the same file name, which would overwrite each other there. conda-build gets --extra-meta remote_url=... sha=... when GitHub Actions sets GITHUB_SERVER_URL, GITHUB_REPOSITORY and GITHUB_SHA, so each conda package records the commit it was built from in info/about.json. ci/hooks.py tests each wheel in a venv made from conda-forge's Python, as pip users install it. Every python: entry in conda_build_config.yaml names its ABI, which ci/matrix.py requires. ci/matrix.py reads x86 as any x86 CPU and osx64 as undefined, as conda-build does, and refuses a conda-build skip that uses osx64. A pull request also builds and tests the newest free-threaded version, and every push and tag run gets a concurrency group of its own.
95766aa to
b568107
Compare
Adds
ci/and.github/workflows/intel-ci.yml, which build the conda packages and the wheels for Linux and Windows, on every declared Python, and test each in a clean environment. The recipe skips macOS, asconda-forge/mkl_umath-feedstockdoes.conda-recipe/is now the only recipe: the conda-forge one that wasconda-recipe-cf/, which takes its version and build number fromgit describe, as the recipe it replaces did. Its Linux build threads MKL withgnu_threadandllvm-openmp, as conda-forge's packages do; Windows and the wheels useintel_threadwithlibiomp5.conda-package.ymlandconda-package-cf.ymlare removed, asintel-ci.ymlbuilds and tests the recipe.python:list inconda_build_config.yaml. The wheels follow thepyproject.tomlclassifiers: 3.10 to 3.15 plus 3.14t and 3.15t.dpcpp_win-64, whichci/install_icx.pyinstalls. They leave MKL and the compiler runtime out.ci/build.pyadds their requirements to each wheel through[tool.gh-ci.pin-compatible]:mklfrom the mkl-devel version they were built with, andintel-openmpandintel-cmplr-lib-rtfrom the icx version, each up to its next major. The Linux before-all andci/install_icx.pyprint icx's version forci/build.py. icx links the Linux libraries with full RELRO, whichci/build.pychecks, along with a library search path inside$ORIGIN.meson.buildaddsLibraryto CMake's search path on Windows, where mkl-devel puts its CMake config.ci/test.pyrunspytest --pyargs mkl_umathon the installed package.mainand tags build them all.info/about.json.ci/benchmark.pyruns the asv suite inbenchmarks/against an installed package, for running the benchmarks outsideintel-ci.yml, which doesn't run it: one quick sample of each benchmark for a pull request, a full measurement for a tag.Miniforge is pinned to 26.7.2-0, as 26.9.4-0 makes
setup-miniconda'sconda initfail on Windows runners.Validation: run 44355589179 on
b568107builds and tests the reduced pull request matrix on Linux and Windows. Not run yet: the full matrix, which onlymainand tags build.