Skip to content

Build and test conda packages and wheels - #294

Open
jharlow-intel wants to merge 1 commit into
mainfrom
ci/intel-python-ci
Open

jharlow-intel wants to merge 1 commit into
mainfrom
ci/intel-python-ci

Conversation

@jharlow-intel

@jharlow-intel jharlow-intel commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator

Adds ci/ and .github/workflows/intel-ci.yml, which build the conda packages and the wheels for Linux and Windows, on every declared Python, and test each in a clean environment. The recipe skips macOS, as conda-forge/mkl_umath-feedstock does.

  • conda-recipe/ is now the only recipe: the conda-forge one that was conda-recipe-cf/, which takes its version and build number from git describe, as the recipe it replaces did. Its Linux build threads MKL with gnu_thread and llvm-openmp, as conda-forge's packages do; Windows and the wheels use intel_thread with libiomp5. conda-package.yml and conda-package-cf.yml are removed, as intel-ci.yml builds and tests the recipe.
  • The conda packages build Python 3.10 to 3.15 plus 3.14t, from the python: list in conda_build_config.yaml. The wheels follow the pyproject.toml classifiers: 3.10 to 3.15 plus 3.14t and 3.15t.
  • Wheels build with cibuildwheel and icx 2026.2: on Linux from Intel's oneAPI yum repository, on Windows from conda-forge's dpcpp_win-64, which ci/install_icx.py installs. They leave MKL and the compiler runtime out. ci/build.py adds their requirements to each wheel through [tool.gh-ci.pin-compatible]: mkl from the mkl-devel version they were built with, and intel-openmp and intel-cmplr-lib-rt from the icx version, each up to its next major. The Linux before-all and ci/install_icx.py print icx's version for ci/build.py. icx links the Linux libraries with full RELRO, which ci/build.py checks, along with a library search path inside $ORIGIN.
  • meson.build adds Library to CMake's search path on Windows, where mkl-devel puts its CMake config.
  • ci/test.py runs pytest --pyargs mkl_umath on the installed package.
  • Each test job runs for every package that built, even when another build failed. A pull request builds the oldest and newest Python plus the newest free-threaded one; main and tags build them all.
  • conda-build records the source commit in each package's info/about.json.
  • ci/benchmark.py runs the asv suite in benchmarks/ against an installed package, for running the benchmarks outside intel-ci.yml, which doesn't run it: one quick sample of each benchmark for a pull request, a full measurement for a tag.

Miniforge is pinned to 26.7.2-0, as 26.9.4-0 makes setup-miniconda's conda init fail on Windows runners.

Validation: run 44355589179 on b568107 builds and tests the reduced pull request matrix on Linux and Windows. Not run yet: the full matrix, which only main and tags build.

@jharlow-intel
jharlow-intel marked this pull request as ready for review October 23, 2026 20:42
@jharlow-intel jharlow-intel self-assigned this Oct 28, 2026
@jharlow-intel
jharlow-intel force-pushed the ci/intel-python-ci branch 12 times, most recently from 02315bd to f53b1ce Compare November 3, 2026 18:00
@jharlow-intel jharlow-intel changed the title Build and test conda-recipe-cf for Intel Python CI Build and test conda packages and wheels Nov 5, 2026
@jharlow-intel
jharlow-intel force-pushed the ci/intel-python-ci branch 4 times, most recently from d7e1dc3 to 96e092f Compare November 9, 2026 21:50
Comment thread pyproject.toml Outdated
# What the wheels need at run time, which conda packages get from the recipe:
# mkl >= the mkl-devel version the build used, < its next major
pin-compatible = { mkl = "mkl-devel" }
wheel-requires = ["intel-openmp >=2026.2,<2027", "intel-cmplr-lib-rt >=2026.2,<2027"]

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On Linux icx isn't pinned (dnf install intel-oneapi-compiler-dpcpp-cpp, compiler/latest), but on Windows it's 2026.2.0, and the runtime requirements here are hardcoded to >=2026.2,<2027. With the next oneAPI release the Linux wheels will build with a newer icx and still allow the old runtime. dpnp#3151 pins icx to 2026.2 and gets these from the compiler version through pin-compatible. Could we do the same here?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done as in dpnp: Linux installs intel-oneapi-compiler-dpcpp-cpp-2026.2. The before-all and ci/install_icx.py print the compiler's version under Installed SDK versions:, and [tool.gh-ci.pin-compatible] maps intel-openmp and intel-cmplr-lib-rt to it, so the floors follow the icx the wheels were built with.

Comment thread conda-recipe/meta.yaml Outdated
@@ -1,18 +1,22 @@
{% set version = "0.6.0" %}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The other repos take the version from GIT_DESCRIBE_TAG/GIT_DESCRIBE_NUMBER, and so did the old recipe. With 0.6.0 and build 0 hardcoded, every build from main looks the same as the 0.6.0 release. The "takes its version from git describe" comment in intel-ci.yml is also wrong now.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, fixed: the version and build number come from GIT_DESCRIBE_TAG and GIT_DESCRIBE_NUMBER again, so the intel-ci.yml comment is true again.

Comment thread conda-recipe/build.sh
--no-index \
--prefix "${PREFIX}" \
CC=icx CXX=icpx $PYTHON -m pip install --no-build-isolation --no-deps \
-Csetup-args="-Dmkl_threading=gnu_thread" \

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Linux conda package now uses gnu_thread with llvm-openmp, while the old recipe, Windows and the wheels use intel_thread with libiomp5. Is that intended? If these packages go to the Intel channel, it might be worth mentioning in the description.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Intended: it's conda-forge's recipe, and mkl_umath-feedstock's build.sh passes the same -Dmkl_threading=gnu_thread with llvm-openmp in host. Windows and the wheels keep intel_thread. The description now says so.

Comment thread pyproject.toml
[tool.cython-lint]
ignore = ['E722'] # do not use bare 'except'
max-line-length = 80
[tool.cibuildwheel]

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[tool.cython-lint] got removed here. Was that on purpose? cython-lint still runs in pre-commit, now with its default settings.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not on purpose; it got lost in a rebase. Restored.

Comment thread .github/workflows/intel-ci.yml Outdated
conda-remove-defaults: "true"

- name: Install conda-build
run: conda install -y -q -n base --override-channels -c conda-forge conda-build

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mkl-service pins conda-build=26.9.1. Should we pin it here too?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pinned to 26.9.1, as in mkl-service.

Comment thread conda-recipe/meta.yaml Outdated
- {{ pin_compatible('mkl') }}
- {{ pin_compatible('intel-openmp') }}
- {{ pin_compatible('intel-cmplr-lib-rt') }}
# NumPy 1 vs 2 loops are chosen at compile time (USE_NUMPY_2), so a build

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

USE_NUMPY_2 was removed in gh-289, so this comment is out of date. Same for "CI overrides it with --numpy" in conda_build_config.yaml, since nothing passes --numpy now.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed both.

Comment thread ci/config.yml
wheel: "mkl_umath {platform} Wheels Python {python}"
test:
requires: [pytest]
benchmark:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

intel-ci.yml doesn't run benchmark.py or read this section. Could ci/README.md mention that, like in dpnp#3151? The description also makes it sound like PRs run the benchmarks.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ci/README.md now says so, in dpnp#3151's words, and the description no longer reads as if pull requests run the benchmarks.

Comment thread AGENTS.md
- **Workflows:** `.github/workflows/`
- `conda-package.yml` — main conda build/test pipeline
- `conda-package-cf.yml` — conda-forge-oriented build/test pipeline
- `intel-ci.yml` — Intel CI: builds and tests the conda packages and the wheels; see `ci/README.md`

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing tests against the Intel channel anymore, but line 80 still says Intel's NumPy is required. Maybe update it and mention in the description that the Intel channel build is gone?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated line 80 and the matching line under Distribution: NumPy 2 from conda-forge or PyPI, which the conda packages and the wheels are tested with. The description already says conda-recipe/ is now the conda-forge recipe only.

Adds a ci/ directory and .github/workflows/intel-ci.yml. The workflow builds
conda-recipe/ with conda-build 26.9.1 and the wheels with cibuildwheel, for
every platform and Python version the recipe and pyproject.toml declare, and
tests each package in a clean environment with ci/test.py.

conda-recipe/ is now the only recipe, the conda-forge one that was
conda-recipe-cf/; the Intel-channel recipe, conda-package.yml, which copied
wheels out through WHEELS_OUTPUT_FOLDER, and conda-package-cf.yml are removed,
as intel-ci.yml builds and tests the same recipe, running the test jobs for
each package that built even when another build failed.
conda_build_config.yaml now lists the Python versions. The recipe takes its
version and build number from git describe, as the recipe it replaces did.
Its Linux build threads MKL with gnu_thread and llvm-openmp, as conda-forge's
packages do; Windows and the wheels use intel_thread.

The wheels link MKL's layers, the Intel OpenMP runtime and icx's runtime.
[build-system] requires mkl-devel, and [tool.gh-ci.pin-compatible] maps mkl
to it and intel-openmp and intel-cmplr-lib-rt to icx: ci/build.py adds
Requires-Dist >= the version the build installed, < its next major, the way
conda's run_exports pin it, and fails a build that logs no version. It reads
mkl-devel's from python -m build's list, and icx's from the "Installed SDK
versions:" list the Linux before-all and ci/install_icx.py print; icx is
2026.2 on both. In [project] the runtime wheels would fail the recipe's pip
check, as conda-forge's packages have no pip metadata. ci/build.py takes
METADATA's line ending from the headers. The repair commands leave MKL's and
the compiler runtime's libraries out, and the Linux wheels are tagged
manylinux_2_28 only, like the mkl wheel.

icx already links the Linux libraries with full RELRO and a stack protector,
so the wheels need no extra flags. ci/build.py refuses a wheel whose ELF
files lack full RELRO. It keeps an RPATH or RUNPATH entry only when it is
$ORIGIN or a directory under it, as meson-python keeps the isolated build
env's absolute lib directory, and refuses a file left with none. It rewrites
the wheels in scratch, so a refused wheel never reaches --output, and
refuses two built packages with the same file name, which would overwrite
each other there.

conda-build gets --extra-meta remote_url=... sha=... when GitHub Actions
sets GITHUB_SERVER_URL, GITHUB_REPOSITORY and GITHUB_SHA, so each conda
package records the commit it was built from in info/about.json.

ci/hooks.py tests each wheel in a venv made from conda-forge's Python, as pip
users install it. Every python: entry in conda_build_config.yaml names its
ABI, which ci/matrix.py requires. ci/matrix.py reads x86 as any x86 CPU and
osx64 as undefined, as conda-build does, and refuses a conda-build skip that
uses osx64. A pull request also builds and tests the newest free-threaded
version, and every push and tag run gets a concurrency group of its own.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants