Repository navigation
GPU Test Runner and Related Fixes #9112
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: dev
Are you sure you want to change the base?
Changes from all commits
b6f53b7
c9fdb83
f0d79b9
9c598ff
773760a
94fc798
c6f79f4
dbad3b9
10d8f54
460f12d
a98caf1
0975082
70deb93
11c4f87
e63c08e
fd6a40f
b496991
fb1c54f
525755f
01992d0
a088e37
940a1ef
0c4436f
8caf239
3060de1
837ea35
7866574
2dd33ee
6d67368
3635022
0e2b96c
cdc1e6f
6d1b06b
33591b7
ddb15b8
bff8977
af4096c
5498977
5647d37
5f42f7b
e6e9275
4ab6b76
ba204e3
4e5d527
0d68092
083d17f
002f040
62e09fa
1c8b222
db4269f
be49e72
f31f656
0d7dd16
5f628c6
5c6b039
258b2c7
181b2ea
47e09b2
2744fde
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,102 @@ | ||
| name: tests | ||
|
|
||
| on: | ||
| schedule: | ||
| - cron: "0 2 * * 3" # at 02:00 UTC Wednesday | ||
| workflow_dispatch: | ||
|
|
||
| permissions: | ||
| contents: read | ||
|
|
||
| concurrency: | ||
| # automatically cancel the previously triggered workflows when there's a newer version | ||
| group: cicd-tests-gpu-${{ github.event.pull_request.number || github.ref }} | ||
| cancel-in-progress: true | ||
|
|
||
| # Supported versions of Python and PyTorch are listed here for use below and as documentation | ||
| env: | ||
| # supported versions of Python | ||
| PYTHON_VER1: '3.11' | ||
| PYTHON_VER2: '3.12' | ||
| PYTHON_VER3: '3.13' | ||
| # PYTHON_VER4: '3.14' # TODO: not compatible with Torchscript, re-enable once Torchscript removed from MONAI | ||
| # PYTHON_VER5: '3.15' | ||
| # supported versions of PyTorch | ||
| PYTORCH_VER1: '2.12.1' | ||
| PYTORCH_VER2: '2.13.0' | ||
| PYTORCH_VER3: '2.14.1' | ||
| PYTORCH_VER4: '2.15.0' | ||
| TORCHVISION_VER1: '0.27.1' # used for testing with lowest PyTorch version (PYTORCH_VER1), update as needed | ||
|
|
||
| TQDM_DISABLE: 1 # squelch TQDM output to reduce log file size | ||
|
|
||
| PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION: python # https://github.com/Project-MONAI/MONAI/issues/4354 | ||
|
|
||
| BUILD_MONAI: 1 # always build MONAI and ensure it tries to use the compiled extensions | ||
| FORCE_CUDA: 1 # ensure CUDA code is built | ||
| CUDA_HOME: '/usr/local/cuda' # ensure the CUDA directory is definitely found | ||
| TORCH_CUDA_ARCH_LIST: "7.5 8.0 8.6 8.9 9.0+PTX 12.0" # specify supported architectures | ||
|
|
||
| jobs: | ||
|
|
||
| full-dep-gpu: | ||
| runs-on: [self-hosted, linux, gpu] | ||
| strategy: | ||
| fail-fast: false | ||
| timeout-minutes: 360 | ||
| steps: | ||
| - uses: actions/checkout@v7 | ||
| - name: Set up Python ${{ env.PYTHON_VER1 }} | ||
| uses: actions/setup-python@v6 | ||
| with: | ||
| python-version: ${{ env.PYTHON_VER1 }} | ||
| cache: 'pip' | ||
| env: | ||
| DEBIAN_FRONTEND: 'noninteractive' | ||
| - name: Install the complete dependencies | ||
| run: | | ||
| python -m pip install --upgrade pip wheel wheel_stub | ||
| python -m pip install torch==${PYTORCH_VER1} torchvision==${TORCHVISION_VER1} | ||
| python -m pip install --pre -U itk # TODO: needed? | ||
| python monai/config/print_dependencies.py build-system | xargs pip install --no-build-isolation | ||
| # tests below use the MONAI in the current directory, -e ensures the compiled code stays there and is found | ||
| python -m pip install --no-build-isolation torch==${PYTORCH_VER1} torchvision==${TORCHVISION_VER1} -e .[all,testing,nvimgcodec] | ||
| shell: bash | ||
| - name: Run (almost) all tests | ||
| run: | | ||
| python -c 'import monai._C; cap=monai._C.max_compute_capability(); assert cap > 0, f"Capabilities too low: {cap}"' | ||
| nvidia-smi || true | ||
| python monai/config/check_env.py --env --monai | ||
| ./runtests.sh -u --net | ||
| # env: # FIXME: NGC test failing with "RuntimeError: NGC API response is not ok. Failed to get token." Need to fix key? | ||
| # NGC_API_KEY: ${{ secrets.NGC_API_KEY }} | ||
| # NGC_ORG: ${{ secrets.NGC_ORG }} | ||
| # NGC_TEAM: ${{ secrets.NGC_TEAM }} | ||
|
|
||
| full-dep-gpu-hyena: | ||
| runs-on: [self-hosted, linux, gpu] | ||
| strategy: | ||
| fail-fast: false | ||
| timeout-minutes: 30 | ||
| steps: | ||
| - uses: actions/checkout@v7 | ||
| - name: Set up Python ${{ env.PYTHON_VER1 }} | ||
| uses: actions/setup-python@v6 | ||
| with: | ||
| python-version: ${{ env.PYTHON_VER1 }} | ||
| cache: 'pip' | ||
| env: | ||
| DEBIAN_FRONTEND: 'noninteractive' | ||
| - name: Install the complete dependencies | ||
| run: | | ||
| python -m pip install --upgrade pip wheel wheel_stub | ||
| python -m pip install torch==${PYTORCH_VER3} torchvision | ||
| python monai/config/print_dependencies.py build-system | xargs pip install --no-build-isolation | ||
| # tests below use the MONAI in the current directory, -e ensures the compiled code stays there and is found | ||
| python -m pip install --no-build-isolation -e .[testing,hyena] | ||
| shell: bash | ||
| - name: Run Hyena tests | ||
| run: | | ||
| python -c "from monai.networks.blocks.hyena import is_nvsubquadratic_available; \ | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Could we also assert |
||
| assert is_nvsubquadratic_available(), 'nvsubquadratic must be importable'" | ||
| python -m unittest tests.networks.blocks.test_hyena_block tests.networks.nets.test_hyena_nd_unetr tests.networks.nets.test_swin_unetr | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -170,3 +170,5 @@ runs | |
| *.pth | ||
|
|
||
| *zarr/* | ||
|
|
||
| uv.lock | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -114,6 +114,12 @@ def print_config(file=sys.stdout): | |
| fprint(f"MONAI flags: {HAS_EXT=}, {USE_COMPILED=}, {USE_META_DICT=}", file=file) | ||
| fprint(f"MONAI rev id: {monai.__revision_id__}", file=file) | ||
| fprint(f"MONAI __file__: {monai.__file__}", file=file) | ||
|
|
||
| if HAS_EXT: | ||
| fprint(f"monai._C.max_compute_capability: {monai._C.max_compute_capability()}", file=file) | ||
| fprint(f"monai._C.compiled_with_cuda: {monai._C.compiled_with_cuda()}", file=file) | ||
| fprint(f"monai._C.monai_torch_version: {monai._C.monai_torch_version()}", file=file) | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The macro behind this is Suggestion: either format the value as a dotted version ( |
||
|
|
||
| fprint("\nOptional dependencies:", file=file) | ||
| for k, v in get_optional_config_values().items(): | ||
| fprint(f"{k} version: {v}", file=file) | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -361,7 +361,8 @@ def forward(self, pred: torch.Tensor, target: torch.Tensor) -> torch.Tensor: | |
| wa, pa, wb, pb = self.parzen_windowing(pred, target) # (batch, num_sample, num_bin), (batch, 1, num_bin) | ||
|
|
||
| pab = torch.bmm(wa.permute(0, 2, 1), wb.to(wa)).div(wa.shape[1]) # (batch, num_bins, num_bins) | ||
| papb = torch.bmm(pa.permute(0, 2, 1), pb.to(pa)) # (batch, num_bins, num_bins) | ||
| papb = (pa.permute(0, 2, 1) * pb).to(pab) # (batch, num_bins, num_bins) | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The broadcast product here is the same outer product as the Suggestion: add a one-line comment noting that
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Would be better to retain |
||
|
|
||
| mi = torch.sum( | ||
| pab * torch.log((pab + self.smooth_nr) / (papb + self.smooth_dr) + self.smooth_dr), dim=(1, 2) | ||
| ) # (batch) | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -69,7 +69,7 @@ all = [ | |
| "itk>=5.2", | ||
| "jsonschema", | ||
| "lmdb", | ||
| "matplotlib>=3.6.3", | ||
| "matplotlib>=3.11.2", | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Keep both extras installable on Python 3.10. The project still declares Python 3.10 support ( Set a matplotlib minimum that Python 3.10 can resolve. If the newer floor is required on Python 3.11 and later, add a
|
||
| "MetricsReloaded @ git+https://github.com/Project-MONAI/MetricsReloaded@monai-support", | ||
| "mlflow>=3.15.2", | ||
| "nibabel", | ||
|
|
@@ -124,7 +124,7 @@ itk = ["itk>=5.2"] | |
| jsonschema = ["jsonschema"] | ||
| lmdb = ["lmdb"] | ||
| lpips = ["torchvision"] # compatibility extra; LPIPS is implemented within MONAI | ||
| matplotlib = ["matplotlib>=3.6.3"] | ||
| matplotlib = ["matplotlib>=3.11.2"] | ||
| # per #5882 this requirement must be on one line to get removed by sed in weekly-preview.yml | ||
| metrics_reloaded = ["MetricsReloaded @ git+https://github.com/Project-MONAI/MetricsReloaded@monai-support"] | ||
| mlflow = ["mlflow>=3.15.2"] | ||
|
|
@@ -138,6 +138,9 @@ nni = [ | |
| # required for GPU accelerated decoding of compressed DICOM images | ||
| nvimgcodec = [ | ||
| "pydicom", | ||
| # Tighter requirements than just the "cucim" optional dependency to get around a bug with cupy, also cucim>26.8 will | ||
| # require Numpy 2.x. | ||
| "cucim-cu13>=26.6.0; platform_system == 'Linux' and python_version >= '3.11'", | ||
| "cupy-cuda13x[ctk]!=14.1.0; platform_system == 'Linux'", | ||
| "nvidia-nvimgcodec-cu13[all]>=0.8.0; platform_system == 'Linux'" | ||
| ] | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These build settings are defined at workflow level, so the Hyena job (30 minute timeout) also enables
BUILD_MONAIandFORCE_CUDAand rebuilds the CUDA extension it never exercises. Related points in the same block:PYTHON_VER2,PYTHON_VER3,PYTORCH_VER2andPYTORCH_VER4are declared but never referenced,CUDA_HOMEandTORCH_CUDA_ARCH_LISThardcode one runner's layout, and the install step usespip install --pre -U itkwith a "TODO: needed?" comment.Suggestion: move
BUILD_MONAI,FORCE_CUDA,CUDA_HOMEandTORCH_CUDA_ARCH_LISTinto thefull-dep-gpujob, drop the unused version entries, and either pin the itk install or remove it.