Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
version: 2
updates:
- package-ecosystem: github-actions
directory: /
schedule:
interval: monthly
groups:
actions:
patterns: ["*"]
52 changes: 36 additions & 16 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,13 @@ on:
pull_request:
workflow_dispatch:

permissions:
contents: read

env:
NUMPY_VERSION: "2.5.4"

# Actions are pinned to commit SHAs; Dependabot keeps them up to date.
jobs:
# cv2 is built with the Python Limited API, so one build per
# (OpenCV, profile, arch) serves every Python >= 3.12.
Expand All @@ -21,12 +25,12 @@ jobs:
arch: [x86_64, arm64]
runs-on: ${{ matrix.arch == 'arm64' && 'ubuntu-24.04-arm' || 'ubuntu-24.04' }}
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- run: ./build.sh 3.12 ${{ matrix.arch }} opencv ${{ matrix.opencv }} ${{ matrix.profile }}
- uses: actions/upload-artifact@v4
- uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: opencv-${{ matrix.opencv }}-${{ matrix.profile }}-${{ matrix.arch }}
path: artifacts/*.tar.gz
path: artifacts/*
if-no-files-found: error

numpy:
Expand All @@ -37,15 +41,16 @@ jobs:
arch: [x86_64, arm64]
runs-on: ${{ matrix.arch == 'arm64' && 'ubuntu-24.04-arm' || 'ubuntu-24.04' }}
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- run: ./build.sh ${{ matrix.python }} ${{ matrix.arch }} numpy ${{ env.NUMPY_VERSION }}
- uses: actions/upload-artifact@v4
- uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: numpy-${{ matrix.python }}-${{ matrix.arch }}
path: artifacts/*.tar.gz
path: artifacts/*
if-no-files-found: error

# Smoke-test and zip inside each target runtime.
# Smoke-test and zip inside each target runtime, then invoke every zip
# through the Lambda Runtime Interface Emulator with /opt mounted read-only.
package:
needs: [opencv, numpy]
strategy:
Expand All @@ -55,18 +60,19 @@ jobs:
arch: [x86_64, arm64]
runs-on: ${{ matrix.arch == 'arm64' && 'ubuntu-24.04-arm' || 'ubuntu-24.04' }}
steps:
- uses: actions/checkout@v4
- uses: actions/download-artifact@v4
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
with:
pattern: opencv-*-${{ matrix.arch }}
path: artifacts
merge-multiple: true
- uses: actions/download-artifact@v4
- uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
with:
name: numpy-${{ matrix.python }}-${{ matrix.arch }}
path: artifacts
- run: ./build.sh ${{ matrix.python }} ${{ matrix.arch }} package
- uses: actions/upload-artifact@v4
- run: python3 scripts/test_layer.py dist
- uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: layers-${{ matrix.python }}-${{ matrix.arch }}
path: dist/*
Expand All @@ -90,8 +96,10 @@ jobs:
runs-on: ubuntu-24.04
permissions:
contents: write
id-token: write
attestations: write
steps:
- uses: actions/download-artifact@v4
- uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
with:
pattern: layers-*
path: dist
Expand All @@ -101,13 +109,25 @@ jobs:
run: |
sha256sum *.zip > SHA256SUMS
{
echo "| File | Profile | Python | Arch | OpenCV | NumPy | Unzipped (MB) | Zipped (MB) |"
echo "|---|---|---|---|---|---|---|---|"
jq -rs 'sort_by(.file)[] | "| \(.file) | \(.profile) | \(.python) | \(.arch) | \(.opencv) | \(.numpy) | \(.unzipped_mb) | \(.zipped_mb) |"' *.json
echo "| File | Profile | Python | Arch | OpenCV | NumPy | Zipped (MB) | Unzipped (MB) | Import (ms) |"
echo "|---|---|---|---|---|---|---|---|---|"
ls *.json | grep -v '\.cdx\.json$' | xargs jq -rs 'sort_by(.file)[] | "| \(.file) | \(.profile) | \(.python) | \(.arch) | \(.opencv) | \(.numpy) | \(.zipped_mb) | \(.unzipped_mb) | \(.lambda_test.import_ms) |"'
echo
echo "Import time was measured in the Lambda Runtime Interface Emulator on GitHub-hosted runners, not on AWS Lambda."
echo
echo "Each zip has a CycloneDX SBOM (\`.cdx.json\`) and a build provenance attestation:"
echo
echo '```bash'
echo "gh attestation verify <file>.zip -R ${GITHUB_REPOSITORY}"
echo '```'
} > ../notes.md
- uses: softprops/action-gh-release@v2
- uses: actions/attest-build-provenance@4d101475d8b20a2381f78447822ac1eab6504dd8 # v4.2.2
with:
subject-path: dist/*.zip
- uses: softprops/action-gh-release@8a0cd651e5b2365bd640854a3fa5dfc70e4875f3 # v3.0.4
with:
files: |
dist/*.zip
dist/*.cdx.json
dist/SHA256SUMS
body_path: notes.md
63 changes: 59 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,8 @@ Minimal headless OpenCV layers for AWS Lambda, built from source.

- **Small**: OpenCV and its codecs are linked statically into a single stripped `cv2.abi3.so`, without FFmpeg or OpenBLAS.
- **Fast cold start**: Precompiled bytecode is included, because Lambda cannot write a bytecode cache to `/opt`.
- **Runtime-matched**: Built and smoke-tested inside the official `public.ecr.aws/lambda/python` images.
- **Reproducible**: Deterministic zips with `SHA256SUMS`, built by GitHub Actions.
- **Runtime-matched**: Built inside the official `public.ecr.aws/lambda/python` images, and every zip is invoked through the Lambda Runtime Interface Emulator before release.
- **Verifiable**: Pinned sources and build tools, deterministic zips, `SHA256SUMS`, a CycloneDX SBOM per zip, and GitHub build provenance attestations.


## Quick Start
Expand Down Expand Up @@ -105,6 +105,12 @@ The layers keep only what image processing on Lambda needs.
| Haar cascade XML files | Ship the XML files you need with your function. `cv2.CascadeClassifier` is only available with 4.14.0, because OpenCV 5 removed it from the main repository |
| `python3.11` and older | These runtimes run on Amazon Linux 2 and are not supported |

> [!WARNING]
> **NumPy is built without BLAS.** This is fine for image processing, where NumPy mostly holds pixel arrays for OpenCV. If your function uses NumPy as a compute engine (large matrix products, decompositions, solvers), these layers are the wrong choice. Use the official `opencv-python-headless` and `numpy` wheels, which bundle OpenBLAS.

> [!WARNING]
> **x86_64 binaries require AVX2.** `cv2` is compiled with `CPU_BASELINE=AVX2` and has no fallback path. Every AWS Lambda x86_64 host supports AVX2, but other machines may not, such as older CPUs or some emulators. On a CPU without AVX2, `import cv2` stops with an error that the build does not support the current CPU. The `arm64` builds use the default NEON baseline.


## How It Stays Small

Expand All @@ -118,6 +124,48 @@ The layers keep only what image processing on Lambda needs.
| `.pyc` compiled with `unchecked-hash` | Zip rounds file times to 2 seconds, so timestamp-checked bytecode would be ignored on every cold start |


## Testing

Every zip goes through two checks before it can be merged or released. CI fails if either one fails.

| Check | Where | What it verifies |
|---|---|---|
| Smoke test | Build container, before zipping | `cv2` and `numpy` import, and basic image operations work |
| Lambda invocation | Fresh `public.ecr.aws/lambda/python` container with the zip extracted to a read-only `/opt`, invoked through the Runtime Interface Emulator with 1024 MB of memory | See below |

The Lambda invocation runs [`tests/lambda/handler.py`](tests/lambda/handler.py) twice, once cold and once warm, and checks:

- `cv2` and `numpy` load from `/opt/python`, and the precompiled bytecode is present
- JPEG, PNG, and WebP round trips
- ORB matching recovers a known shift through `findHomography` (`standard`, `dnn`)
- `cv2.dnn.blobFromImage` (`dnn`)
- Each profile has its modules and lacks the ones it excludes, such as `cv2.dnn` in `standard` and `cv2.VideoCapture` everywhere
- Import time, a 12 MP decode → resize → blur → WebP pipeline, and peak memory, which are recorded in each zip's `.json`

> [!NOTE]
> The emulator runs on GitHub-hosted runners, so the timings are indicative only. Cold start, memory, and latency on AWS Lambda itself are not measured.

To run the Lambda invocation locally after a build, run `python3 scripts/test_layer.py dist`.


## Supply Chain

| Item | How |
|---|---|
| Source archives | The OpenCV and NumPy source archives are verified against SHA256 hashes pinned in [`scripts/build_layer.py`](scripts/build_layer.py) before building |
| Build tools | CMake, Ninja, Meson, meson-python, and Cython versions are pinned. NumPy builds without build isolation, so only the pinned tools are used |
| Build environment | The Lambda image digest, GCC version, and tool versions are recorded in each zip's `.json` |
| GitHub Actions | Actions are pinned to commit SHAs and updated by Dependabot |
| SBOM | Each release includes a CycloneDX SBOM (`.cdx.json`) per zip, listing OpenCV, NumPy, and the third-party libraries linked into `cv2` |
| Provenance | Each zip has a GitHub build provenance attestation |

`SHA256SUMS` proves that a download is intact. The attestation also proves that the zip was built by this repository's workflow:

```bash
gh attestation verify opencv-4.14.0-minimal-python3.13-arm64.zip -R 076923/aws-lambda-python-opencv-layer
```


## Build from Source

Requires Docker. Building `arm64` on an `x86_64` machine uses QEMU emulation and is slow.
Expand All @@ -129,7 +177,12 @@ Requires Docker. Building `arm64` on an `x86_64` machine uses QEMU emulation and

```
dist/opencv-4.14.0-minimal-python3.13-arm64.zip
dist/opencv-4.14.0-minimal-python3.13-arm64.json # versions and sizes
dist/opencv-4.14.0-minimal-python3.13-arm64.json # versions, sizes, build info, test results
dist/opencv-4.14.0-minimal-python3.13-arm64.cdx.json # CycloneDX SBOM
```

```bash
python3 scripts/test_layer.py dist # invoke every zip through the Lambda emulator
```

<details>
Expand Down Expand Up @@ -172,7 +225,9 @@ opencv-{opencv}-{profile}-python{python}-{arch}.zip

## Release

Push a `v*` tag. GitHub Actions builds all 36 zips and attaches them to the release with `SHA256SUMS` and a size table.
Push a `v*` tag. GitHub Actions builds and tests all 36 zips, then attaches them to the release with their SBOMs, `SHA256SUMS`, provenance attestations, and a size table.

To change the OpenCV or NumPy version, update the workflow matrix and add the SHA256 of the new source archive to `OPENCV_SOURCES` or `NUMPY_SOURCES` in [`scripts/build_layer.py`](scripts/build_layer.py).

```bash
git tag v2026.11.0
Expand Down
9 changes: 8 additions & 1 deletion build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,15 @@ esac
cd "$(dirname "$0")"
mkdir -p artifacts dist

IMAGE="public.ecr.aws/lambda/python:${PY}"
docker pull -q --platform "$PLATFORM" "$IMAGE" > /dev/null
# Record the exact image digest in the build info.
DIGEST=$(docker image inspect --format '{{index .RepoDigests 0}}' "$IMAGE")

docker run --rm --platform "$PLATFORM" \
-e LAMBDA_IMAGE="$DIGEST" \
-e HOST_UID="$(id -u)" -e HOST_GID="$(id -g)" \
-v "$PWD:/work" -w /work \
--entrypoint python3 \
"public.ecr.aws/lambda/python:${PY}" \
"$IMAGE" \
scripts/build_layer.py "$@"
Loading