CKS: pin the autoscaler and CCM in the binaries ISO builder by bhouse-nexthop · Pull Request #14649 · apache/cloudstack

bhouse-nexthop · 2026-11-09T21:19:12Z

Description

This PR pins the cluster autoscaler and the cloud controller manager (CCM) that scripts/util/create-kubernetes-binaries-iso.sh bundles into the CKS binaries ISO, and writes a list of the ISO's contents into the ISO. Because the autoscaler now changes with the Kubernetes version, a cluster upgrade also redeploys the autoscaler from the new ISO.

Fixes #14648

Related: #14598 fixes the same download problems in the node-side scripts (deploy-provider, autoscale-kube-cluster, deploy-csi-driver). This PR doesn't touch those scripts, so the two don't conflict.

What was wrong

Autoscaler:
- The manifest came from kubernetes/autoscaler master, and its image is apache/cloudstack-kubernetes-autoscaler:latest.
- That tag was last pushed in March 2022 and is a 1.24.0-alpha development build. So every ISO shipped the same old autoscaler, whatever its Kubernetes version.
- The cluster-autoscaler only supports the Kubernetes minor version it was released with.
CCM:
- The manifest came from cloudstack-kubernetes-provider main, with whatever image main named on the day of the build.
- For a long time that was v1.1.0, which has a node-initialisation bug (Fix providerID implementation cloudstack-kubernetes-provider#58, fixed in v1.2.0).
No record: nothing in the ISO said which versions it held, so two ISOs for the same Kubernetes version could differ.
Silent download failures: most downloads used curl without -f, so a 404 page was saved and bundled as a manifest or binary.

What changed

Autoscaler (ISO builder)

By default, the script picks the newest cluster-autoscaler-<minor>.<patch> release tag for the ISO's Kubernetes minor version.
- It finds the tag with git ls-remote, so no API token is needed.
- Today, 1.31.x gets 1.31.5 and 1.33.x gets 1.33.6.
AUTOSCALER_VERSION=1.33.4 picks an exact release instead.
- The script warns if the release is for a different minor version.
- The build stops with an error if no release exists for the minor version, for example a new Kubernetes minor before its autoscaler release.
The manifest comes from that release tag.
- I checked that the CloudStack example (cloudprovider/cloudstack/examples/cluster-autoscaler-standard.yaml) exists at the latest tag of every minor from 1.20 to 1.37, so no fallback manifest is needed.
- Before 1.20 the example doesn't exist at the release tags, so builds for those (long EOL) versions now fail on the download.
The image is replaced with the upstream image of the same release, registry.k8s.io/autoscaling/cluster-autoscaler:v<version>.
- The example names the unversioned apache/cloudstack-kubernetes-autoscaler:latest at every tag, so it can't be used as is.
- The upstream image is built with all cloud providers, CloudStack included.
- It runs ./cluster-autoscaler from /, as the manifest expects.
- It is published for amd64 and arm64.
Extra RBAC.
- The CloudStack example's ClusterRole hasn't kept up with newer releases. They also watch VolumeAttachments (from 1.32) and DRA objects (DRA is on by default from 1.35).
- Without read access to those, the autoscaler never finishes syncing its caches and never scales (Failed to watch *v1.VolumeAttachment kubernetes/autoscaler#7663, ClusterRole doesn't have volumeattachments resource permissions to the Cluster Autoscaler kubernetes/autoscaler#8678). The AWS examples were fixed the same way.
- The script appends a cluster-autoscaler-extra ClusterRole and binding with that read access. At 1.36.2 that makes the CloudStack example's permissions match the AWS example's.
- It only adds read permissions, so it is harmless on older versions.

CCM (ISO builder)

The default is v1.2.0, the current release and the first one with the node-initialisation fix.
On CKS that fix has no visible effect yet. CKS starts kubelets without --cloud-provider=external, so the CCM never initialises nodes: providerID stays unset and nodes never get the uninitialized taint, whatever the CCM version. The CCM's load balancer support works either way. Changing the kubelet flag is out of scope for this PR.
CCM_VERSION=v1.3.0-rc1 picks another release tag.
The manifest comes from that tag, and its image is set to apache/cloudstack-kubernetes-provider:<tag>. This is needed because a tag's manifest doesn't always name its own image: v1.3.0-rc1's names v1.2.0.

versions.txt in the ISO

It lists the Kubernetes, CNI plugins, crictl, CCM, CSI driver, autoscaler and etcd versions.
It also lists the URL of every file downloaded, and every bundled image with its digest.
It is not a .yaml file, so the node scripts that copy *.yaml from the ISO ignore it.

Download and build robustness (ISO builder)

Downloads use curl -fsSL --retry 3, one file per call, so a failed download stops the build. A multi-URL curl would only report the last file's status.
The curl | sed pipelines for kubelet.service and 10-kubeadm.conf now download first and then edit, so a failed download is no longer hidden by the pipe.
The CNI fallback to the old archive name is kept.
The CSI and etcd downloads move from wget to the same helper.
The ISO is assembled in a mktemp -d directory, removed on exit, instead of a fixed /tmp/iso. Files left behind by a failed build no longer end up in the next ISO.
The image list is deduplicated before pulling. The CSI manifest names two images twice.
Pulled images are removed with ctr image rm --sync.
- Without --sync, containerd's asynchronous garbage collection could delete a layer that the next image shares, between that image's pull and its export.
- That failed my test builds with ctr: failed to get reader: content digest ...: not found.
The positional arguments are unchanged. Both overrides are environment variables.

Upgrade (KubernetesClusterUpgradeWorker)

upgrade-kubernetes.sh already copies the new ISO's autoscaler.yaml to the control node, but the running autoscaler was never touched. With :latest everywhere, that didn't matter.
Now the autoscaler's image changes with the Kubernetes version. After all nodes are upgraded and the new version is saved, a cluster with autoscaling enabled gets its autoscaler redeployed from the new ISO.
- It runs autoscale-kube-cluster -e with the cluster's id, min and max from the database, the same command as enabling autoscaling.
- It runs while the cluster is still Upgrading, so autoscaling can't be changed through the API in the meantime.
- A failure is logged as a warning and leaves the previous autoscaler running. It doesn't fail the upgrade.
The command string is now built by one helper that both callers share.

Not changed here:

Node-side fallbacks: autoscale-kube-cluster and deploy-provider still download from master/main when the ISO doesn't have the manifests. CKS: fix the add-on scripts' download and option handling #14598 changes those same lines, so pinning them is left until it is merged.
Kubelet unit files: kubelet.service and 10-kubeadm.conf still come from shapeblue/cloudstack-nonoss main. Their URLs are now recorded in versions.txt.
aarch64 ISOs: images are still pulled for the build host's platform, and etcd is always the amd64 archive.
CCM on upgrade: it is not redeployed during an upgrade, as before.
Upgrading onto an old ISO: an upgrade to a version whose ISO was built by the old script redeploys that ISO's :latest autoscaler. That is the same autoscaler a cluster created from that ISO would get.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
Build/CI
Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Bug Severity

How Has This Been Tested?

ISO builds. I built ISOs with the modified script on an Ubuntu x86_64 host with containerd 2.x. I then extracted versions.txt, autoscaler.yaml, provider.yaml and the image tarballs from each ISO with isoinfo.

Build	Autoscaler (manifest tag / image)	CCM (manifest tag / image)
1.31.14, defaults	1.31.5 / `registry.k8s.io/autoscaling/cluster-autoscaler:v1.31.5`	v1.2.0 / `apache/cloudstack-kubernetes-provider:v1.2.0`
1.33.13, defaults, with etcd 3.5.21	1.33.6 / `...cluster-autoscaler:v1.33.6`	v1.2.0 / `...:v1.2.0`
1.33.13, `AUTOSCALER_VERSION=v1.33.4 CCM_VERSION=1.3.0-rc1`	1.33.4 / `...cluster-autoscaler:v1.33.4`	v1.3.0-rc1 / `...:v1.3.0-rc1` (the tag's manifest named v1.2.0)

The 1.31 build was done with Calico 3.29.3 and Dashboard 2.7.0. Later builds used flannel and Headlamp manifests, because Docker Hub's anonymous pull limit stopped the Calico/Dashboard builds partway.

For each ISO I checked the following:

autoscaler.yaml and provider.yaml name exactly those images, with imagePullPolicy: IfNotPresent.
The <min>:<max>:<cluster-id> placeholders are intact.
The autoscaler manifest parses into the six original objects plus the extra ClusterRole and binding.
docker/ holds cluster-autoscaler:v<version>.tar and cloudstack-kubernetes-provider:<tag>.tar, and the image name inside each tarball matches the manifest.
versions.txt has the right versions and URLs.
- There is one line per image, each with a 64-hex digest.
- The etcd version and URL appear only when etcd was requested.
The number of image tarballs equals the number of image lines.
No build directory is left behind.

Failure cases. Each of these stops the build with curl: (22) ... 404 and a non-zero exit:

CCM_VERSION=v9.9.9
AUTOSCALER_VERSION=1.33.99
a dashboard URL that doesn't exist

Builder pieces tested on their own:

A minor version with no autoscaler release (1.99) stops with the error asking for AUTOSCALER_VERSION.
AUTOSCALER_VERSION=1.32.7 on 1.33 prints the mismatch warning.
CNI v0.7.5 falls back to the old cni-plugins-amd64-<ver>.tgz name.

Unit tests. New tests in KubernetesClusterUpgradeWorkerTest cover updateAutoscaler:

it does nothing when autoscaling is disabled or min/max is unset;
it runs autoscale-kube-cluster -i <uuid> -e -M <max> -m <min>;
it doesn't throw when SSH fails or returns an error.

All 82 tests in the kubernetes-service module pass, and checkstyle is clean.

Not tested:

Deploying these ISOs on a live CKS cluster.
In particular, running the upstream autoscaler image with the CloudStack provider against a real cluster, and a real upgrade that redeploys it.
The RBAC addition is based on the autoscaler source and the upstream issues above.

How did you try to break this feature and the system with this change?

Existing ISOs and clusters: unaffected at create time. The node side reads the same file names (autoscaler.yaml, provider.yaml, docker/*.tar).
Existing build commands: the positional arguments are unchanged. Without the new variables, builds get version-matched defaults instead of master/main.
A minor version before its autoscaler release: the build fails with a clear message instead of bundling a mismatched autoscaler. AUTOSCALER_VERSION overrides it.
A change to an upstream manifest's layout: setting the image checks that the manifest has exactly one image: line and fails otherwise, instead of silently bundling the wrong image.
A manifest without a trailing newline: the appended RBAC starts with a blank line, so --- can't join the last line.
Upgrade with autoscaling disabled: nothing changes.
Upgrade where the redeploy fails: a warning is logged and the upgrade still succeeds with the old autoscaler running.
On an air-gapped cluster: the redeploy runs only after every node in the upgrade has imported the new ISO's images. Nodes marked for manual upgrade are the exception, but the rolling update keeps the old pod until the new one is ready.

create-kubernetes-binaries-iso.sh downloaded the cluster-autoscaler manifest from kubernetes/autoscaler master and the CCM manifest from cloudstack-kubernetes-provider main, and bundled whatever images those named. The autoscaler image was apache/cloudstack-kubernetes-autoscaler :latest, a 2022 development build, for every Kubernetes version, and the CCM was whatever main held on the day of the build (v1.1.0 for a long time, which never initialises nodes). Nothing recorded what went in. - Autoscaler: use the newest cluster-autoscaler release for the ISO's Kubernetes minor version (or AUTOSCALER_VERSION), take the CloudStack example manifest from that release tag, and bundle the upstream registry.k8s.io image of the same release, which includes the CloudStack provider. Add a ClusterRole for the VolumeAttachment and DRA read access that newer releases need and the example lacks. - CCM: pin to v1.2.0 (or CCM_VERSION) and set the manifest's image to that tag. - Write versions.txt into the ISO with every version, URL and image digest. - Download with curl -f, one file per call, so a failed download stops the build instead of bundling an error page. Build in a fresh mktemp directory, and remove pulled images with --sync so containerd's garbage collection can't delete layers the next pull shares. - Upgrade: the autoscaler image now changes with the Kubernetes version, so once every node is upgraded, redeploy the autoscaler of a cluster with autoscaling enabled from the new ISO, with the cluster's min, max and id. A failure is logged and doesn't fail the upgrade. Signed-off-by: Brad House <[email protected]>

codecov · 2026-11-09T21:29:15Z

Codecov Report

❌ Patch coverage is 85.71429% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 18.32%. Comparing base (d02adcd) to head (5a49669).

Files with missing lines	Patch %	Lines
.../actionworkers/KubernetesClusterUpgradeWorker.java	87.50%	1 Missing and 1 partial ⚠️
...KubernetesClusterResourceModifierActionWorker.java	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff            @@
##               4.22   #14649   +/-   ##
=========================================
  Coverage     18.32%   18.32%           
- Complexity    16644    16646    +2     
=========================================
  Files          5939     5939           
  Lines        537768   537787   +19     
  Branches      65929    65931    +2     
=========================================
+ Hits          98551    98560    +9     
- Misses       428008   428018   +10     
  Partials      11209    11209

Flag	Coverage Δ
uitests	`4.08% <ø> (ø)`
unittests	`19.41% <85.71%> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

bhouse-nexthop · 2026-11-10T01:42:20Z

Test report: pinned autoscaler and CCM in the binaries ISO, on a live cluster

Summary: I built two ISOs with this PR's create-kubernetes-binaries-iso.sh and registered both. A cluster created from the first runs the autoscaler release that matches its Kubernetes minor version, and the CCM at v1.2.0. The autoscaler syncs with no RBAC or cache errors. It scaled the cluster up from 1 to 3 workers for Pending pods, and back down to 1 once they were gone. Upgrading the cluster to the second ISO redeployed the autoscaler with the newer image, keeping the cluster's min, max and id.

Environment

A CloudStack 24.0 (main) build with this PR, alongside some of my other open PRs. The management server and all three agents were upgraded in place to this build.
The PR targets 4.22; it wasn't run on a 4.22 build. The ISO script and KubernetesClusterUpgradeWorker are the same on both branches apart from this PR.
Three KVM hosts, Ceph RBD primary storage, one management server.
ISOs built on an Ubuntu 24.04 x86_64 host with containerd's ctr. The arguments were the same as for the lab's existing ISOs: CNI 1.9.1, Calico v3.30.7 and Headlamp 0.40.1.
A fresh cluster with 1 control node and 1-3 workers, the CKS system VM template and an isolated network. Autoscaling ran with min 1 and max 3.

Results

Check	Result
Build `setup-v1.32.13` and `setup-v1.33.13` with defaults	pass, 139 s and 113 s
`versions.txt` in each ISO	pass. It lists the versions, URLs and image digests
Autoscaler and CCM pinned in the ISOs	pass: 1.32.13 -> `cluster-autoscaler:v1.32.7` and 1.33.13 -> `v1.33.6`; CCM `v1.2.0` in both
Cluster created from the 1.32.13 ISO, CCM	pass. `apache/cloudstack-kubernetes-provider:v1.2.0` runs, and a `LoadBalancer` Service gets a CloudStack LB rule
Enable autoscaling (1..3)	pass. Pinned image, `--nodes=1:3:<cluster id>`, no RBAC or cache-sync errors
Scale-up on Pending pods	pass, 1 -> 3 workers; all pods Running 232 s after they were created
Scale-down when idle	pass, 3 -> 1 workers 654 s after the pods were deleted (default idle times)
Upgrade 1.32.13 -> 1.33.13 (the second ISO)	pass, 323 s. The autoscaler was redeployed with `v1.33.6`, keeping `--nodes=1:3:<cluster id>`
Node `providerID` and the `uninitialized` taint	not applicable to CKS; see the CCM section

ISOs

Each ISO has versions.txt at the root, next to autoscaler.yaml and provider.yaml. The 1.33.13 one:

AUTOSCALER_VERSION=1.33.6
AUTOSCALER_MANIFEST_URL=https://raw.githubusercontent.com/kubernetes/autoscaler/cluster-autoscaler-1.33.6/cluster-autoscaler/cloudprovider/cloudstack/examples/cluster-autoscaler-standard.yaml
AUTOSCALER_IMAGE=registry.k8s.io/autoscaling/cluster-autoscaler:v1.33.6
CCM_VERSION=v1.2.0
CCM_MANIFEST_URL=https://raw.githubusercontent.com/apache/cloudstack-kubernetes-provider/v1.2.0/deployment.yaml
CCM_IMAGE=apache/cloudstack-kubernetes-provider:v1.2.0
...
IMAGE=registry.k8s.io/autoscaling/cluster-autoscaler:v1.33.6@sha256:7b44e0f0...
IMAGE=docker.io/apache/cloudstack-kubernetes-provider:v1.2.0@sha256:09d3042f...

It also has the Kubernetes, CNI, crictl, Calico, Headlamp, CSI and kubelet unit URLs, and one IMAGE= line with a digest for every bundled image.

docker/ has cluster-autoscaler_v1.32.7.tar and cluster-autoscaler_v1.33.6.tar respectively, and cloudstack-kubernetes-provider_v1.2.0.tar in both.
autoscaler.yaml names only the pinned image, with imagePullPolicy: IfNotPresent. It keeps --nodes=<min>:<max>:<cluster-id>, and has the six objects of the upstream example plus the cluster-autoscaler-extra ClusterRole and binding: read access to volumeattachments, deviceclasses, resourceslices and resourceclaims.
provider.yaml names apache/cloudstack-kubernetes-provider:v1.2.0.

CCM

On the new cluster, the CCM deployment runs apache/cloudstack-kubernetes-provider:v1.2.0. The existing clusters, built from ISOs made with the old script, run v1.1.0.
A LoadBalancer Service. The CCM acquired a public IP and created a CloudStack LB rule (<svc-uid>-tcp-8080, 8080 -> NodePort, Active). The Service got that IP as its EXTERNAL-IP within 12 s. Deleting the Service removed the rule.
providerID stays empty, on every node, as on the clusters running v1.1.0.
- CKS starts the kubelets without --cloud-provider=external (KUBELET_EXTRA_ARGS=--cgroup-driver=systemd). So no node gets the node.cloudprovider.kubernetes.io/uninitialized taint, and nothing is left behind.
- The CCM's node controller only initialises nodes that have that taint, whatever its version. So the providerID fix in v1.2.0 doesn't come into play on a CKS cluster as it is set up today.
- This PR doesn't change how the kubelet is started, so I don't count it against the PR. The PR description says v1.1.0 "does not initialise nodes", which suggests v1.2.0 would initialise them on CKS. It doesn't on its own.
I didn't see any effect on the autoscaler. It found the cluster's VMs, scaled up and down, and only logged Failed to get nodegroup for <vm id>: Unable to find node ... warnings for the nodes, as it does without a providerID.

Autoscaler

Enabling autoscaling with scaleKubernetesCluster autoscalingenabled=true minsize=1 maxsize=3 deployed it from the ISO's manifest:

registry.k8s.io/autoscaling/cluster-autoscaler:v1.32.7
["./cluster-autoscaler","--v=4","--stderrthreshold=info","--cloud-provider=cloudstack","--cloud-config=/config/cloud-config",
 "--skip-nodes-with-local-storage=false","--nodes=1:3:<cluster id>"]

Log checks over the whole run (2,628 lines by the end of the scale-down):
- no Failed to watch or Failed to list lines, no forbidden errors, and no cache-sync timeouts. The only forbidden matches were the lastScaleDownForbidden... field in the status lines;
- one E line, Failed to get node infos for groups: Unable to find node .... It came 45 s after a scaled-down node's VM was removed, and the autoscaler carried on normally.
Scale-up:
- I created a 3-replica Deployment requesting 1200m CPU each. One pod fit on the single worker and two were Pending.
- The autoscaler logged Final scale-up plan: [{<cluster id> 1->3 (max: 3)}] and Scale-up: setting group <cluster id> size to 3. That started a KUBERNETES.CLUSTER.SCALE job in CloudStack.
- Two workers were added. The cluster went Running -> Scaling -> Running with size 3, and all 3 pods were Running on 3 workers 232 s after the Deployment was created.
Scale-down:
- I deleted the Deployment.
- About 10 minutes later the autoscaler logged Scale-down: removing empty node for both new workers. The cluster went Scaling (3) -> Scaling (2) -> Running with size 1, 654 s after the deletion.
- The upstream example doesn't set --scale-down-unneeded-time or --scale-down-delay-after-add, so the defaults applied. I didn't edit the manifest.

Upgrade redeploys the autoscaler

upgradeKubernetesCluster from the 1.32.13 ISO to the 1.33.13 ISO took 323 s, and ended Running with both nodes on v1.33.13.

Afterwards the autoscaler deployment had a new pod:

before: registry.k8s.io/autoscaling/cluster-autoscaler:v1.32.7  --nodes=1:3:<cluster id>  (started 00:27:02)
after:  registry.k8s.io/autoscaling/cluster-autoscaler:v1.33.6  --nodes=1:3:<cluster id>  (started 00:52:25)

The new pod's log had no forbidden, watch or list, or cache-sync errors.
The first attempt on the control node failed with kubectl apply of the network manifest:
```
... is forbidden: User "kubernetes-admin" cannot get resource "customresourcedefinitions" ...
RBAC: [clusterrole.rbac.authorization.k8s.io "cluster-admin" not found, ...]
```
This was right after kubeadm upgrade apply restarted the API server, before its RBAC cache had synced. The upgrade worker's existing retry succeeded on the next attempt. It's not related to the autoscaler change.

Cleanup: the cluster and both Kubernetes versions were deleted. Neither left a VM, network or ISO behind.

Result: pass. The ISO records what it holds and bundles the autoscaler that matches its Kubernetes version and CCM v1.2.0. The autoscaler they deploy scales a CKS cluster up and down, and an upgrade moves it to the new version's autoscaler.

Not covered

Overrides (AUTOSCALER_VERSION, CCM_VERSION), aarch64, and building with etcd. Only default builds were run on the live cluster.
Kubernetes 1.35 and later, where the DRA read access in the extra ClusterRole matters most. The CKS template here was used with 1.32 and 1.33.
Upgrading a cluster onto an ISO built by the old script.

boring-cyborg Bot added the component:kubernetes label Nov 9, 2026

DaanHoogland linked an issue Nov 11, 2026 that may be closed by this pull request

CKS: the binaries ISO builder bundles an unpinned autoscaler and CCM, and records nothing about what it contains #14648

Open

DaanHoogland added this to the 4.22.3 milestone Nov 11, 2026

Navigation Menu

Repository navigation

CKS: pin the autoscaler and CCM in the binaries ISO builder - #14649

CKS: pin the autoscaler and CCM in the binaries ISO builder#14649
bhouse-nexthop wants to merge 1 commit into
apache:4.22from
bhouse-nexthop:cks-iso-pin-addons

bhouse-nexthop commented Nov 9, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Nov 9, 2026 •

edited

Loading

Uh oh!

bhouse-nexthop commented Nov 10, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

bhouse-nexthop commented Nov 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

What was wrong

What changed

Types of changes

Feature/Enhancement Scale or Bug Severity

Bug Severity

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

codecov Bot commented Nov 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

bhouse-nexthop commented Nov 10, 2026

Test report: pinned autoscaler and CCM in the binaries ISO, on a live cluster

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

bhouse-nexthop commented Nov 9, 2026 •

edited

Loading

codecov Bot commented Nov 9, 2026 •

edited

Loading