Repository navigation
ci: verify one DevWorkspace e2e test scenarios based on minikube on github actions #1742
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
e34e0e9
c8fc667
5ed2d08
7634b78
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,204 @@ | ||
| # | ||
| # Copyright (c) 2019-2026 Red Hat, Inc. | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); | ||
| # you may not use this file except in compliance with the License. | ||
| # You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
| # | ||
|
|
||
| name: Install pull request controller on minikube | ||
| description: >- | ||
| Start a small minikube cluster, install this checkout's controller image, | ||
| and clone the DevWorkspace operator load-tests repository so a later step | ||
| can create one DevWorkspace. The controller and webhook deployments are | ||
| pinned to imagePullPolicy Never so the cluster runs the image built from | ||
| the pull request instead of quay.io/devfile/devworkspace-controller:next. | ||
| inputs: | ||
| image-tarball: | ||
| description: gzip-compressed docker save archive of the controller image | ||
| required: true | ||
| image-ref: | ||
| description: Image reference stored in the archive and installed as DWO_IMG | ||
| required: false | ||
| default: docker.io/library/devworkspace-controller:ci | ||
| kubernetes-version: | ||
| description: Kubernetes version started by minikube | ||
| required: false | ||
| default: v1.33.0 | ||
| dwo-namespace: | ||
| description: Namespace the operator is installed into | ||
| required: false | ||
| default: devworkspace-controller | ||
| load-test-repo: | ||
| description: Load-test repository cloned after the operator is ready | ||
| required: false | ||
| default: https://github.com/rohankanojia-testing/devworkspace-operator-load-tests.git | ||
| load-test-ref: | ||
| description: Git ref cloned from the load-test repository | ||
| required: false | ||
| default: main | ||
|
|
||
| outputs: | ||
| load-test-dir: | ||
| description: Absolute path of the cloned load-test repository | ||
| value: ${{ steps.clone-load-tests.outputs.dir }} | ||
|
|
||
| runs: | ||
| using: composite | ||
| steps: | ||
| - name: Setup Minikube-Kubernetes | ||
| uses: manusa/actions-setup-minikube@001b9f30165b2a875b835ec5183ced75b94bc269 # v2.19.0 | ||
| with: | ||
| minikube version: v1.39.0 | ||
| kubernetes version: ${{ inputs.kubernetes-version }} | ||
| github token: ${{ github.token }} | ||
|
|
||
| - name: Enable metrics-server | ||
| shell: bash | ||
| run: | | ||
| minikube addons enable metrics-server | ||
| kubectl rollout status deployment/metrics-server -n kube-system --timeout=300s | ||
| - name: Install cert-manager | ||
| shell: bash | ||
| run: | | ||
| kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.18.2/cert-manager.yaml | ||
| kubectl rollout status deployment/cert-manager -n cert-manager --timeout=300s | ||
| kubectl rollout status deployment/cert-manager-webhook -n cert-manager --timeout=300s | ||
| kubectl rollout status deployment/cert-manager-cainjector -n cert-manager --timeout=300s | ||
| for _ in $(seq 1 30); do | ||
| if kubectl api-versions | grep -q '^cert-manager.io/v1$'; then | ||
| exit 0 | ||
| fi | ||
| sleep 5 | ||
| done | ||
| echo "cert-manager.io/v1 did not become available" | ||
| exit 1 | ||
| - name: Enable ingress | ||
| shell: bash | ||
| run: | | ||
| minikube addons enable ingress | ||
| kubectl rollout status deployment/ingress-nginx-controller -n ingress-nginx --timeout=300s | ||
| - name: Install deployment tools and k6 | ||
| shell: bash | ||
| run: | | ||
| sudo apt-get update | ||
| sudo apt-get install -y jq gettext-base | ||
| pip install yq || pip install --break-system-packages yq || true | ||
| # Ubuntu 26 apt ignores the keybox written by gpg --recv-keys. Dearmor | ||
| # the published key into the binary keyring format apt accepts. | ||
| curl -fsSL https://dl.k6.io/key.gpg | gpg --dearmor | sudo tee /usr/share/keyrings/k6-archive-keyring.gpg >/dev/null | ||
| sudo chmod 644 /usr/share/keyrings/k6-archive-keyring.gpg | ||
| echo "deb [signed-by=/usr/share/keyrings/k6-archive-keyring.gpg] https://dl.k6.io/deb stable main" | sudo tee /etc/apt/sources.list.d/k6.list | ||
| sudo apt-get update | ||
| sudo apt-get install -y k6 | ||
| k6 version | ||
| - name: Load the pull request image into minikube | ||
| shell: bash | ||
| env: | ||
| IMAGE_TARBALL: ${{ inputs.image-tarball }} | ||
| IMAGE_REF: ${{ inputs.image-ref }} | ||
| run: | | ||
| set -euo pipefail | ||
| archive="${RUNNER_TEMP}/devworkspace-controller-ci.tar" | ||
| gzip -dc "${IMAGE_TARBALL}" > "${archive}" | ||
| minikube image load "${archive}" | ||
| rm -f "${archive}" | ||
| echo "Images visible to minikube:" | ||
| minikube image ls | ||
| if ! minikube image ls | grep -q 'devworkspace-controller:ci'; then | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Load check greps for a hardcoded devworkspace-controller:ci instead of the configurable image-ref input. If a caller overrides image-ref, the step exits 1 even though the image loaded fine, and the error prints ${IMAGE_REF} which never matches what was searched. Only the default tag works. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Verify the configured image reference, not the hard-coded tag. If a caller sets
|
||
| echo "minikube does not have ${IMAGE_REF}" | ||
| exit 1 | ||
| fi | ||
| - name: Install DevWorkspace Operator from this checkout | ||
| shell: bash | ||
| env: | ||
| NAMESPACE: ${{ inputs.dwo-namespace }} | ||
| DWO_IMG: ${{ inputs.image-ref }} | ||
| run: | | ||
| set -euo pipefail | ||
| echo "Installing ${DWO_IMG} into namespace ${NAMESPACE}" | ||
| kubectl config current-context | ||
| make install | ||
| # The generated deployment and the webhook Deployment created at | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. CreateWebhookServerDeployment runs on every controller startup and, on AlreadyExists, issues client.Update with getSpecDeployment() which hardcodes ImagePullPolicy: PullAlways. If the controller-manager pod restarts after the webhook was pinned to Never, the webhook Deployment reverts to Always → ImagePullBackOff (CI image isn't in any registry) → breaks verify-webhook/verify-backup. This is the one most likely to cause flaky CI. |
||
| # controller startup both set imagePullPolicy Always. Always ignores | ||
| # the image loaded into minikube, so pin Never after install. The | ||
| # controller writes the webhook Deployment once during startup and | ||
| # does not reconcile it afterwards. | ||
| wait_and_pin_image() { | ||
| local deploy="$1" | ||
| local container="$2" | ||
| local deadline=$((SECONDS + 360)) | ||
| local idx policy image | ||
| while (( SECONDS < deadline )); do | ||
| if ! kubectl -n "${NAMESPACE}" get deployment "${deploy}" >/dev/null 2>&1; then | ||
| echo "waiting for deployment/${deploy}" | ||
| sleep 5 | ||
| continue | ||
| fi | ||
| idx="$(kubectl -n "${NAMESPACE}" get deployment "${deploy}" -o json | jq -r --arg name "${container}" ' | ||
| .spec.template.spec.containers | ||
| | to_entries[] | ||
| | select(.value.name == $name) | ||
| | .key')" | ||
| if [[ -z "${idx}" || "${idx}" == "null" ]]; then | ||
| echo "container ${container} not found on deployment/${deploy}" | ||
| kubectl -n "${NAMESPACE}" get deployment "${deploy}" -o yaml | ||
| return 1 | ||
| fi | ||
| kubectl -n "${NAMESPACE}" patch deployment "${deploy}" --type=json \ | ||
| -p="[{\"op\":\"replace\",\"path\":\"/spec/template/spec/containers/${idx}/imagePullPolicy\",\"value\":\"Never\"}]" | ||
| sleep 2 | ||
| policy="$(kubectl -n "${NAMESPACE}" get deployment "${deploy}" -o jsonpath="{.spec.template.spec.containers[${idx}].imagePullPolicy}")" | ||
| image="$(kubectl -n "${NAMESPACE}" get deployment "${deploy}" -o jsonpath="{.spec.template.spec.containers[${idx}].image}")" | ||
| echo "${deploy}: image=${image} imagePullPolicy=${policy}" | ||
| if [[ "${policy}" == "Never" && "${image}" == "${DWO_IMG}" ]]; then | ||
| if kubectl -n "${NAMESPACE}" rollout status "deployment/${deploy}" --timeout=180s; then | ||
| policy="$(kubectl -n "${NAMESPACE}" get deployment "${deploy}" -o jsonpath="{.spec.template.spec.containers[${idx}].imagePullPolicy}")" | ||
| image="$(kubectl -n "${NAMESPACE}" get deployment "${deploy}" -o jsonpath="{.spec.template.spec.containers[${idx}].image}")" | ||
| if [[ "${policy}" == "Never" && "${image}" == "${DWO_IMG}" ]]; then | ||
| return 0 | ||
| fi | ||
| fi | ||
| fi | ||
| sleep 5 | ||
| done | ||
| echo "timed out pinning deployment/${deploy} to ${DWO_IMG}" | ||
| kubectl -n "${NAMESPACE}" get deploy,pods -o wide || true | ||
| kubectl -n "${NAMESPACE}" logs -l app.kubernetes.io/name=devworkspace-controller --tail=100 || true | ||
| return 1 | ||
| } | ||
| wait_and_pin_image devworkspace-controller-manager devworkspace-controller | ||
| wait_and_pin_image devworkspace-webhook-server webhook-server | ||
| kubectl get pods -n "${NAMESPACE}" -o wide | ||
| - name: Clone load tests | ||
| id: clone-load-tests | ||
| shell: bash | ||
| env: | ||
| LOAD_TEST_REPO: ${{ inputs.load-test-repo }} | ||
| LOAD_TEST_REF: ${{ inputs.load-test-ref }} | ||
| run: | | ||
| set -euo pipefail | ||
| dest="${RUNNER_TEMP}/devworkspace-operator-load-tests" | ||
| git clone --depth 1 --branch "${LOAD_TEST_REF}" "${LOAD_TEST_REPO}" "${dest}" | ||
| git -C "${dest}" log -1 --oneline | ||
| echo "dir=${dest}" >> "${GITHUB_OUTPUT}" | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,44 @@ | ||
| #!/bin/bash | ||
| # | ||
| # Copyright (c) 2019-2026 Red Hat, Inc. | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); | ||
| # you may not use this file except in compliance with the License. | ||
| # You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
| # | ||
|
|
||
| # Dump the objects that explain a failed minikube verification. Invoked with | ||
| # `if: failure()`, so a missing cluster must not fail the step. | ||
| set +e | ||
|
|
||
| namespace="${DWO_NAMESPACE:-devworkspace-controller}" | ||
|
|
||
| echo "===== pods =====" | ||
| kubectl get pods -A -o wide | ||
|
|
||
| echo "===== controller logs =====" | ||
| kubectl logs -n "${namespace}" -l app.kubernetes.io/name=devworkspace-controller --tail=200 | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Give each diagnostics probe a timeout. If the API server lists pods but stops responding to the controller log request, the script waits without reaching the webhook logs, workspace snapshots, or warning events.
|
||
|
|
||
| echo "===== webhook logs =====" | ||
| kubectl logs -n "${namespace}" -l app.kubernetes.io/name=devworkspace-webhook-server --tail=200 | ||
|
|
||
| echo "===== devworkspaces =====" | ||
| kubectl get devworkspaces -A -o yaml | ||
|
|
||
| echo "===== devworkspaceoperatorconfigs =====" | ||
| kubectl get devworkspaceoperatorconfigs -A -o yaml | ||
|
|
||
| echo "===== backup jobs =====" | ||
| kubectl get jobs -A -l controller.devfile.io/backup-job=true -o yaml | ||
|
|
||
| echo "===== warning events =====" | ||
| kubectl get events -A --field-selector type=Warning | ||
|
|
||
| exit 0 | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Minor/robustness: The trailing || true swallows the case where both pip install attempts fail. Since yq is required downstream by make install → generate_deployment.sh, a total install failure here won't actually be tolerated, it'll resurface ~40 lines later as a cryptic yq-not-found error during deployment generation, which is harder to trace back to this step.
Suggest dropping || true and verifying the install so it fails fast with a clear message, while keeping the real fallback chain (plain → --break-system-packages):
pip install yq || pip install --break-system-packages yq
yq --version
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
|| trueis intentional. This job does not needyqfor the install that follows.make installrunsgenerate_deployment.shwith no arguments. That script listsyqinrequired_binand printsERROR: Program yq is required when it is missing, then continues. It only exits becauseyqis missing when--split-yamlis set, and make install does not pass that flag. The otheryqcall is behind--generate-olm, which this job does not use.