Skip to content

HIVE-29591: Create Hive precommit image and move Dockerfile into the hive repository - #6953

Merged
abstractdog merged 8 commits into
apache:masterfrom
abstractdog:HIVE-29591-precommit-image
Nov 16, 2026
Merged

abstractdog merged 8 commits into
apache:masterfrom
abstractdog:HIVE-29591-precommit-image

Conversation

@abstractdog

@abstractdog abstractdog commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

  • A precommit image in dev-support/docker/ci/runner. It is based on Debian trixie-slim and contains:

    • a Temurin 21 JDK, without jmods and man pages
    • Maven
    • the tools the precommit stages and tests use: the docker CLI, git, openssl, python3, rsync, tini and xmlstarlet
  • A GitHub Actions workflow, precommit-image.yml. On PRs that change the Dockerfile, it builds the image. On master it publishes the image for amd64 and arm64 as apache/hive-ci-runner:<version>, where
    <version> is the org.opencontainers.image.version label of the Dockerfile. A published the workflow fails if the tag already exists. A manual run can publish a candidate under atag of its own.

  • The Jenkinsfile runs in apache/hive-ci-runner:1 instead of `ayushtkn/hive-dev-box:e

    • The tag is pinned; a new image is rolled out by a follow-up commit that bumps it.
    • The dev-box-specific lines are gone (sw java, /etc/profile.d/*.sh, `git config extr
    • The Maven options are now passed through MAVEN_ARGS (Maven 3.9+), in place of the dev-box mvn wrapper.
    • A new IMAGE build parameter lets a run use a candidate image.
  • The init@derby and init@postgres stages are removed. DbInstallBase already covers schematool on more databases, and InformationSchemaWithPrivilegeTestBase covers the SYS/INFORMATION_SCHEMA
    initialization. This also keeps Hadoop out of the image.

  • cachingprintstream.q and stack_trace.q normalize the exec failure message with qt:replace:

    • Recent JDK updates report a failed exec as `Exec failed, error: 2 (No such file or dire
    • Older ones report error=2, No such file or directory.

    This line is the only difference in the output, so the tests still check the same failure.

  • A README on how to change, try and roll out the image.

  • The docker and helm action pins in docker-images.yml and hive-postgres-tpcds-metastore.yml are bumped to the versions precommit-image.yml uses. Some of the old pins were close to their expiry dates
    in the ASF allowlist.

Why are the changes needed?

The precommit image used to be built from a separate repository (hive-dev-box) and pushed to a personal Docker Hub account. Keeping the Dockerfile in the Hive repository and publishing it from CI makes the
image reviewable and reproducible, and removes the dependency on a personal account. See HI

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Locally, in the image with a docker:dind sidecar:

  • mvn install -DskipTests -Pitests,dist succeeds.
  • These qtests pass:
    • newline.q (python TRANSFORM) on TestMiniLlapCliDriver
    • jdbc_project_pushdown.q (postgres in docker) on TestMiniLlapLocalCliDriver
    • stack_trace.q on TestNegativeLlapCliDriver
    • cachingprintstream.q on TestNegativeLlapLocalCliDriver
  • The kudu-handler tests pass on amd64 (18 tests). They are skipped on aarch64, as before.

A full precommit run with the candidate image (IMAGE=abstractdog/hive-precommit:HIVE-29591-v3):
https://ci.hive.apache.org/job/hive-precommit/job/PR-6953/12/

@abstractdog abstractdog changed the title HIVE-29591: Create Hive precommit image and move Dockerfile into the … HIVE-29591: Create Hive precommit image and move Dockerfile into the hive repository Nov 9, 2026
@abstractdog
abstractdog force-pushed the HIVE-29591-precommit-image branch from 46def98 to 101acd8 Compare November 9, 2026 13:59
@abstractdog
abstractdog force-pushed the HIVE-29591-precommit-image branch from 101acd8 to 84fa9dc Compare November 10, 2026 10:02
@abstractdog
abstractdog force-pushed the HIVE-29591-precommit-image branch from 84fa9dc to 698ecdd Compare November 11, 2026 11:11
@abstractdog
abstractdog requested a review from zabetak November 11, 2026 11:21

@zabetak zabetak left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, pending tests!

Comment thread .github/workflows/precommit-image.yml Outdated
Comment on lines +64 to +67
if docker manifest inspect "$image" > /dev/null 2>&1; then
echo "::error::$image is already published, and a published tag is never overwritten. Bump org.opencontainers.image.version in the Dockerfile (or pick another tag for a manual run)."
exit 1
fi

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can disable tag overwrite directly in DockerHub if we want to.

abstractdog and others added 7 commits November 13, 2026 07:22
- Move the image to dev-support/docker/ci/runner.
- Drop the init@<db> stages and precommit-init-metastore.sh (DbInstallBase already covers
  schematool), and with them Hadoop from the image.
- Drop smoke-test.sh.
- Version the image with its org.opencontainers.image.version label: publish
  apache/hive-ci-runner:<version>, never overwrite a published tag, and pin the tag in the
  Jenkinsfile.
- Trim the README to the steps people actually follow.

Co-Authored-By: Claude Code <[email protected]>
docker 20.10 cannot extract the current mcr.microsoft.com/mssql/server:2019-latest
layers ("archive/tar: invalid tar header", microsoft/mssql-docker#978), which fails
TestSchemaToolForMetastore[mssql].
A push to a fork's master only syncs apache/hive, and the tag check fails there once
apache/hive has published the version. Forks still build the image on their PRs and
manual runs.

Co-Authored-By: Claude Code <[email protected]>
… does not need a new version

When the tag is published already, the workflow fails only if the Dockerfile changed (against
the target branch of a PR, or the previous head of a push); otherwise it builds the image but
skips publishing it. A manual run with an existing tag still fails.
Comment thread .github/workflows/precommit-image.yml Resolved
…t, which the Jenkinsfile uses

A change of the Dockerfile no longer needs a version bump, nor a follow-up pull request that
bumps the Jenkinsfile: on master the workflow publishes apache/hive-ci-runner:<UTC timestamp>-<commit>
and moves apache/hive-ci-runner:latest to it. A manual run on another branch publishes a
candidate without moving latest. Only the newest 5 <UTC timestamp>-<commit> tags are kept.

Co-Authored-By: Claude Code <[email protected]>
@sonarqubecloud

Copy link
Copy Markdown

@abstractdog
abstractdog merged commit 7d315d7 into apache:master Nov 16, 2026
8 of 9 checks passed
@abstractdog

Copy link
Copy Markdown
Contributor Author

merged, waiting for action to build and push initial latest image: https://github.com/apache/hive/actions/runs/43885083693

@abstractdog

Copy link
Copy Markdown
Contributor Author

merged, waiting for action to build and push initial latest image: https://github.com/apache/hive/actions/runs/43885083693

yay, initial image is there: https://hub.docker.com/layers/apache/hive-ci-runner/latest/images/sha256-eb599f384ccaf4201a0585999a9b3e68f5cbb9f63b07933364d637d649f7e7c7

kuczoram pushed a commit to kuczoram/hive that referenced this pull request Nov 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants