Skip to content

docs: Add comments to docker-to-squash.py - #9284

Open
dragonknightk wants to merge 2 commits into
apache:masterfrom
dragonknightk:docs-docker-to-squash-comments
Open

dragonknightk wants to merge 2 commits into
apache:masterfrom
dragonknightk:docs-docker-to-squash-comments

Conversation

@dragonknightk

@dragonknightk dragonknightk commented Nov 18, 2026 •

Copy link
Copy Markdown

Fixes #7570

What is the purpose of the change

This adds comments and docstrings to docker-to-squash.py, as requested in the review of #3366, with no code changes. They cover the script's prerequisites, the HDFS layout, the image-tag-to-hash format, the main sub-commands and the non-obvious helpers.

Where the current behaviour is a bug, the docstring marks it as a "Known issue" and links a follow-up issue. This keeps the docs from turning a bug into the contract. The fixes are left to separate PRs.

How was the change tested

Comparing the AST of the old and new file with docstrings removed shows no difference, so only comments and docstrings changed. python3 -m py_compile passes.

bin/test_docker_to_squash.py passes on Python 3.9. On Python 3.10 and later it fails at import with and without this change, because of the collections.Iterable import.

Each known issue was reproduced on Python 3.9 against the unchanged script, with stub hadoop and skopeo commands in place of a cluster and a registry.

AI Disclosure

Human-led contribution. Grok Bot, working for dragonknightk, assisted with this change, including the code, tests, and description. All submitted content was reviewed by a human before submission.

ʕ•ᴥ•ʔ

Closes issue 7570

## What is the purpose of the change

This adds comments and docstrings to docker-to-squash.py, as requested in the review of PR 3366, with no code changes. They cover the script's prerequisites, the HDFS layout, the image-tag-to-hash format, the main sub-commands and the non-obvious helpers. The shell_command docstring states that exceptions raised in its try block, including the one for raise_on_error, are discarded by the return in the finally clause, so callers relying on raise_on_error do not see failures.

## How was the change tested

Comparing the AST of the old and new file with docstrings removed shows no difference, and the diff only adds lines. python3 -m py_compile passes. bin/test_docker_to_squash.py passes on Python 3.9. On Python 3.10 and later it fails at import with and without this change, because the script imports Iterable from collections.

## AI Disclosure

Human-led contribution. Grok Bot, working for dragonknightk, assisted with this change, including the code, tests, and description. All submitted content was reviewed by a human before submission.

Signed-off-by: KK <[email protected]>
@rzo1 rzo1 added documentation Improvements or additions to documentation skip-changelog labels Nov 18, 2026
@rzo1 rzo1 added this to the 3.3.0 milestone Nov 18, 2026

@rzo1 rzo1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thx for the contrib. The new docs are accurate and really help with this script. One concern: a few docstrings describe bugs as the intended contract, and once that's in the docstring it tends to stay. Could you add a short "Known issue" note where the behaviour is a bug and open a follow-up issue for each, so the docs don't lock that behaviour in? Details inline. Happy to merge after that, and fixes in a separate PR would be very welcome.

Comment thread bin/docker-to-squash.py Outdated
that stream goes to the console and is returned as None. stderr also goes to the console
when LOG_LEVEL is DEBUG. The process is killed after timeout_sec seconds.

Because the return statement is in the finally clause, any exception raised in the try

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a real bug, not intended behaviour: callers passing raise_on_error=True never see the failure. Could you phrase it as "Known issue (#…): …" and file an issue? Otherwise someone reading the docstring later will treat it as the contract.

Comment thread bin/docker-to-squash.py Outdated
Return (manifest, manifest hash) for image using `skopeo inspect --raw`.

If skopeo returns a manifest list (a multi-arch image), the amd64 entry is picked and
fetched again by digest as <image>@<digest>. Here image loses its tag and, if it contains

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here: re-fetching from the default registry (and dropping library from library/ubuntu) looks like a bug in image.split("/", 1)[-1], not something we want to document as behaviour. Please mark it as a known issue and link an issue.

Comment thread bin/docker-to-squash.py Outdated
def read_image_tag_to_hash(image_tag_to_hash):
"""
Parse the local image-tag-to-hash file (format in the module docstring) and return
(hash_to_tags, tag_to_hash). Malformed lines are skipped. If a tag appears on more than

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On Python 3 this isn't quite what happens. str(line) on a bytes object gives "b'tag:hash\\n'". Lines with a #comment still parse, but lines without one end in \n', fail the sha256 check and are skipped as "malformed", so their tags are silently lost. Please mention it as a known issue rather than "malformed lines are skipped".

Comment thread bin/docker-to-squash.py Outdated


def write_local_image_tag_to_hash(image_tag_to_hash, hash_to_tags):
"""Write hash_to_tags to the local file, sorted, dropping manifest hashes with no tags left."""

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI hash_to_tags.iteritems() doesn't exist in Python 3, so this function currently raises AttributeError. Together with the collections.Iterable import, every operation that changes the tag mapping fails on Python 3. Worth one issue covering the Python 3 problems (iteritems, str(bytes), collections.Iterable).

Closes issue 7570

## What is the purpose of the change

This adds comments and docstrings to docker-to-squash.py, as requested in the review of PR 3366, with no code changes. They cover the script's prerequisites, the HDFS layout, the image-tag-to-hash format, the main sub-commands and the non-obvious helpers.

Where the current behaviour is a bug, the docstring marks it as a "Known issue" and links a follow-up issue. This keeps the docs from turning a bug into the contract. The fixes are left to separate PRs.

- issue 9289: the return in the finally clause of shell_command discards exceptions, so raise_on_error=True never raises.
- issue 9290: when skopeo returns a manifest list, get_manifest_from_docker_image fetches the amd64 image again by digest but drops the registry or namespace from its name, so it can pull from the wrong registry or repository.
- issue 9291: the script does not run on Python 3, because of the collections.Iterable import, long and several bytes/str problems.
- issue 9292: get_working_dir creates a new, empty directory on every call, so push-update never finds the images that pull-build left.

## How was the change tested

Comparing the AST of the old and new file with docstrings removed shows no difference, so only comments and docstrings changed. python3 -m py_compile passes.

bin/test_docker_to_squash.py passes on Python 3.9. On Python 3.10 and later it fails at import with and without this change, because of the collections.Iterable import.

Each known issue was reproduced on Python 3.9 against the unchanged script, with stub hadoop and skopeo commands in place of a cluster and a registry.

## AI Disclosure

Human-led contribution. Grok Bot, working for dragonknightk, assisted with this change, including the code, tests, and description. All submitted content was reviewed by a human before submission.

Signed-off-by: dragonknightk <[email protected]>
@dragonknightk

Copy link
Copy Markdown
Author

@rzo1 Thanks for the review. I've marked the bugs as known issues and opened follow-ups: #9289, #9290, #9291, #9292. PTAL ʕ•ᴥ•ʔ

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation skip-changelog

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[STORM-3788] Add more comments in docker-to-squash.py

2 participants