Repository navigation
docs: Add comments to docker-to-squash.py - #9284
dragonknightk wants to merge 2 commits into
Conversation
Closes issue 7570 ## What is the purpose of the change This adds comments and docstrings to docker-to-squash.py, as requested in the review of PR 3366, with no code changes. They cover the script's prerequisites, the HDFS layout, the image-tag-to-hash format, the main sub-commands and the non-obvious helpers. The shell_command docstring states that exceptions raised in its try block, including the one for raise_on_error, are discarded by the return in the finally clause, so callers relying on raise_on_error do not see failures. ## How was the change tested Comparing the AST of the old and new file with docstrings removed shows no difference, and the diff only adds lines. python3 -m py_compile passes. bin/test_docker_to_squash.py passes on Python 3.9. On Python 3.10 and later it fails at import with and without this change, because the script imports Iterable from collections. ## AI Disclosure Human-led contribution. Grok Bot, working for dragonknightk, assisted with this change, including the code, tests, and description. All submitted content was reviewed by a human before submission. Signed-off-by: KK <[email protected]>
rzo1
left a comment
There was a problem hiding this comment.
Thx for the contrib. The new docs are accurate and really help with this script. One concern: a few docstrings describe bugs as the intended contract, and once that's in the docstring it tends to stay. Could you add a short "Known issue" note where the behaviour is a bug and open a follow-up issue for each, so the docs don't lock that behaviour in? Details inline. Happy to merge after that, and fixes in a separate PR would be very welcome.
| that stream goes to the console and is returned as None. stderr also goes to the console | ||
| when LOG_LEVEL is DEBUG. The process is killed after timeout_sec seconds. | ||
|
|
||
| Because the return statement is in the finally clause, any exception raised in the try |
There was a problem hiding this comment.
This is a real bug, not intended behaviour: callers passing raise_on_error=True never see the failure. Could you phrase it as "Known issue (#…): …" and file an issue? Otherwise someone reading the docstring later will treat it as the contract.
| Return (manifest, manifest hash) for image using `skopeo inspect --raw`. | ||
|
|
||
| If skopeo returns a manifest list (a multi-arch image), the amd64 entry is picked and | ||
| fetched again by digest as <image>@<digest>. Here image loses its tag and, if it contains |
There was a problem hiding this comment.
Same here: re-fetching from the default registry (and dropping library from library/ubuntu) looks like a bug in image.split("/", 1)[-1], not something we want to document as behaviour. Please mark it as a known issue and link an issue.
| def read_image_tag_to_hash(image_tag_to_hash): | ||
| """ | ||
| Parse the local image-tag-to-hash file (format in the module docstring) and return | ||
| (hash_to_tags, tag_to_hash). Malformed lines are skipped. If a tag appears on more than |
There was a problem hiding this comment.
On Python 3 this isn't quite what happens. str(line) on a bytes object gives "b'tag:hash\\n'". Lines with a #comment still parse, but lines without one end in \n', fail the sha256 check and are skipped as "malformed", so their tags are silently lost. Please mention it as a known issue rather than "malformed lines are skipped".
|
|
||
|
|
||
| def write_local_image_tag_to_hash(image_tag_to_hash, hash_to_tags): | ||
| """Write hash_to_tags to the local file, sorted, dropping manifest hashes with no tags left.""" |
There was a problem hiding this comment.
FYI hash_to_tags.iteritems() doesn't exist in Python 3, so this function currently raises AttributeError. Together with the collections.Iterable import, every operation that changes the tag mapping fails on Python 3. Worth one issue covering the Python 3 problems (iteritems, str(bytes), collections.Iterable).
Closes issue 7570 ## What is the purpose of the change This adds comments and docstrings to docker-to-squash.py, as requested in the review of PR 3366, with no code changes. They cover the script's prerequisites, the HDFS layout, the image-tag-to-hash format, the main sub-commands and the non-obvious helpers. Where the current behaviour is a bug, the docstring marks it as a "Known issue" and links a follow-up issue. This keeps the docs from turning a bug into the contract. The fixes are left to separate PRs. - issue 9289: the return in the finally clause of shell_command discards exceptions, so raise_on_error=True never raises. - issue 9290: when skopeo returns a manifest list, get_manifest_from_docker_image fetches the amd64 image again by digest but drops the registry or namespace from its name, so it can pull from the wrong registry or repository. - issue 9291: the script does not run on Python 3, because of the collections.Iterable import, long and several bytes/str problems. - issue 9292: get_working_dir creates a new, empty directory on every call, so push-update never finds the images that pull-build left. ## How was the change tested Comparing the AST of the old and new file with docstrings removed shows no difference, so only comments and docstrings changed. python3 -m py_compile passes. bin/test_docker_to_squash.py passes on Python 3.9. On Python 3.10 and later it fails at import with and without this change, because of the collections.Iterable import. Each known issue was reproduced on Python 3.9 against the unchanged script, with stub hadoop and skopeo commands in place of a cluster and a registry. ## AI Disclosure Human-led contribution. Grok Bot, working for dragonknightk, assisted with this change, including the code, tests, and description. All submitted content was reviewed by a human before submission. Signed-off-by: dragonknightk <[email protected]>
Fixes #7570
What is the purpose of the change
This adds comments and docstrings to docker-to-squash.py, as requested in the review of #3366, with no code changes. They cover the script's prerequisites, the HDFS layout, the image-tag-to-hash format, the main sub-commands and the non-obvious helpers.
Where the current behaviour is a bug, the docstring marks it as a "Known issue" and links a follow-up issue. This keeps the docs from turning a bug into the contract. The fixes are left to separate PRs.
How was the change tested
Comparing the AST of the old and new file with docstrings removed shows no difference, so only comments and docstrings changed. python3 -m py_compile passes.
bin/test_docker_to_squash.py passes on Python 3.9. On Python 3.10 and later it fails at import with and without this change, because of the collections.Iterable import.
Each known issue was reproduced on Python 3.9 against the unchanged script, with stub hadoop and skopeo commands in place of a cluster and a registry.
AI Disclosure
Human-led contribution. Grok Bot, working for dragonknightk, assisted with this change, including the code, tests, and description. All submitted content was reviewed by a human before submission.
ʕ•ᴥ•ʔ