Skip to content

docker-to-squash.py: don't swallow exceptions in shell_command - #9293

Open
linzch3 wants to merge 1 commit into
apache:masterfrom
linzch3:fix-9289-shell-command-finally
Open

linzch3 wants to merge 1 commit into
apache:masterfrom
linzch3:fix-9289-shell-command-finally

Conversation

@linzch3

@linzch3 linzch3 commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

What is the purpose of the change

Fixes #9289.

shell_command in bin/docker-to-squash.py returned from its finally clause. A return in finally discards any exception in flight, so raise_on_error=True never raised: shell_command(["false"], False, False, True) just returned (b'', b'', 1). Because of that check_dependencies reported success when hadoop or /usr/sbin/mksquashfs was missing, and failed hadoop commands (e.g. in atomic_upload_mv_to_hdfs) went unnoticed.

Changes:

  • Move the return out of finally, so the timer is still cancelled but exceptions propagate.
  • In the except block, only re-raise when raise_on_error is set (with from ex). With raise_on_error=False, a command that can't be started (e.g. hadoop not installed) still returns rc -1 like today, so does_hdfs_entry_exist(..., raise_on_error=False) and the other False callers keep their current behavior.
  • Update test_shell_command, which asserted the old behavior: with raise_on_error=False both badcmd and false return a non-zero rc, with raise_on_error=True both raise.

How was the change tested

  • python -m unittest -v test_docker_to_squash in bin/ with Python 3.9: passes. The updated test fails on master (Exception not raised).
  • On Python 3.10+ the module can't be imported at all (from collections import Iterable), that's one of the other follow-ups from docs: Add comments to docker-to-squash.py #9284, so I left it alone here.

Upgrade notes

An AI assistant helped draft this change; I reviewed it and take responsibility for it.

Happy to adjust anything, lmk.

The return in the finally clause discarded the exception, so
raise_on_error=True never raised. Return after the try instead, and only
re-raise launch errors when raise_on_error is set so callers passing False
still get the -1 return code.

Fixes apache#9289
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

docker-to-squash.py: shell_command swallows exceptions, so raise_on_error=True never raises

1 participant