Repository navigation
Stop the archive save run when the Wayback Machine turns it away - #539
Conversation
When web.archive.org refuses connections or rate limits, the run kept walking the list at 5 s intervals, which kept the block alive and spent 13 minutes on 23 URLs while saving 6. It also exited 0, so the workflow counted as successful, moved the last-run anchor forward and lost the other 17 captures for good. - wayback.save marks a result as blocked on a connection error or HTTP 429/509, and gives up connecting after 10 s. - archive_save waits a cool-off (60 s) and tries the page once more; turned away again, it stops, lists the URLs not tried in the summary and exits 1, so the next run starts from the last successful one. A page the archive fails to capture still only gets reported. - #fragment links count as their page, so hamburg.python.pizza is saved once instead of three times. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_016a2XGhTo4nmqhECRZjqgXd
for more information, see https://pre-commit.ci
|
Routine review 1. RequestNo linked issue; the request is the PR body: stop the "Archive New Conference URLs" run when web.archive.org turns it away, so the run fails and the next one picks up the unsaved URLs. Instances: 2. Checked against
3. Needs your decision
4. FindingsImportant: None. Nits
5. TestsThe new tests cover the invariant on every instance: blocked versus page failure in Base ( Change, same files: Change, full suite (REVIEW.md command):
Other checks on the head: CI Status, Data Validation, Python Tests, CodeQL and labeler succeeded. 6. Bounds
Generated by Claude Code |
|
Both nits checked. CI on
Generated by Claude Code |
requests re-raises a read timeout while the body downloads as ConnectionError(ReadTimeoutError), so a stall after the headers was marked blocked and could stop the run. Only a read timeout before the headers was handled as a page failure. main now also returns 1 from a named stopped flag instead of checking the last result. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_016a2XGhTo4nmqhECRZjqgXd
The 60 s cool-off and retry made a run against a blocking archive take about 2 minutes and rarely helped: a block that lifts within a minute is the exception. Stop at the first refusal instead. The run fails within seconds and the next run saves the pages from the last successful one. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_016a2XGhTo4nmqhECRZjqgXd
The last "Archive New Conference URLs" run spent over 13 minutes on 23 URLs and saved 6. web.archive.org refused most connections, and the script kept requesting one URL every 5 s, so the block never lifted. It still exited 0, which moved the last-run anchor forward and dropped the other 17 captures for good: those URLs are no longer new, and their save points had passed.
Changes
wayback.savemarks a result asblockedwhen the connection is refused or fails, or when the archive answers HTTP 429/509. It now gives up on connecting after 10 s. The read timeout stays at 60 s. A read timeout is notblocked, including one while the body downloads, which requests raises asConnectionError(ReadTimeoutError).archive_savestops at the firstblockedresult, lists the URLs it did not try in the step summary and exits 1. The workflow run fails, so the next run starts from the last successful one and picks those pages up. A page the archive fails to capture (403, 523, read timeout) is still only reported.#fragmentlink counts as its page, sohamburg.python.pizza/,/#sponsorsand/#cocare saved once, not three times.When the archive refuses connections, the run now fails within seconds; it took over 13 minutes before. One case is still slow: if connecting hangs, the run fails after the 10 s connect timeout.
Tests
pytest tests/test_archive_save.py tests/test_wayback.py: 60 passed. The new tests cover blocked and non-blocked save results (including a read timeout in the body), the stop with exit 1 and its "Not tried" list, a page failure that does not stop the run, and fragment folding. All pre-commit hooks pass.🤖 Generated with Claude Code
https://claude.ai/code/session_016a2XGhTo4nmqhECRZjqgXd