Skip to content

Stop the archive save run when the Wayback Machine turns it away - #539

Merged
JesperDramsch merged 4 commits into
mainfrom
ai-agent/blissful-cray-7cq6ft
Nov 17, 2026
Merged

JesperDramsch merged 4 commits into
mainfrom
ai-agent/blissful-cray-7cq6ft

Conversation

@JesperDramsch

@JesperDramsch JesperDramsch commented Nov 17, 2026 •

Copy link
Copy Markdown
Owner

The last "Archive New Conference URLs" run spent over 13 minutes on 23 URLs and saved 6. web.archive.org refused most connections, and the script kept requesting one URL every 5 s, so the block never lifted. It still exited 0, which moved the last-run anchor forward and dropped the other 17 captures for good: those URLs are no longer new, and their save points had passed.

Changes

  • wayback.save marks a result as blocked when the connection is refused or fails, or when the archive answers HTTP 429/509. It now gives up on connecting after 10 s. The read timeout stays at 60 s. A read timeout is not blocked, including one while the body downloads, which requests raises as ConnectionError(ReadTimeoutError).
  • archive_save stops at the first blocked result, lists the URLs it did not try in the step summary and exits 1. The workflow run fails, so the next run starts from the last successful one and picks those pages up. A page the archive fails to capture (403, 523, read timeout) is still only reported.
  • A #fragment link counts as its page, so hamburg.python.pizza/, /#sponsors and /#coc are saved once, not three times.

When the archive refuses connections, the run now fails within seconds; it took over 13 minutes before. One case is still slow: if connecting hangs, the run fails after the 10 s connect timeout.

Tests

pytest tests/test_archive_save.py tests/test_wayback.py: 60 passed. The new tests cover blocked and non-blocked save results (including a read timeout in the body), the stop with exit 1 and its "Not tried" list, a page failure that does not stop the run, and fragment folding. All pre-commit hooks pass.

🤖 Generated with Claude Code

https://claude.ai/code/session_016a2XGhTo4nmqhECRZjqgXd

When web.archive.org refuses connections or rate limits, the run kept
walking the list at 5 s intervals, which kept the block alive and spent
13 minutes on 23 URLs while saving 6. It also exited 0, so the workflow
counted as successful, moved the last-run anchor forward and lost the
other 17 captures for good.

- wayback.save marks a result as blocked on a connection error or
  HTTP 429/509, and gives up connecting after 10 s.
- archive_save waits a cool-off (60 s) and tries the page once more;
  turned away again, it stops, lists the URLs not tried in the summary
  and exits 1, so the next run starts from the last successful one.
  A page the archive fails to capture still only gets reported.
- #fragment links count as their page, so hamburg.python.pizza is
  saved once instead of three times.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_016a2XGhTo4nmqhECRZjqgXd
@github-actions github-actions Bot added the Tests label Nov 17, 2026

Copy link
Copy Markdown
Owner Author

Routine review

1. Request

No linked issue; the request is the PR body: stop the "Archive New Conference URLs" run when web.archive.org turns it away, so the run fails and the next one picks up the unsaved URLs. Instances: utils/wayback.py save (blocked for connection errors and HTTP 429/509, 10 s connect timeout, 60 s read timeout kept), utils/archive_save.py main (60 s --cool-off, one retry, stop, "Not tried" list, exit 1), and #fragment links folded into their page in entry_urls.

2. Checked against

  • Date: Tue Nov 17 15:53:43 UTC 2026 (date).
  • requests 2.34.2 as installed in the sandbox (requests/models.py, Response.iter_content): a ReadTimeoutError while reading the body is re-raised as requests.ConnectionError; issubclass(requests.ConnectTimeout, requests.ConnectionError) is True. Read from the installed source, no URL fetched.

3. Needs your decision

  1. The fragment folding (entry_urls now applies urldefrag and deduplicates) is not part of the title's ask. It is small, tested, and cuts requests during a rate limit. Keep it in this PR, or split it into its own? Recommendation: keep it.

4. Findings

Important: None.

Nits

  • utils/wayback.py:202: a read timeout while the response body is downloading is raised by requests as ConnectionError (see Checked against), so it counts as blocked. The run waits 60 s, tries once more, and then stops with exit 1. The body says a read timeout is "still only reported". test_read_timeout_is_not_blocked mocks only ReadTimeout, the timeout before the headers arrive. If the archive sends headers and then stalls, that one page fails the run. It is rare and the next run catches up, so this is a judgement call; one option is to treat a ConnectionError whose cause is ReadTimeoutError as not blocked.
  • utils/archive_save.py:300-302: the docstring says it returns 1 "when the archive turned the run away", but the code returns 1 only when the last result is blocked. That is the same thing at present, since the loop breaks on the second refusal. A named flag would make that clear.

5. Tests

The new tests cover the invariant on every instance: blocked versus page failure in save (429, 523, read timeout, connection error), retry-then-succeed, two refusals stop with exit 1 and a "Not tried" list, a page failure does not stop the run, and fragment folding. On the base revision of utils/archive_save.py and utils/wayback.py they fail, and on the change they pass:

Base (tests/test_archive_save.py tests/test_wayback.py):

FAILED tests/test_archive_save.py::TestNewUrls::test_fragment_is_the_same_page
FAILED tests/test_archive_save.py::TestTurnedAway::test_retry_after_cool_off
FAILED tests/test_archive_save.py::TestTurnedAway::test_stops_and_fails_when_turned_away_twice
FAILED tests/test_archive_save.py::TestTurnedAway::test_page_failure_does_not_stop_the_run
FAILED tests/test_wayback.py::TestSave::test_refused - AttributeError: 'SaveR...
FAILED tests/test_wayback.py::TestSave::test_page_failure_is_not_blocked - At...
FAILED tests/test_wayback.py::TestSave::test_read_timeout_is_not_blocked - At...
FAILED tests/test_wayback.py::TestSave::test_connection_error - AttributeErro...
8 failed, 52 passed in 0.26s

Change, same files: 60 passed in 0.12s

Change, full suite (REVIEW.md command): 2634 passed, 4 skipped, 3 xfailed, 2 xpassed, 251 warnings in 121.64s (0:02:01)

uvx [email protected] check utils tests: All checks passed!

pre-commit.ci - pr: failure; the Pre-commit Checks check run also failed. The cause is isort (--force-single-line-imports, rev 5.13.2 from .pre-commit-config.yaml), run locally:

ERROR: /home/user/pr/utils/archive_save.py Imports are incorrectly sorted and/or formatted.
@@ -44,7 +44,6 @@
 from datetime import timedelta
 from datetime import timezone
 from pathlib import Path
-
 from urllib.parse import urldefrag

Other checks on the head: CI Status, Data Validation, Python Tests, CodeQL and labeler succeeded.

6. Bounds

  • I checked out head 5c05089 (rev-parse matched) and read the PR through the REST API.
  • I read the full diff, the unchanged parts of utils/archive_save.py it touches (normalise, due_urls, new_urls), and .github/workflows/archive-new-urls.yml. The workflow anchors on gh run list --status success, so exit 1 does hold the anchor back as the body says.
  • I did not run the data check (sort_yaml.py) because the PR changes no _data file. I did not run the workflow against the live archive.
  • Pre-existing: when a run stops, the captures that succeeded before the stop are requested again on the next run, because the anchor does not move. That costs requests, not correctness.

Generated by Claude Code

Copy link
Copy Markdown
Owner Author

Both nits checked. CI on 89231d2 is green: pre-commit.ci pushed the isort fix, and the tree matches what I had locally.

  • Read timeout during the body (utils/wayback.py): confirmed in requests 2.34.2, where Response.iter_content turns ReadTimeoutError into ConnectionError. A stall after the headers therefore counts as blocked and can stop the run, which contradicts the PR body. The fix is to treat a ConnectionError whose cause is a ReadTimeoutError as a page failure, with a test that raises it that way. Since it is marked a nit, it isn't getting its own push. It goes into the next push on this PR, or into a push of its own if you'd like it before merging.
  • Return value (utils/archive_save.py): agreed. A stopped flag will go in the same push.

Generated by Claude Code

requests re-raises a read timeout while the body downloads as
ConnectionError(ReadTimeoutError), so a stall after the headers was
marked blocked and could stop the run. Only a read timeout before the
headers was handled as a page failure. main now also returns 1 from a
named stopped flag instead of checking the last result.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_016a2XGhTo4nmqhECRZjqgXd
The 60 s cool-off and retry made a run against a blocking archive take
about 2 minutes and rarely helped: a block that lifts within a minute is
the exception. Stop at the first refusal instead. The run fails within
seconds and the next run saves the pages from the last successful one.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_016a2XGhTo4nmqhECRZjqgXd
@JesperDramsch
JesperDramsch merged commit 6cc8d25 into main Nov 17, 2026
10 checks passed
@JesperDramsch
JesperDramsch deleted the ai-agent/blissful-cray-7cq6ft branch November 17, 2026 16:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants