Repository navigation
experiment: slow tier two tests at a time per runner (do not merge) - #582
Closed
andrewpark-scaleai wants to merge 8 commits into
Closed
andrewpark-scaleai wants to merge 8 commits into
andrewpark-scaleai wants to merge 8 commits into
Conversation
andrewpark-scaleai
added a commit
that referenced
this pull request
Nov 10, 2026
…made local_stack_at's teardown force-removed every container named agent-env-local-*, which is every local sandbox's compose containers on this machine's Docker. Run two tests at a time and it removed the other worker's gateway mid-deploy (PR #582: multi_env_computers_test's three tests, 'Gateway container not running'). It now records the ids of the LocalSandboxes made while the stack is up and removes only theirs. Co-Authored-By: Claude <[email protected]>
andrewpark-scaleai
force-pushed
the
andrew-park/experiment-slow-tier-n2
branch
from
November 10, 2026 00:22
29b2ce5 to
5fc32f9
Compare
…ather than per-shard caches The slow tier saved each runner's whole registry to the Actions cache, keyed by shard and week. The same image landed on up to three shards, so the entries (1.6 to 6.3GB each) overflowed the repository's 10GB, evicted each other, and most shards built every image from scratch. - The composer copies an image of a key the env's image store lacks from AGENT_ENV_COMPUTER_IMAGE_MIRROR, read anonymously, before it would build it; a mirror it can't read leaves the image to be built. - registry_api.copy_image copies an image between registries by its digest, sending only the blobs the target lacks and each manifest byte for byte. - After its tests pass on main, each slow shard shares the images it built or copied to ghcr.io/scaleapi/agentenv-ci-computers under their keys and <key>-seen-<date>. - computer-image-mirror.yml deletes, daily, the versions no run has used for 7 days, keeping a kept index's platform manifests. Co-Authored-By: Claude <[email protected]>
Co-Authored-By: Claude <[email protected]>
…image in the mirror The slow tier's times are from PR #580's run 42716509774, attempt 2, which built no image; the installer's from the same run; the gateway's and unit tier's from main's run 42715765775, after #575 and #576. Co-Authored-By: Claude <[email protected]>
…fore its credentials go Co-Authored-By: Claude <[email protected]>
…rocess made multi_env_computers_test compared every container on the machine before and after a failed redeploy, so another pytest worker's sandboxes coming and going failed it when the slow tier ran two tests at a time (PR #582). tst/util/local_sandboxes.record notes the LocalSandboxes a process makes and containers() names theirs; the test and local_stack_at, which #584 taught the same, both use it. Co-Authored-By: Claude <[email protected]>
…tainers fails Co-Authored-By: Claude <[email protected]>
andrewpark-scaleai
force-pushed
the
andrew-park/experiment-slow-tier-n2
branch
from
November 10, 2026 02:05
5fc32f9 to
74dca92
Compare
…do not merge) Co-Authored-By: Claude <[email protected]>
andrewpark-scaleai
force-pushed
the
andrew-park/experiment-slow-tier-n2
branch
from
November 10, 2026 02:43
74dca92 to
29c56c4
Compare
Contributor
Author
|
Superseded by #600, which runs the slow tier two tests at a time on top of main. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The goal is to find out whether the slow tier can run two tests at a time on each of its 12 runners. Its tests run one at a time today because many share fixed image tags, artifact ids and Docker names.
Setup
This branch is #580 (the shared computer image mirror and the refreshed durations table) plus #584 (the shared local stack removes only its own sandboxes). On top of that:
pytest -n 2 --dist loadfile, so a file's tests, and its module fixtures, stay on one worker.Results
The first run, on main's per-shard cache, found three things:
multi_env_computers_test.pytests failed on one shard. test: have the local stack remove only the sandboxes its own process made #584 fixes that cause:local_stack_atremoved every local sandbox on the machine.I'll run it three more times to look for further collisions, then close this PR.
This PR is not for merging.