[andrewpark-scaleai]

Why

The goal is a slow tier that can run two tests at a time on each runner. On #582's rerun with -n 2, one shard failed multi_env_computers_test.py::test_two_computers_keep_serving_through_restarts_and_a_failed_redeploy_and_close_ends_them:

AssertionError: ({'agent-local-154e6343'}, …)
Extra items in the left set: 'agent-local-154e6343'
Extra items in the right set: 'agent-local-30e0f552'

The test compared every container on the machine before and after a failed redeploy, so the other worker's sandboxes coming and going failed it.

What changes

• tst/util/local_sandboxes.py provides two helpers:
  • recorded_sandbox_ids(monkeypatch) lists the LocalSandbox ids a process makes.
  • container_names_carrying(ids) names their containers. Every local sandbox container carries its sandbox's id: agent-<id>, relay-<id>, and the gateway stack's agent-env-<id>-…. It runs docker ps with check=True, so a Docker failure raises rather than reading as "no containers".
• multi_env_computers_test.py counts only the containers of the sandboxes its test made, through a made fixture. So "nothing left up" is asserted as == set().
• local_stack_at, which test: have the local stack remove only the sandboxes its own process made #584 scoped the same way, now uses the same helper.

Verification

• An independent review traced every container the local backend creates. Each carries its sandbox's id. Both places that construct a LocalSandbox record it, module fixtures make no sandboxes, and the narrowed check is at least as strict as before for every container this test makes.
• Both files collect, and ruff check passes.
• experiment: slow tier two tests at a time per runner (do not merge) #582 will rerun with this change to look for further collisions.

Mutation check

The new checks must still catch a leak, so I ran the two changed tests against a mutant whose LocalSandbox.terminate() returns at once. This was a local run of main + #580 + this PR, on Python 3.12, with a local registry and the computer image mirror, as the slow tier runs.

• Baseline: both tests pass (369s).
• Mutant: both tests fail, and the failures list the leaked containers by name, so the recorded sandbox ids match real containers.
  • test_a_computer_that_fails_mid_deploy_fails_the_deploy_and_leaves_nothing_up fails at _containers_made(made) == set() and lists agent-env-local-2044d505-…-pgweb-1 and others.
  • test_two_computers_keep_serving_through_restarts_and_a_failed_redeploy_and_close_ends_them fails at after_failure == deployed_before, which lists the failed redeploy's agent-env-local-d64414bc-… and agent-local-… containers.

Confidence Score: 5/5

The PR appears safe to merge; the latest change fixes the reported cleanup problem.

Summary

The PR limits container checks and cleanup to sandboxes created by the current test process.

• Sandbox tests check only containers from their own process.
• Local stack cleanup removes only its own gateway containers.

_{Reviews (8) · Last reviewed commit: "test: still reset the stores and remove ..." · Reviewed by Greptile}