Skip to content

ci: run the slow tier's tests two at a time on each shard - #600

Open
andrewpark-scaleai wants to merge 1 commit into
mainfrom
andrew-park/slow-tier-n2
Open

andrewpark-scaleai wants to merge 1 commit into
mainfrom
andrew-park/slow-tier-n2

Conversation

@andrewpark-scaleai

@andrewpark-scaleai andrewpark-scaleai commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Why

The goal is a faster slow tier on the same 12 runners, with no test weakened. Each slow shard runs one test at a time today, while its runner uses at most 4 GB of its 16 GB.

What changes

The slow step runs pytest -n 2 --dist loadfile. Each shard runs two test files at a time. A file's tests, and its module fixtures and pytest.mark.dependency chains, stay on one worker. No test or assertion changes.

Evidence

The #582 experiment ran this setup on main plus #580:

  • The last run passed on all 12 shards. The slowest job took 840s; on main the slowest shards take about 1100s. Peak memory was 2.7 to 3.5 GB of 16 GB, with no OOM kills.
  • One earlier run failed because a test counted every container on the machine. test: count and remove only the containers of sandboxes this test's process made #585 fixed that.
  • One earlier run failed on shards 4 and 10 in apt-heavy image builds, each after about 11 minutes, on separate runners. An independent review found no state the two workers share in either step: the resolver runs apt in its own container and work folder, and the base image tag carries a uuid. It is most likely an apt mirror flake. Its stderr was not kept, so this is not proven.

Neither experiment run tested this exact commit, so this PR's own local-backends run is the check. I'll report its per-shard times here.

Known limits

  • A shard can't finish faster than its biggest file. chromium_app_test.py takes about 550s.
  • Host-port choice is serialized per state root, so two workers can in rare cases pick the same free port. The fast tier already runs -n auto with the same exposure.

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge; no blocking failures were found.

Summary

Runs two slow-test workers on each of the existing 12 shards.

  • Each slow integration shard runs two test files at once.

andrewpark-scaleai acknowledged the known host-port collision exposure. andrewpark-scaleai also disclosed the earlier apt failures and missing stderr; no concrete failure caused by this change was established.

Reviews (1) · Last reviewed commit: "ci: run the slow tier's tests two at a t..." · Reviewed by Greptile

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant