[vahid-ahmadi]

Refs #243 (demo-critical: hosted score_reform(model="obr") takes over 10 minutes).

What was found

The OBR session measured the hosted pieces (obr-for-macro-243.md in the shared notes). The OBR solve takes about 3 minutes; the rest is the PolicyEngine population costing at 2–3 minutes per year, against "~6 s per run" in the docs. Reading policyengine 6.11.1's source shows a large part of the reason:

• calculate_decile_impacts (policyengine/outputs/decile_impact.py:209–210) calls Simulation.ensure() on the simulations it is passed.
• ensure() (core/simulation.py:194) checks only policyengine's in-process _cache, then a saved output file. A simulation finished with run() is in neither, so it runs again and then save()s its output beside the input dataset (common/model_version.py:55).
• Macro runs the baseline and reform with run() and then passes them to calculate_decile_impacts. So every costing ran the reform twice, and the baseline twice on first use per year. On the hosted server each run also wrote a ~27 MB file to the shared Modal volume under a random name that is never read again. A read-only modal volume ls shows 1,725 such files since July (about 45 GB), 77 of them today.

Change

• core._pe_register_run(sim) adds a finished run to policyengine.core.simulation._cache under its storage_id, so the later ensure() reuses it. The result is one run per simulation and no file written. It is called after the baseline run, the reform run and the incidence-overlay run. If the private name moves (it is pinned to policyengine 6.11.1), it does nothing and behaviour falls back to the old double run.
• Runtime text: population_reform_impact now says two to three minutes hosted and points to start_job(tool="pe_population_impact"). score_reform's obr entry says over ten minutes on the default window and suggests start_job or years=1–2. start_job names the job tool for each slow tool.

Results are unchanged: the decile step reads the same run it would otherwise have repeated deterministically.

Also: the incidence tools run as jobs

frbus_shock_incidence, hank_shock_incidence, svar_inflation_incidence and define_scenario_incidence each end in a population microsimulation. In the demo pass a direct hosted frbus_shock_incidence call timed out, and while it held the server, model_summary, hank_summary and list_reform_parameters timed out too (they returned normally afterwards). They are now on jobs.JOB_TOOLS under their own names, and start_job documents them.

Decision log (Fable 5.2, asked because Vahid asked for no questions)

• Question: for the demo, should score_reform's runtime be handled with (A) docs and demo guidance, (B) parallelising the five per-year costings, (C) stopping per-run output files on the shared volume, or a combination?
• Fable's answer: A + C now; defer B until Report stage timings in the OBR score_reform result #253's stage timings confirm the cause.
• Reasoning: B is speculative and risks memory in a 4-CPU/8 GB container. C is small and low-risk, stops unbounded volume growth, and may help speed. Don't delete the existing files in the PR (a separate destructive decision). Remove the "~6 s" claim, but don't hard-code "~12 min", which would change again.
• How this PR implements C: registering the run stops the second run and the write together, rather than redirecting the output folder. That also removes the duplicate compute, which is likely a large part of the 2–3 minutes. I couldn't time it locally (the local venv has policyengine 4.22.3, and there is no HF token). Report stage timings in the OBR score_reform result #253's timing_seconds will show the effect on the first hosted run after both merge.

Not done (needs Vahid)

• Deleting the 1,725 orphaned files (~45 GB) on policyengine-macro-pe-data. The per-year datasets enhanced_frs_2023_24_year_2026…2030.h5 and populace_us_2024_year_2026.h5 must stay.
• The live test's 600 s budget: Remote smoke tests: run obr_shock and microsim score_reform through jobs #237 rewrites that test, so switching it to years=1 with a realistic budget belongs there.

Tests

New test_pe_register_run.py (with a stand-in policyengine module). Integration (excluding remote): 456 passed. Merging redeploys Modal.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
macromod	Ready	Preview	Nov 15, 2026 1:38pm UTC