Skip to content

Stop dotnet restore hanging when MSBuild worker nodes inherit its output - #15

Closed
GrahamTheCoder wants to merge 2 commits into
masterfrom
claude/busy-ramanujan-b0w0p8
Closed

GrahamTheCoder wants to merge 2 commits into
masterfrom
claude/busy-ramanujan-b0w0p8

Conversation

@GrahamTheCoder

@GrahamTheCoder GrahamTheCoder commented Nov 18, 2026 •

Copy link
Copy Markdown
Owner

Summary

MultiFileSolutionAndProjectTests sometimes stopped making progress, and after 5 minutes the blame hang timeout reported it as "Test host process crashed". The hang is in dotnet restore, which runs before any conversion code, so the code under test isn't involved.

Root cause

A hang dump showed no thread doing work: CPU use was 0%, no locks were held, and no thread was in CodeConverter code. dumpasync showed the test stuck here:

ConvertVbLibraryOnlyAsync -> MultiFileTestFixture.ConvertProjectsWhereAsync
 -> MsBuildWorkspaceConverter.ConvertProjectsWhereAsync -> SolutionLoader.AnalyzeSolutionAsync -> RunDotnetRestoreAsync
 -> ProcessRunner.GetOutputAsync -> Task.WhenAll(process.WaitForExitAsync(), stdOutComplete.Task, stdErrComplete.Task)
      Process.WaitForExitAsync -> WaitUntilOutputEOF -> PipeStream.ReadAsync   (stdout and stderr both pending)
  • The dotnet restore process had already exited (ProcessWaitState._exited = 1).
  • Its stdout and stderr pipes never reached EOF, because another process still held their write ends.
  • The restore's environment, read from the dump, contained MSBUILDENSURESTDOUTFORTASKPROCESSES=1. dotnet test sets this for its children, and the testhost passes it on.
  • With that variable set, MSBuild launches worker nodes without redirecting their streams (NodeLauncher.cs#L65-L67, Traits.cs#L487-L490). The /nodeReuse:true nodes therefore inherit our pipes and keep running after the build, waiting to be reused.
  • It's intermittent because new nodes are only launched when no compatible idle node is available.

Repro

I ran dotnet restore on the characterization solution with piped output, the testhost's environment, and a random MSBUILDNODEHANDSHAKESALT, which forces MSBuild to launch new nodes:

Variant Restore exit stdout/stderr EOF
Environment as in the testhost rc 0 none 15s after exit (hangs every time)
MSBUILDENSURESTDOUTFORTASKPROCESSES removed rc 0 straight away
MSBUILDDISABLENODEREUSE=1 rc 0 straight away
-nodeReuse:false rc 0 straight away
--disable-build-servers rc 0 straight away

Fix

ProcessRunner now sets MSBUILDDISABLENODEREUSE=1 for every process it starts. Worker nodes then exit when the build finishes, the pipes close and the wait completes. Setting it in ProcessRunner covers every GetOutputAsync caller: the restore, dotnet msbuild -getProperty and git diff. The XML doc comment records the reasoning.

I left out a timeout on waiting for EOF after the process exits. Process.WaitForExitAsync already waits for EOF internally, so a timeout would mean replacing it with our own handling of the exit event. It could also cut off output that a legitimately slow child is still writing. With the cause fixed, that complexity isn't worth it.

Needs your decision

Node reuse is now off for anything ProcessRunner starts, including the CLI's restore outside tests. The cost is starting new nodes on each run, about a second for a restore.

Tests

  • New regression test ProcessRunnerTests.GetOutputAsyncReturnsWhenBuildLeavesNewWorkerNodesRunningAsync. It doesn't run a second restore. Instead, it writes two SDK-less projects to a temp directory and runs dotnet msbuild -m:2 through GetOutputAsync, building them in parallel so that MSBuild launches a worker node. The environment has MSBUILDENSURESTDOUTFORTASKPROCESSES=1 and a unique handshake salt. The test waits at most 1 minute for the output to end.
    • Before the fix it fails: "dotnet msbuild output didn't end within 00:01:00".
    • After the fix it passes in about 0.6s. It needs no NuGet or SDK evaluation, and isn't in the MSBuild collection because it uses its own directory.
  • The full suite was run 3 times with --blame-hang --blame-hang-timeout 5m, against the first version of this PR, which used a restore-based test. Every run gave 1426 passed, 5 skipped and 0 failed, which matches the baseline of 1425/5/0 plus the new test.

Generated by Claude Code

ProcessRunner waits for the child's redirected stdout/stderr to reach EOF.
When MSBUILDENSURESTDOUTFORTASKPROCESSES=1 is inherited (dotnet test sets
it), MSBuild launches worker nodes without redirecting their streams, so
newly launched reusable nodes keep our pipes open after the build ends and
the wait never completes. It hung MultiFileSolutionAndProjectTests
intermittently, whenever restore had to launch new nodes.

Disable node reuse for processes we run, so the nodes exit with the build.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_019FEr8FVAmHV9bscZ82CChg
…ad of a restore

A parallel build of two SDK-less projects is enough to make MSBuild launch a
worker node, so the test no longer adds a second restore of the
characterization solution to the run. It takes ~0.6s, needs no NuGet or SDK
evaluation, and no longer needs the MSBuild collection since it uses its own
temp directory. It still fails without MSBUILDDISABLENODEREUSE.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_019FEr8FpweGikrBKYdcSCS1
@GrahamTheCoder

Copy link
Copy Markdown
Owner Author

Superseded: icsharpcode#1323

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants