Skip to content

Overlap the sequential waits inside the decompiler test infrastructure - #4034

Open
christophwille wants to merge 7 commits into
masterfrom
decompiler-tests-overlap
Open

Overlap the sequential waits inside the decompiler test infrastructure#4034
christophwille wants to merge 7 commits into
masterfrom
decompiler-tests-overlap

Conversation

@christophwille

@christophwille christophwille commented Aug 19, 2026

Copy link
Copy Markdown
Member

The "Round 2: sequential awaits inside the test infra" commits of #3940, split out on their own. They do not depend on #3940's NUnit worker-count / fixture-parallelism change (which is not being pursued): RoundtripAssembly and the matrix runners are already Parallelizable(ParallelScope.All) on master, and the overlaps below are per-test or per-process, independent of how many workers NUnit runs. All seven apply to master without conflicts and are cherry-picked verbatim; only ICSharpCode.Decompiler.Tests is touched (Helpers/Tester.cs, Helpers/RoslynToolset.cs, CorrectnessTestRunner.cs, RoundtripAssembly.cs).

commit what why
Synchronize toolset registration dictionaries lock around the two Dictionary.Add in RoslynToolset/RefAssembliesToolset.Fetch; lookups stay lock-free (they only happen after Initialize completed) prerequisite of the next one
Overlap toolset downloads and TestRunner builds in Tester.Initialize the nine NuGet fetches and the TestRunner builds start eagerly and are awaited in one Task.WhenAll (every failure surfaced, not just the first); only the two Windows RID TestRunner builds stay sequential with each other (shared obj/, implicit restores would race on project.assets.json) Initialize gates every test in the suite; on a cold machine / CI it serialized all downloads and builds
Cache the vswhere-based MSBuild lookup Lazy<Task<string>> (ExecutionAndPublication); the non-Windows Assert.Ignore stays outside the cache so it is raised per test, not memoized as a faulted task the parallel roundtrip fixture spawned one vswhere.exe per test for a process-invariant answer
Run original and decompiled executables concurrently Tester.StartRun returns the in-flight run (fault pre-observed, so an abandoned run after an upstream failure cannot raise UnobservedTaskException); RunAndCompareOutput gets an overload taking that task and awaits both with a plain WhenAll (keeps NUnit Ignore semantics); exit codes are still asserted in the original order, so failure output is unchanged the two runs are independent processes with separately buffered output
Start the original executable before decompiling in correctness tests RunCS/RunVB/RunIL start the original exe right after the first compile; for mcs configs the .exe.config write moves ahead of the run start (the runtime reads it at launch) while the compiler-option mutation stays after the decompile, which must see the original options the decompile/recompile stages only read the original binary
Make the roundtrip testAction asynchronous Func<string, Task> instead of GetAwaiter().GetResult() on NUnit worker threads cleanup; enables the next one
Overlap the pristine-executable run with the roundtrip pipeline RunWithOutput starts the reference exe from the ILSpy-tests checkout before the whole-project decompile + MSBuild rebuild (nothing in that pipeline writes to the input dir); the submodule-missing guard is hoisted into EnsureTestDirAvailable() and runs before the early start, so those tests still report Ignored the pristine run used to wait for the multi-minute decompile it does not depend on

Verification

Local, 24-thread Windows 11 box, Debug, ILSpy-tests checked out (roundtrips and the mcs matrix run: 140 mcs cases passed, which exercises the .exe.config-before-run reordering), workstation GC (this branch is off master; #4033 is separate):

run result wall sum of per-test durations (TRX)
master, warm (same day, same box) 4254 / 0 / 20 553 s 11,532 s
this branch, cold (bin/.../{roslyn,vswhere,netfx} deleted, exercises the concurrent Initialize) 4254 / 0 / 20 554 s 10,819 s
this branch, warm 4254 / 0 / 20 512 s 10,663 s

The 20 skips are the usual environment-gated ones. The wall-time difference is within a few percent and is not the point; the per-test overlap shows up as ~7% less summed test time for the same work. The local "cold" Initialize is served from ILSpy-tests/nuget and the already-built TestRunner, so it is only a correctness check of the concurrent path here (4 s); CI is where the downloads actually happen.

CI

Run https://github.com/icsharpcode/ILSpy/actions/runs/32240841471: all four jobs green (Windows Debug/Release, Linux, macOS).

decompiler test host master (run 32177067040) this branch
Windows Debug 1393 s 1373 s
Windows Release 664 s 766 s
Linux (own step) 6m19 6m25

No measurable CI wall-time effect either way: the Windows numbers sit inside the run-to-run spread of that shared 4-core runner (master's own Release test step has ranged 11m10-15m26 across recent runs), and on Linux the suite is CPU-bound with most of its matrix skipped. The value of these commits is structural (no blocking waits on worker threads, no per-test vswhere.exe, setup downloads and builds overlapped, original/decompiled runs concurrent), not a headline number.

Not taken from #3940: the NUnit LevelOfParallelism/Parallelizable(Fixtures)/Order commit and the Defender doc (dropped), the block-invariant change (reverted there), server GC (#4033), the fixture-host hardening (#4032), the concurrent .NET Framework scan and the diagnostics sampler (reverted there).

🤖 Generated with Claude Code

Tester.Initialize is about to issue the toolset Fetch calls concurrently;
each Fetch ends by registering its install path in a plain Dictionary,
which is not safe for concurrent writers. Lookups need no lock: they only
happen after Initialize has awaited all registrations.

Assisted-by: Claude:claude-fable-5:Claude Code
The setup fixture gates every test in the suite, and on a cold machine it
serialized nine NuGet fetches plus two self-contained TestRunner builds.
The fetches extract into disjoint directories and the builds depend on no
fetched toolset, so everything now runs concurrently and is awaited once.
Only the two Windows RID builds stay sequential with each other: they
share the TestRunner project's obj/ directory, and their implicit
restores would race on project.assets.json.

Assisted-by: Claude:claude-fable-5:Claude Code
Every roundtrip test spawned its own vswhere.exe to answer a question
that is invariant for the lifetime of the process. Lazy<Task<string>>
with ExecutionAndPublication guarantees a single spawn even when the
parallel roundtrip fixture hits the lookup from several tests at once.

Assisted-by: Claude:claude-fable-5:Claude Code
RunAndCompareOutput awaited the two runs back to back, but they are
independent processes with separately buffered output. The new StartRun
helper also lets callers begin the original run even earlier and hand
the in-flight task to the comparison; it pre-observes the task fault so
a run abandoned after an upstream failure cannot surface as an
UnobservedTaskException. Plain WhenAll (no error aggregation) keeps
NUnit Ignore semantics when both runs raise IgnoreException, and the
exit codes are still asserted in the original order, so failure output
is unchanged.

Assisted-by: Claude:claude-fable-5:Claude Code
The original binary is complete once the first compile (or ilasm)
finishes, and the decompile/recompile stages only read it, so its
execution now overlaps them instead of waiting at the very end of the
pipeline. For mcs configurations the .exe.config write moves ahead of
the run start - the runtime reads it at process launch - while the
compiler-option mutation stays after the decompile, which must see the
original options.

Assisted-by: Claude:claude-fable-5:Claude Code
The RunWithTest/RunWithOutput lambdas blocked an NUnit worker thread
with GetAwaiter().GetResult() on inherently async work. Passing a
Func<string, Task> lets RunInternal await the action, and enables
handing an already-running execution into the comparison.

Assisted-by: Claude:claude-fable-5:Claude Code
In RunWithOutput roundtrip tests the reference executable from the
ILSpy-tests checkout ran only after the whole-project decompile and the
MSBuild rebuild had finished, although nothing in that pipeline writes
to the input directory. Its execution now starts first and overlaps the
multi-minute decompile. The submodule-missing guard moves ahead of the
early start so those tests still report Ignored, not a faulted launch.

Assisted-by: Claude:claude-fable-5:Claude Code
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant