[BUG] dotnet test never finishes on macOS ARM64 — two distinct test-host wedges, both in Mcp* classes

[erikdarlingdata]

dotnet test does not finish on macOS ARM64. It wedges partway through and the run never returns. CI is ubuntu-latest and is unaffected, which is why this has stayed invisible — it only costs whoever runs the suite locally, and it costs them a core.

Splitting this out of #437, where it turned up.

Two distinct signatures, not one

This is the part that took the longest to see, and it matters because 5f84116 documents only the first.

1. The GC-suspension livelock — the one 5f84116 describes. Spins at ~107-108% CPU. Sampling a wedged host shows 16 frames of SuspendEE / CheckActivationSafePoint, plus a .NET BGC thread. Reproduced under xunit.v3 4.0.0 / Microsoft.Testing.Platform.

2. A blocked wedge, which is something else. Under the current runner (VSTest, xunit.v3 3.2.2) I isolated a hang to McpPlanPathPolicyTests where every thread sits in ThreadNativeWait / Monitor_Wait, 0% CPU, no SuspendEE anywhere in the sample. That is not a livelock, it is a deadlock, and the two want different fixes.

I could not narrow #2 past the class. It survives excluding OpenAsync_TimesOutUnresponsiveNativeRoots and OpenAsync_ValidatesAndReturnsTheSameOpenedHandle, so it is not simply the infinite-Task.Delay stub or the symlink-swap test.

A lead worth chasing first

The wedged host had a live child process:

39273  PlanViewer.Core.Tests          <- wedged, 107% CPU
 └ 39275  dotnet planview.dll mcp serve   <- real MCP server, spawned by the tests

The suite starts real planview mcp serve subprocesses, and both hangs are in Mcp* classes. CI logs show the same thing from the other end — every run ends with the runner reaping orphans:

Terminate orphan process: pid (8721) (dotnet)
Terminate orphan process: pid (8722) (dotnet)
...

A child process that outlives, or fails to complete, its handshake would explain a 0%-CPU all-threads-blocked wedge much more naturally than a runtime bug does. I would look there before looking at CoreCLR.

No test-runner timeout stops signature 1

Measured against a genuinely wedged host, not assumed:

mechanism	setting	outcome
MTP hangdump	--hangdump-timeout 90s	ran 2m55s at 107% CPU, never fired
MTP session timeout	--timeout 60s	ran 8 minutes, never fired

MTP does put the timeout in a separate controller process (confirmed in the process tree) — that was the property 5f84116 argued was essential — and it still cannot win. Same reason the commit gives for xUnit's own timeouts: the diagnostics the platform needs are served by the execution engine that is suspended.

--blame-hang-timeout under VSTest did not fire on it either.

Honest gap: I have not shown the current TestSessionTimeout watchdog kills a livelocked host. 5f84116 verified it at 1ms on a healthy run, which proves the property is wired up, not that it can terminate a wedge. Both runners may be equally powerless here. Worth testing deliberately before trusting the 15-minute backstop.

Reproducing

dotnet test tests/PlanViewer.Core.Tests with no filter, on macOS ARM64 (.NET 10.0.10, Apple silicon). Not every run wedges; it took several attempts to land each signature. Reproduces on unmodified dev — the first run I did, before touching anything, hung this way.

Filtered runs are fine, which is the current workaround: everything except Mcp*/PlanRepl* passes (279 tests).

Why it is worth fixing rather than documenting

CONTRIBUTING.md tells contributors to run dotnet test. On a Mac that command does not return, and leaves a process pinning a core after they Ctrl-C the runner.

[erikdarlingdata]

Root cause found, and it is not a test-infrastructure problem. It is a broken P/Invoke that corrupts memory, in a security check.

fcntl is variadic, and this declaration is not

OpenedFilePathResolver.GetMacOsPath resolves an open handle back to its real path with F_GETPATH:

[DllImport("libc", EntryPoint = "fcntl", SetLastError = true)]
private static extern int Fcntl(int fileDescriptor, int command, byte[] buffer);

fcntl(2) is int fcntl(int fildes, int cmd, ...) — the third argument is variadic. On Apple arm64 the variadic ABI differs from AAPCS64: named arguments go in registers, variadic arguments go on the stack. A fixed-signature P/Invoke puts buffer in x2, and the callee never looks there.

Standalone repro, outside the test suite entirely — this declaration, one open file:

Arch      : Arm64
real path : /var/folders/px/.../T/probe-d37aacae….sqlplan
FIXED  rc=0 errno=0 len=0 -> ''

rc=0 means fcntl reported success. F_GETPATH on success writes the path into the buffer it was handed. Ours came back empty, so it wrote up to MAXPATHLEN bytes to whatever address happened to be in that stack slot. That is an arbitrary ~1KB write on every call.

It explains the distribution exactly

The three tests in McpPlanPathPolicyTests that pass are precisely the three that never reach GetFinalPath; the two that reach it both wedge. I ran this as a prediction before checking, not as a description afterwards:

test	opens a file → reaches GetFinalPath?	result
GetPathComparisonForPlatform	no (pure string theory)	passes, 3s
ContainsWindowsAlternateDataStream	no (pure string theory)	passes, 3s
OpenAsync_DeniesAdvertisedEmptyRoots	no — denied before opening	passes, 3s
OpenAsync_AllowsAPlanUnderTheFilesystemRoot	yes	HUNG (5/5)
OpenAsync_ValidatesAndReturnsTheSameOpenedHandle	yes	HUNG (2/2)

It is deterministic, not intermittent — I checked that too, because I had assumed the opposite: 5 consecutive runs of the class, 5 hangs.

It also explains why CI never sees it. Linux takes the /proc/self/fd branch and Windows takes GetFinalPathNameByHandle; only macOS goes through fcntl, and only arm64 has the mismatched ABI. ubuntu-latest cannot reproduce this.

And it explains the two signatures in the issue body: an arbitrary write into the runtime's own memory is a textbook way to get both a GC-suspension livelock and an unrelated-looking deadlock out of the same defect, nondeterministically, depending on what was in that stack slot.

The security half, which matters more than the hang

GetFinalPath is not a convenience. It is the TOCTOU re-validation — open the handle, then confirm the path the kernel actually opened is still inside the advertised roots, so a symlink swapped between check and open cannot smuggle a read out of bounds. OpenAsync_ValidatesAndReturnsTheSameOpenedHandle is the test for exactly that.

On macOS arm64 that check has never worked. It returns "", and Path.GetFullPath("") throws ArgumentException — which is not in OpenAsync's catch ... when (exception is IOException or UnauthorizedAccessException) filter, so it propagates instead of being handled. It fails closed, so it is not an exploitable bypass, but the guarantee the test claims to be proving is not being provided on that platform, and the FileStream leaks on the way out.

Fixing it

The declaration cannot be repaired in place — .NET has no varargs P/Invoke on this target (__arglist throws InvalidProgramException: Vararg calling convention not supported, which I confirmed). The options are a native shim, or not using a variadic function:

Preferred: compare identity instead of resolving the path. fstat on the handle and stat on the lexical path, then compare st_dev + st_ino. That answers the TOCTOU question directly — "is the thing I opened the thing I checked?" — and neither call is variadic. It also drops the platform-specific path-string parsing on all three platforms.

Whatever the fix, it needs a test that would have caught this: the current one asserts on .Label and passes on Linux and Windows while the underlying call has never worked on Apple silicon.

General rule worth writing down: no variadic libc function via plain DllImport in this repo. fcntl, open, ioctl, printf-family. They will appear to work on x64 and silently corrupt memory on Apple silicon.

[erikdarlingdata]

Fixed in #443. One correction to my comment above first, because the fix I recommended there is wrong and it sounds plausible enough that someone would build it.

fstat/stat device+inode comparison does not work here. stat() re-resolves the path. If a symlink was swapped before the open, stat() follows the swap too, so the fd's inode and the path's inode match and the check detects nothing — precisely the attack it was supposed to catch. Resolving the descriptor back to its real path is the only thing that answers "where does this handle actually live", which is the question the TOCTOU check exists to ask. I had the diagnosis right and the remedy backwards.

The actual fix is proc_pidfdinfo from libproc — same answer as F_GETPATH, fixed signature, so ordinary marshalling is correct. Repairing the fcntl declaration in place is not on the table: .NET has no varargs P/Invoke on this target, __arglist throws Vararg calling convention not supported.

Two details worth knowing if you touch that file:

• The struct offsets are derived in a comment from <sys/proc_info.h> rather than left as magic numbers, and the returned size is checked against the expected 1200 rather than trusted. A future macOS layout change now fails loudly instead of quietly returning wrong bytes — which is the exact failure mode this file just came out of.
• macOS answers with the canonical path: /private/var/... where Path.GetTempPath() reports /var/.... Roots already get canonicalized through ResolveLinkTarget, so containment still matches, but a string comparison against the path you passed in will fail for reasons unrelated to the defect.

Result:

	before	after
full dotnet test on macOS ARM64	never finished	16s, 305 passed, 0 failed
McpPlanPathPolicyTests	5/5 hangs	3/3 passes, 3s each

Both wedge signatures in the issue body are gone, which is the confirmation that they were one defect and not two: an arbitrary ~1KB write into runtime memory produces a GC-suspension livelock or an unrelated-looking deadlock depending on what was in that stack slot.

Leaving this open until #443 lands.

[erikdarlingdata]

Fixed by #443, shipped in v1.20.0.

dotnet test finishes on macOS now — 307 tests in about 13 seconds, where before it did not complete at all. Both wedge signatures in the issue body are gone, which is the confirmation they were one defect and not two: a variadic fcntl called through a fixed-signature DllImport wrote ~1KB through an arbitrary pointer on every call, and an arbitrary write into runtime memory surfaces as a GC-suspension livelock or an unrelated-looking deadlock depending on what was in that stack slot.

The rule it earned is written into CONTRIBUTING.md now, so the next person does not rediscover it: no variadic libc function through a plain DllImport — fcntl, open, ioctl, the printf family. They look fine on x64 and silently corrupt memory on Apple silicon.