Set FUTEX_WAITERS under the bucket lock in LOCK_PI

[xalestar]

Fixes #413.

pthread-tls-probe hung because the guest main thread was parked in FUTEX_LOCK_PI and nothing woke it. It never reached pthread_join. The lock is one that Rosetta's runtime takes around its own work: the futex word is at 0x8000001a2440, inside the translator's range, and the worker thread takes and releases it 13 times before it exits.

futex_lock_pi_inner set FUTEX_WAITERS before it took the bucket lock, and under the lock it only checked that the word was still owned. The loop that sets the bit also left without setting it when it read an unowned word. An owner that released and retook the lock in guest code between those reads left the waiter parked on an owned word with the bit clear. The owner's next release was then the guest-side CAS from its tid to 0, which makes no syscall, so FUTEX_UNLOCK_PI never ran. The waiter's 100 ms poll leaves only for exit_group, a signal, FUTEX_OWNER_DIED, or a dead owner tid, and a word of 0 is none of those.

A temporary trace, not part of this change, showed the same three reads in each of three hangs: the fast-path CAS saw 0x2, the loop read 0, and the read under the bucket lock saw 0x2 again, with no FUTEX_UNLOCK_PI after it.

The bit is now set and the waiter enqueued under one hold of the bucket lock. With the bit set the owner can only release through FUTEX_UNLOCK_PI, which takes the same lock after its release and so finds the waiter.

tests/test-futex-pi.c gains one case: two threads take and release one PI lock and enter the kernel only on contention, until 2000 acquisitions have gone through FUTEX_LOCK_PI. That call carries a 2 s deadline, so a lost wake is reported as a failure and does not hang the suite. The case also fails when no acquisition needed the kernel, so it cannot pass without contention.

Based on main at b57f367. Measured on an Apple M5 Pro, macOS 27.0.1, with no other elfuse test running:

• Loop from the issue: 6 of 300 runs timed out on main; 0 of 1000 timed out with this change.
• make test-rosetta-glibc: 5 runs in a row pass, 7 of 7 probes each.
• New case in test-futex-pi: reported a lost wake in 20 of 20 runs on main, and in 0 of 20 runs with this change.
• make check: all sections pass.
• make test-matrix-elfuse-aarch64: 296 passed, 1 failed, 10 skipped. The failure is test-sigio on the TCP out-of-band SIGURG case, which is intermittent on both trees: standalone it failed 9 of 20 runs on unmodified main and 10 of 20 runs on this branch.
• make test-matrix-elfuse-x86_64: 77 passed, 1 failed, 1 skipped. The failure is audit-known-limitations (rt_sigreturn assertion inside Rosetta, rc=133). It fails 3 of 3 runs on unmodified main and 3 of 3 runs on this branch.
• make lint, make analyze, make check-format, .ci/check-cppcheck.sh and make verify-futexdeadline pass. make analyze reports nothing in src/runtime/futex.c.

Two more commits close gaps found in review:

• The timeout path of FUTEX_LOCK_PI cleared FUTEX_WAITERS after it dropped the bucket lock, so a waiter that enqueued in that gap could park on an owned word without the bit. It now clears under the lock. This has no test: a stress case of 2000 rounds passed 16 of 16 runs with only this fix reverted, 6 of them with every core loaded, because the gap is a few instructions long.
• FUTEX_LOCK_PI on a word holding FUTEX_WAITERS with a TID of 0 spun in the host, since the CAS(0->TID) fast path never matches it. It now takes the word over, as Linux does in futex_lock_pi_atomic. The new test-futex-pi case hangs on the previous head and passes with the change.

After these, make check, make lint, make check-format, .ci/check-cppcheck.sh and make verify-futexdeadline pass again.

Not changed here, and left for follow-up pull requests because none of them is needed for this hang:

• FUTEX_UNLOCK_PI writes 0 and wakes one waiter; it does not hand the lock to that waiter. With two or more waiters the others stay parked on a word without FUTEX_WAITERS.
• The owner-died recovery writes 0 to the word, which can remove the bit while another waiter is parked.

---

Summary by cubic

Fixes #413 by setting FUTEX_WAITERS and enqueuing the waiter under one hold of the bucket lock in FUTEX_LOCK_PI. Previously a waiter could park on an owned word with the bit clear when the owner released and retook the lock in guest code between the bit set and the enqueue; the owner's next release then made no syscall, so FUTEX_UNLOCK_PI never ran and the waiter hung. pthread-tls-probe hit this.

Two related lost-wake paths are closed the same way:

• The FUTEX_LOCK_PI timeout path clears the bit under the bucket lock, so a waiter that enqueues after reading it can't be left parked without it.
• FUTEX_LOCK_PI takes over a word that has FUTEX_WAITERS but no owner, as Linux does, instead of spinning on the CAS(0->TID) fast path.

Adds cases to tests/test-futex-pi.c: one runs two threads through 2000 contended PI acquisitions with a 2 s deadline on each FUTEX_LOCK_PI call, requiring all 2000 to have contended so a lost wake reports a failure instead of hanging the suite; a second covers takeover of a FUTEX_WAITERS-only word.

^{Written for commit edfd4ea. Summary will update on new commits.}

[jserv]

The timeout exit still breaks this invariant: it unlinks the waiter, drops b->lock, and only then calls futex_clear_waiters_bit. A second waiter that takes the lock in that gap sees FUTEX_WAITERS already set, takes the futex_pi_has_waiters break and parks. The bit is then cleared under it, and the owner's next release is a guest-side CAS that makes no syscall, which is the same lost wake this change fixes. Moving the futex_clear_waiters_bit(word) call above pthread_mutex_unlock(&b->lock) in that branch is safe for the same reason the marking loop is safe under the lock, and it would make this comment true. Otherwise, qualify the comment and keep it on the follow-up list the description already carries.

[xalestar]

Moved futex_clear_waiters_bit(word) above pthread_mutex_unlock(&b->lock) in 097546b, so the check for other waiters and the clear happen under one hold of the bucket lock, and the comment holds as written. There is no test for this path: a stress case that raced an expired FUTEX_LOCK_PI against a parking one passed 16 of 16 runs with only this change reverted, because the gap between the unlock and the clear is a few instructions.

[jserv]

Thank @xalestar for contributing!