Repository navigation
tests: replace wall-clock correctness assertions and sleep-based race coordination #1680
Description
Problem
Some correctness tests infer synchronization, nonblocking behavior, or concurrent overlap from elapsed wall-clock time or fixed sleeps. Host scheduling, driver initialization, and runner load can cause false failures; a sleeping/descheduled worker can also let a broken implementation pass without exercising the claimed interleaving.
The motivating failure is #1604's dropping_in_flight_future_returns_promptly_and_retires_owners: GPU CI log. On head eddfe1495, dropping took 200.937405 ms against a 100 ms assertion while a helper opened the GPU gate after 200 ms. The unchanged test passed 30 consecutive isolated runs on a local RTX 5090 (driver 610.43.02). That demonstrates differing outcomes, not that the CI failure is harmless: its cause remains unestablished.
This is follow-up test-hardening work associated with #1461/#1604 and release tracking #1606. It does not request an implementation change in the current host-interop PRs or waive their CI failures.
Initial inventory
Audited revision: ede42672a6f68256b92a9403d6dc19955a3e460b (shared host stack). The following are concrete candidates; this is a starting inventory, not a claim that every timer or listed test has produced a flaky failure.
| Pattern | Location | What to address |
|---|---|---|
| Elapsed-time assertions | cuda-async/tests/drop_in_flight.rs | dropping_in_flight_future_returns_promptly_and_retires_owners: 200 ms automatic gate release, drop must take <100 ms. dropping_unpolled_future_submits_nothing: drop must take <50 ms. |
| Elapsed-time assertions | cuda-core/tests/stream_query.rs | 500 ms sleeping callback; query must return in <250 ms and while the callback remains unfinished. |
| Elapsed-time assertions | cuda-core/tests/shared_allocation.rs | Buffer/stream drop must take <2 seconds while a gate holds GPU work. Replace elapsed-time proof with observed return-before-release and retained-owner state. |
| Elapsed-time assertions | cutile-rs/cutile/tests/gpu/submission_lifetimes.rs | async_on success/error and partial-submission error/panic paths impose <1 second assertions despite already having explicit gates and owner checks. |
| Timed non-completion as evidence | cuda-core/tests/device_buffer.rs | observe_completion / finish_gated_worker treat no message within 100 ms as proof that initialization/free is blocked. A delayed worker can hide a missing fence. Also audit the 1-second completion assertions. |
| Sleep-based overlap | cuda-async/tests/execute_once.rs | slow_counted_op sleeps 150 ms to supposedly guarantee overlapping executors; shared/unzip tests should observe the intended interleaving explicitly. |
| Sleep-based overlap | cuda-async/tests/concurrent_capture.rs | 50 ms sleeps attempt overlapping captures; 100 ms attempts to keep capture active while another thread creates a stream. Confirm both participants reached the required state. |
| Sleep-based state transitions | cuda-async/tests/reactor_correctness.rs | 200 ms sleep assumes cancelled completion slots have recycled; 2 ms gaps assume the scanner parks. Observe those states instead of assuming sleep caused them. |
| Additional CPU concurrency/retry tests | cuda-oxide/crates/cuda-host/src/ltoir_cache.rs; cuda-oxide/crates/cuda-artifact-finalizer/src/ptx.rs | Cache single-flight test uses a 30 ms build delay to encourage contention. ETXTBSY retry test releases a write descriptor after 30 ms, which can miss the retry path or exhaust its budget under scheduling delay. |
| Additional CuTe validation fixtures | cuda-oxide/cute-rs-preview/models/qwen_decoder_cute/src/model.rs; cuda-oxide/cute-rs-preview/models/qwen_decoder_cute/src/decode_linear_grouped.rs | 100 ms receive timeouts are used as evidence that a drop/projection is blocked behind a foreign producer. Also audit the delayed-producer sleeps in talker_decode_validation.rs:220 and predictor_graph_validation.rs:326. |
Direction and acceptance
- Audit correctness tests across the host, Tile, and SIMT workspaces for elapsed-time assertions, timed non-completion observations, and sleeps used to claim a particular interleaving; extend the inventory and account for each candidate.
- Replace fixed-delay ordering with explicit handshakes, channels/barriers, observable completion/retirement states, or test hooks. Establish that the operation actually reached the point under test; a thread-start notification alone is not proof it attempted the operation.
- For nonblocking drop/query, hold GPU work behind a controlled gate, observe the host operation returning while the gate remains closed and owners remain retained, then release the gate and verify eventual cleanup. Use hooks/model tests where black-box observation cannot establish the needed ordering.
- Retain generous watchdogs to stop hangs and release gates/join workers on failures. A watchdog expiry should report the observed states and fail the run; it must not silently open the gate and allow a test to pass. Timeouts are bounds on a test run, not the evidence that the behavior is correct.
- Preserve detection of actual blocking, premature resource release, missing fences, duplicate execution, and missed wakeups. Validate the replacements against deliberately broken implementations or targeted negative controls, and under varied host scheduling/load.
- Keep performance measurements in explicitly identified benchmarks. Do not convert ordinary correctness tests into latency benchmarks or merely increase thresholds/add retries to hide a failure.
Not every existing timer needs replacement: bounded waits for eventual completion and deadlock watchdogs are useful, as are benchmark timers and diagnostic timing. For example, cutile/tests/gpu/warmup.rs::multi_thread_dedup_timing_evidence already asserts the compilation count, with timings only printed; that is not an elapsed-time correctness assertion. Review deadline-only cleanup polling (including device_buffer_leaks.rs and cuda-host/tests/shared_launch_lifetimes.rs) separately, preserving its actual state checks.
Additional integration occurrence
On #1613 head e16b73a9c, the GPU run failed dropping_in_flight_future_returns_promptly_and_retires_owners at 200.853427 ms against the 100 ms assertion. The same runtime/test code passed GPU CI in #1684. The final integration candidate d2c8120bf changes only ancestry and a duplicate changelog note; the exact test passed 30/30 isolated local runs there. The CI cause remains unresolved; isolated success does not establish whether scheduling or actual blocking caused the failure. No thresholds or test selection were changed.
The full GPU CI suite passed on final #1613 candidate d2c8120bf35ebb1fa1e6b9bb0b240f425a6fa91b, including the unchanged drop-timing assertion. All 76 applicable checks are green. This confirms the candidate run passed, but does not identify the cause of the earlier intermittent failure; this test-design issue remains open.
Activity
Unify the async host runtime and remove cuda_async::simt (#1604)