Skip to content

ci(verify): run the wall-clock crash and realtime playback tests first and alone - #2983

Merged
richiemcilroy merged 1 commit into
mainfrom
ci/verify-wall-clock-tests-alone
Oct 23, 2026
Merged

richiemcilroy merged 1 commit into
mainfrom
ci/verify-wall-clock-tests-alone

Conversation

@richiemcilroy

@richiemcilroy richiemcilroy commented Oct 23, 2026 •

Copy link
Copy Markdown
Member

Two macOS Engine verify tests measure against the wall clock, and they fail when they share the runner with the rest of the suite:

  • crash_safety hands frames to a real-time VideoToolbox session on a 30 fps schedule. Its crash/equality case lost 3 to 7 consecutive frames in the in-process arm (for example [100, 101, 102, 213, 214, 215, 216]).
  • realtime_loop counts the frames playback presents in real time. realtime30-from20/avf presented 65% of its frames against a 75% floor.

Both pass when run.sh reruns them alone. Every crash/equality failure in the last 80 Engine verify runs has the same pattern: 6 of 6 failed in the shared run and passed in the solo rerun. The solo reruns were also 7 to 19 s faster. realtime_loop failed shared and passed solo in #2953. In the shared runs, the seek storms, chaos_recording and the export jitter tests were running at the same time, and they load the same 6 vCPUs and media engine.

This PR gives both tests threads-required = "num-test-threads" and priority = 100 in the default and ci nextest profiles. They start first in their shard and nothing else runs alongside them. The thresholds and checks are unchanged, and no cases are marked as expected failures.

Cost: about 140 s (crash_safety) and 19 s (realtime_loop) of serial time, in whichever shard each one lands in. Both used to overlap with other tests.

The other recurring macOS failure, crash/recorder-kill/*/static-screen (fragment-span), was fixed in the harness by #2785.

On this PR's macOS run (fast tier, on top of #2785), nextest started crash_safety first (1 of 61, 132.8 s, pass) and realtime_loop second (18.3 s, pass) before any other test, and macos / build and test passed.

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge.

Summary

Runs crash_safety and realtime_loop first and alone within each nextest shard.

  • The two wall-clock tests run first and alone.

Reviews (2) · Last reviewed commit: "ci(verify): run the wall-clock crash and..." · Reviewed by Greptile

@vercel

vercel Bot commented Oct 23, 2026

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
cap-status Ready Ready Preview Oct 23, 2026 12:22pm UTC

@richiemcilroy

Copy link
Copy Markdown
Member Author

@greptileai review

@richiemcilroy
richiemcilroy marked this pull request as ready for review October 23, 2026 17:10
@richiemcilroy
richiemcilroy merged commit 0b825e0 into main Oct 23, 2026
42 checks passed

This branch was successfully deployed

2 active deployments
Preview – cap-web — 40b199bf Deployed Oct 23, 2026 by vercel[bot]
Preview – cap-status — 40b199bf Deployed Oct 23, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant