test(studio): measure render drift against a producer frame in the edit accuracy bench - #4764
Draft
miguel-heygen wants to merge 4 commits into
Draft
miguel-heygen wants to merge 4 commits into
miguel-heygen wants to merge 4 commits into
Conversation
5 of 6 tasks
miguel-heygen
force-pushed
the
test/studio-edit-accuracy-render-drift
branch
2 times, most recently
from
September 30, 2026 12:53
d2b3984 to
adc50c8
Compare
6 of 7 tasks
miguel-heygen
force-pushed
the
test/studio-edit-accuracy-render-drift
branch
from
September 30, 2026 13:16
adc50c8 to
1f83860
Compare
miguel-heygen
force-pushed
the
test/studio-edit-accuracy-render-drift
branch
from
September 30, 2026 15:06
1f83860 to
90fbb68
Compare
…it accuracy bench Finds the target's box in a producer-captured frame by its flat colour, with sub-pixel edges from area moments, and compares it with the reloaded preview. The baseline gains the render column from the same full run on main.
…a case A producer capture failure used to turn the case into a harness error that wiped every accuracy metric. It now sets render to null with renderError, fails render alone, and is counted in the table.
miguel-heygen
force-pushed
the
test/studio-edit-accuracy-render-drift
branch
from
September 30, 2026 16:22
90fbb68 to
70a12df
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds render drift to the edit accuracy benchmark, its fifth metric. After each case's edit and a fresh Studio load, the producer renders the saved file at the same playhead, and the benchmark compares where the element lands in that frame with where the reloaded preview shows it.
baseline.jsongains the render column. It comes from the same full run on main 14e8700 as #4760's baseline: render passes 618/660.Stacked on the edit accuracy benchmark PR (#4760); review that one first.
Why
A preview can show one thing while the export shows another. The other metrics only look at Studio, so they cannot catch a case where the preview and the rendered video disagree about the same file.
Related work
Refs #4760 (the benchmark this extends). The CI gate follows in its own PR.
How
createCaptureSession+captureFrameToBuffer), the pathscripts/preview-capture.tsuses. It runs in the same pinned headless shell Studio is driven in, with JPEG output. PNG is the engine's transparent mode.unsettled.renderis null andrenderErrorholds the message. The case's other metrics still count, and the table counts these cases.clip-path.playerReadyTimeout. The bench passes 10 s instead. The frame is identical: the measured box matches to every printed digit.Test plan
renderError). Undo, drop and reload still score, and there is no harness error.tl.set(... {width, height}, 0)into a zero-length timeline.fallow audit --base origin/mainand the comment ratchet pass.Before
Without render drift, the benchmark saw only Studio. This is the reloaded Studio preview for the worst case, which the Studio-only metrics accept.
After
The producer frame for the same saved file, at the same playhead, shows the element resized. Render drift now reports the 110 px gap, and the table has the render column.