Skip to content

test(studio): measure render drift against a producer frame in the edit accuracy bench - #4764

Draft
miguel-heygen wants to merge 4 commits into
test/studio-edit-accuracy-benchfrom
test/studio-edit-accuracy-render-drift
Draft

miguel-heygen wants to merge 4 commits into
test/studio-edit-accuracy-benchfrom
test/studio-edit-accuracy-render-drift

Conversation

@miguel-heygen

@miguel-heygen miguel-heygen commented Sep 30, 2026 •

Copy link
Copy Markdown
Collaborator

What

Adds render drift to the edit accuracy benchmark, its fifth metric. After each case's edit and a fresh Studio load, the producer renders the saved file at the same playhead, and the benchmark compares where the element lands in that frame with where the reloaded preview shows it. baseline.json gains the render column. It comes from the same full run on main 14e8700 as #4760's baseline: render passes 618/660.

Stacked on the edit accuracy benchmark PR (#4760); review that one first.

Why

A preview can show one thing while the export shows another. The other metrics only look at Studio, so they cannot catch a case where the preview and the rendered video disagree about the same file.

Related work

Refs #4760 (the benchmark this extends). The CI gate follows in its own PR.

How

  • The producer frame comes from the engine's own capture session (createCaptureSession + captureFrameToBuffer), the path scripts/preview-capture.ts uses. It runs in the same pinned headless shell Studio is driven in, with JPEG output. PNG is the engine's transparent mode.
  • The element's box is found by its flat fixture colour over the whole frame:
    • coverage per pixel comes from luminance, which JPEG keeps at full resolution;
    • area moments give the centre and both sides, corrected by 1/12 px² for pixel coverage;
    • so a rotated or cropped rectangle is measured the same way as an axis-aligned one.
  • The reloaded preview is measured once it has held still for 1 s, as every snapshot in test(studio): an edit accuracy benchmark for manual editing #4760 is. If the reload never settles, render fails with it, flagged unsettled.
  • If the producer cannot render the case, only render fails: render is null and renderError holds the message. The case's other metrics still count, and the table counts these cases.
  • Rotated elements compare axis-aligned boxes. The producer page's DOM rect is recorded as a diagnostic only, because it ignores clip-path.
  • A composition with no GSAP never registers a timeline, so the capture session would wait its full 45 s playerReadyTimeout. The bench passes 10 s instead. The frame is identical: the measured box matches to every printed digit.

Test plan

  • A producer failure injected into two cases fails render alone (renderError). Undo, drop and reload still score, and there is no harness error.
  • Pixel box against the producer's own DOM rect on an unrotated, uncropped case: within 0.01 px.
  • A 1 px shift injected into the producer page reads as render drift 1.02 (clean 0.02). A 3 px shift injected into the reloaded preview reads as 3.01.
  • Restacked on the settle fix: a 4-case run gives render 0.03 px on the clean cases and 50.0 px on resize-hold-center-r0-root-z100, with no harness errors.
  • Full grid on main (before the settle fix): render passes 618/660. Worst is 110.05 px (resize-hold-pct-r30-root-z100): Studio's preview does not show the size the producer renders from the same file. The resize writes tl.set(... {width, height}, 0) into a zero-length timeline.
  • oxlint, oxfmt, fallow audit --base origin/main and the comment ratchet pass.
  • Unit tests added/updated. None: the injections above prove the metric.
  • Comments follow CONTRIBUTING.md "Comments".

Before

Without render drift, the benchmark saw only Studio. This is the reloaded Studio preview for the worst case, which the Studio-only metrics accept.

Reloaded Studio preview of the worst render drift case

After

The producer frame for the same saved file, at the same playhead, shows the element resized. Render drift now reports the 110 px gap, and the table has the render column.

Producer frame of the same saved file

Edit accuracy baseline table with render drift

@miguel-heygen
miguel-heygen force-pushed the test/studio-edit-accuracy-render-drift branch 2 times, most recently from d2b3984 to adc50c8 Compare September 30, 2026 12:53
@miguel-heygen
miguel-heygen force-pushed the test/studio-edit-accuracy-render-drift branch from adc50c8 to 1f83860 Compare September 30, 2026 13:16
miguel-heygen added a commit that referenced this pull request Sep 30, 2026
… settle fix

Full grid on main 14e8700 with the settle fix, press jump and the smoothness control: 70/660 pass, 314/660 pass everything but smoothness, 0 harness errors, 0 unsettled. Scored without render, which #4764 adds.
@miguel-heygen
miguel-heygen force-pushed the test/studio-edit-accuracy-render-drift branch from 1f83860 to 90fbb68 Compare September 30, 2026 15:06
…it accuracy bench

Finds the target's box in a producer-captured frame by its flat colour, with sub-pixel edges from area moments, and compares it with the reloaded preview. The baseline gains the render column from the same full run on main.
… on main

Same full run on main 14e8700 as #4760's baseline, with the render column: render passes 618/660, 70/660 pass, 314/660 pass everything but smoothness.
…a case

A producer capture failure used to turn the case into a harness error that wiped every accuracy metric. It now sets render to null with renderError, fails render alone, and is counted in the table.
@miguel-heygen
miguel-heygen force-pushed the test/studio-edit-accuracy-render-drift branch from 90fbb68 to 70a12df Compare September 30, 2026 16:22

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant