test(studio): an edit accuracy benchmark for manual editing

[miguel-heygen]

What

A benchmark that measures how accurately manual editing in Studio follows the person doing it. One command drives real pointer and keyboard gestures in the built CLI Studio, one generated project per case, and scores each case in composition pixels.

This first part ships the grid, the project generator, the runner and five of the six metrics, plus baseline.json from a full run on main. Render drift (the check against a producer frame) is the stacked follow-up. The CI gate is a separate later PR.

bun run --cwd packages/studio test:edit-accuracy -- --grid full --jobs 4

Flags: --grid full|pr, --shard i/n, --jobs N, --filter <regex on case id>, --out <dir>, --port <first>, --cli <cli.js>.

Why

Nothing measured whether a move, resize, rotate, crop or nudge lands where the pointer put it, survives undo and a reload, or stays smooth. Manual edit fixes had no score to hold them to, and the benchmark is that score. Later edit PRs should raise it and never lower it.

Related work

Refs the stacked render drift PR (to follow). The CI gate and ratchet over baseline.json come next, in their own PR.

How

Grid (660 cases)

• Every case is one flat-coloured element in a fresh tmp copy of a generated project, so no checked-in fixture is ever edited.
• GSAP: none, an x/y tween spanning the playhead, or a gsap.set hold.
• Placement: stylesheet translate in px, translate in %, left/top: 50% with a -50% translate, or GSAP xPercent/yPercent: -50. The last is not applicable without GSAP.
• Rotation 0 or 30deg; root or nested composition; preview zoom 50%, 100% or 200% through Studio's own Ctrl+wheel zoom.
• Gestures:
  • move;
  • corner resize, dragged along the diagonal;
  • rotate;
  • crop of the right edge;
  • arrow nudge ×5.
• Undo and redo run in every case, before any reload.
• Studio has corner handles only (the edge strips crop), so there is no edge-resize gesture.

Measurement

• Positions come from the element's rendered quad (CDP DOM.getContentQuads), mapped to composition px through the root composition's own quad.
• Pointer steps run one per animation frame, in lockstep with the measurement.
• Studio is the built CLI, driven by real input events. WebMCP studio_seek is used only to seek.
• One Chrome for everything: the engine's pinned headless shell.
• Snapping is turned off through Studio's own toggle, because a snap moves the box off the pointer on purpose.

Metrics (all in composition px)

1. Tracking: max per-frame distance between the grabbed point and the pointer.
2. Drop jump: last drag frame against the settled commit.
3. Reload drift: the commit against a fresh Studio load.
4. Undo fidelity: undo is byte-identical to the file before the gesture, redo to the committed file, and each box lands within 0.5 px.
5. Smoothness: frame-interval p95, read from the rAF callback's own clock (Chrome stamps a late frame with the vsync it missed, which hides a stall), plus long tasks.

A case passes at ≤ 0.5 px on 1-3, byte-identical undo and redo with the box ≤ 0.5 px, and frame p95 ≤ 16.7 ms. The table also counts cases passing everything except smoothness, because smoothness gating is still an open decision.

Test plan

• Full grid on main ae91433: 660 cases in 44 min with 4 jobs. Five cases hit a harness error: Studio swapped the preview iframe mid-read, which the runner now survives. They were rerun and spliced into baseline.json.
• Every metric catches an injected error:

Injection	Metric	Clean	Injected
move commit +3 px (Studio build)	drop	0.00	3.02
every other pointer move dropped	tracking	0.74	6.13
preview shifted 3 px after reload	reload	0.00	3.02
undo writes one extra byte (server build)	undo bytes	identical	differ
40 ms main-thread stall per move	smooth p95	17.1 ms	59.2 ms

• Noise floor, 10 cases × 3 runs:
  • tracking, drop, reload and undo repeat to within 0.01 px;
  • two nested no-GSAP cases are bimodal (Studio lands one of two outcomes, 0 or 108 px);
  • smoothness p95 ranges 16.9 to 47.4 ms under box load.
• The refactored runner reproduces the full run's values to 0.01 px on 9 deterministic cases.
• oxlint, oxfmt, fallow audit --base origin/main and the comment ratchet pass.
• Unit tests added/updated. None: the benchmark itself is the test, and the injections above prove it.
• Comments follow CONTRIBUTING.md "Comments".

Baseline on main: 0/660 pass all five checks here; 304/660 pass everything but smoothness.

Metric	Pass	Worst (px, or ms for smooth)	Worst case
tracking	466	1.0	resize-none-pct-r0-nested-z50
drop	557	222.6	resize-none-pct-r0-root-z50
reload	579	216.6	resize-none-pct-r30-root-z100
undo	517	199.0	crop-tween-center-r30-root-z50
smooth	0	74.9	move-hold-center-r30-nested-z100

What the failures are:

• Every tracking failure is at most 1 px: Studio rounds the live drag position and size to whole composition px.
• The large drop and reload errors are resizes of %-translated elements.
• The undo worst is the stylesheet translate applied again on top of GSAP's folded x/y after an undo.
• Smoothness fails on frame jitter and box load.

Before

There was no benchmark. This is one of its generated case projects in Studio before any gesture, which is where every case starts.

After

The baseline table from the full run on main, then two of the worst cases as the bench captured them: a resize that jumped 222.6 px on release, and a crop whose undo left the element displaced.