Skip to content

feat(web): AgentPrism trace viewer for eval runs - #161

Open
JesseVent wants to merge 1 commit into
supabase:mainfrom
JesseVent:trace-viewer-clean
Open

feat(web): AgentPrism trace viewer for eval runs#161
JesseVent wants to merge 1 commit into
supabase:mainfrom
JesseVent:trace-viewer-clean

Conversation

@JesseVent

@JesseVent JesseVent commented Aug 3, 2026

Copy link
Copy Markdown

Summary

Adds a trace viewer to the web results app: every result row gets a "View span tree" link that opens the full span tree for that run — system/user messages, every tool call with its input/output, assistant turns, and check results — with per-span duration, token usage, and status.

What's added

Core (packages/core)

  • trace-viewer.ts — adapter that turns an EvalResult (+ its raw transcript) into an AgentPrism span tree (evalResultToTraceSpans), plus tests (trace-viewer.test.ts).
  • transcript/types.ts / parsers/adapt.ts — a TokenUsage type and usage?/ts? fields threaded through TranscriptEvent/TranscriptPart, so per-turn token counts and timestamps survive the transcript → trace pipeline instead of only being available as run-level aggregates.
  • agents/{claude-code,codex,opencode}/parser.ts — each agent parser now extracts real per-turn usage (extractUsage/extractTurnUsage) from its own transcript format.
  • index.ts — re-exports the above, plus per-step usage/timestamp capture in the AI SDK agent path.

Export (apps/framework/scripts/export-results.ts)

  • Alongside the aggregate eval-results.json, writes one apps/web/src/data/traces/<evalId>.json per eval via evalResultToTraceSpans, preferring a run that actually recorded a transcript over one that didn't (so the viewer shows a real span tree, not an empty no-skills run). Lazy/one-per-file so the aggregate bundle stays lean — the web app fetches a trace only when its row is opened. New --no-traces flag skips writing them.

Web (apps/web)

  • components/agent-prism/** — the AgentPrism component set (span cards, tree view, details view with Attributes/In-Out/Raw tabs, theme tokens) vendored into the repo.
  • components/trace-panel.tsx — the integration point: fetches a trace's JSON lazily, revives Date fields (JSON round-trips them as strings, which breaks AgentPrism's duration math), and renders TraceViewer full-screen with an Escape-to-close overlay.
  • components/results/eval-details.tsx — adds the "View span tree" link.
  • Tool call input/output defaults to a human-readable "Plain" view with real line breaks (multi-line shell commands/output are unreadable as JSON's escaped \n form); "JSON" stays available as a tab for exact, copy-pasteable output.
  • Trimmed the vendored TraceViewer to fit how we actually use it: it's built for a multi-trace list (pick between several loaded traces), but we only ever open one trace at a time, so the always-one-item trace-list sidebar (and, on mobile, its matching "back to list" dead-end button) is hidden — the span tree gets the full width instead.

How to try it

pnpm install
pnpm --filter web dev

Open the results table, click any row, click "View span tree".

image

Agent Prism Trace Visualiser

image ## Notes
  • Rebased cleanly onto current main: kept main's regenerated eval-results.json/regression-eval-results.json instead of the stale snapshot this originally carried, and dropped everything that had leaked in from an unrelated, not-yet-merged skill-trigger-suite branch (trigger trace fixtures, EXPERIMENT_SUITES/url-state.ts entries, a README section) — this PR is the viewer only.
  • pnpm --filter core test (131/131), pnpm --filter web run typecheck, and pnpm --filter web run build all pass. One pre-existing eval-results.test.ts data-consistency assertion fails on this branch — confirmed it fails identically on plain main today, unrelated to this change.

Adds an AgentPrism-based trace visualization UI (apps/web) and the core
adapter that turns any EvalResult into a trace span tree.

Rebased cleanly onto current main:
- Kept main's regenerated regression-eval-results.json and eval-results.json
  instead of the stale snapshot this carried, since main has since refreshed
  both independently.
- Dropped the "trigger" experiment-suite trace fixtures and the "trigger"
  entries in EXPERIMENT_SUITES/url-state, since that suite isn't merged into
  main's core schema yet and lands separately.
- Merged the adjacent (non-conflicting) codex parser test additions from
  main's supabase#130 alongside this branch's usage-tracking tests.

Claude-Session: https://claude.ai/code/session_01WaER4ZeRLmDSwC5GFzkeFQ
@JesseVent
JesseVent requested a review from a team August 3, 2026 15:54
@vercel

vercel Bot commented Aug 3, 2026

Copy link
Copy Markdown

@JesseVent is attempting to deploy a commit to the Supabase Team on Vercel.

A member of the Team first needs to authorize it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant