Skip to content

Latest commit

 

History

History
385 lines (325 loc) · 20.5 KB

SPEC.md

File metadata and controls

385 lines (325 loc) · 20.5 KB

promptdiff - Design Spec

Status: early implementation. The original spike proved the claude -p mechanism; the current repo now contains a real run command, a scenario-driven compare command, deterministic graders, sandbox setup, timeout and budget bounds, pluggable runners (headless Claude Code and OpenAI-compatible endpoints), and tests for the core local behavior.

1. Problem

AI agent pipelines generate proposed improvements to prompts, skills, and enforcement hooks. The chronic failure mode is that proposed edits are shipped on faith or never shipped at all because there is no cheap proof that the edit changes behavior without damaging scenarios that already worked.

promptdiff is the proof step between "we think this skill edit helps" and "ship it." It does not collect findings, rank recurrence, or manage tuning rounds. Those can live in a larger system. This repo focuses on the eval harness.

2. Eval Altitudes

The eval method should match the change being tested:

Change altitude Eval method Cost
Enforcement hook Deterministic unit test 1 run, near-free
Skill or agent prompt text N-run stochastic A/B over pass rates N x arms x per-run cost

promptdiff targets the second row. It runs one agent with baseline skill text and one agent with proposed skill text against the same scenario set.

3. Runners

Model access goes through a small Runner interface: system prompt + user prompt + bounds in, RunResult (output text, cost, turns, duration, models) out. Each runner declares capabilities:

  • sandboxTools: the runner executes tools inside the sandbox cwd (needed for artifact mode, command graders against agent-written files, and any non-empty tools list)
  • skillRegistry: the runner has a harness-managed skill registry and an appendable default system prompt (needed for install delivery)
  • images: the runner can attach image files to the user message (needed for scenarios with images / run --image)

The engine validates a scenario's demands against the selected runner's capabilities before any paid run, so unsupported combinations fail loudly at startup instead of producing a silently tool-less arm. Runners are selected with --runner <name> or a top-level "runner" scenario field.

3.1 claude-p (default)

Shells out to headless Claude Code:

claude -p "<fixture work item>" \
  --system-prompt-file <assembled prompt file> \
  --model <model> \
  --output-format json \
  --tools <tools> \
  --max-budget-usd <amount> \
  [--max-turns <n>]

The spawned process runs with the per-run sandbox as its actual working directory. Extra directories can be granted with --add-dir, but they are not used as a substitute for sandboxing.

The runner captures final output text, total_cost_usd, turn count, duration, and model usage keys.

A maxTurns cap (scenario field, per-case override, or --max-turns) is passed through as --max-turns. A run that stops at the cap (result subtype error_max_turns, on any exit code) is returned as a normal result flagged exhaustedTurns — the engine records it and scores it as a failed run without invoking the grader, so partial output cannot pass by accident.

3.2 openai

Sends one POST <baseUrl>/chat/completions request to any OpenAI-compatible endpoint (OpenAI, ollama, vLLM, llama.cpp, OpenRouter, ...). The assembled system prompt and the scenario prompt become the system and user messages. --base-url/$OPENAI_BASE_URL select the server; $OPENAI_API_KEY is sent as a bearer token when set (local servers need none).

Text mode only — no tools, no sandbox execution, no skill registry, so it pairs with text graders. These endpoints report tokens, not USD, so pricing is user-declared: scenario pricing (per-model USD per million input/output tokens) or run --price in,out. Priced runs compute real cost from response usage, enforce maxBudgetUsd post-hoc (a completed completion cannot be aborted mid-request; over-budget runs error rather than retry), and fail loudly when a priced endpoint returns no usage. Unpriced runs report 0 — true for local servers — and are bounded only by being single completions. Raw usage is always preserved in the raw result.

Vision: this is the only runner declaring the images capability. Scenario images (or run --image) are embedded as base64 data-URI image_url content parts ahead of the prompt text — jpg/jpeg/png/webp/gif, validated for existence at load time. Two scenario fields tune the endpoint: requestParams merges extra fields into the request body (spread first, so it can never clobber model or messages — pin temperature here to keep pass-rate deltas about the prompt), and retries (default 2) re-attempts transient failures — timeouts, connection errors, HTTP 429/5xx — with exponential backoff and a per-attempt timeout, so one overloaded-endpoint response cannot abort a whole compare. Deterministic failures (other 4xx, malformed responses) fail immediately without burning retries. claude-p is excluded on purpose: the CLI manages its own transport, and retrying non-zero exits would re-run budget aborts.

4. A/B Variable

Two delivery modes, matching two distinct questions about a skill edit:

inline (default) — compliance. Skill variants are inlined into the system prompt instead of being loaded by the Skill tool at runtime:

system prompt = agent body with frontmatter stripped
              + baseline or proposed skill body with frontmatter stripped

That keeps the comparison controlled. The baseline and proposed arms differ by the skill text under test, not by whichever skill version happens to be installed on disk. The limitation is the flip side of the control: an inlined skill is always in context, so inline evals cannot measure whether the skill would have been invoked — and frontmatter (including the description that drives model-invoked triggering) is stripped entirely.

install — invocation. The skill directory is copied, frontmatter intact, into <sandbox>/.claude/skills/<name>, where headless Claude's project-skill discovery registers it. The agent body is delivered with --append-system-prompt rather than --system-prompt-file, because replacing the system prompt would strip the harness machinery (skill registry, Skill tool) that this mode exists to exercise. Arms differ only by which variant directory lands in the per-run sandbox registry. This is the mode for A/B testing description/trigger wording — the "does it fire at all" question. It requires tools and is incompatible with --tools "".

Known contamination source in install mode: same-named user-level skills load in both arms; the installer warns when it detects one.

Model as the A/B variable. The arms can also differ by model, runner, or endpoint instead of (or in addition to) skill text. A shared top-level "skills" array holds the instruction set constant, and the record form of "baseline"/"proposed" carries per-arm "model", "runner", and "baseUrl" (each falling back to the shared top-level value). Each arm is validated against its own runner's capabilities before any paid run, and the summary labels each arm with its model (and runner, when they differ).

Cross-runner confound: a claude-p arm is agentic — tools, multiple turns — while an openai arm is a single completion. Pass rates on text-graded scenarios compare meaningfully, but the mechanics differ and cost does not compare at all (the openai runner reports $0; the summary notes this when exactly one arm is openai).

5. Scenario Format

A compare file defines:

  • agent file
  • baseline skill files, empty to compare against no skill, or a shared skill set both arms inherit
  • proposed skill files
  • model (shared, or per arm for model comparisons)
  • run count
  • sandbox root and optional seed workspace
  • target and regression scenarios
  • deterministic graders

The first scenario defaults to target; later scenarios default to regression. Explicit kind is preferred.

Target assertions:

  1. baseline must not fully pass, otherwise the gap was not reproduced
  2. proposed must improve the target pass rate

Regression assertions:

  1. proposed must not fall below baseline pass rate
  2. proposed must not produce the artifact in a smaller share of its runs than baseline (see No artifact)

Every case also carries a two-tailed Fisher exact p for its pass/fail table; the summary labels deltas with p > 0.05 as explainable by sampling noise. This is a label, not a gate — assertions and exit codes are unchanged, but a receipt now says how thin its evidence is. Scenarios may declare productionModel; arms testing a different model are flagged in the summary. --report ndjson --report-out <file> appends per-scenario records (arms, rates, cost, sampling p, rendered-prompt sha256 hashes) as append-only run history. --receipts <dir> writes per-scenario receipt files (overwritten each run) recording the content hash of every prompt file tested plus the verdict — content addressing instead of hand-maintained prompt_version strings, so a consuming repo's CI can require a passing receipt for each shipped prompt's current hash. Reports are history; receipts are current state. --raw-out <dir> (compare and measure) additionally writes each run's full runner result JSON — token usage categories, modelUsage, subtype — one file per completed run, for token-level analysis that the digested summaries cannot support; files land as runs finish, so a partial invocation still leaves its completed records. --transcript-out <dir> (compare and measure, streamEvents runners only) additionally captures the runner's per-event stream as NDJSON, one <scenario>_<arm>_<n>.stream.jsonl per run written event by event — per-turn usage and tool calls, which the aggregate result object cannot carry. A run killed mid-stream keeps its partial file.

compare scenarios assert nothing: they exist to report both arms' pass rates and the delta, for comparisons (typically model-vs-model) where neither direction is claimed in advance.

Measure mode. promptdiff measure runs a single instruction set through the same scenario machinery and reports per-case pass rates with no delta and no assertions — the "characterize before you change" half of prompt testing. It loads the compare format with a single-arm allowance (skills or baselineSkills alone suffices) and always exits 0 on completion. Identical-arm compares are not a substitute: at small n they emit directional verdicts from pure sampling noise.

Template rendering. render.vars — top-level and/or per scenario, with the scenario winning per var — binds {{name}} placeholders across the agent body, inlined skill text, and scenario prompts. This is what lets scenarios point at production prompt files (which are full of pipeline placeholders) instead of hand-rendered copies that drift. Values resolve file-first relative to the scenario file and are read at load time; anything that isn't an existing file is a literal. Rendering is opt-in and strict: with any render block present, unbound placeholders abort before any paid run; without one, braces pass through untouched. Substituted content is never re-scanned, so fixtures containing braces neither expand recursively nor false-positive the unbound check. Incompatible with install delivery, which copies skill files verbatim.

6. Grading

Prefer deterministic graders over LLM judges.

Text graders inspect the run's final output. JSON graders parse that output — the last balanced JSON value when prose surrounds it, since reasoning models narrate around their answer — and check assert path assertions of the form <path> <op> <literal> (ops == != > >= < <= contains; [*] is existential: any element may satisfy the assertion, including for !=). Assertion grammar is validated at config load, before any paid run; missing paths and type mismatches at grade time fail the assertion with a message, never throw. Like text graders, json graders demand no sandbox tools and work on every runner. Command graders run inside the sandbox after the model invocation and check the exit code of a local command such as bun test, go test ./..., or a fixture-specific script. The final output is also written into the sandbox and exposed to command graders as $PROMPTDIFF_OUTPUT_FILE, so completion-only runs can be command-graded (scenario mode: "text" keeps the tools demand at zero).

Grader files. The command contract (run in the sandbox, exit code is the verdict, stderr is the diagnostics) stays the base layer; any executable in any language can grade. A grader file is a layer on top of it for the common case of checking one artifact from TS/JS. The file's default export is grade(artifactPath, { name: ({ result }) => { ... } }), and a scenario selects one grader with { "file": "./grade.eval.ts", "name": "..." } (no type needed; "type": "file" is accepted). result offers assert(condition, message, context?) for semantic checks, json(), text, path, and the literal checks shouldHave, shouldNotHave, and shouldMatch. assert is the primary check in docs and examples on purpose: the strongest graders in use are semantic (range coverage, calibrated regex rulers), and substring sugar alone would push authors toward brittle string matching.

The engine compiles a file grader to a command grader: <bun> <promptdiff> grade --file <abs path> --name <name>, run in the same sandbox cwd with the same timeout handling. promptdiff grade resolves the artifact against its cwd (the sandbox, or the grader's cwd inside it), reads it, runs the named grader, and collects every check rather than stopping at the first failure. It exits 0 when all checks pass, 1 when any fail (each failure on one stderr line with its context, then a one-line verdict that becomes the run's grade message), 77 when the artifact does not exist, and 2 when the file or name is unusable. A grader that makes no checks fails. An exception inside a grader is recorded as one more failure. Under promptdiff grade, the specifiers @theaiteam/promptdiff and promptdiff resolve to the running copy through a Bun runtime plugin, so a grader file needs no local install and cannot drift from the runner's version. Scenario loading checks each file and name in a child process (promptdiff grade --list) before any paid run. The baseline cache key includes the grader file's content hash (not the files it imports).

No artifact. A grader that finds nothing to grade reports no-artifact, a third outcome beside pass and assertion-failed. File graders report it when the artifact path does not exist. Command graders report it by exiting 77 (also exported as $PROMPTDIFF_NO_ARTIFACT_EXIT_CODE), unless their expectExitCode is 77, which keeps its old meaning. No existing command grader changes between pass and fail: an exit 77 was a failure before and is still not a pass. A no-artifact run is never a pass, so a negative check cannot pass on an agent that produced nothing, and it is not counted as a failure, so reproduction rates are not inflated by runs with no signal. ArmSummary.noArtifact counts these runs; passRate is passes over graded runs (totalRuns - noArtifact), and Fisher's p uses the graded counts. Summaries print 4/4 pass (100%), 1 no-artifact and list each such run. compare adds a NOTE with both arms' no-artifact counts whenever either is non-zero, and fails target and regression assertions for an arm with no graded run at all, which has no pass rate to compare. A regression assertion also fails when proposed produces the artifact in a smaller share of its runs than baseline, since excluding those runs from the pass rate would otherwise hide a change that stops the agent writing the artifact.

LLM judges are implemented, WITH mandatory calibration, for judgments that cannot be expressed as local checks (semantic and style rubrics that regex both under- and over-catches). The design premise is that an uncalibrated judge is worse than the regex it replaces — same wrongness, more confidence, higher cost — so judge graders refuse to grade until proven:

  • A judge grader is { "type": "judge", "rubric": <markdown file>, "model": <judge model>, "runner": <claude-p|openai>, "baseUrl"?, "minAccuracy"? (default 0.9) }. The rubric becomes the judge's system prompt verbatim, plus a fixed harness instruction demanding one JSON verdict object; the graded output is the user prompt. The judge model is explicit on purpose and never defaults to the arm's model (self-grading bias). The last balanced JSON object in the reply wins (reasoning models add prose); no valid verdict fails the graded run, never passes it.
  • Labeled fixtures live in a sibling directory, <rubric-stem>.fixtures/pass/*.md (outputs the judge must call clean) and .../fail/*.md (outputs it must flag). promptdiff calibrate runs the judge over every fixture and writes <rubric>.calibration.json next to the rubric — committable, keyed by rubric content hash.
  • The gate: at compare/measure startup, before any paid run, every judge grader must have a calibration record that exists, matches the current rubric sha256 and the spec's model/runner, and clears minAccuracy on BOTH classes. Per-class bars on purpose: a judge that passes everything is 100% on the pass class and 0% on the fail class; overall accuracy hides it.
  • v1 judges are absolute (grade one output against the rubric); pairwise comparison is deferred. The calibration gate is the escalation signal: a rubric whose absolute judge cannot clear the bar is the case for pairwise.

A judge adds one billed model call per graded run (its cost is added to the run's cost), so deterministic graders remain the default choice.

7. Architecture

promptdiff          # Bun executable shim
src/cli.ts          # command parsing and user-facing orchestration
src/args.ts         # strict local flag parser
src/prompt.ts       # frontmatter stripping and prompt assembly
src/index.ts        # package entry for grader files (exports grade)
src/grade.ts        # grader-file authoring API and check collection
src/runner/         # provider-coupled runner code
src/engine/         # compare loop, config loading, graders, sandbox lifecycle
test/               # Bun tests

The compare engine depends only on the runner interface and its declared capabilities; it never imports provider SDKs or provider-specific code. src/runner/index.ts is the registry that maps runner names to implementations.

Baseline arms are usually frozen by construction, so compare --cache (opt-in; results under --cache-dir, default .promptdiff/cache) reuses recorded baseline-arm results instead of re-paying for them on every iteration of the proposed prompt. src/engine/cache.ts keys each case on a sha256 over everything that could change the outcome — rendered baseline system prompt, scenario prompt, baseline model/runner/baseUrl, run count, tools/mode/delivery, grader spec, image contents, and a deterministic tree hash of the sandbox seed (plus the baseline skill trees under install delivery, where skill text never enters the system prompt). Hits are real recorded ArmSummaries, marked (cached) on the summary; the proposed arm always runs fresh. Deleting the cache directory busts it.

8. Current Limitations

  • Pass-rate assertions are intentionally simple. They are good enough to catch decisive effects at small N, but there is no statistical test yet.
  • Scenario authoring is manual JSON. A future tuning loop should generate these files from accepted findings.
  • Command graders run trusted local commands from scenario files, and loading a scenario imports its grader files. Do not run untrusted scenario files.
  • The cache key hashes a grader file but not modules it imports; editing a helper module alone does not bust the baseline cache.
  • The openai runner is a single completion per run: no tool use, so it can only answer text-graded questions. Tool-using evals on non-Claude models would need an agentic runner (e.g. wrapping another agent CLI).
  • Fixture coverage remains the bottleneck. Missing fixtures mean missing regression protection.

9. Decisions

  • Standalone Bun repo rather than embedding in a larger mission runner.
  • Claude Code -p runner first; OpenAI-compatible endpoints second. Runner capabilities are validated up front rather than degraded silently.
  • Inline skills for variant control.
  • Deterministic graders first; LLM judges only behind a mandatory calibration gate (absolute mode in v1, pairwise deferred).
  • Per-run sandbox cwd, timeout, and budget bounds are mandatory for paid runs.