feat(harness): add Harness as Code package - #89
Draft
phodal wants to merge 22 commits into
Draft
Conversation
Add a Langium-based Harness as Code compiler, versioned IR resolver, and Qoder/Pi SDK executors with honest materialization semantics and fail-closed host and runtime handling. Wire the package into the root workspace and cross-platform CI, and harden package verification. Implements docs/specs/2026-08-14-harness-as-code-v0.1.md. Validated with npm run check, focused documentation-link tests, npm audit, a cold package install, and a native Qoder SDK smoke. Co-authored-by: Codex (GPT 5.6 Sol) <codex@openai.com>
Add a package-local generation skill, compact language contract, and deterministic compiler/resolver validator, and include them in the @qoder-ai/harness tarball. Cover valid, invalid, missing-composition, and honest degradation behavior. Implements docs/specs/2026-08-15-harness-dsl-generation-skill.md. Validated with the 42-test package suite, skill validation, agent asset lint, documentation-link checks, npm pack inspection, a cold package install, and an isolated fresh-agent forward test. Co-authored-by: Codex (GPT 5.6 Sol) <codex@openai.com>
Implements docs/specs/2026-08-15-harness-coding-compare.md with a full-surface DSL fixture and a frozen Qoder README comparison runner. Trials use isolated repositories, bounded tool permissions, deterministic grading, redacted traces, runtime receipts, and durable verdict artifacts. Validated with npm run check, a native Qoder H0/H1 run where both real README outcomes scored 100, a permission-model regrade, and a cold tarball install/import smoke. Co-authored-by: Codex (GPT 5.6 Sol) <codex@openai.com>
Add a named qoder-minimal-v1 executor contract and variant-level compare profiles so runtime effects can be measured independently of harness composition. The profile fixes a four-tool surface, isolates optional SDK context, and records non-secret runtime evidence. Implements docs/specs/2026-08-15-qoder-minimal-runtime-profile.md. Validation passed in an isolated staged checkout with npm run harness:generated, npm run harness:build, npm run harness:test (56 tests), and the documentation link graph (6 checks). Co-authored-by: Codex (GPT 5.6 Sol) <codex@openai.com>
…n defects Make the smallest useful .harness document five lines: a bare materializes a component as advisory prompt guidance, mechanism and strength are optional, one binding can name several hosts, and a composition can extend another. The README now teaches that minimal form first and routes to the full surface only when an author needs it.
Replace the v0.1 authored surface (component / binding / plugin / composition) with host-neutral core semantics: a harness assembles one workflow, its logical agent roles, and their capability requirements; skill, tool, and mcp are the three standard capability kinds; runtime names a concrete host with its adapter and execution style; target is a deployment statement; binding is the adapter-layer mapping. The core DSL no longer defines a generic plugin. Host plugin and extension concepts stay in the host's own namespace as dotted binding mechanisms (qoder.plugin, pi.extension, deepseek.plugin, prime.python-skill) behind the runtime's adapter, so DeepSeek's Cordis plugin runtime and Pi's package/extension split remain isolated implementation details. Workflow implementation and capability exposure become independent dimensions: a workflow is either a declarative graph (edges, event routes, stop conditions) or a programmatic controller, and a runtime exposes capabilities through tool calling or programmatic calling in a host language. Resolution enforces deployability instead of pretending a Deno workflow can be translated into Prime's Python, and points at a matching runtime or external ACP driving. The proven v0.1 machinery is kept: the strength ladder, preferred / minimum / on-degrade policy, the declared-versus-materialized boundary with its advisory cap, deny-wins permission merging (now including MCP transport grants), canonical hashing, and the immutable revision. Realizations are recorded per agent and capability, and IR_VERSION moves to 0.2.0. Validated with npm run build, npm run typecheck, the 78 package tests covering every new rejection and resolution path, check:generated, the repository documentation link graph, and the authoring skill validator against the rewritten examples. Co-authored-by: QoderAI (Qwen 3.8 Max) <qoder_ai@qoder.com>
Add host-neutral streaming events, an AG-UI SSE adapter, and a React studio for live runs and comparison evidence. Stream Qoder partial messages and bounded tool results into keyboard-expandable cards with honest failed, unavailable, truncated, and interrupted states. Harden the loopback browser boundary against client-controlled Host trust, return structured 413 responses for oversized bodies, and add a built-app Playwright regression at 390px. Wire the new packages into CI and the protected GitHub Actions publication selector; local workflows only build, test, pack, or dry-run. Implements docs/specs/2026-08-15-harness-ui-studio.md. Validated with npm run check, the doc-link graph test, workspace pack dry-runs, workflow YAML parsing, and the Playwright interaction test. Co-authored-by: Codex (GPT 5.6 Sol) <codex@openai.com>
The execution layer was a one-shot batch surface: HarnessExecutor.execute ran one prompt to completion, so interactive surfaces could not hold a conversation against a resolved revision, the executor interface carried no version of its own, and runtime capability gaps surfaced only as free-form warning strings. Introduce harness-adapter-v1: HarnessAdapterV1.doStart binds a resolved revision to a live host session that accepts multiple sequential prompt turns, each emitting its own complete run-event sequence under the existing HarnessRunEmitter invariants. Optional behavior now degrades with a typed HarnessCapabilityUnsupportedError carrying adapterId and capability instead of silence, and the framework owns the working directory rather than letting adapters derive their own paths. The Qoder and Pi executors become adapters plus a thin runOnce wrapper, so the batch surface keeps its exact result shape, receipts, metrics, and event sequences. Each adapter's preamble policy follows its host: the Qoder SDK is stateless per query, so an ephemeral session re-sends the preamble every turn while a persisted one sends it once; Pi keeps one createAgentSession for the whole session and sends it on the first turn only. Pi exposes no abort surface, so a turn abortSignal is refused as turn-abort rather than ignored. The sandbox-ownership precondition of the upstream design is deliberately not adopted: binding to natively running hosts is this DSL's core scenario. Turn suspend/continue and AI SDK harness interop are left to later specs. Implements docs/specs/2026-08-15-harness-executor-session-adapter.md. Validated with npm run build and 103 package tests, including the 17 pre-existing exec tests unchanged and 10 new adapter tests covering per-turn event framing, preamble policy, typed capability refusal, graceful-stop degradation, abort bridging, and host mismatch; harness-ui (20) and harness-studio (25) build and pass against the new export surface. Co-authored-by: Claude Code (Claude Agent) <noreply@anthropic.com>
Lock revisions to bundle, source, and adapter facts; fail closed during materialization; keep Qoder turns in one live query; and derive Compare verdicts from paired evidence. This review-driven change has no external Story and implements docs/specs/2026-08-15-harness-execution-closure-blockers.md. Validated with npm run check, preview health/module smoke, Harness Studio Playwright coverage, and a live two-turn Qoder SDK session. Co-authored-by: Codex (GPT 5.6 Sol) <codex@openai.com>
Separate the browser-safe Harness core from host adapters and devtools, move adapter facts into a pure registry, and require injected UI executors. Add trusted-fixture sandbox receipts and bridge persisted harness runs into the Inspector evidence chain with self-contained comparison artifacts. Implements docs/specs/2026-08-15-harness-as-code-architecture-hardening.md. Validated with the root, Harness, Harness UI, Harness Studio, browser, packaging, cold-install, preview, generated-code, module-graph, and documentation-link checks recorded in the spec. Co-authored-by: Codex (GPT 5.6 Sol) <codex@openai.com>
A source-backed skill was locked byte-for-byte, resolved as advisory guidance, and recorded as delivered while the executor put only its path into the prompt. The executor now reads the declared SKILL.md under the source root, bounded at 32 KiB and truncated with an explicit run warning, and inlines the text; an undeliverable source fails the run before the host SDK loads, and Pi materialization copies the skill's real files. Also closes the surrounding honesty and boundary gaps: resolution rejects a descriptor whose adapterId differs from the runtime's adapter, freezes only a cloned revision, and counts a permission grant as enforced only for the exact domain/access pair; compilation rejects stop-free declarative workflows and agent roles the workflow never references; and the AG-UI servers refuse a non-loopback bind without --unsafe-allow-remote. Spec: docs/specs/2026-08-16-harness-review-remediation.md Test: 143 harness + 26 harness-ui + 29 harness-studio tests pass Co-authored-by: QoderAI (Qwen 3.8 Max) <qoder_ai@qoder.com>
Add a package-owned plan/apply contract that binds an exact Git commit to an exact Pi JSONL entry, revalidates both inputs, and continues the session in an isolated detached worktree. Persist the continued session and receipt under Git metadata, create a one-parent result commit on a namespaced ref, and expose a bounded Node-only API and CLI without changing the browser-safe package root. Validated with the full @qoder-ai/harness suite (15 files, 152 tests), package dry-run contents, manifest checks, doc-link checks, and pack verification. Co-authored-by: Codex (GPT 5.6 Sol) <codex@openai.com>
Use fileURLToPath for file URL conversion and normalize separators only at assertion boundaries so the Harness package tests exercise the same contracts on Windows, macOS, and Linux.\n\nValidated with: npx vitest run test/compare.test.ts test/module-graph.test.ts\n\nCo-authored-by: Codex (GPT 5.6 Sol) <codex@openai.com>
Assert the redacted file_path value after JSONL parsing so Windows JSON escaping is not mistaken for duplicate path separators.\n\nValidated with: npx vitest run test/compare.test.ts test/module-graph.test.ts\n\nCo-authored-by: Codex (GPT 5.6 Sol) <codex@openai.com>
Convert the built entry file URL with fileURLToPath before walking the Harness UI module graph, avoiding duplicate drive prefixes on Windows.\n\nValidated with: npx vitest run test/module-graph.test.ts\n\nCo-authored-by: Codex (GPT 5.6 Sol) <codex@openai.com>
Thread source locking and delivery through harness-ui, harness-studio, and compare using docs/specs/2026-08-16-harness-ui-studio-compare-source-skill-delivery.md. Resolve skill paths from the harness file directory and validate actual SKILL.md delivery across all three surfaces. Validated with npm run check. Co-authored-by: Codex (GPT 5.6 Sol) <codex@openai.com>
Replace the checkout-sensitive harness text rewrites with a direct source-backed compare fixture documented in docs/specs/2026-08-16-harness-ui-studio-compare-source-skill-delivery.md. This keeps Windows CRLF conversion from dropping the skill binding while preserving the delivery behavior under test. Validated with npm run check (1,325 root tests; 152 harness tests; 29 UI tests; 31 Studio tests; package verification passed). Co-authored-by: Codex (GPT 5.6 Sol) <codex@openai.com>
Build the expected default source roots with Node's native path resolver so the Harness UI and Studio tests pass on Windows while preserving the explicit override contract. Validated with both package test suites. Co-authored-by: Codex (GPT 5.6 Sol) <codex@openai.com>
Implements the checkpoint-anchored multi-lane experiment contract, runner, and Studio Workbench described in docs/specs/2026-08-17-harness-studio-checkpoint-compare.md. The change keeps checkpoint identity separate from sandbox creation, streams lane-scoped ACP events, and correlates tool paths without overstating historical provenance. Validated with 179 harness tests, 39 Studio tests, three browser flows, six doc-link tests, real Better Harness lanes, Canvas preview health checks, and an npm pack dry run. Dirty-state patch replay remains an explicit pending slice in the spec. Co-authored-by: Codex (GPT 5.6 Sol) <codex@openai.com>
Replace the v0.2 authoring surface with explicit language versions and deployments, honest workflow modes, kind-specific capability realization, and exact tool contracts. Preserve revision, source-lock, and adapter preflight while migrating examples, documentation, the generation skill, and the harness-ui consumer. Implements docs/specs/2026-08-17-harness-dsl-v0.3-semantic-contraction.md. Validation covered 156 harness tests, 29 harness-ui tests, 1325 root tests, the documentation link graph, and package verification. Co-authored-by: Codex (GPT 5.6 Sol) <codex@openai.com>
Turn the Studio experiment surface into a two-stage Compare flow. A Builder discovers bounded project history through a server-side checkpoint adapter, resolves one opaque item into a source-neutral checkpoint projection, and locks it into an immutable content-addressed experiment definition; resolve and lock create no worktree, sandbox, or result ref, so isolated copies still appear only on Run. The Workbench then compares one focused Baseline/Candidate pair across Summary, Trace, and Evidence, with Reference/Baseline/Candidate as the single role vocabulary and per-comparison verdicts instead of a global one. Provider event shapes (ACP, AG-UI, Anthropic) are normalized at the server boundary so React holds no provider branches, and comparability, role selection, pair results, relation counts, resource ledgers, and canonical event folding move into pure tested modules split from the React controller. ADR-0005 records the boundary this rests on: Git/worktree is the first checkpoint source adapter, not the product contract, and discovery, lock, and materialization stay separate phases. Review of the change also fixed three defects. A run completion now settles only its own run's calls, so parallel trials within one lane can no longer mark each other's in-flight calls result-unavailable. The narrow-viewport browser flow waits for the collapsed context rail instead of measuring before React flushes the media-query state. The Builder treatment assertion expects the real profile ids the manifest carries, and the spec's stale harness and Studio test counts were corrected to the observed runs. Implements docs/specs/2026-08-17-harness-studio-checkpoint-compare.md slices 4 through 6. Validated with 1325 root tests, 59 Studio tests, 156 harness tests, three browser flows, the documentation link graph, typecheck, and a clean whitespace check. Dirty-state patch replay remains an explicit pending slice, and a third fresh run still lacks its own unfocused role. Co-authored-by: QoderAI (Qoder Agent) <qoder_ai@qoder.com>
Restructure Harness Inspector Session Detail into numbered Run Cells with a collapsible Process trace, an evidence-bounded Outcome, and a read-only Session outline, and restructure the Harness Studio Workbench into a notebook bar, shared Context, Run Comparison cell, and Compare Result cell with a right-hand Checkpoints rail. Both follow the spec in docs/specs/2026-08-18-notebook-session-and-studio.md. The session projection now keeps one canonical Turn event order, as decided in docs/adrs/session-notebook-evidence-projection.md: assistant messages and tool requests stay in observed source order, and an assistant message is promoted to the terminal response only when it is the final retained event in its Turn. A Turn whose last observed event is a tool call reports responseStatus "incomplete" instead of presenting an intermediate message as the result. Counts now separate intermediate responses, tool calls, and total retained process events, and the report model carries the new fields with a conservative fallback for reports written before them. Outcome states that a session-scoped patch is unavailable rather than deriving a diff from the mutable worktree. Assistant prose renders through a sanitized Markdown renderer that escapes raw HTML and drops link targets outside http, https, mailto, anchor, and relative paths. The Studio checkpoint rail button now carries call and event counts in its accessible name, so the Playwright locator for the Trace "Calls" lens is pinned to an exact name match. Validated with npx vitest run (95 files, 1327 tests), npm run harness-studio:test (59 tests), and npm run harness-studio:test:browser (3 tests). Co-authored-by: QoderAI (Qoder Agent) <qoder_ai@qoder.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
@qoder-ai/harness/session-executorcontract andharness-session-executorCLISession checkpoint POC boundaries
read,ls,edit, andwrite; Bash, extensions, skills, prompt templates, delete, rename, and.gitaccess are disabledValidation
npm test -w @qoder-ai/harness— 15 files, 152 tests passednpx vitest run test/plugins/plugin-manifests.test.mjs test/skills-docs/doc-link-graph.test.mjs— 2 files, 9 tests passednpm pack --dry-run --ignore-scripts -w @qoder-ai/harness --json— compiled API, declarations, and CLI includednpm run pack:verify— npm and runtime bundles verifiedThe checkpoint suite uses temporary Git repositories and the real Pi
SessionManagerbranch/fork API, but deliberately does not make a paid/live model request.