Establish a product-level coding-agent benchmark for Workshop workflows

[Teakowa]

Goal

Establish an agent benchmark for Wright that measures whether a general-purpose coding agent can complete realistic Overwatch Workshop development tasks using the project and Wright itself, without Workshop-specific prompt or skill injection.

The benchmark should make agentic capability a product-level acceptance surface: if an otherwise general coding agent can understand, implement, diagnose, modify, and verify non-trivial Workshop requirements through Wright, that is strong evidence that Wright exposes useful semantic tooling rather than merely existing as a collection of commands.

Context

Wright's product direction includes diagnostics, lint/static analysis, inspect/semantic query, validated source edits, agent/embedding workflows, and compilation where supported by the owning engine.

A meaningful agent benchmark should exercise these capabilities together through realistic Workshop tasks.

The benchmark should answer one concrete product question:

Can a general coding agent, without Workshop-specific prompt or skill injection, use Wright and the project itself to complete realistic Workshop development work correctly?

This is not intended to prove that a model has memorized Workshop. The benchmark should reveal whether Wright provides enough discoverable semantic information, diagnostics, queries, validation, and edit support for an agent to work effectively without a separate Workshop knowledge pack.

Scope

Benchmark model

Define a reproducible benchmark harness around:

natural-language requirement
→ agent explores the project and Wright capabilities
→ agent creates, understands, or modifies source
→ Wright check / inspect / lint / analyze / edit / compile as applicable
→ agent reacts to diagnostics and query results
→ final project is validated

The benchmark must distinguish Wright/product failures from owning-engine gaps.

• Workshop semantics and canonical behavior belong to workshop-rs.
• OverPy semantics belong to opy-rs.
• DEL/OSTW semantics belong to deltin-rs.
• Agent/tooling integration and orchestration belong to Wright.

No Workshop-specific injection

The primary benchmark condition should not preload the agent with:

• a Workshop-specific system prompt;
• a Workshop/OverPy/OSTW skill pack;
• a generated all-functions reference inserted into context;
• hidden task-specific hints or solution notes.

The agent may use:

• the normal repository contents and project instructions available to an ordinary coding agent;
• Wright's released/documented agent, CLI, or service surfaces;
• source-language/project documentation that is naturally part of the repository or discoverable through the product workflow.

The exact agent/model/version and environment must be recorded for reproducibility, but the benchmark contract must not depend on one vendor or model family.

Scenario families

Start with scenarios that test different parts of the product rather than only greenfield generation.

1. Greenfield implementation

   • Generate a complete mode or feature from natural-language requirements.
   • Include requirements that require Workshop-specific feasibility decisions rather than syntax transcription.
   • Use realistic interactions such as lifecycle handling, hero restrictions, HUDs, projectiles, raycasts, state transitions, dynamic health, effects, and Workshop limits.

2. Existing-project understanding

   • Ask the agent to explain or locate behavior in an existing Workshop/OPY/DEL project.
   • Exercise symbols, references, semantic queries, control flow, settings, and domain facts where supported.

3. Existing-project modification

   • Request a bounded semantic change in an existing project.
   • Verify the intended behavior changes while unrelated source structure remains preserved.

4. Diagnosis and repair

   • Provide a project with a real or representative defect.
   • Require the agent to use Wright diagnostics/queries, identify the root cause, apply a fix, and re-check the result.

Language coverage

Benchmark scenarios should be conceptually language-neutral where practical.

The same user-level requirement may be exercised through:

• raw Workshop;
• OverPy;
• DEL/OSTW;

only when the corresponding owning implementation declares the required capability supported.

Do not duplicate or fake missing source-language semantics in Wright merely to make a benchmark pass.

Verification

Prefer deterministic assertions over subjective model-quality scoring.

Useful result fields may include:

• required behaviors satisfied / failed;
• source parses and checks successfully;
• semantic diagnostics remaining;
• unsupported or hallucinated APIs used;
• required correction rounds;
• unsafe/destructive edits;
• intended edit preservation checks;
• compilation success when compilation is part of the scenario;
• generated Workshop structural/cost assertions where relevant;
• whether the agent correctly refused an impossible or unsupported requirement.

Use ordinary tests and benchmark artifacts. Do not create a parallel verification framework.

Where a requirement can only be verified in the Overwatch runtime, keep that distinction explicit instead of treating static validation as proof of runtime behavior.

Comparative modes

The harness should make controlled comparisons possible, for example:

general agent only
vs
general agent + repository
vs
general agent + repository + Wright

Additional experiments with specialized Workshop guidance may be useful later, but the primary Wright product acceptance condition is the no-specialized-injection path.

The comparison exists to identify what capability Wright adds; it is not a model leaderboard.

Non-goals

• Building a generic agent framework or autonomous planner inside Wright.
• Shipping model-provider integrations as part of the benchmark itself.
• Training or fine-tuning a model.
• Making any large Workshop-specific prompt or knowledge pack a prerequisite for success.
• Treating one successful stochastic model run as proof of Wright correctness.
• Replacing owner-level unit, regression, compatibility, integration, or real-project tests.
• Requiring live Overwatch runtime validation for scenarios whose claims are fully static.
• Adding Wright-side semantic workarounds for incomplete language engines.
• Ranking models or optimizing the benchmark for a specific model vendor.

Acceptance criteria

• A benchmark contract defines the allowed agent context, Wright surfaces, environment metadata, scenario format, result format, and deterministic validation rules.
• The primary benchmark condition can run a general coding agent without Workshop-specific prompt/skill injection.
• The initial suite includes at least one scenario from each applicable family: greenfield implementation, existing-project understanding, bounded modification, and diagnosis/repair.
• At least one initial scenario represents a non-trivial, realistic Workshop game-mode requirement rather than a syntax-only exercise.
• Scenario failures can be attributed to the owning layer rather than collapsed into a single agent score.
• Deterministic checks cover syntax/semantic validity and scenario-specific requirements; runtime-only claims are identified separately.
• The benchmark records correction/tool-use behavior sufficiently to distinguish one-shot generation from an actual Wright-assisted agent loop.
• The harness can compare a no-Wright baseline with the Wright-assisted condition without changing the task itself.
• The benchmark can run against supported raw Workshop and source-language workflows without Wright duplicating owner semantics.
• Results are suitable for identifying concrete product or engine gaps and creating owner Issues from those gaps.
• The benchmark is not required to block ordinary PR CI; execution cadence and cost should be chosen based on measured usefulness and stability.

Dependencies / ownership

• Wright owns the benchmark harness, agent-facing product workflow, semantic query/edit orchestration, and benchmark result contract.
• Language owners remain responsible for syntax/semantic/compiler behavior exposed through Wright.
• Related Wright work: Roadmap to v1.0: stable Workshop tooling platform #134 (1.0 product roadmap) and Ship a versioned agent contract over the shared tool service #407 (versioned agent contract).