This guide takes a new task from an idea to a reviewable PR. For agent-assisted
work, invoke $create-searchswe-task or ask your agent to follow the
task-authoring skill.
To run existing tasks, start with the evaluation quick start.
Fork the repository, clone your fork, and create a contribution branch.
Before scaffolding, inspect one or two similar reviewed tasks under tasks/ for
structure. Record their paths and relevance in VALIDATION.md, and establish
your own task's data, metrics and resource limits. The
authoring contract
provides the detailed package requirements.
Use a Python 3.12+ environment and run these commands from the checkout root:
pip install -e .
python .agents/skills/create-searchswe-task/scripts/scaffold_task.py example-search \
--author 'Alice Example' --hardware cpuCreate each task at tasks/<task-name>/. Choose a unique name of at most five
lowercase hyphen-separated words, starting with a letter and otherwise using
letters/digits. The task- prefix and all are reserved. Check current tasks
and active PRs for collisions. A PR may include multiple tasks.
Use --hardware gpu when execution needs a GPU, and repeat --author for
coauthors. Record actual authors in task.toml and use a Git email linked to
your GitHub account. The scaffold sets task.name to search-swe/<task-name>;
complete its prompts and verifier before running the task.
Define a concrete engineering goal, starting inputs, submission interface, measurable outcomes and resource budget. Keep an ordinary task contribution within its package and reuse the shared CPU/GPU images. The launcher prepares the selected coding harness; task Dockerfiles add the task's dependencies. Verifier judge dependencies keep their independent pins.
| File or directory | What to provide |
|---|---|
raw-instruction.md, instruction.md |
The task specification and a clear agent-facing request covering its requirements |
task.toml |
Name, authors, version, resource limits, artifact paths and service configuration |
assets.json |
Fixed input paths, sources, immutable revisions, sizes and SHA-256 hashes |
environment/ |
Reproducible agent setup, starter code and resource documentation |
tests/ |
A separate verifier, task-specific grader, trajectory integrity audit and protected evaluation inputs |
README.md |
Task metadata, difficulty, verification and references |
VALIDATION.md |
Design decisions, checks, model feedback, revisions and evidence for the final setting |
Keep the README sections generated by the scaffold: Task Metadata,
Change Log, Difficulty Explanation, Verification Explanation, and
References, under the task title. New tasks start at 1.0.0 with a dated
Added initial task entry; keep the README and task.toml versions aligned.
List the papers, datasets or methods used, or None.
Keep credentials, downloaded inputs/models, job outputs and author-only
solutions outside Git. Isolate verifier inputs and grading code from the
coding agent and submitted program. Declare permitted external services in the
task's resource documentation. Use
metadata.configuration
in task.toml to give the CLI setup descriptions, guidance and required flags
for task-specific environment variables.
Fixed inputs can reference public upstream Hugging Face datasets directly.
Record provenance and usage terms, then pin the exact filename, full immutable
40-character commit SHA, size and SHA-256 in assets.json. Model files retain
their original repository pins. For new redistributable inputs, use a personal
public dataset following the development-data guide.
Keep referenced files available throughout review and while released tasks
still depend on them. Coordinate any official hosting with the maintainer in
the PR.
Start with the package checks:
python scripts/validation/check_tasks.py tasks/example-search
git diff --checkFollow the validation steps to check Compose configuration, container builds, and verifier known-good and negative cases. Run task-specific regressions; changes to shared tools also need the applicable repository tests.
Then run a coding-model trial with the CLI:
search-swe run example-searchChoose the harness, provider, model and reasoning setting within your resource
and API budget. The CLI guides credential setup, restores fixed inputs
and verifies their checksums. For GPU trials, use --gpu-devices to select an
available device. Each trial needs a fresh output directory.
If required keys are missing, the authoring agent must identify their variable
names, purpose and local configuration steps before the dependent run. Enter
keys locally through the CLI or .env; keep their values out of chat and PRs.
See the service access rules when enabling
OpenRouter submission APIs.
Inspect the trajectory, artifacts, scores and judge health. Use that feedback
to fix ambiguous instructions, missing resources or grading defects while
preserving the intended difficulty. Revalidate affected checks and repeat the
model trial when changes affect the task or scoring. Record the model settings,
final tested revision, results, feedback and resulting changes in VALIDATION.md,
along with any unrun checks and their blockers. The
model-evaluation workflow
describes the feedback loop in detail.
Open a PR from your contribution branch to main. Include:
- The task name, objective and CPU/GPU requirements.
- Input sources, immutable revisions and applicable usage terms.
- A link to
VALIDATION.md, with grader checks, model-trial results and the final tested revision; identify outstanding checks explicitly. - Any hosting or setup help needed from maintainers.
Enable Allow edits from maintainers where available so review fixes and asset pins can be updated in the same PR. Otherwise, apply agreed review changes in your branch. Keep validation evidence current as the task changes.
Maintainers use the task-review skill for asset publication and merge checks.