Skip to content

Latest commit

 

History

History
133 lines (105 loc) · 6.72 KB

tasks.md

File metadata and controls

133 lines (105 loc) · 6.72 KB

Contribute a task

Documentation index

This guide takes a new task from an idea to a reviewable PR. For agent-assisted work, invoke $create-searchswe-task or ask your agent to follow the task-authoring skill. To run existing tasks, start with the evaluation quick start.

1. Create the task package

Fork the repository, clone your fork, and create a contribution branch. Before scaffolding, inspect one or two similar reviewed tasks under tasks/ for structure. Record their paths and relevance in VALIDATION.md, and establish your own task's data, metrics and resource limits. The authoring contract provides the detailed package requirements.

Use a Python 3.12+ environment and run these commands from the checkout root:

pip install -e .
python .agents/skills/create-searchswe-task/scripts/scaffold_task.py example-search \
  --author 'Alice Example' --hardware cpu

Create each task at tasks/<task-name>/. Choose a unique name of at most five lowercase hyphen-separated words, starting with a letter and otherwise using letters/digits. The task- prefix and all are reserved. Check current tasks and active PRs for collisions. A PR may include multiple tasks.

Use --hardware gpu when execution needs a GPU, and repeat --author for coauthors. Record actual authors in task.toml and use a Git email linked to your GitHub account. The scaffold sets task.name to search-swe/<task-name>; complete its prompts and verifier before running the task.

2. Complete the task and its inputs

Define a concrete engineering goal, starting inputs, submission interface, measurable outcomes and resource budget. Keep an ordinary task contribution within its package and reuse the shared CPU/GPU images. The launcher prepares the selected coding harness; task Dockerfiles add the task's dependencies. Verifier judge dependencies keep their independent pins.

File or directory What to provide
raw-instruction.md, instruction.md The task specification and a clear agent-facing request covering its requirements
task.toml Name, authors, version, resource limits, artifact paths and service configuration
assets.json Fixed input paths, sources, immutable revisions, sizes and SHA-256 hashes
environment/ Reproducible agent setup, starter code and resource documentation
tests/ A separate verifier, task-specific grader, trajectory integrity audit and protected evaluation inputs
README.md Task metadata, difficulty, verification and references
VALIDATION.md Design decisions, checks, model feedback, revisions and evidence for the final setting

Keep the README sections generated by the scaffold: Task Metadata, Change Log, Difficulty Explanation, Verification Explanation, and References, under the task title. New tasks start at 1.0.0 with a dated Added initial task entry; keep the README and task.toml versions aligned. List the papers, datasets or methods used, or None.

Keep credentials, downloaded inputs/models, job outputs and author-only solutions outside Git. Isolate verifier inputs and grading code from the coding agent and submitted program. Declare permitted external services in the task's resource documentation. Use metadata.configuration in task.toml to give the CLI setup descriptions, guidance and required flags for task-specific environment variables.

Fixed inputs can reference public upstream Hugging Face datasets directly. Record provenance and usage terms, then pin the exact filename, full immutable 40-character commit SHA, size and SHA-256 in assets.json. Model files retain their original repository pins. For new redistributable inputs, use a personal public dataset following the development-data guide. Keep referenced files available throughout review and while released tasks still depend on them. Coordinate any official hosting with the maintainer in the PR.

3. Validate, run a model and refine

Start with the package checks:

python scripts/validation/check_tasks.py tasks/example-search
git diff --check

Follow the validation steps to check Compose configuration, container builds, and verifier known-good and negative cases. Run task-specific regressions; changes to shared tools also need the applicable repository tests.

Then run a coding-model trial with the CLI:

search-swe run example-search

Choose the harness, provider, model and reasoning setting within your resource and API budget. The CLI guides credential setup, restores fixed inputs and verifies their checksums. For GPU trials, use --gpu-devices to select an available device. Each trial needs a fresh output directory.

If required keys are missing, the authoring agent must identify their variable names, purpose and local configuration steps before the dependent run. Enter keys locally through the CLI or .env; keep their values out of chat and PRs. See the service access rules when enabling OpenRouter submission APIs.

Inspect the trajectory, artifacts, scores and judge health. Use that feedback to fix ambiguous instructions, missing resources or grading defects while preserving the intended difficulty. Revalidate affected checks and repeat the model trial when changes affect the task or scoring. Record the model settings, final tested revision, results, feedback and resulting changes in VALIDATION.md, along with any unrun checks and their blockers. The model-evaluation workflow describes the feedback loop in detail.

4. Submit the PR

Open a PR from your contribution branch to main. Include:

  • The task name, objective and CPU/GPU requirements.
  • Input sources, immutable revisions and applicable usage terms.
  • A link to VALIDATION.md, with grader checks, model-trial results and the final tested revision; identify outstanding checks explicitly.
  • Any hosting or setup help needed from maintainers.

Enable Allow edits from maintainers where available so review fixes and asset pins can be updated in the same PR. Otherwise, apply agreed review changes in your branch. Keep validation evidence current as the task changes.

Maintainers use the task-review skill for asset publication and merge checks.