Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -39,8 +39,8 @@ AGENT_OPENAI_API_KEY=
# Claude Code direct mode and Pi's anthropic provider: official Anthropic API key.
AGENT_ANTHROPIC_API_KEY=

# OpenRouter Agent mode (--provider openrouter) for Codex, Claude Code or Pi. Separate from
# OPENROUTER_API_KEY used by task submissions and ANSWER_JUDGE_API_KEY.
# OpenRouter coding-agent key for Codex, Claude Code, Pi or Antigravity CLI.
# Task submissions use OPENROUTER_API_KEY; answer judging uses ANSWER_JUDGE_API_KEY.
AGENT_OPENROUTER_API_KEY=

# Optional Codex or Claude Code reasoning effort; default omits the override.
Expand Down
2 changes: 1 addition & 1 deletion docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ task with the CLI or your coding agent.
| Guide | Use it to |
| --- | --- |
| [Quick start](evaluation/quickstart.md) | Install, choose CLI or agent operation, and inspect your first results |
| [CLI reference](evaluation/cli.md) | Configure settings, select tasks, and schedule sequential or parallel batches |
| [CLI reference](evaluation/cli.md) | Configure settings, run batches, and regrade recorded trials |
| [Supported agents and providers](evaluation/agents.md) | Choose harnesses, versions and provider routes; configure API keys or Claude Code/Codex subscriptions |
| [Submission model access](evaluation/submission-models.md) | Apply the required OpenRouter model allowlists to submission API keys |
| [Data and models](evaluation/assets.md) | Plan disk space, download, verify, and restore fixed inputs |
Expand Down
17 changes: 17 additions & 0 deletions docs/evaluation/agents.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ The default is Codex + OpenRouter + `openai/gpt-6-luna` (`max` reasoning).
| Claude Code | `direct` | Claude model at the official Anthropic API | `AGENT_ANTHROPIC_API_KEY` |
| Pi | `openrouter` | OpenRouter text/tool models; full OpenRouter ID | `AGENT_OPENROUTER_API_KEY` |
| Pi | `direct` | `provider/model` from Pi's bundled API-key catalog | Selected provider's key and endpoint settings |
| Antigravity CLI | `openrouter` | OpenRouter text/tool models using Chat Completions | `AGENT_OPENROUTER_API_KEY` |
| Any registered Harbor agent | `native` | Native model ID, or the agent's default | Selected agent's `AGENT_*` variables and optional native configuration |

## Harness versions and installation
Expand Down Expand Up @@ -142,6 +143,22 @@ routes share the Search-SWE event-stream-to-ATIF converter.
Inspect the chosen agent's official
setup when it needs project files, a subscription or cloud credentials.

## Antigravity CLI through OpenRouter

Use CLI 0.3.1 or later in the 0.3.x series with a checkout containing this
integration. Set `AGENT_OPENROUTER_API_KEY` in the checkout's private `.env`.

```bash
search-swe run TASK --agent antigravity-cli --provider openrouter \
--model google/gemini-4-argon --reasoning-effort high
```

The route uses a loopback bridge in the agent container to translate agy's
Gemini requests to OpenRouter Chat Completions. Use the full OpenRouter model
ID. Effort accepts `default` (agy's model default), `low`, or `high`.
The bridge transfers tool calls, results, and reasoning details across turns.
OpenRouter is the coding-model endpoint for this route.

## Claude Code and Codex subscriptions

Both use `--provider native` with explicit credentials in `--agent-config`.
Expand Down
40 changes: 35 additions & 5 deletions docs/evaluation/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,36 @@ New tasks are discovered from the checkout and normally need no CLI upgrade.
The target checkout must match the installed CLI's release series. `update`
updates repository files; package upgrades remain separate Python installations.

## Regrade a recorded trial

Pass the directory of one completed local Harbor trial. The CLI discovers the
task from the recorded result and uses the verifier in the selected checkout.

```bash
search-swe regrade /path/to/original-trial --dry-run
search-swe regrade /path/to/original-trial
```

Keep the original `config.json`, `result.json`, `agent/`, and
`artifacts/manifest.json` with its recorded artifact files. The preview checks
that the current verifier's declared inputs are present, including the ATIF
trajectory for tasks that require it. Regrade rejects
`large-scale-formula-retrieval` during preflight because its verifier needs
private agent-phase memory evidence outside the recorded trial.

Run with the task's verifier credentials and network settings in `.env`
or `--env-file`. The verifier uses the task's usual asset preparation and
CPU/GPU requirements. Use `--offline` for cached assets, `--cpu-set` for a
CPU pool, or `--gpu-device` for a specific GPU.

The CLI starts only the verifier and preserves the original trial. The new
trial and its `regrade.json` summary are written under `jobs/regrades/` or a
fresh `--output` directory. The summary records both trial IDs, the verifier
checkout revision, and rewards. Measurement validity is recorded when the
verifier reports it explicitly. A failed verifier phase or a primary evaluation
invalidated by answer-judge errors causes a nonzero exit status. Review the new
verifier logs before using the result, including when the reward is zero.

## Interactive setup

`search-swe` or `search-swe run` opens the wizard:
Expand All @@ -78,9 +108,9 @@ keys to configure; `run` selects the tasks to evaluate. Use each command's
`--help` for its flags.

The default is **Codex + OpenRouter + `openai/gpt-6-luna`, `max` reasoning**.
Codex, Claude Code and Pi provide integrated OpenRouter routes. All registered
Harbor agents are available through `--provider native`; use `search-swe agents`
to discover them. See the
Codex, Claude Code, Pi and Antigravity CLI provide integrated OpenRouter routes.
Use `search-swe agents` to discover Harbor agents available through
`--provider native`. See the
[agent/provider matrix](https://github.com/VectorSpaceLab/Search-SWE/blob/main/docs/evaluation/agents.md)
for direct routes, model scope and [harness versions/cache](https://github.com/VectorSpaceLab/Search-SWE/blob/main/docs/evaluation/agents.md#harness-versions-and-installation). `search-swe pi-models` lists Pi providers;
add `--provider NAME` for exact model IDs.
Expand All @@ -90,8 +120,8 @@ add `--provider NAME` for exact model IDs.
Search by name or model ID, highlight with ↓/↑ and select with Enter. An empty
field keeps the displayed model; full IDs can also be entered manually.
Suggestions use text, tool and API metadata: Responses for Codex, Messages for
Claude Code, and Chat Completions for Pi. Model selection makes no inference
request; actual compatibility depends on the model/provider.
Claude Code, and Chat Completions for Pi and Antigravity CLI. Model selection
makes no inference requests. Runtime compatibility depends on the model and provider.

The public catalog is cached for 24 hours at
`$XDG_CACHE_HOME/search-swe/openrouter-models.json` (default
Expand Down
2 changes: 1 addition & 1 deletion docs/evaluation/quickstart.md
Original file line number Diff line number Diff line change
Expand Up @@ -105,7 +105,7 @@ URL for a branch or tag. Match the skill to your CLI and checkout.
### Update the skill

The skill records its matching CLI release in `SKILL.md` as `metadata.version`
(currently `0.3.0`), with compatible CLI/checkout series `0.3.x`.
(currently `0.3.1`), with compatible CLI/checkout series `0.3.x`.
For a project installation, run these commands in your coordinating agent's
workspace:

Expand Down
5 changes: 3 additions & 2 deletions docs/evaluation/runtime.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,13 +58,14 @@ The single-task backend uses `--openrouter` for OpenRouter; the CLI uses
| `--agent codex --openrouter` | Full OpenRouter ID | `AGENT_OPENROUTER_API_KEY` |
| `--agent claude-code --openrouter` | Full OpenRouter ID with Messages/tool support | `AGENT_OPENROUTER_API_KEY` |
| `--agent pi --openrouter` | Full OpenRouter ID with Chat Completions/tool support | `AGENT_OPENROUTER_API_KEY` |
| `--agent antigravity-cli --openrouter` | Full OpenRouter ID with Chat Completions/tool support | `AGENT_OPENROUTER_API_KEY` |
| `--agent codex` | Direct endpoint's model ID | `AGENT_OPENAI_BASE_URL`, `AGENT_OPENAI_API_KEY` |
| `--agent claude-code` | Official Anthropic model ID | `AGENT_ANTHROPIC_API_KEY` |
| `--agent pi` | Pi `provider/model` ID | Selected provider's key and endpoint settings |
| `--agent NAME --native` | Native Harbor model ID or agent default | Agent-only `AGENT_*` variables and optional `--agent-config PATH` |

Use `search-swe agents` to discover Harbor's registered agents. Agents beyond the
three integrated routes select native mode automatically in this backend.
Use `search-swe agents` to discover Harbor's registered agents. Antigravity CLI
selects its native route unless `--openrouter` is passed.
`--agent-version VERSION` or `SEARCH_SWE_AGENT_VERSION` overrides the central
[harness default](agents.md#harness-versions-and-installation); `default` resets it.
The five centrally managed CLIs are installed and version-verified during setup,
Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"

[project]
name = "search-swe"
version = "0.3.0"
version = "0.3.1"
description = "Configure and run Search-SWE coding-agent evaluations"
readme = "docs/evaluation/cli.md"
requires-python = ">=3.12"
Expand Down
Loading
Loading