manykarim · Nov 12, 2026 · Nov 10, 2026 · Nov 10, 2026 · Nov 11, 2026 · Nov 11, 2026
diff --git a/.env.example b/.env.example
@@ -7,12 +7,17 @@ CLAUDE_CODE_OAUTH_TOKEN=sk-ant-oat01-...
 # is unset. Only uncomment and set if you specifically want API-key auth.
 # ANTHROPIC_API_KEY=sk-ant-...
 
+# OpenRouter API key, for runs on an OpenRouter model
+# (--model deepseek/deepseek-v4.1-flash). Claude credentials are not passed to
+# those sessions. Set a credit limit on the key at openrouter.ai.
+# OPENROUTER_API_KEY=sk-or-v1-...
+
 # Default model for evaluation runs. Must be one of:
-#   - claude-haiku-4-5-20251001  (cheap, fast; default for PR/narrow tasks, trigger evals)
-#   - claude-sonnet-5            (higher fidelity; for realistic/adversarial tiers)
+#   - claude-haiku-5-5   (cheap, fast; default for PR/narrow tasks, trigger evals)
+#   - claude-sonnet-5-5  (higher fidelity; for realistic/adversarial tiers)
 # claude-opus-5-5 is a manual, explicit opt-in only
 # (--model claude-opus-5-5 --allow-opus --max-cost-usd <cap>) — see ADR-004.
-CLAUDE_MODEL_DEFAULT=claude-haiku-4-5-20251001
+CLAUDE_MODEL_DEFAULT=claude-haiku-5-5
 
 # Log level for the harness (DEBUG, INFO, WARNING, ERROR).
 RF_SKILL_EVAL_LOG_LEVEL=INFO

diff --git a/RELEASING.md b/RELEASING.md
@@ -29,6 +29,13 @@ Older tags stay where they are: `v1.1.0` to `v3.0.0` are `rf-agentskills` conten
    ```
 4. `release.yml` checks that the tag matches the plugin's `plugin.json`, builds the plugin's archives, attaches them to a GitHub release and moves the `stable` and `latest` branches.
 
+Before step 3, check the skills on Claude. CI evaluates PRs on DeepSeek V4.1 Flash, through OpenRouter, so run a manual Claude evaluation of the release candidate's branch and check its gate against the Claude baselines:
+
+```bash
+gh workflow run skill-evaluation.yml --ref <branch> \
+  -f provider=claude -f model=claude-sonnet-5-5 -f tiers=narrow,realistic -f arms=treatment,baseline
+```
+
 A pushed tag is the ref users pin a marketplace to: never move or delete one.
 
 ## Installer release

diff --git a/docs/ci/architecture/adr/ADR-005-ci-integration.md b/docs/ci/architecture/adr/ADR-005-ci-integration.md
@@ -7,6 +7,36 @@
 
 ---
 
+## Amendment 2026-11-10 — `openrouter-ci-eval-tier`
+
+PR and weekly runs move to a cheap OpenRouter tier when the
+`OPENROUTER_API_KEY` secret is set, so the evals no longer depend on a
+maintainer's Claude subscription. The 2026-09-27 tiers below stay as they
+are; only the model and the credentials change.
+
+- **Provider.** Preflight picks `openrouter` when the OpenRouter key is set,
+  `claude` otherwise; a manual dispatch input `provider` overrides it. Each
+  eval job gets only that provider's credentials.
+- **Model.** `deepseek/deepseek-v4.1-flash` for every tier, with turn-limit
+  factor 1.5 (`--max-turns-factor`). Claude runs (Haiku, Sonnet, Opus) are a
+  manual dispatch with `provider: claude`, and part of the release checklist.
+- **Baselines.** Kept per model: `plugins/<plugin>/evals/baselines/` for
+  Claude, `plugins/<plugin>/evals/baselines/openrouter/deepseek-v4.1-flash/`
+  for DeepSeek. The gate never compares one model with another.
+- **Cost.** The harness prices OpenRouter runs at list prices (Claude Code's
+  own figure prices them as Claude); caps are `PR_NARROW_CAP_USD_OPENROUTER`,
+  `PR_TRIGGER_CAP_USD_OPENROUTER` and `WEEKLY_CAP_USD_OPENROUTER`. Each job
+  records the key's spend before and after, and the key's credit limit on
+  OpenRouter is the hard cap.
+- **Why.** A spike (run 42822700497, rf-core narrow, both arms, N=3) completed
+  all 72 sessions on DeepSeek for $0.84; the skills loaded in every treatment
+  run and the treatment–baseline delta stayed visible on 8 of 12 tasks.
+- **Risk.** Anthropic doesn't support non-Claude models in Claude Code. Each
+  OpenRouter job starts with a one-session smoke test; if a Claude Code
+  release breaks the path, dispatch with `provider: claude`.
+
+---
+
 ## Amendment 2026-09-27 — `strengthen-skill-eval-harness`
 
 The three planned workflows are implemented as **one** workflow,

diff --git a/docs/ci/local-testing.md b/docs/ci/local-testing.md
@@ -12,6 +12,7 @@ PR. For the full user guide, see [usage.md](usage.md).
   - [OAuth token expiry](#oauth-token-expiry)
   - [Rate-limit exhaustion](#rate-limit-exhaustion)
   - [`uv sync` cache issues](#uv-sync-cache-issues)
+  - [Running on OpenRouter](#running-on-openrouter)
 - [Iterating on a failing task](#iterating-on-a-failing-task)
 - [Adding a new skill to the eval task bank](#adding-a-new-skill-to-the-eval-task-bank)
 
@@ -182,6 +183,24 @@ uv lock --check
 If `uv lock --check` fails on `main`, file an issue — CI enforces
 lock integrity and should have caught it.
 
+### Running on OpenRouter
+
+Without a Claude subscription, run on DeepSeek V4.1 Flash through OpenRouter.
+Set `OPENROUTER_API_KEY` (in `.env` or exported), give the key a credit limit
+at openrouter.ai, and pass the model and the turn-limit factor CI uses:
+
+```bash
+uv run rf-skill-eval doctor --provider openrouter
+uv run rf-skill-eval run-batch --tier narrow --arms treatment --runs 3 \
+  --model deepseek/deepseek-v4.1-flash --max-turns-factor 1.5 --max-cost-usd 2 \
+  --output eval/runs/or-$(date +%s)
+```
+
+Compare the results with the DeepSeek baselines (`gate` finds them from the
+runs' model), not the Haiku ones. The costs in the report are estimates at
+OpenRouter's list prices; the key's activity page on openrouter.ai shows what
+was charged. See [usage.md](usage.md#openrouter-tier).
+
 ---
 
 ## Iterating on a failing task

diff --git a/docs/ci/usage.md b/docs/ci/usage.md
@@ -22,6 +22,7 @@ same machinery on every PR.
 - [Adding a New Task](#adding-a-new-task)
 - [CI Reference](#ci-reference)
 - [Model Policy](#model-policy)
+- [OpenRouter tier](#openrouter-tier)
 - [Cost Expectations](#cost-expectations)
 - [Related Documents](#related-documents)
 
@@ -123,6 +124,9 @@ comment out `CLAUDE_CODE_OAUTH_TOKEN` and uncomment
 `ANTHROPIC_API_KEY` instead. See [faq.md](faq.md#subscription) for
 details.
 
+To run on the cheap OpenRouter model instead, set `OPENROUTER_API_KEY`; see
+[OpenRouter tier](#openrouter-tier).
+
 ### 3. Install Python dependencies
 
 ```bash
@@ -362,8 +366,10 @@ tokens above `--token-budget` (default +30 %; the PR job uses +75 %), incomplete
 significantly below the baseline — Fisher exact p < 0.05 on run-level loads,
 or more than one query for baselines without run counts — a shipped skill
 listed by name only in a skill listing, cost over `--max-cost-usd`) or 3 (`rebaseline-needed` only: the task
-definition, fixture or model changed, or no baseline entry exists — never a
-pass). Baseline files are promoted through a reviewed PR
+definition, fixture, model or turn-limit factor changed, or no baseline entry exists — never a
+pass). Without `--baseline`, `gate` and `baseline update` use the runs' model's
+folder: `<evals>/baselines/` for Claude, `<evals>/baselines/openrouter/<model name>/`
+for an OpenRouter model. Baseline files are promoted through a reviewed PR
 ([`eval/baselines/README.md`](../../eval/baselines/README.md)).
 
 ---
@@ -406,13 +412,19 @@ amendment).
 
 | Trigger | Jobs | Scope | Arms | N | Model | Cap |
 |---|---|---|---|---|---|---|
-| `pull_request` | `preflight` → `pr-eval` | narrow tasks of changed skills + `plugin` canaries (all narrow tasks when the harness changed); validation-split trigger evals when a SKILL.md `description` changed | treatment | 3 | `claude-haiku-5-5` | `PR_NARROW_CAP_USD`=10, `PR_TRIGGER_CAP_USD`=20 |
-| `schedule` (Sun 04:00 UTC) | `preflight` → `full-eval` | all tiers + all trigger sets | treatment, baseline | 3 | tier defaults | 12 + 12 + 8 + 8 = $40 |
-| `workflow_dispatch` | `preflight` → `full-eval` | inputs `tiers`, `arms`, `runs`, `model`, `allow_opus`, `max_cost_usd`, `triggers` | input | input | input | `max_cost_usd` per invocation |
+| `pull_request` | `preflight` → `pr-eval` | narrow tasks of changed skills + `plugin` canaries (all narrow tasks when the harness changed); validation-split trigger evals when a SKILL.md `description` changed | treatment | 3 | `deepseek/deepseek-v4.1-flash` when `OPENROUTER_API_KEY` is set, else `claude-haiku-5-5` | OpenRouter: `PR_NARROW_CAP_USD_OPENROUTER`=2, `PR_TRIGGER_CAP_USD_OPENROUTER`=2; Claude: `PR_NARROW_CAP_USD`=10, `PR_TRIGGER_CAP_USD`=20 |
+| `schedule` (Sun 04:00 UTC) | `preflight` → `full-eval` | all tiers + all trigger sets | treatment, baseline | 3 | DeepSeek when its key is set, else tier defaults | OpenRouter: `WEEKLY_CAP_USD_OPENROUTER`=3 per invocation; Claude: 12 + 12 + 8 + 8 = $40 |
+| `workflow_dispatch` | `preflight` → `full-eval` | inputs `provider`, `tiers`, `arms`, `runs`, `model`, `allow_opus`, `max_cost_usd`, `triggers` | input | input | `provider: claude` + `model`, or DeepSeek | `max_cost_usd` per invocation |
 | all | `harness-tests` | ruff, mypy, `pytest tests/eval` (no API calls) | – | – | – | – |
 
-- The PR job gates against `eval/baselines/narrow.json` and posts one PR
-  comment (report + gate result + trigger report).
+- Preflight picks the provider once: OpenRouter when `OPENROUTER_API_KEY` is
+  set, otherwise Claude; a manual dispatch with `provider` overrides it. Each
+  eval job gets only that provider's credentials.
+- The PR job gates against the provider's baselines
+  (`plugins/<plugin>/evals/baselines/narrow.json` for Claude,
+  `plugins/<plugin>/evals/baselines/openrouter/deepseek-v4.1-flash/narrow.json`
+  for OpenRouter) and posts one PR comment (report + gate result + trigger
+  report, and the OpenRouter spend).
 - The weekly job uploads the report and a `candidate-baseline-<run>`
   artifact; a maintainer promotes it with a PR. The PR gate becomes a
   **required** check only after two weekly baselines have been recorded.
@@ -431,6 +443,12 @@ gh workflow run skill-evaluation.yml -f tiers=narrow -f arms=treatment -f runs=5
 gh workflow run skill-evaluation.yml -f tiers=narrow -f model=claude-opus-5-5 -f allow_opus=true -f max_cost_usd=20
 # Force API-key auth:
 gh workflow run skill-evaluation.yml -f use_api_key=true
+# A Claude run while the OpenRouter key is set (before a release):
+gh workflow run skill-evaluation.yml -f provider=claude -f tiers=narrow,realistic -f model=claude-sonnet-5-5
+# Record the OpenRouter baselines (N=6):
+gh workflow run skill-evaluation.yml -f provider=openrouter -f runs=6
+# Re-record one plugin only:
+gh workflow run skill-evaluation.yml -f provider=openrouter -f runs=6 -f plugins=rf-desktop
 ```
 
 ---
@@ -449,6 +467,56 @@ gh workflow run skill-evaluation.yml -f use_api_key=true
   It never runs on PRs or the weekly schedule (≈5× Sonnet per token).
 - Retired ids (`claude-haiku-4-5`, `claude-sonnet-4-6`) are rejected with a
   migration hint.
+- OpenRouter models (`deepseek/deepseek-v4.1-flash`) are run time only; a task
+  YAML that declares one fails to load. See [OpenRouter tier](#openrouter-tier).
+
+---
+
+## OpenRouter tier
+
+The harness can run every session on DeepSeek V4.1 Flash through OpenRouter,
+so the evals run from an OpenRouter key instead of a Claude subscription. CI
+uses it by default when the `OPENROUTER_API_KEY` secret is set. A spike run of
+the rf-core narrow tier (both arms, N=3, 72 sessions) cost $0.84.
+
+```bash
+export OPENROUTER_API_KEY=sk-or-v1-...           # or put it in .env
+uv run rf-skill-eval doctor --provider openrouter
+uv run rf-skill-eval run-batch --tier narrow --arms treatment --runs 3 \
+  --model deepseek/deepseek-v4.1-flash --max-turns-factor 1.5 --max-cost-usd 2 \
+  --output eval/runs/or
+uv run rf-skill-eval gate --runs-dir eval/runs/or   # uses baselines/openrouter/deepseek-v4.1-flash/
+```
+
+- **Provider.** An OpenRouter model id selects the OpenRouter provider;
+  `--provider openrouter|anthropic` only states it, and a mismatch is refused.
+  The runner points Claude Code at `https://openrouter.ai/api`, maps every
+  Claude model alias and the subagent model to the chosen model, and doesn't
+  pass `CLAUDE_CODE_OAUTH_TOKEN` or `ANTHROPIC_API_KEY` to the session. A run
+  whose usage names any other model is recorded as incomplete
+  (`provider_model`).
+- **Allowed models.** Only `deepseek/deepseek-v4.1-flash`. Adding one is a
+  code change: an entry in `OPENROUTER_MODELS`, a price in
+  `src/rf_skill_eval/domain/pricing.py`, and its own baselines.
+- **Cost.** Claude Code prices every session as Claude (the spike reported
+  $29.66). For OpenRouter runs the harness prices the token counts at
+  OpenRouter's list prices instead; reports label that figure as an estimate.
+  List prices are an upper bound (the spike priced at $1.33 against $0.84
+  charged). `--max-cost-usd` uses the estimate. CI also records the key's
+  real spend before and after each job; the key's credit limit is the hard
+  cap.
+- **Turn limits.** DeepSeek takes more turns than Haiku: 25 of the 72 spike
+  sessions stopped at the limit. `--max-turns-factor 1.5` scales each task's
+  `max_turns` (rounded up); the factor is stored with the runs and the
+  baselines, and a different factor means `rebaseline-needed`. Reports show
+  turn-limit stops per task and arm. `OPENROUTER_TURNS_FACTORS` in the
+  workflow can override the factor per plugin. None does now: rf-desktop at
+  2.0 used the extra turns and passed no more often.
+- **What it tells you.** Results say how the skills work with DeepSeek, not
+  with Claude. Run a Claude check before a release (see RELEASING.md).
+  Anthropic doesn't support non-Claude models in Claude Code, which prints
+  `unrecognized_model` on each session; a Claude Code release could break this
+  path, and the CI smoke step would catch it.
 
 ---
 

diff --git a/openspec/changes/openrouter-ci-eval-tier/.openspec.yaml b/openspec/changes/openrouter-ci-eval-tier/.openspec.yaml
@@ -0,0 +1,2 @@
+schema: spec-driven
+created: 2026-11-10