Repository navigation
test-model-routing: replace retired Copilot model gpt-5.4-mini with gpt-5.6-luna #10366
Description
Problem
test-model-routing.yml has failed on every push to main since 19 Oct 2026. Each run fails at the same step: "Run an effort-routed Copilot inference through AWF" in the job "Exercise live routing, policy, and Copilot inference". AWF exits with code 78 before the agent command runs:
[INFO] Waiting for model routing selection...
[ERROR] Fatal error: InfrastructureReadinessError: Model routing failed (model_policy_violation): The model policy excludes every available model
[cause]: RoutingFailureExitError: ... at toRoutingFailureExit (dist/routing/bootstrap.js:70:12)
- Last passing run: 39298157165 (
01b6e9af, 19 Oct 17:54 UTC). - First failing run: 39311298054 (
2d770644, 19 Oct 19:28 UTC). - Latest failing run: 39468350283 (
a6e3ce95). The four runs on 20 Oct fail the same way.
Because this step fails, the rest of the job (cross-endpoint sub-agents, two-candidate routing, the preserved-selection assertions, and the Copilot CLI agent run) has been skipped since 19 Oct. The live routing path on main is therefore not covered end to end.
Root cause
The step's apiProxy.allowedModels contains only github-copilot/gpt-5.4-mini. GitHub deprecated that model on 2026-10-19 (changelog, 18 Sep 2026): "GPT-5.4 mini | 2026-10-19 | suggested alternative GPT-5.6 Luna". The last passing run started at 17:54 UTC that day and the first failing run at 19:28 UTC.
- The api-proxy builds the routing catalogue from its live Copilot
/modelsdiscovery (createRoutingCatalogueincontainers/api-proxy/routing-catalogue.js). buildRoutingCandidatesincontainers/api-proxy/routing-candidates.jsthen keeps only the models allowed by the policy. With no allowed model left, it throwsmodel_policy_violation.- The router is never consulted: in the failing run,
awf-routerbecame healthy and AWF failed while waiting for the selection.
Further evidence that the model was retired upstream, not broken by a code change:
- The commits between the last passing and the first failing run (
01b6e9af...2d770644) only touch Gemini settings, chroot GID handling, ARC/DinD docs and AI-credit pricing. None touches routing, the api-proxy catalogue or the router. - On 20 Oct, the
duplicate-code-detectorrun 39502260929 logged the api-proxy model inventory for this org. It lists 31 Copilot models, includinggpt-5.6-luna,gpt-5.6-terra,gpt-5.6-sol,gpt-5.3-codexandclaude-haiku-4.5. It does not listgpt-5.4-mini(orgpt-5.4-nano). That workflow still setsmodel: gpt-5.4-mini, and its token usage was recorded asmodel=gpt-5.6-luna. - The earlier step "Run a routed Copilot inference through AWF" still passes, because it only allows
claude-haiku-4.5, which is still offered.
Why workflows with model: gpt-5.4-mini still succeed while routing fails:
- Non-routed workflows send
gpt-5.4-ministraight to Copilot, which serves the request with the suggested alternative. gh-aw's model resolution (status=exact) checks the configured name, not the live inventory, and the api-proxy records the served model,gpt-5.6-luna. - Routing checks the policy against the live
/modelsinventory before any request is sent. A retired model is no longer listed there, so a policy that allows only that model is rejected.
Proposed fix
Change only .github/workflows/test-model-routing.yml: replace every Copilot use of gpt-5.4-mini in the "Exercise live routing, policy, and Copilot inference" job with gpt-5.6-luna. That covers the allowedModels lists, the asserted model (github-copilot/...) and wire_model, the expected-model lists, and the request bodies and success messages, in these steps:
- "Run an effort-routed Copilot inference through AWF"
- "Translate cross-endpoint Copilot sub-agent requests through AWF"
- "Route between two Copilot candidates and allow a deviating choice"
- "Assert preserved routing selection and terminal request records"
- "Run the Copilot CLI agent on the routed selection"
gpt-5.6-luna keeps each step's intent:
- It is in this org's live inventory (above).
- It is in the bundled router tables in
containers/router/routing/with every effort level. - Copilot
/modelslists it withsupported_endpoints/responsesandws:/responsesandcapabilities.supports.reasoning_effortnone, low, medium, high, xhigh, max.
So it is still a Responses-only GPT model with effort control, the counterpart to the Chat Completions-only claude-haiku-4.5. The step's effort ranking already includes max, and the endpoint logic (/responses for the GPT model, /chat/completions for Claude) is unchanged.
Out of scope, to keep this a single fix:
- The OpenAI and Anthropic native-provider job (
openai/gpt-5.4,anthropic/claude-sonnet-5) uses provider keys rather than Copilot, and isn't affected. contribution-check,copilot-token-usage-analyzer,duplicate-code-detectorandsmoke-cloud-hypervisoralso setmodel: gpt-5.4-mini. They keep running because Copilot serves those requests with another model, so they don't fail. Updating them is a separate change.
Acceptance criteria
- A push to
mainthat triggerstest-model-routing.yml, or aworkflow_dispatchonmain, passes the effort-routed step./reflectreportsstatus: selectedwithmodel: github-copilot/gpt-5.6-luna,endpoint: /responsesand a supported effort. - The steps that are skipped today run and pass, including the cross-endpoint translation in both directions and the Copilot CLI agent run.
- No Copilot step in the job references
gpt-5.4-mini.
Context
This came up while verifying #10358 (PR #10359, which ships the Go router). After that PR merges, test-model-routing is the end-to-end check for the Go router. It can't cover the effort-routing and later steps until this is fixed.
Activity
ci(routing): replace retired gpt-5.4-mini with gpt-5.6-luna in test-m…