Skip to content

test-model-routing: replace retired Copilot model gpt-5.4-mini with gpt-5.6-luna #10366

Closed

Description

@rabeckett

Problem

test-model-routing.yml has failed on every push to main since 19 Oct 2026. Each run fails at the same step: "Run an effort-routed Copilot inference through AWF" in the job "Exercise live routing, policy, and Copilot inference". AWF exits with code 78 before the agent command runs:

[INFO] Waiting for model routing selection...
[ERROR] Fatal error: InfrastructureReadinessError: Model routing failed (model_policy_violation): The model policy excludes every available model
  [cause]: RoutingFailureExitError: ... at toRoutingFailureExit (dist/routing/bootstrap.js:70:12)
  • Last passing run: 39298157165 (01b6e9af, 19 Oct 17:54 UTC).
  • First failing run: 39311298054 (2d770644, 19 Oct 19:28 UTC).
  • Latest failing run: 39468350283 (a6e3ce95). The four runs on 20 Oct fail the same way.

Because this step fails, the rest of the job (cross-endpoint sub-agents, two-candidate routing, the preserved-selection assertions, and the Copilot CLI agent run) has been skipped since 19 Oct. The live routing path on main is therefore not covered end to end.

Root cause

The step's apiProxy.allowedModels contains only github-copilot/gpt-5.4-mini. GitHub deprecated that model on 2026-10-19 (changelog, 18 Sep 2026): "GPT-5.4 mini | 2026-10-19 | suggested alternative GPT-5.6 Luna". The last passing run started at 17:54 UTC that day and the first failing run at 19:28 UTC.

  • The api-proxy builds the routing catalogue from its live Copilot /models discovery (createRoutingCatalogue in containers/api-proxy/routing-catalogue.js).
  • buildRoutingCandidates in containers/api-proxy/routing-candidates.js then keeps only the models allowed by the policy. With no allowed model left, it throws model_policy_violation.
  • The router is never consulted: in the failing run, awf-router became healthy and AWF failed while waiting for the selection.

Further evidence that the model was retired upstream, not broken by a code change:

  • The commits between the last passing and the first failing run (01b6e9af...2d770644) only touch Gemini settings, chroot GID handling, ARC/DinD docs and AI-credit pricing. None touches routing, the api-proxy catalogue or the router.
  • On 20 Oct, the duplicate-code-detector run 39502260929 logged the api-proxy model inventory for this org. It lists 31 Copilot models, including gpt-5.6-luna, gpt-5.6-terra, gpt-5.6-sol, gpt-5.3-codex and claude-haiku-4.5. It does not list gpt-5.4-mini (or gpt-5.4-nano). That workflow still sets model: gpt-5.4-mini, and its token usage was recorded as model=gpt-5.6-luna.
  • The earlier step "Run a routed Copilot inference through AWF" still passes, because it only allows claude-haiku-4.5, which is still offered.

Why workflows with model: gpt-5.4-mini still succeed while routing fails:

  • Non-routed workflows send gpt-5.4-mini straight to Copilot, which serves the request with the suggested alternative. gh-aw's model resolution (status=exact) checks the configured name, not the live inventory, and the api-proxy records the served model, gpt-5.6-luna.
  • Routing checks the policy against the live /models inventory before any request is sent. A retired model is no longer listed there, so a policy that allows only that model is rejected.

Proposed fix

Change only .github/workflows/test-model-routing.yml: replace every Copilot use of gpt-5.4-mini in the "Exercise live routing, policy, and Copilot inference" job with gpt-5.6-luna. That covers the allowedModels lists, the asserted model (github-copilot/...) and wire_model, the expected-model lists, and the request bodies and success messages, in these steps:

  • "Run an effort-routed Copilot inference through AWF"
  • "Translate cross-endpoint Copilot sub-agent requests through AWF"
  • "Route between two Copilot candidates and allow a deviating choice"
  • "Assert preserved routing selection and terminal request records"
  • "Run the Copilot CLI agent on the routed selection"

gpt-5.6-luna keeps each step's intent:

  • It is in this org's live inventory (above).
  • It is in the bundled router tables in containers/router/routing/ with every effort level.
  • Copilot /models lists it with supported_endpoints /responses and ws:/responses and capabilities.supports.reasoning_effort none, low, medium, high, xhigh, max.

So it is still a Responses-only GPT model with effort control, the counterpart to the Chat Completions-only claude-haiku-4.5. The step's effort ranking already includes max, and the endpoint logic (/responses for the GPT model, /chat/completions for Claude) is unchanged.

Out of scope, to keep this a single fix:

  • The OpenAI and Anthropic native-provider job (openai/gpt-5.4, anthropic/claude-sonnet-5) uses provider keys rather than Copilot, and isn't affected.
  • contribution-check, copilot-token-usage-analyzer, duplicate-code-detector and smoke-cloud-hypervisor also set model: gpt-5.4-mini. They keep running because Copilot serves those requests with another model, so they don't fail. Updating them is a separate change.

Acceptance criteria

  • A push to main that triggers test-model-routing.yml, or a workflow_dispatch on main, passes the effort-routed step. /reflect reports status: selected with model: github-copilot/gpt-5.6-luna, endpoint: /responses and a supported effort.
  • The steps that are skipped today run and pass, including the cross-endpoint translation in both directions and the Copilot CLI agent run.
  • No Copilot step in the job references gpt-5.4-mini.

Context

This came up while verifying #10358 (PR #10359, which ships the Go router). After that PR merges, test-model-routing is the end-to-end check for the Go router. It can't cover the effort-routing and later steps until this is fixed.

Activity

  1. added a commit that references this issue on Oct 21, 2026
    fe55b45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions