Pin the e2e provider-launch tests to fixed CI MPSes (Haiku / Nano) - #471
Open
masonc08 wants to merge 4 commits into
Open
Pin the e2e provider-launch tests to fixed CI MPSes (Haiku / Nano)#471masonc08 wants to merge 4 commits into
masonc08 wants to merge 4 commits into
Conversation
TestModelProviderLaunch picked whichever non-relayed service listed first, so the inference it ran was non-deterministic (any provider, any model) and its cost uncontrolled. Pin each path to a dedicated CI MPS that exposes only its cheapest model, so runs are reproducible and cost ~a cent each: - claude -> main.ucode.ci_e2e_anthropic_nonrelay_mps (api-key Anthropic, Haiku 4.5 only); route_root_model pins ANTHROPIC_MODEL to the one allowed target. - codex -> main.ucode.ci_openai_mps (api-key OpenAI, gpt-5-nano only); pinned via state["codex_default_model"]. Each skips (rather than fails) when its MPS is absent, so e2e runs against other workspaces still work. Both were verified to route 200 on the CI workspace. Co-authored-by: Isaac <no-reply@databricks.com>
Both its callers (the claude and codex provider tests) now pin a fixed CI MPS, so the "pick the first non-relayed service" helper is unused. Its three imports stay used by TestModelProviderServicesDiscovery, so nothing is orphaned. Co-authored-by: Isaac <no-reply@databricks.com>
masonc08
marked this pull request as ready for review
September 2, 2026 21:24
masonc08
enabled auto-merge (squash)
September 2, 2026 21:40
masonc08
force-pushed
the
masonc08/e2e-pin-ci-anthropic-mps
branch
from
September 2, 2026 21:48
cb3fb8e to
7c80f3e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
TestModelProviderLaunchpicked whichever non-relayed MPS listed first, so the inference it ran was non-deterministic (any provider, any model) and its cost uncontrolled. Pin each path to a dedicated CI MPS that exposes only its cheapest model — reproducible, and ~a cent per run.main.ucode.ci_e2e_anthropic_nonrelay_mps(api-key Anthropic, Haiku 4.5 only).route_root_modelpinsANTHROPIC_MODELto the one allowed target, since Claude Code's default tier isn't in the allowlist.main.ucode.ci_openai_mps(api-key OpenAI, gpt-5-nano only), pinned viastate["codex_default_model"].Each skips (not fails) when its MPS is absent, so e2e runs against other workspaces still work.
How do you know it works?
Both models verified to route HTTP 200 through their MPS on the CI workspace (
eng-ml-inference-team-us-east-1):claude-haiku-4-5-20251001via/ai-gateway/anthropic/v1/messages,gpt-5-nanovia/ai-gateway/codex/v1/responses.ruffclean; all 31 e2e tests still collect. The tests themselves run in CI'se2ejob.This pull request and its description were written by Isaac.