Add non-relay Anthropic MPS e2e tests - #472
Draft
masonc08 wants to merge 1 commit into
Draft
Conversation
TestAnthropicNonRelayMps (binary-free, direct Anthropic Messages calls through the pinned CI MPS): a real Haiku inference (200 + model echoed back), allowlist enforcement (a disallowed model -> 403 "not in the allowed models list"), and discovery (listed, usable-for-claude, non-relayed). Also turns the two provider-launch cases into explicit model-selection checks via ucode's resolver: a bare launch defaults to the only servable tier (Haiku), and --model haiku resolves to the one declared target. The MPS exact-matches its allowlist, so a successful launch is itself proof Haiku was used. Verified live on the CI workspace: the three binary-free tests pass and both resolver assertions pass. The launch step can't run on a box with OS-managed Claude settings (write_tool_config refuses the override); it runs in the CI e2e job. Co-authored-by: Isaac <no-reply@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Stacked on #471 (the CI-MPS pins), which this depends on for the
CI_ANTHROPIC_MPS/CI_ANTHROPIC_MODELconstants — hence the base is that branch, notmain. Retarget tomainonce #471 lands.Adds e2e coverage for the non-relay (api-key) Anthropic MPS path:
TestAnthropicNonRelayMps— binary-free, direct Anthropic Messages calls through the pinned CI MPS:test_direct_inference_routes_to_haiku— real inference: HTTP 200 + responsemodel == claude-haiku-4-5-20251001.test_rejects_model_outside_allowlist— a disallowed model → 403 "not in the allowed models list".test_mps_is_listed_usable_and_nonrelayed— discovery: listed, usable-for-claude, non-relayed.test_launch_claude_through_provider— no--modeldefaults to the only servable tier (Haiku).test_launch_claude_model_haiku_through_provider—--model haikuresolves to the declared Haiku target.The MPS exact-matches its allowlist, so a successful launch is itself proof Haiku was used.
How do you know it works?
Verified live on the CI workspace: the three binary-free tests pass and both resolver assertions pass. The binary launch step runs in the CI e2e job (it can't run where OS-managed Claude settings are present, e.g. a dev box).
This pull request and its description were written by Isaac.