Skip to content

Connection test probes a hard-coded premium model when the connection has no enabled model (OpenRouter → anthropic/claude-sonnet-5) #5493

Description

@sunrioa

What happened

I created an OpenRouter connection, had not enabled any model yet, and clicked "Test connection".
The OpenRouter activity log then showed a request to Claude Sonnet 5 that I never selected
anywhere in Maka: 9 prompt tokens, 13 completion tokens, $0.000148, app "Unknown".

Maka's own usage statistics have no record of that call (connection tests are not written to the
usage tables), so it took a while to trace it back to the connection probe.

Root cause, verified against main @ 5b9db1c:

  • resolveConnectionTestModel in packages/runtime/src/test-connection.ts picks the probe model in
    this order (line 94): explicit model → the connection's enabled models → the provider's
    hard-coded fallbackModels → the connection's discovered inventory.
  • For OpenRouter, fallbackModels is openrouterModelIds
    (packages/core/src/provider-registry.ts lines 516–519, wired at line 1441), whose first entry
    is anthropic/claude-sonnet-5.
  • Because the hard-coded list is consulted before the discovered inventory, a fresh connection
    with no enabled model is probed with Sonnet 5 even when Maka has already fetched the account's
    inventory (444 models in my case, including several :free ones).
  • The probe body is Hi with max_tokens: 16 (lines 420–421 / 461–462), which matches the
    9 / 13 token counts exactly. At OpenRouter's $2/M input and $10/M output for Sonnet 5 the cost is
    9 × 0.000002 + 13 × 0.00001 = $0.000148, the amount in the log.

This contradicts the principle documented right above that function (line 75, referring to #1584):
the probe must not be redirected onto a model the user never chose. The hard-coded fallback does
exactly that for every new connection.

Expected instead:

  1. Probe an enabled model when one exists (already the case).
  2. If none is enabled but an inventory was fetched, pick from that inventory, preferring a :free
    model (or, once pricing is stored, the cheapest one) over the hard-coded list.
  3. Only fall back to the hard-coded list when there is no inventory at all (legacy connections).
  4. Optionally tell the user which model the test is about to use, or ask them to pick one first.

Impact beyond the (small) unexpected charge: OpenRouter keys can carry a model allow-list, and some
accounts don't have every provider enabled. When the hard-coded model is rejected, Maka records the
result as a failed connection although the key is fine.

How to reproduce

  1. Settings → Connections → add an OpenRouter connection with a valid API key.
  2. Fetch the model list, but do not enable any model.
  3. Click "Test connection" (the command-palette action "Test connection · OpenRouter" behaves the same;
    neither passes a model).
  4. Open https://openrouter.ai/activity: a request to anthropic/claude-sonnet-5 with 9 prompt /
    13 completion tokens appears, sent by Maka's probe.
  5. Settings → Usage in Maka shows no corresponding call.

Environment

  • Maka Desktop 0.2.0, local packaged build from a branch based on main (behaviour verified unchanged on main @ 5b9db1c)
  • Runtime Host: local
  • macOS 26.5 (Darwin 25.5.0), Apple Silicon
  • Provider: OpenRouter, connection created with the model inventory fetched and no model enabled

Logs, screenshots, or additional context

OpenRouter activity row: Sep 17, 10:01 local · Claude Sonnet 5 · "Claude Platform on AWS" ·
9 tok / 13 tok · $0.000148.

Image

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions