diff --git a/brain/knowledge/ai-intelligence/ai-agents.md b/brain/knowledge/ai-intelligence/ai-agents.md index b791144568ca..42d478fd9b21 100644 --- a/brain/knowledge/ai-intelligence/ai-agents.md +++ b/brain/knowledge/ai-intelligence/ai-agents.md @@ -19,7 +19,7 @@ A flow step type (the `run_agent` action of `@activepieces/piece-ai`) that runs | `AGENT_BUILDER` | `conversation.modelName` | tier id | | `FLOW_STEP` | `agent.published.modelName`, or the step's own `input.aiProviderModel.model` | concrete id | -Three things follow. `run_agent` has two modes: pointed at a saved agent (`agentId` set) the step sends no model and the agent row decides; inline, it sends `input.aiProviderModel.model`. An agent row carries two configs: agent-page chat reads `agent.draft`, a `run_agent` step reads `agent.published` (via `resolvePublishedAgent`), so the same agent can already run different models through different doors. The fast model crosses every door: `firstStepUsesFastModel` (`run-agent-turn.ts`) runs the first step of almost every turn on the fast tier, flow steps included, whatever the user picked. `execute-agent-run.ts` persists `config.tier.id` onto `conversation.modelName` when the job carried no `modelName`, so conversations already hold tier ids today (inline `run_agent`, eval runs, every `AGENT_BUILDER`), read back by `chat-tool-billing.ts`, `chat-analytics-sync.ts` and `pinnedModelOf`. +Three things follow. `run_agent` has two modes: pointed at a saved agent (`agentId` set) the step sends no model and the agent row decides; inline, it sends `input.aiProviderModel.model`. An agent row carries two configs: agent-page chat reads `agent.draft`, a `run_agent` step reads `agent.published` (via `resolvePublishedAgent`), so the same agent can already run different models through different doors. The fast model crosses every door: `firstStepUsesFastModel` (`run-agent-turn.ts`) runs the first step of almost every turn on the fast tier, flow steps included, whatever the user picked. `execute-agent-run.ts` persists `config.tier.id` onto `conversation.modelName` when the job carried no `modelName`, so conversations already hold tier ids today (inline `run_agent`, eval runs, every `AGENT_BUILDER`), read back by `chat-tool-billing.ts` and `chat-analytics-sync.ts`. **A persisted tier id is not a pin.** `pinnedModelOf` (`rpc/rpc-shared.ts`) ignores `conversation.modelName` for a conversation with an agent: that value is the resolved default, and once the console removes that tier it would reach a catalogue-less own-key provider as a literal model id. A saved agent's run model is the agent's own half (draft for AGENT, published for FLOW_STEP); only an agentless inline step reads `conversation.modelName`, which is the step's configured model. ### Tool types (AgentTool discriminated union) diff --git a/brain/knowledge/ai-intelligence/ai-providers.md b/brain/knowledge/ai-intelligence/ai-providers.md index 6c9620816dea..9718d93288f3 100644 --- a/brain/knowledge/ai-intelligence/ai-providers.md +++ b/brain/knowledge/ai-intelligence/ai-providers.md @@ -58,9 +58,12 @@ the release's bundled tiers serve instead. `modelTierCatalog.current(surface)` (`packages/server/utils/src/model-tier-catalog.ts`) is the single accessor: synchronous, never throws, serves the cached file or `ACTIVEPIECES_CHAT_TIERS`, and refreshes in the background every 15 minutes with a 5-minute back-off after a failure. `warmUp()` runs at API boot from -`aiProviderService.setup()`, bounded to 2 s so a dead CDN cannot block boot. Today only `warmUp()` calls it: the -resolution helpers in `ee/agent/agent-model-resolution.ts` (`findTier`, `resolveTier`, `FAST_TIER_ID`) still read -the bundled list and are the next thing to move. `AP_MODEL_TIERS_URL` overrides +`aiProviderService.setup()`, bounded to 2 s so a dead CDN cannot block boot. The resolution family in +`ee/agent/agent-model-resolution.ts` (`findTier`, `resolveTier`, `resolveNamedModelId`, `resolveModelIdForProvider`, +`defaultModelIdForProvider`, `resolveModelIdForAnalytics`) and the fast-model helpers in `agent-helpers.ts` read it, +every one with a required `surface` argument. `surfaceOf({ source })` in that file is the only place the run +source maps to a list (FLOW_STEP and AGENT to flow, CHAT and AGENT_BUILDER to chat), and `agent-config-rpc.ts` +derives `namesItsOwnModel` from it. `AP_MODEL_TIERS_URL` overrides the URL. The file is published from the console in `ap-analytics`; see decision 000041. ### Gotchas - **A catalog change takes up to ~2 days to reach a dropdown, through three caches in series.** CDN edge (`--cache-control max-age=3600`, 1h) → the server's in-memory catalog (`CATALOG_TTL_MS`, 24h) → `modelsCache` (flushed by the nightly `cron.schedule('0 0 * * *')`). So "I republished but the UI still shows the old price" is expected, not a bug; restarting the API short-circuits the two in-process layers. The edge TTL is deliberately 1h and not the 604800 that `publish-embed-sdk.yml` uses — that path is version-stamped, ours is a stable key rewritten in place, so a week-long edge cache would pin stale prices for a week. @@ -70,17 +73,19 @@ the URL. The file is published from the console in `ap-analytics`; see decision - **A new provider needs an entry in the generator's `MODELS_DEV_PROVIDER` map or it silently ships with no metadata.** models.dev provider ids do not match ours: `bedrock` → `amazon-bedrock`, `qwen` → `alibaba`, `moonshot` → `moonshotai`, `activepieces` → aliased onto `openrouter` at lookup time. The six OpenAI-compatible vendors were merged before the map was updated and produced exactly this — the run prints `no upstream source: …`, which is the thing to read after adding a provider. `cloudflare-gateway` and `custom` legitimately have no source. - **The catalog object must be published before the code that reads it ships.** There is no bundled copy, so until a `workflow_dispatch` run puts it on the CDN, every install — including local dev — shows no metadata at all. - **The pricing console shows bundled constants when no file is published, and says nothing.** `overview()` in `ai-pricing-service.ts` has three states, and two of them render the same numbers: **absent** (`NoSuchKey`) silently seeds the editor from `DEFAULT_PRICING_DRAFT`, the console's own checked-in copy of the tiers and weights, while **unreadable** (any other S3 error) shows those same constants *plus* a banner and blocks publishing. Only **present** is live data. So a screen full of plausible tiers is not evidence that anything was ever published — `curl -I https://cdn.activepieces.com/ai/pricing.json` is. The absent state is deliberate, it is how the first publish bootstraps, but it means "I can see the tiers" and "the prices are live" are unrelated facts. Worse, **absent and misconfigured are the same state**: `cdnClient.getJson` catches `NoSuchKey` and returns `undefined`, so a wrong bucket or an endpoint with the bucket in its path reads as "nothing published yet" and hides behind the defaults. The tell is the History tab — `list` does *not* catch `NoSuchKey`, so the same misconfiguration that reads as empty on the pricing page returns a 500 there. A 500 on history with a normal-looking pricing page means the S3 addressing is wrong, not that the history is empty. -- **Deleting the live tier file is safe for customers and confusing for everyone else.** `ai/pricing.json` was deleted on 2026-09-24 to drop a leftover test tier, and Activepieces kept serving the right tiers throughout: the 404 makes `fetchTiers` throw, the failure is logged and backed off, and `current()` keeps serving the last good copy the process fetched; only a process that never fetched one, a fresh boot during the outage, runs the `ACTIVEPIECES_CHAT_TIERS` shipped with the release. Two things do change. Each server logs 'Failed to refresh the AI model tiers file' (or 'Failed to load…' on a process that never got a copy) on every failed attempt, and attempts happen at boot and whenever `current()` is called with a stale or missing copy, at most once per five minutes (`FAILURE_BACKOFF_MS`), so a permanent 404 is retried the same way as a blip. Today only `warmUp()` calls it, so that is one line per boot. And the console reads the missing key as **absent**, so the next publish starts again at `version: 1` while the `ai/pricing/history/` objects from the old numbering are still there — the version number is `live.version + 1`, not a counter, so it is only meaningful between two publishes that never lost the live key. To remove one bad tier, edit and publish; do not delete the object. +- **Deleting the live tier file is safe for customers and confusing for everyone else.** `ai/pricing.json` was deleted on 2026-09-24 to drop a leftover test tier, and Activepieces kept serving the right tiers throughout: the 404 makes `fetchTiers` throw, the failure is logged and backed off, and `current()` keeps serving the last good copy the process fetched; only a process that never fetched one, a fresh boot during the outage, runs the `ACTIVEPIECES_CHAT_TIERS` shipped with the release. Two things do change. Each server logs 'Failed to refresh the AI model tiers file' (or 'Failed to load…' on a process that never got a copy) on every failed attempt, and attempts happen at boot and whenever `current()` is called with a stale or missing copy, at most once per five minutes (`FAILURE_BACKOFF_MS`), so a permanent 404 is retried the same way as a blip. And the console reads the missing key as **absent**, so the next publish starts again at `version: 1` while the `ai/pricing/history/` objects from the old numbering are still there — the version number is `live.version + 1`, not a counter, so it is only meaningful between two publishes that never lost the live key. To remove one bad tier, edit and publish; do not delete the object. - **`CDN_S3_*` and `AP_S3_*` are two different buckets on two different providers, and mixing them puts public objects in customer data.** `cdn.activepieces.com` is a **DigitalOcean Spaces** bucket, `activepieces-cdn` in `fra1`, fronted by Cloudflare — a 200 from it carries `x-rgw-object-type` and `x-do-cdn-uuid`, which is how you tell. That is where `ai/model-catalog.json`, `ai/pricing.json` and `sdk/embed/*.js` live, and the credentials are the `CDN_S3_*` GitHub secrets. `AP_S3_BUCKET` is something else entirely: the **Cloudflare R2** file-storage bucket (`ap-files-prod` in production) holding run logs, step checkpoints, webhook payloads and flow bundles. The names are similar enough that the pricing console was once configured with the R2 file-storage key by mistake, which would have written a `public-read` `ai/pricing.json` into the customer-data bucket while `cdn.activepieces.com` stayed 404. When wiring anything that publishes to the CDN, confirm the endpoint is the Spaces region origin (`https://fra1.digitaloceanspaces.com`, never with the bucket in the path — the SDK appends `Bucket` itself and the doubled path returns `NoSuchKey` on every call). - **The weekly sync workflow can fail and nobody is told.** `sync-model-catalog.yml` has no alert on failure, and the only signal that it broke is a 404 on the object, which every reader swallows. Its run on 2026-08-31 failed with `InvalidAccessKeyId` — the `CDN_S3_ACCESS_KEY_ID` secret in this repo does not match a live key — so `ai/model-catalog.json` has never actually landed. Check `gh run list --workflow=sync-model-catalog.yml` before believing the CDN is the problem. **That same `CDN_S3_*` credential also publishes the AI tier file `ai/pricing.json` from `ap-analytics`** (decision [000039](../decisions/000041-ai-model-tiers-are-published-to-the-cdn-from-a-second-console.md)), so one dead key silently breaks model metadata *and* the published tiers at once: the console hides model costs (`model-catalog-service.ts` logs 'model prices will be hidden' and returns an empty list) while Activepieces keeps serving the tiers bundled in the release. Two features, one credential, no alarm on either. - **A fresh Cloud platform cannot connect any AI provider, and the UI says nothing about why.** The admin page at `/platform/ai` sets `allowWrite = platform.plan.aiProvidersEnabled`, and that column defaults to **false** on the `free` plan (migration `1776…AddDefaultToAiProvidersEnabled`), so the connect button is simply absent rather than disabled-with-a-reason. On Cloud the flag is owned by Autumn (`autumn-utils.ts` lists it among the synced features), so there is nothing to configure locally and no `DEV_ENTERPRISE_PLAN` escape hatch on main. For local Cloud testing, flip it directly: `UPDATE platform_plan SET "aiProvidersEnabled" = true`. Then connect a **BYO key**, not the managed `ACTIVEPIECES` provider — per *Provider visibility* above, the managed row is hidden whenever `OPENROUTER_PROVISION_KEY` is unset, which is the normal local state, and a hidden provider makes `getChatProvider()` return null. Anything gated on a chat provider (chat itself, personalization research) stays silently disabled until a BYO row exists. - **Since the AI steps moved to the worker, the server sees every step's model id, and `ai-execute-controller` is the one place to act on it.** The piece POSTs `/v1/ai/execute` with `provider` and `modelId`, the controller builds the `EXECUTE_AI` job from that body, and the worker runs whatever model id the job carries. So anything that needs to rewrite or validate the model a flow step is about to use belongs there, not in the piece — one server-side edit covers all five AI steps, and an old pinned piece bundle keeps working because it only forwards the string it stored. This is what makes storing a tier id in a step cheap: the bundle never has to understand it. Note the agent step does **not** come through here — it resolves separately over `getAgentConfig`, in `agent-config-rpc`. Two conditions sit under that. **An old bundle only forwards the string from `0.11.0` up** — `AI_PIECE_COST_BILLING_VERSION`, the first version whose `ai-step.ts` POSTs `v1/ai/execute`. A step pinned below it runs the pre-worker code that calls the provider from the sandbox, so a tier id there would reach OpenRouter as the literal string. That case is already closed: `migrate-v26-ai-piece-cost-billing` bumps every saved `piece-ai` step below `0.11.0` up to it, which is why a later migration can rewrite what a step stores without touching `pieceVersion` for the runtime. **And resolve on the API, never in the worker**: `ExecuteAiRequest.modelId` is a bare `z.string()` so a tier id passes validation, `getEffectiveProviderAndModel` and `aiUtils.buildWebSearchToolsOrThrow` both read `data.modelId` as a real model, and the `resolveAiProvider` RPC does not carry the model at all, so using it as the seam would mean widening the worker contract. - **Managed billing can only price an OpenRouter call, and a call it cannot price is simply not billed.** `openRouterCostOf` (`server/utils/activepieces-ai-cost.ts`) reads `providerMetadata['openrouter'].usage.cost`, with one fallback that GETs `openrouter.ai/api/v1/generation?id=` and takes `total_cost`. If neither answers, `createBillingInfo` returns `undefined` and the turn is never reported — it runs, the customer keeps the output, and nothing is charged. Nothing logs "unbilled". This is what actually welds the managed provider to OpenRouter, more than the client does: `create-language-model.ts` shares one branch for `OPENROUTER` and `ACTIVEPIECES`, every tier `modelId` is an OpenRouter id, and `openrouter-api.ts` mints the per-platform key from `OPENROUTER_PROVISION_KEY`. So putting managed credits on any other gateway needs a second cost extractor and a second generation lookup **first**, or every call through it bills zero. For reference, no Vercel AI Gateway code exists here at all — the only gateway in the repo is Cloudflare's (`CLOUDFLARE_GATEWAY`, the `ai-gateway-provider` package), and `@ai-sdk/gateway` appears only as a transitive dep of `ai` that nothing imports. A BYOK user can already reach Vercel's gateway today through `AIProviderName.CUSTOM`, which takes a base URL plus hand-typed model ids. -- **A tier id that no longer exists resolves two different ways, and only the run source decides which.** `getAgentConfig` branches on `namesItsOwnModel` (true for `AgentRunSource.FLOW_STEP` and `AGENT`): those go through `resolveNamedModelId`, which **throws** `The model "X" is not available on Activepieces AI credits` for an id it cannot find, while every other source goes through `resolveModelIdForProvider` → `resolveTier`, which **silently returns the default tier**. So deleting a published tier moves existing chats onto the default with no error and no notice, and that is genuinely safe only while flow steps and agents still store concrete model ids. The moment a step stores `premium` it takes the throwing branch and every run of that flow fails, because the candidate list is model ids and never contains a tier id. Whoever lands the tier-id switch has to decide what a removed tier means before the console keeps telling admins that removing one is safe. +- **A tier id that no longer exists resolves to the default tier of its surface, on every door, with a warning.** `resolveNamedModelId` (the path FLOW_STEP and AGENT take) tells a tier id from a model id by the slash rule: a bare slug is a tier id and goes through `resolveTier`, which returns the surface default for an id the file no longer carries and logs `Tier is no longer published; running the surface default` with the tier id and surface; a slashed id is a managed model id and is still refused with `The model "X" is not available on Activepieces AI credits` unless the release ships it or a published tier on that surface carries it. Every other source goes through `resolveModelIdForProvider` → `resolveTier`, the same default. So removing a tier from the file never fails a flow or an agent; it moves them onto the default until they are re-picked. The console may keep saying that removing a tier is safe. Decided 2026-09-26 with PR 2 of the configurable tiers work. - **The console's Publish writes the live key directly; there is no draft, no staging and no filter.** `ai-pricing-service.ts` in `ap-analytics` writes `LIVE_PRICING_KEY` on Publish, and its "add tier" row seeds `id: tier-N`, `label: New tier`, so a half-finished experiment is one click from every server's cached tier list, with only the CDN `max-age=300` plus the 15-minute in-process TTL in between. Test against a local file with `AP_MODEL_TIERS_URL` (add the host to `AP_SSRF_ALLOW_LIST` for loopback), never by publishing. - **The AI piece's managed dropdown only ever offers tier models, so every stored managed model id came from a tier pick.** `props.ts` (`packages/pieces/community/ai/src/lib/common/props.ts`) builds its options from `ACTIVEPIECES_CHAT_TIERS`, the constant compiled into the release, in two places: `managedModelLabel` supplies the label, and the same lookup is the `.filter` that decides which models are offered at all. There is no way to pin a model deliberately through it. Two consequences. A re-pointed tier is invisible to the AI piece in both directions: a saved step keeps its concrete model id, and a new step cannot pick the new model because it matches no bundled tier. And any migration from stored model ids to tier ids has a trivially complete mapping, since each stored id is one of the three bundled tier model ids of the release it was saved on. Until the dropdown emits tier ids, the AI piece only ever follows the release. - **`AIProviderModel` is declared twice, and a field added to one half is stripped on the wire.** There is one in `core/piece-types` (what pieces import, since they cannot import `@activepieces/shared`) and one in `core/shared` (what the server and the response schema use). They are not re-exports of each other — today the shared copy carries `metadata` and the piece-types copy does not. Fastify serialises the response against the *shared* schema and drops every key it does not declare, so adding a field to the piece-types copy alone type-checks in the piece, type-checks in the server, and arrives as `undefined`. A published tier label was nearly shipped that way. Any new field on a provider model needs both files. - **No image model belongs to a chat tier, so a tier-based filter empties the image picker.** The managed provider is OpenRouter underneath and `openrouter-provider.listModels` tags a model `IMAGE` when its `architecture.output_modalities` says so — `ACTIVEPIECES` is deliberately absent from `NO_IMAGE_GENERATION_PROVIDERS`, and `google/gemini-2.5-flash-image` and friends are real managed models. But `ACTIVEPIECES_CHAT_TIERS` is three *chat* tiers, so no image model can ever match one. Any filter of the form "a managed model must belong to a tier" therefore drops every image model silently, leaving `aiProps({ modelType: 'image' })` with an empty dropdown and no error. Gate that filter on `modelType === 'text'`. +- **The api test suite runs on the bundled tiers, never the live file.** `packages/server/api/vitest.setup.ts` points `AP_MODEL_TIERS_URL` at `model-tiers.invalid`, so boot warm-up fails fast and every test resolves against `ACTIVEPIECES_CHAT_TIERS`. CI runners have egress, and the first time a resolver read the file in CI the live `defaultTierId` was `fast`, which turned a test asserting the release default (Expert) red with no code change behind it. A test that wants the published shape mocks `modelTierCatalog` (`test/unit/app/ee/agent/model-tier-fixture.ts`); an integration test that boots the app cannot be handed a file. +- **A published tier without `nativeModelId` keeps every BYOK key on the release's model for that tier id.** `nativeModelIdFor` (`ee/agent/agent-model-resolution.ts`) takes the tier's own `nativeModelId`, else the `nativeModelId` of the bundled `ACTIVEPIECES_CHAT_TIERS` entry with the same id, else null, which sends a curated provider to its first curated model and a catalogue-less provider to the first text model its key lists. The fallback is keyed by tier id only, so two lists sharing an id resolve to the same native model on every own-key provider, whatever their managed models are. The BYOK tier map (PR 6 of the configurable tiers work) is the real fix. - **`premium` pairs `modelId: anthropic/claude-opus-4.8` with `nativeModelId: claude-opus-4-7` on purpose.** The managed tier runs Opus 4.8 through OpenRouter; a customer's own Anthropic key runs Opus 4.7. The reason is cost on the customer's key: Heavy on a BYOK key must not silently move the customer onto the most expensive model without them choosing it. The console's `defaults.ts` carries the same pair. Do not "fix" either copy. - **Every provider auth schema is written twice, and only one copy is the one the server reads.** `packages/core/utils/src/lib/ai-provider-credentials.ts` (plain zod) backs the API and the web app; `packages/core/piece-types/src/lib/ai-providers.ts` (`zod/mini`, so `.check(z.minLength(1))` and `z.optional(x)` instead of `.min(1)` / `.optional()`) is what `@activepieces/pieces-framework` re-exports to pieces and the engine. Adding a credential field to only one leaves the other silently stripping it — zod objects drop unknown keys, so the upsert still returns 200 and the failure surfaces later as a provider-side auth error. Change both. @@ -113,7 +118,7 @@ the URL. The file is published from the console in `ap-analytics`; see decision - **The OpenAI-compatible provider can speak the Responses API, and `@ai-sdk/openai-compatible` is not how you get there.** That SDK exposes only `chatModel` (`/chat/completions`) and `completionModel` — no responses model in any version. The route is `@ai-sdk/openai`, which accepts a custom `baseURL` + `headers` and exposes `.responses()`; the Cloudflare gateway's openai branch already did this before the `apiStyle: 'chat' | 'responses'` config field existed. Header precedence is the subtlety: the SDK builds `{ Authorization: Bearer , ...options.headers }`, so a caller header wins on `Authorization` (the Bedrock case, where the key *is* the bearer token) but a custom `apiKeyHeader` such as `x-api-key` leaves the SDK's default `Authorization` riding along. `withUserAgentSuffix` drops `undefined` values, but `OpenAIProviderSettings.headers` is typed `Record`, so the way to strip it is the header-stripping `fetch` the Cloudflare branch uses — no cast needed, and worth reaching for straight away rather than writing the duplicate header up as an accepted limitation. Strip only when `apiKeyHeader` is not itself Authorization, compared **case-insensitively**: HTTP header names are, and an admin can type `authorization`. Confirmed shape, since it is easy to assert on the wrong key: the SDK lowercases everything, so a mis-set provider really does emit `{authorization: 'Bearer ', 'x-api-key': ''}` — the credential twice. - **AWS's OpenAI-compatible surface is two endpoints, and "Bedrock Mantle" is a real AWS product name, not a customer's nickname.** `bedrock-runtime.{region}.amazonaws.com/openai/v1` (AWS-recommended) and `bedrock-mantle.{region}.api.aws/v1` both serve Chat Completions *and* Responses, both authenticate with a Bedrock API key as `Authorization: Bearer`, and both take an unmodified OpenAI SDK. Mantle alone adds server-side tools and web search, `background=true` async inference, and Projects/Workspaces; bedrock-runtime alone has Guardrails, cross-Region inference and intelligent prompt routing. So a ticket claiming "Mantle only supports responses" is half right — it serves chat/completions too, and the real loss is the Responses-only capabilities. - **Vertex is a different door onto the same models, not a different vendor.** The `GOOGLE` provider is the Gemini Developer API (`generativelanguage.googleapis.com`, static API key); `VERTEX` is Vertex AI (`{region}-aiplatform.googleapis.com/v1beta1/projects/{project}/locations/{region}/publishers/google`, service-account OAuth2 with ~1h tokens) — the same Gemini models, but billed to the customer's GCP account and inheriting IAM, VPC-SC, CMEK, data residency and audit logs, plus Model Garden's Claude/Llama/Mistral. That rotating token is why the CUSTOM provider can never reach Vertex: `buildOpenAICompatibleHeaders` injects one static header, which is why the workaround used to be a LiteLLM proxy. `@ai-sdk/google-vertex` does the JWT-to-token exchange itself, so the native provider needs no proxy. Vertex models are entered by hand (`MANUAL_MODEL_PROVIDERS`) on purpose: availability is project-, region- and Model-Garden-specific, and a `publishers/google` listing would miss exactly the third-party models that motivate choosing Vertex. **But "same models" only holds for Gemini.** Model Garden also serves Anthropic, Meta, Mistral and xAI, and those do not share Gemini's API surface — `@ai-sdk/google-vertex` ships separate `/anthropic`, `/maas` and `/xai` entry points for exactly that reason, so one `createVertex(...)` call does not cover Vertex. Because models are typed in by hand, a Claude id reaches the factory as an ordinary string and, unrouted, gets built with the Gemini client and fails at the endpoint. `create-language-model.ts` routes ids containing `claude` to `createVertexAnthropic`; `vertexClientFor` picks the client from the id shape, so all three are routed. Because the shapes are unambiguous the check is a lookup, not a tuned heuristic — and note the publisher prefix has to be tested *before* the `claude` one, or a MaaS path containing "claude" routes to the Anthropic client. Routing is cheap because the id shapes are distinct, not because a heuristic was tuned: `gemini-2.5-pro` is bare, Anthropic carries an `@date` (`claude-3-5-sonnet@20241022`), and MaaS is publisher-prefixed and suffixed (`meta/llama-4-scout-17b-16e-instruct-maas`). The publisher prefix is a stronger discriminator than the `includes('claude')` match currently shipped, and it looks like an argument for listing models rather than taking them by hand — but it is not, see below: the listing is scoped to a single publisher, so Model Garden ids stay hand-typed and the string matching stays with them. **Listing was considered and rejected**, and the reason is worth keeping so it does not get reopened, and **the response shape does not need a GCP account to learn** — every Google API publishes an unauthenticated discovery document: `curl -s 'https://aiplatform.googleapis.com/$discovery/rest?version=v1beta1'` returns ~5MB of JSON whose `schemas` are the authoritative request/response types. `publishers.models.list` returns `{ publisherModels: PublisherModel[], nextPageToken }`, and `PublisherModel` is `{ name: 'publishers/google/models/', versionId, versionState (STABLE|UNSTABLE), launchStage (GA|PUBLIC_PREVIEW|PRIVATE_PREVIEW|EXPERIMENTAL), openSourceCategory, supportedActions, predictSchemata, frameworks, parent }`. **There is no `displayName` on it** — the one in `google-vertexai`'s piece comes from `@google/genai`'s `Model` type, which is a different shape, so copying that code gives every row a blank label with nothing failing. `displayName` exists only on the nested `parent`, which is the base model a tuned model derives from. The decisive fact is in that same doc: `publishers.models.list` takes `parent` matching `^publishers/[^/]+$` — **one publisher per call**. Listing everything therefore means hardcoding the set of publishers to enumerate, which is the same shape of hardcoding it was meant to remove, and Model Garden models would still need hand entry, leaving two mechanisms where there is now one. `VERTEX` stays in `MANUAL_MODEL_PROVIDERS` alongside `CUSTOM` and `CLOUDFLARE_GATEWAY`, which fits: Vertex model availability really is per-project. Reach for the discovery doc before a live probe on any `*.googleapis.com` integration; a curl against the real endpoint needs **billing attached to the project even for a read** (`403 BILLING_DISABLED`), so it is the slower path and it is not free to set up. The `x-goog-user-project` header is not optional when probing with a personal login: without it aiplatform answers `403 SERVICE_DISABLED` naming `consumer: projects/32555940559`, which is Google's shared gcloud CLI project rather than yours, and the message talks about quota projects rather than the missing header. This is a user-credential quirk only — a service account carries its own `project_id`, so the provider itself never hits it. **The multi-vendor story is text only.** Vertex's image models are Google's own Imagen (`imagen-3.0-*`, `imagen-4.0-*`), not Model Garden's third parties, and `createVertex(...)` also exposes `imageModel`, `video`, `speech`, `transcription` and `textEmbeddingModel` — none of which we wire today. Imagen is wired (`createVertex(...).imageModel()`, with a `VERTEX` case in the piece's image switch, so `VERTEX` is no longer in `NO_IMAGE_GENERATION_PROVIDERS`); video, speech, transcription and embeddings are not. Embeddings still have no `DEFAULT_EMBEDDING_MODELS` entry, so that path paths fail with an explanatory message instead of reaching a switch that has no case for it. -- **Chat-tier resolution answers "which model" from three places, and only one of them is the key.** `resolveModelIdForProvider` (`ee/agent/agent-model-resolution.ts`, spread into `agentHelpers`) now prefers the resolved key's own configured text models when its config carries a `models` array — that is the `MANUAL_MODEL_PROVIDERS` case (Vertex, Custom, Cloudflare Gateway), where the admin typed the exact ids the key exposes. **An empty catalog is not a missing one**, and collapsing the two is how the first cut of this went wrong: a helper returning `string[] | undefined` treated "lists zero text models" the same as "has no catalog", so a key configured with only image models fell through to the curated list and resolved to a Gemini id it never offered. An admin-listed catalog is the whole truth about a key, so an empty one now refuses the turn with a message rather than guessing. Otherwise it falls back to the static `ALLOWED_CHAT_MODELS_BY_PROVIDER`, and if the provider is missing from that too it returns the raw tier id with its vendor prefix stripped — which is how a Vertex key once resolved to `claude-sonnet-4-6` and handed it to the Gemini client. That map is `Partial<`, so omitting a chat-capable provider compiles cleanly and fails only at the endpoint. **The key's own `modelScope`/`modelIds` allow-list is applied last, to whichever candidate list was chosen**, and a resolution with nothing left refuses the turn rather than returning a model the key forbids. `GetProviderConfigResponse` carries both fields for that, populated at all three construction sites (`getChatProvider`, `getConfigOrThrow`, `enrichWithKeysIfNeeded`); adding fields there is backward compatible because the AI piece reads the response as a typed shape rather than parsing it. I first recorded this as too big for a provider PR — "a wire contract the engine and the AI piece both consume" — having never measured it. It was about thirty lines across four files. Measure before recording something as out of scope; the estimate outlives the guess. **The resolver is exported and has five call sites across four files, and only two of them are runs.** `agent-rpc-handlers` resolves the chat turn's model *and* its fast model, and `chat-personalization-service` resolves both for research — those four must be given the resolved row's `config`, `modelScope` and `modelIds`, or the key's constraints are bypassed on exactly the paths a user takes. `resolveTierModel` inside `agent-helpers` is the fifth. `agent-service` also calls it but holds only a provider name and sets a stored default on a draft, so it is correctly left thin (`agent-draft-ai` did too, until #15676 deleted it). Adding a parameter here means grepping `resolveModelIdForProvider|resolveFastModelId` across `packages/server`, not editing the call site in front of you — threading only `resolveTierModel` looks complete, passes every test, and still leaves the real chat path unconstrained. One trap when touching this function: a tier id (`fast`, `smart`) is **not** a model id, so the preferred pick must be the resolved native id unless the selection is itself among the candidates — passing `selectedModel` straight through makes `resolveFastModelId` return the first curated model instead of the fast tier's. +- **Chat-tier resolution answers "which model" from three places, and only one of them is the key.** `resolveModelIdForProvider` (`ee/agent/agent-model-resolution.ts`, spread into `agentHelpers`) now prefers the resolved key's own configured text models when its config carries a `models` array — that is the `MANUAL_MODEL_PROVIDERS` case (Vertex, Custom, Cloudflare Gateway), where the admin typed the exact ids the key exposes. **An empty catalog is not a missing one**, and collapsing the two is how the first cut of this went wrong: a helper returning `string[] | undefined` treated "lists zero text models" the same as "has no catalog", so a key configured with only image models fell through to the curated list and resolved to a Gemini id it never offered. An admin-listed catalog is the whole truth about a key, so an empty one now refuses the turn with a message rather than guessing. Otherwise it falls back to the static `ALLOWED_CHAT_MODELS_BY_PROVIDER`, and if the provider is missing from that too it returns the raw tier id with its vendor prefix stripped — which is how a Vertex key once resolved to `claude-sonnet-4-6` and handed it to the Gemini client. That map is `Partial<`, so omitting a chat-capable provider compiles cleanly and fails only at the endpoint. **The key's own `modelScope`/`modelIds` allow-list is applied last, to whichever candidate list was chosen**, and a resolution with nothing left refuses the turn rather than returning a model the key forbids. `GetProviderConfigResponse` carries both fields for that, populated at all three construction sites (`getChatProvider`, `getConfigOrThrow`, `enrichWithKeysIfNeeded`); adding fields there is backward compatible because the AI piece reads the response as a typed shape rather than parsing it. I first recorded this as too big for a provider PR — "a wire contract the engine and the AI piece both consume" — having never measured it. It was about thirty lines across four files. Measure before recording something as out of scope; the estimate outlives the guess. **The resolver is exported and has five call sites across four files, and only two of them are runs.** `agent-rpc-handlers` resolves the chat turn's model *and* its fast model, and `chat-personalization-service` resolves both for research — those four must be given the resolved row's `config`, `modelScope` and `modelIds`, or the key's constraints are bypassed on exactly the paths a user takes. `resolveTierModel` inside `agent-helpers` is the fifth. `agent-service` also calls it but holds only a provider name and sets a stored default on a draft, so it is correctly left thin (`agent-draft-ai` did too, until #15676 deleted it). Adding a parameter here means grepping `resolveModelIdForProvider|resolveFastModelId` across `packages/server`, not editing the call site in front of you — threading only `resolveTierModel` looks complete, passes every test, and still leaves the real chat path unconstrained. The `surface` argument was added as a required key on the whole family for exactly that reason: a missed call site is a compile error, except inside a `vi.mock` factory, which is not type-checked and has to be updated by hand. One trap when touching this function: a tier id (`fast`, `smart`) is **not** a model id, so the preferred pick must be the resolved native id unless the selection is itself among the candidates — passing `selectedModel` straight through makes `resolveFastModelId` return the first curated model instead of the fast tier's. - **A failed credential validation tells the admin nothing, for every provider except Cloudflare Gateway.** `aiProviderService.validateProviderCredentials` gates the upstream message behind `includeHttpErrorInMessage`, which is `provider === CLOUDFLARE_GATEWAY` and nothing else, so everyone else gets a bare `Failed to validate credentials for `. The cause is not lost — it is logged one line earlier (`log.error({ error }, '[aiProviderService#validateProviderCredentials] ...')`) and passed as the `httpErrorResponse` error param — but **web never renders `httpErrorResponse`**, so the only way to diagnose a rejected key is the server log. Grep the log for `validateProviderCredentials` before assuming the provider integration is broken — that text is the whole diagnosis, and it is often not about credentials at all. Confirmed case: a brand-new xAI team with no credits purchased answers `GET /v1/models` with `403 permission-denied — Your newly created team doesn't have any credits or licenses yet`, naming the console page that fixes it, and we render that as "Failed to validate credentials for xAI" — sending the admin off to regenerate a key that was never wrong. Vendors also phrase real key failures inconsistently (xAI uses `400 Incorrect API key provided`, not a 401). The corollary: **a provider that saves without error is not a working provider.** A no-credits 403 and a bad key are indistinguishable in the UI, so only an actual generation proves a key end to end. This is an admin-only surface (`platformAdminOnly`), so there is little reason to keep hiding it. - **A provider's logo is an asset someone has to upload, not something the code ships.** `AiProviderInfo.logoUrl` in `packages/web/src/features/agents/ai-providers.ts` is a plain string rendered into an ``, and every provider points at `https://cdn.activepieces.com/pieces/.png` — nothing is bundled. Adding a provider therefore carries a cross-team dependency with no compile-time or test signal: a slug with no asset behind it renders a broken-image icon in the platform admin list, and only a live request tells you. Check the URL with `curl -o /dev/null -w '%{http_code}'` before assuming it works — a vendor that already ships as a *piece* usually has its logo there already (`deepseek.png`, `grok-xai.png` did), so start the upload request only for the genuinely missing ones. A Vite asset import also satisfies `logoUrl` (see `GoogleIcon` in `platform/security/sso/index.tsx`) and removes the runtime CDN dependency for air-gapped installs, but it diverges from every other provider — treat it as a fallback, not the default. - **The manual-models panel stores a model id and nothing else, so every model an admin adds is typed `TEXT`.** `MANUAL_MODEL_PROVIDERS` (Vertex, Custom, Cloudflare Gateway) persist a `ProviderModelConfig` of `{ modelId, modelName, modelType }`, but since #14825 the UI keeps only `string[]`: `ManualModelList` is a chip list of ids, and `config-detail.tsx` rebuilds the objects on save with `?? { modelId, modelName: modelId, modelType: TEXT }`. The old `universal-pieces/model-form-popover.tsx` had all three fields; replacing it with chips dropped two of them, and no schema or test noticed because the reconstruction is type-correct. The visible damage is not just a mislabelled model: `cloudflareGatewayProvider.validateConnection` pings **only text models** with a `/chat/completions` request, so one image model added through the UI is validated as chat, fails, and takes the whole key down with `These models have issues: ...` — the admin cannot save a key that has an image model on it at all. Anything reading `modelType` downstream (the agent text-model filter, `ap-list-ai-models`, the builder's image pickers) is wrong the same way. Fixed by holding `ProviderModelConfig[]` in the draft rather than `string[]` (no migration; stored rows always had the right shape), so the standing rule is that **a manual-models UI must carry `modelType` all the way to the payload** — a panel that collapses models to ids reintroduces exactly this, and nothing downstream will complain until an admin adds an image model. Keys saved while the bug was live still hold image models typed `TEXT`; the type badge on each chip flips one in place, which is why it is editable rather than display-only. @@ -185,7 +190,7 @@ the URL. The file is published from the console in `ap-analytics`; see decision - **`/v1/ai-tools` is registered only in the CLOUD and ENTERPRISE branches of `app.ts`, but the AI Center page that reads it is not edition-gated** — so a Community admin opening the Capabilities tab fired `useAiToolConfigs`, got Fastify's `Route not found`, and the page showed the global "Failed to load data" dialog of the day. Shipped that way from #13911 until the tab was gated on `ApFlagId.EDITION` in the page. What made this class of bug hard to place is that the surface was then page-independent (raised from `QueryCache.onError` in `query-client.ts`) and React Query's 3 default retries meant it landed several seconds later on whatever page you navigated to next (the report was against `/platform/general`). The tab now renders `DataFetchErrorState` in place instead, so a repeat would at least accuse the right page. When a screenshot's edition is in doubt, read the sidebar: **Billing & subscription** and **Usage** carry a lock only when `edition === COMMUNITY`, every other lock there is plan-driven. Any new EE-only route needs its UI entry point gated the same way, `enabled:` on the query or hiding the surface. - **The AI Center providers page renders its whole state machine off one query, so every branch needs an `isLoading` gate above it.** `ProvidersTab` derives the empty state, the grouped list *and* the `?config=` detail view from `useAiProviderConfigs()`. While that query is pending, `data` is `undefined` — `configs` is `[]`, so the empty "Connect your first provider" screen paints for a frame, and a hard refresh on a `?config=` URL finds no matching row and falls through to the list before flipping to the detail page. The parent `SuspenseWrapper` does not cover it: the route chunk and the suspense-backed projects collection resolve first, then the component mounts with no providers yet. Any new branch added to this page belongs below the `if (isLoading) return ` gate, not beside it. - **A save bar in the AI Center needs a full-height column, not `fixed` and not bare `sticky`.** The page sits inside *two* nested scrollers — `PlatformLayout`'s `flex flex-col flex-1 overflow-auto` and the AI Center's own `flex-1 overflow-auto` column — so `fixed inset-x-0` centres the bar on the viewport, roughly a sidebar-width left of the content column it belongs to. Swapping it to `sticky bottom-4` fixes the centring for free but pins nothing when the content is shorter than the viewport: the bar just lands wherever the content ends, mid-page. The working shape is `min-h-full` + `flex flex-col` on the tab-content wrapper, `flex-1` on the `TabsContent`, `grow` on the panel root, then `sticky bottom-4 mt-auto` on the bar — `mt-auto` handles the short page, `sticky` handles the long one. `min-h-full` resolves only because every ancestor up to `SidebarInset h-full` has a definite height; that chain is what makes the whole thing work, so check it before reusing this elsewhere. -- **A chat *tier* is our own idea, so it has no meaning on a provider we ship no curated model list for — resolve it by asking the key what it offers.** `ACTIVEPIECES_CHAT_TIERS` (fast/smart/premium) only maps to a real id through `ALLOWED_CHAT_MODELS_BY_PROVIDER`, which covers Anthropic, OpenAI, Google, Vertex and Activepieces. Everything else — Bedrock, Azure, Mistral, xAI, DeepSeek and the other OpenAI-compatible vendors — has an empty candidate list. The resolver used to invent an id from the tier (strip the vendor prefix, dots to dashes: `anthropic/claude-haiku-4.5` -> `claude-haiku-4-5`), which Bedrock rejected as "The provided model identifier is invalid" (GIT-1840, Pylon 5836). #15355 removed the invention and left a throw in its place, which was worse: every agent turn on those providers died with `ENTITY_NOT_FOUND: this AI provider key allows no text model a chat turn can run on` before the configured model handled anything (Pylon 6069). **The missing ingredient was never a smarter guess, it was the key's own model list.** `agentHelpers.resolveModelId` is now the single resolver for every surface: it tries our catalogue first and, only when that has nothing to say, reads `aiProviderService.listModels` (process-cached, cleared nightly by cron, already filtered by the key's `modelScope`), preferring a model whose id *contains* the tier's `nativeModelId` — Bedrock spells them `eu.anthropic.claude-sonnet-4-6-...`, so a tier still selects the model it names wherever the vendor hosts it, and otherwise degrades to the first text model the way `pickAllowedModel` already does for an admin-listed catalogue. Picking from a list the provider returned is the opposite of the fabrication #15355 removed. **Two traps, both learned the hard way.** First, an agent surface and generic chat are not symmetric: an agent *carries* a real model id (its picker lists `listModels` output unfiltered, precisely because these providers have no `ALLOWED_CHAT_MODELS_BY_PROVIDER` entry) while chat carries only a tier name, because `ChatModelSelector` falls back to tier options when `getCuratedChatModels` returns `undefined`. Second, **being listed proves the key can see a model, not that the account can invoke it** — so where a model is already settled the fast round reuses it rather than choosing again (`getAgentConfig` passes the model the turn resolved; the tool RPC reads the run's own model off the conversation or the agent via `pinnedModelOf` — and it has to read the *same* half the run did: an agent-page chat runs on `agent.draft` (`agent-conversation-controller.ts`), a Run Agent step on `agent.published`, so reading the wrong one silently lends the tool a stale model whenever the two differ). Resolving the fast tier independently once picked a Haiku the account could not invoke and failed a turn whose configured Sonnet worked fine. A borrowed model still has to pass the key's current `modelScope`, or resolution falls through to the list, which applies it. Nothing crosses the worker, so a rolling deploy has no version-skew window. +- **A chat *tier* is our own idea, so it has no meaning on a provider we ship no curated model list for — resolve it by asking the key what it offers.** `ACTIVEPIECES_CHAT_TIERS` (fast/smart/premium) only maps to a real id through `ALLOWED_CHAT_MODELS_BY_PROVIDER`, which covers Anthropic, OpenAI, Google, Vertex and Activepieces. Everything else — Bedrock, Azure, Mistral, xAI, DeepSeek and the other OpenAI-compatible vendors — has an empty candidate list. The resolver used to invent an id from the tier (strip the vendor prefix, dots to dashes: `anthropic/claude-haiku-4.5` -> `claude-haiku-4-5`), which Bedrock rejected as "The provided model identifier is invalid" (GIT-1840, Pylon 5836). #15355 removed the invention and left a throw in its place, which was worse: every agent turn on those providers died with `ENTITY_NOT_FOUND: this AI provider key allows no text model a chat turn can run on` before the configured model handled anything (Pylon 6069). **The missing ingredient was never a smarter guess, it was the key's own model list.** `agentHelpers.resolveModelId` is now the single resolver for every surface: it tries our catalogue first and, only when that has nothing to say, reads `aiProviderService.listModels` (process-cached, cleared nightly by cron, already filtered by the key's `modelScope`), preferring a model whose id *contains* the tier's `nativeModelId` — Bedrock spells them `eu.anthropic.claude-sonnet-4-6-...`, so a tier still selects the model it names wherever the vendor hosts it, and otherwise degrades to the first text model the way `pickAllowedModel` already does for an admin-listed catalogue. Picking from a list the provider returned is the opposite of the fabrication #15355 removed. **Two traps, both learned the hard way.** First, an agent surface and generic chat are not symmetric: an agent *carries* a real model id (its picker lists `listModels` output unfiltered, precisely because these providers have no `ALLOWED_CHAT_MODELS_BY_PROVIDER` entry) while chat carries only a tier name, because `ChatModelSelector` falls back to tier options when `getCuratedChatModels` returns `undefined`. Second, **being listed proves the key can see a model, not that the account can invoke it** — so where a model is already settled the fast round reuses it rather than choosing again (`getAgentConfig` passes the model the turn resolved; the tool RPC gets the run's own model from the worker, `modelId` on `executePieceTool`). One more trap, found in review on 2026-09-27: a stored name is not the model the run ran on. The worker stamps the resolved *tier id* on an agentless inline run's conversation, so reading it back as a model hands a bare `smart` to a catalogue-less own-key provider, and no rule on the string can save it, because a deployment may be named `smart` too. So nothing is read off the conversation or the agent for the fast round any more: the worker says which model the turn resolved, `modelId` on `ExecutePieceToolRequest` (core-execution), and the API borrows exactly that. The field is required, not optional, because the dispatch gate keeps worker and app on one release, so an older worker never reaches a newer API. Resolving the fast tier independently once picked a Haiku the account could not invoke and failed a turn whose configured Sonnet worked fine. A borrowed model still has to pass the key's current `modelScope`, or resolution falls through to the list, which applies it. Nothing crosses the worker, so a rolling deploy has no version-skew window. - `packages/server/api/src/app/ai/` — backend module: provider controller, service, entity, module, plus the sibling ai-tool-config files - `packages/server/api/src/app/ai/providers/` — per-vendor strategies keyed by `AIProviderName` - `packages/server/api/src/app/platform/billing-provider.ts` — `assertCreditsAndAppSumoNotExceeded` credit gate diff --git a/brain/knowledge/engineering/ci-pr-review-hygiene.md b/brain/knowledge/engineering/ci-pr-review-hygiene.md index b3c72cb7ff69..29c60561a18a 100644 --- a/brain/knowledge/engineering/ci-pr-review-hygiene.md +++ b/brain/knowledge/engineering/ci-pr-review-hygiene.md @@ -84,6 +84,7 @@ Enforcement is the **`Codeowners review` repository ruleset** (active on the def - **Docs videos belong on `cdn.activepieces.com/videos/docs/`, never in the repo — and `docs/resources/` is the legacy path that suggests otherwise.** Every `