diff --git a/.github/workflows/sync-model-catalog.yml b/.github/workflows/sync-model-catalog.yml index e9e3b267c525..77c354ea4468 100644 --- a/.github/workflows/sync-model-catalog.yml +++ b/.github/workflows/sync-model-catalog.yml @@ -60,3 +60,14 @@ jobs: run: | STATUS=$(curl -s -o /dev/null -w "%{http_code}" "https://cdn.activepieces.com/ai/model-catalog.json?run=$GITHUB_RUN_ID" || true) [ "$STATUS" = "200" ] || { echo "Publish verification failed: HTTP $STATUS"; exit 1; } + + - name: Notify Discord on failure + if: failure() + env: + WEBHOOK: ${{ secrets.DISCORD_ON_CALL_WEBHOOK }} + BRANCH: ${{ github.ref_name }} + RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} + run: | + jq -n --arg branch "$BRANCH" --arg url "$RUN_URL" \ + '{content: "🚨 **Sync AI Model Catalog Workflow Failed** 🚨\n\n**Branch:** \($branch)\n**Action URL:** \($url)\n\nThe AI model catalog sync did not finish. Please check the logs for more details."}' \ + | curl --fail --silent --show-error -H "Content-Type: application/json" -X POST -d @- "$WEBHOOK" diff --git a/brain/knowledge/ai-intelligence/ai-agents.md b/brain/knowledge/ai-intelligence/ai-agents.md index d4d06442fe81..8ac923d67ee0 100644 --- a/brain/knowledge/ai-intelligence/ai-agents.md +++ b/brain/knowledge/ai-intelligence/ai-agents.md @@ -15,9 +15,9 @@ A flow step type (the `run_agent` action of `@activepieces/piece-ai`) that runs | Door (`AgentRunSource`) | Model comes from | Stored as | |---|---|---| | `CHAT` | `conversation.modelName`, picked in the UI | tier id | -| `AGENT` | `agent.draft.modelName`; no picker is shown while chatting (`ai-chat-box.tsx` hides it when `agentId` is set) | concrete id | +| `AGENT` | `agent.draft.modelName`; no picker is shown while chatting (`ai-chat-box.tsx` hides it when `agentId` is set) | tier id on the managed provider; concrete id on an own key or a row saved before ENG-754 | | `AGENT_BUILDER` | `conversation.modelName` | tier id | -| `FLOW_STEP` | `agent.published.modelName`, or the step's own `input.aiProviderModel.model` | concrete id | +| `FLOW_STEP` | `agent.published.modelName`, or the step's own `input.aiProviderModel.model` | same as `AGENT` | Three things follow. `run_agent` has two modes: pointed at a saved agent (`agentId` set) the step sends no model and the agent row decides; inline, it sends `input.aiProviderModel.model`. An agent row carries two configs: agent-page chat reads `agent.draft`, a `run_agent` step reads `agent.published` (via `resolvePublishedAgent`), so the same agent can already run different models through different doors. The fast model crosses every door: `firstStepUsesFastModel` (`run-agent-turn.ts`) runs the first step of almost every turn on the fast tier, flow steps included, whatever the user picked. `execute-agent-run.ts` persists `config.tier.id` onto `conversation.modelName` when the job carried no `modelName`, so conversations already hold tier ids today (inline `run_agent`, eval runs, every `AGENT_BUILDER`), read back by `chat-tool-billing.ts` and `chat-analytics-sync.ts`. **A persisted tier id is not a pin.** `pinnedModelOf` (`rpc/rpc-shared.ts`) ignores `conversation.modelName` for a conversation with an agent: that value is the resolved default, and once the console removes that tier it would reach a catalogue-less own-key provider as a literal model id. A saved agent's run model is the agent's own half (draft for AGENT, published for FLOW_STEP); only an agentless inline step reads `conversation.modelName`, which is the step's configured model. @@ -30,6 +30,9 @@ Three things follow. `run_agent` has two modes: pointed at a saved agent (`agent - **PredefinedInputsStructure** β€” per-field `AGENT_DECIDE` / `CHOOSE_YOURSELF` / `LEAVE_EMPTY` baked into the tool so the agent knows which inputs it controls. ### Gotchas + +- **A managed agent row or Run Agent step stores a tier id (`smart`) from ENG-754 on; nothing converts the rows saved before.** `AIModelSelector` (`web/src/features/agents/ai-model/`) lists the published flow tiers by id for the managed provider and never calls `/models` for it; a blank picker pre-fills the published `flow.defaultTierId` once the tiers query settles (it used to take the first tier, Fast), and `withDefaultModel` (`agent-service.ts`) seeds the same default tier id on create. Own-key providers still store concrete ids: `resolveNamedModelId` passes their name through verbatim, so a tier id there would reach the provider as a model name. An older row keeps its concrete id: it runs that model (slash rule), shows the raw id in the picker, and does not follow a re-point until the ENG-753 migration. +- **A flow-surface run takes its thinking budget from the tier it names, not the default tier.** `resolveRunTier` (`agent-model-resolution.ts`) builds `config.tier` for FLOW_STEP and AGENT runs: a managed tier id gives that tier's `thinkingBudget`, `id` and `modelId`; a concrete id, an own-key model, a tier the file dropped, or no model at all gives the flow default, as every flow-surface run did before ENG-754. So a Heavy agent thinks at Heavy's budget and reports `premium` in `chat.tier`, while a Heavy row saved as `anthropic/claude-opus-4.8` still thinks at Expert's until it is migrated. - **A chat turn's data lives in jsonb, so recording something per turn needs no migration.** `agent_conversation.messages` is a `jsonb` column and each entry is a `PersistedAgentMessageSchema`, which already carries optional per-turn fields (`thinkingDurationMs`, `feedback`). Adding another optional one is a zod field plus a `@activepieces/shared` patch bump β€” no column, no migration, no rollback flag. The conversation row itself only has `modelName`, which is why "record it on the conversation" is usually the wrong instinct: a user can switch tier mid-conversation, so anything that describes one turn belongs on the message, not the row. - **But the server does not build those messages, so "no migration" does not mean "cheap".** `messages` arrives already assembled from the worker over RPC (`saveAgentMessages` in `rpc/conversation-rpc.ts`), and the server only persists it and bills off it. Putting a new per-turn field on a message therefore means changing the RPC request schema and the worker that fills it, not editing a server file. Costing this off the jsonb shape alone gave an estimate of about an hour for work that is really a change to the engine contract. Check who authors the object before you price a change to it. @@ -72,7 +75,7 @@ Three things follow. `run_agent` has two modes: pointed at a saved agent (`agent - **The curated list is a guess about capability, so the code learns.** A rejection calls `noteReasoningIsMandatory`, which records the id **only if it is curated** so a caller-supplied model id cannot grow the set, and the existing one-shot `shouldRetryStream` re-issues the turn with a minimal budget. `gemini-3.7-flash` is curated *and* reasoning-native, which is exactly the case the list alone would miss. - **`satisfies` against the vendor's own reasoning type is near-worthless here.** `{ effort: 'none' }`, `{ enabled: false, max_tokens: n }` and `{ enabled: false, effort: 'minimal' }` all satisfy it, because the vendor supports disabling and it is the *endpoints* that do not. The guard has to be a closed union of the three directives we actually send. -- **A tool's *property keys* are user text too, and nothing guards them.** `mcpToolNameUtils` covers the tool NAME; the input schema's keys come straight from `property.name`, the field labels a customer typed into the MCP trigger, at `agent-run-controller.ts` (`resolveFlowTools`) and `mcp-server-builder.ts` (`registerFlowTools`), which build the shape identically. A field called `Email Sender` produces a key with a space, which Anthropic rejects on the same `^[a-zA-Z0-9_.-]{1,64}$` rule it applies to names, so that flow-as-tool step can never succeed. Sanitising is only half a fix: the payload is handed to the flow as its trigger body, so `{{trigger['Email Sender']}}` in existing flows needs a keyβ†’label map to translate back. Fix it in one shared helper, because the MCP server ships the same invalid schema to any Anthropic-backed client. +- **A tool's *property keys* are user text too, so they go through `mcp-tool-input.ts`.** Field labels a customer typed into the MCP trigger (`Email Sender`) are not valid schema keys on Anthropic. `modelKeyByPropertyName` sanitizes them with `mcpToolNameUtils.toProviderSafeIdentifier`, keeping them unique, with a positional fallback and a length cap. `toFlowPayload` maps the model's args back to the original labels, so `{{trigger['Email Sender']}}` keeps working. The MCP server and agent flow tools both use it; build any new flow-as-tool schema through it. - **Do not raise `ai` past 7.0.69 β€” from `ai@7.0.70` the SDK stops auto-executing tools, and it fails silently.** Bisected on 2026-09-21 while trying to reach `experimental_evaluate`. On 7.0.44 (what the repo pins) and every version through 7.0.69, a tool call executes normally. From **7.0.70** onward, including 7.0.103–7.0.107, `generateText`/`streamText` still *recognise* the call β€” `staticToolCalls=1`, a `tool-call` part is emitted with correctly parsed input β€” but `execute` is never invoked and `toolResults` stays `0`. No error, no warning. The changelog entry for 7.0.70 is `a828527: Prevent automatic tool execution when a model call ends with an unsafe finish reason`, but the gate fires for every `finishReason` tried, `tool-calls` and `stop` included. Reproduced in a clean project outside the monorepo, on both `MockLanguageModelV3` and `MockLanguageModelV4`, on zod 3 and zod 4; `needsApproval: false` and `toolApproval: () => false` do not restore it, and neither does emitting the full `tool-input-start`/`delta`/`end` lifecycle before the `tool-call`. The alarm is `packages/server/worker/test/lib/agent-eval/agent-tool-call-repair.test.ts` β€” 4 of its 5 tests go red, and they look like a *repair* regression, which is the wrong trail: a plain valid tool call does not execute either. **Consequence: `experimental_evaluate` (first shipped in 7.0.103) cannot be reached without breaking the agent**, so anything needing typed evaluation calls the gateway's `/v4/ai/evaluation-model` endpoint over `safeHttp` instead of taking the bump. Re-test the table above before any future `ai` upgrade; if a release fixes it, that is the one to jump to. diff --git a/brain/knowledge/ai-intelligence/ai-providers.md b/brain/knowledge/ai-intelligence/ai-providers.md index 3693e5633703..167ec72035fa 100644 --- a/brain/knowledge/ai-intelligence/ai-providers.md +++ b/brain/knowledge/ai-intelligence/ai-providers.md @@ -78,7 +78,7 @@ after a file without chat keys it seeds the chat list from the live flow list. - **The pricing console shows bundled constants when no file is published, and says nothing.** `overview()` in `ai-pricing-service.ts` has three states, and two of them render the same numbers: **absent** (`NoSuchKey`) silently seeds the editor from `DEFAULT_PRICING_DRAFT`, the console's own checked-in copy of the tiers and weights, while **unreadable** (any other S3 error) shows those same constants *plus* a banner and blocks publishing. Only **present** is live data. So a screen full of plausible tiers is not evidence that anything was ever published β€” `curl -I https://cdn.activepieces.com/ai/pricing.json` is. The absent state is deliberate, it is how the first publish bootstraps, but it means "I can see the tiers" and "the prices are live" are unrelated facts. Worse, **absent and misconfigured are the same state**: `cdnClient.getJson` catches `NoSuchKey` and returns `undefined`, so a wrong bucket or an endpoint with the bucket in its path reads as "nothing published yet" and hides behind the defaults. The tell is the History tab β€” `list` does *not* catch `NoSuchKey`, so the same misconfiguration that reads as empty on the pricing page returns a 500 there. A 500 on history with a normal-looking pricing page means the S3 addressing is wrong, not that the history is empty. - **Deleting the live tier file is safe for customers and confusing for everyone else.** `ai/pricing.json` was deleted on 2026-09-24 to drop a leftover test tier, and Activepieces kept serving the right tiers throughout: the 404 makes `fetchTiers` throw, the failure is logged and backed off, and `current()` keeps serving the last good copy the process fetched; only a process that never fetched one, a fresh boot during the outage, runs the `ACTIVEPIECES_CHAT_TIERS` shipped with the release. Two things do change. Each server logs 'Failed to refresh the AI model tiers file' (or 'Failed to load…' on a process that never got a copy) on every failed attempt, and attempts happen at boot and whenever `current()` is called with a stale or missing copy, at most once per five minutes (`FAILURE_BACKOFF_MS`), so a permanent 404 is retried the same way as a blip. And the console reads the missing key as **absent**, so the next publish starts again at `version: 1` while the `ai/pricing/history/` objects from the old numbering are still there β€” the version number is `live.version + 1`, not a counter, so it is only meaningful between two publishes that never lost the live key. To remove one bad tier, edit and publish; do not delete the object. - **`CDN_S3_*` and `AP_S3_*` are two different buckets on two different providers, and mixing them puts public objects in customer data.** `cdn.activepieces.com` is a **DigitalOcean Spaces** bucket, `activepieces-cdn` in `fra1`, fronted by Cloudflare β€” a 200 from it carries `x-rgw-object-type` and `x-do-cdn-uuid`, which is how you tell. That is where `ai/model-catalog.json`, `ai/pricing.json` and `sdk/embed/*.js` live, and the credentials are the `CDN_S3_*` GitHub secrets. `AP_S3_BUCKET` is something else entirely: the **Cloudflare R2** file-storage bucket (`ap-files-prod` in production) holding run logs, step checkpoints, webhook payloads and flow bundles. The names are similar enough that the pricing console was once configured with the R2 file-storage key by mistake, which would have written a `public-read` `ai/pricing.json` into the customer-data bucket while `cdn.activepieces.com` stayed 404. When wiring anything that publishes to the CDN, confirm the endpoint is the Spaces region origin (`https://fra1.digitaloceanspaces.com`, never with the bucket in the path β€” the SDK appends `Bucket` itself and the doubled path returns `NoSuchKey` on every call). -- **The weekly sync workflow can fail and nobody is told.** `sync-model-catalog.yml` has no alert on failure, and the only signal that it broke is a 404 on the object, which every reader swallows. Its run on 2026-08-31 failed with `InvalidAccessKeyId` β€” the `CDN_S3_ACCESS_KEY_ID` secret in this repo does not match a live key β€” so `ai/model-catalog.json` has never actually landed. Check `gh run list --workflow=sync-model-catalog.yml` before believing the CDN is the problem. **That same `CDN_S3_*` credential also publishes the AI tier file `ai/pricing.json` from `ap-analytics`** (decision [000039](../decisions/000041-ai-model-tiers-are-published-to-the-cdn-from-a-second-console.md)), so one dead key silently breaks model metadata *and* the published tiers at once: the console hides model costs (`model-catalog-service.ts` logs 'model prices will be hidden' and returns an empty list) while Activepieces keeps serving the tiers bundled in the release. Two features, one credential, no alarm on either. +- **The weekly sync workflow now pages Discord on failure, but the alert needs `DISCORD_ON_CALL_WEBHOOK` in this repo.** `sync-model-catalog.yml` ends with the same `if: failure()` step as `release-pieces.yml`. Before that it had no alert, and the only signal that it broke is a 404 on the object, which every reader swallows. Its run on 2026-08-31 failed with `InvalidAccessKeyId` β€” the `CDN_S3_ACCESS_KEY_ID` secret in this repo does not match a live key β€” so `ai/model-catalog.json` has never actually landed. Check `gh run list --workflow=sync-model-catalog.yml` before believing the CDN is the problem. **That same `CDN_S3_*` credential also publishes the AI tier file `ai/pricing.json` from `ap-analytics`** (decision [000039](../decisions/000041-ai-model-tiers-are-published-to-the-cdn-from-a-second-console.md)), so one dead key silently breaks model metadata *and* the published tiers at once: the console hides model costs (`model-catalog-service.ts` logs 'model prices will be hidden' and returns an empty list) while Activepieces keeps serving the tiers bundled in the release. Two features, one credential, no alarm on either. - **A fresh Cloud platform cannot connect any AI provider, and the UI says nothing about why.** The admin page at `/platform/ai` sets `allowWrite = platform.plan.aiProvidersEnabled`, and that column defaults to **false** on the `free` plan (migration `1776…AddDefaultToAiProvidersEnabled`), so the connect button is simply absent rather than disabled-with-a-reason. On Cloud the flag is owned by Autumn (`autumn-utils.ts` lists it among the synced features), so there is nothing to configure locally and no `DEV_ENTERPRISE_PLAN` escape hatch on main. For local Cloud testing, flip it directly: `UPDATE platform_plan SET "aiProvidersEnabled" = true`. Then connect a **BYO key**, not the managed `ACTIVEPIECES` provider β€” per *Provider visibility* above, the managed row is hidden whenever `OPENROUTER_PROVISION_KEY` is unset, which is the normal local state, and a hidden provider makes `getChatProvider()` return null. Anything gated on a chat provider (chat itself, personalization research) stays silently disabled until a BYO row exists. - **Since the AI steps moved to the worker, the server sees every step's model id, and `ai-execute-controller` is the one place to act on it.** The piece POSTs `/v1/ai/execute` with `provider` and `modelId`, the controller builds the `EXECUTE_AI` job from that body, and the worker runs whatever model id the job carries. So anything that needs to rewrite or validate the model a flow step is about to use belongs there, not in the piece β€” one server-side edit covers all five AI steps, and an old pinned piece bundle keeps working because it only forwards the string it stored. This is what makes storing a tier id in a step cheap: the bundle never has to understand it. Note the agent step does **not** come through here β€” it resolves separately over `getAgentConfig`, in `agent-config-rpc`. Two conditions sit under that. **An old bundle only forwards the string from `0.11.0` up** β€” `AI_PIECE_COST_BILLING_VERSION`, the first version whose `ai-step.ts` POSTs `v1/ai/execute`. A step pinned below it runs the pre-worker code that calls the provider from the sandbox, so a tier id there would reach OpenRouter as the literal string. That case is already closed: `migrate-v26-ai-piece-cost-billing` bumps every saved `piece-ai` step below `0.11.0` up to it, which is why a later migration can rewrite what a step stores without touching `pieceVersion` for the runtime. **And resolve on the API, never in the worker**: `ExecuteAiRequest.modelId` is a bare `z.string()` so a tier id passes validation, `getEffectiveProviderAndModel` and `aiUtils.buildWebSearchToolsOrThrow` both read `data.modelId` as a real model, and the `resolveAiProvider` RPC does not carry the model at all, so using it as the seam would mean widening the worker contract. - **Managed billing can only price an OpenRouter call, and a call it cannot price is simply not billed.** `openRouterCostOf` (`server/utils/activepieces-ai-cost.ts`) reads `providerMetadata['openrouter'].usage.cost`, with one fallback that GETs `openrouter.ai/api/v1/generation?id=` and takes `total_cost`. If neither answers, `createBillingInfo` returns `undefined` and the turn is never reported β€” it runs, the customer keeps the output, and nothing is charged. Nothing logs "unbilled". This is what actually welds the managed provider to OpenRouter, more than the client does: `create-language-model.ts` shares one branch for `OPENROUTER` and `ACTIVEPIECES`, every tier `modelId` is an OpenRouter id, and `openrouter-api.ts` mints the per-platform key from `OPENROUTER_PROVISION_KEY`. So putting managed credits on any other gateway needs a second cost extractor and a second generation lookup **first**, or every call through it bills zero. For reference, no Vercel AI Gateway code exists here at all β€” the only gateway in the repo is Cloudflare's (`CLOUDFLARE_GATEWAY`, the `ai-gateway-provider` package), and `@ai-sdk/gateway` appears only as a transitive dep of `ai` that nothing imports. A BYOK user can already reach Vercel's gateway today through `AIProviderName.CUSTOM`, which takes a base URL plus hand-typed model ids. @@ -216,3 +216,4 @@ after a file without chat keys it seeds the chat list from the live flow list. Paths verified 2026-07-26. - **`OPENROUTER_PROVISION_KEY` is a management key and cannot make inference calls; only the per-platform keys it mints can.** OpenRouter states it outright: management keys are for creating, listing, updating and deleting other keys and are rejected by every completion or decisions endpoint. So "we already have an OpenRouter key" is true for provisioning and false for anything that needs to call a model at the instance level (the AI Router's Jev call, for example): that needs an ordinary inference key from the same account, created in the dashboard or via the same `POST /keys` the instance already uses, with a credit `limit`. The AI Router does exactly that since 2026-09-23: it runs on the platform's managed key where credits are on, else the admin's own OpenRouter row, as a worker `EXECUTE_AI` job billed like any AI step. - **`z.enum(SomeTsEnum).exclude([SomeTsEnum.MEMBER])` compiles and excludes nothing at the type level; pass the key string, `.exclude(['MEMBER'])`.** zod's `exclude` is typed over `keyof T` and returns `ZodEnum>`. A TS enum member is its own literal type, not the string `'MEMBER'`, so `Omit` finds no key to drop and the inferred value type keeps every member, while the runtime check does exclude it. The symptom is one step removed from the cause: a switch over the parsed field is suddenly non-exhaustive (`switch-exhaustiveness-check` in lint, `TS2366` in the build) even though the branch you removed can never arrive. `/v1/ai/execute` hit this when `ROUTE` was carved out of `AiStepAction` for #15707. Neither vitest nor a package-local eslint run without type information reports it; only the build does. +- **The managed Activepieces row can never restrict its models, so a finding that the managed picker offers a model the key forbids is not reachable.** The row is inserted with `modelScope: 'all', modelIds: []`, `create()` refuses `provider === ACTIVEPIECES` (`aiProvider.activepiecesIsManaged`), and `update()` handles only `enabledForChat` for it and returns before any scope field is read (`ai-provider-service.ts`). Every published tier therefore runs on it, and any `modelScope` filtering on the managed path is inert. Reviewers (Greptile on #15871) raise this as a P1; answer with these three facts rather than adding a filter. diff --git a/brain/knowledge/decisions/000032-an-agent-is-a-project-scoped-row-that-flow-steps-reference-live.md b/brain/knowledge/decisions/000032-an-agent-is-a-project-scoped-row-that-flow-steps-reference-live.md index 31aba2f8cd27..2d1a011bab16 100644 --- a/brain/knowledge/decisions/000032-an-agent-is-a-project-scoped-row-that-flow-steps-reference-live.md +++ b/brain/knowledge/decisions/000032-an-agent-is-a-project-scoped-row-that-flow-steps-reference-live.md @@ -34,7 +34,8 @@ rejected alternative was exactly that: dropdown prefills the step, link then gon "allow unattended writes" flag, not re-litigating the reference. - **Moving a flow between projects breaks the link** β€” the id is project-local, and git sync carries connections but not agents. Hence `externalId` on the row from day one; project state must upsert - agents by `(projectId, externalId)`, and until it does the import must fail loudly. + agents by `(projectId, externalId)`, and until it does the import must fail loudly. Publish already refuses a + missing agent (`assertAgentsResolveInProject`), but release and replace drop `republishFlow`'s error today. - Two orphaned tables (`agent`, `agent_run`) from the 2025 module are dropped so the entity can take the obvious name. `breaking = true` for rollback safety, no `⛓️‍πŸ’₯ breaking-change` label. - **Autonomy stays a flow** β€” "Run on a schedule" scaffolds a real flow with a linked step, so there is diff --git a/brain/knowledge/engineering/ci-pr-review-hygiene.md b/brain/knowledge/engineering/ci-pr-review-hygiene.md index 44c1c6ddc11c..c89783fbb7e6 100644 --- a/brain/knowledge/engineering/ci-pr-review-hygiene.md +++ b/brain/knowledge/engineering/ci-pr-review-hygiene.md @@ -85,7 +85,8 @@ Enforcement is the **`Codeowners review` repository ruleset** (active on the def - **Docs videos belong on `cdn.activepieces.com/videos/docs/`, never in the repo β€” and `docs/resources/` is the legacy path that suggests otherwise.** Every `