Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions .github/workflows/sync-model-catalog.yml
Original file line number Diff line number Diff line change
Expand Up @@ -60,3 +60,14 @@ jobs:
run: |
STATUS=$(curl -s -o /dev/null -w "%{http_code}" "https://cdn.activepieces.com/ai/model-catalog.json?run=$GITHUB_RUN_ID" || true)
[ "$STATUS" = "200" ] || { echo "Publish verification failed: HTTP $STATUS"; exit 1; }

- name: Notify Discord on failure
if: failure()
env:
WEBHOOK: ${{ secrets.DISCORD_ON_CALL_WEBHOOK }}
BRANCH: ${{ github.ref_name }}
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
run: |
jq -n --arg branch "$BRANCH" --arg url "$RUN_URL" \
'{content: "🚨 **Sync AI Model Catalog Workflow Failed** 🚨\n\n**Branch:** \($branch)\n**Action URL:** \($url)\n\nThe AI model catalog sync did not finish. Please check the logs for more details."}' \
| curl --fail --silent --show-error -H "Content-Type: application/json" -X POST -d @- "$WEBHOOK"
9 changes: 6 additions & 3 deletions brain/knowledge/ai-intelligence/ai-agents.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,9 +15,9 @@ A flow step type (the `run_agent` action of `@activepieces/piece-ai`) that runs
| Door (`AgentRunSource`) | Model comes from | Stored as |
|---|---|---|
| `CHAT` | `conversation.modelName`, picked in the UI | tier id |
| `AGENT` | `agent.draft.modelName`; no picker is shown while chatting (`ai-chat-box.tsx` hides it when `agentId` is set) | concrete id |
| `AGENT` | `agent.draft.modelName`; no picker is shown while chatting (`ai-chat-box.tsx` hides it when `agentId` is set) | tier id on the managed provider; concrete id on an own key or a row saved before ENG-754 |
| `AGENT_BUILDER` | `conversation.modelName` | tier id |
| `FLOW_STEP` | `agent.published.modelName`, or the step's own `input.aiProviderModel.model` | concrete id |
| `FLOW_STEP` | `agent.published.modelName`, or the step's own `input.aiProviderModel.model` | same as `AGENT` |

Three things follow. `run_agent` has two modes: pointed at a saved agent (`agentId` set) the step sends no model and the agent row decides; inline, it sends `input.aiProviderModel.model`. An agent row carries two configs: agent-page chat reads `agent.draft`, a `run_agent` step reads `agent.published` (via `resolvePublishedAgent`), so the same agent can already run different models through different doors. The fast model crosses every door: `firstStepUsesFastModel` (`run-agent-turn.ts`) runs the first step of almost every turn on the fast tier, flow steps included, whatever the user picked. `execute-agent-run.ts` persists `config.tier.id` onto `conversation.modelName` when the job carried no `modelName`, so conversations already hold tier ids today (inline `run_agent`, eval runs, every `AGENT_BUILDER`), read back by `chat-tool-billing.ts` and `chat-analytics-sync.ts`. **A persisted tier id is not a pin.** `pinnedModelOf` (`rpc/rpc-shared.ts`) ignores `conversation.modelName` for a conversation with an agent: that value is the resolved default, and once the console removes that tier it would reach a catalogue-less own-key provider as a literal model id. A saved agent's run model is the agent's own half (draft for AGENT, published for FLOW_STEP); only an agentless inline step reads `conversation.modelName`, which is the step's configured model.

Expand All @@ -30,6 +30,9 @@ Three things follow. `run_agent` has two modes: pointed at a saved agent (`agent
- **PredefinedInputsStructure** — per-field `AGENT_DECIDE` / `CHOOSE_YOURSELF` / `LEAVE_EMPTY` baked into the tool so the agent knows which inputs it controls.

### Gotchas

- **A managed agent row or Run Agent step stores a tier id (`smart`) from ENG-754 on; nothing converts the rows saved before.** `AIModelSelector` (`web/src/features/agents/ai-model/`) lists the published flow tiers by id for the managed provider and never calls `/models` for it; a blank picker pre-fills the published `flow.defaultTierId` once the tiers query settles (it used to take the first tier, Fast), and `withDefaultModel` (`agent-service.ts`) seeds the same default tier id on create. Own-key providers still store concrete ids: `resolveNamedModelId` passes their name through verbatim, so a tier id there would reach the provider as a model name. An older row keeps its concrete id: it runs that model (slash rule), shows the raw id in the picker, and does not follow a re-point until the ENG-753 migration.
- **A flow-surface run takes its thinking budget from the tier it names, not the default tier.** `resolveRunTier` (`agent-model-resolution.ts`) builds `config.tier` for FLOW_STEP and AGENT runs: a managed tier id gives that tier's `thinkingBudget`, `id` and `modelId`; a concrete id, an own-key model, a tier the file dropped, or no model at all gives the flow default, as every flow-surface run did before ENG-754. So a Heavy agent thinks at Heavy's budget and reports `premium` in `chat.tier`, while a Heavy row saved as `anthropic/claude-opus-4.8` still thinks at Expert's until it is migrated.
- **A chat turn's data lives in jsonb, so recording something per turn needs no migration.** `agent_conversation.messages` is a `jsonb` column and each entry is a `PersistedAgentMessageSchema`, which already carries optional per-turn fields (`thinkingDurationMs`, `feedback`). Adding another optional one is a zod field plus a `@activepieces/shared` patch bump — no column, no migration, no rollback flag. The conversation row itself only has `modelName`, which is why "record it on the conversation" is usually the wrong instinct: a user can switch tier mid-conversation, so anything that describes one turn belongs on the message, not the row.
- **But the server does not build those messages, so "no migration" does not mean "cheap".** `messages` arrives already assembled from the worker over RPC (`saveAgentMessages` in `rpc/conversation-rpc.ts`), and the server only persists it and bills off it. Putting a new per-turn field on a message therefore means changing the RPC request schema and the worker that fills it, not editing a server file. Costing this off the jsonb shape alone gave an estimate of about an hour for work that is really a change to the engine contract. Check who authors the object before you price a change to it.

Expand Down Expand Up @@ -72,7 +75,7 @@ Three things follow. `run_agent` has two modes: pointed at a saved agent (`agent
- **The curated list is a guess about capability, so the code learns.** A rejection calls `noteReasoningIsMandatory`, which records the id **only if it is curated** so a caller-supplied model id cannot grow the set, and the existing one-shot `shouldRetryStream` re-issues the turn with a minimal budget. `gemini-3.7-flash` is curated *and* reasoning-native, which is exactly the case the list alone would miss.
- **`satisfies` against the vendor's own reasoning type is near-worthless here.** `{ effort: 'none' }`, `{ enabled: false, max_tokens: n }` and `{ enabled: false, effort: 'minimal' }` all satisfy it, because the vendor supports disabling and it is the *endpoints* that do not. The guard has to be a closed union of the three directives we actually send.

- **A tool's *property keys* are user text too, and nothing guards them.** `mcpToolNameUtils` covers the tool NAME; the input schema's keys come straight from `property.name`, the field labels a customer typed into the MCP trigger, at `agent-run-controller.ts` (`resolveFlowTools`) and `mcp-server-builder.ts` (`registerFlowTools`), which build the shape identically. A field called `Email Sender` produces a key with a space, which Anthropic rejects on the same `^[a-zA-Z0-9_.-]{1,64}$` rule it applies to names, so that flow-as-tool step can never succeed. Sanitising is only half a fix: the payload is handed to the flow as its trigger body, so `{{trigger['Email Sender']}}` in existing flows needs a key→label map to translate back. Fix it in one shared helper, because the MCP server ships the same invalid schema to any Anthropic-backed client.
- **A tool's *property keys* are user text too, so they go through `mcp-tool-input.ts`.** Field labels a customer typed into the MCP trigger (`Email Sender`) are not valid schema keys on Anthropic. `modelKeyByPropertyName` sanitizes them with `mcpToolNameUtils.toProviderSafeIdentifier`, keeping them unique, with a positional fallback and a length cap. `toFlowPayload` maps the model's args back to the original labels, so `{{trigger['Email Sender']}}` keeps working. The MCP server and agent flow tools both use it; build any new flow-as-tool schema through it.

- **Do not raise `ai` past 7.0.69 — from `ai@7.0.70` the SDK stops auto-executing tools, and it fails silently.** Bisected on 2026-09-21 while trying to reach `experimental_evaluate`. On 7.0.44 (what the repo pins) and every version through 7.0.69, a tool call executes normally. From **7.0.70** onward, including 7.0.103–7.0.107, `generateText`/`streamText` still *recognise* the call — `staticToolCalls=1`, a `tool-call` part is emitted with correctly parsed input — but `execute` is never invoked and `toolResults` stays `0`. No error, no warning. The changelog entry for 7.0.70 is `a828527: Prevent automatic tool execution when a model call ends with an unsafe finish reason`, but the gate fires for every `finishReason` tried, `tool-calls` and `stop` included. Reproduced in a clean project outside the monorepo, on both `MockLanguageModelV3` and `MockLanguageModelV4`, on zod 3 and zod 4; `needsApproval: false` and `toolApproval: () => false` do not restore it, and neither does emitting the full `tool-input-start`/`delta`/`end` lifecycle before the `tool-call`. The alarm is `packages/server/worker/test/lib/agent-eval/agent-tool-call-repair.test.ts` — 4 of its 5 tests go red, and they look like a *repair* regression, which is the wrong trail: a plain valid tool call does not execute either. **Consequence: `experimental_evaluate` (first shipped in 7.0.103) cannot be reached without breaking the agent**, so anything needing typed evaluation calls the gateway's `/v4/ai/evaluation-model` endpoint over `safeHttp` instead of taking the bump. Re-test the table above before any future `ai` upgrade; if a release fixes it, that is the one to jump to.

Expand Down
Loading
Loading