[pull] main from activepieces:main

brain/knowledge/ai-intelligence/ai-agents.md

@@ -4,11 +4,11 @@ icon: 🤖
 
 # AI Agents
 
-A flow step type (backed by `@activepieces/piece-agent`) that runs an LLM-driven autonomous loop. Given a prompt, tools, an AI provider/model, and optional structured-output fields, it runs a ReAct-style loop (up to `maxSteps`) where the model can call any configured tool before producing a final answer.
+A flow step type (the `run_agent` action of `@activepieces/piece-ai`) that runs an LLM-driven autonomous loop. Given a prompt, tools, an AI provider/model, and optional structured-output fields, it runs a ReAct-style loop (up to `maxSteps`) where the model can call any configured tool before producing a final answer.
 
 ### How it works
 
-- No backend entity of its own — the whole configuration lives inside the flow version's step settings. The step is a `PIECE` action on `@activepieces/piece-agent`; `settings.input` holds `agentTools`, `structuredOutput`, `prompt`, `maxSteps`, `aiProviderModel` (`{ provider, model }`), and optional `webSearch`.
+- A step carries its own configuration in the flow version's step settings: `settings.input` holds `agentTools`, `structuredOutput`, `prompt`, `maxSteps` and `aiProviderModel` (`{ provider, model, configId }`). A saved **Agent** is a separate thing — a project-scoped row (`agent` table, `ee/agent/agent-entity.ts`) that Chat and the Agents page use; a flow step does not read it.
 - Configured entirely in the Flow Builder (`web/src/app/builder/step-settings/agent-settings/`); a test panel runs a single agent step. `AgentTimeline` renders `AgentStepBlock[]` from the output as markdown blocks + expandable tool-call cards.
 
 ### Tool types (AgentTool discriminated union)

brain/knowledge/ai-intelligence/ai-mcp.md

@@ -26,8 +26,8 @@ Platform admins configure LLM backends for AI pieces; auto-provisions an "Active
 
 Platform-level AI assistant that manages projects via natural language, streaming over WebSocket and using the project's MCP server as its tool surface.
 
-- **Execution model (key gotcha)**: the LLM loop runs in the **worker**, not the API. Controller enqueues `EXECUTE_CHAT_AGENT` → worker `run-chat-turn.ts` runs `streamText()` → chunks stream back via RPC → `CHAT_MESSAGE_CHUNK` websocket (filtered by `runId`). `chat-service.ts` only does conversation CRUD.
-- **Entities**: `ChatConversation` (per platform+user, optional project scope, messages as JSONB `ModelMessage[]`, compaction summary). `chat_rollout_user` tracks the cloud beta cohort (capped at 200 distinct users who sent a message).
+- **Execution model (key gotcha)**: the LLM loop runs in the **worker**, not the API. Controller enqueues `WorkerJobType.EXECUTE_AGENT_RUN` → worker `execute-agent-run.ts` → `run-agent-turn.ts` runs `streamText()` → chunks stream back via RPC → `CHAT_MESSAGE_CHUNK` websocket (filtered by `runId`). `agent-conversation-service.ts` only does conversation CRUD.
+- **Entities**: `AgentConversation` (table `agent_conversation`, per platform+user, optional project scope, messages as JSONB `ModelMessage[]`, compaction summary). `chat_rollout_user` tracks the cloud beta cohort (capped at 200 distinct users who sent a message).
 - **Integration/gotchas**: EE/Cloud only (needs `chatEnabled`, or cloud rollout/grandfather); refuses PGLite dev DB — needs Postgres + Redis. Two-phase (discovery/build) tool gating; Redis pub/sub approval gates for display cards + write-action previews; MCP tools no longer gated (just timeout-wrapped). Server-managed connections — LLM never sees credential externalIds. Web search rides the configured LLM credential (no second BYOK).
 
 ### Knowledge Base

brain/knowledge/ai-intelligence/ai-providers.md

@@ -44,11 +44,11 @@ Lets platform admins configure one or more LLM backends for AI pieces in flows.
 - **A failed credential validation tells the admin nothing, for every provider except Cloudflare Gateway.** `aiProviderService.validateProviderCredentials` gates the upstream message behind `includeHttpErrorInMessage`, which is `provider === CLOUDFLARE_GATEWAY` and nothing else, so everyone else gets a bare `Failed to validate credentials for <name>`. The cause is not lost — it is logged one line earlier (`log.error({ error }, '[aiProviderService#validateProviderCredentials] ...')`) and passed as the `httpErrorResponse` error param — but **web never renders `httpErrorResponse`**, so the only way to diagnose a rejected key is the server log. Grep the log for `validateProviderCredentials` before assuming the provider integration is broken — that text is the whole diagnosis, and it is often not about credentials at all. Confirmed case: a brand-new xAI team with no credits purchased answers `GET /v1/models` with `403 permission-denied — Your newly created team doesn't have any credits or licenses yet`, naming the console page that fixes it, and we render that as "Failed to validate credentials for xAI" — sending the admin off to regenerate a key that was never wrong. Vendors also phrase real key failures inconsistently (xAI uses `400 Incorrect API key provided`, not a 401). The corollary: **a provider that saves without error is not a working provider.** A no-credits 403 and a bad key are indistinguishable in the UI, so only an actual generation proves a key end to end. This is an admin-only surface (`platformAdminOnly`), so there is little reason to keep hiding it.
 - **A provider's logo is an asset someone has to upload, not something the code ships.** `AiProviderInfo.logoUrl` in `packages/web/src/features/agents/ai-providers.ts` is a plain string rendered into an `<img src>`, and every provider points at `https://cdn.activepieces.com/pieces/<slug>.png` — nothing is bundled. Adding a provider therefore carries a cross-team dependency with no compile-time or test signal: a slug with no asset behind it renders a broken-image icon in the platform admin list, and only a live request tells you. Check the URL with `curl -o /dev/null -w '%{http_code}'` before assuming it works — a vendor that already ships as a *piece* usually has its logo there already (`deepseek.png`, `grok-xai.png` did), so start the upload request only for the genuinely missing ones. A Vite asset import also satisfies `logoUrl` (see `GoogleIcon` in `platform/security/sso/index.tsx`) and removes the runtime CDN dependency for air-gapped installs, but it diverges from every other provider — treat it as a fallback, not the default.
 - **`AIProviderConfig` is an *untagged* `z.union`, so a new provider's config schema must sit ahead of the empty ones — and "empty" includes a schema whose every field is optional.** Zod strips unknown keys and a union returns the first member that parses, so `AnthropicProviderConfig` (`z.object({})`) matches *any* object: list it before a `{ baseUrl?: string }` config and a configured base URL is silently reduced to `{}` — no error, no log, the admin's override just stops existing on the next read. The file carries an `Order matters` comment, but it says "empty ones last", which reads as though only a literal `z.object({})` is at risk. The safe rule is to insert any new config immediately after the last schema with a *required* field (today `BedrockProviderConfig`). `ProviderConfigUnion` is discriminated on `provider` and so is immune; only the two untagged unions (`AIProviderConfig`, `AIProviderAuthConfig`) bite. Both live twice — `packages/core/shared/.../management/ai-providers/index.ts` (zod classic) and `packages/core/piece-types/.../ai-providers.ts` (zod/mini, the copy pieces use) — and every provider edit has to land in both. **There is a third copy the shared package does not own:** `createFormSchema` in the admin dialog (`.../setup/ai/universal-pieces/upsert-provider-dialog.tsx`) re-declares a per-provider schema, branching explicitly on Azure / Cloudflare / Custom / Bedrock and falling through to a generic case whose `config` is a union of three empty objects. **A provider with a non-empty config and no branch there loses that config entirely** — `zodResolver` hands react-hook-form the *parsed* value, so the strip happens before submit and the setting is never sent, with no error anywhere. Fixing the shared union does not fix this one; grep for every union of config schemas when adding a provider. The dialog only diverges from the correct `ProviderConfigUnion` to make auth optional in edit mode, so collapsing it onto the shared discriminated union is the real repair. (Testing that file directly is awkward: importing it pulls in a transitive dep that touches `document` at import time, which a node-env vitest cannot load — the schema factory would have to move out of the component file first.)
-- **A failed `enrichWithKeysIfNeeded()` is self-sustaining, and it takes chat down with it.** `createKey` runs on the chat hot path — `chatHelpers.resolveChatProvider` → `getChatProvider` calls it whenever the platform's managed ACTIVEPIECES row has no `apiKey` — and the `save` happens *after* the OpenRouter call, so a failure persists nothing and the next chat turn calls `createKey` again. There is also no distributed lock or cache, so concurrent turns for one platform each mint a live key and only the last is saved; the orphans keep spending provisioning quota. Seen in prod 2026-07-30: `keys-modify-api-rpd-v2` 429 (OpenRouter's key create/modify bucket, 10k/day on the provision key — a *separate* limit from inference), which killed every chat turn for the affected platform in `getChatConfig` before the first token, with no recovery until the bucket reset at 00:00 UTC.
+- **A failed `enrichWithKeysIfNeeded()` is self-sustaining, and it takes chat down with it.** `createKey` runs on the chat hot path — `agentHelpers.resolveChatProvider` → `aiProviderService.getChatProvider` calls it whenever the platform's managed ACTIVEPIECES row has no `apiKey` — and the `save` happens *after* the OpenRouter call, so a failure persists nothing and the next chat turn calls `createKey` again. There is also no distributed lock or cache, so concurrent turns for one platform each mint a live key and only the last is saved; the orphans keep spending provisioning quota. Seen in prod 2026-07-30: `keys-modify-api-rpd-v2` 429 (OpenRouter's key create/modify bucket, 10k/day on the provision key — a *separate* limit from inference), which killed every chat turn for the affected platform in `getAgentConfig` before the first token, with no recovery until the bucket reset at 00:00 UTC.
 - **`openrouter-api.ts` uses raw `fetch`** — no timeout, no retry, no `tryCatch`, and it bypasses the repo's `safeHttp` rule for outbound HTTP in `packages/server/api`. A `getKey` 408 from OpenRouter escapes the admin `increaseAiCredits` path as an unhandled rejection.
 - **Chat model tiers are Activepieces-only.** `ACTIVEPIECES_CHAT_TIERS` (`fast`/`smart`/`premium`, shown as Fast/Expert/Heavy) hold OpenRouter-shaped Anthropic ids, so they only mean anything for the ACTIVEPIECES and OPENROUTER chat providers. Any provider that declares `ALLOWED_CHAT_MODELS_BY_PROVIDER` (openai, anthropic, google) picks a real model id from that list instead. Naively stripping the tier's vendor prefix for every provider is what once sent `claude-haiku-4-5` to OpenAI and broke every message.
 - **A short model list in the key's picker is the vendor's catalog, not a truncation.** `listModels` returns whatever the provider's own `/models` endpoint gives and filters nothing except the key's `modelScope` allow-list. Anthropic ships roughly a dozen models, OpenAI ~80 (mostly embeddings/tts/whisper), while OpenRouter is an aggregator and returns 400+ from every vendor it proxies — so the counts differ by an order of magnitude by design. Anthropic pages at 20 by default, which is why its request pins `?limit=1000`. If the list shows exactly three Claude models, that is the *chat* dropdown reading the curated `ANTHROPIC_CHAT_MODELS`, a different surface from the admin picker.
-- **Read the chat model list through `aiProviderUtils.getCuratedChatModels({ provider })`.** It is the one accessor the server resolver (`chatHelpers.resolveModelIdForProvider`) and the chat dropdown share, so the two cannot drift; it returns `{ id, label }` or `undefined` — never an empty list, so callers may treat a result as non-empty. Labels come from the hardcoded `CHAT_MODEL_LABELS` (falling back to the id) rather than `AIProviderModel.name`, because the live `listModels` response cannot supply one for every provider: anthropic returns `display_name` and google `displayName`, but OpenAI's `/v1/models` returns ids only.
+- **Read the chat model list through `aiProviderUtils.getCuratedChatModels({ provider })`.** It is the one accessor the server resolver (`agentHelpers.resolveModelIdForProvider`) and the chat dropdown share, so the two cannot drift; it returns `{ id, label }` or `undefined` — never an empty list, so callers may treat a result as non-empty. Labels come from the hardcoded `CHAT_MODEL_LABELS` (falling back to the id) rather than `AIProviderModel.name`, because the live `listModels` response cannot supply one for every provider: anthropic returns `display_name` and google `displayName`, but OpenAI's `/v1/models` returns ids only.
 - **`conversation.modelName` carries either a tier id or a real model id** — it is a free string with no discriminator. A legacy tier id resolves to the tier's equivalent model when the provider ships it, else the provider's first curated model, so old conversations keep working after a provider switch. Note `premium` maps to opus 4.8, which the native anthropic list does not carry, so a legacy `premium` on anthropic lands on Sonnet.
 - Chat-provider resolution is **first `enabledForChat` row wins**, *not* "prefer ACTIVEPIECES". All three branches of `findAvailableChatProviderRow` reduce to that: when the managed provider is visible the function returns `chatProviders[0]` whatever it is, so a platform with `[openai, activepieces]` both chat-enabled resolves to **openai**. The client mirror is `aiProviderQueries.useChatProvider()` (`providers.find((p) => p.enabledForChat)`) — always read the resolved chat provider through it rather than re-deriving the rule inline. **`enabledForChat` on a deduped project entry must be an OR across that provider's keys, never the top-ranked key's flag** — ranking (`selected` > `except` > `all`, newest first) and chat selection answer different questions, so reading `rows[0].enabledForChat` makes the client report "no provider configured" whenever the chat-enabled key is not the ranking winner, while the server (`findAvailableChatProviderRow`, which queries `enabledForChat: true` directly) happily serves the turn. Invisible with one key per provider. Both sides lean on an unordered `findBy()`: there is no `ORDER BY`, so "first" is not guaranteed stable when several providers are chat-enabled.
 - Listing providers is **not a pure read**: both `listConfigs` and `listForProject` go through `listVisibleRows`, which inserts the ACTIVEPIECES provider row when `aiCreditsEnabled && !activepiecesExists`. A `GET /v1/ai-providers` can therefore create a row. It also applies the hidden-provider filter (`plan.embeddingEnabled` hides the managed provider), which is why the client can trust its output without re-checking flags.

brain/knowledge/flows-execution/chat.md

@@ -7,13 +7,13 @@ icon: 💬
 A platform-level AI chat assistant that manages Activepieces projects via natural language. Streams LLM responses over WebSocket and exposes project resources (flows, tables, connections, runs) as callable tools through the project's MCP server. Conversations persist per-user with cross-session memory (personal instructions + remembered facts injected into every turn), compaction, attachments, multi-project context, and two-phase tool gating. EE/Cloud only (not registered in CE).
 
 ### Execution model (read first)
-The chat LLM loop runs in the **worker**, not the API. Send path: `agent-conversation-controller.ts` (`POST /conversations/:id/messages`) enqueues a `WorkerJobType.EXECUTE_CHAT_AGENT` job → worker `execute-agent-run.ts` calls `getChatConfig` RPC, assembles tools, runs `run-agent-turn.ts` (shared `streamText()` DI loop) → chunks stream back via `sendChatEvent` RPC → websocket `CHAT_MESSAGE_CHUNK` (filtered by `runId`) → frontend reducer. `agent-conversation-service.ts` only does conversation CRUD + persistence.
+The chat LLM loop runs in the **worker**, not the API. Send path: `agent-conversation-controller.ts` (`POST /conversations/:id/messages`) enqueues a `WorkerJobType.EXECUTE_AGENT_RUN` job → worker `execute-agent-run.ts` calls `getAgentConfig` RPC, assembles tools, runs `run-agent-turn.ts` (shared `streamText()` DI loop) → chunks stream back via `sendAgentEvent` RPC → websocket `CHAT_MESSAGE_CHUNK` (filtered by `runId`) → frontend reducer. `agent-conversation-service.ts` only does conversation CRUD + persistence.
 
 ### Entities & services
 - **ChatPersonalization** (`chat_personalization`) — first-run onboarding: role + company, background research, researched empty-state cards. See [chat personalization](./chat-personalization.md).
 - **AgentConversation** (`agent_conversation`) — per-user, per-platform, optionally per-project; `status` STREAMING/IDLE/ERROR, `activeRunId`, `messages` (ModelMessage[] JSONB), `uiMessages`, `summary`/`summarizedUpToIndex` for compaction.
 - **ChatRolloutUser** (`chat_rollout_user`) — cloud rollout cohort; `chattedAt` drives the cap.
-- **UserMemory** (`user_memory`) — one row per (platformId, userId): `instructions` (nullable text) + `memories` (jsonb string[]); capped at 50 facts × 280 chars and 4000 chars of instructions (`chatHelpers.capMemories`).
+- **UserMemory** (`user_memory`) — one row per (platformId, userId): `instructions` (nullable text) + `memories` (jsonb string[]); capped at 50 facts × 280 chars and 4000 chars of instructions (`agentHelpers.capMemories`).
 - Tool logic in `ee/agent/`; shared tool phase/classification in `core/shared/.../ee/agent/`.
 
 ### How it works
@@ -26,7 +26,7 @@ The chat LLM loop runs in the **worker**, not the API. Send path: `agent-convers
 
 ### Turn liveness — three independent timers (get this right)
 A turn is kept alive / reclaimed by three separate mechanisms in `execute-agent-run.ts`; confusing them causes "chat randomly stops" bugs:
-- **Heartbeat** (`HEARTBEAT_INTERVAL_MS` 15s): a `setInterval` that bumps `conversation.updated` (via `heartbeatChatConversation` RPC) + sends an empty keepalive chunk, so a live-but-slow turn is never reclaimed as stale.
+- **Heartbeat** (`HEARTBEAT_INTERVAL_MS` 15s): a `setInterval` that bumps `conversation.updated` (via `heartbeatAgentConversation` RPC) + sends an empty keepalive chunk, so a live-but-slow turn is never reclaimed as stale.
 - **DB stale-recovery** (`STREAMING_STALENESS_TIMEOUT_MS` 90s, `agent-helpers.ts`): on-read (`getConversationOrThrow`) + a per-minute sweep flip any STREAMING conversation whose `updated` is >90s old back to IDLE. The heartbeat is what holds this off.
 - **Stream idle watchdog** (`STREAM_IDLE_TIMEOUT_MS` 90s, in `streamChunksToClient`): aborts the turn if the drain-stream reader is silent 90s. It must be SUSPENDED while legitimate silent work is in flight — pending tool calls AND in-flight reasoning (`reasoning-start`→`reasoning-end`). **Reasoning-awareness was missing and caused the bug where long "thinking" on the Expert tier randomly aborted a healthy turn** (a >90s gap between reasoning deltas looked like a wedge). Backstop for a genuine mid-reasoning wedge is `MAX_TURN_WALL_CLOCK_MS` (20 min).