[pull] main from activepieces:main

brain/knowledge/ai-intelligence/testing-model-tiers-locally.md

@@ -0,0 +1,15 @@
+---
+icon: 🧪
+---
+# Testing Model Tiers Locally
+
+How to see the published tier file drive chat, agents and the pickers on a dev box, without publishing anything. The server fetches through `safeHttp`, which blocks loopback, so the allow-list line is not optional.
+
+1. Write a tier file with both lists. Put something in it that the release does not ship, such as a chat-only tier `turbo` and a relabelled `smart` ("Expert (from file)"), so you can tell the file from the bundled constants at a glance. Every tier needs `id`, `label`, `modelId` (an OpenRouter id) and `thinkingBudget`; the file needs `version`, `publishedAt`, `publishedBy`, `tiers`, `defaultTierId`, and `chatTiers` with `chatDefaultTierId` together or not at all.
+2. Serve it: `python3 -m http.server 8787` in that folder.
+3. In `.env.dev`: `AP_MODEL_TIERS_URL=http://127.0.0.1:8787/tiers.json` and `AP_SSRF_ALLOW_LIST=127.0.0.1`. Start with `npm run dev`. The boot log must not say "Failed to load the AI model tiers file".
+4. Chat page: the picker shows the chat list, with the chat default selected on a fresh chat. Agent model picker: the flow list only. A tier on one list and not the other proves the surfaces are told apart.
+5. Send one chat message without picking. The `Chat config resolved` log line carries `tier.id`, `surface` and the model id the turn ran on.
+6. To see a re-point, change a `modelId` in the file, restart the backend (the catalog cache is in-process, 15 minutes) and reload the page (the web refetches every 15 minutes). Same label, new model in the log.
+
+The Activepieces provider must be enabled for chat under Platform, AI, which needs `AP_EDITION=cloud` and `AP_OPENROUTER_PROVISION_KEY` on the dev box.

brain/knowledge/decisions/000041-ai-model-tiers-are-published-to-the-cdn-from-a-second-console.md

@@ -53,8 +53,11 @@ the bucket; a key scoped to `ai/pricing*` is the follow-up.
   in ap-analytics, `packages/server/utils/src/model-tier-catalog.ts` here. Change one without the
   other and Activepieces rejects the file and silently keeps serving the old tiers. The reader
   ignores keys it does not declare, which is what kept the console publishable while it still sent
-  the credit fields; ap-analytics dropped those on 2026-09-20 and added `nativeModelId`, so the two
-  schemas name the same five tier fields again.
+  the credit fields; ap-analytics dropped those on 2026-09-20. Since 2026-09-27 the two schemas
+  name the same nine top-level keys (`version`, `publishedAt`, `publishedBy`, `tiers`,
+  `defaultTierId`, `fastTierId`, `chatTiers`, `chatDefaultTierId`, `chatFastTierId`). Per tier they
+  differ by one field: the console dropped `nativeModelId` again on 2026-09-27, while Activepieces
+  keeps it optional and falls back to the bundled tier's value.
 - A tier change is no longer visible in `git log` of this repo. The audit trail is the CDN history
   prefix.
 - The reader **fails closed**: bad schema, no tiers, or a missing default tier all fall back to the
@@ -82,4 +85,4 @@ the bucket; a key scoped to `ai/pricing*` is the follow-up.
   `CLAUDE.md` and the package names all say config-console. It was created as `config-console` and
   renamed on 2026-09-02. A clone made before the rename pushes into a 404; fix it with
   `git remote set-url origin git@github.com:activepieces/ap-analytics.git`.
-- **Internal tools run on DigitalOcean, but not all the same way.** `activepieces/discover` uses a **droplet** driven by Actions over SSH (`.github/workflows/deploy.yml` — build, rsync, `pm2 reload`), with nginx and certbot in front. `ap-analytics` uses **App Platform** instead: DO builds the Dockerfile, redeploys on push by itself, and issues TLS, so there is no server to log into and the repo carries no deploy workflow at all. Neither repo says which it is, so check the DO dashboard before assuming. Runbook: `DEPLOYMENT.md` in ap-analytics.
+- **Internal tools run on DigitalOcean, but not all the same way.** `activepieces/discover` uses a **droplet** driven by Actions over SSH (`.github/workflows/deploy.yml` — build, rsync, `pm2 reload`), with nginx and certbot in front. `ap-analytics` uses **App Platform** instead: DO builds the Dockerfile, redeploys on push by itself, and issues TLS, so there is no server to log into and the repo carries no deploy workflow at all. That repo takes no pull requests: commits go straight to `main`, and the push is the deploy. Neither repo says which it is, so check the DO dashboard before assuming. Runbook: `DEPLOYMENT.md` in ap-analytics.

brain/knowledge/memory.md

@@ -1,5 +1,7 @@
 - 2026-09-26 — PR 2 of the configurable tiers work (`feat/tier-resolution`, ENG-767) moved chat, agent and Run Agent model resolution onto the published tier file, with a required `surface` argument on the whole `agent-model-resolution` family and `surfaceOf({ source })` as the one source-to-list rule. Three calls made that day, recorded on the AI Providers page: a removed tier resolves to its surface default with a warning on every door; the `chat-tool-billing.ts` tier-label and FLOW_STEP-provider bug stays as it is and lives in ENG-772; the end-to-end re-point check runs on canary after merge with `AP_MODEL_TIERS_URL`, not locally.
 - 2026-09-24 — `test-unit` (api unit tests) only joined the PR pipeline on 2026-09-20 (#15653). A branch whose last CI run predates that has never had its unit tests run in CI; expect first-time failures when it merges `main`, as #15550 did (a test mock disagreeing with the real code, not a merge problem).
+- 2026-09-27 — PR 3 of the configurable tiers work (`feat/tier-endpoint`, ENG-768): `GET /v1/ai-providers/tiers` serves both lists and both web pickers read it; a new chat no longer hardcodes `smart`. Found on the way: the live `ai/pricing.json` republished on 2026-09-26 carried `defaultTierId: fast`, which never reached web chats only because the web sent `smart` explicitly; it is being republished as `smart`.
+- 2026-09-28 — PR 4 of the configurable tiers work (`feat/tier-ai-piece`, ENG-752): the AI piece's managed text dropdown (piece 0.12.0, `minimumSupportedRelease` 0.93.0) offers the published flow tiers and stores the tier id, falling back to the bundled tier ids, never `/models`. `aiModelResolution.resolveTierModelId` (`src/app/ai/`, CE) turns the id into today's flow model in `ai-execute-controller` before enqueue, skips image steps, and logs `model.id` and `tier.id` on both branches. Merge gate: `origin/main`'s root version must be below 0.93.0 when it merges.
 - 2026-09-20 — Found while planning the AI tier work: managed AI piece steps under-billed for about 23 days, between #14942 (2026-08-24) and #15494 (2026-09-16). #14942 changed the AI piece's `provider` prop to store an object (`{ provider, configId }`), but `flow-run-ai-usage-extractor` read that field with a string-only helper, so it resolved to `'unknown'`, failed the `provider !== ACTIVEPIECES` check in `resolveAiCreditWeight`, and charged **1 credit** per step instead of the model's table weight. It hit the five non-agent AI steps only, and only those configured or re-saved inside that window — a step saved earlier still held a plain string and billed correctly. `run_agent` was never affected, because it stores `aiProviderModel` instead. Nothing logged an error, and every test in the file passed a string provider, so the suite was green the whole time. The code is gone (#15494 deleted the extractor and bills on real provider cost now), so there is nothing left to fix — but the cloud credit events for that window are still wrong, and nobody has reconciled them.
 - 2026-09-20 — The live `ai/pricing.json` (version 1, published 2026-09-14) carried a leftover test tier `tier-4` / "New tier" on `anthropic/claude-opus-4.8`, created by clicking "add tier" in the pricing console and publishing. It was removed on 2026-09-24 by deleting the whole object, so the key is 404 until someone publishes again from the console; every server falls back to the bundled tiers and warns every five minutes until then. Next time, edit the row and publish; do not delete the object.
 - 2026-09-06 — Follow-up (Option B, "stub-writer cascade"): fix `engine-run-callback-service.ts` `ensureLogsFileExists` (line 84-86) so a fresh log-file stub isn't written with `{ steps: {}, tags: [] }` when the underlying file is missing — that stub is what the next BullMQ retry downloads and trips `EmptyResumeStateError` on. Options: (a) store `internalError` on `flow_run` itself so no stub is needed; (b) use a distinguishable stub shape (`executionState: null`) that the engine reader rejects fast; (c) `UnrecoverableError` in worker on state-loss errors so BullMQ doesn't retry the stub. Handles manual-admin-ops case (case #5 in the retention-hunt discussion) and any non-retention internal-error that would otherwise loop. Ping user once C.2/W.1/#2/#4 have landed.

brain/knowledge/pieces-engine/pieces.md

@@ -23,7 +23,7 @@ The metadata catalog of automation integrations ("pieces") — each a named inte
 - EE/Cloud per-piece and per-action/trigger visibility flows through `resolveVisibility` (`ee/pieces/filters/piece-filtering-utils.ts`), which returns a `VisibilityPolicy` or `null` on CE / when `platformId`/`projectId` is nil (callers treat `null` as no filtering). The policy is derived from the project's **piece set** (via `project.pieceSetId`, falling back to the platform Default).
 - Install and sync also enqueue a tool-search reindex, but only when `isToolSearchEnabled()`; no-op otherwise.
 - `delete` removes all versions sharing the name on that platform, and only for `CUSTOM` pieces the caller owns.
-- **A piece silently vanishes from the list when its `minimumSupportedRelease` is ahead of the root `package.json` version.** `fetchLatestPieces` filters every piece through `isSupportedRelease(apVersionUtil.getCurrentRelease(), piece)`. Pieces are routinely merged targeting the *next* release, so on `main` a couple dozen are invisible locally until the version bump lands. No warning is logged — it just isn't there.
+- **A piece silently vanishes from the list when its `minimumSupportedRelease` is ahead of the root `package.json` version.** `fetchLatestPieces` filters every piece through `isSupportedRelease(apVersionUtil.getCurrentRelease(), piece)`. Pieces are routinely merged targeting the *next* release, so on `main` a couple dozen are invisible locally until the version bump lands. No warning is logged — it just isn't there. The same gate hides the version from exact-version lookups (`findExactVersion`), so a flow step saved on it fails to load on an older server, and `AP_DEV_PIECES` does not bypass it: the database copies are dropped by name and the dev copy by release, so the piece vanishes entirely. To test a piece that targets the next release, bump the root `package.json` version locally and do not commit it. A server that already reports that release but lacks the server code the piece needs is the dangerous case; check `origin/main`'s root version at merge time.
 - **`/v1/pieces` is uncached, so a new `piece_metadata` column must be added to `PIECE_COLUMNS_WITHOUT_TRANSLATIONS` or it silently will not appear in the list.** `pieceCache` caches only the *registry* (name/version/platformId tuples) for `get`/`findExactVersion`/`registry`; the list path never touches it, and the `dedupe()` helper in `piece-metadata-service.ts` is in-flight-only (it deletes the key in `finally`), so it collapses concurrent callers but caches nothing between requests. Every request therefore re-runs the whole pipeline against the DB. Because of that, `fetchLatestCompatiblePiecesFromDB` selects an explicit column list that deliberately omits `i18n` — that column holds every locale for the piece (~50 MB across the catalogue, written by `pieceInstallService`) and the list path discards it, so loading it was pure waste. Non-English requests now pull only the one locale in a second query (`pm."i18n" -> :locale`) and hand it to `pieceTranslation.translatePiece`, which also shrinks its `JSON.parse(JSON.stringify(...))` deep clone. The cost is that the explicit select is a list to keep in sync with the entity.
 - **Piece sorting mutates its input, which is why the list result cannot simply be cached.** `pieceSorting`'s comparators and `reverseIfDesc` sort in place (`pieces.sort(...)`, `pieces.reverse()`). That is safe only because the array is rebuilt per request; making them non-mutating is a prerequisite for ever caching the built list, or requests would reorder shared state under each other. Same trap in `pieceTranslation.translatePiece`: when the piece has no entry for the locale it returns the *original* object rather than a clone, so any `piece.i18n = undefined` after it would mutate the shared row — take the no-translations branch before calling it.
 - **DynamicProperties clears its value before it knows the new schema, so the merge source must be a snapshot.** `DynamicPropertiesImplementation` re-fetches the child schema on every refresher change, clearing the form value synchronously and re-populating it in the mutation callback. The merge source for `getDefaultValueForProperties` has to be a `lastKnownValue` ref captured *before* the clear — reading `form.getValues()` in the callback sees the cleared `null` and defaults every child (GIT-1514). The snapshot must be spread-cloned: RHF `getValues(name)` hands back the live object and the clear's `setValue(...child, null)` mutates it in place. Guard the ref with `isNil` so it survives rapid successive changes, where later effect runs already observe `null`.

packages/core/execution/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@activepieces/core-execution",
-  "version": "0.23.3",
+  "version": "0.23.4",
   "type": "commonjs",
   "main": "./dist/src/index.js",
   "scripts": {

packages/core/execution/src/lib/workers/worker-contract.ts

@@ -159,6 +159,9 @@ export type AgentConfigResponse = {
     modelId: string
     fastModelId: string
     imageModelId?: string
+    searchCredentials?: AiProviderCredentials
+    searchModelId?: string
+    imageCredentials?: AiProviderCredentials
     systemPrompt: string
     messages: unknown[]
     allMessages: unknown[]

packages/core/shared/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@activepieces/shared",
-  "version": "0.185.1",
+  "version": "0.186.0",
   "type": "commonjs",
   "sideEffects": false,
   "main": "./dist/src/index.js",

packages/core/shared/src/lib/management/ai-tools/index.ts

@@ -13,6 +13,7 @@ export enum AiToolProvider {
     FIRECRAWL = 'firecrawl',
     APIFY = 'apify',
     FAL = 'fal',
+    AI_PROVIDER = 'ai_provider',
 }
 
 export const AiToolAuthConfig = z.object({
@@ -23,6 +24,12 @@ export type AiToolAuthConfig = z.infer<typeof AiToolAuthConfig>
 export const AiToolProviderConfig = z.record(z.string(), z.unknown())
 export type AiToolProviderConfig = z.infer<typeof AiToolProviderConfig>
 
+export const AiProviderToolConfig = z.object({
+    aiProviderId: z.string().min(1, formErrors.required),
+    modelId: z.string().min(1, formErrors.required).optional(),
+})
+export type AiProviderToolConfig = z.infer<typeof AiProviderToolConfig>
+
 export const AiToolConfig = z.object({
     ...BaseModelSchema,
     platformId: z.string(),
@@ -46,7 +53,7 @@ export type AiToolConfigWithoutSensitiveData = z.infer<typeof AiToolConfigWithou
 export const CreateAiToolConfigRequest = z.object({
     capability: z.enum(AiToolCapability),
     provider: z.enum(AiToolProvider),
-    auth: AiToolAuthConfig,
+    auth: AiToolAuthConfig.optional(),
     config: AiToolProviderConfig.optional(),
     enabled: z.boolean().optional(),
 })
@@ -73,3 +80,8 @@ export const GetEnabledAiToolsResponse = z.object({
     imageGeneration: ResolvedAiTool.optional(),
 })
 export type GetEnabledAiToolsResponse = z.infer<typeof GetEnabledAiToolsResponse>
+
+export type AiProviderToolChoices = {
+    webSearch?: AiProviderToolConfig
+    imageGeneration?: AiProviderToolConfig
+}

packages/pieces/community/ai/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@activepieces/piece-ai",
-  "version": "0.11.0",
+  "version": "0.12.0",
   "type": "commonjs",
   "main": "./dist/src/index.js",
   "types": "./dist/src/index.d.ts",

packages/pieces/community/ai/src/index.ts

@@ -12,7 +12,7 @@ import { runAgent } from "./lib/actions/agents/run-agent";
 export const ai = createPiece({
   displayName: "AI",
   auth: PieceAuth.None(),
-  minimumSupportedRelease: '0.92.0',
+  minimumSupportedRelease: '0.93.0',
   categories: [
     PieceCategory.ARTIFICIAL_INTELLIGENCE,
     PieceCategory.UNIVERSAL_AI,