diff --git a/brain/knowledge/ai-intelligence/ai-providers.md b/brain/knowledge/ai-intelligence/ai-providers.md index 0f4148f782bb..3693e5633703 100644 --- a/brain/knowledge/ai-intelligence/ai-providers.md +++ b/brain/knowledge/ai-intelligence/ai-providers.md @@ -63,8 +63,11 @@ the background every 15 minutes with a 5-minute back-off after a failure. `warmU `defaultModelIdForProvider`, `resolveModelIdForAnalytics`) and the fast-model helpers in `agent-helpers.ts` read it, every one with a required `surface` argument. `surfaceOf({ source })` in that file is the only place the run source maps to a list (FLOW_STEP and AGENT to flow, CHAT and AGENT_BUILDER to chat), and `agent-config-rpc.ts` -derives `namesItsOwnModel` from it. `AP_MODEL_TIERS_URL` overrides -the URL. The file is published from the console in `ap-analytics`; see decision 000041. +derives `namesItsOwnModel` from it. The AI piece's text steps resolve through `aiModelResolution.resolveTierModelId` +(`src/app/ai/ai-model-resolution.ts`, CE), always on the flow surface, in `ai-execute-controller.ts` before the job is +enqueued. `AP_MODEL_TIERS_URL` overrides the URL. The file is published from the console in `ap-analytics`; see +decision 000041. The console publishes both lists and both fast pointers on every Publish, and on its first load +after a file without chat keys it seeds the chat list from the live flow list. ### Gotchas - **A catalog change takes up to ~2 days to reach a dropdown, through three caches in series.** CDN edge (`--cache-control max-age=3600`, 1h) → the server's in-memory catalog (`CATALOG_TTL_MS`, 24h) → `modelsCache` (flushed by the nightly `cron.schedule('0 0 * * *')`). So "I republished but the UI still shows the old price" is expected, not a bug; restarting the API short-circuits the two in-process layers. The edge TTL is deliberately 1h and not the 604800 that `publish-embed-sdk.yml` uses — that path is version-stamped, ours is a stable key rewritten in place, so a week-long edge cache would pin stale prices for a week. - **`Sync AI Model Catalog` is the only workflow that proves the CDN S3 credentials still work — a green `Publish Embed SDK` proves nothing.** Both read the same `CDN_S3_ACCESS_KEY_ID` / `CDN_S3_SECRET_ACCESS_KEY` / `CDN_S3_BUCKET` / `CDN_S3_ENDPOINT` secrets, but `publish-embed-sdk.yml` guards its upload job on `needs.check.outputs.publish == 'true'`, which is false whenever the version in `packages/ee/embed-sdk/package.json` is already on the CDN. Its daily cron then finishes in about nine seconds with the upload skipped and the run still marked success, so the key can rot for weeks behind a green tick — that is how the model-catalog cron came to fail with `InvalidAccessKeyId` on three consecutive Mondays while the embed SDK looked healthy. To date a dead key, find the last embed-SDK run whose `publish` job actually ran (`gh run list --workflow=publish-embed-sdk.yml`, look for a duration in minutes rather than seconds) and compare it against the first model-catalog failure. To test a rotated key, dispatch `sync-model-catalog.yml` — it uploads unconditionally. @@ -80,13 +83,17 @@ the URL. The file is published from the console in `ap-analytics`; see decision - **Since the AI steps moved to the worker, the server sees every step's model id, and `ai-execute-controller` is the one place to act on it.** The piece POSTs `/v1/ai/execute` with `provider` and `modelId`, the controller builds the `EXECUTE_AI` job from that body, and the worker runs whatever model id the job carries. So anything that needs to rewrite or validate the model a flow step is about to use belongs there, not in the piece — one server-side edit covers all five AI steps, and an old pinned piece bundle keeps working because it only forwards the string it stored. This is what makes storing a tier id in a step cheap: the bundle never has to understand it. Note the agent step does **not** come through here — it resolves separately over `getAgentConfig`, in `agent-config-rpc`. Two conditions sit under that. **An old bundle only forwards the string from `0.11.0` up** — `AI_PIECE_COST_BILLING_VERSION`, the first version whose `ai-step.ts` POSTs `v1/ai/execute`. A step pinned below it runs the pre-worker code that calls the provider from the sandbox, so a tier id there would reach OpenRouter as the literal string. That case is already closed: `migrate-v26-ai-piece-cost-billing` bumps every saved `piece-ai` step below `0.11.0` up to it, which is why a later migration can rewrite what a step stores without touching `pieceVersion` for the runtime. **And resolve on the API, never in the worker**: `ExecuteAiRequest.modelId` is a bare `z.string()` so a tier id passes validation, `getEffectiveProviderAndModel` and `aiUtils.buildWebSearchToolsOrThrow` both read `data.modelId` as a real model, and the `resolveAiProvider` RPC does not carry the model at all, so using it as the seam would mean widening the worker contract. - **Managed billing can only price an OpenRouter call, and a call it cannot price is simply not billed.** `openRouterCostOf` (`server/utils/activepieces-ai-cost.ts`) reads `providerMetadata['openrouter'].usage.cost`, with one fallback that GETs `openrouter.ai/api/v1/generation?id=` and takes `total_cost`. If neither answers, `createBillingInfo` returns `undefined` and the turn is never reported — it runs, the customer keeps the output, and nothing is charged. Nothing logs "unbilled". This is what actually welds the managed provider to OpenRouter, more than the client does: `create-language-model.ts` shares one branch for `OPENROUTER` and `ACTIVEPIECES`, every tier `modelId` is an OpenRouter id, and `openrouter-api.ts` mints the per-platform key from `OPENROUTER_PROVISION_KEY`. So putting managed credits on any other gateway needs a second cost extractor and a second generation lookup **first**, or every call through it bills zero. For reference, no Vercel AI Gateway code exists here at all — the only gateway in the repo is Cloudflare's (`CLOUDFLARE_GATEWAY`, the `ai-gateway-provider` package), and `@ai-sdk/gateway` appears only as a transitive dep of `ai` that nothing imports. A BYOK user can already reach Vercel's gateway today through `AIProviderName.CUSTOM`, which takes a base URL plus hand-typed model ids. - **A tier id that no longer exists resolves to the default tier of its surface, on every door, with a warning.** `resolveNamedModelId` (the path FLOW_STEP and AGENT take) tells a tier id from a model id by the slash rule: a bare slug is a tier id and goes through `resolveTier`, which returns the surface default for an id the file no longer carries and logs `Tier is no longer published; running the surface default` with the tier id and surface; a slashed id is a managed model id and is still refused with `The model "X" is not available on Activepieces AI credits` unless the release ships it or a published tier on that surface carries it. Every other source goes through `resolveModelIdForProvider` → `resolveTier`, the same default. So removing a tier from the file never fails a flow or an agent; it moves them onto the default until they are re-picked. The console may keep saying that removing a tier is safe. Decided 2026-09-26 with PR 2 of the configurable tiers work. -- **The console's Publish writes the live key directly; there is no draft, no staging and no filter.** `ai-pricing-service.ts` in `ap-analytics` writes `LIVE_PRICING_KEY` on Publish, and its "add tier" row seeds `id: tier-N`, `label: New tier`, so a half-finished experiment is one click from every server's cached tier list, with only the CDN `max-age=300` plus the 15-minute in-process TTL in between. Test against a local file with `AP_MODEL_TIERS_URL` (add the host to `AP_SSRF_ALLOW_LIST` for loopback), never by publishing. -- **The AI piece's managed dropdown only ever offers tier models, so every stored managed model id came from a tier pick.** `props.ts` (`packages/pieces/community/ai/src/lib/common/props.ts`) builds its options from `ACTIVEPIECES_CHAT_TIERS`, the constant compiled into the release, in two places: `managedModelLabel` supplies the label, and the same lookup is the `.filter` that decides which models are offered at all. There is no way to pin a model deliberately through it. Two consequences. A re-pointed tier is invisible to the AI piece in both directions: a saved step keeps its concrete model id, and a new step cannot pick the new model because it matches no bundled tier. And any migration from stored model ids to tier ids has a trivially complete mapping, since each stored id is one of the three bundled tier model ids of the release it was saved on. Until the dropdown emits tier ids, the AI piece only ever follows the release. +- **The console's Publish writes the live key directly; there is no draft, no staging and no filter.** `ai-pricing-service.ts` in `ap-analytics` writes `LIVE_PRICING_KEY` on Publish, and its "add tier" row seeds `id: tier-N`, `label: New tier`, so a half-finished experiment is one click from every server's cached tier list, with only the CDN `max-age=300` plus the 15-minute in-process TTL in between. Test against a local file with `AP_MODEL_TIERS_URL` (serve it on `127.0.0.1` and add `127.0.0.1` to `AP_SSRF_ALLOW_LIST`, which takes IPs or CIDRs, not hostnames, so `localhost` is still refused), never by publishing. +- **A console older than the two-list change, or a rollback to one, strips the chat keys on read, and its next Publish drops them with no diff line.** The console parses the live file with a zod object, and zod strips keys it does not declare, so `chatTiers`, `chatDefaultTierId` and `chatFastTierId` vanish from the draft the moment an old console reads it. Publish writes the draft back, and the history diff shows nothing because that console never knew the keys. Activepieces then serves chat from the flow list again, silently, since the chat keys are optional. After any console rollback, republish from a current build and check `curl -s https://cdn.activepieces.com/ai/pricing.json | jq keys` lists all nine keys. +- **A managed text step stores a tier id from AI piece 0.12.0 on, and every managed model id stored before that came from a tier pick.** For the managed provider and a text step, `props.ts` (`packages/pieces/community/ai/src/lib/common/props.ts`) reads `GET /v1/ai-providers/tiers` with the engine token and offers the `flow` list as `{ label, value: tier.id }`. On any failure (a server without the route, a network error, an empty list) it offers the release's `ACTIVEPIECES_CHAT_TIERS`, still as tier ids, so a step saved on 0.12.0 always holds a tier id. `/models` is still called for managed image steps (`managedImageModelLabel` filters to `ACTIVEPIECES_IMAGE_TIERS`) and for own-key providers. Up to 0.11.0 the dropdown filtered `/models` to the bundled tier models and stored the concrete id, with no way to pin a model deliberately. So the migration from stored model ids to tier ids (ENG-753) has a trivially complete mapping: each stored id is one of the three bundled tier model ids of the release it was saved on. Until it runs, those steps follow the release, not the file. +- **A managed text step on 0.12.0 that still holds a concrete model id shows 'Select AI Model' and keeps running that model.** The options are tier ids now. `searchable-select.tsx` renders the placeholder when the value matches no option and keeps the value, and `dynamic-dropdown-piece-property.tsx` only clears it when a refresher (here `provider`) changes. The run is unaffected, because the slash rule passes a concrete id through. It only happens when something bumps `pieceVersion` in place, as a flow migration does. Options are fetched against the step's saved exact version, so a step left on 0.11.0 keeps its old options, and the update dialog's minor bump resets every input (`getInputAfterVersionChange`, `MINOR_OR_MAJOR`), so the user re-picks the model with everything else. +- **The AI step resolver skips image steps, because image tiers reuse the text tier ids.** `ACTIVEPIECES_IMAGE_TIERS` are `fast` / `smart` / `premium` too, and `aiJobFor`'s shared `modelId` feeds all five actions. Resolving a `GENERATE_IMAGE` job against the flow list would hand an image step a text model. The controller gates on `body.action`; any future image-tier resolution needs its own list. - **`AIProviderModel` is declared twice, and a field added to one half is stripped on the wire.** There is one in `core/piece-types` (what pieces import, since they cannot import `@activepieces/shared`) and one in `core/shared` (what the server and the response schema use). They are not re-exports of each other — today the shared copy carries `metadata` and the piece-types copy does not. Fastify serialises the response against the *shared* schema and drops every key it does not declare, so adding a field to the piece-types copy alone type-checks in the piece, type-checks in the server, and arrives as `undefined`. A published tier label was nearly shipped that way. Any new field on a provider model needs both files. - **No image model belongs to a chat tier, so a tier-based filter empties the image picker.** The managed provider is OpenRouter underneath and `openrouter-provider.listModels` tags a model `IMAGE` when its `architecture.output_modalities` says so — `ACTIVEPIECES` is deliberately absent from `NO_IMAGE_GENERATION_PROVIDERS`, and `google/gemini-2.5-flash-image` and friends are real managed models. But `ACTIVEPIECES_CHAT_TIERS` is three *chat* tiers, so no image model can ever match one. Any filter of the form "a managed model must belong to a tier" therefore drops every image model silently, leaving `aiProps({ modelType: 'image' })` with an empty dropdown and no error. Gate that filter on `modelType === 'text'`. - **The api test suite runs on the bundled tiers, never the live file.** `packages/server/api/vitest.setup.ts` points `AP_MODEL_TIERS_URL` at `model-tiers.invalid`, so boot warm-up fails fast and every test resolves against `ACTIVEPIECES_CHAT_TIERS`. CI runners have egress, and the first time a resolver read the file in CI the live `defaultTierId` was `fast`, which turned a test asserting the release default (Expert) red with no code change behind it. A test that wants the published shape mocks `modelTierCatalog` (`test/unit/app/ee/agent/model-tier-fixture.ts`); an integration test that boots the app cannot be handed a file. - **A published tier without `nativeModelId` keeps every BYOK key on the release's model for that tier id.** `nativeModelIdFor` (`ee/agent/agent-model-resolution.ts`) takes the tier's own `nativeModelId`, else the `nativeModelId` of the bundled `ACTIVEPIECES_CHAT_TIERS` entry with the same id, else null, which sends a curated provider to its first curated model and a catalogue-less provider to the first text model its key lists. The fallback is keyed by tier id only, so two lists sharing an id resolve to the same native model on every own-key provider, whatever their managed models are. The BYOK tier map (PR 6 of the configurable tiers work) is the real fix. - **`premium` pairs `modelId: anthropic/claude-opus-4.8` with `nativeModelId: claude-opus-4-7` on purpose.** The managed tier runs Opus 4.8 through OpenRouter; a customer's own Anthropic key runs Opus 4.7. The reason is cost on the customer's key: Heavy on a BYOK key must not silently move the customer onto the most expensive model without them choosing it. The console's `defaults.ts` carries the same pair. Do not "fix" either copy. +- **Whatever the console publishes is in every Cloud customer's picker within about 20 minutes on a fresh page load, and about 35 minutes on a page left open: no draft, no staging, no filter.** `GET /v1/ai-providers/tiers` (`ListModelTiers` in `ai-provider-controller.ts`) returns `modelTierCatalog.current(surface)` for both lists, keyed `flow` and `chat`, to USER and ENGINE principals with no platform or project scope, because the tiers are global; the response schema puts only `id`, `label` and `modelId` on the wire. The web reads it through one hook, `aiProviderQueries.useModelTiers(surface)` (`features/platform-admin/hooks/ai-provider-hooks.ts`): the chat selector passes `chat`, the agent picker passes `flow`, and nothing but a test checks which one a caller passed, since the surface argument the server requires does not reach this layer. Both pickers show the release's bundled tiers while the call is in flight or has failed, so they never render empty. The web query is stale after 15 minutes and refetches on that interval while the tab is visible (`staleTime` alone only gates the mount, focus and reconnect triggers, so a tab left open would never refetch), which is where the two bounds come from: CDN `max-age` 5 min plus the server's 15-min TTL for a fresh load, plus one 15-min client tick for an open page, and the server serves its old copy once while it refreshes. The AI piece's own model dropdown reads the same route from piece 0.12.0 (the props.ts bullet above). - **Every provider auth schema is written twice, and only one copy is the one the server reads.** `packages/core/utils/src/lib/ai-provider-credentials.ts` (plain zod) backs the API and the web app; `packages/core/piece-types/src/lib/ai-providers.ts` (`zod/mini`, so `.check(z.minLength(1))` and `z.optional(x)` instead of `.min(1)` / `.optional()`) is what `@activepieces/pieces-framework` re-exports to pieces and the engine. Adding a credential field to only one leaves the other silently stripping it — zod objects drop unknown keys, so the upsert still returns 200 and the failure surfaces later as a provider-side auth error. Change both. - **A schema change in `packages/core/utils` is invisible to another package's tests until you build it.** Sibling packages resolve `@activepieces/core-utils` through its `main` (`dist/src/index.js`), so vitest in `core/ai-providers` keeps parsing against the stale build and a new optional field reads back as dropped. `cd packages/core/utils && npm run build` first. The API dev server is the opposite: `serve` runs under tsx with the tsconfig path alias, so it reads `src` directly — but it only watches `packages/core/shared/src/**`, not `core/utils`, so a core-utils-only edit needs a restart to take effect. The same staleness makes a **branch** look green when it is not: `dist` is gitignored, so after `git checkout` it still holds whatever the previous branch built, and a test that depends on a field added by a sibling PR passes locally and fails in CI. Rebuild `core/utils` after every branch switch before trusting a run. @@ -173,7 +180,7 @@ the URL. The file is published from the console in `ap-analytics`; see decision - **Chat model tiers are Activepieces-only.** `ACTIVEPIECES_CHAT_TIERS` (`fast`/`smart`/`premium`, shown as Fast/Expert/Heavy) hold OpenRouter-shaped Anthropic ids, so they only mean anything for the ACTIVEPIECES and OPENROUTER chat providers. Any provider that declares `ALLOWED_CHAT_MODELS_BY_PROVIDER` (openai, anthropic, google) picks a real model id from that list instead. Naively stripping the tier's vendor prefix for every provider is what once sent `claude-haiku-4-5` to OpenAI and broke every message. - **A short model list in the key's picker is the vendor's catalog, not a truncation.** `listModels` returns whatever the provider's own `/models` endpoint gives and filters nothing except the key's `modelScope` allow-list. Anthropic ships roughly a dozen models, OpenAI ~80 (mostly embeddings/tts/whisper), while OpenRouter is an aggregator and returns 400+ from every vendor it proxies — so the counts differ by an order of magnitude by design. Anthropic pages at 20 by default, which is why its request pins `?limit=1000`. If the list shows exactly three Claude models, that is the *chat* dropdown reading the curated `ANTHROPIC_CHAT_MODELS`, a different surface from the admin picker. - **Read the chat model list through `aiProviderUtils.getCuratedChatModels({ provider })`.** It is the one accessor the server resolver (`agentHelpers.resolveModelIdForProvider`) and the chat dropdown share, so the two cannot drift; it returns `{ id, label }` or `undefined` — never an empty list, so callers may treat a result as non-empty. Labels come from the hardcoded `CHAT_MODEL_LABELS` (falling back to the id) rather than `AIProviderModel.name`, because the live `listModels` response cannot supply one for every provider: anthropic returns `display_name` and google `displayName`, but OpenAI's `/v1/models` returns ids only. -- **`conversation.modelName` carries either a tier id or a real model id** — it is a free string with no discriminator. A legacy tier id resolves to the tier's equivalent model when the provider ships it, else the provider's first curated model, so old conversations keep working after a provider switch. Note `premium` maps to opus 4.8, which the native anthropic list does not carry, so a legacy `premium` on anthropic lands on Sonnet. +- **`conversation.modelName` carries either a tier id or a real model id** — it is a free string with no discriminator. A legacy tier id resolves to the tier's equivalent model when the provider ships it, else the provider's first curated model, so old conversations keep working after a provider switch. Note `premium` maps to opus 4.8, which the native anthropic list does not carry, so a legacy `premium` on anthropic lands on Sonnet. A new web chat sends no model at all until the user picks one (`use-chat.ts` starts `modelName` at null): the server resolves the published chat default for that turn and the worker stamps the tier it ran onto the conversation, so the chat default lives in one place, the published file, and the picker shows it as selected by reading `chat.defaultTierId` from the tiers endpoint. Before that change the web sent `smart` explicitly on every new chat, which is why a published `defaultTierId` never reached web chats. **That holds only where the picker shows tiers.** An own-key provider with a curated model list shows concrete models, but a null model still resolves the published chat default tier and then that tier's `nativeModelId`, which is in the Anthropic curated list: a published default of `fast` ran Haiku on every Anthropic-key chat while the picker showed Sonnet, and the worker's tier stamp kept it there. So a new chat on a curated key sends the model the picker shows (`chatUtils.newChatModelName`); the tier picker sends null, and so does any chat whose picker is hidden (an agent or the agent builder: a saved agent's job carries the agent's own model anyway). The chat box does not mount until the chat provider has loaded, or a message sent in that window would go out with no model. Caught in review on #15838. - Chat-provider resolution is **first `enabledForChat` row wins, else the visible managed ACTIVEPIECES row** — `pickChatRow`, shared by `findAvailableChatProviderRow` and both list endpoints. So a platform with `[openai, activepieces]` both chat-enabled still resolves to **openai** (an explicit pick beats the fallback), while a platform with nothing flagged resolves to the managed row instead of to nothing. The flag a list endpoint reports is therefore the *resolved* one, not the stored column — `ai_provider.enabledForChat` can say `false` on the row chat actually runs on, and `ensureManagedProviderRow` no longer writes it. The client mirror is `aiProviderQueries.useChatProvider()` (`providers.find((p) => p.enabledForChat)`) — always read the resolved chat provider through it rather than re-deriving the rule inline. A project entry is chat-enabled when it owns the picked row, whichever of its keys carries the flag: ranking (`selected` > `except` > `all`, newest first) and chat selection answer different questions, so reading `rows[0].enabledForChat` makes the client report "no provider configured" whenever the chat-enabled key is not the ranking winner. `findBy()` has no `ORDER BY`, so "first" is not guaranteed stable when several providers are chat-enabled. - Listing providers is **not a pure read**: both `listConfigs` and `listForProject` go through `listVisibleRows`, which inserts the ACTIVEPIECES provider row when `aiCreditsEnabled && !activepiecesExists`. A `GET /v1/ai-providers` can therefore create a row. It also applies the hidden-provider filter (`plan.embeddingEnabled` hides the managed provider), which is why the client can trust its output without re-checking flags. - **A managed chat turn is charged in two events, not one.** The model cost arrives per model call from diff --git a/brain/knowledge/ai-intelligence/testing-model-tiers-locally.md b/brain/knowledge/ai-intelligence/testing-model-tiers-locally.md new file mode 100644 index 000000000000..05ce0b69e125 --- /dev/null +++ b/brain/knowledge/ai-intelligence/testing-model-tiers-locally.md @@ -0,0 +1,15 @@ +--- +icon: 🧪 +--- +# Testing Model Tiers Locally + +How to see the published tier file drive chat, agents and the pickers on a dev box, without publishing anything. The server fetches through `safeHttp`, which blocks loopback, so the allow-list line is not optional. + +1. Write a tier file with both lists. Put something in it that the release does not ship, such as a chat-only tier `turbo` and a relabelled `smart` ("Expert (from file)"), so you can tell the file from the bundled constants at a glance. Every tier needs `id`, `label`, `modelId` (an OpenRouter id) and `thinkingBudget`; the file needs `version`, `publishedAt`, `publishedBy`, `tiers`, `defaultTierId`, and `chatTiers` with `chatDefaultTierId` together or not at all. +2. Serve it: `python3 -m http.server 8787` in that folder. +3. In `.env.dev`: `AP_MODEL_TIERS_URL=http://127.0.0.1:8787/tiers.json` and `AP_SSRF_ALLOW_LIST=127.0.0.1`. Start with `npm run dev`. The boot log must not say "Failed to load the AI model tiers file". +4. Chat page: the picker shows the chat list, with the chat default selected on a fresh chat. Agent model picker: the flow list only. A tier on one list and not the other proves the surfaces are told apart. +5. Send one chat message without picking. The `Chat config resolved` log line carries `tier.id`, `surface` and the model id the turn ran on. +6. To see a re-point, change a `modelId` in the file, restart the backend (the catalog cache is in-process, 15 minutes) and reload the page (the web refetches every 15 minutes). Same label, new model in the log. + +The Activepieces provider must be enabled for chat under Platform, AI, which needs `AP_EDITION=cloud` and `AP_OPENROUTER_PROVISION_KEY` on the dev box. diff --git a/brain/knowledge/decisions/000041-ai-model-tiers-are-published-to-the-cdn-from-a-second-console.md b/brain/knowledge/decisions/000041-ai-model-tiers-are-published-to-the-cdn-from-a-second-console.md index 21210624a7e5..13fb0ae5f899 100644 --- a/brain/knowledge/decisions/000041-ai-model-tiers-are-published-to-the-cdn-from-a-second-console.md +++ b/brain/knowledge/decisions/000041-ai-model-tiers-are-published-to-the-cdn-from-a-second-console.md @@ -53,8 +53,11 @@ the bucket; a key scoped to `ai/pricing*` is the follow-up. in ap-analytics, `packages/server/utils/src/model-tier-catalog.ts` here. Change one without the other and Activepieces rejects the file and silently keeps serving the old tiers. The reader ignores keys it does not declare, which is what kept the console publishable while it still sent - the credit fields; ap-analytics dropped those on 2026-09-20 and added `nativeModelId`, so the two - schemas name the same five tier fields again. + the credit fields; ap-analytics dropped those on 2026-09-20. Since 2026-09-27 the two schemas + name the same nine top-level keys (`version`, `publishedAt`, `publishedBy`, `tiers`, + `defaultTierId`, `fastTierId`, `chatTiers`, `chatDefaultTierId`, `chatFastTierId`). Per tier they + differ by one field: the console dropped `nativeModelId` again on 2026-09-27, while Activepieces + keeps it optional and falls back to the bundled tier's value. - A tier change is no longer visible in `git log` of this repo. The audit trail is the CDN history prefix. - The reader **fails closed**: bad schema, no tiers, or a missing default tier all fall back to the @@ -82,4 +85,4 @@ the bucket; a key scoped to `ai/pricing*` is the follow-up. `CLAUDE.md` and the package names all say config-console. It was created as `config-console` and renamed on 2026-09-02. A clone made before the rename pushes into a 404; fix it with `git remote set-url origin git@github.com:activepieces/ap-analytics.git`. -- **Internal tools run on DigitalOcean, but not all the same way.** `activepieces/discover` uses a **droplet** driven by Actions over SSH (`.github/workflows/deploy.yml` — build, rsync, `pm2 reload`), with nginx and certbot in front. `ap-analytics` uses **App Platform** instead: DO builds the Dockerfile, redeploys on push by itself, and issues TLS, so there is no server to log into and the repo carries no deploy workflow at all. Neither repo says which it is, so check the DO dashboard before assuming. Runbook: `DEPLOYMENT.md` in ap-analytics. +- **Internal tools run on DigitalOcean, but not all the same way.** `activepieces/discover` uses a **droplet** driven by Actions over SSH (`.github/workflows/deploy.yml` — build, rsync, `pm2 reload`), with nginx and certbot in front. `ap-analytics` uses **App Platform** instead: DO builds the Dockerfile, redeploys on push by itself, and issues TLS, so there is no server to log into and the repo carries no deploy workflow at all. That repo takes no pull requests: commits go straight to `main`, and the push is the deploy. Neither repo says which it is, so check the DO dashboard before assuming. Runbook: `DEPLOYMENT.md` in ap-analytics. diff --git a/brain/knowledge/engineering/ci-pr-review-hygiene.md b/brain/knowledge/engineering/ci-pr-review-hygiene.md index 29c60561a18a..44c1c6ddc11c 100644 --- a/brain/knowledge/engineering/ci-pr-review-hygiene.md +++ b/brain/knowledge/engineering/ci-pr-review-hygiene.md @@ -35,6 +35,7 @@ Enforcement is the **`Codeowners review` repository ruleset** (active on the def - **Reading the parallel test job's log: trust the turbo `Failed:` line at the tail, not the scary lines in the middle.** The job runs about 21 turbo tasks at once, so `gh run view --job --log-failed` interleaves all of them. `FATAL ERROR: Reached heap limit` and `Caught fatal signal 9` next to `test/lib/sandbox/*.test.ts` are the sandbox suite killing isolates on purpose; `Error: Boom inside engine trigger hook` and the `HttpError` stderr dumps are fixture output. Strip the prefixes (`sed 's/^[^\t]*\t[^\t]*\t//'`), grep ` FAIL ` for the real assertion, and read the `Tasks: … Failed: ` summary to know which package actually failed. Engine handler tests timing out at 20 s in that job are load, not hangs: `flow-skip-progress.test.ts` (28 s) and then `flow-rerun.test.ts` timed out on consecutive runs of #15707 while each finishes in 0.6 s locally. The CI step starts `turbo run test` and the API's `test-unit test-ce test-ee test-cloud` in parallel with `&` on a 4-vCPU runner, and a branch that touches `packages/core/*` invalidates about 60 downstream tasks, so the engine suites compete with four Postgres-backed API suites for the same cores; main mostly hits the turbo cache and never sees it. The engine's `testTimeout` is now 60 s, the value the API's vitest config already used for the same job; the real lever, if it recurs, is not running the two turbo invocations concurrently. - **A spurious `core` request on a pieces PR is not always the lockfile — check for a second root file.** [#14558](https://github.com/activepieces/activepieces/pull/14558) looked like the lockfile case but its non-pieces files were `bun.lock` *and* `tsconfig.base.json`; the `core` request landed 6s after the commit that touched the tsconfig, not after the pieces push. Per-piece `paths` mappings generated into root `tsconfig.base.json` mean a pieces change can still reach a core-owned file, and no CODEOWNERS pattern can fix that — the file holds real compiler options and CODEOWNERS has no sub-file granularity. - **Greptile's Confidence Score prose is cumulative, so a low score mixes resolved findings with live ones — triage each claim separately rather than trusting or dismissing the number.** It edits one summary comment in place, and its "Files Needing Attention" list keeps naming findings that are already resolved and outdated: #14825 sat at 2/5 citing three files, two of which were a closed P1 and a duplicate view of the third. Read the *unresolved* review threads (`reviewThreads(first:60) { isResolved isOutdated }` over GraphQL — the REST comments endpoint carries no resolution state) and judge from those; re-trigger the review to refresh the score. It also re-raises the same class of finding each round with a new comment id, so a fix on one thread does not silence its sibling. **The score recomputes on a push, never on a reply** — so a finding Greptile *itself* withdraws still counts against it. On [#15243](https://github.com/activepieces/activepieces/pull/15243) the second pass read the pushed fix, filed zero new comments, and replied "this is not a valid finding… Withdrawing the comment" on its own P2; the summary comment was edited 67 seconds later, but only the footer (`Reviews (2)`, last-reviewed-commit link) changed — the headline stayed 2/5 and still justified itself with the finding just retracted. So compare the summary's `updated_at` against the review `submitted_at` from `gh api repos/:owner/:repo/pulls//reviews`: a summary newer than the last review pass has usually only had its footer bumped. But do not read "cumulative" as "stale": a 2/5 on [#14934](https://github.com/activepieces/activepieces/pull/14934) listed three files, two carrying findings already fixed and one naming a real bypass nobody had tested — the approve path published without the guard the direct path ran. The resolved-thread count is the right gate for *merging*, and a weak signal about *content*, because a finding Greptile states only in the summary never becomes a thread to resolve. Read the summary's file list even at zero unresolved threads, and check each named file against the code. The same applies to a human reviewer's tool-generated summary: Amr's review on #15707 said "Blockers: 4" and named them, but only two had an inline thread; the other two never left his tool. Count the threads against the summary before triaging, and answer the missing ones in a PR comment so the record shows they were checked. +- **Greptile marks down a PR whose title describes the code instead of what the user gets.** A custom rule in the Greptile dashboard (not in this repo) deducts half a point and posts a P2 "Implementation-focused PR title" on a title like `feat(ai): the managed model dropdown stores a tier, not a model`. Write the title as the user-visible change and keep the Conventional Commit prefix, lowercase after the type ("Validate PR title" and commitlint check it), e.g. `feat(ai): managed AI steps follow their tier when it moves to a newer model`. Fix it with `gh pr edit --title`; the `edited` event reruns the title check, and no commit is needed. - **`api` and `worker` resolve `@activepieces/shared` and the `core-*` packages from `dist`, so a package-local `tsc --noEmit` there reports whatever was built last.** Both `packages/server/api/tsconfig.app.json` and `packages/server/worker/tsconfig.lib.json` set `"paths": {}`, clearing the inherited source mappings, so an export added to `packages/core/shared/src` shows up as `TS2305: has no exported member` until that package is rebuilt — the error names your file and is not about your file. `web` is not affected for the packages it actually imports: `vite.config.mts` aliases `@activepieces/shared`, `@activepieces/pieces-framework`, `core-utils`, `core-formula`, `core-piece-types` and `core-execution` to their `src`. Nothing else is aliased there, so check `vite.config.mts` before assuming a package is exempt — `@activepieces/ai-providers` (note the name: no `core-` prefix) has no Vite alias and resolves through `dist`, though it is mapped to source in `tsconfig.base.json` and web does not import it today. `npx turbo run build --filter=` needs no manual pre-build, since `build` declares `dependsOn: ["^build"]` and builds dependencies first; only the direct `tsc` invocation can lie, and `npx turbo run build --filter=@activepieces/shared` is the cure. **The web editor and `tsc` are affected too, one hop out.** `packages/web/tsconfig.app.json` maps only `@activepieces/shared`, `pieces-framework` and a few pieces to `src`; `shared/src` then imports `@activepieces/core-utils`, which resolves through `node_modules` to `dist`. So after a sync that changes a `core-utils` schema (a `sessionToken` field on the Bedrock auth, #15747), the editor reports `'sessionToken' does not exist in type` on main's own web code while `vite` is fine. Rebuild the core packages and restart the TS server; the code needs no change. - **The `main` CI job builds web, worker, api and engine before it runs a single test, so a build error there fails the whole job with no test output to point at it.** One missing enum import in a worker file surfaced only as `worker#build` exiting 2. Before pushing a change that moves code between files, run `npx turbo run build --filter=web --filter=worker --filter=api --filter=@activepieces/engine` — the same set the job builds — rather than trusting the per-package typecheck you happened to run before the move. Note that the `web` half of that command needs more heap than node gives it by default: on a 14 GB machine `vite build` transforms all ~7k modules and then dies with `node::OOMErrorHandler` / `SIGABRT` during bundling, which reads as a broken change rather than a full laptop. Run it as `NODE_OPTIONS=--max-old-space-size=8192 npx turbo run build --filter=web` and it finishes in well under a minute. CI runners have the headroom, so this only ever bites locally. - **`check-migrations` runs against your own PGlite database, so a leftover index from work you abandoned fails the gate locally while CI is green.** The check runs migrations then asks TypeORM to generate one, and any difference between your database and the entity metadata counts as drift — including an index a reverted branch created and never dropped. The report names it (`DROP INDEX "public".""` in the generated `up`). CI starts from an empty database, so this class of failure is local-only. The database is `~/.activepieces/pglite` by default, which is also the dev server's, so drop the specific index rather than resetting the directory: a tiny CJS script with `PGlite.create({ dataDir })` and `DROP INDEX IF EXISTS` is enough, and it must run inside `packages/server/api` where the dependency resolves. @@ -84,7 +85,7 @@ Enforcement is the **`Codeowners review` repository ruleset** (active on the def - **Docs videos belong on `cdn.activepieces.com/videos/docs/`, never in the repo — and `docs/resources/` is the legacy path that suggests otherwise.** Every `