[pull] main from activepieces:main

brain/knowledge/flows-execution/action-run.md

@@ -15,7 +15,7 @@ At this stage action runs are **execution only — nothing is persisted**. The c
 - **Priority** `high`, not `critical`, so action runs never outrank the builder interactions a human is actively waiting on.
 
 ### Gotchas
-- **The budget is one end-to-end deadline, not two timeouts with a margin.** `actionRunService` stamps `expiresAt = now + 120s` onto the job and waits 130s. The extra 10s only covers the sandbox kill and the pubsub hop home — it is *not* an allowance for queueing, resolution or provisioning, because those spend the same 120s: `createSandboxRuntime.execute` clamps the run to what is left of `expiresAt` after provisioning **and boot**, and throws `SANDBOX_EXECUTION_TIMEOUT` without starting the engine if nothing is left. **Do not re-shape this as "sandbox 120s + a fudge factor".** That was the original bug: the watcher's clock starts at enqueue and the sandbox's at run-start, so a cold piece install (easily >10s) expired the caller first while the action kept running and writing — and the retry that invites duplicates the write. The worker keeps its own 120s cap for a deadline it cannot trust (missing from an older API, or inflated by clock skew).
+- **The budget is one end-to-end deadline, not two timeouts with a margin.** `actionRunService` stamps `expiresAt = now + AP_FLOW_TIMEOUT_SECONDS` onto the job and waits that plus 10s. The extra 10s only covers the sandbox kill and the pubsub hop home — it is *not* an allowance for queueing, resolution or provisioning, because those spend the same budget: `createSandboxRuntime.execute` clamps the run to what is left of `expiresAt` after provisioning **and boot**, and throws `SANDBOX_EXECUTION_TIMEOUT` without starting the engine if nothing is left. **Do not re-shape this as "sandbox budget + a fudge factor".** That was the original bug: the watcher's clock starts at enqueue and the sandbox's at run-start, so a cold piece install (easily >10s) expired the caller first while the action kept running and writing — and the retry that invites duplicates the write. The worker no longer carries a cap of its own; `execute-action.ts` derives `timeoutInSeconds` from `expiresAt` alone.
 - **Derive the clamp where the timer is armed. Every phase between the two is a hole.** The only thing enforcing the run budget is the SIGKILL timer armed *inside* `sandbox.execute`; `remainingTimeoutInSeconds` was called before `sandbox.start()`, so the engine got its full remaining budget counted from a later instant and user code ran until `expiresAt + bootMs`. Boot is the norm, not the tail — `canReuseSandbox` is false for `SANDBOX_PROCESS` unless `AP_REUSE_SANDBOX=true`, so every production action run pays bind retries, two *unbounded* isolate `execPromise` calls and a 30s connect cap, comfortably past the 10s grace. The pre-boot check survives only as a fast path (skip a pointless spawn); the authoritative clamp is recomputed after the `sandboxStart` timed block. Note the recomputed throw lands inside the `try`, so it invalidates a freshly booted box instead of parking it warm — one cold boot on a rare path, accepted over restructuring boot out of the `try`.
 - **Do not bound provision or the connect wait by the deadline without reworking the error taxonomy first.** It looks like the obvious next step — `spawnWithKill` already takes `timeoutMs` and `waitForConnection`'s 30s is a bare literal — but both raise a plain `Error`, which fails `isSandboxTimeout` in `execute-action.ts` and is rethrown, so the caller gets `INTERNAL_ERROR` / *"the engine crashed while loading or executing the piece"* — the exact misreport the next gotcha forbids, in place of an honest `neverStarted`. Capping the connect wait also does not bound boot: it is the *last* of three boot phases, behind the unbounded isolate calls. Neither buys any safety, because a setup overrun already yields `neverStarted` with no user code executed.
 - A watcher timeout maps to `TIMEOUT`, **never** `INTERNAL_ERROR` — reporting "the engine crashed while loading or executing the piece" when nothing ran sends the calling agent off debugging the piece instead of retrying.
@@ -39,6 +39,12 @@ At this stage action runs are **execution only — nothing is persisted**. The c
 - **The sweep is convergent, not coordinated, because a worker cannot lock.** N sweepers over one shared `./cache` is the steady state (compose `replicas: 5`; Helm's default `rollout` mounts one RWO PVC into every replica), and a worker process has no Redis, so `distributedLock` and system jobs are both unavailable — see [workers § Gotchas](../execution-runtime/workers.md). Safety is structural instead: every step is idempotent (`force: true`, ENOENT-tolerant, mtime re-checked immediately before `rm`) and eviction recomputes its target from the live `readdir` rather than accumulating across deletions, so two simultaneous sweepers pick the same oldest set, delete it once, and neither evicts past the cap. Do not add state that accumulates across deletions — the removed byte accounting was exactly that, and it over-evicted whenever a peer deleted a dir first. Timer jitter is unnecessary for the same reason: concurrent sweeps cost duplicated `stat` calls, nothing more.
 - **Directories from every earlier layout are deliberately left to leak.** Bare-`sha256`, `mcp-flow-version-id` and `ar_`-prefixed dirs only exist on machines that ran intermediate commits of the branch that added them — none of those layouts ever reached `main`, so there is nothing in the field to migrate and no reclaim path was written. The sweeper does *not* reclaim them: the name-sniffing branch that did had no TTL and no mtime re-check, and `mcp-flow-version-id` is still a live constant on `main` (`DEFAULT_MCP_DATA.flowVersionId`), so the day anything provisions code under it that branch would `rm -rf` it every 30 minutes. On a dev box that ran those commits, `rm -rf cache/v12/codes` is the cleanup.
 - **A cache hit must prove the artifact exists — do not "simplify" the `compiledArtifactPresent` check out of `code-builder`.** `cacheState`'s memo is module-scoped with no invalidation API, and a hit returns without touching disk. Delete a step dir and the memo still reports `cacheHit: true`, nothing rebuilds, and the engine `require`s a missing `index.js` — failing that snippet on that worker until the process restarts. Process-local invalidation would not be enough either: the reference `docker-compose.yml` gives `app` and five `worker` replicas the same `./cache` bind mount, so one container's sweeper deletes a directory another has memoised. The one `stat` per hit is the only thing that makes deletion — by the sweeper, by an operator, by a reset volume — recoverable.
+- **The budget is `AP_FLOW_TIMEOUT_SECONDS`, not a knob of its own, and the layers must stay ordered.** An action run is one step of automation, so it gets the budget a flow step gets — `actionRunService` and `runFlowAsTool` both read the same prop. The order that has to hold, outermost first: **worker RPC timeout for `LONG_RUNNING_RPC_METHODS` (budget + `LONG_RUNNING_RPC_MARGIN_MS`) > watcher wait (budget + `WATCHER_GRACE_MS`) > action budget > engine run**. The margin is not decoration — the app spends time *outside* the action on the same call (the `pieceInputFiller` model call, which has no timeout of its own), so a margin-free deadline expires the caller while the action is still legitimately running. Do not give any layer its own env var; move the whole stack with the one knob.
+- **This raised every ceiling; it did not restore an old one.** Issue #15127 reports that 0.86 ran agent tools under `AP_FLOW_TIMEOUT_SECONDS` — the history does not support that. `runPieceTool` and the configured-piece-tool path were *introduced* in #14613, and before it an agent's piece action went over the MCP client into `actionRunService` at the same 120s. 0.88 added a second, lower ceiling (a 60s RPC timeout in the worker) that made the new feature unusable from birth. Worth knowing before anyone repeats "we just restored 0.86" in a changelog.
+- **socket.io's ack timeout is per-call, so a long tool call needs a bigger timeout, not a different transport.** `socket.timeout(ms)` sets `flags.timeout`, `_registerAckCallback` reads it, and `emit` clears `flags` afterwards — so `createRpcClient` taking `RpcTimeout = number | ((method: string) => number)` is the entire fix, worker-side only. A deferred variant (ack immediately, deliver the result later on an `rpc-result` event keyed by a `callId`) was built and reverted: it bought nothing, needed both sides upgraded to work, and introduced a crash — the result promise has no handler attached until *after* the ack resolves, so a disconnect inside the ack window rejects it unhandled and Node's default `--unhandled-rejections=throw` kills the worker (which, in a `WORKER_AND_APP` container, takes the whole instance down via `docker-entrypoint.sh`). Disconnect is already handled for free: `Socket.onclose` calls `_clearAcks`, which rejects every `emitWithAck` ack with `socket has been disconnected`.
+- **Keep the `handler threw` prefix in the RPC error text.** `createConfiguredPieceTools` greps for it to choose between telling the model "that action failed" and "it may already have run, do not call it again" — the second is what stops an agent re-running a side effect it cannot see the result of.
+- **The code cache's active-execution window must cover the run budget, or a long code action deletes its own build.** `ACTION_RUN_CACHE_ACTIVE_WINDOW_MS` (15 min) exempts fresh dirs from eviction, and mtime is stamped at *provision* and never refreshed during the run — so a run longer than the window ages out of its own exemption while executing, and if the tree is over `ACTION_RUN_CACHE_MAX_DIRS` the sweeper removes the directory under it, failing the run on a missing `index.js`. Unreachable while the budget was a hard 120s; reachable the moment the budget followed `AP_FLOW_TIMEOUT_SECONDS`, which operators do set above 15 minutes. `sweep` takes `activeWindowMs`, and the worker passes `max(window, budget)`. Only action runs are exposed — the sweeper's entire scope is `codes/action-runs/`, so flow-version code caches are untouched.
+- **`EXECUTE_ACTION` is exempt from the project concurrency limiter, and raising the budget scaled that.** `RATE_LIMIT_WORKER_JOB_TYPES` is `[EXECUTE_FLOW]` only, and `rate-limiter-interceptor` is gated on `AP_PROJECT_RATE_LIMITER_ENABLED` (default `false`). So an action run holds a `WORKER_JOBS` slot for the full budget with no per-project cap — a slow endpoint driven through MCP `ap_run_action` can occupy every slot on an instance for that long. Adding `EXECUTE_ACTION` to the list is not a one-liner: `shouldContinue` narrows to `ExecuteFlowJobData` and reads `environment`, which `ExecuteActionJobData` does not have.
 - **`EXECUTE_ACTION` is not project-group routable.** `PROJECT_GROUP_ROUTABLE_JOB_TYPES` is `{EXECUTE_FLOW, EXECUTE_WEBHOOK}`, so action runs land on the platform/shared queue even when the project has dedicated workers — unlike the temporary-flow path they replaced, which routed as `EXECUTE_FLOW`. Matters when a project's own workers are the only ones that can reach its network.
 - **`actionRunMode` disables the two flow-only behaviours** in `piece-executor.ts`: the progress reporter becomes a no-op (no flow run to stream to), and waitpoints are rejected by `assertActionRunCannotSuspend` as a plain `Error` (USER-level) so the step ends FAILED, not INTERNAL_ERROR — "this action only works inside a flow" is a usage error, not an engine bug, and must not page oncall.
 - **`createWaitpointHook` must throw *synchronously*, from the returned function's body and not from inside an `async` body. Do not refactor it into a single `async` closure.** This is why the hook is split into a sync wrapper plus `submitWaitpoint`. The deprecated `pause()` shim at `packages/pieces/framework/src/lib/context/versioning.ts` (`buildLegacyPauseHook`, kept until 2026-10-12) does `context.run.createWaitpoint({...}).catch(() => process.exit(1))`. If the rejection arrives as a rejected promise, that `.catch` attaches and **kills the worker process**; thrown synchronously, it propagates as a FAILED step instead. Any piece still on `context.run.pause()` hits this path.

brain/knowledge/flows-execution/flow-runs.md

@@ -42,7 +42,7 @@ CE has full run tracking. Cloud may enforce retention windows; bulk-retry admin
 Entry point: `flowRunService`, defined in `flow-run-service.ts` and wired through `flow-run-module.ts`.
 
 - `packages/server/api/src/app/flows/flow-run/` — controller, service, entity, hooks, side effects, runs queue, AI usage extractor/tracker
-- `packages/server/api/src/app/flows/flow-run/waitpoint/` — resume routes, the `/confirm` page, and its theme hooks
+- `packages/server/api/src/app/waitpoints/` — the waitpoint module: entity, service, resume routes, the `/confirm` page, its theme hooks, and the `RESUME_DELAY_WAITPOINT` handler
 - `packages/core/execution/src/lib/flow-run/` — `FlowRun` type, request dtos, execution types (`StepOutput`, `FlowExecution`), zstd log serializer
 - `packages/server/engine/src/lib/helper/logging-utils.ts` — produces the truncated-input placeholder the web run-details tab detects
 - `packages/server/api/src/app/ee/billing-usage-report/` — daily EE job emitting per-platform run counts to PostHog (`TOTAL_RUNS_PER_DAY`, captured and flushed in platform batches)

docs/install/configure-operate/production-setup.mdx

@@ -68,6 +68,7 @@ Past roughly 80 workers the worker fleet stops being the only dial: throughput k
 | Limit | Default | Env var |
 |---|---|---|
 | Flow run timeout | 600 s | `AP_FLOW_TIMEOUT_SECONDS` |
+| Single action run timeout | 600 s | `AP_FLOW_TIMEOUT_SECONDS` |
 | Sync webhook response | 30 s | `AP_WEBHOOK_TIMEOUT_SECONDS` |
 | Max webhook payload | 25 MB | `AP_MAX_WEBHOOK_PAYLOAD_SIZE_MB` |
 | Step file size | 25 MB | `AP_MAX_FILE_SIZE_MB` |

docs/install/reference/breaking-changes.mdx

@@ -24,6 +24,18 @@ The `SMTP Blacklist Sender` checkbox is replaced by a `Blocked Sender Addresses`
 
 Nothing to configure or migrate, and no existing step stops working. Review any flow whose Create or Update Contact step relied on the old behaviour: a step that was silently wiping attributes will now leave them intact, and a step you were relying on to blacklist contacts must have the checkbox explicitly ticked. To block a sender for a contact, list that sender's address in `Blocked Sender Addresses` — it must be an active sender in your Brevo account, or Brevo answers *"One of the sender is invalid or inactive"*.
 
+#### `AP_FLOW_TIMEOUT_SECONDS` now also bounds a single action run
+
+An action run — an agent's configured piece tool, a chat action, or MCP `ap_run_action` — used to be capped at a hardcoded 120 seconds, and a flow run started as an MCP tool at a hardcoded 300 seconds. Neither could be changed. Any agent tool whose action took longer than a minute failed outright with `RPC [executePieceTool] failed (timeout: 60000ms)`, because a second hardcoded limit in the worker cut it off first.
+
+Both hardcoded limits are gone. All of them now follow `AP_FLOW_TIMEOUT_SECONDS`, the same setting that bounds a flow run, which defaults to 600 seconds. An action run therefore gets the same budget as a flow step.
+
+The limit going up is the change to be aware of: a slow or hung action now occupies a worker slot for up to `AP_FLOW_TIMEOUT_SECONDS` instead of 120 seconds, and action runs are not covered by the per-project concurrency limiter. On a small worker pool, several slow actions can hold execution capacity for longer than they used to.
+
+#### What you need to do
+
+Nothing to configure or migrate, and no new environment variable — a default installation gets the fix with no action. Two things to consider if they apply to you. If you raised `AP_FLOW_TIMEOUT_SECONDS` for long flows, note that agent tools, chat actions and MCP action runs now inherit that same ceiling; lower it if you do not want them held that long. If you put Activepieces behind a reverse proxy and use MCP `ap_run_action`, an action that runs past the proxy's read timeout (nginx defaults to 60 seconds, Cloudflare to 100) returns a gateway error to the MCP client while the action keeps running — raise `proxy_read_timeout` if you rely on long action runs over MCP.
+
 #### A Run Agent step whose turn dies now fails its flow run
 
 A Run Agent step whose turn died — no AI provider configured for the chosen vendor, a revoked API key, a model the provider no longer serves — used to report success. The step returned the failed agent result rather than raising it, so the step read SUCCEEDED, the run completed SUCCEEDED, and later steps acted on a payload that carried the failure inside it. Such a run is now FAILED, and the agent's error message is the step's error.

docs/install/reference/environment-variables.mdx

@@ -148,7 +148,7 @@ run timeouts, and the network egress posture for user code. Read
 | `AP_PREWARM_CACHE_ON_STARTUP` | Pre-fill the worker's local piece and code cache on startup by resolving every enabled flow on the platform, instead of filling it lazily on each flow's first run. Enabling it removes the one-time cold-start latency of the first run after a worker (re)starts, but the warm-up itself costs memory and CPU proportional to the number of enabled flows — on instances with many flows it can pin the worker at its CPU limit for the duration of the warm-up and spike memory enough to OOM-kill small workers, especially when all workers restart at once during an upgrade. Keep it disabled unless your instance has a modest number of enabled flows, your workers have memory headroom, and first-run latency after deploys matters to you. | `false` |
 | `AP_SANDBOX_MEMORY_LIMIT` | Maximum memory (KB) a single sandboxed engine process can use. Each process runs at most one execution at a time. | `1048576` |
 | `AP_SANDBOX_PROPAGATED_ENV_VARS` | Comma-separated environment variables propagated into sandboxed code. For pieces, keep everything in the authentication object so it works across instances. | `None` |
-| `AP_FLOW_TIMEOUT_SECONDS` | Maximum runtime for a single flow run, in seconds. | `600` |
+| `AP_FLOW_TIMEOUT_SECONDS` | Maximum runtime for a single flow run, in seconds. Also bounds a single action run — an agent's piece tool, a chat action, or MCP `ap_run_action` — so raising it lets those hold a worker slot for the same length of time. | `600` |
 | `AP_TRIGGER_TIMEOUT_SECONDS` | Maximum runtime for a trigger's polling, in seconds. | `60` |
 | `AP_DEFAULT_CONCURRENT_JOBS_LIMIT` | Default maximum concurrent runs per project. Can be overridden per project in settings. | `5` |
 | `AP_PROJECT_RATE_LIMITER_ENABLED` | Enforce per-project rate limits to prevent excessive usage. | `false` |

docs/install/reference/limits.mdx

@@ -39,6 +39,7 @@ it.
 | Limit | Cloud | Env var | Self-hosted default |
 |---|---|---|---|
 | Flow run timeout | 10 min | `AP_FLOW_TIMEOUT_SECONDS` | `600` |
+| Single action run timeout | 10 min | `AP_FLOW_TIMEOUT_SECONDS` | `600` |
 | Worker process memory | 1 GB | the worker container's memory cap | `1 GB` |
 | Paused flow lifetime | 30 days | `AP_PAUSED_FLOW_TIMEOUT_DAYS` | `30` |
 | Execution data retention | 30 days | `AP_EXECUTION_DATA_RETENTION_DAYS` | `30` |

packages/core/execution/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@activepieces/core-execution",
-  "version": "0.17.0",
+  "version": "0.18.0",
   "type": "commonjs",
   "main": "./dist/src/index.js",
   "scripts": {