[pull] main from activepieces:main

.agents/skills/debug-failed-run/SKILL.md

@@ -46,6 +46,22 @@ Use the **`Logs`** source (`id: 6a2a91b1d37162f45ad78233`; key columns `Body`, `
 
 Scope the time window to the job's `processedAt`/`finishedAt` from Step 1 (± a few minutes) to keep queries cheap.
 
+**Host correlation** — every log carries `LogAttributes['host']` (format `<host_ip>-<container_hex>`, e.g. `91.98.135.134-e5cbf708af8d`), stamped by the global evlog host enricher. When a failure looks resource-shaped (OOM, RPC timeout, sandbox crash, silent hang), correlate the failing job's host with its `system.snapshot` metrics at that time:
+
+```sql
+-- Given the failed job's host + finish time, what did that container's memory + event loop look like?
+SELECT Timestamp, LogAttributes['memRssMb'] AS rss_mb,
+       LogAttributes['memHeapUsedMb'] AS heap_mb,
+       LogAttributes['eventLoopDelayP99Ms'] AS lag_ms
+FROM default.otel_logs
+WHERE LogAttributes['event'] = 'system.snapshot'
+  AND LogAttributes['host'] = '<host_from_failed_job>'
+  AND Timestamp BETWEEN <finishedAt> - INTERVAL 15 MINUTE AND <finishedAt> + INTERVAL 1 MINUTE
+ORDER BY Timestamp
+```
+
+RSS climbing toward the container's cgroup ceiling (~900 MB on workers, ~2 GB on app) or `lag_ms` spiking above ~1 s just before the failure = infra cause, not the code path. The reverse — pull failed jobs on a host whose RSS crossed a threshold — surfaces a leaking container that is dropping a small percentage of runs silently.
+
 > **Those two names exist only in the script's report** (`debug-failed-job.js` formats them from the raw job). If you bypass the script and read jobs straight off the queue with `bullmq` — `q.getJobs(['failed'], …)` — the properties are **`job.processedOn` / `job.finishedOn`**. `job.processedAt` is silently `undefined`, not an error, so a hand-rolled scan reports every failure as having no timestamp and you fall back to `job.timestamp` (enqueue time) without noticing. That shifts failures earlier by however long the queue backlog is (8+ minutes on cloud) and will make a fix look like it did not take effect. Bucket by `finishedOn` when you want "is this still happening". You're looking for the engine/worker log lines that bracket the failure — sandbox crashes, OOM ("no space"/heap), RPC timeouts, connection refresh failures.
 
 ## Step 3 — Trace the failure into this repo

CLAUDE.md

@@ -100,9 +100,14 @@ When running in `--mode=cloud`, do not use OAuth2 connections — the OAuth prov
 - If the PR includes any contributions to pieces (integrations under `packages/pieces`), also add the appropriate pieces label (in addition to the primary label above):
   - **`🧩 area/third-party-pieces`** — for third-party integrations (most pieces under `packages/pieces/community/`)
   - **`🧩 area/core-pieces`** — for core pieces (under `packages/pieces/core/`)
-- **Always fill the "Breaking change?" section of the PR template** — tick exactly one box (the `breaking-change-check` CI job fails if it is left unedited). A change is breaking if a self-hoster or API consumer must take action: removed/renamed API fields or endpoints, dropped columns, new required fields, removed/required env vars, or default/limit/behaviour changes. If it is breaking:
-  - also apply the **`⛓️‍💥 breaking-change`** label (in addition to the primary label above), and
-  - add an entry to `docs/install/reference/breaking-changes.mdx` describing what changed and the action required. CI enforces that the label and the docs entry travel together.
+- **The PR template has TWO required sections and `breaking-change-check` fails if either is left unedited** — `### Breaking change?` *and* `### Security impact?`. Tick exactly one box in each (ticking two in one section also fails). Copy the option text from `.github/pull_request_template.md`; the parser matches `- [x] no` / `- [x] yes` under each heading, so keep the `no`/`yes` as the first word after the checkbox.
+  - `Breaking change?` has **three** options: `no`, `yes — technical` (removed/renamed API field or endpoint, dropped column, new required field, removed/required env var), `yes — functional` (default/limit/behaviour change, new self-hosted setup step).
+  - `Security impact?` has **two**: `no`, `yes — security-sensitive` (touches authentication, secrets, permissions/roles, cryptography, SSRF / outbound HTTP, or file handling). This one is a **declaration only** — no label, no docs entry, nothing else is coupled to it.
+- **If the change is breaking, the label and the docs entry must travel together** — the check's coupling is bidirectional and it also cross-checks the template answer, so label, docs entry and ticked box must all agree:
+  - apply the **`⛓️‍💥 breaking-change`** label (in addition to the primary label above), **and**
+  - add an entry to `docs/install/reference/breaking-changes.mdx`. It must have a `####` title **and** at least one body line — a bare heading, a `---`, or loose prose does not count as an entry.
+  - Ticking `no` while carrying the label fails, and so does adding a docs entry without the label.
+- **Never `gh run rerun` a `pull_request`-triggered check to pick up a PR description edit.** A rerun replays the *original* event payload, so it re-reads the stale body and fails again — and because it is a later attempt on an older run, its failure shadows a newer run that already passed. Edit the body (which fires `edited`, a trigger these workflows listen for) and let the fresh run supersede. To check the answer locally without any run at all: `PR_BODY="$(gh pr view <n> --json body --jq .body)" PR_LABELS="$(gh pr view <n> --json labels --jq '[.labels[].name]')" BASE_REF=main npx tsx tools/scripts/breaking-change-check.ts`
 - **Non-rollbackable migrations are a separate axis** from customer-facing breaking changes: a migration that runs destructive DDL (`DROP TABLE`/`DROP COLUMN`, `ADD ... NOT NULL` without `DEFAULT`, etc.) must set `breaking = true` on the migration class — this is the rollback-safety flag (used by `rollback-migrations.ts` and the release rollback note), enforced by `check-migration-rollback.ts`. It does **not** by itself require the `⛓️‍💥 breaking-change` label; decide that from the upgrade-impact question above.
 
 ## Database Migrations

brain/knowledge/data-storage-observability/evlog-metrics.md

@@ -0,0 +1,73 @@
+---
+icon: 📈
+---
+
+# Logging & Metrics (evlog)
+
+All structured logging goes through **evlog**. One wide event per unit of work (HTTP request, worker job, flow run) with all context as fields, shipped to a single remote drain. Metrics are derived from log fields on the backend — we do not run a separate metrics pipeline for application signals, with one exception (BullMQ queue depth as native OTLP gauges).
+
+### Setup
+
+- `evlogSetup.init` (`packages/server/utils/src/evlog-setup.ts`) wires `initLogger` with service/version/env, level + sampling, a redact list (secrets, tokens, `authorization`, `connection.value`, axios internals), and a drain wrapper (`wrapDrainWithHost`) that stamps `os.hostname()` on every event whose `host` field is undefined. Uses the drain, not `enricherPlugin`, because `enricherPlugin.enrich` hooks only run on the HTTP request-finish path; plain `log.info/warn/error` and `createLogger().emit()` go through `emitWideEvent`, which invokes drains but not enrichers, and workers don't register `evlogFastify` at all — a `host` enricher would be a worker-side no-op. Events that already carry a host (e.g. `system.snapshot`, which sets it at the call site) pass through unchanged.
+- Sampling: info sampled per `AP_SAMPLE_RATE_INFO`; warn/error always kept; extra keep rules for `status ≥ 400` and `duration > 2000ms`.
+- Field-naming schema is a hard contract — see `packages/server/CLAUDE.md` "Structured Logging Field Schema". One concept = one dotted path (`flowRun.id`, `piece.name`, `job.type`), durations end in `Ms`, bytes in `Bytes`, reserved keys are auto-populated.
+
+### Drains
+
+`packages/server/utils/src/evlog-drains.ts` picks one remote destination by env, first match wins: **Axiom → HyperDX → Loki → Better Stack → generic OTLP**. All batch through a shared pipeline (100 / 5s, retry ×3, buffer 5000). In dev, an additional NDJSON file drain writes to `.evlog/logs/` so `/analyze-logs` and jq work locally.
+
+### Log-based metrics
+
+The drain destination (HyperDX/ClickStack, Loki, Better Stack) derives metrics from log fields at query time — that is the whole reason the field schema is strict. Typical shapes:
+
+- `count() where error.code = 'X'` → error rate
+- `p95(durationMs) group by path` → latency SLO
+- `count() group by piece.name, flowRun.status` → per-piece failure rate
+
+### System snapshot (process metrics on ClickStack)
+
+Process-level CPU / memory / event-loop lag reach ClickStack **as evlog events**, not native metrics. Both API (`packages/server/api/src/app/helper/system-snapshot.ts`) and worker (`packages/server/worker/src/lib/utils/system-snapshot.ts`) run a `setInterval(..., 60_000)` that emits one wide event with:
+
+- `event: 'system.snapshot'`, `host`
+- `memRssMb`, `memHeapUsedMb`, `memHeapTotalMb`
+- `eventLoopDelayP99Ms` (from `perf_hooks.monitorEventLoopDelay`)
+- `queueCounts` (BullMQ, API only)
+
+Emitted with `_forceKeep: true` so info-sampling cannot drop it. Dashboards and alerts for RSS / heap / event-loop lag are built on this event.
+
+### BullMQ queue depth (real OTLP metrics)
+
+The only signal that leaves as a **native metric**, not a log. Same 60s tick calls `otelQueueMetrics.push` (`packages/server/api/src/app/helper/otel-queue-metrics.ts`) which POSTs an OTLP gauge `bullmq.job.count` with `queue` + `state` attributes to `OTEL_EXPORTER_OTLP_ENDPOINT/v1/metrics`. Gated by `AP_OTEL_QUEUE_METRICS_ENABLED`. Queue-depth cardinality (queues × states) would bloat wide events, and gauges are the natural shape.
+
+### Worker → App healthcheck (in-band, not evlog)
+
+CPU / RAM / disk / per-executor RSS travel from worker to app **piggyback on every `apiClient.poll(machineInfo)` call** — see `buildMachineInfo` in `packages/server/worker/src/lib/worker.ts` (uses `systemUsage` from `packages/server/utils/src/system-usage.ts`). App caches it in `app-machine-cache.ts` and surfaces the fleet at `GET /v1/health/system`. This powers the Platform health page; it does **not** go to ClickStack.
+
+### What lives where
+
+| Signal | Source | Destination |
+| --- | --- | --- |
+| Latency / error / business KPIs | evlog wide events per request/job | ClickStack via drain — log-based metric |
+| Process RSS / heap / event-loop lag | `system.snapshot` every 60s | ClickStack via drain — log-based metric |
+| BullMQ queue depth by state | `otelQueueMetrics.push` every 60s | OTLP `/v1/metrics` — native gauge |
+| Worker CPU / RAM / disk / executors | `apiClient.poll(machineInfo)` | App cache → `GET /v1/health/system` |
+| Managed Postgres / Redis CPU / mem | DigitalOcean managed metrics | Betterstack (via `do-alerts@` group) |
+| Hetzner host CPU / mem | host agent, if any | not in ClickStack |
+
+### Gotchas
+
+- **Missing `_forceKeep` drops your metric.** A recurring info-level snapshot without `_forceKeep: true` is subject to sampling — the value silently thins out at whatever `sampleRateInfo` is set to, and charts read as gaps or an inflated mean. Every periodic-metric event must set it.
+- **Field-name drift breaks every log-based metric.** The same id logged as `runId` in one place and `flowRun.id` in another cannot be joined at query time. The schema in `packages/server/CLAUDE.md` is enforced by convention only — the drain will not warn.
+- **`hyperdxToken` wins over `otlpEnabled`.** `resolveRemote` picks the first matching drain (Axiom → HyperDX → Loki → Better Stack → OTLP), so setting an Axiom or HyperDX token silently disables `AP_OTEL_ENABLED`. To send logs to a generic OTLP collector, unset the other provider tokens.
+- **The Betterstack drain requires both `betterstackToken` and `betterstackHost`.** Setting only one silently falls through to the next drain.
+- **Worker machine info is in-band with polling, not scheduled.** If a worker stops polling (crash, network partition), its numbers on the health page freeze — there is no separate heartbeat.
+
+### Key files
+
+- `packages/server/utils/src/evlog-setup.ts` — `evlogSetup.init` / `.flush`
+- `packages/server/utils/src/evlog-drains.ts` — drain selection
+- `packages/server/utils/src/system-usage.ts` — cgroup CPU / RAM readers
+- `packages/server/api/src/app/helper/system-snapshot.ts` — API 60s tick
+- `packages/server/worker/src/lib/utils/system-snapshot.ts` — worker 60s tick
+- `packages/server/api/src/app/helper/otel-queue-metrics.ts` — OTLP gauge push
+- `packages/server/api/src/app/helper/app-machine-cache.ts` — fleet cache for `/v1/health/system`

brain/knowledge/data-storage-observability/index.md

@@ -34,6 +34,10 @@ Security-relevant actions persisted to `audit_event`, queryable by platform admi
 
 Platform reporting: daily runs, active flows/users, time-saved estimates. `PlatformAnalyticsReport` cached (5-min TTL) refreshed under a distributed lock; separate daily cron (12:00 UTC) tallies per-piece usage into `pieceMetadata.usage`. minutesSaved = runs × flow.timeSavedPerRun. Powers `/impact` (Summary/Trends/Details). Gated by `analyticsEnabled` — NOT in CE. Frontend queries carry `enabled: platform.plan.analyticsEnabled`.
 
+### Logging & Metrics (evlog)
+
+All structured logging goes through **evlog** — one wide event per unit of work, one remote drain (Axiom / HyperDX / Loki / Better Stack / OTLP; first match wins). Metrics on ClickStack are **log-based**: dashboards and alerts query numeric fields (`durationMs`, `memRssMb`, `eventLoopDelayP99Ms`) on wide events. Both API and worker emit a 60s `system.snapshot` event with process RSS / heap / event-loop lag so those charts exist at all. BullMQ queue depth is the one signal exported as a **native OTLP gauge** (`bullmq.job.count`), because per-queue-per-state cardinality would bloat wide events. Worker CPU / RAM travels in-band on the poll healthcheck (not to ClickStack) and surfaces on `GET /v1/health/system`.
+
 ### Product Telemetry
 
 Anonymous product analytics to PostHog, from both the browser and the app container. Gated per platform by `platform_configuration.isProductTelemetryEnabled`, edited at Platform Admin > Infrastructure > Configurations; `AP_TELEMETRY_ENABLED` survives only as the value a platform's row is *born* with. See [000033](../decisions/000033-platform-configuration-rows-are-authoritative-and-created-on-first-read.md).
@@ -81,3 +85,4 @@ Streams platform/project events to webhook URLs in real time — internal AP flo
 - **Knowledge Base** — documents chunked into vector embeddings for AI search
 - **Analytics** — usage reporting
 - **Audit Logs** — the persisted security-action record
+- **Logging & Metrics (evlog)** — wide events, drains, log-based metrics, `system.snapshot`, OTLP queue gauge