fix: post-11.07 audit — 8 findings (webhook keyset cursor, helm PG egress/PDB/selector, control-plane clock & retry, E4 settle window)

.github/dependabot.yml

@@ -46,6 +46,16 @@ updates:
       # have explicit floors in pyproject.toml driven by CVE fixes
       # (see commit e8b1237 and the A06 dependency-profiles block).
       # Allow minor + patch within the >= floor, never major.
+      #
+      # NOTE (audit 2026-07-17 #8): a Dependabot `ignore` applies to BOTH
+      # version updates AND security updates — there is no per-ignore
+      # `applies-to` (that key exists only on `groups`). So a CVE fixed ONLY in
+      # a major bump of these deps gets no automatic security-update PR. This is
+      # an ACCEPTED tradeoff: the `safety` + `pip-audit` jobs in security.yml
+      # scan these deps and FAIL on such a CVE, surfacing it for a deliberate
+      # manual major bump (the whole reason their majors are gated). Do NOT
+      # "fix" by dropping the ignore — that re-introduces the unwanted
+      # major-version PR churn these floors exist to prevent.
       - dependency-name: "langchain-core"
         update-types: ["version-update:semver-major"]
       - dependency-name: "langchain-text-splitters"

CHANGELOG.md

@@ -4,6 +4,54 @@ All notable changes to AgentFlow are documented in this file.
 
 ## [Unreleased]
 
+### Fixes — post-11.07 adversarial audit (2 latent P1, 4 P2, 1 low)
+
+Four-surface adversarial review of everything merged since the 11.07 audit
+(control-plane, serving/webhook, helm/k8s, CI). All findings fixed and locally
+verified (ruff/lint/targeted unit tests + helm render); no live-cluster run.
+Full report: `audit_2026-07-17_post11.md`.
+
+- **Webhook journal scan — composite keyset cursor (P1, was a silent wedge).**
+  The scan cursor is now `(processed_at, event_id)` and fetches strictly after
+  it, so a single second holding ≥ `scan_batch_size` rows can no longer re-pin a
+  second-granular cursor and silently drop every later webhook. Portable
+  OR-decomposition predicate (row-value tuples don't transpile to ClickHouse); a
+  regression test seeds a larger-than-one-batch second and is red on the old
+  code, green on the fix. Follow-up (adversarial review): a strict keyset alone
+  would silently drop same-second rows that become visible after the frontier
+  passed their second (UUID event ids are not monotonic; ClickHouse
+  `processed_at` is second-granular), so every dispatcher fetch is bounded by a
+  DB-clock **settle watermark** — `processed_at <= now() - INTERVAL
+  AGENTFLOW_WEBHOOK_SETTLE_SECONDS` (default 3 s) — and the frontier crosses
+  only closed seconds. The settle value must exceed writer stamp-to-visibility
+  lag plus writer↔DB clock skew.
+- **NetworkPolicy — PostgreSQL egress (P1, was a hardened-prod outage).** A
+  `:5432` egress rule now renders when `controlPlane.store=postgres`; without it,
+  `networkPolicy.enabled=true` on the postgres profile denied every pod's PG
+  connect and the control-plane pool never opened.
+- **Control-plane retry backoff — DB clock (P2).** `next_attempt_at` is computed
+  in SQL (`now() + make_interval`) instead of the app pod's clock, mirroring the
+  lease columns, so NTP skew no longer collapses or over-delays the
+  webhook/outbox backoff.
+- **Replica-correctness verify — settle window (P2).** E4 Checks 3/4 dwell a
+  settle window and assert the count never exceeds one, instead of asserting on
+  the first sighting — which passed a delayed split delivery/page undetected.
+- **PodDisruptionBudget — api-scoped (P2).** The PDB selects `component: api`, so
+  a drain cannot evict both API pods while a lone worker satisfies the budget.
+- **API Deployment selector — stable component label (P2).**
+  `app.kubernetes.io/component: api` is now unconditional, so the immutable
+  `spec.selector` no longer changes when `worker.enabled` is toggled. One-time
+  recreate for pre-split installs is documented in
+  `docs/migration/helm-component-selector.md` (no pre-upgrade hook — it would
+  fire on every upgrade).
+- **Webhook outcome write — transient retry (P2).**
+  `record_webhook_delivery_outcome` retries transient DB errors like
+  `record_api_usage`, so a blip after a successful POST no longer strands the
+  row+lease for the full claim window.
+- **Dependabot `ignore` — documented security-update scope (low).** Commented
+  that an `ignore` also suppresses security-update PRs (there is no per-ignore
+  `applies-to`); an accepted tradeoff, still caught by the safety/pip-audit jobs.
+
 ### Perf — paced 1 h @ 100 eps produce (apply 99.5)
 
 - Mac compose (1 Flink TM), fresh volumes: 360 000 events at `--pace-eps 100`

docs/migration/helm-component-selector.md

@@ -0,0 +1,86 @@
+# Helm migration: API Deployment selector now carries `component: api` unconditionally
+
+## Applies to
+
+Operators upgrading an **existing** AgentFlow Helm release that was installed
+with a chart **before** this one, where `worker.enabled=false` (the default).
+Fresh installs and releases already running `worker.enabled=true` are **not**
+affected.
+
+## What changed and why
+
+The API Deployment (`templates/deployment.yaml`) now adds
+`app.kubernetes.io/component: api` to its pod template labels **and** its
+`spec.selector.matchLabels` **unconditionally**. Previously that label was added
+only when `worker.enabled=true`.
+
+This fixes two audit findings:
+
+- **#7 — immutable selector toggle-break.** `spec.selector` is immutable in
+  Kubernetes. When the label was gated on `worker.enabled`, flipping that value
+  on a live release changed the selector, so `helm upgrade` aborted with
+  `spec.selector is immutable`. Making the label unconditional keeps the selector
+  **constant** across the `worker.enabled` toggle, so the split can be enabled or
+  disabled in place.
+- **#6 — component-scoped PodDisruptionBudget.** The PDB now selects
+  `component: api`, guaranteeing at least one *API* pod survives a voluntary
+  disruption. That guarantee only holds if API pods **always** carry the label,
+  which is why the pod-template label had to become unconditional too.
+
+It also keeps API and worker pods in disjoint selector sets, so the two
+Deployments can never adopt each other's pods.
+
+## The one-time breaking upgrade
+
+Because `spec.selector` is immutable, the **first** upgrade of a pre-existing
+`worker.enabled=false` release onto this chart changes the selector from
+`{name, instance}` to `{name, instance, component=api}`. The API server rejects
+the in-place change:
+
+```
+Deployment.apps "<release>-agentflow" is invalid: spec.selector: Invalid value:
+v1.LabelSelector{...}: field is immutable
+```
+
+`helm upgrade` fails and rolls back; nothing is left half-applied. Recreate the
+API Deployment **once** to move past it. This is a one-time step — subsequent
+upgrades are normal.
+
+### Option A — zero-downtime (recommended for multi-replica / production)
+
+Orphan the existing pods so they keep serving, let the recreated Deployment
+stand up new pods, then reap the orphans:
+
+```bash
+kubectl delete deployment <release>-agentflow -n <namespace> --cascade=orphan
+helm upgrade <release> <chart> [same flags as before]
+# wait for the new (component=api) pods to become Ready, then remove the
+# orphaned old pods — they lack component=api so the new Deployment won't adopt
+# them and they will not be cleaned up automatically:
+kubectl get pods -n <namespace> -l app.kubernetes.io/name=agentflow \
+  -L app.kubernetes.io/component
+kubectl delete pod <old-pod-names> -n <namespace>
+```
+
+### Option B — simplest (brief serving gap)
+
+For a single-replica default install where a short gap is acceptable:
+
+```bash
+kubectl delete deployment <release>-agentflow -n <namespace>
+helm upgrade <release> <chart> [same flags as before]
+```
+
+## Notes
+
+- The chart does **not** ship a `pre-upgrade` hook to automate the delete. A hook
+  that deletes the primary API Deployment would fire on **every** upgrade (helm
+  hooks are not conditional on "did the selector change"), turning a one-time
+  migration into a recurring outage. The manual step above runs exactly once, by
+  operator decision.
+- The Service selector is unaffected (Service selectors are mutable): it already
+  narrows to `component: api` when `worker.enabled=true` and otherwise selects on
+  `{name, instance}`, which still matches the API pods.
+- Related NetworkPolicy change in the same release: a PostgreSQL `:5432` egress
+  rule is now rendered when `controlPlane.store=postgres`. No migration action —
+  it only adds an allowed egress.

docs/serving-bridge.md

@@ -177,16 +177,35 @@ scans and pushes kept one entry per event forever. All journal consumers are
 now bounded:
 
 - **Webhook dispatcher** — incremental cursor scan: each pass fetches at most
-  `scan_batch_size` (1000) rows at/after the `processed_at` high-water mark
-  (`min_processed_at`, inclusive — the journal is second-granular on
-  ClickHouse, and the seen-set dedups the re-fetched cursor second). The
-  cursor advances over the contiguous prefix of rows that end the pass seen
-  and **freezes at the first row whose durable enqueue failed**, so that row
-  is re-fetched and retried next pass — the retry-forever semantics the full
-  scan provided. An all-seen full batch still advances the cursor, so a seen
-  frontier wider than one batch cannot pin the window. Startup
-  (`mark_existing_events_seen`) seeds the cursor from the newest batch instead
-  of enumerating the journal.
+  `scan_batch_size` (1000) rows **strictly after** a composite
+  `(processed_at, event_id)` keyset cursor, in `processed_at, event_id ASC`
+  order. The cursor advances over the contiguous prefix of rows that end the
+  pass seen and **freezes at the first row whose durable enqueue failed**, so
+  that row is re-fetched and retried next pass — the retry-forever semantics the
+  full scan provided. The keyset (not a bare second-granular high-water mark) is
+  what lets the frontier advance **within** a single second: an inclusive
+  `processed_at >=` cursor floored to the second was re-pinned forever by any
+  second holding ≥ `scan_batch_size` rows, silently dropping every later webhook
+  for every tenant (audit 2026-07-17 #1). The predicate is the portable
+  OR-decomposition `t > ts OR (t = ts AND event_id > id)` — a row-value tuple
+  does not transpile to ClickHouse. A strict keyset is only lossless over
+  seconds no writer will stamp again — event ids are UUIDs (not monotonic), so
+  a frontier inside the still-open second would permanently exclude any
+  same-second row that becomes visible later with a lower id. Every dispatcher
+  fetch is therefore bounded by a DB-clock **settle watermark**
+  (`processed_at <= now() - INTERVAL N SECOND`,
+  `AGENTFLOW_WEBHOOK_SETTLE_SECONDS`, default 3 s; `0` disables the bound
+  entirely — tests only; must exceed writer stamp-to-visibility lag +
+  writer↔DB clock skew, and a violation is silent (a late-visible row behind
+  the frontier is simply never delivered), so treat the invariant as an
+  operating requirement, not a tunable. On a non-UTC DuckDB host the
+  session-local frame is non-monotonic across the autumn DST fold: delivery
+  of rows stamped inside the fold window is delayed (never dropped) by up to
+  the fold width. Worst-case added delivery latency = settle. Startup (`mark_existing_events_seen`) seeds the cursor
+  from the newest **settled** batch instead of enumerating the journal —
+  unsettled rows deliver once settled (a restart race is not lost; the durable
+  enqueue's idempotent key suppresses re-POSTs); the seen-set is now a
+  secondary safety net, with the keyset as the primary dedup.
 - **Seen-sets** — `BoundedSeenSet` (`src/serving/seen_events.py`): capped,
   FIFO-with-refresh eviction. Eviction is safe because webhook enqueue is
   idempotent on its primary key (inline delivery fires only for freshly

helm/agentflow/templates/NOTES.txt

@@ -0,0 +1,45 @@
+AgentFlow {{ .Chart.AppVersion }} has been deployed as release "{{ .Release.Name }}".
+
+Service (in-cluster):
+  {{ include "agentflow.fullname" . }}.{{ .Release.Namespace }}.svc:{{ .Values.service.port }}
+
+Control-plane store: {{ .Values.controlPlane.store }}
+{{- if .Values.worker.enabled }}
+Process-role split: ENABLED (API Deployment role=api, worker Deployment role=worker).
+{{- end }}
+{{- if .Values.networkPolicy.enabled }}
+NetworkPolicy: ENABLED (default-deny + explicit egress{{ if eq .Values.controlPlane.store "postgres" }}, including PostgreSQL:{{ .Values.networkPolicy.egressPorts.postgres }}{{ end }}).
+{{- end }}
+
+{{- if .Release.IsUpgrade }}
+
+>>> UPGRADE NOTE — one-time API Deployment recreate may be required <<<
+
+This release makes the API Deployment's spec.selector carry
+`app.kubernetes.io/component: api` UNCONDITIONALLY. spec.selector is IMMUTABLE.
+If the release you just upgraded FROM was installed with an earlier chart where
+`worker.enabled=false` (selector was name+instance only), the API server will
+reject this upgrade with:
+
+    Deployment.apps "{{ include "agentflow.fullname" . }}" is invalid:
+    spec.selector: Invalid value: ... field is immutable
+
+If that happened, the upgrade did NOT apply. Recreate the API Deployment once —
+pick ONE:
+
+  # A) zero-downtime: orphan the old pods, let the new Deployment take over,
+  #    then reap the orphans after the new pods are Ready.
+  kubectl delete deployment {{ include "agentflow.fullname" . }} \
+    -n {{ .Release.Namespace }} --cascade=orphan
+  helm upgrade {{ .Release.Name }} <chart> [same flags]
+  # ...verify new pods Ready, then delete leftover old pods by hand.
+
+  # B) simplest (brief serving gap on a single-replica install):
+  kubectl delete deployment {{ include "agentflow.fullname" . }} \
+    -n {{ .Release.Namespace }}
+  helm upgrade {{ .Release.Name }} <chart> [same flags]
+
+Releases already running the split (worker.enabled=true) are unaffected — their
+selector already carried component=api. Fresh installs are unaffected.
+See docs/migration/helm-component-selector.md for detail.
+{{- end }}

helm/agentflow/templates/deployment.yaml

@@ -15,9 +15,12 @@ metadata:
   name: {{ include "agentflow.fullname" . }}
   labels:
     {{- include "agentflow.labels" . | nindent 4 }}
-    {{- if .Values.worker.enabled }}
+    # component=api is UNCONDITIONAL (not gated on worker.enabled): it keeps this
+    # Deployment's immutable spec.selector stable across a worker.enabled toggle
+    # (helm upgrade would otherwise fail "spec.selector is immutable"), keeps API
+    # vs worker pods disjoint so the two Deployments never adopt each other's
+    # pods, and lets the api-scoped PodDisruptionBudget always match these pods.
     app.kubernetes.io/component: api
-    {{- end }}
 spec:
   replicas: {{ .Values.replicaCount }}
   strategy:
@@ -28,16 +31,12 @@ spec:
   selector:
     matchLabels:
       {{- include "agentflow.selectorLabels" . | nindent 6 }}
-      {{- if .Values.worker.enabled }}
       app.kubernetes.io/component: api
-      {{- end }}
   template:
     metadata:
       labels:
         {{- include "agentflow.selectorLabels" . | nindent 8 }}
-        {{- if .Values.worker.enabled }}
         app.kubernetes.io/component: api
-        {{- end }}
         {{- with .Values.podLabels }}
         {{- toYaml . | nindent 8 }}
         {{- end }}

helm/agentflow/templates/networkpolicy.yaml

@@ -50,4 +50,12 @@ spec:
     - ports:
         - protocol: TCP
           port: {{ .Values.networkPolicy.egressPorts.otlp }}
+    {{- if eq .Values.controlPlane.store "postgres" }}
+    # PostgreSQL control-plane store (when controlPlane.store=postgres): the
+    # shared webhook/alert/outbox/usage pool. Without this rule every API and
+    # worker pod's connect to PG:5432 is denied and the pool never opens.
+    - ports:
+        - protocol: TCP
+          port: {{ .Values.networkPolicy.egressPorts.postgres }}
+    {{- end }}
 {{- end }}

helm/agentflow/templates/poddisruptionbudget.yaml

@@ -10,4 +10,13 @@ spec:
   selector:
     matchLabels:
       {{- include "agentflow.selectorLabels" . | nindent 6 }}
+      # Scope to the API role only. selectorLabels alone (name+instance) also
+      # match the worker pods, so with worker.enabled a single minAvailable:1
+      # budget spanning {api ∪ worker} lets a drain evict BOTH API pods while the
+      # lone worker satisfies it → serving surface hits zero. component=api keeps
+      # this budget a guarantee about request-serving pods. The worker is a
+      # background singleton (delivery loops claim rows in PG and resume after a
+      # reschedule); a minAvailable:1 PDB on a single-replica worker would instead
+      # make its node undrainable, so workers deliberately get no PDB.
+      app.kubernetes.io/component: api
 {{- end }}

helm/agentflow/values.yaml

@@ -101,8 +101,10 @@ podDisruptionBudget:
 
 # NetworkPolicy enforces a default-deny baseline plus explicit ingress on the
 # HTTP port and egress to dependencies (DNS, Kubernetes API, Redis, Kafka,
-# ClickHouse, OTLP collector). Disabled by default so existing clusters
-# without a NetworkPolicy controller are not broken; turn on per environment.
+# ClickHouse, OTLP collector, and — on the postgres control-plane profile —
+# PostgreSQL). Disabled by default so existing clusters without a NetworkPolicy
+# controller are not broken; turn on per environment. The postgres:5432 egress
+# rule is rendered only when controlPlane.store=postgres (harmless otherwise).
 networkPolicy:
   enabled: false
   ingressFromNamespaces:
@@ -112,6 +114,7 @@ networkPolicy:
     kafka: 9092
     clickhouse: 8123
     otlp: 4317
+    postgres: 5432
 
 # Serving engine (ADR 0006 / 0007 / 0009). The shipped demo runs ClickHouse;
 # the chart's default stays the safe single-node DuckDB profile because the