Audit hardening: tenant isolation, TLS-enforcing production profile, PG pooling & migrations, hash-pinned supply chain, real SLIs#189
Merged
Conversation
SearchIndex.search() returns mappings, but the router post-filtered them
with getattr(result, "entity_type", None) — always None for a dict, so the
`or` short-circuited and every forbidden row passed. A key scoped to
`order` got user/product/session ids and snippets from /v1/search whenever
it sent no explicit entity_types filter; the direct entity endpoints kept
answering 403, so only search leaked.
The key allowlist is now passed into the index and applied before scoring,
so forbidden documents never enter the candidate set. That also fixes a
second defect: the old post-filter ran after [:limit], so forbidden rows
consumed result slots and could crowd out every row the key was allowed to
see.
Policy, now pinned by tests: entity and catalog_field documents follow the
allowlist; metric documents stay visible to scoped keys (/v1/metrics/* is
not entity-scoped); an empty allowlist returns no entity-scoped document.
search() returns list[SearchHit] (TypedDict) so mypy rejects the attribute
access that silently disabled the filter.
Verified: the new regression tests fail on the pre-fix code with
`assert {'product', 'session', 'user'} == set()`.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…fect (P0-2) QueryEngine.__init__ ran the serving DDL and seeded demo rows on every boot — against the embedded store AND whatever external backend was configured, and before anything read AGENTFLOW_DEMO_MODE, which is why that flag appeared to do nothing. So the serving identity needed CREATE/ALTER/INSERT on the production store, several booting replicas could each find an empty table and seed it, and a fresh production ClickHouse got demo orders for no better reason than being empty. The constructor now lays down only the embedded schema — that store is created in-process and has no other provisioner — and sends nothing at all to an external backend. A test fails if a single statement reaches ClickHouse on boot, so the API image can run against a read-only ClickHouse user. Seeding is opt-in through AGENTFLOW_SEED_ON_BOOT (default off), separate from AGENTFLOW_DEMO_MODE, which keeps its own meaning (public demo key + read-only guard). The suite turns it on in conftest, because its fixtures assert on the canonical demo entities. Provisioning moves out of band: * new `python -m src.serving.provision [--schema] [--seed]` — idempotent, and it provisions every store the API reads (on the ClickHouse profile the embedded DuckDB still carries control-plane state, so both get their schema); * `make demo` runs it with --seed; * docker-compose.prod.yml gains a `serving-init` one-shot service the API waits on; * Helm gains a pre-install/pre-upgrade migration Job, so a failed migration fails the release instead of letting pods come up against a store they would have silently created. serving.clickhouse.migrationUser/migrationPasswordKey let the DDL identity differ from the serving one. ClickHouseSink still ensures the schema (the writer holds the grants and cannot run without the tables) but no longer decides an empty store deserves demo rows. ServingBackend splits initialize_demo_data() into ensure_schema() + seed_demo_data(); the old method stays as the two composed, for the writer and the CLI. Verified: 2136 passed, 56 skipped (4 errors are Docker-only tests on a host without Docker) · mypy 131 files clean · ruff clean · helm lint + template. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… (P0-3) /v1/lineage, /v1/slo, /v1/search and the health collector read the embedded DuckDB directly, whatever SERVING_BACKEND said. On the ClickHouse profile the API therefore split in half: entity and metric answered from ClickHouse, while lineage reconstructed provenance, SLO computed an error budget, and health reported freshness — all from a DuckDB holding nothing but demo rows. Lineage even labelled the enrichment layer system="duckdb" on a ClickHouse deployment. A plausible wrong answer is worse than an outage. New semantic_layer/journal.py (JournalReader): every pipeline_events read goes through ServingBackend. QueryEngine.backend / .journal are the front doors, and a static ratchet test (AST, not grep) now fails on any private reach from a read surface. A behavioural test injects a backend holding rows that exist nowhere in DuckDB and asserts every surface returns those rows. The health collector never checked the serving store at all — it checked Kafka, Flink and Iceberg, and opened its own read-only DuckDB at DUCKDB_PATH for freshness and quality (an unrelated database on the ClickHouse profile, a brand-new empty one on the :memory: default). It now has a `serving` component and reads the journal through the active backend. /v1/health always answered 200 — its status lives in the payload — and both Kubernetes probes AND the Compose healthcheck pointed at it, so a replica with a dead ClickHouse looked healthy to every orchestrator watching it. Readiness is now /health/ready (503 when the serving store is unreachable or unprovisioned — the error P0-2's removal of boot-time DDL makes possible) and liveness /health/live, which stays dependency-free so a ClickHouse outage cannot roll every pod. ControlPlaneStore.ping() added (no-op embedded, SELECT 1 on PG). The search index's entity scan is bounded (AGENTFLOW_SEARCH_SCAN_LIMIT, default 10k) and logs truncation: it was an unbounded SELECT * that grows with the serving data (audit P1-6). Notes on the SQL: the journal is DuckDB-flavoured with values bound as ? where the backend binds and _quote_literal-escaped where it does not (ClickHouse's execute(params=...) is a documented no-op), so entity_id from the URL path still binds exactly as before. The one inlined value is the time window, an int from config/slo.yaml: INTERVAL ? is a DuckDB syntax error and CAST(? AS INTERVAL) has no ClickHouse translation. quantile_cont(col, q) now transpiles to the parametric quantile(q)(col) — the SLO latency SLI was the one journal read with no valid ClickHouse form. Freshness subtracts against the store's own clock: the two stores keep naive journal timestamps in different zones. Tests coupled to the old architecture were moved onto it rather than deleted: the health checks' "unexpected errors propagate" invariant, lineage's "scan is offloaded to a worker thread" invariant, and the SLO router's schema-adaptive measurements all still hold, now through the backend. Verified: full unit+integration suite green apart from the three architecture-coupled tests above, which are rewritten and pass (23/23) · mypy 133 files clean · ruff clean · helm lint + template (probes render to /health/ready and /health/live) · all four journal queries exercised against a live DuckDB. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Tenant isolation was a schema qualification: TenantRouter mapped a tenant to a
`duckdb_schema` and SQLBuilderMixin rewrote `orders_v2` into `"acme"."orders_v2"`.
That was the only mechanism, and it did not work on either store.
Nothing in `src/` ever issues CREATE SCHEMA — only four test fixtures do. So the
tenant schema did not exist at runtime, and since every key in config/api_keys.yaml
carried `tenant: "acme-corp"`, *every* authenticated entity read was failing:
get_entity(tenant=None) -> HIT ORD-20260404-1001 # auth disabled
get_entity(tenant='acme-corp') -> ValueError: Table '"acme"."orders_v2"' ...
On ClickHouse the same name meant a database nobody creates (UNKNOWN_TABLE); drop
the qualification and every tenant shares one table — and one ReplacingMergeTree
key, so two tenants' rows with the same order_id are two versions of one row and
the later insert destroys the earlier one. That is data loss, and no read-side
filter can undo it.
The suite was green throughout, because its keys named tenants that are not in
tenants.yaml at all: the qualification resolved to nothing and the reads fell
through to the shared table. A boundary no test can tell apart from its own
absence is not a boundary.
One model now, both stores (ADR-004): a `tenant_id` column that leads the write
key — ORDER BY (tenant_id, <pk>) on ClickHouse, PRIMARY KEY (tenant_id, <pk>) on
DuckDB, where the pipeline upserts with INSERT OR REPLACE.
- Reads go through one chokepoint. `_qualify_table` returns a scoped relation,
not a name: `(SELECT * EXCLUDE (tenant_id) FROM orders_v2 WHERE tenant_id = 'x')
AS "orders_v2"`. `_scope_sql` does the same inside metric templates and NL SQL.
EXCLUDE keeps tenant_id out of `SELECT *`, so entity contracts are unchanged and
the two stores stay column-identical (sqlglot maps EXCLUDE -> ClickHouse EXCEPT).
- An unscoped read still fails closed: `_holds_foreign_tenant_rows` refuses a read
with no tenant context when the table holds rows of more than the default tenant.
- Writes stamp the tenant (`event_tenant()`, a leaf module both the pipeline and
the sink can reach). Aggregates group by it: refresh_user_aggregates took a set
of user ids and a global GROUP BY user_id, which would sum two tenants' orders
into one total and write it back to both.
- Search indexes every tenant (one corpus per process), so the tenant travels on
the document and is filtered before scoring, like the P0-4 allowlist.
- ClickHouse cannot prepend to a sorting key, and CREATE TABLE IF NOT EXISTS
silently keeps the old one, so `assert_tenant_key` refuses to serve a store that
predates the key and `provision --migrate` rebuilds it (staging table, backfill
under 'default', atomic EXCHANGE TABLES). DuckDB cannot alter a PRIMARY KEY, so
it says so and names the fix.
- local_pipeline kept a second, hand-maintained copy of the same five tables; it
had already drifted. It now calls DuckDBBackend.ensure_schema().
- Demo keys carried `acme-corp` while the seed and the live stream write under
`default`; they now name the tenant their data is actually under.
ruff clean, mypy clean (134 files), 2144 passed (4 Docker-only errors, unchanged).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…not there (P0-1 tail) d085a9d made the boundary physical -- `tenant_id` leading the write key on both stores. This closes what it left open: the proof, the dead mechanism it replaced, and a validator that did not validate. Proven, on a live ClickHouse. tests/integration/test_clickhouse_tenant_isolation_live.py plants two tenants in one store with the same order_id, user_id, session_id and product_id, then drives entity, timeline, metric, historical, NL, pagination, batch, search, lineage and SLO under both keys, plus qualified-SQL, CTE-shadowing and recursive-CTE escape attempts. 33 passed against ClickHouse 25.3.14.14. It provisions its own database so the foreign-tenant rows it plants cannot make an unscoped read elsewhere in the session fail closed. The property suite was still asserting the old model, and was red. Rewritten against the column model, and turned into an actual property test: it generates tenants and entity ids rather than sampling two, and rows accumulate across examples, so the invariant is checked against a store holding many tenants' colliding ids rather than exactly two. It immediately found a real bug. `_TENANT_ID_RE.match()` accepted "acme\n": Python's `$` also matches *before* a trailing newline, so a string that is a different tenant than "acme" passed validation and would have become its own silent partition. Now `fullmatch`. Not an injection -- quote_sql_literal still escapes -- but a boundary validator that accepts what it says it rejects is not one. duckdb_schema is gone: from TenantDefinition, from config/tenants.yaml, from the chart's shipped values, and TenantRouter.get_duckdb_schema() is deleted. It named the isolation mechanism, nothing read it, and nothing ever provisioned what it named -- a trap for whoever configures the next tenant. Two compatibility seams stay on purpose: pydantic ignores unknown keys, so an old tenants.yaml still loads; and the Helm values schema still *accepts* the key (tenant items are additionalProperties: false, so dropping it would reject values written for the old model) with a description saying it is ignored. scripts/restore.py asserted that a restored store had a schema per tenant, parsed out of that same duckdb_schema field -- and the backup workflow's fixture fabricated exactly those schemas, so the check passed only because the fixture had made it pass. When the field went away the regex matched nothing, the expected set went empty, and the check silently accepted anything, including a pipeline store with no serving tables. Both now use the product's own DDL: the fixture writes two tenants into one tenant_id-keyed table, restore asserts the serving tables come back carrying the column, and it refuses a store with no serving tables at all. A check that cannot fail is not a check. Docs say what is true per store. README, STATUS, security-audit.md (its "Tenant Isolation" section described the broken mechanism as a strength), release-readiness.md and helm-deployment.md: the boundary is implemented on both stores, proven on DuckDB by example and by property, and proven on ClickHouse by the live suite above. ruff check + format clean (src tests scripts sdk integrations warehouse) -- tests/unit/test_clickhouse_sink.py had been left unformatted by d085a9d, which would have failed the CI lint gate. mypy clean (134 files). 2181 passed, 89 skipped, 0 failed (4 errors = kafka/iceberg, need Docker). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…re a backup (P1-2, P1-4) P1-4. Dockerfile.api did `COPY config /app/config` with no allowlist, while pyproject.toml already excludes config/api_keys.yaml, config/webhooks.yaml and config/tenants.yaml from the sdist for the same reason -- bcrypt key hashes and webhook signing secrets. .dockerignore did not mirror that, so a future plaintext secret in one of those files would have been baked into an immutable image layer. .dockerignore now excludes the same three paths that scripts/check_release_artifacts.py already treats as forbidden release members. Compose and Helm are unaffected: both mount the real files over the image. The image also ran as root. Helm compensated (runAsUser: 10001); `docker run` did not. It now creates and switches to a non-root agentflow user at the same uid/gid, with /app/data pre-chowned so a fresh Compose named volume inherits it. Its HEALTHCHECK still asked /v1/health -- which always answers 200 and puts its verdict in the payload -- so a standalone container called itself healthy while the serving backend was unreachable and every read was failing. Helm and docker-compose.prod.yml had already been moved to /health/ready by e8664b2; the image was the last place still asking a question that cannot get a no. (P0-3 leftover, fixed here because it is the same file.) P1-2. "Nightly Backup" never touched a deployed environment: it built synthetic DuckDB fixtures on an ephemeral runner, tarred them, and uploaded a 7-day Actions artifact. That is a real regression test for the backup/restore code path and a fiction as a backup. Renamed to what it is, and docs/disaster-recovery.md drops the RPO 24h / RTO 15-30m claims it made on that strength -- nothing was ever measured against a real store, and DR covers neither ClickHouse nor the PostgreSQL control plane. Said plainly instead of implied away. scripts/backup.py swept all of config/ into the archive, api_keys.yaml included: credential material distributed by tar, Actions artifact and S3. It now reuses check_release_artifacts.FORBIDDEN_MEMBER_PATTERNS rather than keeping a second list to drift. No real ClickHouse/PostgreSQL backup was designed here -- that needs infrastructure this repo does not have. Honesty about the gap is the deliverable. New: tests/unit/test_docker_secret_policy.py (static; Docker is unavailable on the dev host) and tests/unit/test_backup.py, which asserts the archive holds no secret config, cross-checks it with find_forbidden_members(), and round-trips backup -> verify -> restore. Verified: 22 targeted tests pass; ruff check + format clean; workflow YAML parses. The image itself is not built here -- build-smoke does that on the PR. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
P1-5. CI ran ruff on src/ tests/ scripts/ sdk/ and left integrations/ and warehouse/ unlinted -- and a real import-sorting error was already sitting in that blind spot (integrations/.../crewai/tools.py). Fixed, and both paths are now under the lint gate. 21 of 35 workflow jobs had no timeout-minutes, including the core unit, integration, security and release jobs. All of them do now; a new tests/unit/test_workflow_timeouts.py fails if one is added without. The TypeScript SDK's typecheck, tests and build ran only in the publish workflow (on a tag). Every PR got `npm audit` and nothing else, so a broken SDK could merge and only fail at release. A new required-shaped `sdk-ts` job runs npm ci, typecheck, test, build and `npm pack --dry-run` on every PR. Wiring it into branch protection is a repo-settings change the owner still has to make. P1-3. The Safety scan covered core dependencies, the SDK and base integrations but not the extras that actually ship or deploy -- cloud, postgres, the root `integrations` extra, flink, load, contract. The matrix now resolves and scans each one. .github/dependabot.yml claimed that GitHub security advisories "bypass the schedule and open immediately". They do not: Dependabot *security* updates are a separate repo-level toggle, and the audit confirmed via the GitHub API that it is DISABLED on this repo. The comment no longer asserts something false and names the one command that would make it true. Flipping it is a repo-settings change, gated on the owner: gh api --method PUT repos/brownjuly2003-code/agentflow/automated-security-fixes Not done: the Python lock itself. `[tool.uv] environments` pins resolution to the 3.11-3.13 the project actually claims (without it, `uv lock` builds a universal resolution including hypothetical future interpreters, where apache-beam and the pyarrow pin genuinely conflict) -- but uv.lock is not generated yet, and nothing in CI references one, so this is a prerequisite landed on its own rather than a half-resolved lock committed as if it worked. Verified: ruff check + format clean across all six paths; every workflow YAML parses; targeted tests pass. The workflows themselves are not executed here. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
_NEXT_SESSION.md is working notes for the next agent -- untracked, but not ignored, which is one `git add -A` away from a public repository. The audit report next to it (/audit_*.md) was already named for the same reason. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… never install (P1-3) uv.lock is one resolution for Python 3.11-3.13; requirements-docker.lock is its hash-pinned export (cloud,postgres) and the only source Dockerfile.api installs third-party code from (--require-hashes, then the project wheel --no-deps, then pip check inside the build). CI job lock-check re-derives the whole chain; security.yml pip-audit reads the pinned set with nothing left to resolve. The [flink] extra is removed because it never existed as an installable thing: apache-flink 2.3.0 -> beam <=2.61 caps pyarrow<17 against core's pyarrow>=17, so a fresh 'pip install .[flink]' cannot resolve on any Python. The Flink runtime image gets its own pinned manifest (src/processing/flink_jobs/requirements.txt, asserted against ARG FLINK_VERSION at build), and Safety scans that file instead. Verified: uv lock --check; export == committed file; fresh py3.11 venv pip install --require-hashes + pip check green; pip-audit over the lock: no known vulnerabilities. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…rsioned migrations, process roles (P1-1) PostgresControlPlaneStore checks connections out of a bounded psycopg_pool.ConnectionPool (AGENTFLOW_CONTROLPLANE_PG_POOL_MIN/MAX/ TIMEOUT_SECONDS, default 1/10/10s) instead of psycopg.connect() per method call; transaction semantics are unchanged and pool pressure is scrapeable (agentflow_pg_pool_*) and alertable (ControlPlanePoolSaturated, UsageRowsDropped). record_api_usage_batch is one executemany in one transaction instead of a connection per row. Schema DDL becomes a monotonic migration ledger (control_plane_schema_version) serialized across replicas by a transaction-scoped advisory lock; a pre-versioning database upgrades in place via the IF NOT EXISTS baseline. AGENTFLOW_PROCESS_ROLE=api|worker|all splits serving replicas from the one process running the delivery loops, and refuses the embedded profile where those loops exist nowhere else. The ledger deliberately makes ensure_*_schema() non-restorative — a dropped table is a failure to see, not to heal — so the fault-injection probes now repair their own damage with raw baseline DDL. Live-verified on PostgreSQL 17.10: 37/37 probes, including pool bound under 16 concurrent writers, a 256-row batch sharing one xmin, four replicas racing a fresh migration run, and both split-role boot shapes. Unit 1913+ green, integration green (4 known Docker ERRORs), ruff+format 429 files clean, mypy 134 files clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The periodic tick full-scanned and re-tokenized every entity table every 60 seconds, per replica, forever. refresh() reads pipeline_events past a cursor instead: a quiet journal costs one bounded read and zero scans; a small change-set costs targeted IN re-reads of exactly the changed rows (scan_entity_rows_by_ids, through the active backend), applied copy-on-write so a concurrent search never observes a half-applied batch, with document frequencies maintained incrementally and the result pinned byte-equivalent to a full rebuild. The full pass survives where it is honest: cold cursor, journal-window overflow, an oversized change-set, and a scheduled every-N-ticks safety net for journal-bypassing writers and deletions (bounded staleness instead of silent staleness). Verified: 10 new unit probes including rebuild-equivalence, deletion, tenant collision, atomic swap, and a tracemalloc scale probe (10-row refresh over a 3000-row corpus allocates <1/5 of a rebuild's peak, zero wholesale scans); live ClickHouse 25.3 suite now 34/34 — a post-boot row is indexed incrementally through the real sqlglot round trip and visible to its own tenant only. mypy clean; unit+property 1921+19 green (one timing flake green in isolation, same class as the pinned one). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
/v1/slo's 'current' was threshold/measured off one point aggregate — not a
share of good events, and error_budget_remaining applied the budget formula
to a number with a nonstandard meaning. Now every SLI is good/valid over
the window: latency = events at or under the threshold (p95 demoted to
diagnostic), errors = non-5xx share, freshness = time-weighted — the share
of observed seconds during which the newest journal row was at most
threshold old, reconstructed exactly from event gaps. Missing data reads
'unknown' with null numbers instead of a fabricated 0.0. Multi-window burn
rates (1h/6h/3d) mark at_risk when the (1h,6h) pair exceeds 14.4 or the
(6h,3d) pair exceeds 6 — a month of clean traffic no longer hides an hour
burning budget at 25x. good/valid/unit ship in the payload so the numbers
are auditable.
Two real defects found on the way, both pinned: DATE_DIFF('second')
truncation floored every sub-second gap to zero and systematically
under-reported freshness on a busy journal (now milliseconds); and on
ClickHouse LAG transpiles to lagInFrame, which hands the FIRST row a
zero-date instead of NULL — without the epoch guard that row booked a
phantom threshold-sized credit (found live on 25.3, pinned by an
exact-numbers live probe).
Verified: SLO unit suite rewritten (22 green, incl. fast-burn paging and
unknown semantics), integration 5 green, live ClickHouse suite 35/35,
unit+property full sweep green, mypy clean, openapi.json regenerated.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… (P2-1) The runtime moved to 2.0.0 while the docs kept describing older builds: FastAPI/OpenAPI froze at 1.0.0, SECURITY.md supported a v1.x line that is no longer current, release-readiness.md sat on v1.4.0 with a 12-check count (branch protection has 13, incl. build-smoke), security-audit.md cited the PII masker deleted on 2026-07-01, architecture.md called the in-process 1.06 s figure a production default and DuckDB the default backend, deployment.md described a make demo that no longer exists, and the startup log said auth was "open" when no-keys actually fail-closes with 503. - src/version.py: single version source (pyproject first — editable metadata goes stale on bump; installed wheels fall through to importlib.metadata). FastAPI/OpenAPI now report the package version. - Startup log states the real posture: fail_closed_no_keys / open only under AGENTFLOW_AUTH_DISABLED. - SECURITY.md, release-readiness.md, security-audit.md, architecture.md (incl. the U+FFFD box art and the fail-open rate-limiter claim — the limiter fails closed to a per-process cap), deployment.md brought to current truth; X-PII-Masked changelog entries note the header retired. - tests/unit/test_docs_single_source_of_truth.py: the CI ratchet — version cohesion across app/openapi.json/Chart.yaml/pyproject, SECURITY.md current line, release-readiness self-consistency, removed-path scan over current-state docs (ADRs/perf reports stay historical), no replacement characters, authoritative links resolve. Failed 7/9 on the pre-fix tree; mkdocs.yml points at it. Deliberately not done (over-engineering for a solo reference repo): docs/claims.yaml evidence registry and a docs/ tree restructure into current/evidence/archive. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…t-class (P2-3)
The backend has supported CLICKHOUSE_SECURE since H-C2, but nothing
required it and nothing could express it in the chart: a scale-profile
deployment spoke HTTP Basic Auth to an external ClickHouse unless the
operator hand-rolled extraEnv, Redis/Postgres TLS was never checked at
all, and a wildcard CORS origin rode into any profile with credentials
on.
- src/serving/transport_policy.py: AGENTFLOW_PROFILE (demo|dev|
production; demo mode implies demo, default dev). On production the
boot refuses plaintext transport to an external ClickHouse/Redis/
PostgreSQL — loopback exempt (the bytes never leave the host), every
problem reported at once, deliberate exceptions named per store in
AGENTFLOW_INSECURE_TRANSPORT_OK. Wildcard CORS with credentials is
demo-only and dies at import, not at request time. Wired into the API
lifespan (before any store/client is built), the provision CLI (same
wire, same gate), and production+demo-mode refuses to boot outright.
- ClickHouse client: ca_cert (CLICKHOUSE_CA_CERT / serving.yaml
ca_cert) builds the TLS context from ONLY the operator's PEM bundle —
a public CA can no longer impersonate a private-CA endpoint; hostname
verification stays on; a bad bundle path refuses to construct.
- Helm: serving.clickhouse.secure + serving.clickhouse.tls.{caSecret,
caKey} render CLICKHOUSE_SECURE and a read-only CA mount into BOTH
consumers of the wire (API deployment and provision Job);
config.profile arms the app-side gate. values.schema.json stays
strict (additionalProperties:false, profile enum).
- Proof: 33 unit tests (policy matrix, boot wiring via TestClient, TLS
context pins against a checked-in test CA — the one deliberate *.pem
gitignore exception, certificate only) and a live probe against
ClickHouse 25.3 behind a private CA (3/3: private bundle sufficient,
system store refuses it, 127.0.0.1 vs DNS:localhost refused).
Full unit sweep 1958 passed, property 13, helm renders pinned by
test_serving_clickhouse_tls_render_is_first_class.
Non-goals, recorded: client certificates (terminate mTLS at the
ingress/mesh), an SNI override (urllib offers no clean seam; the cert
must match the configured host).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ope (P2-4) The S3 backend hard-coded key=infrastructure/terraform.tfstate for every environment, so a dev plan and a production apply would read and write the same state object. The Makefile and the workflow also disagreed on where tfvars live (root dev/prod.tfvars vs environments/*.tfvars), and a committed root prod.tfvars full of placeholder values looked like a production input packet. - backend "s3" carries no key any more: init MUST name the environment's state object (-backend-config="key=env/<environment>/terraform.tfstate"); a bare init fails instead of silently sharing state. Wired into Makefile deploy-dev/deploy-prod and both workflow init sites. CI's terraform-validate uses -backend=false and is unaffected (fmt -check and validate both green locally). - Root prod.tfvars removed: production inputs exist only as the operator-provided environments/prod.tfvars (never committed; the .example gained the github_org/github_repo lines the root file had). deploy-prod now resolves the same path the workflow does and explains itself when the file is absent. dev.tfvars stays as the local sandbox scaffold, and aws-oidc-setup.md says so. - The workflow is named for what it provisions — Terraform (streaming-infrastructure-reference) — Kafka/Flink/object storage/ monitoring; API/ClickHouse/PostgreSQL/Redis/ingress remain operator-owned and apply stays if:false pending external approval. Not done from P2-4, deliberately: bootstrapping the state bucket/lock table and a full runtime deployment — the audit offers reference-scope as the honest alternative, and that is what this names. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
DORA Metrics
|
The load/perf seeder carried a private copy of the serving DDL with single-column primary keys. The P0-1 boot gate (assert_tenant_key) correctly refuses such a store, so both CI gates that boot the API over a seeded file — load-test and perf-smoke — died waiting for /v1/health. The schema now comes from DuckDBBackend.ensure_schema(), the single owner of the serving DDL; inserts list their columns explicitly so tenant_id takes its DEFAULT — the tenant the read path resolves when a request names none. Verified end-to-end locally: seed, boot, health 200, /v1/entity and /v1/search serve the seeded rows. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The strict-schema probe pinned the exact validator message. Helm
changed it ("Additional property bogus is not allowed" became
"additional properties 'bogus' not allowed"), so the test failed on
CI's newer helm while the schema did exactly its job. Assert the
meaning — nonzero exit, the key named, rejected as an additional
property — under both wordings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… CVE trivy-action builds the SARIF "with all severities" unless limit-severities-for-sarif is set, silently dropping both severity: HIGH,CRITICAL and ignore-unfixed from the scan the exit code comes from — an unfixable MEDIUM in the Debian base (liblzma5 CVE-2026-34743, no fix released) failed a gate that declares itself HIGH,CRITICAL-only. Setting the flag makes the declared filters real. Safety: ignore SFTY-20260217-93940 (pyarrow IPC use-after-free) in the flink-runtime bucket only — beam caps pyarrow<17 there so the fixed 23.0.1 cannot install, while the API image and every extra already run pyarrow 24 under pip-audit. Reason and removal condition documented inline, same pattern as the existing 88512 ignore; the string-ID ignore mechanics verified live against a pyarrow==16.1.0 pin. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ackage gate The branch added scan_entity_rows / scan_entity_rows_by_ids (P0-3 read contract, P1-6 incremental refresh) to the query package, but their tests live with the search index — outside the six files the per-package coverage gate runs — so the gate slid to 89% against its fail-under=90. Cover them where the gate looks, in the same Mock-host idiom, plus fetch_orders_by_status which had no direct unit coverage at all: entity_queries.py 79% -> 100%, package 89% -> 94%, so the gate no longer sits one statement from its threshold. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Third copy of the same failure shape: seed_benchmark_fixtures runs after the canonical owner has laid down the tenant schema, but its positional VALUES still counted six columns against the now seven-column serving tables (tenant_id leads, audit P0-1), so perf-check died in the binder the first time it ran on this branch. Columns are now listed explicitly and tenant_id takes its DEFAULT, same as the load seeder. Verified by running the full benchmark locally: seed, boot, warmup and the 60s canonical run all pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hardening pass across security, reliability, CI supply chain, docs, and infra — 15 commits, each independently verified (unit + property + live Postgres 17 / ClickHouse 25.3 probes).
Security
/v1/searchclosed (d6de01a) — search results are now tenant-scoped at the backend contract level.d085a9d,0854355) — tenant isolation enforced physically in DuckDB and ClickHouse; dead soft-isolation mechanism removed; proven on live CH.691db8a) —AGENTFLOW_PROFILE=productionrejects plaintext transport to external ClickHouse/Redis/Postgres (loopback exempt, explicit per-store override); wildcard CORS + credentials dies at import outside demo; CH client builds its TLS context exclusively from the operator CA bundle with hostname verification always on. Helm gets first-class TLS values. Live probe against CH 25.3 behind a private CA: 3/3.e425a32).Reliability
AGENTFLOW_PROCESS_ROLE=api|worker|allsplit (bdd3b29) — 37/37 live PG probes.47f7815); read surfaces and health go through the backend contract (e8664b2).e18d36a) — journal-cursor refresh, targeted scans, copy-on-write + atomic swap; full rebuild retained for cold/overflow paths./v1/sloserves real SLIs (8e53dad) — good/valid ratios per window, time-weighted freshness, multi-window burn rates; two live CH arithmetic bugs (DATE_DIFF truncation, lagInFrame zero-date) found and pinned by tests.Supply chain & CI
uv.lock+ hash-pinned Docker installs (09b3c03) — third-party deps install only from the hash-pinned lock;lock-checkCI job;pip-auditover the lock; unresolvable[flink]extra removed in favor of a dedicated runtime manifest.cbfc126).Docs & infra
24f6b5a) — one version source, ratchet test (test_docs_single_source_of_truth.py) that fails on doc drift.9c8cc17) —-backend-configkeys per env, bare init fails loudly instead of silently sharing state; placeholder prod tfvars removed; workflow renamed to its honest scope.Verification
pip check🤖 Generated with Claude Code