From a1b81d7e4fb8b60796baa0772aa4727c81f99598 Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Fri, 2 Oct 2026 10:44:13 -0400 Subject: [PATCH 1/9] docs(adr): make the audit store swappable to a warehouse (ADR 0009) ADR 0006 kept Postgres as the only audit store of record and fed an external sink as a tee. That leaves years of audit history in the operational database, writes no locked archive, keeps dashboard and export load on Postgres, and gives two incomplete copies: the tee skips platform-level events and the auth resolver's raw-SQL inserts. ADR 0009 adds swap values to AUDIT_SINK (bigquery, clickhouse). Under a swap value the warehouse is the store of record and Postgres keeps only a transactional queue row, committed in the caller's transaction and drained in batches by a relay (at-least-once, idempotent by event id, ordered per organization, deleted after the adapter acknowledges). Adapters live in the accounts service behind one read/write interface, reads move with writes under service-enforced tenant scope, and each event type declares a retention class that routes it to warehouse tiers and a locked object-storage archive. postgres stays the default and both is unchanged. ADR 0006 is marked superseded in part (its sink/tee decision); ADR 0003 gets a pointer that its in-Postgres analytics stance now holds for the postgres default only. Docs only. Co-Authored-By: Claude Opus 5.5 --- .../adr/0003-typed-audit-event-registry.md | 1 + .../0006-audit-sink-and-retention-tiers.md | 2 +- module/docs/adr/0009-audit-store-swap.md | 241 ++++++++++++++++++ 3 files changed, 243 insertions(+), 1 deletion(-) create mode 100644 module/docs/adr/0009-audit-store-swap.md diff --git a/module/docs/adr/0003-typed-audit-event-registry.md b/module/docs/adr/0003-typed-audit-event-registry.md index 1d50d2050..1b9b4b7a8 100644 --- a/module/docs/adr/0003-typed-audit-event-registry.md +++ b/module/docs/adr/0003-typed-audit-event-registry.md @@ -1,6 +1,7 @@ # ADR 0003: Typed audit-event registry — single-table STI, a code-owned catalog, and in-Postgres analytics - Status: Proposed +- Refined by: [ADR 0009](./0009-audit-store-swap.md) — a deployment may select a warehouse as its audit store, so "analytics stays in Postgres" holds for the `postgres` default only - Date: 2026-08-18 - Task: closes the review-and-planning deliverable of #170. Implementation is deferred to follow-up tickets that this ADR authorizes but does not open. diff --git a/module/docs/adr/0006-audit-sink-and-retention-tiers.md b/module/docs/adr/0006-audit-sink-and-retention-tiers.md index deb73c329..b081f354a 100644 --- a/module/docs/adr/0006-audit-sink-and-retention-tiers.md +++ b/module/docs/adr/0006-audit-sink-and-retention-tiers.md @@ -1,6 +1,6 @@ # ADR 0006: Audit sink and retention tiers — a swappable compliance backend, Postgres as the atomic hot source of truth -- Status: Proposed +- Status: Proposed; superseded in part by [ADR 0009](./0009-audit-store-swap.md) — the sink/tee decision ("a tee, not a swap": Decision item 2, option B) - Date: 2026-09-02 - Task: prepares the audit performance/size and compliance workstreams (see #430). Refines — for the **sink and retention dimension only** — the diff --git a/module/docs/adr/0009-audit-store-swap.md b/module/docs/adr/0009-audit-store-swap.md new file mode 100644 index 000000000..ae36551e0 --- /dev/null +++ b/module/docs/adr/0009-audit-store-swap.md @@ -0,0 +1,241 @@ +# ADR 0009: Audit store swap — a warehouse as the audit store of record, Postgres as its transactional queue + +- Status: Proposed +- Date: 2026-10-02 +- Supersedes, in part: [ADR 0006](./0006-audit-sink-and-retention-tiers.md) — its + sink decision, "a tee, not a swap" (Decision item 2, option B), and for a + deployment that selects a swap value, its retention tiers (Decision item 3). + ADR 0006's admission policy and sizing stand. +- Refines: [ADR 0003](./0003-typed-audit-event-registry.md) — "analytics stays in + Postgres" now holds for the `postgres` default only. The registry and the + single write path are unchanged. +- It does not change code. + +## Context + +ADR 0006 kept Postgres as the only audit store of record and added an opt-in +tee: with `AUDIT_SINK=both`, the emitter enqueues an export job beside each +org-scoped audit row, and a worker posts it to an operator-configured HTTP +endpoint (`HTTPAuditSink`), one event per request. That keeps the write-path +guarantee, and it remains the right answer for a deployment that only needs a +copy somewhere else. It is not enough for a deployment whose audit must be kept +for years, locked, and queried heavily: + +1. **Long retention.** Postgres holds every audit row for the + `data_retention_policies` window (365 days by default; `RunRetention` drops + whole monthly partitions). A compliance window of several years keeps that + history in the operational database — ADR 0006's sizing puts seven years at + ~2 TB for a small deployment and ~22 TB for a medium one. The tee does not + shrink Postgres; it adds a copy. +2. **A locked archive.** A compliance regime may require a write-once copy that + no operator can alter or delete before its window ends. The kit writes no + such copy; the tee's destination is whatever sits behind an HTTP endpoint. +3. **Query load.** The activity list, dashboard aggregation + (`AggregateAuditLog`: grouped counts over time buckets and payload + predicates), exports and readable-source queries all run against the + operational database. Dashboards make audit an analytical workload on the + instance that serves every mutation. +4. **One store to query.** Under the tee there are two copies and neither is + complete. The warehouse copy lacks platform-level events (no organization): + the job platform admits a global-scope job only from the privileged worker + pool, which cannot enqueue on the caller's transaction, so `work.go` warns + that these stay Postgres-only. It also lacks the rows the auth resolver + writes with raw SQL (`pkg/auth/pg/resolver.go`), which never pass through the + emitter. Postgres lacks everything older than its window. Every reader has + to know which copy answers which question. + +The requirement ADR 0006 refused to trade away still stands: no state change +commits without its compliance record, and no external system can join a +Postgres transaction. + +## Decision + +1. **Swap, not tee.** `AUDIT_SINK` gains two swap values, `bigquery` and + `clickhouse`. Under a swap value the selected warehouse is the audit store of + record and Postgres no longer keeps audit history. `postgres` stays the + default for every deployment that does not opt in. `both` — ADR 0006's tee — + keeps its current behaviour, for compatibility. `external` stays refused. + +2. **Postgres keeps only the transactional queue.** `EmitTx` still writes inside + the caller's transaction (and `Emit` inside its own, for the observational + types of ADR 0003's durability amendment), but what it writes is a + short-lived queue row, not a permanent audit row. The queue row commits + together with the domain event that webhook delivery fans out from, exactly + as the audit row does today, so "no state change without its compliance + record" holds unchanged. Every audit write goes through the queue — including + platform-level (no-organization, control-plane) events, which the swap must + not leave Postgres-only the way the tee does. + + A relay drains the queue to the selected adapter: + - in **batches**; + - **at-least-once, idempotent by event id** — a redelivered event leaves one + record; + - **ordered per organization** (platform-level events form one more ordering + key); + - a queue row is **deleted only after the adapter acknowledges** the write. + + The idempotency reservation (`audit_event_idempotency`) stays in Postgres + beside the queue row, as it is today. + +3. **Adapters live inside the kit.** They are Go, in the accounts service, behind + one `AuditStore` interface carrying both halves: + - **write** — append a batch; + - **read** — list (keyset-paginated, as `QueryAuditLog` is), aggregate (as + `AggregateAuditLog`), export, and the readable-source queries + (`LatestSourceSyncRequests`). + + Two adapters ship: **BigQuery** (managed, Google Cloud) and **ClickHouse** + (self-hosted or managed, any cloud or on-premises). The existing Postgres + reads become the third implementation of the same interface and the + reference the other two are measured against. + +4. **Reads move with writes.** Under a swap value the activity list, dashboard + aggregation, exports and readable-source queries read from the adapter. + Postgres row-level security does not reach a warehouse, so tenant scope is + enforced in the service on every query: an organization's read always + carries its organization predicate, only the platform-admin path may span + organizations, and the adapter refuses a query that arrives without a scope. + Adapter-native row policies (BigQuery row-level access policies, ClickHouse + row policies) are defence in depth, not the boundary. + +5. **Retention tiers.** Every registered event type — code-owned, or declared + by a solution or a composed module — declares a **retention class** in the + typed registry (ADR 0003): + - **security** — authentication, failed authentication, permission and role + changes, admin actions, data exports, configuration changes; + - **content** — everything else. + + The retention class is its own field. `Category`'s `security` value names a + narrower set and is not reused for it. + + The warehouse keeps two tables: + - an **events** table holding the envelope — actor, organization, event + type, target id, timestamp, result — plus a hash of the details, and the + full details as well for security-class types; kept for the compliance + window; + - a **details** table holding the full details of content-class types, kept + for a shorter, configurable window. + + A **locked object-storage archive** holds the compliance copy: full for + security-class events, content-free (envelope plus hash) for content-class + events. Locked means write-once under a retention lock the operator cannot + shorten: GCS Bucket Lock, S3 Object Lock, Azure Blob immutability policies, + or an object-lock-capable store on-premises. + + Windows are deployment configuration, not code. + +6. **ClickHouse operating requirements.** + - Inserts are batched — the relay's batches, `async_insert`, or both. A + single-row insert creates a data part per insert and ends in + too-many-parts errors. + - Tables are replicated (`Replicated*MergeTree`), coordinated by ClickHouse + Keeper. + - Tables are partitioned by month, and table TTL expires each tier at its + window. + - Growing past one shard is a planned operation, not an automatic one — or + the deployment uses a managed ClickHouse with shared storage. + +7. **History migration.** Switching a deployment from `postgres` to a swap value + includes a one-time copy of the existing audit rows into the adapter and the + archive, verified per organization by row counts and hashes. Afterwards the + Postgres audit table holds only queue rows. + +8. **Conformance.** Every adapter passes one shared test suite: append + idempotency (a redelivered batch leaves one record per event id), batch + append, list and aggregate parity with the Postgres implementation over the + same fixture, refusal of a query without tenant scope, and retention-class + routing (a content-class event's full details land only in the details table, + and only its content-free form in the archive). + +## Alternatives considered + +The queue between the transaction and the warehouse (item 2): + +| Alternative | Rejected because | +| --- | --- | +| Local disk spool in the accounts process | Crash gap between the transaction commit and the spool write: the mutation is committed and its record is nowhere durable. | +| Synchronous write to the warehouse in the request | A warehouse outage either fails the mutation or loses the event; there is no third outcome. | +| A message broker | The same dual write — commit, then publish — with the same gap, plus a new system to run. | +| Change data capture on the Postgres WAL | Still needs the Postgres row in order to capture it, and replication slots plus a connector are heavier to run than a relay over a table. | + +Where the adapters live (item 3): + +| Alternative | Rejected because | +| --- | --- | +| A separate warehouse service the kit calls | One more deployable per environment, a second queue in front of it, a second permission check, and every read becomes a network hop. Worth it only if producers or readers outside the kit needed the store directly; none do. | +| One managed warehouse per cloud (BigQuery, Redshift, Synapse) | One adapter per cloud to build and hold at parity. ClickHouse covers every non-Google environment, on-premises included, with one adapter. | + +The sink shape itself: + +| Alternative | Rejected because | +| --- | --- | +| Keep the tee as the only option (ADR 0006) | Fails the four needs under Context. It stays available as `both`. | + +## Consequences + +**What changes in code** (paths under `module/services/accounts/code/`): + +- `work.go` — `configuredAuditSink` admits `bigquery` and `clickhouse`, builds + the adapter and starts the relay. The `external` refusal stays; its message, + which today says Postgres is the source of truth under every value, is + reworded. +- `pkg/business/audit.go` — under a swap value, the emitter's write inserts a + queue row in place of the permanent audit row, for org-scoped and + platform-level entries alike. `EmitTx`'s transaction contract, + `VerifyAuditWiring` and the domain-event publish are unchanged. + `AggregateAuditLogForReader` runs the reader's grant check and the aggregate + in one tenant transaction today; under a swap value the check stays in + Postgres and the aggregate goes to the adapter, so the two no longer share a + snapshot. +- `pkg/auth/pg/resolver.go` — `emitSsoJitAudit` and `emitUserRegistered` insert + audit rows with raw SQL on the resolution transaction. They must write queue + rows too, or the swap loses them. +- `pkg/infra/postgres_audit.go` — `QueryAuditLog` and `AggregateAuditLog` become + the Postgres implementation of `AuditStore`. The aggregate's category + dimension reads `audit_event_types` in SQL; an adapter takes it from the + registry in the service instead. +- `pkg/infra/postgres_readable_sources.go` — `LatestSourceSyncRequests` joins + `audit_events` to `principals` in one query, inside the source-read snapshot. + Under a swap value it becomes two reads — actor ids from the adapter, display + names from Postgres — outside that snapshot. +- `pkg/business/audit_registry.go` — `AuditEventDefinition` gains the retention + class, required the way `Durability` is. +- `pkg/business/retention.go` — partition drop keeps running for the Postgres + store. Under a swap value, tier expiry belongs to the warehouse (TTL or table + expiration) and to the archive's lock. +- `store/migrations` — the queue's storage. `audit_events` is append-only: the + `audit_events_no_delete` and `audit_events_no_update` triggers reject row + deletes, and the tenant and control-plane roles hold `SELECT` and `INSERT` + only. Delete-after-acknowledge needs a delete path those rules do not grant; + providing one must not weaken the append-only rule for the `postgres` + default. The tenant policy (`audit_events_tenant`, in `1_baseline.up.sql`) + keeps guarding queue rows; it no longer guards stored history, which the + service-side scope of item 4 does. +- The envelope's **result** has no column today: some types carry it in their + name (`saas.approval.denied`), others in an `outcome` payload field. The + events table needs it as a column, so each type has to say where its result + comes from. + +**What stays:** + +- `postgres` is the default and behaves exactly as today: audit rows in Postgres, + read under row-level security, retention by partition drop. +- `both` behaves exactly as today (`pkg/business/audit_jobs.go`, + `pkg/business/audit_http_sink.go`), including its org-scoped-only coverage. +- The typed registry and its payload validation, the durability rule, the + idempotency reservation, and webhook fan-out. +- ADR 0006's admission policy and sizing. [ADR 0008](./0008-datasource-sync-actor-provenance.md)'s + rule that a sync's actor lives only in the audit trail — under a swap value, + that trail is the adapter. + +**Costs accepted:** + +- Under a swap value, reads trail writes: an event is listed once the relay has + delivered it, not when its mutation commits. +- A deployment that opts in runs two more systems — the warehouse and the + archive bucket — and a relay whose lag needs watching. +- Each adapter must stay at parity with the Postgres implementation; the + conformance suite is what holds it there. +- Left to the implementation: whether the relay writes the archive copy as part + of the acknowledged batch, or the archive is filled from the warehouse + afterwards. From eb417245146eaf9574fb6ee7ddea59fb67c5c0cd Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Fri, 2 Oct 2026 10:47:29 -0400 Subject: [PATCH 2/9] docs(adr): settle the audit queue, history removal and archive write in ADR 0009 The queue is its own short-lived table, not audit_events: audit_events keeps its append-only triggers and grants and receives no new rows under a swap value, so no trigger is relaxed for any postgres or both deployment. The history migration removes copied rows by detaching and dropping the verified monthly partitions, the mechanism RunRetention already uses. The relay writes each batch to the warehouse and as one object to the locked archive, and deletes the batch's queue rows only after both acknowledge. The envelope drops its result field: an event's outcome stays in its type name or its payload's outcome field. Relaxing the no-delete trigger and filling the archive from the warehouse later are recorded as rejected alternatives. Co-Authored-By: Claude Opus 5.5 --- module/docs/adr/0009-audit-store-swap.md | 110 +++++++++++++---------- 1 file changed, 61 insertions(+), 49 deletions(-) diff --git a/module/docs/adr/0009-audit-store-swap.md b/module/docs/adr/0009-audit-store-swap.md index ae36551e0..90abd729d 100644 --- a/module/docs/adr/0009-audit-store-swap.md +++ b/module/docs/adr/0009-audit-store-swap.md @@ -58,21 +58,31 @@ Postgres transaction. 2. **Postgres keeps only the transactional queue.** `EmitTx` still writes inside the caller's transaction (and `Emit` inside its own, for the observational - types of ADR 0003's durability amendment), but what it writes is a - short-lived queue row, not a permanent audit row. The queue row commits - together with the domain event that webhook delivery fans out from, exactly - as the audit row does today, so "no state change without its compliance - record" holds unchanged. Every audit write goes through the queue — including + types of ADR 0003's durability amendment), but what it writes is a row in a + **separate, short-lived queue table** — not a permanent row in + `audit_events`. The queue row commits together with the mutation and with + the domain event that webhook delivery fans out from, exactly as the audit + row does today, so "no state change without its compliance record" holds + unchanged. Every audit write goes through the queue — including platform-level (no-organization, control-plane) events, which the swap must not leave Postgres-only the way the tee does. - A relay drains the queue to the selected adapter: + `audit_events` keeps its append-only triggers and its grants unchanged; under + a swap value it simply receives no new rows. The queue being its own table + is what keeps the append-only guarantee intact for every `postgres` and + `both` deployment: no trigger is relaxed to make room for a delete. + + A relay drains the queue: - in **batches**; - **at-least-once, idempotent by event id** — a redelivered event leaves one - record; + record in the warehouse; - **ordered per organization** (platform-level events form one more ordering key); - - a queue row is **deleted only after the adapter acknowledges** the write. + - each batch is **appended to the warehouse and written as one object to the + locked archive** (item 5) — the archive copy is written within the batch, + not filled in later; + - the batch's queue rows are **deleted only after both writes are + acknowledged**. The idempotency reservation (`audit_event_idempotency`) stays in Postgres beside the queue row, as it is today. @@ -110,15 +120,16 @@ Postgres transaction. The warehouse keeps two tables: - an **events** table holding the envelope — actor, organization, event - type, target id, timestamp, result — plus a hash of the details, and the - full details as well for security-class types; kept for the compliance - window; + type, target id, timestamp — plus a hash of the details, and the full + details as well for security-class types; kept for the compliance window. + An event's outcome stays where it is today: in the event type's name + (`saas.approval.denied`) or in the payload's `outcome` field; - a **details** table holding the full details of content-class types, kept for a shorter, configurable window. - A **locked object-storage archive** holds the compliance copy: full for - security-class events, content-free (envelope plus hash) for content-class - events. Locked means write-once under a retention lock the operator cannot + A **locked object-storage archive** holds the compliance copy, one object + per relay batch: full for security-class events, content-free (envelope plus + hash) for content-class events. Locked means write-once under a retention lock the operator cannot shorten: GCS Bucket Lock, S3 Object Lock, Azure Blob immutability policies, or an object-lock-capable store on-premises. @@ -136,9 +147,12 @@ Postgres transaction. the deployment uses a managed ClickHouse with shared storage. 7. **History migration.** Switching a deployment from `postgres` to a swap value - includes a one-time copy of the existing audit rows into the adapter and the - archive, verified per organization by row counts and hashes. Afterwards the - Postgres audit table holds only queue rows. + includes a one-time copy of the existing `audit_events` rows into the + warehouse and the archive, verified per organization by row counts and by + per-row hashes. Once verified, the copied monthly partitions are detached + and dropped — the mechanism `RunRetention` already uses — so no row is + deleted and the append-only triggers stay as they are. From then on + `audit_events` receives no new rows, and Postgres holds only the queue. 8. **Conformance.** Every adapter passes one shared test suite: append idempotency (a redelivered batch leaves one record per event id), batch @@ -157,6 +171,8 @@ The queue between the transaction and the warehouse (item 2): | Synchronous write to the warehouse in the request | A warehouse outage either fails the mutation or loses the event; there is no third outcome. | | A message broker | The same dual write — commit, then publish — with the same gap, plus a new system to run. | | Change data capture on the Postgres WAL | Still needs the Postgres row in order to capture it, and replication slots plus a connector are heavier to run than a relay over a table. | +| Queue rows in `audit_events`, with the no-delete trigger relaxed for them | Weakens the append-only guarantee on the table every `postgres` and `both` deployment keeps its history in, to serve a mode those deployments do not use. | +| Fill the archive from the warehouse later, on a schedule | Opens a window in which the compliance copy lags the store of record, and adds a second job to watch. | Where the adapters live (item 3): @@ -176,45 +192,41 @@ The sink shape itself: **What changes in code** (paths under `module/services/accounts/code/`): - `work.go` — `configuredAuditSink` admits `bigquery` and `clickhouse`, builds - the adapter and starts the relay. The `external` refusal stays; its message, - which today says Postgres is the source of truth under every value, is - reworded. -- `pkg/business/audit.go` — under a swap value, the emitter's write inserts a - queue row in place of the permanent audit row, for org-scoped and - platform-level entries alike. `EmitTx`'s transaction contract, - `VerifyAuditWiring` and the domain-event publish are unchanged. - `AggregateAuditLogForReader` runs the reader's grant check and the aggregate - in one tenant transaction today; under a swap value the check stays in - Postgres and the aggregate goes to the adapter, so the two no longer share a - snapshot. + the adapter and the archive writer, and starts the relay. The `external` + refusal stays; its message, which today says Postgres is the source of truth + under every value, is reworded. +- `pkg/business/audit.go` — under a swap value, the emitter's write inserts into + the queue table in place of `audit_events`, for org-scoped and platform-level + entries alike. `EmitTx`'s transaction contract, `VerifyAuditWiring` and the + domain-event publish are unchanged. `AggregateAuditLogForReader` runs the + reader's grant check and the aggregate in one tenant transaction today; under + a swap value it becomes grant check in Postgres, then the aggregate in the + warehouse — two steps, no shared snapshot. - `pkg/auth/pg/resolver.go` — `emitSsoJitAudit` and `emitUserRegistered` insert - audit rows with raw SQL on the resolution transaction. They must write queue - rows too, or the swap loses them. + into `audit_events` with raw SQL on the resolution transaction, bypassing the + emitter. Under a swap value they must go through the queue, or the swap loses + them. - `pkg/infra/postgres_audit.go` — `QueryAuditLog` and `AggregateAuditLog` become the Postgres implementation of `AuditStore`. The aggregate's category dimension reads `audit_event_types` in SQL; an adapter takes it from the registry in the service instead. - `pkg/infra/postgres_readable_sources.go` — `LatestSourceSyncRequests` joins `audit_events` to `principals` in one query, inside the source-read snapshot. - Under a swap value it becomes two reads — actor ids from the adapter, display - names from Postgres — outside that snapshot. + Under a swap value it becomes a warehouse query for the latest actor ids, then + a join to display names in the service from Postgres — outside that snapshot. - `pkg/business/audit_registry.go` — `AuditEventDefinition` gains the retention class, required the way `Durability` is. -- `pkg/business/retention.go` — partition drop keeps running for the Postgres - store. Under a swap value, tier expiry belongs to the warehouse (TTL or table +- `pkg/business/retention.go` — partition drop keeps running for `audit_events`, + and is also how the history migration (item 7) removes copied partitions. + Under a swap value, tier expiry belongs to the warehouse (TTL or table expiration) and to the archive's lock. -- `store/migrations` — the queue's storage. `audit_events` is append-only: the - `audit_events_no_delete` and `audit_events_no_update` triggers reject row - deletes, and the tenant and control-plane roles hold `SELECT` and `INSERT` - only. Delete-after-acknowledge needs a delete path those rules do not grant; - providing one must not weaken the append-only rule for the `postgres` - default. The tenant policy (`audit_events_tenant`, in `1_baseline.up.sql`) - keeps guarding queue rows; it no longer guards stored history, which the - service-side scope of item 4 does. -- The envelope's **result** has no column today: some types carry it in their - name (`saas.approval.denied`), others in an `outcome` payload field. The - events table needs it as a column, so each type has to say where its result - comes from. +- `store/migrations` — a new queue table, under the same tenant row-level + security as `audit_events` (`audit_events_tenant`, in `1_baseline.up.sql`), + with delete granted to the relay. `audit_events` itself is untouched: its + `audit_events_no_delete` and `audit_events_no_update` triggers and its + `SELECT`/`INSERT`-only grants stay as they are. Stored history in the + warehouse is guarded by the service-side scope of item 4, not by these + policies. **What stays:** @@ -236,6 +248,6 @@ The sink shape itself: archive bucket — and a relay whose lag needs watching. - Each adapter must stay at parity with the Postgres implementation; the conformance suite is what holds it there. -- Left to the implementation: whether the relay writes the archive copy as part - of the acknowledged batch, or the archive is filled from the warehouse - afterwards. +- A batch whose warehouse append fails after its archive object was written is + retried whole, and a locked object cannot be replaced — so the archive is + at-least-once like the relay and can hold an event more than once. From 0c1953c7f7f3fd73b5cdb45fd51fb18895907f2b Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Fri, 2 Oct 2026 10:48:25 -0400 Subject: [PATCH 3/9] docs(adr): settle duplicate archive objects in ADR 0009 Archive objects are named per batch, and a retried batch writes a new object, so the archive may hold an event more than once. Every archive reader, including the history-copy verification and any compliance export, deduplicates by event id; the per-event hash makes duplicates provably identical. Conditional create-if-absent object names are recorded as a rejected alternative: they need the same batch composition on every retry, for no compliance gain. Co-Authored-By: Claude Opus 5.5 --- module/docs/adr/0009-audit-store-swap.md | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/module/docs/adr/0009-audit-store-swap.md b/module/docs/adr/0009-audit-store-swap.md index 90abd729d..d0950f7d8 100644 --- a/module/docs/adr/0009-audit-store-swap.md +++ b/module/docs/adr/0009-audit-store-swap.md @@ -129,9 +129,15 @@ Postgres transaction. A **locked object-storage archive** holds the compliance copy, one object per relay batch: full for security-class events, content-free (envelope plus - hash) for content-class events. Locked means write-once under a retention lock the operator cannot - shorten: GCS Bucket Lock, S3 Object Lock, Azure Blob immutability policies, - or an object-lock-capable store on-premises. + hash) for content-class events. Locked means write-once under a retention + lock the operator cannot shorten: GCS Bucket Lock, S3 Object Lock, Azure Blob + immutability policies, or an object-lock-capable store on-premises. + + Archive objects are named per batch. A retried batch writes a new object, so + the archive may hold an event more than once. Every archive reader — + including the history-copy verification (item 7) and any compliance export — + deduplicates by event id, and the per-event hash makes duplicates provably + identical. Windows are deployment configuration, not code. @@ -173,6 +179,7 @@ The queue between the transaction and the warehouse (item 2): | Change data capture on the Postgres WAL | Still needs the Postgres row in order to capture it, and replication slots plus a connector are heavier to run than a relay over a table. | | Queue rows in `audit_events`, with the no-delete trigger relaxed for them | Weakens the append-only guarantee on the table every `postgres` and `both` deployment keeps its history in, to serve a mode those deployments do not use. | | Fill the archive from the warehouse later, on a schedule | Opens a window in which the compliance copy lags the store of record, and adds a second job to watch. | +| Conditional create-if-absent archive object names, so a retry cannot duplicate | Needs the same batch composition on every retry — more moving parts for no compliance gain. | Where the adapters live (item 3): @@ -248,6 +255,3 @@ The sink shape itself: archive bucket — and a relay whose lag needs watching. - Each adapter must stay at parity with the Postgres implementation; the conformance suite is what holds it there. -- A batch whose warehouse append fails after its archive object was written is - retried whole, and a locked object cannot be replaced — so the archive is - at-least-once like the relay and can hold an event more than once. From 510b7ae82017f229b768b76f3109595131ce13f2 Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Fri, 2 Oct 2026 19:20:20 -0400 Subject: [PATCH 4/9] docs(adr): settle warehouse duplicates, declared retention classes and swap settings in ADR 0009 The warehouse may hold an event more than once after a redelivery: BigQuery collapses a repeated insert id only briefly, and a relay that restarts between its writes and its delete delivers the batch again. Every read returns each event once by event id, as archive readers already do, and the conformance suite tests that instead of append idempotency. Committed write streams are rejected: stream state to carry across restarts for no compliance gain. A solution- or module-declared type states its retention class in its declaration; one that states none is content, any other value is refused, and a declared class only grows. A code-owned type still cannot register without one. A deployment selecting a swap value provides the warehouse settings, AUDIT_ARCHIVE_URL (its scheme picks the archive writer, gs:// for GCS), AUDIT_CONTENT_RETENTION_DAYS and AUDIT_DEPLOYMENT_ID. Co-Authored-By: Claude Opus 5.5 --- module/docs/adr/0009-audit-store-swap.md | 46 +++++++++++++++++++----- 1 file changed, 37 insertions(+), 9 deletions(-) diff --git a/module/docs/adr/0009-audit-store-swap.md b/module/docs/adr/0009-audit-store-swap.md index d0950f7d8..dc34378ea 100644 --- a/module/docs/adr/0009-audit-store-swap.md +++ b/module/docs/adr/0009-audit-store-swap.md @@ -56,6 +56,16 @@ Postgres transaction. default for every deployment that does not opt in. `both` — ADR 0006's tee — keeps its current behaviour, for compatibility. `external` stays refused. + A deployment that selects a swap value sets, beside `AUDIT_SINK`: the + warehouse (`AUDIT_BIGQUERY_PROJECT` and `AUDIT_BIGQUERY_DATASET` for + `bigquery`), `AUDIT_ARCHIVE_URL` — the locked archive as a URL whose scheme + picks the archive writer, `gs://` for GCS, any scheme without a + writer refused at startup — `AUDIT_CONTENT_RETENTION_DAYS` (the details + window of item 5) and `AUDIT_DEPLOYMENT_ID`, stamped on every record. A + missing setting fails startup. Credentials are the platform's ambient + identity (on Kubernetes, the pod's workload identity), never a key in + configuration. + 2. **Postgres keeps only the transactional queue.** `EmitTx` still writes inside the caller's transaction (and `Emit` inside its own, for the observational types of ADR 0003's durability amendment), but what it writes is a row in a @@ -74,8 +84,9 @@ Postgres transaction. A relay drains the queue: - in **batches**; - - **at-least-once, idempotent by event id** — a redelivered event leaves one - record in the warehouse; + - **at-least-once, keyed by event id** — a batch redelivered after a crash + may leave an event in the warehouse twice, as it may in the archive, and + every read returns each event once by event id (item 4); - **ordered per organization** (platform-level events form one more ordering key); - each batch is **appended to the warehouse and written as one object to the @@ -108,13 +119,27 @@ Postgres transaction. Adapter-native row policies (BigQuery row-level access policies, ClickHouse row policies) are defence in depth, not the boundary. -5. **Retention tiers.** Every registered event type — code-owned, or declared - by a solution or a composed module — declares a **retention class** in the - typed registry (ADR 0003): + Every read returns each event once by event id. The warehouse may hold an + event more than once after a redelivery (item 2): BigQuery collapses a + repeated streaming insert id only within a short window, and a relay that + restarts between its writes and its delete delivers the batch again. The + read half deduplicates exactly as every archive reader does (item 5), and + the per-event hash makes the copies provably identical. + +5. **Retention tiers.** Every registered event type has a **retention class** + in the typed registry (ADR 0003): - **security** — authentication, failed authentication, permission and role changes, admin actions, data exports, configuration changes; - **content** — everything else. + A code-owned type declares its class beside its durability, and cannot be + registered without one. A type declared by a solution (its manifest) or a + composed module (its declaration call) states its class in that + declaration; one that states none is **content**, and a value outside the + two is refused. A declared class only grows, as a `pii` mark does: a + re-declaration may raise `content` to `security` and is refused if it would + lower it, omission included. + The retention class is its own field. `Category`'s `security` value names a narrower set and is not reused for it. @@ -160,9 +185,9 @@ Postgres transaction. deleted and the append-only triggers stay as they are. From then on `audit_events` receives no new rows, and Postgres holds only the queue. -8. **Conformance.** Every adapter passes one shared test suite: append - idempotency (a redelivered batch leaves one record per event id), batch - append, list and aggregate parity with the Postgres implementation over the +8. **Conformance.** Every adapter passes one shared test suite: read + deduplication (after a batch is appended twice, every read returns each + event once by event id), batch append, list and aggregate parity with the Postgres implementation over the same fixture, refusal of a query without tenant scope, and retention-class routing (a content-class event's full details land only in the details table, and only its content-free form in the archive). @@ -180,6 +205,7 @@ The queue between the transaction and the warehouse (item 2): | Queue rows in `audit_events`, with the no-delete trigger relaxed for them | Weakens the append-only guarantee on the table every `postgres` and `both` deployment keeps its history in, to serve a mode those deployments do not use. | | Fill the archive from the warehouse later, on a schedule | Opens a window in which the compliance copy lags the store of record, and adds a second job to watch. | | Conditional create-if-absent archive object names, so a retry cannot duplicate | Needs the same batch composition on every retry — more moving parts for no compliance gain. | +| Committed write streams with offsets, so a redelivery cannot duplicate in the warehouse | Stream state the relay must carry across restarts and recover on every one — more machinery for no compliance gain, since every read already deduplicates by event id, as archive readers do. | Where the adapters live (item 3): @@ -222,7 +248,9 @@ The sink shape itself: Under a swap value it becomes a warehouse query for the latest actor ids, then a join to display names in the service from Postgres — outside that snapshot. - `pkg/business/audit_registry.go` — `AuditEventDefinition` gains the retention - class, required the way `Durability` is. + class, required the way `Durability` is. A declared type's class comes from + its declaration — the manifest event, or the module declaration message — + and is stored on its `audit_event_types` row, `content` when it states none. - `pkg/business/retention.go` — partition drop keeps running for `audit_events`, and is also how the history migration (item 7) removes copied partitions. Under a swap value, tier expiry belongs to the warehouse (TTL or table From 21feb102c3bba8c492463df12b30f397d8f5fc65 Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Sat, 3 Oct 2026 20:16:37 -0400 Subject: [PATCH 5/9] docs(adr): settle how ADR 0009 reads BigQuery The service that reads the store of record also appends to it, so it holds bigquery.tables.updateData; with bigquery.jobs.create beside it, that identity could run DELETE, UPDATE and MERGE. ADR 0009 now decides that BigQuery reads run no query job: they use the Storage Read API with a server-side row restriction (deployment, organization, time window, event types), and the service sorts, pages, deduplicates and aggregates. It names the read grants, and rejects a second, query-capable identity for reads. Co-Authored-By: Claude Opus 5.5 --- module/docs/adr/0009-audit-store-swap.md | 27 ++++++++++++++++++++++-- 1 file changed, 25 insertions(+), 2 deletions(-) diff --git a/module/docs/adr/0009-audit-store-swap.md b/module/docs/adr/0009-audit-store-swap.md index dc34378ea..614fa89b3 100644 --- a/module/docs/adr/0009-audit-store-swap.md +++ b/module/docs/adr/0009-audit-store-swap.md @@ -126,6 +126,22 @@ Postgres transaction. read half deduplicates exactly as every archive reader does (item 5), and the per-event hash makes the copies provably identical. + BigQuery reads run no query job. They go through the BigQuery Storage Read + API. Each read session carries a server-side row restriction: the + deployment, the caller's organization (unless the read is the platform + read), the time window, so partitions outside it are pruned, and the event + types when the read names them. The service then sorts, pages, + deduplicates, joins content details and aggregates the rows it streams back, + with the semantics of the Postgres reads. + + The reason is the grants. The service that reads is the service that + appends, so it holds `bigquery.tables.updateData`. With + `bigquery.jobs.create` beside it, that identity could run DELETE, UPDATE and + MERGE against the store of record. So the identity is granted + `bigquery.datasets.get`, `bigquery.tables.get`, `bigquery.tables.getData`, + `bigquery.tables.list`, and `bigquery.readsessions.create`, `.getData` and + `.update`, and no job permission at all. + 5. **Retention tiers.** Every registered event type has a **retention class** in the typed registry (ADR 0003): - **security** — authentication, failed authentication, permission and role @@ -214,6 +230,12 @@ Where the adapters live (item 3): | A separate warehouse service the kit calls | One more deployable per environment, a second queue in front of it, a second permission check, and every read becomes a network hop. Worth it only if producers or readers outside the kit needed the store directly; none do. | | One managed warehouse per cloud (BigQuery, Redshift, Synapse) | One adapter per cloud to build and hold at parity. ClickHouse covers every non-Google environment, on-premises included, with one adapter. | +How BigQuery is read (item 4): + +| Alternative | Rejected because | +| --- | --- | +| A second, query-capable identity for reads | Adds a second principal to grant, rotate and audit. And the process that appends would then hold job creation through it too: the DML risk the append-only invariant exists to prevent. | + The sink shape itself: | Alternative | Rejected because | @@ -245,8 +267,9 @@ The sink shape itself: registry in the service instead. - `pkg/infra/postgres_readable_sources.go` — `LatestSourceSyncRequests` joins `audit_events` to `principals` in one query, inside the source-read snapshot. - Under a swap value it becomes a warehouse query for the latest actor ids, then - a join to display names in the service from Postgres — outside that snapshot. + Under a swap value it becomes a warehouse read of each source's latest actor + id — outside that snapshot — then a join to display names from Postgres, + inside it. - `pkg/business/audit_registry.go` — `AuditEventDefinition` gains the retention class, required the way `Durability` is. A declared type's class comes from its declaration — the manifest event, or the module declaration message — From 25fc80d422c4ef89da2c2fd366adf62e9fd65b79 Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Sat, 3 Oct 2026 21:24:19 -0400 Subject: [PATCH 6/9] docs(adr): record ClickHouse audit settings --- module/docs/adr/0009-audit-store-swap.md | 27 ++++++++++++++---------- 1 file changed, 16 insertions(+), 11 deletions(-) diff --git a/module/docs/adr/0009-audit-store-swap.md b/module/docs/adr/0009-audit-store-swap.md index 614fa89b3..d86918dcb 100644 --- a/module/docs/adr/0009-audit-store-swap.md +++ b/module/docs/adr/0009-audit-store-swap.md @@ -58,13 +58,16 @@ Postgres transaction. A deployment that selects a swap value sets, beside `AUDIT_SINK`: the warehouse (`AUDIT_BIGQUERY_PROJECT` and `AUDIT_BIGQUERY_DATASET` for - `bigquery`), `AUDIT_ARCHIVE_URL` — the locked archive as a URL whose scheme - picks the archive writer, `gs://` for GCS, any scheme without a - writer refused at startup — `AUDIT_CONTENT_RETENTION_DAYS` (the details - window of item 5) and `AUDIT_DEPLOYMENT_ID`, stamped on every record. A - missing setting fails startup. Credentials are the platform's ambient - identity (on Kubernetes, the pod's workload identity), never a key in - configuration. + `bigquery`; `AUDIT_CLICKHOUSE_DSN` and `AUDIT_EVENTS_RETENTION_DAYS` for + `clickhouse`), `AUDIT_ARCHIVE_URL` — the locked archive as a URL whose scheme + picks the archive writer (`gs://` for GCS or `s3://` for S3; + any scheme without a writer is refused at startup) — + `AUDIT_CONTENT_RETENTION_DAYS` (the details window of item 5) and + `AUDIT_DEPLOYMENT_ID`, stamped on every record. ClickHouse may additionally + set `AUDIT_CLICKHOUSE_CLUSTER` for replicated tables. The events retention + must be at least the content-detail window. A missing required setting + fails startup. Credentials are the platform's ambient identity (on + Kubernetes, the pod's workload identity), never a key in configuration. 2. **Postgres keeps only the transactional queue.** `EmitTx` still writes inside the caller's transaction (and `Emit` inside its own, for the observational @@ -171,8 +174,9 @@ Postgres transaction. A **locked object-storage archive** holds the compliance copy, one object per relay batch: full for security-class events, content-free (envelope plus hash) for content-class events. Locked means write-once under a retention - lock the operator cannot shorten: GCS Bucket Lock, S3 Object Lock, Azure Blob - immutability policies, or an object-lock-capable store on-premises. + lock the operator cannot shorten. The first two writers target GCS Bucket + Lock and S3 Object Lock. Azure Blob immutability policies still need a + writer; an on-premises S3-compatible lock must be qualified before use. Archive objects are named per batch. A retried batch writes a new object, so the archive may hold an event more than once. Every archive reader — @@ -186,8 +190,9 @@ Postgres transaction. - Inserts are batched — the relay's batches, `async_insert`, or both. A single-row insert creates a data part per insert and ends in too-many-parts errors. - - Tables are replicated (`Replicated*MergeTree`), coordinated by ClickHouse - Keeper. + - A single-node deployment uses MergeTree. When + `AUDIT_CLICKHOUSE_CLUSTER` is set, tables use `ReplicatedMergeTree` with + ClickHouse Keeper; the cluster must exist before the adapter starts. - Tables are partitioned by month, and table TTL expires each tier at its window. - Growing past one shard is a planned operation, not an automatic one — or From ac94469b8b214abed019f05edf2171c0d3dca347 Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Mon, 5 Oct 2026 16:33:28 -0400 Subject: [PATCH 7/9] docs(adr): correct ADR 0009 settings, retention and relay ordering Settings: AUDIT_CLICKHOUSE_DSN carries the user and password, so it is a secret delivered as one rather than plain configuration; an s3:// archive needs AWS_REGION (and AWS_ENDPOINT_URL_S3 for a compatible store); the relay tuning settings AUDIT_RELAY_BATCH_SIZE and AUDIT_RELAY_MAX_WAIT are listed; the events/content retention ordering is stated as ClickHouse-only. Retention: on BigQuery the events table has no kit-set expiry; on ClickHouse AUDIT_EVENTS_RETENTION_DAYS sets a TTL. The locked archive is the compliance record, security events in full and content events as envelope plus SHA-256. The security class lists what the registry declares and names it the source of truth. Design: the archive is written before the warehouse; history removal drops an explicit list of verified partitions in one locking, re-counting database function; a permanently rejected row moves to a quarantine table; the queue is monitored under every sink; BigQuery reads hold bounded memory; the role-catalog importer resolves the sink as the service does. Cross-references: the tee is option C of ADR 0006's Options considered, not B; the sink loader is audit_sink.go and auditsink.Load, not work.go. Co-Authored-By: Claude Sonnet 5.5 --- .../0006-audit-sink-and-retention-tiers.md | 2 +- module/docs/adr/0009-audit-store-swap.md | 170 ++++++++++++------ 2 files changed, 121 insertions(+), 51 deletions(-) diff --git a/module/docs/adr/0006-audit-sink-and-retention-tiers.md b/module/docs/adr/0006-audit-sink-and-retention-tiers.md index b081f354a..59f998675 100644 --- a/module/docs/adr/0006-audit-sink-and-retention-tiers.md +++ b/module/docs/adr/0006-audit-sink-and-retention-tiers.md @@ -1,6 +1,6 @@ # ADR 0006: Audit sink and retention tiers — a swappable compliance backend, Postgres as the atomic hot source of truth -- Status: Proposed; superseded in part by [ADR 0009](./0009-audit-store-swap.md) — the sink/tee decision ("a tee, not a swap": Decision item 2, option B) +- Status: Proposed; superseded in part by [ADR 0009](./0009-audit-store-swap.md) — the sink/tee decision ("a tee, not a swap": Decision item 2; option C of Options considered) - Date: 2026-09-02 - Task: prepares the audit performance/size and compliance workstreams (see #430). Refines — for the **sink and retention dimension only** — the diff --git a/module/docs/adr/0009-audit-store-swap.md b/module/docs/adr/0009-audit-store-swap.md index d86918dcb..4b7b4dea8 100644 --- a/module/docs/adr/0009-audit-store-swap.md +++ b/module/docs/adr/0009-audit-store-swap.md @@ -3,8 +3,9 @@ - Status: Proposed - Date: 2026-10-02 - Supersedes, in part: [ADR 0006](./0006-audit-sink-and-retention-tiers.md) — its - sink decision, "a tee, not a swap" (Decision item 2, option B), and for a - deployment that selects a swap value, its retention tiers (Decision item 3). + sink decision, "a tee, not a swap" (Decision item 2; option C of its Options + considered), and for a deployment that selects a swap value, its retention + tiers (Decision item 3). ADR 0006's admission policy and sizing stand. - Refines: [ADR 0003](./0003-typed-audit-event-registry.md) — "analytics stays in Postgres" now holds for the `postgres` default only. The registry and the @@ -56,18 +57,35 @@ Postgres transaction. default for every deployment that does not opt in. `both` — ADR 0006's tee — keeps its current behaviour, for compatibility. `external` stays refused. - A deployment that selects a swap value sets, beside `AUDIT_SINK`: the - warehouse (`AUDIT_BIGQUERY_PROJECT` and `AUDIT_BIGQUERY_DATASET` for - `bigquery`; `AUDIT_CLICKHOUSE_DSN` and `AUDIT_EVENTS_RETENTION_DAYS` for - `clickhouse`), `AUDIT_ARCHIVE_URL` — the locked archive as a URL whose scheme - picks the archive writer (`gs://` for GCS or `s3://` for S3; - any scheme without a writer is refused at startup) — - `AUDIT_CONTENT_RETENTION_DAYS` (the details window of item 5) and - `AUDIT_DEPLOYMENT_ID`, stamped on every record. ClickHouse may additionally - set `AUDIT_CLICKHOUSE_CLUSTER` for replicated tables. The events retention - must be at least the content-detail window. A missing required setting - fails startup. Credentials are the platform's ambient identity (on - Kubernetes, the pod's workload identity), never a key in configuration. + A deployment that selects a swap value sets, beside `AUDIT_SINK`: + - the warehouse: `AUDIT_BIGQUERY_PROJECT` and `AUDIT_BIGQUERY_DATASET` for + `bigquery`; `AUDIT_CLICKHOUSE_DSN` and `AUDIT_EVENTS_RETENTION_DAYS` for + `clickhouse`, which may additionally set `AUDIT_CLICKHOUSE_CLUSTER` for + replicated tables; + - `AUDIT_ARCHIVE_URL` — the locked archive as a URL whose scheme picks the + archive writer (`gs://` for GCS or `s3://` for S3; any + scheme without a writer is refused at startup); + - `AUDIT_CONTENT_RETENTION_DAYS` — the details window of item 5; + - `AUDIT_DEPLOYMENT_ID`, stamped on every record; + - optionally, the relay's tuning: `AUDIT_RELAY_BATCH_SIZE` (the most events + one delivery carries; default 500) and `AUDIT_RELAY_MAX_WAIT` (how long a + partial batch waits for more events before it is delivered anyway; + default 5s, at most 60s). + + A missing required setting fails startup. `AUDIT_EVENTS_RETENTION_DAYS` is a + ClickHouse setting only — BigQuery's events table has no expiry the kit sets + (item 5) — and ClickHouse requires it to be at least the content-detail + window, since details must not outlive their event. + + Credentials never go in configuration. BigQuery and GCS use the platform's + ambient identity (on Kubernetes, the pod's workload identity). An `s3://` + archive uses the AWS SDK's default credential chain and also needs + `AWS_REGION` — startup refuses without it — and `AWS_ENDPOINT_URL_S3` when + the bucket is in an S3-compatible store rather than S3 itself. The one + credential that has no ambient identity is `AUDIT_CLICKHOUSE_DSN`, which + carries the ClickHouse user and password: it is a secret, delivered to the + service the way the deployment delivers its other secrets, never plain + configuration, and never logged or echoed in an error. 2. **Postgres keeps only the transactional queue.** `EmitTx` still writes inside the caller's transaction (and `Emit` inside its own, for the observational @@ -92,11 +110,28 @@ Postgres transaction. every read returns each event once by event id (item 4); - **ordered per organization** (platform-level events form one more ordering key); - - each batch is **appended to the warehouse and written as one object to the - locked archive** (item 5) — the archive copy is written within the batch, - not filled in later; + - each batch is **written as one object to the locked archive first, then + appended to the warehouse** (item 5) — the archive copy is written within + the batch, not filled in later, and a failure between the two writes + repeats the archive write, not the warehouse's. The relay remembers which + writes the batch at the head of the queue has had acknowledged, so an + outage of one side does not write that batch to the other side again; only + a restart forgets, which is the redelivery above; - the batch's queue rows are **deleted only after both writes are - acknowledged**. + acknowledged**; + - a row the warehouse **permanently rejects** — one that fails alone while + the other rows of the same attempt succeed — does not stall the queue. It + moves, in one transaction, to an `audit_event_quarantine` table beside the + queue and is counted in `saas.audit_queue.quarantined`. The kit never + deletes a quarantined row; replaying one is future work. + + The queue is monitored: `saas.audit_queue.depth` and + `saas.audit_queue.oldest_age` report how much is waiting and for how long. + When the queue cannot be read they go absent rather than staying at their + last value, and each failed read counts in + `saas.audit_queue.snapshot_errors`. The monitoring runs under every sink, not + only a swap value: under a non-warehouse sink a non-empty queue — rows left + by an earlier swap, which nothing drains — is logged at startup. The idempotency reservation (`audit_event_idempotency`) stays in Postgres beside the queue row, as it is today. @@ -135,7 +170,8 @@ Postgres transaction. read), the time window, so partitions outside it are pruned, and the event types when the read names them. The service then sorts, pages, deduplicates, joins content details and aggregates the rows it streams back, - with the semantics of the Postgres reads. + with the semantics of the Postgres reads, in bounded memory whatever the size + of the history in the window. The reason is the grants. The service that reads is the service that appends, so it holds `bigquery.tables.updateData`. With @@ -147,9 +183,15 @@ Postgres transaction. 5. **Retention tiers.** Every registered event type has a **retention class** in the typed registry (ADR 0003): - - **security** — authentication, failed authentication, permission and role - changes, admin actions, data exports, configuration changes; - - **content** — everything else. + - **security** — the record of who could get in and what they were allowed to + do: authentication (including failures), credentials issued and revoked, + permission, role, membership, sharing and delegation changes, + administrative and operator actions, data leaving the platform (exports, + replays), and configuration changes. The registry's declarations + (`pkg/business/audit_registry.go`) are the source of truth for which types + are in this class; + - **content** — everything else: what happened to a tenant's own content and + operations. A code-owned type declares its class beside its durability, and cannot be registered without one. A type declared by a solution (its manifest) or a @@ -165,18 +207,26 @@ Postgres transaction. The warehouse keeps two tables: - an **events** table holding the envelope — actor, organization, event type, target id, timestamp — plus a hash of the details, and the full - details as well for security-class types; kept for the compliance window. - An event's outcome stays where it is today: in the event type's name - (`saas.approval.denied`) or in the payload's `outcome` field; + details as well for security-class types. An event's outcome stays where it + is today: in the event type's name (`saas.approval.denied`) or in the + payload's `outcome` field. How long the table keeps a row depends on the + warehouse. On ClickHouse, `AUDIT_EVENTS_RETENTION_DAYS` sets the table's + TTL. On BigQuery the kit sets no expiry on the events table; a deployment + that wants one sets a table or partition expiration on its dataset; - a **details** table holding the full details of content-class types, kept - for a shorter, configurable window. - - A **locked object-storage archive** holds the compliance copy, one object - per relay batch: full for security-class events, content-free (envelope plus - hash) for content-class events. Locked means write-once under a retention - lock the operator cannot shorten. The first two writers target GCS Bucket - Lock and S3 Object Lock. Azure Blob immutability policies still need a - writer; an on-premises S3-compatible lock must be qualified before use. + for the shorter content window, `AUDIT_CONTENT_RETENTION_DAYS` — a + partition expiration on BigQuery, a table TTL on ClickHouse. + + The **compliance record is the locked object-storage archive**, one object + per relay batch: the full event for security-class types, and for + content-class types the envelope plus the SHA-256 of the details, never the + details themselves. Locked means write-once under a retention lock the + operator cannot shorten, set to the deployment's compliance window (ADR + 0006's sizing assumes seven years). The warehouse's events table serves + reads; it is not what a compliance regime relies on. The first two writers + target GCS Bucket Lock and S3 Object Lock. Azure Blob immutability policies + still need a writer; an on-premises S3-compatible lock must be qualified + before use. Archive objects are named per batch. A retried batch writes a new object, so the archive may hold an event more than once. Every archive reader — @@ -184,7 +234,8 @@ Postgres transaction. deduplicates by event id, and the per-event hash makes duplicates provably identical. - Windows are deployment configuration, not code. + The content window is deployment configuration, not code; so is the lock + period of the archive bucket, and the events TTL on ClickHouse. 6. **ClickHouse operating requirements.** - Inserts are batched — the relay's batches, `async_insert`, or both. A @@ -193,18 +244,24 @@ Postgres transaction. - A single-node deployment uses MergeTree. When `AUDIT_CLICKHOUSE_CLUSTER` is set, tables use `ReplicatedMergeTree` with ClickHouse Keeper; the cluster must exist before the adapter starts. - - Tables are partitioned by month, and table TTL expires each tier at its - window. + - Tables are partitioned by month, and a table TTL expires each table at its + window: `AUDIT_EVENTS_RETENTION_DAYS` for events, + `AUDIT_CONTENT_RETENTION_DAYS` for details. - Growing past one shard is a planned operation, not an automatic one — or the deployment uses a managed ClickHouse with shared storage. 7. **History migration.** Switching a deployment from `postgres` to a swap value includes a one-time copy of the existing `audit_events` rows into the warehouse and the archive, verified per organization by row counts and by - per-row hashes. Once verified, the copied monthly partitions are detached - and dropped — the mechanism `RunRetention` already uses — so no row is - deleted and the append-only triggers stay as they are. From then on - `audit_events` receives no new rows, and Postgres holds only the queue. + per-row hashes. Once verified, the copied monthly partitions are removed + with `DROP TABLE`, as `RunRetention` drops expired ones, so no row is + deleted and the append-only triggers stay as they are. The removal is one + database function (migration 20) that takes an explicit list of the + verified partitions, locks them, re-counts each against the count that was + verified, and refuses on any mismatch — a partition that gained a row since + verification is never dropped. `--through`, the cutoff of the copy, accepts + completed months only. From then on `audit_events` receives no new rows, and + Postgres holds only the queue. 8. **Conformance.** Every adapter passes one shared test suite: read deduplication (after a batch is appended twice, every read returns each @@ -251,10 +308,15 @@ The sink shape itself: **What changes in code** (paths under `module/services/accounts/code/`): -- `work.go` — `configuredAuditSink` admits `bigquery` and `clickhouse`, builds - the adapter and the archive writer, and starts the relay. The `external` - refusal stays; its message, which today says Postgres is the source of truth - under every value, is reworded. +- `audit_sink.go` and `pkg/auditstore/auditsink` — `auditsink.Load` admits + `bigquery` and `clickhouse`, reads and validates their settings, and + `auditsink.Open` builds the adapter and the archive writer; `audit_sink.go` + starts the relay on them. The `external` refusal stays; its message, which + today says Postgres is the source of truth under every value, is reworded. + The one-time history copy and the role-catalog importer resolve the sink + through the same `auditsink.Load` as the service, so a deployment configures + the swap once and neither tool writes to a different store than the service + reads; the importer never falls back silently to `audit_events`. - `pkg/business/audit.go` — under a swap value, the emitter's write inserts into the queue table in place of `audit_events`, for org-scoped and platform-level entries alike. `EmitTx`'s transaction contract, `VerifyAuditWiring` and the @@ -279,13 +341,18 @@ The sink shape itself: class, required the way `Durability` is. A declared type's class comes from its declaration — the manifest event, or the module declaration message — and is stored on its `audit_event_types` row, `content` when it states none. -- `pkg/business/retention.go` — partition drop keeps running for `audit_events`, - and is also how the history migration (item 7) removes copied partitions. - Under a swap value, tier expiry belongs to the warehouse (TTL or table - expiration) and to the archive's lock. +- `pkg/business/retention.go` — partition drop keeps running for `audit_events`; + the history migration (item 7) removes copied partitions with `DROP TABLE` + as well, through its own verified-partition function. Under a swap value, + expiry belongs to the warehouse where it sets one — the details table on + both, the events table on ClickHouse only — and to the archive's lock. - `store/migrations` — a new queue table, under the same tenant row-level security as `audit_events` (`audit_events_tenant`, in `1_baseline.up.sql`), - with delete granted to the relay. `audit_events` itself is untouched: its + with delete granted to the relay; its down migration refuses to drop a + non-empty queue, since the rows in it are audit records no warehouse holds + yet. A second migration (21) adds the `audit_event_quarantine` table of item + 2, and migration 20 the partition-removal function of item 7. `audit_events` + itself is untouched: its `audit_events_no_delete` and `audit_events_no_update` triggers and its `SELECT`/`INSERT`-only grants stay as they are. Stored history in the warehouse is guarded by the service-side scope of item 4, not by these @@ -308,6 +375,9 @@ The sink shape itself: - Under a swap value, reads trail writes: an event is listed once the relay has delivered it, not when its mutation commits. - A deployment that opts in runs two more systems — the warehouse and the - archive bucket — and a relay whose lag needs watching. + archive bucket — and a relay whose lag and quarantine table need watching. +- On BigQuery the events table is never expired by the kit and grows until the + deployment sets an expiry on its dataset; the locked archive, not that table, + is the compliance record. - Each adapter must stay at parity with the Postgres implementation; the conformance suite is what holds it there. From c20b198e5d4ea0af2ad90cbfe5f993c9e05fe441 Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Tue, 6 Oct 2026 15:17:28 -0400 Subject: [PATCH 8/9] docs(adr): match ADR 0009 to the relay, grants and read bounds as built State what the audit relay, the BigQuery and ClickHouse adapters and the operator tools do, so the decision record and the code agree: - Delivery is not ordered. The queue gate is transaction visibility, not sequence number; both stores key a record by event id and reads sort by event time. - The relay keeps per-row acknowledgements. A row is quarantined only when the store reports it refused that row for good, or its details cannot be serialized; every other failure is retried with backoff. The quarantine reason starts with its class, and replay is not built. - The monitored series include the quarantine size, and all three go absent when the queue cannot be read. - The BigQuery identity also holds tables.create and tables.updateData, and the events table is refused at startup if its partitions expire. - ClickHouse on a cluster uses insert_quorum=auto, insert_quorum_parallel=0 and select_sequential_consistency=1; async_insert must not be enabled; a two-replica cluster stops appending with one replica down. - The role-catalog importer reads AUDIT_SINK through RequireMode and refuses an unset value, so a deployment on the postgres default sets it. The resolver records through RecordTx rather than raw SQL. - What is bounded is a BigQuery read window (WindowBytes, 64 MiB): a read with one instant over the budget fails, an export over 32 MiB is refused and has no time range, and the history copy reads a day outside the budget. - Migrations 18 to 21 are listed with their grants and down guards. Co-Authored-By: Claude Sonnet 5.5 --- module/docs/adr/0009-audit-store-swap.md | 218 +++++++++++++++-------- 1 file changed, 146 insertions(+), 72 deletions(-) diff --git a/module/docs/adr/0009-audit-store-swap.md b/module/docs/adr/0009-audit-store-swap.md index 4b7b4dea8..1251d51a9 100644 --- a/module/docs/adr/0009-audit-store-swap.md +++ b/module/docs/adr/0009-audit-store-swap.md @@ -41,8 +41,8 @@ for years, locked, and queried heavily: the job platform admits a global-scope job only from the privileged worker pool, which cannot enqueue on the caller's transaction, so `work.go` warns that these stay Postgres-only. It also lacks the rows the auth resolver - writes with raw SQL (`pkg/auth/pg/resolver.go`), which never pass through the - emitter. Postgres lacks everything older than its window. Every reader has + records on its resolution transaction (`pkg/auth/pg/resolver.go`), which carry + no tee job. Postgres lacks everything older than its window. Every reader has to know which copy answers which question. The requirement ADR 0006 refused to trade away still stands: no state change @@ -68,9 +68,14 @@ Postgres transaction. - `AUDIT_CONTENT_RETENTION_DAYS` — the details window of item 5; - `AUDIT_DEPLOYMENT_ID`, stamped on every record; - optionally, the relay's tuning: `AUDIT_RELAY_BATCH_SIZE` (the most events - one delivery carries; default 500) and `AUDIT_RELAY_MAX_WAIT` (how long a - partial batch waits for more events before it is delivered anyway; - default 5s, at most 60s). + one delivery carries; default 500, at most 5000) and `AUDIT_RELAY_MAX_WAIT` + (how long a partial batch waits for more events before it is delivered + anyway; default 5s, at most 60s). + + The role-catalog importer, a deploy step that records audit events apart from + the service, is the one process with no default: it refuses to run when + `AUDIT_SINK` is unset or blank. A deployment on the `postgres` default + therefore sets `AUDIT_SINK=postgres` explicitly wherever the importer runs. A missing required setting fails startup. `AUDIT_EVENTS_RETENTION_DAYS` is a ClickHouse setting only — BigQuery's events table has no expiry the kit sets @@ -103,32 +108,55 @@ Postgres transaction. is what keeps the append-only guarantee intact for every `postgres` and `both` deployment: no trigger is relaxed to make room for a delete. - A relay drains the queue: - - in **batches**; + A relay drains the queue, one relay at a time under a lease: + - in **batches**, taken oldest queue row first; - **at-least-once, keyed by event id** — a batch redelivered after a crash may leave an event in the warehouse twice, as it may in the archive, and every read returns each event once by event id (item 4); - - **ordered per organization** (platform-level events form one more ordering - key); + - **in no guaranteed order.** The relay reads a row only when its writing + transaction took its id before the oldest transaction still running, so it + never reads a row that has not committed. That gate is transaction + visibility, not sequence number: a row can be delivered ahead of a lower + sequence number whose transaction has yet to commit. Nothing depends on + delivery order: both stores key a record by event id, and every read sorts + by event time; - each batch is **written as one object to the locked archive first, then appended to the warehouse** (item 5) — the archive copy is written within the batch, not filled in later, and a failure between the two writes - repeats the archive write, not the warehouse's. The relay remembers which - writes the batch at the head of the queue has had acknowledged, so an - outage of one side does not write that batch to the other side again; only - a restart forgets, which is the redelivery above; + repeats the archive write, not the warehouse's. The relay keeps, for each + queued row, which of the two writes has acknowledged it and what it decided + about it, so an outage of one side does not write that row to the other side + again; only a restart forgets, which is the redelivery above; - the batch's queue rows are **deleted only after both writes are acknowledged**; - - a row the warehouse **permanently rejects** — one that fails alone while - the other rows of the same attempt succeed — does not stall the queue. It - moves, in one transaction, to an `audit_event_quarantine` table beside the - queue and is counted in `saas.audit_queue.quarantined`. The kit never - deletes a quarantined row; replaying one is future work. - - The queue is monitored: `saas.audit_queue.depth` and - `saas.audit_queue.oldest_age` report how much is waiting and for how long. - When the queue cannot be read they go absent rather than staying at their - last value, and each failed read counts in + - a row is **set aside only on the store's own word.** The relay quarantines + a row only when the store reports that it permanently refused that row + (`business.PermanentRowRejection`), or when the row's details cannot be + serialized. BigQuery reports a row it lists as `invalid` in a streaming + insert, a row too large for any request, and a value that cannot be + encoded; ClickHouse reports a value its driver cannot encode, and a row + refused with an error code that says the row's own content is malformed, + found by splitting the refused block down to single rows. Every other + failure — a transport or server error, a throttle, a quota, a timeout, a + failed quorum, an error the store does not recognize — is retried with + backoff and sets nothing aside, and the relay infers nothing from how often a + row failed or from what else the store accepted. After a write that names + refused rows, the relay sets those aside and writes the rest again; + - a set-aside row does not stall the queue. It moves, whole, in the same + transaction that deletes the delivered rows, to an `audit_event_quarantine` + table beside the queue, and is counted in `saas.audit_queue.quarantined`. + The reason kept with it starts with a class: `warehouse_rejected_archived:` + (the store refused the row; the archive already holds a copy, and so does the + quarantine) or `unserializable_not_archived:` (the details cannot be + serialized, so the archive's line for the event cannot be written and the + quarantine holds the only copy). The kit never deletes a quarantined row; + replaying one is not built. + + The queue is monitored: `saas.audit_queue.depth`, `saas.audit_queue.oldest_age` + and `saas.audit_queue.quarantined` report how much is waiting, for how long, + and how many rows are set aside. When the queue cannot be read — the latest + read failed, or none succeeded within twice the read interval — the three go + absent rather than staying at their last value, and each failed read counts in `saas.audit_queue.snapshot_errors`. The monitoring runs under every sink, not only a swap value: under a non-warehouse sink a non-empty queue — rows left by an earlier swap, which nothing drains — is logged at startup. @@ -140,8 +168,8 @@ Postgres transaction. one `AuditStore` interface carrying both halves: - **write** — append a batch; - **read** — list (keyset-paginated, as `QueryAuditLog` is), aggregate (as - `AggregateAuditLog`), export, and the readable-source queries - (`LatestSourceSyncRequests`). + `AggregateAuditLog`), export, and the readable-source query + (`LatestSourceSyncEvents`, which feeds `LatestSourceSyncRequests`). Two adapters ship: **BigQuery** (managed, Google Cloud) and **ClickHouse** (self-hosted or managed, any cloud or on-premises). The existing Postgres @@ -165,19 +193,38 @@ Postgres transaction. the per-event hash makes the copies provably identical. BigQuery reads run no query job. They go through the BigQuery Storage Read - API. Each read session carries a server-side row restriction: the - deployment, the caller's organization (unless the read is the platform - read), the time window, so partitions outside it are pruned, and the event - types when the read names them. The service then sorts, pages, + API. Each read session carries a server-side row restriction: the deployment, + the caller's organization (unless the read is the platform read), the time + window, so partitions outside it are pruned, and the envelope filters the read + names — event types, actor, resource, client. The service then sorts, pages, deduplicates, joins content details and aggregates the rows it streams back, - with the semantics of the Postgres reads, in bounded memory whatever the size - of the history in the window. + with the semantics of the Postgres reads. + + What is bounded is a read window, not a read. A read walks time in windows, + newest first, and counts what one window holds in memory — the content details + it joins and, for a read that must see every event of the window before it can + answer (an aggregation, an export), the events and the set that deduplicates + them — against a byte budget, `ReadConfig.WindowBytes` (64 MiB by default; the + service exposes no setting for it). A window that passes the budget is read + again as two halves. What outlives a window is the answer: a page, an + aggregation's buckets, an export's events. Three limits remain: + - A read whose events at a single instant alone pass the budget cannot be + split further and fails (`ErrReadTooDense`). + - An export is the one answer as large as the history it matches. One export + gathers at most 32 MiB of event text from either warehouse + (`AuditExportMaxBytes`) and is refused with `ResourceExhausted` past it. The + export request carries no time range, so an actor or an event type is all + that narrows it. + - The history copy (item 7) reads and holds a whole window of the store at a + time outside this budget. The reason is the grants. The service that reads is the service that appends, so it holds `bigquery.tables.updateData`. With `bigquery.jobs.create` beside it, that identity could run DELETE, UPDATE and MERGE against the store of record. So the identity is granted - `bigquery.datasets.get`, `bigquery.tables.get`, `bigquery.tables.getData`, + `bigquery.datasets.get`, `bigquery.tables.create` (the adapter creates its two + tables when they are missing), `bigquery.tables.get`, + `bigquery.tables.updateData` (the append), `bigquery.tables.getData`, `bigquery.tables.list`, and `bigquery.readsessions.create`, `.getData` and `.update`, and no job permission at all. @@ -211,8 +258,9 @@ Postgres transaction. is today: in the event type's name (`saas.approval.denied`) or in the payload's `outcome` field. How long the table keeps a row depends on the warehouse. On ClickHouse, `AUDIT_EVENTS_RETENTION_DAYS` sets the table's - TTL. On BigQuery the kit sets no expiry on the events table; a deployment - that wants one sets a table or partition expiration on its dataset; + TTL. On BigQuery the kit creates the events table with no partition + expiration, and refuses at startup an events table whose partitions expire, + since its identity holds no `bigquery.tables.update` to change one; - a **details** table holding the full details of content-class types, kept for the shorter content window, `AUDIT_CONTENT_RETENTION_DAYS` — a partition expiration on BigQuery, a table TTL on ClickHouse. @@ -228,8 +276,9 @@ Postgres transaction. still need a writer; an on-premises S3-compatible lock must be qualified before use. - Archive objects are named per batch. A retried batch writes a new object, so - the archive may hold an event more than once. Every archive reader — + Archive objects are named per batch and created only if absent. A batch + delivered again after a restart is composed anew and written as a new object, + so the archive may hold an event more than once. Every archive reader — including the history-copy verification (item 7) and any compliance export — deduplicates by event id, and the per-event hash makes duplicates provably identical. @@ -238,12 +287,28 @@ Postgres transaction. period of the archive bucket, and the events TTL on ClickHouse. 6. **ClickHouse operating requirements.** - - Inserts are batched — the relay's batches, `async_insert`, or both. A - single-row insert creates a data part per insert and ends in - too-many-parts errors. - - A single-node deployment uses MergeTree. When - `AUDIT_CLICKHOUSE_CLUSTER` is set, tables use `ReplicatedMergeTree` with - ClickHouse Keeper; the cluster must exist before the adapter starts. + - Inserts are batched: one insert per table per relay batch. A single-row + insert creates a data part per insert and ends in too-many-parts errors. + `async_insert` is not used and must not be enabled for the adapter's user or + profile: the relay deletes a queue row on the insert's acknowledgement, and + an insert acknowledged before its data is stored would let the relay delete + the only copy. + - A single-node deployment uses MergeTree and sends neither the quorum nor + the sequential-read setting below. When `AUDIT_CLICKHOUSE_CLUSTER` is set, + tables are created `ON CLUSTER` as `ReplicatedMergeTree` with ClickHouse + Keeper; the cluster must exist before the adapter starts. Every insert then + carries `insert_quorum=auto` (a majority of the replicas must hold it), + `insert_quorum_parallel=0` (one quorum insert at a time) and a 60-second + `insert_quorum_timeout`, and every read carries + `select_sequential_consistency=1`, so a read is served only by a replica that + holds every acknowledged insert; a replica that has not caught up fails the + read, which the caller retries. A majority of two replicas is both of them, + so on a two-replica cluster one replica down stops appends: the insert fails + at the quorum timeout, the relay retries with backoff, and the queue grows + until the replica returns. + - The adapter's ClickHouse user holds `CREATE TABLE`, `INSERT` and `SELECT` on + the database and no `ALTER`, `DELETE`, `UPDATE`, `TRUNCATE` or `DROP`, so it + cannot rewrite the store of record. - Tables are partitioned by month, and a table TTL expires each table at its window: `AUDIT_EVENTS_RETENTION_DAYS` for events, `AUDIT_CONTENT_RETENTION_DAYS` for details. @@ -259,16 +324,19 @@ Postgres transaction. database function (migration 20) that takes an explicit list of the verified partitions, locks them, re-counts each against the count that was verified, and refuses on any mismatch — a partition that gained a row since - verification is never dropped. `--through`, the cutoff of the copy, accepts - completed months only. From then on `audit_events` receives no new rows, and - Postgres holds only the queue. + verification is never dropped. `-through`, the cutoff of the copy, accepts + completed months (UTC) only. The copy and its read-back hold one UTC day of + the deployment's events in memory at a time, outside the read budget of + item 4. From then on `audit_events` receives no new rows, and Postgres holds + only the queue. 8. **Conformance.** Every adapter passes one shared test suite: read deduplication (after a batch is appended twice, every read returns each - event once by event id), batch append, list and aggregate parity with the Postgres implementation over the - same fixture, refusal of a query without tenant scope, and retention-class - routing (a content-class event's full details land only in the details table, - and only its content-free form in the archive). + event once by event id), batch append, list, aggregate, export and + readable-source parity with the Postgres implementation over the same + fixture, refusal of a query without tenant scope, and retention-class routing + (a content-class event's full details land only in the details table, and + only its content-free form in the archive). ## Alternatives considered @@ -282,7 +350,7 @@ The queue between the transaction and the warehouse (item 2): | Change data capture on the Postgres WAL | Still needs the Postgres row in order to capture it, and replication slots plus a connector are heavier to run than a relay over a table. | | Queue rows in `audit_events`, with the no-delete trigger relaxed for them | Weakens the append-only guarantee on the table every `postgres` and `both` deployment keeps its history in, to serve a mode those deployments do not use. | | Fill the archive from the warehouse later, on a schedule | Opens a window in which the compliance copy lags the store of record, and adds a second job to watch. | -| Conditional create-if-absent archive object names, so a retry cannot duplicate | Needs the same batch composition on every retry — more moving parts for no compliance gain. | +| Archive object names fixed by a batch's content, so a redelivery after a restart cannot duplicate | Needs the same batch composition on every redelivery — more moving parts for no compliance gain. | | Committed write streams with offsets, so a redelivery cannot duplicate in the warehouse | Stream state the relay must carry across restarts and recover on every one — more machinery for no compliance gain, since every read already deduplicates by event id, as archive readers do. | Where the adapters live (item 3): @@ -313,10 +381,13 @@ The sink shape itself: `auditsink.Open` builds the adapter and the archive writer; `audit_sink.go` starts the relay on them. The `external` refusal stays; its message, which today says Postgres is the source of truth under every value, is reworded. - The one-time history copy and the role-catalog importer resolve the sink - through the same `auditsink.Load` as the service, so a deployment configures - the swap once and neither tool writes to a different store than the service - reads; the importer never falls back silently to `audit_events`. + The one-time history copy resolves the sink through the same + `auditsink.Load` as the service, so a deployment configures the swap once and + the copy never writes to a different store than the service reads. The + role-catalog importer reads `AUDIT_SINK` alone, through + `auditsink.RequireMode`, and refuses an unset or blank value rather than + defaulting to `postgres`: under a swap value a default would write its events + to `audit_events`, where nothing reads them. - `pkg/business/audit.go` — under a swap value, the emitter's write inserts into the queue table in place of `audit_events`, for org-scoped and platform-level entries alike. `EmitTx`'s transaction contract, `VerifyAuditWiring` and the @@ -324,10 +395,11 @@ The sink shape itself: reader's grant check and the aggregate in one tenant transaction today; under a swap value it becomes grant check in Postgres, then the aggregate in the warehouse — two steps, no shared snapshot. -- `pkg/auth/pg/resolver.go` — `emitSsoJitAudit` and `emitUserRegistered` insert - into `audit_events` with raw SQL on the resolution transaction, bypassing the - emitter. Under a swap value they must go through the queue, or the swap loses - them. +- `pkg/auth/pg/resolver.go` — `emitSsoJitAudit` and `emitUserRegistered` record + their events on the resolution transaction through the emitter's `RecordTx` + rather than inserting into `audit_events` themselves. `RecordTx` writes the + record alone — the `audit_events` row, or under a swap value the queue row — + with no domain event and no tee job, so the swap does not lose them. - `pkg/infra/postgres_audit.go` — `QueryAuditLog` and `AggregateAuditLog` become the Postgres implementation of `AuditStore`. The aggregate's category dimension reads `audit_event_types` in SQL; an adapter takes it from the @@ -346,17 +418,19 @@ The sink shape itself: as well, through its own verified-partition function. Under a swap value, expiry belongs to the warehouse where it sets one — the details table on both, the events table on ClickHouse only — and to the archive's lock. -- `store/migrations` — a new queue table, under the same tenant row-level - security as `audit_events` (`audit_events_tenant`, in `1_baseline.up.sql`), - with delete granted to the relay; its down migration refuses to drop a - non-empty queue, since the rows in it are audit records no warehouse holds - yet. A second migration (21) adds the `audit_event_quarantine` table of item - 2, and migration 20 the partition-removal function of item 7. `audit_events` - itself is untouched: its - `audit_events_no_delete` and `audit_events_no_update` triggers and its - `SELECT`/`INSERT`-only grants stay as they are. Stored history in the - warehouse is guarded by the service-side scope of item 4, not by these - policies. +- `store/migrations` — migration 18 adds the queue table, under the same tenant + row-level security as `audit_events` (`audit_events_tenant`, in + `1_baseline.up.sql`), with insert granted to the writers and select and delete + to the relay; its down migration refuses to drop a non-empty queue, since the + rows in it are audit records no warehouse holds yet. Migration 19 adds the + retention class to `audit_event_types`. Migration 20 adds the + verified-partition removal function of item 7. Migration 21 adds the + `audit_event_quarantine` table of item 2, which only the relay's role can read + or insert into; its down migration refuses to drop a non-empty quarantine. + `audit_events` itself is untouched: its `audit_events_no_delete` and + `audit_events_no_update` triggers and its `SELECT`/`INSERT`-only grants stay as + they are. Stored history in the warehouse is guarded by the service-side scope + of item 4, not by these policies. **What stays:** @@ -376,8 +450,8 @@ The sink shape itself: delivered it, not when its mutation commits. - A deployment that opts in runs two more systems — the warehouse and the archive bucket — and a relay whose lag and quarantine table need watching. -- On BigQuery the events table is never expired by the kit and grows until the - deployment sets an expiry on its dataset; the locked archive, not that table, - is the compliance record. +- On BigQuery the kit never expires the events table, and refuses to start on + one whose partitions expire, so the table grows with the history; the locked + archive, not that table, is the compliance record. - Each adapter must stay at parity with the Postgres implementation; the conformance suite is what holds it there. From 7806c048eccdd6e501a6b047461432a10125c2ca Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Tue, 6 Oct 2026 15:35:29 -0400 Subject: [PATCH 9/9] docs(adr): renumber the audit migrations ADR 0009 cites to 19 through 22 Main took migration 18 for resource indicators, so the four audit migrations the store change adds are now 19 (queue), 20 (retention class), 21 (verified-partition removal) and 22 (quarantine). ADR 0009 names them by number in the history-migration item and in the store/migrations paragraph; both now match the numbers the migrations will land under. Co-Authored-By: Claude Sonnet 5.5 --- module/docs/adr/0009-audit-store-swap.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/module/docs/adr/0009-audit-store-swap.md b/module/docs/adr/0009-audit-store-swap.md index 1251d51a9..391d72424 100644 --- a/module/docs/adr/0009-audit-store-swap.md +++ b/module/docs/adr/0009-audit-store-swap.md @@ -321,7 +321,7 @@ Postgres transaction. per-row hashes. Once verified, the copied monthly partitions are removed with `DROP TABLE`, as `RunRetention` drops expired ones, so no row is deleted and the append-only triggers stay as they are. The removal is one - database function (migration 20) that takes an explicit list of the + database function (migration 21) that takes an explicit list of the verified partitions, locks them, re-counts each against the count that was verified, and refuses on any mismatch — a partition that gained a row since verification is never dropped. `-through`, the cutoff of the copy, accepts @@ -418,13 +418,13 @@ The sink shape itself: as well, through its own verified-partition function. Under a swap value, expiry belongs to the warehouse where it sets one — the details table on both, the events table on ClickHouse only — and to the archive's lock. -- `store/migrations` — migration 18 adds the queue table, under the same tenant +- `store/migrations` — migration 19 adds the queue table, under the same tenant row-level security as `audit_events` (`audit_events_tenant`, in `1_baseline.up.sql`), with insert granted to the writers and select and delete to the relay; its down migration refuses to drop a non-empty queue, since the - rows in it are audit records no warehouse holds yet. Migration 19 adds the - retention class to `audit_event_types`. Migration 20 adds the - verified-partition removal function of item 7. Migration 21 adds the + rows in it are audit records no warehouse holds yet. Migration 20 adds the + retention class to `audit_event_types`. Migration 21 adds the + verified-partition removal function of item 7. Migration 22 adds the `audit_event_quarantine` table of item 2, which only the relay's role can read or insert into; its down migration refuses to drop a non-empty quarantine. `audit_events` itself is untouched: its `audit_events_no_delete` and