From a1b81d7e4fb8b60796baa0772aa4727c81f99598 Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Fri, 2 Oct 2026 10:44:13 -0400 Subject: [PATCH 01/69] docs(adr): make the audit store swappable to a warehouse (ADR 0009) ADR 0006 kept Postgres as the only audit store of record and fed an external sink as a tee. That leaves years of audit history in the operational database, writes no locked archive, keeps dashboard and export load on Postgres, and gives two incomplete copies: the tee skips platform-level events and the auth resolver's raw-SQL inserts. ADR 0009 adds swap values to AUDIT_SINK (bigquery, clickhouse). Under a swap value the warehouse is the store of record and Postgres keeps only a transactional queue row, committed in the caller's transaction and drained in batches by a relay (at-least-once, idempotent by event id, ordered per organization, deleted after the adapter acknowledges). Adapters live in the accounts service behind one read/write interface, reads move with writes under service-enforced tenant scope, and each event type declares a retention class that routes it to warehouse tiers and a locked object-storage archive. postgres stays the default and both is unchanged. ADR 0006 is marked superseded in part (its sink/tee decision); ADR 0003 gets a pointer that its in-Postgres analytics stance now holds for the postgres default only. Docs only. Co-Authored-By: Claude Opus 5.5 --- .../adr/0003-typed-audit-event-registry.md | 1 + .../0006-audit-sink-and-retention-tiers.md | 2 +- module/docs/adr/0009-audit-store-swap.md | 241 ++++++++++++++++++ 3 files changed, 243 insertions(+), 1 deletion(-) create mode 100644 module/docs/adr/0009-audit-store-swap.md diff --git a/module/docs/adr/0003-typed-audit-event-registry.md b/module/docs/adr/0003-typed-audit-event-registry.md index 1d50d2050..1b9b4b7a8 100644 --- a/module/docs/adr/0003-typed-audit-event-registry.md +++ b/module/docs/adr/0003-typed-audit-event-registry.md @@ -1,6 +1,7 @@ # ADR 0003: Typed audit-event registry — single-table STI, a code-owned catalog, and in-Postgres analytics - Status: Proposed +- Refined by: [ADR 0009](./0009-audit-store-swap.md) — a deployment may select a warehouse as its audit store, so "analytics stays in Postgres" holds for the `postgres` default only - Date: 2026-08-18 - Task: closes the review-and-planning deliverable of #170. Implementation is deferred to follow-up tickets that this ADR authorizes but does not open. diff --git a/module/docs/adr/0006-audit-sink-and-retention-tiers.md b/module/docs/adr/0006-audit-sink-and-retention-tiers.md index deb73c329..b081f354a 100644 --- a/module/docs/adr/0006-audit-sink-and-retention-tiers.md +++ b/module/docs/adr/0006-audit-sink-and-retention-tiers.md @@ -1,6 +1,6 @@ # ADR 0006: Audit sink and retention tiers — a swappable compliance backend, Postgres as the atomic hot source of truth -- Status: Proposed +- Status: Proposed; superseded in part by [ADR 0009](./0009-audit-store-swap.md) — the sink/tee decision ("a tee, not a swap": Decision item 2, option B) - Date: 2026-09-02 - Task: prepares the audit performance/size and compliance workstreams (see #430). Refines — for the **sink and retention dimension only** — the diff --git a/module/docs/adr/0009-audit-store-swap.md b/module/docs/adr/0009-audit-store-swap.md new file mode 100644 index 000000000..ae36551e0 --- /dev/null +++ b/module/docs/adr/0009-audit-store-swap.md @@ -0,0 +1,241 @@ +# ADR 0009: Audit store swap — a warehouse as the audit store of record, Postgres as its transactional queue + +- Status: Proposed +- Date: 2026-10-02 +- Supersedes, in part: [ADR 0006](./0006-audit-sink-and-retention-tiers.md) — its + sink decision, "a tee, not a swap" (Decision item 2, option B), and for a + deployment that selects a swap value, its retention tiers (Decision item 3). + ADR 0006's admission policy and sizing stand. +- Refines: [ADR 0003](./0003-typed-audit-event-registry.md) — "analytics stays in + Postgres" now holds for the `postgres` default only. The registry and the + single write path are unchanged. +- It does not change code. + +## Context + +ADR 0006 kept Postgres as the only audit store of record and added an opt-in +tee: with `AUDIT_SINK=both`, the emitter enqueues an export job beside each +org-scoped audit row, and a worker posts it to an operator-configured HTTP +endpoint (`HTTPAuditSink`), one event per request. That keeps the write-path +guarantee, and it remains the right answer for a deployment that only needs a +copy somewhere else. It is not enough for a deployment whose audit must be kept +for years, locked, and queried heavily: + +1. **Long retention.** Postgres holds every audit row for the + `data_retention_policies` window (365 days by default; `RunRetention` drops + whole monthly partitions). A compliance window of several years keeps that + history in the operational database — ADR 0006's sizing puts seven years at + ~2 TB for a small deployment and ~22 TB for a medium one. The tee does not + shrink Postgres; it adds a copy. +2. **A locked archive.** A compliance regime may require a write-once copy that + no operator can alter or delete before its window ends. The kit writes no + such copy; the tee's destination is whatever sits behind an HTTP endpoint. +3. **Query load.** The activity list, dashboard aggregation + (`AggregateAuditLog`: grouped counts over time buckets and payload + predicates), exports and readable-source queries all run against the + operational database. Dashboards make audit an analytical workload on the + instance that serves every mutation. +4. **One store to query.** Under the tee there are two copies and neither is + complete. The warehouse copy lacks platform-level events (no organization): + the job platform admits a global-scope job only from the privileged worker + pool, which cannot enqueue on the caller's transaction, so `work.go` warns + that these stay Postgres-only. It also lacks the rows the auth resolver + writes with raw SQL (`pkg/auth/pg/resolver.go`), which never pass through the + emitter. Postgres lacks everything older than its window. Every reader has + to know which copy answers which question. + +The requirement ADR 0006 refused to trade away still stands: no state change +commits without its compliance record, and no external system can join a +Postgres transaction. + +## Decision + +1. **Swap, not tee.** `AUDIT_SINK` gains two swap values, `bigquery` and + `clickhouse`. Under a swap value the selected warehouse is the audit store of + record and Postgres no longer keeps audit history. `postgres` stays the + default for every deployment that does not opt in. `both` — ADR 0006's tee — + keeps its current behaviour, for compatibility. `external` stays refused. + +2. **Postgres keeps only the transactional queue.** `EmitTx` still writes inside + the caller's transaction (and `Emit` inside its own, for the observational + types of ADR 0003's durability amendment), but what it writes is a + short-lived queue row, not a permanent audit row. The queue row commits + together with the domain event that webhook delivery fans out from, exactly + as the audit row does today, so "no state change without its compliance + record" holds unchanged. Every audit write goes through the queue — including + platform-level (no-organization, control-plane) events, which the swap must + not leave Postgres-only the way the tee does. + + A relay drains the queue to the selected adapter: + - in **batches**; + - **at-least-once, idempotent by event id** — a redelivered event leaves one + record; + - **ordered per organization** (platform-level events form one more ordering + key); + - a queue row is **deleted only after the adapter acknowledges** the write. + + The idempotency reservation (`audit_event_idempotency`) stays in Postgres + beside the queue row, as it is today. + +3. **Adapters live inside the kit.** They are Go, in the accounts service, behind + one `AuditStore` interface carrying both halves: + - **write** — append a batch; + - **read** — list (keyset-paginated, as `QueryAuditLog` is), aggregate (as + `AggregateAuditLog`), export, and the readable-source queries + (`LatestSourceSyncRequests`). + + Two adapters ship: **BigQuery** (managed, Google Cloud) and **ClickHouse** + (self-hosted or managed, any cloud or on-premises). The existing Postgres + reads become the third implementation of the same interface and the + reference the other two are measured against. + +4. **Reads move with writes.** Under a swap value the activity list, dashboard + aggregation, exports and readable-source queries read from the adapter. + Postgres row-level security does not reach a warehouse, so tenant scope is + enforced in the service on every query: an organization's read always + carries its organization predicate, only the platform-admin path may span + organizations, and the adapter refuses a query that arrives without a scope. + Adapter-native row policies (BigQuery row-level access policies, ClickHouse + row policies) are defence in depth, not the boundary. + +5. **Retention tiers.** Every registered event type — code-owned, or declared + by a solution or a composed module — declares a **retention class** in the + typed registry (ADR 0003): + - **security** — authentication, failed authentication, permission and role + changes, admin actions, data exports, configuration changes; + - **content** — everything else. + + The retention class is its own field. `Category`'s `security` value names a + narrower set and is not reused for it. + + The warehouse keeps two tables: + - an **events** table holding the envelope — actor, organization, event + type, target id, timestamp, result — plus a hash of the details, and the + full details as well for security-class types; kept for the compliance + window; + - a **details** table holding the full details of content-class types, kept + for a shorter, configurable window. + + A **locked object-storage archive** holds the compliance copy: full for + security-class events, content-free (envelope plus hash) for content-class + events. Locked means write-once under a retention lock the operator cannot + shorten: GCS Bucket Lock, S3 Object Lock, Azure Blob immutability policies, + or an object-lock-capable store on-premises. + + Windows are deployment configuration, not code. + +6. **ClickHouse operating requirements.** + - Inserts are batched — the relay's batches, `async_insert`, or both. A + single-row insert creates a data part per insert and ends in + too-many-parts errors. + - Tables are replicated (`Replicated*MergeTree`), coordinated by ClickHouse + Keeper. + - Tables are partitioned by month, and table TTL expires each tier at its + window. + - Growing past one shard is a planned operation, not an automatic one — or + the deployment uses a managed ClickHouse with shared storage. + +7. **History migration.** Switching a deployment from `postgres` to a swap value + includes a one-time copy of the existing audit rows into the adapter and the + archive, verified per organization by row counts and hashes. Afterwards the + Postgres audit table holds only queue rows. + +8. **Conformance.** Every adapter passes one shared test suite: append + idempotency (a redelivered batch leaves one record per event id), batch + append, list and aggregate parity with the Postgres implementation over the + same fixture, refusal of a query without tenant scope, and retention-class + routing (a content-class event's full details land only in the details table, + and only its content-free form in the archive). + +## Alternatives considered + +The queue between the transaction and the warehouse (item 2): + +| Alternative | Rejected because | +| --- | --- | +| Local disk spool in the accounts process | Crash gap between the transaction commit and the spool write: the mutation is committed and its record is nowhere durable. | +| Synchronous write to the warehouse in the request | A warehouse outage either fails the mutation or loses the event; there is no third outcome. | +| A message broker | The same dual write — commit, then publish — with the same gap, plus a new system to run. | +| Change data capture on the Postgres WAL | Still needs the Postgres row in order to capture it, and replication slots plus a connector are heavier to run than a relay over a table. | + +Where the adapters live (item 3): + +| Alternative | Rejected because | +| --- | --- | +| A separate warehouse service the kit calls | One more deployable per environment, a second queue in front of it, a second permission check, and every read becomes a network hop. Worth it only if producers or readers outside the kit needed the store directly; none do. | +| One managed warehouse per cloud (BigQuery, Redshift, Synapse) | One adapter per cloud to build and hold at parity. ClickHouse covers every non-Google environment, on-premises included, with one adapter. | + +The sink shape itself: + +| Alternative | Rejected because | +| --- | --- | +| Keep the tee as the only option (ADR 0006) | Fails the four needs under Context. It stays available as `both`. | + +## Consequences + +**What changes in code** (paths under `module/services/accounts/code/`): + +- `work.go` — `configuredAuditSink` admits `bigquery` and `clickhouse`, builds + the adapter and starts the relay. The `external` refusal stays; its message, + which today says Postgres is the source of truth under every value, is + reworded. +- `pkg/business/audit.go` — under a swap value, the emitter's write inserts a + queue row in place of the permanent audit row, for org-scoped and + platform-level entries alike. `EmitTx`'s transaction contract, + `VerifyAuditWiring` and the domain-event publish are unchanged. + `AggregateAuditLogForReader` runs the reader's grant check and the aggregate + in one tenant transaction today; under a swap value the check stays in + Postgres and the aggregate goes to the adapter, so the two no longer share a + snapshot. +- `pkg/auth/pg/resolver.go` — `emitSsoJitAudit` and `emitUserRegistered` insert + audit rows with raw SQL on the resolution transaction. They must write queue + rows too, or the swap loses them. +- `pkg/infra/postgres_audit.go` — `QueryAuditLog` and `AggregateAuditLog` become + the Postgres implementation of `AuditStore`. The aggregate's category + dimension reads `audit_event_types` in SQL; an adapter takes it from the + registry in the service instead. +- `pkg/infra/postgres_readable_sources.go` — `LatestSourceSyncRequests` joins + `audit_events` to `principals` in one query, inside the source-read snapshot. + Under a swap value it becomes two reads — actor ids from the adapter, display + names from Postgres — outside that snapshot. +- `pkg/business/audit_registry.go` — `AuditEventDefinition` gains the retention + class, required the way `Durability` is. +- `pkg/business/retention.go` — partition drop keeps running for the Postgres + store. Under a swap value, tier expiry belongs to the warehouse (TTL or table + expiration) and to the archive's lock. +- `store/migrations` — the queue's storage. `audit_events` is append-only: the + `audit_events_no_delete` and `audit_events_no_update` triggers reject row + deletes, and the tenant and control-plane roles hold `SELECT` and `INSERT` + only. Delete-after-acknowledge needs a delete path those rules do not grant; + providing one must not weaken the append-only rule for the `postgres` + default. The tenant policy (`audit_events_tenant`, in `1_baseline.up.sql`) + keeps guarding queue rows; it no longer guards stored history, which the + service-side scope of item 4 does. +- The envelope's **result** has no column today: some types carry it in their + name (`saas.approval.denied`), others in an `outcome` payload field. The + events table needs it as a column, so each type has to say where its result + comes from. + +**What stays:** + +- `postgres` is the default and behaves exactly as today: audit rows in Postgres, + read under row-level security, retention by partition drop. +- `both` behaves exactly as today (`pkg/business/audit_jobs.go`, + `pkg/business/audit_http_sink.go`), including its org-scoped-only coverage. +- The typed registry and its payload validation, the durability rule, the + idempotency reservation, and webhook fan-out. +- ADR 0006's admission policy and sizing. [ADR 0008](./0008-datasource-sync-actor-provenance.md)'s + rule that a sync's actor lives only in the audit trail — under a swap value, + that trail is the adapter. + +**Costs accepted:** + +- Under a swap value, reads trail writes: an event is listed once the relay has + delivered it, not when its mutation commits. +- A deployment that opts in runs two more systems — the warehouse and the + archive bucket — and a relay whose lag needs watching. +- Each adapter must stay at parity with the Postgres implementation; the + conformance suite is what holds it there. +- Left to the implementation: whether the relay writes the archive copy as part + of the acknowledged batch, or the archive is filled from the warehouse + afterwards. From eb417245146eaf9574fb6ee7ddea59fb67c5c0cd Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Fri, 2 Oct 2026 10:47:29 -0400 Subject: [PATCH 02/69] docs(adr): settle the audit queue, history removal and archive write in ADR 0009 The queue is its own short-lived table, not audit_events: audit_events keeps its append-only triggers and grants and receives no new rows under a swap value, so no trigger is relaxed for any postgres or both deployment. The history migration removes copied rows by detaching and dropping the verified monthly partitions, the mechanism RunRetention already uses. The relay writes each batch to the warehouse and as one object to the locked archive, and deletes the batch's queue rows only after both acknowledge. The envelope drops its result field: an event's outcome stays in its type name or its payload's outcome field. Relaxing the no-delete trigger and filling the archive from the warehouse later are recorded as rejected alternatives. Co-Authored-By: Claude Opus 5.5 --- module/docs/adr/0009-audit-store-swap.md | 110 +++++++++++++---------- 1 file changed, 61 insertions(+), 49 deletions(-) diff --git a/module/docs/adr/0009-audit-store-swap.md b/module/docs/adr/0009-audit-store-swap.md index ae36551e0..90abd729d 100644 --- a/module/docs/adr/0009-audit-store-swap.md +++ b/module/docs/adr/0009-audit-store-swap.md @@ -58,21 +58,31 @@ Postgres transaction. 2. **Postgres keeps only the transactional queue.** `EmitTx` still writes inside the caller's transaction (and `Emit` inside its own, for the observational - types of ADR 0003's durability amendment), but what it writes is a - short-lived queue row, not a permanent audit row. The queue row commits - together with the domain event that webhook delivery fans out from, exactly - as the audit row does today, so "no state change without its compliance - record" holds unchanged. Every audit write goes through the queue — including + types of ADR 0003's durability amendment), but what it writes is a row in a + **separate, short-lived queue table** — not a permanent row in + `audit_events`. The queue row commits together with the mutation and with + the domain event that webhook delivery fans out from, exactly as the audit + row does today, so "no state change without its compliance record" holds + unchanged. Every audit write goes through the queue — including platform-level (no-organization, control-plane) events, which the swap must not leave Postgres-only the way the tee does. - A relay drains the queue to the selected adapter: + `audit_events` keeps its append-only triggers and its grants unchanged; under + a swap value it simply receives no new rows. The queue being its own table + is what keeps the append-only guarantee intact for every `postgres` and + `both` deployment: no trigger is relaxed to make room for a delete. + + A relay drains the queue: - in **batches**; - **at-least-once, idempotent by event id** — a redelivered event leaves one - record; + record in the warehouse; - **ordered per organization** (platform-level events form one more ordering key); - - a queue row is **deleted only after the adapter acknowledges** the write. + - each batch is **appended to the warehouse and written as one object to the + locked archive** (item 5) — the archive copy is written within the batch, + not filled in later; + - the batch's queue rows are **deleted only after both writes are + acknowledged**. The idempotency reservation (`audit_event_idempotency`) stays in Postgres beside the queue row, as it is today. @@ -110,15 +120,16 @@ Postgres transaction. The warehouse keeps two tables: - an **events** table holding the envelope — actor, organization, event - type, target id, timestamp, result — plus a hash of the details, and the - full details as well for security-class types; kept for the compliance - window; + type, target id, timestamp — plus a hash of the details, and the full + details as well for security-class types; kept for the compliance window. + An event's outcome stays where it is today: in the event type's name + (`saas.approval.denied`) or in the payload's `outcome` field; - a **details** table holding the full details of content-class types, kept for a shorter, configurable window. - A **locked object-storage archive** holds the compliance copy: full for - security-class events, content-free (envelope plus hash) for content-class - events. Locked means write-once under a retention lock the operator cannot + A **locked object-storage archive** holds the compliance copy, one object + per relay batch: full for security-class events, content-free (envelope plus + hash) for content-class events. Locked means write-once under a retention lock the operator cannot shorten: GCS Bucket Lock, S3 Object Lock, Azure Blob immutability policies, or an object-lock-capable store on-premises. @@ -136,9 +147,12 @@ Postgres transaction. the deployment uses a managed ClickHouse with shared storage. 7. **History migration.** Switching a deployment from `postgres` to a swap value - includes a one-time copy of the existing audit rows into the adapter and the - archive, verified per organization by row counts and hashes. Afterwards the - Postgres audit table holds only queue rows. + includes a one-time copy of the existing `audit_events` rows into the + warehouse and the archive, verified per organization by row counts and by + per-row hashes. Once verified, the copied monthly partitions are detached + and dropped — the mechanism `RunRetention` already uses — so no row is + deleted and the append-only triggers stay as they are. From then on + `audit_events` receives no new rows, and Postgres holds only the queue. 8. **Conformance.** Every adapter passes one shared test suite: append idempotency (a redelivered batch leaves one record per event id), batch @@ -157,6 +171,8 @@ The queue between the transaction and the warehouse (item 2): | Synchronous write to the warehouse in the request | A warehouse outage either fails the mutation or loses the event; there is no third outcome. | | A message broker | The same dual write — commit, then publish — with the same gap, plus a new system to run. | | Change data capture on the Postgres WAL | Still needs the Postgres row in order to capture it, and replication slots plus a connector are heavier to run than a relay over a table. | +| Queue rows in `audit_events`, with the no-delete trigger relaxed for them | Weakens the append-only guarantee on the table every `postgres` and `both` deployment keeps its history in, to serve a mode those deployments do not use. | +| Fill the archive from the warehouse later, on a schedule | Opens a window in which the compliance copy lags the store of record, and adds a second job to watch. | Where the adapters live (item 3): @@ -176,45 +192,41 @@ The sink shape itself: **What changes in code** (paths under `module/services/accounts/code/`): - `work.go` — `configuredAuditSink` admits `bigquery` and `clickhouse`, builds - the adapter and starts the relay. The `external` refusal stays; its message, - which today says Postgres is the source of truth under every value, is - reworded. -- `pkg/business/audit.go` — under a swap value, the emitter's write inserts a - queue row in place of the permanent audit row, for org-scoped and - platform-level entries alike. `EmitTx`'s transaction contract, - `VerifyAuditWiring` and the domain-event publish are unchanged. - `AggregateAuditLogForReader` runs the reader's grant check and the aggregate - in one tenant transaction today; under a swap value the check stays in - Postgres and the aggregate goes to the adapter, so the two no longer share a - snapshot. + the adapter and the archive writer, and starts the relay. The `external` + refusal stays; its message, which today says Postgres is the source of truth + under every value, is reworded. +- `pkg/business/audit.go` — under a swap value, the emitter's write inserts into + the queue table in place of `audit_events`, for org-scoped and platform-level + entries alike. `EmitTx`'s transaction contract, `VerifyAuditWiring` and the + domain-event publish are unchanged. `AggregateAuditLogForReader` runs the + reader's grant check and the aggregate in one tenant transaction today; under + a swap value it becomes grant check in Postgres, then the aggregate in the + warehouse — two steps, no shared snapshot. - `pkg/auth/pg/resolver.go` — `emitSsoJitAudit` and `emitUserRegistered` insert - audit rows with raw SQL on the resolution transaction. They must write queue - rows too, or the swap loses them. + into `audit_events` with raw SQL on the resolution transaction, bypassing the + emitter. Under a swap value they must go through the queue, or the swap loses + them. - `pkg/infra/postgres_audit.go` — `QueryAuditLog` and `AggregateAuditLog` become the Postgres implementation of `AuditStore`. The aggregate's category dimension reads `audit_event_types` in SQL; an adapter takes it from the registry in the service instead. - `pkg/infra/postgres_readable_sources.go` — `LatestSourceSyncRequests` joins `audit_events` to `principals` in one query, inside the source-read snapshot. - Under a swap value it becomes two reads — actor ids from the adapter, display - names from Postgres — outside that snapshot. + Under a swap value it becomes a warehouse query for the latest actor ids, then + a join to display names in the service from Postgres — outside that snapshot. - `pkg/business/audit_registry.go` — `AuditEventDefinition` gains the retention class, required the way `Durability` is. -- `pkg/business/retention.go` — partition drop keeps running for the Postgres - store. Under a swap value, tier expiry belongs to the warehouse (TTL or table +- `pkg/business/retention.go` — partition drop keeps running for `audit_events`, + and is also how the history migration (item 7) removes copied partitions. + Under a swap value, tier expiry belongs to the warehouse (TTL or table expiration) and to the archive's lock. -- `store/migrations` — the queue's storage. `audit_events` is append-only: the - `audit_events_no_delete` and `audit_events_no_update` triggers reject row - deletes, and the tenant and control-plane roles hold `SELECT` and `INSERT` - only. Delete-after-acknowledge needs a delete path those rules do not grant; - providing one must not weaken the append-only rule for the `postgres` - default. The tenant policy (`audit_events_tenant`, in `1_baseline.up.sql`) - keeps guarding queue rows; it no longer guards stored history, which the - service-side scope of item 4 does. -- The envelope's **result** has no column today: some types carry it in their - name (`saas.approval.denied`), others in an `outcome` payload field. The - events table needs it as a column, so each type has to say where its result - comes from. +- `store/migrations` — a new queue table, under the same tenant row-level + security as `audit_events` (`audit_events_tenant`, in `1_baseline.up.sql`), + with delete granted to the relay. `audit_events` itself is untouched: its + `audit_events_no_delete` and `audit_events_no_update` triggers and its + `SELECT`/`INSERT`-only grants stay as they are. Stored history in the + warehouse is guarded by the service-side scope of item 4, not by these + policies. **What stays:** @@ -236,6 +248,6 @@ The sink shape itself: archive bucket — and a relay whose lag needs watching. - Each adapter must stay at parity with the Postgres implementation; the conformance suite is what holds it there. -- Left to the implementation: whether the relay writes the archive copy as part - of the acknowledged batch, or the archive is filled from the warehouse - afterwards. +- A batch whose warehouse append fails after its archive object was written is + retried whole, and a locked object cannot be replaced — so the archive is + at-least-once like the relay and can hold an event more than once. From 0c1953c7f7f3fd73b5cdb45fd51fb18895907f2b Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Fri, 2 Oct 2026 10:48:25 -0400 Subject: [PATCH 03/69] docs(adr): settle duplicate archive objects in ADR 0009 Archive objects are named per batch, and a retried batch writes a new object, so the archive may hold an event more than once. Every archive reader, including the history-copy verification and any compliance export, deduplicates by event id; the per-event hash makes duplicates provably identical. Conditional create-if-absent object names are recorded as a rejected alternative: they need the same batch composition on every retry, for no compliance gain. Co-Authored-By: Claude Opus 5.5 --- module/docs/adr/0009-audit-store-swap.md | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/module/docs/adr/0009-audit-store-swap.md b/module/docs/adr/0009-audit-store-swap.md index 90abd729d..d0950f7d8 100644 --- a/module/docs/adr/0009-audit-store-swap.md +++ b/module/docs/adr/0009-audit-store-swap.md @@ -129,9 +129,15 @@ Postgres transaction. A **locked object-storage archive** holds the compliance copy, one object per relay batch: full for security-class events, content-free (envelope plus - hash) for content-class events. Locked means write-once under a retention lock the operator cannot - shorten: GCS Bucket Lock, S3 Object Lock, Azure Blob immutability policies, - or an object-lock-capable store on-premises. + hash) for content-class events. Locked means write-once under a retention + lock the operator cannot shorten: GCS Bucket Lock, S3 Object Lock, Azure Blob + immutability policies, or an object-lock-capable store on-premises. + + Archive objects are named per batch. A retried batch writes a new object, so + the archive may hold an event more than once. Every archive reader — + including the history-copy verification (item 7) and any compliance export — + deduplicates by event id, and the per-event hash makes duplicates provably + identical. Windows are deployment configuration, not code. @@ -173,6 +179,7 @@ The queue between the transaction and the warehouse (item 2): | Change data capture on the Postgres WAL | Still needs the Postgres row in order to capture it, and replication slots plus a connector are heavier to run than a relay over a table. | | Queue rows in `audit_events`, with the no-delete trigger relaxed for them | Weakens the append-only guarantee on the table every `postgres` and `both` deployment keeps its history in, to serve a mode those deployments do not use. | | Fill the archive from the warehouse later, on a schedule | Opens a window in which the compliance copy lags the store of record, and adds a second job to watch. | +| Conditional create-if-absent archive object names, so a retry cannot duplicate | Needs the same batch composition on every retry — more moving parts for no compliance gain. | Where the adapters live (item 3): @@ -248,6 +255,3 @@ The sink shape itself: archive bucket — and a relay whose lag needs watching. - Each adapter must stay at parity with the Postgres implementation; the conformance suite is what holds it there. -- A batch whose warehouse append fails after its archive object was written is - retried whole, and a locked object cannot be replaced — so the archive is - at-least-once like the relay and can hold an event more than once. From 623ba87929ed771de05e2c0ed6943b0971115429 Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Fri, 2 Oct 2026 18:33:46 -0400 Subject: [PATCH 04/69] feat(audit): write the audit store swap to BigQuery through a transactional queue ADR 0009 (#996) makes a warehouse selectable as the audit store of record. This is its write path; reads move in a later change. AUDIT_SINK=bigquery is a new swap value. postgres (the default) and both behave as before; external stays refused with a reworded reason; unknown values, clickhouse included, still fail startup. The swap requires AUDIT_BIGQUERY_PROJECT, AUDIT_BIGQUERY_DATASET, AUDIT_ARCHIVE_BUCKET, AUDIT_DEPLOYMENT_ID and AUDIT_CONTENT_DETAIL_RETENTION_DAYS, reports every missing one at startup, and uses Application Default Credentials only. Under the swap the emitter writes each record into a new audit_event_queue table (store migration 17) on the caller's transaction, beside the domain event, instead of audit_events. The queue carries audit_events' tenant policy; writers may only insert, and only the relay, as app_job_worker, reads and deletes. audit_events, its triggers and grants are untouched. The relay drains the queue in batches (size and max wait configurable), in queue order under one lease, skipping rows whose transactions could still be overtaken. Each batch is written as one JSONL object to the GCS archive, then streamed to BigQuery with the event id as insert id, and its rows are deleted only after both acknowledge; a retry in the same process repeats only the write not yet acknowledged. Two tables, partitioned by day on occurred_at and clustered by org: events (envelope, SHA-256 of the canonical details, details for security-class types) and details (content-class details, expiring at the configured window). Every registered event type now declares a retention class; declared types are content. The resolver's raw-SQL registration and SSO provisioning rows and the role catalog import's rows now go through the emitter, so they reach the queue under the swap and are written exactly as before otherwise. A source scan holds every audit write to the emitter. Co-Authored-By: Claude Opus 5.5 --- AUTHZ.md | 6 + module/DATABASE_AUTHORITY.md | 4 +- module/FEATURES.md | 7 + module/services/accounts/code/audit_sink.go | 228 ++++++++ .../accounts/code/audit_writers_test.go | 80 +++ .../code/cmd/role-catalog-import/main.go | 30 +- module/services/accounts/code/go.mod | 39 +- module/services/accounts/code/go.sum | 87 ++- .../auditstore/bigquerystore/bigquerystore.go | 340 +++++++++++ .../bigquerystore/bigquerystore_test.go | 222 +++++++ .../auditstore/bigquerystore/emulator_test.go | 140 +++++ .../pkg/auditstore/gcsarchive/gcsarchive.go | 190 ++++++ .../auditstore/gcsarchive/gcsarchive_test.go | 134 +++++ .../accounts/code/pkg/auth/pg/resolver.go | 81 +-- .../code/pkg/auth/pg/resolver_audit_test.go | 135 +++++ .../code/pkg/auth/pg/resolver_sso_jit_test.go | 23 +- .../code/pkg/auth/pg/resolver_test.go | 99 ++-- .../accounts/code/pkg/business/audit.go | 69 ++- .../business/audit_durability_gate_test.go | 9 +- .../pkg/business/audit_queued_records_test.go | 163 ++++++ .../code/pkg/business/audit_registry.go | 507 ++++++++-------- .../accounts/code/pkg/business/audit_relay.go | 344 +++++++++++ .../code/pkg/business/audit_relay_test.go | 544 ++++++++++++++++++ .../code/pkg/business/audit_retention_test.go | 73 +++ .../accounts/code/pkg/business/audit_store.go | 161 ++++++ .../code/pkg/business/audit_store_test.go | 94 +++ .../pkg/business/audit_type_resolution.go | 11 + .../code/pkg/business/service_test.go | 1 + .../pkg/business/solution_audit_events.go | 6 + .../accounts/code/pkg/business/store.go | 4 + .../accounts/code/pkg/infra/postgres_audit.go | 40 +- .../code/pkg/infra/postgres_audit_queue.go | 170 ++++++ .../pkg/infra/postgres_audit_queue_test.go | 330 +++++++++++ .../code/pkg/infra/postgres_role_catalog.go | 29 +- .../pkg/infra/postgres_role_catalog_test.go | 28 +- .../pkg/infra/postgres_role_hardening_test.go | 7 + .../code/pkg/relationcatalog/catalog.go | 4 + module/services/accounts/code/work.go | 83 ++- .../services/accounts/code/work_audit_test.go | 115 +++- .../migrations/17_audit_event_queue.down.sql | 4 + .../migrations/17_audit_event_queue.up.sql | 88 +++ 41 files changed, 4282 insertions(+), 447 deletions(-) create mode 100644 module/services/accounts/code/audit_sink.go create mode 100644 module/services/accounts/code/audit_writers_test.go create mode 100644 module/services/accounts/code/pkg/auditstore/bigquerystore/bigquerystore.go create mode 100644 module/services/accounts/code/pkg/auditstore/bigquerystore/bigquerystore_test.go create mode 100644 module/services/accounts/code/pkg/auditstore/bigquerystore/emulator_test.go create mode 100644 module/services/accounts/code/pkg/auditstore/gcsarchive/gcsarchive.go create mode 100644 module/services/accounts/code/pkg/auditstore/gcsarchive/gcsarchive_test.go create mode 100644 module/services/accounts/code/pkg/auth/pg/resolver_audit_test.go create mode 100644 module/services/accounts/code/pkg/business/audit_queued_records_test.go create mode 100644 module/services/accounts/code/pkg/business/audit_relay.go create mode 100644 module/services/accounts/code/pkg/business/audit_relay_test.go create mode 100644 module/services/accounts/code/pkg/business/audit_retention_test.go create mode 100644 module/services/accounts/code/pkg/business/audit_store.go create mode 100644 module/services/accounts/code/pkg/business/audit_store_test.go create mode 100644 module/services/accounts/code/pkg/infra/postgres_audit_queue.go create mode 100644 module/services/accounts/code/pkg/infra/postgres_audit_queue_test.go create mode 100644 module/services/store/migrations/17_audit_event_queue.down.sql create mode 100644 module/services/store/migrations/17_audit_event_queue.up.sql diff --git a/AUTHZ.md b/AUTHZ.md index 777dec73d..ca49e94a8 100644 --- a/AUTHZ.md +++ b/AUTHZ.md @@ -908,6 +908,12 @@ go run ./cmd/role-catalog-import -catalog roles.json -database-url "$DATABASE_UR go run ./cmd/role-catalog-import -catalog roles.json -database-url "$DATABASE_URL" ``` +The importer records one audit event per changed role through the same emitter +the service uses, so it must run under the deployment's `AUDIT_SINK` +(`-audit-sink`, defaulting to `$AUDIT_SINK`): under a swap value (ADR 0009) its +events go to the transactional queue the accounts relay delivers, not to +`audit_events`. + Domain core is `pkg/rolecatalog` (parse + diff + deterministic plan, no DB); `pkg/infra` snapshots current state and applies the plan in one transaction. diff --git a/module/DATABASE_AUTHORITY.md b/module/DATABASE_AUTHORITY.md index 65e4e5c6f..23e22bb26 100644 --- a/module/DATABASE_AUTHORITY.md +++ b/module/DATABASE_AUTHORITY.md @@ -501,7 +501,7 @@ All tenant/user rows remain constrained by their forced RLS policies. | select, insert, update | `onboarding_progress`, `resource_follows`, `sessions`, `users`, `webauthn_ceremonies`, `webauthn_credentials` | | select, insert | `gdpr_requests` (accepted by request traffic, transitioned only by the leased privacy worker under the control plane) | | select, insert, update, delete | `mfa_backup_codes`, `mfa_devices`, `notifications`, `user_identities` | -| insert only | `client_authorization_codes`, `mfa_login_transactions` | +| insert only | `audit_event_queue` (read and deleted only by the audit relay, as `app_job_worker`), `client_authorization_codes`, `mfa_login_transactions` | | no tenant relation authority | `job_attempts`, `job_messages`, `job_state_transitions`, `magic_links`, `membership_integrity_findings` | Retention deletes and token-based pre-auth reads/updates use the audited @@ -526,7 +526,7 @@ executable inventory and this table in the same change. | Scope | Relations | Required database boundary | |---|---|---| | `global` | `audit_event_types`, `bootstrap_state`, `data_retention_policies`, `datasource_credential_budgets`, `email_templates`, `feature_flags`, `identity_providers`, `plan_entitlements`, `plans`, `platform_admins`, `solution_registrations` | No RLS; exact grants | -| `tenant` | `actor_chain_journal`, `actor_chain_revocations`, `api_keys`, `approval_decisions`, `approval_requests`, `audit_event_idempotency`, `audit_events`, `connector_credentials`, `dashboards`, `datasource_account_links`, `datasource_domains`, `datasource_group_bindings`, `datasource_sources`, `delegation_grants`, `domain_events`, `entitlement_overrides`, `github_app_installations`, `github_app_setups`, `installations`, `invitations`, `membership_integrity_findings`, `org_generic_settings`, `org_identity_providers`, `org_settings`, `organization_activations`, `organization_authorization_revisions`, `organization_members`, `organizations`, `principal_authorization_revisions`, `principals`, `record_shares`, `role_assignments`, `role_permissions`, `roles`, `scope_grants`, `scope_nodes`, `source_delegations`, `source_read_revisions`, `subscriptions`, `team_members`, `team_membership_quarantine`, `teams`, `usage_events`, `usage_totals`, `webhook_deliveries`, `webhook_subscriptions`, `work_context_replay` | Enabled and forced RLS with at least one policy | +| `tenant` | `actor_chain_journal`, `actor_chain_revocations`, `api_keys`, `approval_decisions`, `approval_requests`, `audit_event_idempotency`, `audit_event_queue`, `audit_events`, `connector_credentials`, `dashboards`, `datasource_account_links`, `datasource_domains`, `datasource_group_bindings`, `datasource_sources`, `delegation_grants`, `domain_events`, `entitlement_overrides`, `github_app_installations`, `github_app_setups`, `installations`, `invitations`, `membership_integrity_findings`, `org_generic_settings`, `org_identity_providers`, `org_settings`, `organization_activations`, `organization_authorization_revisions`, `organization_members`, `organizations`, `principal_authorization_revisions`, `principals`, `record_shares`, `role_assignments`, `role_permissions`, `roles`, `scope_grants`, `scope_nodes`, `source_delegations`, `source_read_revisions`, `subscriptions`, `team_members`, `team_membership_quarantine`, `teams`, `usage_events`, `usage_totals`, `webhook_deliveries`, `webhook_subscriptions`, `work_context_replay` | Enabled and forced RLS with at least one policy | | `user` | `client_authorization_codes`, `gdpr_requests`, `mfa_backup_codes`, `mfa_devices`, `mfa_login_transactions`, `notifications`, `onboarding_progress`, `resource_follows`, `sessions`, `user_consent_events`, `user_consent_preferences`, `user_identities`, `users`, `webauthn_ceremonies`, `webauthn_credentials` | Enabled and forced RLS with at least one policy | | `pre_auth` | `magic_links`, `waitlist_entries` | Enabled and forced RLS; fail-closed request policy, accessed only by the control-plane role | | `job` | `job_messages` | Enabled and forced RLS with at least one policy; no request relation grant — function-only scoped enqueue plus exact job-worker grants | diff --git a/module/FEATURES.md b/module/FEATURES.md index 7753e98bf..deabe5fcd 100644 --- a/module/FEATURES.md +++ b/module/FEATURES.md @@ -398,6 +398,7 @@ see `JOBS.md` for the exact boundary and sequencing. | Export (JSON/CSV) | ✅ | `audit_export.go` | | Impersonation tracking | ✅ | Records both real actor + viewed-as user | | Replay / event sourcing| ❌ | Audit log is read-only history; not used to reconstruct state | +| Warehouse store of record | 🟡 | `AUDIT_SINK=bigquery` (ADR 0009): each record commits to a Postgres queue on the mutation's transaction and a relay delivers it to BigQuery and a locked GCS archive. Writes only — the activity list, aggregates and exports still read Postgres until reads move; ClickHouse is not built | ### Admin panel @@ -653,6 +654,12 @@ Environment variables consumed by the api: | `POSTHOG_PROJECT_ID` | Positive PostHog project ID; required only in PostHog mode | | `POSTHOG_HOST` | Explicit HTTPS or local capture origin | | `POSTHOG_API_HOST` | Separate PostHog management/deletion origin | +| `AUDIT_SINK` | `postgres` (default), `both` (tee to `AUDIT_EXTERNAL_URL`, optional `AUDIT_EXTERNAL_TOKEN`), or `bigquery` (swap the audit store of record, ADR 0009); `external` is refused | +| `AUDIT_BIGQUERY_PROJECT`, `AUDIT_BIGQUERY_DATASET` | Required under `bigquery`: the store of record. The dataset must exist; the `audit_events` and `audit_event_details` tables are created if missing. Credentials are Application Default Credentials only | +| `AUDIT_ARCHIVE_BUCKET` | Required under `bigquery`: the GCS bucket, under a retention lock the deployment sets, that receives one object per relay batch | +| `AUDIT_DEPLOYMENT_ID` | Required under `bigquery`: stamped on every record and the archive path | +| `AUDIT_CONTENT_DETAIL_RETENTION_DAYS` | Required under `bigquery`: days a content-class event's full details are kept (the details table's partition expiration) | +| `AUDIT_RELAY_BATCH_SIZE`, `AUDIT_RELAY_MAX_WAIT` | Optional under `bigquery`: events per delivery (default 500, at most 5000) and how long a partial batch waits (default `5s`) | | `ERROR_TRACKING_MODE` | Explicit `disabled` or `sentry`; rejects partial config | | `SENTRY_DSN` | Server Sentry DSN, required in Sentry mode | | `OBSERVABILITY_EXPORTER` | Required in-graph collector output: `debug` or `otlphttp`; the collector refuses to start when it is unset | diff --git a/module/services/accounts/code/audit_sink.go b/module/services/accounts/code/audit_sink.go new file mode 100644 index 000000000..e39a32fc0 --- /dev/null +++ b/module/services/accounts/code/audit_sink.go @@ -0,0 +1,228 @@ +package main + +import ( + "accounts/pkg/auditstore/bigquerystore" + "accounts/pkg/auditstore/gcsarchive" + "accounts/pkg/business" + "accounts/pkg/infra" + "context" + "fmt" + "os" + "regexp" + "strconv" + "strings" + "time" + + "cloud.google.com/go/bigquery" + "cloud.google.com/go/storage" +) + +// The audit sink settings. AUDIT_SINK selects the mode (business.AuditSinkMode); +// each mode's settings are required only under it, and a missing one fails +// startup rather than degrading the audit trail. +const ( + envAuditSink = "AUDIT_SINK" + + // AUDIT_SINK=both (ADR 0006). + envAuditExternalURL = "AUDIT_EXTERNAL_URL" + envAuditExternalToken = "AUDIT_EXTERNAL_TOKEN" + + // AUDIT_SINK=bigquery (ADR 0009). Credentials are Application Default + // Credentials only — on Kubernetes, the pod's workload identity — so there + // is deliberately no key-file or token setting. + envAuditBigQueryProject = "AUDIT_BIGQUERY_PROJECT" + envAuditBigQueryDataset = "AUDIT_BIGQUERY_DATASET" + envAuditArchiveBucket = "AUDIT_ARCHIVE_BUCKET" + envAuditDeploymentID = "AUDIT_DEPLOYMENT_ID" + envAuditContentDetailRetentionDays = "AUDIT_CONTENT_DETAIL_RETENTION_DAYS" + // Optional relay tuning. + envAuditRelayBatchSize = "AUDIT_RELAY_BATCH_SIZE" + envAuditRelayMaxWait = "AUDIT_RELAY_MAX_WAIT" +) + +// auditSinkConfig is the resolved audit sink: the mode and, for the modes that +// write outside Postgres, where. +type auditSinkConfig struct { + mode business.AuditSinkMode + // external is the tee's destination under AUDIT_SINK=both. + external business.ExternalAuditSink + // bigQuery is the store of record and archive under AUDIT_SINK=bigquery. + bigQuery *bigQueryAuditSwap +} + +// configuredAuditSink reads AUDIT_SINK and the settings its mode requires. +// "postgres" (the default) keeps the durable emitter alone. "both" adds an +// asynchronous tee to an HTTP endpoint while audit_events stays the store of +// record. "bigquery" swaps the store of record to BigQuery, with Postgres +// keeping only the transactional queue. "external" is refused: no destination +// outside Postgres can join the transaction a change commits in. +func configuredAuditSink() (auditSinkConfig, error) { + mode, err := business.ParseAuditSinkMode(os.Getenv(envAuditSink)) + if err != nil { + return auditSinkConfig{}, err + } + config := auditSinkConfig{mode: mode} + switch mode { + case business.AuditSinkBoth: + config.external, err = configuredExternalAuditSink() + case business.AuditSinkBigQuery: + config.bigQuery, err = configuredBigQueryAuditSwap() + } + if err != nil { + return auditSinkConfig{}, err + } + return config, nil +} + +func configuredExternalAuditSink() (business.ExternalAuditSink, error) { + endpoint := strings.TrimSpace(os.Getenv(envAuditExternalURL)) + if endpoint == "" { + return nil, fmt.Errorf("AUDIT_SINK=both requires %s for the external audit destination", envAuditExternalURL) + } + return business.NewHTTPAuditSink(business.HTTPAuditSinkConfig{ + Endpoint: endpoint, + Token: strings.TrimSpace(os.Getenv(envAuditExternalToken)), + }) +} + +// bigQueryAuditSwap is the validated AUDIT_SINK=bigquery configuration. +type bigQueryAuditSwap struct { + project string + dataset string + archiveBucket string + deploymentID string + contentDetailRetention time.Duration + batchSize int + maxWait time.Duration +} + +// auditDeploymentIDPattern keeps the deployment id usable as an archive path +// segment and a warehouse value without escaping. +var auditDeploymentIDPattern = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$`) + +// configuredBigQueryAuditSwap reads and validates every AUDIT_SINK=bigquery +// setting, reporting all the missing ones at once. +func configuredBigQueryAuditSwap() (*bigQueryAuditSwap, error) { + swap := &bigQueryAuditSwap{ + project: strings.TrimSpace(os.Getenv(envAuditBigQueryProject)), + dataset: strings.TrimSpace(os.Getenv(envAuditBigQueryDataset)), + archiveBucket: strings.TrimSpace(os.Getenv(envAuditArchiveBucket)), + deploymentID: strings.TrimSpace(os.Getenv(envAuditDeploymentID)), + batchSize: business.DefaultAuditRelayBatchSize, + maxWait: business.DefaultAuditRelayMaxWait, + } + retentionDays := strings.TrimSpace(os.Getenv(envAuditContentDetailRetentionDays)) + var missing []string + for _, setting := range []struct{ name, value string }{ + {envAuditBigQueryProject, swap.project}, + {envAuditBigQueryDataset, swap.dataset}, + {envAuditArchiveBucket, swap.archiveBucket}, + {envAuditDeploymentID, swap.deploymentID}, + {envAuditContentDetailRetentionDays, retentionDays}, + } { + if setting.value == "" { + missing = append(missing, setting.name) + } + } + if len(missing) > 0 { + return nil, fmt.Errorf("AUDIT_SINK=bigquery requires %s", strings.Join(missing, ", ")) + } + + days, err := strconv.Atoi(retentionDays) + if err != nil || days < 1 { + return nil, fmt.Errorf("%s must be a whole number of days, at least 1; got %q", envAuditContentDetailRetentionDays, retentionDays) + } + swap.contentDetailRetention = time.Duration(days) * 24 * time.Hour + if !auditDeploymentIDPattern.MatchString(swap.deploymentID) { + return nil, fmt.Errorf("%s must be letters, digits, '.', '_' or '-' (at most 128, starting with a letter or digit); got %q", + envAuditDeploymentID, swap.deploymentID) + } + if strings.Contains(swap.archiveBucket, "/") { + return nil, fmt.Errorf("%s is a bucket name, not a URL or path; got %q", envAuditArchiveBucket, swap.archiveBucket) + } + if raw := strings.TrimSpace(os.Getenv(envAuditRelayBatchSize)); raw != "" { + size, err := strconv.Atoi(raw) + if err != nil || size < 1 || size > business.MaxAuditRelayBatchSize { + return nil, fmt.Errorf("%s must be between 1 and %d; got %q", envAuditRelayBatchSize, business.MaxAuditRelayBatchSize, raw) + } + swap.batchSize = size + } + if raw := strings.TrimSpace(os.Getenv(envAuditRelayMaxWait)); raw != "" { + wait, err := time.ParseDuration(raw) + if err != nil || wait <= 0 || wait > time.Hour { + return nil, fmt.Errorf("%s must be a positive duration of at most an hour, like 5s; got %q", envAuditRelayMaxWait, raw) + } + swap.maxWait = wait + } + return swap, nil +} + +// newBigQueryAuditRelay builds the relay of the bigquery swap value: the +// BigQuery store of record (its tables created if missing), the GCS archive, +// and the queue on the relay's own pool. Both clients use Application Default +// Credentials. The returned close releases them all. +func newBigQueryAuditRelay(ctx context.Context, types business.DeclaredAuditEventTypeReader, swap *bigQueryAuditSwap) (*business.AuditRelay, func(), error) { + var closers []func() + closeAll := func() { + for i := len(closers) - 1; i >= 0; i-- { + closers[i]() + } + } + fail := func(err error) (*business.AuditRelay, func(), error) { + closeAll() + return nil, nil, err + } + + bigQueryClient, err := bigquery.NewClient(ctx, swap.project) + if err != nil { + return fail(fmt.Errorf("audit store: bigquery client: %w", err)) + } + closers = append(closers, func() { _ = bigQueryClient.Close() }) + warehouse, err := bigquerystore.New(bigQueryClient, bigquerystore.Config{ + Dataset: swap.dataset, + ContentDetailRetention: swap.contentDetailRetention, + }) + if err != nil { + return fail(err) + } + ensureCtx, cancel := context.WithTimeout(ctx, time.Minute) + err = warehouse.Ensure(ensureCtx) + cancel() + if err != nil { + return fail(err) + } + + storageClient, err := storage.NewClient(ctx) + if err != nil { + return fail(fmt.Errorf("audit archive: storage client: %w", err)) + } + closers = append(closers, func() { _ = storageClient.Close() }) + archive, err := gcsarchive.New(storageClient, swap.archiveBucket) + if err != nil { + return fail(err) + } + + pool, err := infra.NewAuditRelayPool(ctx) + if err != nil { + return fail(fmt.Errorf("audit relay: database pool: %w", err)) + } + closers = append(closers, pool.Close) + queue, err := infra.NewPostgresAuditQueue(pool) + if err != nil { + return fail(err) + } + + relay, err := business.NewAuditRelay(business.AuditRelayConfig{ + Queue: queue, + Store: warehouse, + Archive: archive, + Types: types, + DeploymentID: swap.deploymentID, + BatchSize: swap.batchSize, + MaxWait: swap.maxWait, + }) + if err != nil { + return fail(err) + } + return relay, closeAll, nil +} diff --git a/module/services/accounts/code/audit_writers_test.go b/module/services/accounts/code/audit_writers_test.go new file mode 100644 index 000000000..84d61525e --- /dev/null +++ b/module/services/accounts/code/audit_writers_test.go @@ -0,0 +1,80 @@ +package main + +import ( + "go/ast" + "go/parser" + "go/token" + "io/fs" + "os" + "path/filepath" + "regexp" + "sort" + "strings" + "testing" + + "github.com/stretchr/testify/require" +) + +// Every audit record in this service is written by the emitter, so that one +// switch — AUDIT_SINK — decides where every record goes (ADR 0009). A writer +// that inserts into audit_events itself keeps writing there under a swap +// value, where the store of record is elsewhere and audit_events receives +// nothing else: its records would reach neither the warehouse nor the archive. +// +// The SQL that writes an audit record lives in exactly one file, and the two +// store methods that run it are called from exactly one other. +func TestAuditRecordsAreWrittenOnlyThroughTheEmitter(t *testing.T) { + const ( + sqlHome = "pkg/infra/postgres_audit.go" + callHome = "pkg/business/audit.go" + ) + auditInsert := regexp.MustCompile(`(?i)\bINSERT\s+INTO\s+(?:public\.)?audit_event(?:s|_queue)\b`) + + var sqlWriters, callers []string + fset := token.NewFileSet() + err := filepath.WalkDir(".", func(path string, entry fs.DirEntry, err error) error { + if err != nil { + return err + } + if entry.IsDir() { + if path == "pkg/gen" || strings.HasPrefix(entry.Name(), ".") && path != "." { + return filepath.SkipDir + } + return nil + } + if !strings.HasSuffix(path, ".go") || strings.HasSuffix(path, "_test.go") { + return nil + } + source, err := os.ReadFile(path) + if err != nil { + return err + } + if path != sqlHome && auditInsert.Match(source) { + sqlWriters = append(sqlWriters, path) + } + parsed, err := parser.ParseFile(fset, path, source, 0) + if err != nil { + return err + } + ast.Inspect(parsed, func(n ast.Node) bool { + call, ok := n.(*ast.CallExpr) + if !ok { + return true + } + sel, ok := call.Fun.(*ast.SelectorExpr) + if !ok || (sel.Sel.Name != "InsertAuditEvent" && sel.Sel.Name != "EnqueueAuditEvent") { + return true + } + if path != callHome { + callers = append(callers, fset.Position(call.Pos()).String()) + } + return true + }) + return nil + }) + require.NoError(t, err) + sort.Strings(sqlWriters) + sort.Strings(callers) + require.Empty(t, sqlWriters, "only %s may hold SQL that writes an audit record; route the others through the emitter (business.AuditRecorder)", sqlHome) + require.Empty(t, callers, "only %s may call the store's audit record writes; route the others through the emitter (business.AuditRecorder)", callHome) +} diff --git a/module/services/accounts/code/cmd/role-catalog-import/main.go b/module/services/accounts/code/cmd/role-catalog-import/main.go index e96dbe9d1..4604e4080 100644 --- a/module/services/accounts/code/cmd/role-catalog-import/main.go +++ b/module/services/accounts/code/cmd/role-catalog-import/main.go @@ -5,11 +5,15 @@ // // Usage: // -// role-catalog-import -catalog roles.json -database-url "$DATABASE_URL" [-dry-run] [-force] +// role-catalog-import -catalog roles.json -database-url "$DATABASE_URL" [-audit-sink "$AUDIT_SINK"] [-dry-run] [-force] // // The connection principal must be a member of app_control_plane (the same // authority migrations run under); built-in roles cannot be written otherwise. // See AUTHZ.md ("Built-in role catalog import") for the format and workflow. +// +// -audit-sink must be the deployment's AUDIT_SINK: under a swap value +// (ADR 0009) the import's audit events go to the transactional queue, which +// the accounts relay delivers, rather than to audit_events. package main import ( @@ -19,6 +23,7 @@ import ( "io" "os" + "accounts/pkg/business" "accounts/pkg/infra" "accounts/pkg/rolecatalog" ) @@ -43,9 +48,15 @@ func run(args []string, stdout, stderr io.Writer) int { databaseURL := fs.String("database-url", os.Getenv("DATABASE_URL"), "Postgres connection URL (defaults to $DATABASE_URL)") dryRun := fs.Bool("dry-run", false, "print the plan without applying it") force := fs.Bool("force", false, "apply removals even when they would delete assignments or wipe the whole catalog") + auditSink := fs.String("audit-sink", os.Getenv("AUDIT_SINK"), "the deployment's AUDIT_SINK (defaults to $AUDIT_SINK; empty is postgres)") if err := fs.Parse(args); err != nil { return 2 } + sinkMode, err := business.ParseAuditSinkMode(*auditSink) + if err != nil { + line(stderr, "role-catalog-import:", err) + return 1 + } if *catalogPath == "" { line(stderr, "role-catalog-import: -catalog is required") @@ -75,7 +86,22 @@ func run(args []string, stdout, stderr io.Writer) int { } defer store.Close() - result, err := store.ImportRoleCatalog(ctx, catalog, infra.ImportOptions{DryRun: *dryRun, Force: *force, Source: *catalogPath}) + var recorderOptions []business.DurableAuditEmitterOption + if sinkMode.Swaps() { + recorderOptions = append(recorderOptions, business.WithQueuedRecords()) + } + // The import records platform events only (built-in roles have no + // organization), which neither the tee nor webhook delivery carries, so + // the recorder needs nothing from the mode but where the record goes. + recorder, err := business.NewDurableAuditEmitter(store, store, recorderOptions...) + if err != nil { + line(stderr, "role-catalog-import:", err) + return 1 + } + + result, err := store.ImportRoleCatalog(ctx, catalog, infra.ImportOptions{ + DryRun: *dryRun, Force: *force, Source: *catalogPath, Audit: recorder, + }) if err != nil { line(stderr, "role-catalog-import:", err) return 1 diff --git a/module/services/accounts/code/go.mod b/module/services/accounts/code/go.mod index 63b667578..989d40d10 100644 --- a/module/services/accounts/code/go.mod +++ b/module/services/accounts/code/go.mod @@ -5,6 +5,8 @@ go 1.27.0 require ( buf.build/gen/go/bufbuild/protovalidate/protocolbuffers/go v1.36.12-20260825204119-511051f7f437.2 buf.build/go/protovalidate v1.4.0 + cloud.google.com/go/bigquery v1.85.0 + cloud.google.com/go/storage v1.69.0 connectrpc.com/connect v1.21.0 connectrpc.com/otelconnect v0.9.0 github.com/codefly-dev/core v0.5.11-0.20260926231217-d481a2d49683 @@ -35,6 +37,7 @@ require ( golang.org/x/net v0.59.0 golang.org/x/sync v0.23.0 golang.org/x/sys v0.48.0 + google.golang.org/api v0.288.0 google.golang.org/genproto/googleapis/api v0.0.0-20260831171406-18b4a7587f8a google.golang.org/genproto/googleapis/rpc v0.0.0-20260831171406-18b4a7587f8a google.golang.org/grpc v1.83.2 @@ -45,22 +48,37 @@ require ( require ( cel.dev/cel-go v0.32.0 // indirect cel.dev/expr v0.25.3 // indirect + cloud.google.com/go v0.123.0 // indirect + cloud.google.com/go/auth v0.20.0 // indirect + cloud.google.com/go/auth/oauth2adapt v0.2.8 // indirect + cloud.google.com/go/compute/metadata v0.9.0 // indirect + cloud.google.com/go/iam v1.12.0 // indirect + cloud.google.com/go/monitoring v1.30.0 // indirect github.com/Azure/azure-sdk-for-go/sdk/azcore v1.23.1 // indirect github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.14.1 // indirect github.com/Azure/azure-sdk-for-go/sdk/internal v1.12.0 // indirect github.com/AzureAD/microsoft-authentication-library-for-go v1.9.0 // indirect + github.com/GoogleCloudPlatform/opentelemetry-operations-go/detectors/gcp v1.35.0 // indirect + github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/metric v0.57.0 // indirect + github.com/GoogleCloudPlatform/opentelemetry-operations-go/internal/resourcemapping v0.57.0 // indirect github.com/Masterminds/semver v1.5.0 // indirect github.com/Masterminds/semver/v3 v3.5.0 // indirect github.com/VividCortex/ewma v1.2.0 // indirect github.com/antlr4-go/antlr/v4 v4.13.1 // indirect + github.com/apache/arrow/go/v15 v15.0.2 // indirect github.com/beorn7/perks v1.0.1 // indirect github.com/blang/semver v3.5.1+incompatible // indirect github.com/cenkalti/backoff/v5 v5.0.3 // indirect github.com/cespare/xxhash/v2 v2.3.0 // indirect github.com/cheggaaa/pb/v3 v3.2.1 // indirect github.com/clipperhouse/uax29/v2 v2.7.0 // indirect + github.com/cncf/xds/go v0.0.0-20260202195803-dba9d589def2 // indirect + github.com/envoyproxy/go-control-plane/envoy v1.37.0 // indirect + github.com/envoyproxy/protoc-gen-validate v1.3.3 // indirect github.com/fatih/color v1.19.0 // indirect + github.com/felixge/httpsnoop v1.1.0 // indirect github.com/fxamacker/cbor/v2 v2.9.3 // indirect + github.com/go-jose/go-jose/v4 v4.1.4 // indirect github.com/go-logr/logr v1.4.4 // indirect github.com/go-logr/stdr v1.2.2 // indirect github.com/go-ole/go-ole v1.3.0 // indirect @@ -81,16 +99,22 @@ require ( github.com/go-openapi/swag/yamlutils v0.29.1 // indirect github.com/go-viper/mapstructure/v2 v2.5.0 // indirect github.com/go-webauthn/x v0.3.0 // indirect + github.com/goccy/go-json v0.10.2 // indirect github.com/gofrs/flock v0.13.1 // indirect + github.com/google/flatbuffers v23.5.26+incompatible // indirect github.com/google/go-github/v89 v89.0.0 // indirect github.com/google/go-querystring v1.2.0 // indirect github.com/google/go-tpm v0.9.8 // indirect + github.com/google/s2a-go v0.1.9 // indirect + github.com/googleapis/enterprise-certificate-proxy v0.3.17 // indirect + github.com/googleapis/gax-go/v2 v2.26.2 // indirect github.com/hashicorp/errwrap v1.1.0 // indirect github.com/hashicorp/go-multierror v1.1.1 // indirect github.com/jackc/pgpassfile v1.0.0 // indirect github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect github.com/jackc/puddle/v2 v2.2.2 // indirect - github.com/kr/text v0.2.0 // indirect + github.com/klauspost/compress v1.19.1 // indirect + github.com/klauspost/cpuid/v2 v2.3.0 // indirect github.com/kylelemons/godebug v1.1.0 // indirect github.com/lufia/plan9stats v0.0.0-20260802145828-341c2f0c90b5 // indirect github.com/mattn/go-colorable v0.1.15 // indirect @@ -100,8 +124,10 @@ require ( github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect github.com/oklog/ulid/v2 v2.1.2 // indirect github.com/philhofer/fwd v1.2.0 // indirect + github.com/pierrec/lz4/v4 v4.1.18 // indirect github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c // indirect github.com/pkg/errors v0.9.1 // indirect + github.com/planetscale/vtprotobuf v0.6.1-0.20240319094008-0393e58bdf10 // indirect github.com/power-devops/perfstat v0.0.0-20260805114148-88456608a4f6 // indirect github.com/prometheus/client_model v0.6.3 // indirect github.com/prometheus/common v0.71.0 // indirect @@ -110,14 +136,18 @@ require ( github.com/santhosh-tekuri/jsonschema/v6 v6.0.3 // indirect github.com/shirou/gopsutil/v3 v3.24.5 // indirect github.com/shoenig/go-m1cpu v0.2.2 // indirect + github.com/spiffe/go-spiffe/v2 v2.7.0 // indirect github.com/tinylib/msgp v1.6.4 // indirect github.com/tklauser/go-sysconf v0.4.0 // indirect github.com/tklauser/numcpus v0.12.0 // indirect github.com/x448/float16 v0.8.4 // indirect github.com/yoheimuta/go-protoparser/v4 v4.14.2 // indirect github.com/yusufpapurcu/wmi v1.2.4 // indirect + github.com/zeebo/xxh3 v1.1.0 // indirect go.opentelemetry.io/auto/sdk v1.2.1 // indirect + go.opentelemetry.io/contrib/detectors/gcp v1.45.0 // indirect go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.71.0 // indirect + go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.70.0 // indirect go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.46.0 // indirect go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.46.0 // indirect go.opentelemetry.io/otel/exporters/stdout/stdouttrace v1.46.0 // indirect @@ -125,6 +155,13 @@ require ( go.uber.org/atomic v1.11.0 // indirect go.yaml.in/yaml/v3 v3.0.5 // indirect golang.org/x/exp v0.0.0-20260824195058-e88cd73687aa // indirect + golang.org/x/mod v0.41.0 // indirect + golang.org/x/oauth2 v0.36.0 // indirect + golang.org/x/telemetry v0.0.0-20260811182544-a038080d80e5 // indirect golang.org/x/term v0.46.0 // indirect golang.org/x/text v0.42.0 // indirect + golang.org/x/time v0.15.0 // indirect + golang.org/x/tools v0.49.0 // indirect + golang.org/x/xerrors v0.0.0-20240903120638-7835f813f4da // indirect + google.golang.org/genproto v0.0.0-20260715232425-e75dac1f907d // indirect ) diff --git a/module/services/accounts/code/go.sum b/module/services/accounts/code/go.sum index dc7e3b233..3b82ece3e 100644 --- a/module/services/accounts/code/go.sum +++ b/module/services/accounts/code/go.sum @@ -6,6 +6,30 @@ cel.dev/cel-go v0.32.0 h1:irvpFKr5EuGPyxeME03ERh0rii1TX+BDAnB9eL3IvNk= cel.dev/cel-go v0.32.0/go.mod h1:DnVip7tpJSsgZymwfT+m1tnEVy3ivAjSMXPx12YrMkU= cel.dev/expr v0.25.3 h1:A2jO8jwOugrrovveCWfj0KEZOfqiLgAcwjpHPhzIGw0= cel.dev/expr v0.25.3/go.mod h1:hrXvqGP6G6gyx8UAHSHJ5RGk//1Oj5nXQ2NI02Nrsg4= +cloud.google.com/go v0.123.0 h1:2NAUJwPR47q+E35uaJeYoNhuNEM9kM8SjgRgdeOJUSE= +cloud.google.com/go v0.123.0/go.mod h1:xBoMV08QcqUGuPW65Qfm1o9Y4zKZBpGS+7bImXLTAZU= +cloud.google.com/go/auth v0.20.0 h1:kXTssoVb4azsVDoUiF8KvxAqrsQcQtB53DcSgta74CA= +cloud.google.com/go/auth v0.20.0/go.mod h1:942/yi/itH1SsmpyrbnTMDgGfdy2BUqIKyd0cyYLc5Q= +cloud.google.com/go/auth/oauth2adapt v0.2.8 h1:keo8NaayQZ6wimpNSmW5OPc283g65QNIiLpZnkHRbnc= +cloud.google.com/go/auth/oauth2adapt v0.2.8/go.mod h1:XQ9y31RkqZCcwJWNSx2Xvric3RrU88hAYYbjDWYDL+c= +cloud.google.com/go/bigquery v1.85.0 h1:zsFsa8jOVkU4c7CWE1cbrfsemtNbM3YRUmtFRYXYN58= +cloud.google.com/go/bigquery v1.85.0/go.mod h1:oBma1P5/b1Jtd8xRLKoyTeNIMlACGHbSMLudzxHGHgc= +cloud.google.com/go/compute/metadata v0.9.0 h1:pDUj4QMoPejqq20dK0Pg2N4yG9zIkYGdBtwLoEkH9Zs= +cloud.google.com/go/compute/metadata v0.9.0/go.mod h1:E0bWwX5wTnLPedCKqk3pJmVgCBSM6qQI1yTBdEb3C10= +cloud.google.com/go/datacatalog v1.33.0 h1:8V80PpoAGdOOr2QhBrp4wZ66MDCbATdAB/fmVmo5rlU= +cloud.google.com/go/datacatalog v1.33.0/go.mod h1:/EMN04S73fZcPdtNg86VYLDrhi2HheMehQtMCS86Klk= +cloud.google.com/go/iam v1.12.0 h1:Aki3bX9aHUDKPHfnRJfDcTdVedvy6quGBQcTqx3DRXk= +cloud.google.com/go/iam v1.12.0/go.mod h1:FEZ4lXpADAC2AIpQY7LANNjjwyQ2jK439CI2VaD+sLY= +cloud.google.com/go/logging v1.19.0 h1:NCqhdVUg3wQ8Cobdf16FDSuTGi3+6+hdSBHrY5TsR6Q= +cloud.google.com/go/logging v1.19.0/go.mod h1:i40NZCHC9Gqvod4yE+yQfDWwlgwW/SrshkkGibCHxcA= +cloud.google.com/go/longrunning v1.2.0 h1:WjYH3YHBGCxGJP9M4dWGHBfXr/cFIjMkNgWcJj7/iMM= +cloud.google.com/go/longrunning v1.2.0/go.mod h1:5KMQALFGOCtFoi2xSOA1u3H7WKlhmckgiyFw7+LGQp0= +cloud.google.com/go/monitoring v1.30.0 h1:r/d+JUbyKmJ8b07iznuKfzVzrIXTWxHQ3lBRm3x2LlY= +cloud.google.com/go/monitoring v1.30.0/go.mod h1:htlUR0QWVMrjFzZmN4LGnMAve9xB/eduwjmINxVZ8RM= +cloud.google.com/go/storage v1.69.0 h1:jAAMC1411HEh78nKsU0Zns+eFj3TnhjAWIhg5Ud/XBM= +cloud.google.com/go/storage v1.69.0/go.mod h1:PELYsxTYm2peE4mwLEC1+mS1dA/kUSRUxNv56rOy44g= +cloud.google.com/go/trace v1.16.0 h1:GmQovzFc5F0CNfl0VLgL64aoTtu7xsM0YajW2GlG9+E= +cloud.google.com/go/trace v1.16.0/go.mod h1:r+bdAn16dKLSV1G2D5v3e58IlQlizfxWrUfjx7kM7X0= connectrpc.com/connect v1.21.0 h1:LhqSJt7jHf5NJBo9Jq/t/9FjcYAideif0mg+qe2jCUs= connectrpc.com/connect v1.21.0/go.mod h1:A2ygJrukXwWy32vkCAAHNVguZrqZ+jeZ9rGRnGR4dN4= connectrpc.com/otelconnect v0.9.0 h1:NggB3pzRC3pukQWaYbRHJulxuXvmCKCKkQ9hbrHAWoA= @@ -24,6 +48,14 @@ github.com/AzureAD/microsoft-authentication-extensions-for-go/cache v0.1.1 h1:WJ github.com/AzureAD/microsoft-authentication-extensions-for-go/cache v0.1.1/go.mod h1:tCcJZ0uHAmvjsVYzEFivsRTN00oz5BEsRgQHu5JZ9WE= github.com/AzureAD/microsoft-authentication-library-for-go v1.9.0 h1:MDT4FxAPve5FnYn6vOL1r7RCRDG+l9cI7a5LlCuHsqA= github.com/AzureAD/microsoft-authentication-library-for-go v1.9.0/go.mod h1:Y33QHnf0FfdVewFFISOGe20mkZbxX4H839o955/PoeI= +github.com/GoogleCloudPlatform/opentelemetry-operations-go/detectors/gcp v1.35.0 h1:bN1gA3of5bXtbnLsRPrwfmbbe7A5UWFlcTHseujLnpc= +github.com/GoogleCloudPlatform/opentelemetry-operations-go/detectors/gcp v1.35.0/go.mod h1:Yj5vHEz/aAepZGliRJsA6uvHAVAQyEwajq9ORCHPxzM= +github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/metric v0.57.0 h1:jLdiS1vO+XJFyDSWRHBx56r4s/NNtcl5J6KyCcWUX/w= +github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/metric v0.57.0/go.mod h1:8lmpHY+1VRoteiOwyrQMDt1YGXOrFKCz+1wJW7n3ODY= +github.com/GoogleCloudPlatform/opentelemetry-operations-go/internal/cloudmock v0.57.0 h1:cSjUzZ7KU8hicTgzaSv9NmSyM9fTVK3y5lsBUl3wOis= +github.com/GoogleCloudPlatform/opentelemetry-operations-go/internal/cloudmock v0.57.0/go.mod h1:dzcEjy1WJ0Q4u9twNR3LcLhNoYMRCrMCMafpxa0TjPQ= +github.com/GoogleCloudPlatform/opentelemetry-operations-go/internal/resourcemapping v0.57.0 h1:RoO5+d7uCmDqovLrHCr2/BuViUXvdcrNxyNM1pN9dDQ= +github.com/GoogleCloudPlatform/opentelemetry-operations-go/internal/resourcemapping v0.57.0/go.mod h1:YqwkQPrWSC7+byyc1VlKbWLBF5JsW5IoL6xUkemYSXk= github.com/Masterminds/semver v1.5.0 h1:H65muMkzWKEuNDnfl9d70GUjFniHKHRbFPGBuZ3QEww= github.com/Masterminds/semver v1.5.0/go.mod h1:MB6lktGJrhw8PrUyiEoblNEGEQ+RzHPF078ddwwvV3Y= github.com/Masterminds/semver/v3 v3.5.0 h1:kQceYJfbupGfZOKZQg0kou0DgAKhzDg2NZPAwZ/2OOE= @@ -36,6 +68,8 @@ github.com/VividCortex/ewma v1.2.0 h1:f58SaIzcDXrSy3kWaHNvuJgJ3Nmz59Zji6XoJR/q1o github.com/VividCortex/ewma v1.2.0/go.mod h1:nz4BbCtbLyFDeC9SUHbtcT5644juEuWfUAUnGx7j5l4= github.com/antlr4-go/antlr/v4 v4.13.1 h1:SqQKkuVZ+zWkMMNkjy5FZe5mr5WURWnlpmOuzYWrPrQ= github.com/antlr4-go/antlr/v4 v4.13.1/go.mod h1:GKmUxMtwp6ZgGwZSva4eWPC5mS6vUAmOABFgjdkM7Nw= +github.com/apache/arrow/go/v15 v15.0.2 h1:60IliRbiyTWCWjERBCkO1W4Qun9svcYoZrSLcyOsMLE= +github.com/apache/arrow/go/v15 v15.0.2/go.mod h1:DGXsR3ajT524njufqf95822i+KTh+yea1jass9YXgjA= github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM= github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw= github.com/blang/semver v3.5.1+incompatible h1:cQNTCjp13qL8KC3Nbxr/y2Bqb63oX6wdnnjpJbkM4JQ= @@ -58,6 +92,8 @@ github.com/clipperhouse/uax29/v2 v2.7.0 h1:+gs4oBZ2gPfVrKPthwbMzWZDaAFPGYK72F0NJ github.com/clipperhouse/uax29/v2 v2.7.0/go.mod h1:EFJ2TJMRUaplDxHKj1qAEhCtQPW2tJSwu5BF98AuoVM= github.com/cloudflare/circl v1.6.3 h1:9GPOhQGF9MCYUeXyMYlqTR6a5gTrgR/fBLXvUgtVcg8= github.com/cloudflare/circl v1.6.3/go.mod h1:2eXP6Qfat4O/Yhh8BznvKnJ+uzEoTQ6jVKJRn81BiS4= +github.com/cncf/xds/go v0.0.0-20260202195803-dba9d589def2 h1:aBangftG7EVZoUb69Os8IaYg++6uMOdKK83QtkkvJik= +github.com/cncf/xds/go v0.0.0-20260202195803-dba9d589def2/go.mod h1:qwXFYgsP6T7XnJtbKlf1HP8AjxZZyzxMmc+Lq5GjlU4= github.com/codefly-dev/core v0.5.11-0.20260926231217-d481a2d49683 h1:/vzymsu5B9NWwdhWRtlU+68VcVEVDmeZNChRCXAUwlE= github.com/codefly-dev/core v0.5.11-0.20260926231217-d481a2d49683/go.mod h1:SVF+t040X+9pAXAqBnNE/JkXwmw9peVuKnHKFSOzyzk= github.com/codefly-dev/gortk v0.2.0 h1:7bOlS5valYz2zil+fZctQNcPCYBcPj86abcw9N8h1hQ= @@ -76,7 +112,6 @@ github.com/containerd/errdefs v1.0.0 h1:tg5yIfIlQIrxYtu9ajqY42W3lpS19XqdxRQeEwYG github.com/containerd/errdefs v1.0.0/go.mod h1:+YBYIdtsnF4Iw6nWZhJcqGSg/dwvV7tyJ/kCkyJ2k+M= github.com/containerd/errdefs/pkg v0.3.0 h1:9IKJ06FvyNlexW690DXuQNx2KA2cUJXx151Xdx3ZPPE= github.com/containerd/errdefs/pkg v0.3.0/go.mod h1:NJw6s9HwNuRhnjJhM7pylWwMyAkmCQvQ4GpJHEqRLVk= -github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E= github.com/cyphar/filepath-securejoin v0.7.0 h1:s0Y3ITPy6sQn5xt54DuYvTF8hu134ooYLUb58DX/HjE= github.com/cyphar/filepath-securejoin v0.7.0/go.mod h1:ymLGms/u3BYaviIiuKFnUx8EkQEZeK6cInNoAPJA3o4= github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= @@ -94,6 +129,14 @@ github.com/docker/go-units v0.5.0 h1:69rxXcBk27SvSaaxTtLh/8llcHD8vYHT7WSdRZ/jvr4 github.com/docker/go-units v0.5.0/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDDbaIK4Dk= github.com/emirpasic/gods v1.18.1 h1:FXtiHYKDGKCW2KzwZKx0iC0PQmdlorYgdFG9jPXJ1Bc= github.com/emirpasic/gods v1.18.1/go.mod h1:8tpGGwCnJ5H4r6BWwaV6OrWmMoPhUl5jm/FMNAnJvWQ= +github.com/envoyproxy/go-control-plane v0.14.0 h1:hbG2kr4RuFj222B6+7T83thSPqLjwBIfQawTkC++2HA= +github.com/envoyproxy/go-control-plane v0.14.0/go.mod h1:NcS5X47pLl/hfqxU70yPwL9ZMkUlwlKxtAohpi2wBEU= +github.com/envoyproxy/go-control-plane/envoy v1.37.0 h1:u3riX6BoYRfF4Dr7dwSOroNfdSbEPe9Yyl09/B6wBrQ= +github.com/envoyproxy/go-control-plane/envoy v1.37.0/go.mod h1:DReE9MMrmecPy+YvQOAOHNYMALuowAnbjjEMkkWOi6A= +github.com/envoyproxy/go-control-plane/ratelimit v0.1.0 h1:/G9QYbddjL25KvtKTv3an9lx6VBE2cnb8wp1vEGNYGI= +github.com/envoyproxy/go-control-plane/ratelimit v0.1.0/go.mod h1:Wk+tMFAFbCXaJPzVVHnPgRKdUdwW/KdbRt94AzgRee4= +github.com/envoyproxy/protoc-gen-validate v1.3.3 h1:MVQghNeW+LZcmXe7SY1V36Z+WFMDjpqGAGacLe2T0ds= +github.com/envoyproxy/protoc-gen-validate v1.3.3/go.mod h1:TsndJ/ngyIdQRhMcVVGDDHINPLWB7C82oDArY51KfB0= github.com/fatih/color v1.19.0 h1:Zp3PiM21/9Ld6FzSKyL5c/BULoe/ONr9KlbYVOfG8+w= github.com/fatih/color v1.19.0/go.mod h1:zNk67I0ZUT1bEGsSGyCZYZNrHuTkJJB+r6Q9VuMi0LE= github.com/felixge/httpsnoop v1.1.0 h1:3YtUj32ZZkqZtt3sZZsClsymw/QDuVfpNhoA31zeORc= @@ -110,6 +153,8 @@ github.com/go-git/go-billy/v5 v5.9.0 h1:jItGXszUDRtR/AlferWPTMN4j38BQ88XnXKbilmm github.com/go-git/go-billy/v5 v5.9.0/go.mod h1:jCnQMLj9eUgGU7+ludSTYoZL/GGmii14RxKFj7ROgHw= github.com/go-git/go-git/v5 v5.19.2 h1:wkfn7vOlUBu8ivAWKBWisTiwJK4jYHzTF8Ndv1LyGqY= github.com/go-git/go-git/v5 v5.19.2/go.mod h1:QqCBE1EFN5ddFmrliLQ3/ntRCUjZU3EJuwuB/jWEHjk= +github.com/go-jose/go-jose/v4 v4.1.4 h1:moDMcTHmvE6Groj34emNPLs/qtYXRVcd6S7NHbHz3kA= +github.com/go-jose/go-jose/v4 v4.1.4/go.mod h1:x4oUasVrzR7071A4TnHLGSPpNOm2a21K9Kf04k1rs08= github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A= github.com/go-logr/logr v1.4.4 h1:tG4xh9yMsRCAiodLVTxyrkzSZ9+o0L1Kg/+cPVcbP/8= github.com/go-logr/logr v1.4.4/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY= @@ -168,6 +213,8 @@ github.com/go-webauthn/webauthn v0.18.0 h1:PC8R3PNLEmjZf++WwcQlo1Z39S9rf8ma69rlw github.com/go-webauthn/webauthn v0.18.0/go.mod h1:ymzZQhx3D/PrDjznemBdQJ23gHTaSDxUchM7sH1lUCg= github.com/go-webauthn/x v0.3.0 h1:Q2X9vbrlP0Ed+QGEzixh1hthGZlDnzVT0XH/9IIQ0kE= github.com/go-webauthn/x v0.3.0/go.mod h1:5OkdSQdOy7taRXWqvNHggtaPffmW94ybu3rZEER4I+I= +github.com/goccy/go-json v0.10.2 h1:CrxCmQqYDkv1z7lO7Wbh2HN93uovUHgrECaO5ZrCXAU= +github.com/goccy/go-json v0.10.2/go.mod h1:6MelG93GURQebXPDq3khkgXZkazVtN9CRI+MGFi0w8I= github.com/gofrs/flock v0.13.1 h1:jjREztyBeSKBZYAC+mgc1laB+xsgy4kYMf3FbKF2UBo= github.com/gofrs/flock v0.13.1/go.mod h1:sf4BFiHwnvgxa25DlQoDqXQnwRMEOwqxRq37P6MzzmE= github.com/golang-jwt/jwt/v5 v5.3.1 h1:kYf81DTWFe7t+1VvL7eS+jKFVWaUnK9cB1qbwn63YCY= @@ -176,6 +223,8 @@ github.com/golang/groupcache v0.0.0-20241129210726-2c02b8208cf8 h1:f+oWsMOmNPc8J github.com/golang/groupcache v0.0.0-20241129210726-2c02b8208cf8/go.mod h1:wcDNUvekVysuuOpQKo3191zZyTpiI6se1N1ULghS0sw= github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek= github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps= +github.com/google/flatbuffers v23.5.26+incompatible h1:M9dgRyhJemaM4Sw8+66GHBu8ioaQmyPLg1b8VwK5WJg= +github.com/google/flatbuffers v23.5.26+incompatible/go.mod h1:1AeVuKshWv4vARoZatz6mlQ0JxURH0Kv5+zNeJKJCa8= github.com/google/gnostic-models v0.6.9 h1:MU/8wDLif2qCXZmzncUQ/BOfxWfthHi63KqpoNbWqVw= github.com/google/gnostic-models v0.6.9/go.mod h1:CiWsm0s6BSQd1hRn8/QmxqB6BesYcbSZxsz9b0KuDBw= github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY= @@ -189,8 +238,16 @@ github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo= github.com/google/go-tpm v0.9.8/go.mod h1:h9jEsEECg7gtLis0upRBQU+GhYVH6jMjrFxI8u6bVUY= github.com/google/go-tpm-tools v0.3.13-0.20230620182252-4639ecce2aba h1:qJEJcuLzH5KDR0gKc0zcktin6KSAwL7+jWKBYceddTc= github.com/google/go-tpm-tools v0.3.13-0.20230620182252-4639ecce2aba/go.mod h1:EFYHy8/1y2KfgTAsx7Luu7NGhoxtuVHnNo8jE7FikKc= +github.com/google/martian/v3 v3.3.3 h1:DIhPTQrbPkgs2yJYdXU/eNACCG5DVQjySNRNlflZ9Fc= +github.com/google/martian/v3 v3.3.3/go.mod h1:iEPrYcgCF7jA9OtScMFQyAlZZ4YXTKEtJ1E6RWzmBA0= +github.com/google/s2a-go v0.1.9 h1:LGD7gtMgezd8a/Xak7mEWL0PjoTQFvpRudN895yqKW0= +github.com/google/s2a-go v0.1.9/go.mod h1:YA0Ei2ZQL3acow2O62kdp9UlnvMmU7kA6Eutn0dXayM= github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0= github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= +github.com/googleapis/enterprise-certificate-proxy v0.3.17 h1:73NfMHdiqo9JFU9+7a5ExpVa10/R29pXfZIaW559nrg= +github.com/googleapis/enterprise-certificate-proxy v0.3.17/go.mod h1:rSEsBUemEBZEexP2y6jPp16LUmUbjmSbcPMQizR0o4k= +github.com/googleapis/gax-go/v2 v2.26.2 h1:ydkmNXxj7bEmmeK5AihkKnWxyOyBR9TDebvp5L5izk8= +github.com/googleapis/gax-go/v2 v2.26.2/go.mod h1:sMKqnMesnKH+3wiRJROcttA+cJoZoGbZl1vDQ8XYtGk= github.com/grpc-ecosystem/grpc-gateway/v2 v2.30.0 h1:/Tnpcb2E0Pz/tN9s3bfEY2Q8ePCEX9iuS+cneUwncnw= github.com/grpc-ecosystem/grpc-gateway/v2 v2.30.0/go.mod h1:zOBXOsUaBSjKgmH4OGzV1esUpR3oUSCPYVd2cUBjKYY= github.com/hashicorp/errwrap v1.0.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4= @@ -251,12 +308,16 @@ github.com/opencontainers/image-spec v1.1.1/go.mod h1:qpqAh3Dmcf36wStyyWU+kCeDgr github.com/pborman/getopt v0.0.0-20170112200414-7148bc3a4c30/go.mod h1:85jBQOZwpVEaDAr341tbn15RS4fCAsIst0qp7i8ex1o= github.com/philhofer/fwd v1.2.0 h1:e6DnBTl7vGY+Gz322/ASL4Gyp1FspeMvx1RNDoToZuM= github.com/philhofer/fwd v1.2.0/go.mod h1:RqIHx9QI14HlwKwm98g9Re5prTQ6LdeRQn+gXJFxsJM= +github.com/pierrec/lz4/v4 v4.1.18 h1:xaKrnTkyoqfh1YItXl56+6KJNVYWlEEPuAQW9xsplYQ= +github.com/pierrec/lz4/v4 v4.1.18/go.mod h1:gZWDp/Ze/IJXGXf23ltt2EXimqmTUXEy0GFuRQyBid4= github.com/pjbgf/sha1cd v0.6.0 h1:3WJ8Wz8gvDz29quX1OcEmkAlUg9diU4GxJHqs0/XiwU= github.com/pjbgf/sha1cd v0.6.0/go.mod h1:lhpGlyHLpQZoxMv8HcgXvZEhcGs0PG/vsZnEJ7H0iCM= github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c h1:+mdjkGKdHQG3305AYmdv1U2eRNDiU2ErMBj1gwrq8eQ= github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c/go.mod h1:7rwL4CYBLnjLxUqIJNnCWiEdr3bn6IUYi15bNlnbCCU= github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4= github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= +github.com/planetscale/vtprotobuf v0.6.1-0.20240319094008-0393e58bdf10 h1:GFCKgmp0tecUJ0sJuv4pzYCqS9+RGSn52M3FUwPs+uo= +github.com/planetscale/vtprotobuf v0.6.1-0.20240319094008-0393e58bdf10/go.mod h1:t/avpk3KcrXxUnYOhZhMXJlSEyie6gQbtLq5NM3loB8= github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= github.com/power-devops/perfstat v0.0.0-20260805114148-88456608a4f6 h1:jL3a8soXdzuTCcRnKhOmtcsVOObdDTFf4O2B403HPRU= github.com/power-devops/perfstat v0.0.0-20260805114148-88456608a4f6/go.mod h1:OmDBASR4679mdNQnz2pUhc2G8CO2JrUAVFDRBDP/hJE= @@ -290,6 +351,8 @@ github.com/shoenig/test v1.7.0 h1:eWcHtTXa6QLnBvm0jgEabMRN/uJ4DMV3M8xUGgRkZmk= github.com/shoenig/test v1.7.0/go.mod h1:UxJ6u/x2v/TNs/LoLxBNJRV9DiwBBKYxXSyczsBHFoI= github.com/skeema/knownhosts v1.3.2 h1:EDL9mgf4NzwMXCTfaxSD/o/a5fxDw/xL9nkU28JjdBg= github.com/skeema/knownhosts v1.3.2/go.mod h1:bEg3iQAuw+jyiw+484wwFJoKSLwcfd7fqRy+N0QTiow= +github.com/spiffe/go-spiffe/v2 v2.7.0 h1:uXe1MflJoHw58wAUvxVlcM7WpKtijWG7I1UidcGh6g4= +github.com/spiffe/go-spiffe/v2 v2.7.0/go.mod h1:47Q0Q9/AqGha8QLHp+kxpH4Wca7X7EnOtlIJy3mxZ3U= github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= @@ -313,10 +376,14 @@ github.com/yoheimuta/go-protoparser/v4 v4.14.2 h1:/P/LlX1CF9NaTWEltGcIZVvNlPbhAB github.com/yoheimuta/go-protoparser/v4 v4.14.2/go.mod h1:AHNNnSWnb0UoL4QgHPiOAg2BniQceFscPI5X/BZNHl8= github.com/yusufpapurcu/wmi v1.2.4 h1:zFUKzehAFReQwLys1b/iSMl+JQGSCSjtVqQn9bBrPo0= github.com/yusufpapurcu/wmi v1.2.4/go.mod h1:SBZ9tNy3G9/m5Oi98Zks0QjeHVDvuK0qfxQmPyzfmi0= +github.com/zeebo/assert v1.3.0 h1:g7C04CbJuIDKNPFHmsk4hwZDO5O+kntRxzaUoNXj+IQ= +github.com/zeebo/assert v1.3.0/go.mod h1:Pq9JiuJQpG8JLJdtkwrJESF0Foym2/D9XMU5ciN/wJ0= github.com/zeebo/xxh3 v1.1.0 h1:s7DLGDK45Dyfg7++yxI0khrfwq9661w9EN78eP/UZVs= github.com/zeebo/xxh3 v1.1.0/go.mod h1:IisAie1LELR4xhVinxWS5+zf1lA4p0MW4T+w+W07F5s= go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ64= go.opentelemetry.io/auto/sdk v1.2.1/go.mod h1:KRTj+aOaElaLi+wW1kO/DZRXwkF4C5xPbEe3ZiIhN7Y= +go.opentelemetry.io/contrib/detectors/gcp v1.45.0 h1:9jR0ZPRok9ryaOQ2Wx8rg5F7Aon59mxrqbVI60/vlBk= +go.opentelemetry.io/contrib/detectors/gcp v1.45.0/go.mod h1:VSme3o2fvSg5bVg0dRzyHaj4Z5EVhG+g2Fde6LKzmQA= go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.71.0 h1:B2h3uqicet1CT2N5TOFhS+Gq++9i0/CLmaxvhmhtP5s= go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.71.0/go.mod h1:dylvB+ZiiwMvsDij9O84Uy7SijLgHMX4mbkncds+4Sw= go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.70.0 h1:LMuyCAyfalSjDyjdC65nK6N0zoTT63+E/u95X0JovZI= @@ -333,6 +400,8 @@ go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.46.0 h1:w53CD go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.46.0/go.mod h1:BOmGMCbAtvcJiSJ+hLuhgPLdDbimnraSl8irz3iY8sY= go.opentelemetry.io/otel/exporters/prometheus v0.68.0 h1:QOf2IftqQwITVRJpnn0M7M9ZCbgWfxz4P7i9C9yc2N4= go.opentelemetry.io/otel/exporters/prometheus v0.68.0/go.mod h1:bgSvqu2TWGXiz7yr5UTMfObH8oqxJWHTnubQ3ef9BO4= +go.opentelemetry.io/otel/exporters/stdout/stdoutmetric v1.45.0 h1:dm9iyzn6tioYZtwqaiBSU0TSI8Yu/8dTIbfG0+B49DY= +go.opentelemetry.io/otel/exporters/stdout/stdoutmetric v1.45.0/go.mod h1:xAvxYjYK28qvt+yu4BYZ/zMmAjwMXINXD6JiMyeB8iI= go.opentelemetry.io/otel/exporters/stdout/stdouttrace v1.46.0 h1:KdRxPiAoMptR3vfWzvjjvutTsSiwbC2uG0496rzZNfo= go.opentelemetry.io/otel/exporters/stdout/stdouttrace v1.46.0/go.mod h1:K/qSA+3G7Eovxi4K09wzrAgkWRnosS0DAOZeEpve7sM= go.opentelemetry.io/otel/metric v1.46.0 h1:yBnkXvgV7AXFILZc5K6IZe/CBFF3OS7BJ8ov6/lj0K8= @@ -361,8 +430,12 @@ golang.org/x/crypto v0.57.0 h1:3ZVCjf8Ggz7zneR/EHRVx68Ctf+2pmIMP2UFhh9cC6M= golang.org/x/crypto v0.57.0/go.mod h1:Fdz0i5U6CoizGwLda9DttjSk6qlZo25zYNtR+ycvuZA= golang.org/x/exp v0.0.0-20260824195058-e88cd73687aa h1:QSyA8ishJCyT21kER9KwNt0b7BM3iRK4x9QXhjN5Fdk= golang.org/x/exp v0.0.0-20260824195058-e88cd73687aa/go.mod h1:zeBbvyFKDaLwa7CH/zI8KXt7gTl14SF7sO08Pl5jBCM= +golang.org/x/mod v0.41.0 h1:qJmnOUb4YB+FsEuM3HcWucdZASCPGhsX6uljO6pog0c= +golang.org/x/mod v0.41.0/go.mod h1:Ek9pY8RKWXwsWvd3rQiHYtMqkjSUV+s1Rj7j4H5Ur6o= golang.org/x/net v0.59.0 h1:5zfYln+w5XCxwrnMMJPufRgNoXEaGxl0wo5GqPXyues= golang.org/x/net v0.59.0/go.mod h1:2DA/G1UfVbCpQPeWTmMPGY7Cs2PkBkwu743bVX5PIVg= +golang.org/x/oauth2 v0.36.0 h1:peZ/1z27fi9hUOFCAZaHyrpWG5lwe0RJEEEeH0ThlIs= +golang.org/x/oauth2 v0.36.0/go.mod h1:YDBUJMTkDnJS+A4BP4eZBjCqtokkg1hODuPjwiGPO7Q= golang.org/x/sync v0.23.0 h1:KameEIfc1IkluZyXWLn39Wd4tURc6GbCiISGiZm2bQk= golang.org/x/sync v0.23.0/go.mod h1:sUUOizhqBxiL6pEWpqNLUiaJn1ShEbZ6BBqskPbjZm0= golang.org/x/sys v0.0.0-20190916202348-b4ddaad3f8a3/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= @@ -370,12 +443,24 @@ golang.org/x/sys v0.0.0-20201204225414-ed752295db88/go.mod h1:h1NjWce9XRLGQEsW7w golang.org/x/sys v0.1.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.48.0 h1:bbX/i/6MgT9BVLM9RT1thmxL04yeTAhbEz4SyadbXoo= golang.org/x/sys v0.48.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og= +golang.org/x/telemetry v0.0.0-20260811182544-a038080d80e5 h1:ZUSxONxc981v7AW7QUg+I9WwZzSTTJ019ENBYr5pV/Q= +golang.org/x/telemetry v0.0.0-20260811182544-a038080d80e5/go.mod h1:LVehoXe41cL5SCVQilsV7Gg6BNG+Js6P9PhSbYTIUkQ= golang.org/x/term v0.46.0 h1:3+OXuTbaKDgwk8jTi3aSLHRlmWqHEUDUtxnbFigO4YE= golang.org/x/term v0.46.0/go.mod h1:+K02xbkittuwc0Am4abfA3Fc+XRGXkvBXNO88NCXPoc= golang.org/x/text v0.42.0 h1:JbOZXgfeCPU9gacVtYliJqOhD+zhrEqK4LfdpmlUZqI= golang.org/x/text v0.42.0/go.mod h1:ojzP1Z+2QtioaF8DTtO8K5q7JWVVYwZKenzujK0Zd0E= +golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U= +golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno= +golang.org/x/tools v0.49.0 h1:3NI7VXzL9+1WZD52Dx2ttoPwD5DWrFGpl9mFZDlmisI= +golang.org/x/tools v0.49.0/go.mod h1:SJNXV9DBKT0UbdttsQjbfJlAE/q+y36++zo3uL3N0Oo= +golang.org/x/xerrors v0.0.0-20240903120638-7835f813f4da h1:noIWHXmPHxILtqtCOPIhSt0ABwskkZKjD3bXGnZGpNY= +golang.org/x/xerrors v0.0.0-20240903120638-7835f813f4da/go.mod h1:NDW/Ps6MPRej6fsCIbMTohpP40sJ/P/vI1MoTEGwX90= gonum.org/v1/gonum v0.17.0 h1:VbpOemQlsSMrYmn7T2OUvQ4dqxQXU+ouZFQsZOx50z4= gonum.org/v1/gonum v0.17.0/go.mod h1:El3tOrEuMpv2UdMrbNlKEh9vd86bmQ6vqIcDwxEOc1E= +google.golang.org/api v0.288.0 h1:glhO/J88obKP5I269W3hB73dvBKrjU56ZfmNlNXpgTU= +google.golang.org/api v0.288.0/go.mod h1:lM2kYRzYUCBY91P9h6VF1PYmvhxii3O5hji37qRvIcY= +google.golang.org/genproto v0.0.0-20260715232425-e75dac1f907d h1:C9v1o0/4quuhOAfmRXA2j+we0PqZIp8traLdeogF3Ms= +google.golang.org/genproto v0.0.0-20260715232425-e75dac1f907d/go.mod h1:Wz2wFJntZFmLGo7pLDXZ3wYk5hyc0Mb+SkHhDDXT+lU= google.golang.org/genproto/googleapis/api v0.0.0-20260831171406-18b4a7587f8a h1:i3TAXhpKc7TUP1VAPiBBrv45kamjoizCC3rOC0cAbOs= google.golang.org/genproto/googleapis/api v0.0.0-20260831171406-18b4a7587f8a/go.mod h1:CvYJHpbzPlT0fb/PsgtAamdwru/GVxUsomFdXTpOTI8= google.golang.org/genproto/googleapis/rpc v0.0.0-20260831171406-18b4a7587f8a h1:3Dnd1cDaZlB68lziofO+bJXpjOy8UfRv8Unt+yH8tQ4= diff --git a/module/services/accounts/code/pkg/auditstore/bigquerystore/bigquerystore.go b/module/services/accounts/code/pkg/auditstore/bigquerystore/bigquerystore.go new file mode 100644 index 000000000..d109e5c5b --- /dev/null +++ b/module/services/accounts/code/pkg/auditstore/bigquerystore/bigquerystore.go @@ -0,0 +1,340 @@ +// Package bigquerystore is the BigQuery adapter of the audit store swap +// (ADR 0009): the write half of business.AuditStore. +// +// The writer's authority is deliberately minimal — bigquery.datasets.get, +// bigquery.tables.create, bigquery.tables.get and bigquery.tables.updateData, +// and never bigquery.jobs.create — so it runs no load, query or DML job. Rows +// are appended with streaming inserts (tabledata.insertAll) and tables are +// created, never altered, at startup. Credentials are Application Default +// Credentials only: the caller builds the client, and on Kubernetes that is the +// pod's workload identity. +package bigquerystore + +import ( + "context" + "errors" + "fmt" + "net/http" + "slices" + "strings" + "time" + + "accounts/pkg/business" + + "cloud.google.com/go/bigquery" + "google.golang.org/api/googleapi" +) + +// Table names. Both live in the configured dataset. +const ( + EventsTable = "audit_events" + DetailsTable = "audit_event_details" +) + +// maxRowsPerInsert is BigQuery's recommended streaming request size; a relay +// batch larger than it is sent as several requests. +const maxRowsPerInsert = 500 + +// timestampLayout is BigQuery's canonical TIMESTAMP text at its microsecond +// precision, which is also Postgres's. +const timestampLayout = "2006-01-02T15:04:05.000000Z" + +// Config names where the store writes. +type Config struct { + // Dataset must already exist: the writer may read it, never create it. + Dataset string + // ContentDetailRetention is the details table's partition expiration: how + // long a content-class event's full details are kept. + ContentDetailRetention time.Duration +} + +// rowInserter is the streaming-insert seam; *bigquery.Inserter satisfies it. +type rowInserter interface { + Put(ctx context.Context, src any) error +} + +// Store appends relay batches to the events and details tables. +type Store struct { + client *bigquery.Client + dataset string + retention time.Duration + events rowInserter + details rowInserter +} + +// New builds a store over client, which the caller constructs with +// Application Default Credentials. +func New(client *bigquery.Client, cfg Config) (*Store, error) { + if client == nil { + return nil, errors.New("bigquery audit store: client is required") + } + if strings.TrimSpace(cfg.Dataset) == "" { + return nil, errors.New("bigquery audit store: dataset is required") + } + if cfg.ContentDetailRetention < 24*time.Hour { + return nil, errors.New("bigquery audit store: content detail retention must be at least one day") + } + dataset := client.Dataset(cfg.Dataset) + events := dataset.Table(EventsTable).Inserter() + details := dataset.Table(DetailsTable).Inserter() + // A row that does not fit the schema fails its whole request rather than + // being dropped or stripped: the batch is retried, never shortened. + events.SkipInvalidRows, events.IgnoreUnknownValues = false, false + details.SkipInvalidRows, details.IgnoreUnknownValues = false, false + return &Store{ + client: client, + dataset: cfg.Dataset, + retention: cfg.ContentDetailRetention, + events: events, + details: details, + }, nil +} + +// EventsSchema is the events table: every event's envelope, the SHA-256 of its +// canonical details, and the details themselves for security-class events. +func EventsSchema() bigquery.Schema { + return bigquery.Schema{ + {Name: "event_id", Type: bigquery.StringFieldType, Required: true}, + {Name: "deployment_id", Type: bigquery.StringFieldType, Required: true}, + {Name: "org_id", Type: bigquery.StringFieldType, Description: "Empty for a platform event."}, + {Name: "actor_id", Type: bigquery.StringFieldType}, + {Name: "actor_type", Type: bigquery.StringFieldType, Required: true}, + {Name: "event_type", Type: bigquery.StringFieldType, Required: true}, + {Name: "schema_version", Type: bigquery.IntegerFieldType, Required: true}, + {Name: "resource", Type: bigquery.StringFieldType, Required: true, Description: "The target's resource type."}, + {Name: "resource_id", Type: bigquery.StringFieldType, Description: "The target's id."}, + {Name: "occurred_at", Type: bigquery.TimestampFieldType, Required: true}, + {Name: "ip_address", Type: bigquery.StringFieldType}, + {Name: "impersonated_by", Type: bigquery.StringFieldType}, + {Name: "is_impersonated", Type: bigquery.BooleanFieldType, Required: true}, + {Name: "client_id", Type: bigquery.StringFieldType}, + {Name: "retention_class", Type: bigquery.StringFieldType, Required: true}, + {Name: "details_sha256", Type: bigquery.StringFieldType, Required: true, Description: "Lowercase hex SHA-256 of the canonical details string."}, + {Name: "details", Type: bigquery.StringFieldType, Description: "Canonical details JSON; security-class events only."}, + } +} + +// DetailsSchema is the details table: the full details of content-class +// events, kept for the content window. +func DetailsSchema() bigquery.Schema { + return bigquery.Schema{ + {Name: "event_id", Type: bigquery.StringFieldType, Required: true}, + {Name: "deployment_id", Type: bigquery.StringFieldType, Required: true}, + {Name: "org_id", Type: bigquery.StringFieldType}, + {Name: "event_type", Type: bigquery.StringFieldType, Required: true}, + {Name: "occurred_at", Type: bigquery.TimestampFieldType, Required: true}, + {Name: "details_sha256", Type: bigquery.StringFieldType, Required: true}, + {Name: "details", Type: bigquery.StringFieldType, Required: true}, + } +} + +// tableSpec is what Ensure creates and what it requires of a table it finds. +type tableSpec struct { + name string + description string + schema bigquery.Schema + expiration time.Duration +} + +func (s *Store) tableSpecs() []tableSpec { + return []tableSpec{ + { + name: EventsTable, + description: "Audit store of record: one row per audit event (ADR 0009).", + schema: EventsSchema(), + }, + { + name: DetailsTable, + description: "Full details of content-class audit events, kept for the content window (ADR 0009).", + schema: DetailsSchema(), + expiration: s.retention, + }, + } +} + +// metadataFor is the table every spec is created with: partitioned by day on +// occurred_at, clustered by organization, and — for the details table — +// expiring each partition at the content window. +func metadataFor(spec tableSpec) *bigquery.TableMetadata { + return &bigquery.TableMetadata{ + Name: spec.name, + Description: spec.description, + Schema: spec.schema, + TimePartitioning: &bigquery.TimePartitioning{ + Type: bigquery.DayPartitioningType, + Field: "occurred_at", + Expiration: spec.expiration, + }, + Clustering: &bigquery.Clustering{Fields: []string{"org_id"}}, + } +} + +// Ensure verifies the dataset exists, creates each table that is missing, and +// refuses a table it finds that does not match: the writer holds no +// bigquery.tables.update, so a mismatch is the deployment's to fix, and a +// details table expiring at a window other than the configured one would keep +// content longer or shorter than the deployment declared. +func (s *Store) Ensure(ctx context.Context) error { + dataset := s.client.Dataset(s.dataset) + if _, err := dataset.Metadata(ctx); err != nil { + if isNotFound(err) { + return fmt.Errorf("bigquery audit store: dataset %q does not exist; the deployment creates it", s.dataset) + } + return fmt.Errorf("bigquery audit store: read dataset %q: %w", s.dataset, err) + } + for _, spec := range s.tableSpecs() { + table := dataset.Table(spec.name) + existing, err := table.Metadata(ctx) + if isNotFound(err) { + if err := table.Create(ctx, metadataFor(spec)); err != nil && !isAlreadyExists(err) { + return fmt.Errorf("bigquery audit store: create table %s: %w", spec.name, err) + } + // Another replica may have created it first; read back what exists. + if existing, err = table.Metadata(ctx); err != nil { + return fmt.Errorf("bigquery audit store: read table %s: %w", spec.name, err) + } + } else if err != nil { + return fmt.Errorf("bigquery audit store: read table %s: %w", spec.name, err) + } + if err := conforms(spec, existing); err != nil { + return fmt.Errorf("bigquery audit store: table %s.%s: %w", s.dataset, spec.name, err) + } + } + return nil +} + +// conforms reports how an existing table differs from what the store writes. +func conforms(spec tableSpec, existing *bigquery.TableMetadata) error { + partitioning := existing.TimePartitioning + if partitioning == nil || partitioning.Field != "occurred_at" { + return errors.New("is not partitioned on occurred_at") + } + if partitioning.Expiration != spec.expiration { + return fmt.Errorf("partitions expire after %s, the configured window is %s", partitioning.Expiration, spec.expiration) + } + columns := make(map[string]*bigquery.FieldSchema, len(existing.Schema)) + for _, field := range existing.Schema { + columns[field.Name] = field + } + for _, want := range spec.schema { + got, ok := columns[want.Name] + if !ok { + return fmt.Errorf("has no column %s", want.Name) + } + if got.Type != want.Type || got.Repeated { + return fmt.Errorf("column %s is %s, want %s", want.Name, got.Type, want.Type) + } + } + return nil +} + +// AppendAuditBatch implements business.AuditStoreWriter. Every record goes to +// the events table; a content-class record's details go to the details table +// instead of the events row. Each row's insert id is its event id, so a batch +// delivered again is collapsed by BigQuery's streaming deduplication. +func (s *Store) AppendAuditBatch(ctx context.Context, batch business.AuditBatch) error { + var events, details []bigquery.ValueSaver + for _, record := range batch.Records { + events = append(events, EventRow(batch.DeploymentID, record)) + if record.Retention != business.RetentionSecurity { + details = append(details, DetailRow(batch.DeploymentID, record)) + } + } + if err := putInChunks(ctx, s.events, events); err != nil { + return fmt.Errorf("bigquery audit store: append to %s: %w", EventsTable, err) + } + if err := putInChunks(ctx, s.details, details); err != nil { + return fmt.Errorf("bigquery audit store: append to %s: %w", DetailsTable, err) + } + return nil +} + +func putInChunks(ctx context.Context, inserter rowInserter, rows []bigquery.ValueSaver) error { + for chunk := range slices.Chunk(rows, maxRowsPerInsert) { + if err := inserter.Put(ctx, chunk); err != nil { + return describePutError(err) + } + } + return nil +} + +// describePutError names the first rejected row: a PutMultiError's own message +// says only how many rows failed. +func describePutError(err error) error { + var multi bigquery.PutMultiError + if errors.As(err, &multi) && len(multi) > 0 { + first := multi[0] + return fmt.Errorf("%w (row %d, insert id %s: %v)", err, first.RowIndex, first.InsertID, first.Errors) + } + return err +} + +// Row is one streaming-insert row keyed by its insert id. +type Row struct { + InsertID string + Values map[string]bigquery.Value +} + +// Save implements bigquery.ValueSaver. +func (r Row) Save() (map[string]bigquery.Value, string, error) { + return r.Values, r.InsertID, nil +} + +// EventRow is a record's events-table row. +func EventRow(deploymentID string, record business.AuditRecord) Row { + entry := record.Entry + values := map[string]bigquery.Value{ + "event_id": entry.ID, + "deployment_id": deploymentID, + "org_id": nullable(entry.OrgID), + "actor_id": nullable(entry.ActorID), + "actor_type": entry.ActorType, + "event_type": string(entry.EventType), + "schema_version": entry.SchemaVersion, + "resource": entry.Resource, + "resource_id": nullable(entry.ResourceID), + "occurred_at": entry.CreatedAt.UTC().Format(timestampLayout), + "ip_address": nullable(entry.IPAddress), + "impersonated_by": nullable(entry.ImpersonatedBy), + "is_impersonated": entry.IsImpersonated, + "client_id": nullable(entry.ClientID), + "retention_class": string(record.Retention), + "details_sha256": record.DetailsSHA256, + "details": nil, + } + if record.Retention == business.RetentionSecurity { + values["details"] = record.Details + } + return Row{InsertID: entry.ID, Values: values} +} + +// DetailRow is a content-class record's details-table row. +func DetailRow(deploymentID string, record business.AuditRecord) Row { + entry := record.Entry + return Row{InsertID: entry.ID, Values: map[string]bigquery.Value{ + "event_id": entry.ID, + "deployment_id": deploymentID, + "org_id": nullable(entry.OrgID), + "event_type": string(entry.EventType), + "occurred_at": entry.CreatedAt.UTC().Format(timestampLayout), + "details_sha256": record.DetailsSHA256, + "details": record.Details, + }} +} + +func nullable(value string) bigquery.Value { + if value == "" { + return nil + } + return value +} + +func isNotFound(err error) bool { + var apiErr *googleapi.Error + return errors.As(err, &apiErr) && apiErr.Code == http.StatusNotFound +} + +func isAlreadyExists(err error) bool { + var apiErr *googleapi.Error + return errors.As(err, &apiErr) && apiErr.Code == http.StatusConflict +} diff --git a/module/services/accounts/code/pkg/auditstore/bigquerystore/bigquerystore_test.go b/module/services/accounts/code/pkg/auditstore/bigquerystore/bigquerystore_test.go new file mode 100644 index 000000000..32677ac9f --- /dev/null +++ b/module/services/accounts/code/pkg/auditstore/bigquerystore/bigquerystore_test.go @@ -0,0 +1,222 @@ +package bigquerystore + +import ( + "context" + "errors" + "fmt" + "testing" + "time" + + "accounts/pkg/business" + + "cloud.google.com/go/bigquery" + "github.com/stretchr/testify/require" +) + +var occurredAt = time.Date(2026, 10, 2, 14, 30, 5, 123456000, time.UTC) + +func record(t *testing.T, class business.AuditRetentionClass, org string) business.AuditRecord { + t.Helper() + r, err := business.NewAuditRecord(business.AuditEntry{ + ID: business.NewIDString(), + OrgID: org, + ActorID: "11111111-1111-1111-1111-111111111111", + ActorType: business.ActorTypeUser, + EventType: business.EventAuthLogin, + SchemaVersion: 1, + Resource: "session", + ResourceID: "session-1", + Payload: map[string]any{"method": "password"}, + CreatedAt: occurredAt, + }, class) + require.NoError(t, err) + return r +} + +func TestEventRowCarriesTheEnvelopeAndOnlySecurityDetails(t *testing.T) { + security := record(t, business.RetentionSecurity, "22222222-2222-2222-2222-222222222222") + row := EventRow("deployment-1", security) + require.Equal(t, security.Entry.ID, row.InsertID, "the event id is the streaming insert id") + require.Equal(t, map[string]bigquery.Value{ + "event_id": security.Entry.ID, + "deployment_id": "deployment-1", + "org_id": "22222222-2222-2222-2222-222222222222", + "actor_id": "11111111-1111-1111-1111-111111111111", + "actor_type": "user", + "event_type": "saas.auth.login", + "schema_version": 1, + "resource": "session", + "resource_id": "session-1", + "occurred_at": "2026-10-02T14:30:05.123456Z", + "ip_address": nil, + "impersonated_by": nil, + "is_impersonated": false, + "client_id": nil, + "retention_class": "security", + "details_sha256": security.DetailsSHA256, + "details": `{"method":"password"}`, + }, row.Values) + + content := record(t, business.RetentionContent, "") + row = EventRow("deployment-1", content) + require.Nil(t, row.Values["details"], "a content-class event's details are not in the events table") + require.Nil(t, row.Values["org_id"], "a platform event has no organization") + require.Equal(t, content.DetailsSHA256, row.Values["details_sha256"], "its hash is") + + values, insertID, err := row.Save() + require.NoError(t, err) + require.Equal(t, content.Entry.ID, insertID) + require.Equal(t, row.Values, values) +} + +func TestDetailRow(t *testing.T) { + content := record(t, business.RetentionContent, "22222222-2222-2222-2222-222222222222") + row := DetailRow("deployment-1", content) + require.Equal(t, content.Entry.ID, row.InsertID) + require.Equal(t, map[string]bigquery.Value{ + "event_id": content.Entry.ID, + "deployment_id": "deployment-1", + "org_id": "22222222-2222-2222-2222-222222222222", + "event_type": "saas.auth.login", + "occurred_at": "2026-10-02T14:30:05.123456Z", + "details_sha256": content.DetailsSHA256, + "details": `{"method":"password"}`, + }, row.Values) +} + +// fakeInserter records each streaming request. +type fakeInserter struct { + requests [][]Row + fail error +} + +func (f *fakeInserter) Put(_ context.Context, src any) error { + if f.fail != nil { + return f.fail + } + savers := src.([]bigquery.ValueSaver) + request := make([]Row, len(savers)) + for i, saver := range savers { + request[i] = saver.(Row) + } + f.requests = append(f.requests, request) + return nil +} + +func (f *fakeInserter) ids() []string { + var ids []string + for _, request := range f.requests { + for _, row := range request { + ids = append(ids, row.InsertID) + } + } + return ids +} + +func TestAppendAuditBatchRoutesDetailsByRetentionClass(t *testing.T) { + events, details := &fakeInserter{}, &fakeInserter{} + store := &Store{events: events, details: details} + security := record(t, business.RetentionSecurity, "") + content := record(t, business.RetentionContent, "") + + require.NoError(t, store.AppendAuditBatch(context.Background(), business.AuditBatch{ + DeploymentID: "deployment-1", + Records: []business.AuditRecord{security, content}, + })) + require.Equal(t, []string{security.Entry.ID, content.Entry.ID}, events.ids(), "every event has an events row, in batch order") + require.Equal(t, []string{content.Entry.ID}, details.ids(), "only a content-class event has a details row") +} + +func TestAppendAuditBatchSendsAtMostFiveHundredRowsPerRequest(t *testing.T) { + events, details := &fakeInserter{}, &fakeInserter{} + store := &Store{events: events, details: details} + batch := business.AuditBatch{DeploymentID: "deployment-1"} + for range 1200 { + batch.Records = append(batch.Records, record(t, business.RetentionContent, "")) + } + require.NoError(t, store.AppendAuditBatch(context.Background(), batch)) + for name, inserter := range map[string]*fakeInserter{"events": events, "details": details} { + var sizes []int + for _, request := range inserter.requests { + sizes = append(sizes, len(request)) + } + require.Equal(t, []int{500, 500, 200}, sizes, name) + } +} + +func TestAppendAuditBatchFailsWholeAndNamesTheRejectedRow(t *testing.T) { + rejected := bigquery.PutMultiError{{InsertID: "event-7", RowIndex: 7, Errors: bigquery.MultiError{errors.New("no such field: extra")}}} + events, details := &fakeInserter{fail: rejected}, &fakeInserter{} + store := &Store{events: events, details: details} + + err := store.AppendAuditBatch(context.Background(), business.AuditBatch{ + DeploymentID: "deployment-1", + Records: []business.AuditRecord{record(t, business.RetentionContent, "")}, + }) + require.ErrorContains(t, err, "row 7, insert id event-7") + require.ErrorContains(t, err, "no such field: extra") + require.Empty(t, details.requests, "a failed events append stops the batch; the relay retries it whole") +} + +func TestTablesArePartitionedByDayOnOccurredAtAndClusteredByOrganization(t *testing.T) { + store := &Store{retention: 90 * 24 * time.Hour} + specs := store.tableSpecs() + require.Len(t, specs, 2) + for _, spec := range specs { + meta := metadataFor(spec) + require.Equal(t, bigquery.DayPartitioningType, meta.TimePartitioning.Type, spec.name) + require.Equal(t, "occurred_at", meta.TimePartitioning.Field, spec.name) + require.Equal(t, []string{"org_id"}, meta.Clustering.Fields, spec.name) + } + require.Equal(t, EventsTable, specs[0].name) + require.Zero(t, metadataFor(specs[0]).TimePartitioning.Expiration, "the events table is kept for the compliance window, which no code shortens") + require.Equal(t, DetailsTable, specs[1].name) + require.Equal(t, 90*24*time.Hour, metadataFor(specs[1]).TimePartitioning.Expiration, "details expire at the content window") +} + +func TestEnsureRefusesATableItCannotWriteAsConfigured(t *testing.T) { + store := &Store{retention: 30 * 24 * time.Hour} + details := store.tableSpecs()[1] + matching := metadataFor(details) + require.NoError(t, conforms(details, matching)) + + for name, mutate := range map[string]func(*bigquery.TableMetadata){ + "different window": func(m *bigquery.TableMetadata) { m.TimePartitioning.Expiration = 400 * 24 * time.Hour }, + "not partitioned": func(m *bigquery.TableMetadata) { m.TimePartitioning = nil }, + "other partition": func(m *bigquery.TableMetadata) { m.TimePartitioning.Field = "ingested_at" }, + "missing column": func(m *bigquery.TableMetadata) { m.Schema = m.Schema[1:] }, + "retyped column": func(m *bigquery.TableMetadata) { + m.Schema = append(bigquery.Schema{}, m.Schema...) + retyped := *m.Schema[4] + retyped.Type = bigquery.DateFieldType + m.Schema[4] = &retyped + }, + } { + existing := metadataFor(details) + mutate(existing) + require.Error(t, conforms(details, existing), name) + } +} + +func TestNewValidatesItsConfiguration(t *testing.T) { + _, err := New(nil, Config{Dataset: "audit", ContentDetailRetention: 24 * time.Hour}) + require.Error(t, err) + client := &bigquery.Client{} + for name, cfg := range map[string]Config{ + "no dataset": {ContentDetailRetention: 24 * time.Hour}, + "sub-day content days": {Dataset: "audit", ContentDetailRetention: time.Hour}, + } { + _, err := New(client, cfg) + require.Error(t, err, name) + } +} + +func ExampleEventRow() { + r, _ := business.NewAuditRecord(business.AuditEntry{ + ID: "00000000-0000-0000-0000-000000000001", ActorType: "system", EventType: business.EventRoleCreated, + Resource: "role", CreatedAt: occurredAt, Payload: map[string]any{"name": "auditor"}, + }, business.RetentionSecurity) + row := EventRow("deployment-1", r) + fmt.Println(row.InsertID, row.Values["occurred_at"], row.Values["details"]) + // Output: 00000000-0000-0000-0000-000000000001 2026-10-02T14:30:05.123456Z {"name":"auditor"} +} diff --git a/module/services/accounts/code/pkg/auditstore/bigquerystore/emulator_test.go b/module/services/accounts/code/pkg/auditstore/bigquerystore/emulator_test.go new file mode 100644 index 000000000..126ec0bb6 --- /dev/null +++ b/module/services/accounts/code/pkg/auditstore/bigquerystore/emulator_test.go @@ -0,0 +1,140 @@ +package bigquerystore + +import ( + "context" + "os" + "strings" + "testing" + "time" + + "accounts/pkg/business" + + "cloud.google.com/go/bigquery" + "github.com/google/uuid" + "github.com/stretchr/testify/require" + "google.golang.org/api/iterator" + "google.golang.org/api/option" +) + +// The adapter against a BigQuery emulator, when one is reachable: +// +// docker run --rm -p 9050:9050 ghcr.io/goccy/bigquery-emulator --project=test-project +// AUDIT_BIGQUERY_EMULATOR_HOST=localhost:9050 go test ./pkg/auditstore/bigquerystore +// +// It is skipped otherwise, and no gate runs it. The emulator does not implement +// BigQuery's streaming insert-id deduplication or partition expiry, so this +// proves the table shapes Ensure creates, its idempotence, its refusals, and +// that a batch's rows land where the retention class sends them — not the +// deduplication window, which is BigQuery's own. +func emulatorClient(t *testing.T) *bigquery.Client { + t.Helper() + host := strings.TrimSpace(os.Getenv("AUDIT_BIGQUERY_EMULATOR_HOST")) + if host == "" { + t.Skip("AUDIT_BIGQUERY_EMULATOR_HOST is not set; see the comment above for how to run against an emulator") + } + client, err := bigquery.NewClient(context.Background(), "test-project", + option.WithEndpoint("http://"+host), option.WithoutAuthentication()) + require.NoError(t, err) + t.Cleanup(func() { _ = client.Close() }) + return client +} + +// emulatorDataset creates a dataset for one test. The adapter itself never +// creates one: the deployment does. +func emulatorDataset(t *testing.T, client *bigquery.Client) string { + t.Helper() + name := "audit_" + strings.ReplaceAll(uuid.NewString(), "-", "")[:12] + require.NoError(t, client.Dataset(name).Create(context.Background(), &bigquery.DatasetMetadata{})) + t.Cleanup(func() { _ = client.Dataset(name).DeleteWithContents(context.Background()) }) + return name +} + +func TestEmulatorEnsureRefusesAMissingDataset(t *testing.T) { + client := emulatorClient(t) + store, err := New(client, Config{Dataset: "audit_absent", ContentDetailRetention: 30 * 24 * time.Hour}) + require.NoError(t, err) + require.ErrorContains(t, store.Ensure(context.Background()), "does not exist") +} + +func TestEmulatorEnsureCreatesBothTablesOnce(t *testing.T) { + client := emulatorClient(t) + ctx := context.Background() + dataset := emulatorDataset(t, client) + store, err := New(client, Config{Dataset: dataset, ContentDetailRetention: 30 * 24 * time.Hour}) + require.NoError(t, err) + + require.NoError(t, store.Ensure(ctx)) + require.NoError(t, store.Ensure(ctx), "a second start finds both tables and accepts them") + + for table, expiration := range map[string]time.Duration{EventsTable: 0, DetailsTable: 30 * 24 * time.Hour} { + meta, err := client.Dataset(dataset).Table(table).Metadata(ctx) + require.NoError(t, err, table) + require.NotNil(t, meta.TimePartitioning, table) + require.Equal(t, "occurred_at", meta.TimePartitioning.Field, table) + require.Equal(t, expiration, meta.TimePartitioning.Expiration, table) + } + + other, err := New(client, Config{Dataset: dataset, ContentDetailRetention: 400 * 24 * time.Hour}) + require.NoError(t, err) + require.ErrorContains(t, other.Ensure(ctx), "partitions expire", + "a details table expiring at another window is refused, never silently kept") +} + +func TestEmulatorAppendRoutesRecordsByRetentionClass(t *testing.T) { + client := emulatorClient(t) + ctx := context.Background() + dataset := emulatorDataset(t, client) + store, err := New(client, Config{Dataset: dataset, ContentDetailRetention: 30 * 24 * time.Hour}) + require.NoError(t, err) + require.NoError(t, store.Ensure(ctx)) + + security := record(t, business.RetentionSecurity, "22222222-2222-2222-2222-222222222222") + content := record(t, business.RetentionContent, "") + require.NoError(t, store.AppendAuditBatch(ctx, business.AuditBatch{ + ID: "batch-1", DeploymentID: "deployment-1", ComposedAt: occurredAt, + Records: []business.AuditRecord{security, content}, + })) + + type eventRow struct { + EventID string `bigquery:"event_id"` + RetentionClass string `bigquery:"retention_class"` + DetailsSHA256 string `bigquery:"details_sha256"` + Details bigquery.NullString `bigquery:"details"` + OrgID bigquery.NullString `bigquery:"org_id"` + OccurredAt time.Time `bigquery:"occurred_at"` + } + events := map[string]eventRow{} + it := client.Dataset(dataset).Table(EventsTable).Read(ctx) + for { + var row eventRow + err := it.Next(&row) + if err == iterator.Done { + break + } + require.NoError(t, err) + events[row.EventID] = row + } + require.Len(t, events, 2) + require.Equal(t, security.Details, events[security.Entry.ID].Details.StringVal, "a security event keeps its details in the events table") + require.False(t, events[content.Entry.ID].Details.Valid, "a content event does not") + require.False(t, events[content.Entry.ID].OrgID.Valid, "a platform event has no organization") + require.Equal(t, content.DetailsSHA256, events[content.Entry.ID].DetailsSHA256) + require.True(t, occurredAt.Equal(events[security.Entry.ID].OccurredAt)) + + var details []string + it = client.Dataset(dataset).Table(DetailsTable).Read(ctx) + for { + var row struct { + EventID string `bigquery:"event_id"` + Details string `bigquery:"details"` + } + err := it.Next(&row) + if err == iterator.Done { + break + } + require.NoError(t, err) + require.Equal(t, content.Details, row.Details) + details = append(details, row.EventID) + } + require.Equal(t, []string{content.Entry.ID}, details, "only the content event has a details row") +} diff --git a/module/services/accounts/code/pkg/auditstore/gcsarchive/gcsarchive.go b/module/services/accounts/code/pkg/auditstore/gcsarchive/gcsarchive.go new file mode 100644 index 000000000..8d3222bc4 --- /dev/null +++ b/module/services/accounts/code/pkg/auditstore/gcsarchive/gcsarchive.go @@ -0,0 +1,190 @@ +// Package gcsarchive writes the audit store swap's locked archive (ADR 0009) to +// a Google Cloud Storage bucket: one JSON Lines object per relay batch. +// +// The writer's authority is storage.objects.create alone. It never overwrites — +// every write carries a does-not-exist precondition — and never deletes; the +// bucket's retention lock (GCS Bucket Lock), which the deployment configures, +// is what keeps anyone else from doing either before the window ends. +// Credentials are Application Default Credentials only: the caller builds the +// client, and on Kubernetes that is the pod's workload identity. +package gcsarchive + +import ( + "bytes" + "context" + "encoding/json" + "errors" + "fmt" + "hash/crc32" + "net/http" + "strings" + "time" + + "accounts/pkg/business" + + "cloud.google.com/go/storage" + "google.golang.org/api/googleapi" +) + +// ContentType is every archive object's media type. +const ContentType = "application/x-ndjson" + +// objectCreator creates one object only if none of that name exists. +type objectCreator interface { + CreateObject(ctx context.Context, name string, body []byte) error +} + +// Archive writes relay batches as archive objects. +type Archive struct { + bucket objectCreator +} + +// New writes into bucket through client, which the caller constructs with +// Application Default Credentials. +func New(client *storage.Client, bucket string) (*Archive, error) { + if client == nil { + return nil, errors.New("gcs audit archive: client is required") + } + if strings.TrimSpace(bucket) == "" { + return nil, errors.New("gcs audit archive: bucket is required") + } + return &Archive{bucket: gcsBucket{handle: client.Bucket(bucket)}}, nil +} + +// WriteAuditBatch implements business.AuditArchive. +func (a *Archive) WriteAuditBatch(ctx context.Context, batch business.AuditBatch) error { + if len(batch.Records) == 0 { + return errors.New("gcs audit archive: refusing an empty batch") + } + name, err := ObjectName(batch) + if err != nil { + return err + } + body, err := Encode(batch) + if err != nil { + return err + } + if err := a.bucket.CreateObject(ctx, name, body); err != nil { + return fmt.Errorf("gcs audit archive: write %s: %w", name, err) + } + return nil +} + +// ObjectName names a batch's object: +// +// ///
/-.jsonl +// +// The date path is when the relay composed the batch, so a day's objects list +// together and in delivery order. The batch id is unique to one delivery +// attempt: a retried batch is a new object, never a rewrite of an old one. +func ObjectName(batch business.AuditBatch) (string, error) { + if !safeSegment(batch.DeploymentID) || !safeSegment(batch.ID) { + return "", fmt.Errorf("gcs audit archive: deployment id %q and batch id %q must be plain path segments", batch.DeploymentID, batch.ID) + } + at := batch.ComposedAt.UTC() + return fmt.Sprintf("%s/%s/%s-%s.jsonl", + batch.DeploymentID, at.Format("2006/01/02"), at.Format("20060102T150405.000000000Z"), batch.ID), nil +} + +func safeSegment(segment string) bool { + if segment == "" || segment == "." || segment == ".." { + return false + } + for _, r := range segment { + if !(r >= 'a' && r <= 'z' || r >= 'A' && r <= 'Z' || r >= '0' && r <= '9' || r == '-' || r == '_' || r == '.') { + return false + } + } + return true +} + +// Line is one archive line: an event's envelope, the SHA-256 of its canonical +// details, and — for a security-class event only — the details themselves. A +// content-class line is content-free. Details is the canonical string as the +// store keeps it, so a reader verifies a line by hashing that string. +type Line struct { + EventID string `json:"event_id"` + DeploymentID string `json:"deployment_id"` + BatchID string `json:"batch_id"` + OrgID string `json:"org_id,omitempty"` + ActorID string `json:"actor_id,omitempty"` + ActorType string `json:"actor_type"` + EventType string `json:"event_type"` + SchemaVersion int `json:"schema_version"` + Resource string `json:"resource"` + ResourceID string `json:"resource_id,omitempty"` + OccurredAt string `json:"occurred_at"` + IPAddress string `json:"ip_address,omitempty"` + ImpersonatedBy string `json:"impersonated_by,omitempty"` + IsImpersonated bool `json:"is_impersonated"` + ClientID string `json:"client_id,omitempty"` + RetentionClass string `json:"retention_class"` + DetailsSHA256 string `json:"details_sha256"` + Details string `json:"details,omitempty"` +} + +// Encode renders a batch as JSON Lines, one line per record in batch order. +func Encode(batch business.AuditBatch) ([]byte, error) { + var buf bytes.Buffer + encoder := json.NewEncoder(&buf) + encoder.SetEscapeHTML(false) + for _, record := range batch.Records { + entry := record.Entry + line := Line{ + EventID: entry.ID, + DeploymentID: batch.DeploymentID, + BatchID: batch.ID, + OrgID: entry.OrgID, + ActorID: entry.ActorID, + ActorType: entry.ActorType, + EventType: string(entry.EventType), + SchemaVersion: entry.SchemaVersion, + Resource: entry.Resource, + ResourceID: entry.ResourceID, + OccurredAt: entry.CreatedAt.UTC().Format(time.RFC3339Nano), + IPAddress: entry.IPAddress, + ImpersonatedBy: entry.ImpersonatedBy, + IsImpersonated: entry.IsImpersonated, + ClientID: entry.ClientID, + RetentionClass: string(record.Retention), + DetailsSHA256: record.DetailsSHA256, + } + if record.Retention == business.RetentionSecurity { + line.Details = record.Details + } + if err := encoder.Encode(line); err != nil { + return nil, fmt.Errorf("gcs audit archive: encode event %s: %w", entry.ID, err) + } + } + return buf.Bytes(), nil +} + +// gcsBucket creates objects in one bucket. +type gcsBucket struct { + handle *storage.BucketHandle +} + +var castagnoli = crc32.MakeTable(crc32.Castagnoli) + +// CreateObject uploads body in one request under a does-not-exist +// precondition, with its CRC32C so a corrupted upload is refused. The client +// retries the request; if a retry finds the object already there, that object +// is this upload's own — the name carries a batch id no other attempt uses — +// so the precondition failure is the acknowledgement it lost. +func (b gcsBucket) CreateObject(ctx context.Context, name string, body []byte) error { + writer := b.handle.Object(name).If(storage.Conditions{DoesNotExist: true}).NewWriter(ctx) + writer.ContentType = ContentType + writer.ChunkSize = len(body) + writer.CRC32C = crc32.Checksum(body, castagnoli) + writer.SendCRC32C = true + if _, err := writer.Write(body); err != nil { + _ = writer.Close() + return err + } + err := writer.Close() + var apiErr *googleapi.Error + if errors.As(err, &apiErr) && apiErr.Code == http.StatusPreconditionFailed { + return nil + } + return err +} diff --git a/module/services/accounts/code/pkg/auditstore/gcsarchive/gcsarchive_test.go b/module/services/accounts/code/pkg/auditstore/gcsarchive/gcsarchive_test.go new file mode 100644 index 000000000..a6f9546c7 --- /dev/null +++ b/module/services/accounts/code/pkg/auditstore/gcsarchive/gcsarchive_test.go @@ -0,0 +1,134 @@ +package gcsarchive + +import ( + "bufio" + "bytes" + "context" + "encoding/json" + "errors" + "testing" + "time" + + "accounts/pkg/business" + + "github.com/stretchr/testify/require" +) + +var composedAt = time.Date(2026, 10, 2, 14, 30, 5, 987654321, time.UTC) + +func batchOf(t *testing.T, id string, classes ...business.AuditRetentionClass) business.AuditBatch { + t.Helper() + batch := business.AuditBatch{ID: id, DeploymentID: "deployment-1", ComposedAt: composedAt} + for _, class := range classes { + record, err := business.NewAuditRecord(business.AuditEntry{ + ID: business.NewIDString(), + OrgID: "22222222-2222-2222-2222-222222222222", + ActorID: "11111111-1111-1111-1111-111111111111", + ActorType: business.ActorTypeUser, + EventType: business.EventAuthLogin, + SchemaVersion: 1, + Resource: "session", + Payload: map[string]any{"method": "password", "note": "&"}, + CreatedAt: composedAt.Add(-time.Second), + }, class) + require.NoError(t, err) + batch.Records = append(batch.Records, record) + } + return batch +} + +func TestObjectNameIsPerBatchUnderTheDeploymentAndDay(t *testing.T) { + name, err := ObjectName(batchOf(t, "4b6d0d1e-0f3c-4c1a-9a55-6f0c2b1d7e10")) + require.NoError(t, err) + require.Equal(t, "deployment-1/2026/10/02/20261002T143005.987654321Z-4b6d0d1e-0f3c-4c1a-9a55-6f0c2b1d7e10.jsonl", name) + + other, err := ObjectName(batchOf(t, "9e2f4a6b-1c3d-4e5f-8a7b-0c1d2e3f4a5b")) + require.NoError(t, err) + require.NotEqual(t, name, other, "a redelivered batch carries a new id, so it is a new object") + + for _, bad := range []business.AuditBatch{ + {ID: "batch", DeploymentID: "../escape", ComposedAt: composedAt}, + {ID: "a/b", DeploymentID: "deployment-1", ComposedAt: composedAt}, + {ID: "", DeploymentID: "deployment-1", ComposedAt: composedAt}, + } { + _, err := ObjectName(bad) + require.Error(t, err, "%+v", bad) + } +} + +func TestEncodeWritesSecurityInFullAndContentWithoutItsDetails(t *testing.T) { + batch := batchOf(t, "batch-1", business.RetentionSecurity, business.RetentionContent) + body, err := Encode(batch) + require.NoError(t, err) + + var lines []Line + scanner := bufio.NewScanner(bytes.NewReader(body)) + for scanner.Scan() { + var line Line + require.NoError(t, json.Unmarshal(scanner.Bytes(), &line)) + lines = append(lines, line) + } + require.Len(t, lines, 2, "one line per record, in batch order") + + security, content := lines[0], lines[1] + require.Equal(t, batch.Records[0].Entry.ID, security.EventID) + require.Equal(t, "security", security.RetentionClass) + require.Equal(t, `{"method":"password","note":"&"}`, security.Details) + require.Equal(t, business.AuditDetailsSHA256(security.Details), security.DetailsSHA256, + "a reader verifies a line by hashing its details string") + + require.Equal(t, batch.Records[1].Entry.ID, content.EventID) + require.Equal(t, "content", content.RetentionClass) + require.Empty(t, content.Details, "a content-class line is content-free") + require.Equal(t, batch.Records[1].DetailsSHA256, content.DetailsSHA256, "its hash proves what the details table holds") + require.NotContains(t, string(body[bytes.IndexByte(body, '\n')+1:]), "password") + + for _, line := range lines { + require.Equal(t, "deployment-1", line.DeploymentID) + require.Equal(t, "batch-1", line.BatchID) + require.Equal(t, "22222222-2222-2222-2222-222222222222", line.OrgID) + require.Equal(t, "2026-10-02T14:30:04.987654321Z", line.OccurredAt) + } +} + +// fakeBucket records created objects and refuses to overwrite one. +type fakeBucket struct { + objects map[string][]byte + fail error +} + +func (b *fakeBucket) CreateObject(_ context.Context, name string, body []byte) error { + if b.fail != nil { + return b.fail + } + if _, exists := b.objects[name]; exists { + return errors.New("precondition failed: object exists") + } + b.objects[name] = body + return nil +} + +func TestWriteAuditBatchWritesOneObjectPerBatch(t *testing.T) { + bucket := &fakeBucket{objects: map[string][]byte{}} + archive := &Archive{bucket: bucket} + + first := batchOf(t, "batch-1", business.RetentionSecurity, business.RetentionContent, business.RetentionContent) + require.NoError(t, archive.WriteAuditBatch(context.Background(), first)) + require.Len(t, bucket.objects, 1, "a whole batch is one object") + name, err := ObjectName(first) + require.NoError(t, err) + want, err := Encode(first) + require.NoError(t, err) + require.Equal(t, want, bucket.objects[name]) + + require.Error(t, archive.WriteAuditBatch(context.Background(), first), "an object is never overwritten") + require.Error(t, archive.WriteAuditBatch(context.Background(), batchOf(t, "batch-2")), "an empty batch is refused") + + bucket.fail = errors.New("bucket unavailable") + require.ErrorContains(t, archive.WriteAuditBatch(context.Background(), batchOf(t, "batch-3", business.RetentionContent)), "bucket unavailable") +} + +func TestNewValidatesItsConfiguration(t *testing.T) { + _, err := New(nil, "bucket") + require.Error(t, err) +} diff --git a/module/services/accounts/code/pkg/auth/pg/resolver.go b/module/services/accounts/code/pkg/auth/pg/resolver.go index f6d2f69f1..77283c2fd 100644 --- a/module/services/accounts/code/pkg/auth/pg/resolver.go +++ b/module/services/accounts/code/pkg/auth/pg/resolver.go @@ -42,6 +42,7 @@ const BootstrapAdminEmailEnv = "BOOTSTRAP_ADMIN_EMAIL" // user row. type Resolver struct { bootstrap BootstrapStore + audit business.AuditRecorder bootstrapAdminEmail *string signupMode auth.SignupMode } @@ -60,6 +61,15 @@ func NewResolver(bootstrap BootstrapStore) *Resolver { return &Resolver{bootstrap: bootstrap} } +// SetAuditRecorder sets how the resolver records the events it writes on the +// resolution transaction — a registration and an SSO just-in-time provisioning. +// It is required: Resolve refuses to run without it, so no provisioning path can +// commit unrecorded. Production passes the service's audit emitter, which writes +// the audit_events row, or under a swap value (ADR 0009) the queue row. +func (r *Resolver) SetAuditRecorder(recorder business.AuditRecorder) { + r.audit = recorder +} + // SetBootstrapAdminEmail makes runtime composition authoritative for the // one-time bootstrap identity. Tests and legacy standalone callers that do not // call this setter retain the environment-based compatibility path. @@ -127,6 +137,9 @@ func (r *Resolver) Resolve(ctx context.Context, c *auth.Claims, intent auth.Inte if r.bootstrap == nil { return nil, errors.New("pgauth: bootstrap store is required") } + if r.audit == nil { + return nil, errors.New("pgauth: audit recorder is required") + } // Authentication is an interactive boundary. Bound the complete database // operation, including pool acquisition, so a stale dependency becomes a @@ -200,7 +213,7 @@ func (r *Resolver) resolveInTx( // can never produce one: re-provisioning a known user into an org they // were removed from is a membership change, not a registration. if !found { - if err := r.emitUserRegistered(ctx, tx, c, userID, orgID, signupMethod); err != nil { + if err := r.emitUserRegistered(ctx, c, userID, orgID, signupMethod); err != nil { return nil, err } } @@ -856,7 +869,7 @@ func (r *Resolver) provisionSsoJit( if err := r.joinOrg(ctx, tx, orgID, userID, policy.defaultRole); err != nil { return uuid.Nil, uuid.Nil, "", err } - if err := r.emitSsoJitAudit(ctx, tx, c, userID, orgID); err != nil { + if err := r.emitSsoJitAudit(ctx, c, userID, orgID); err != nil { return uuid.Nil, uuid.Nil, "", err } return userID, orgID, policy.defaultRole, nil @@ -907,7 +920,7 @@ func (r *Resolver) provisionSsoInvite( if err := r.acceptInvitation(ctx, tx, invID, orgID, userID, role); err != nil { return uuid.Nil, uuid.Nil, "", err } - if err := r.emitSsoJitAudit(ctx, tx, c, userID, orgID); err != nil { + if err := r.emitSsoJitAudit(ctx, c, userID, orgID); err != nil { return uuid.Nil, uuid.Nil, "", err } return userID, orgID, role, nil @@ -951,19 +964,21 @@ func (r *Resolver) joinOrg(ctx context.Context, tx pgx.Tx, orgID, userID uuid.UU // transaction, so the audit trail commits atomically with the membership it // describes. The provider and org travel in the event for enterprise-tenant // forensics. -func (r *Resolver) emitSsoJitAudit(ctx context.Context, tx pgx.Tx, c *auth.Claims, userID, orgID uuid.UUID) error { - payload, err := json.Marshal(map[string]string{"provider": c.Provider}) - if err != nil { - return fmt.Errorf("pgauth: marshal sso jit audit payload: %w", err) - } - _, err = tx.Exec(ctx, ` - INSERT INTO audit_events ( - id, actor_id, actor_type, event_type, schema_version, resource, resource_id, org_id, payload - ) VALUES ($1, $2, 'user', $3, 1, 'organization', $4, $5, $6::jsonb)`, - business.NewID(), userID, string(business.EventAuthSSOJitProvisioned), orgID.String(), orgID, string(payload), - ) - if err != nil { - return fmt.Errorf("pgauth: insert sso jit audit event: %w", err) +// +// ctx carries the resolution transaction (WithAuthBootstrapTx binds it), which +// is where the recorder writes. The recorder writes the record and nothing +// else — no domain event, no tee job — which is all this path has ever written. +func (r *Resolver) emitSsoJitAudit(ctx context.Context, c *auth.Claims, userID, orgID uuid.UUID) error { + if err := r.audit.RecordTx(ctx, business.AuditEntry{ + ActorID: userID.String(), + ActorType: business.ActorTypeUser, + EventType: business.EventAuthSSOJitProvisioned, + Resource: "organization", + ResourceID: orgID.String(), + OrgID: orgID.String(), + Payload: map[string]any{"provider": c.Provider}, + }); err != nil { + return fmt.Errorf("pgauth: record sso jit audit event: %w", err) } return nil } @@ -984,7 +999,6 @@ func (r *Resolver) emitSsoJitAudit(ctx context.Context, tx pgx.Tx, c *auth.Claim // invitation is accepted. func (r *Resolver) emitUserRegistered( ctx context.Context, - tx pgx.Tx, c *auth.Claims, userID, orgID uuid.UUID, signupMethod string, @@ -994,33 +1008,30 @@ func (r *Resolver) emitUserRegistered( // still recorded, with a NULL org_id — the shape audit_events' tenant policy // already admits, scoped to the actor — rather than dropped: the person did // register, and no tenant should count them until they join one. - var org any + org := "" if orgID != uuid.Nil { - org = orgID + org = orgID.String() } fields := map[string]any{"email": strings.ToLower(c.Email)} if signupMethod != "" { fields["signup_method"] = signupMethod } - // The row is written with raw SQL on the resolution transaction, so the - // business emitter's declared-payload check does not run over it; call it - // here so an undeclared key or an out-of-enum value fails the login that - // would otherwise have written an unreadable record. + // The emitter only logs a payload that drifts from its declaration; check it + // strictly here so an undeclared key or an out-of-enum value fails the login + // that would otherwise have written an unreadable record. if err := business.ValidatePayload(business.EventUserRegistered, fields); err != nil { return fmt.Errorf("pgauth: registration audit payload: %w", err) } - payload, err := json.Marshal(fields) - if err != nil { - return fmt.Errorf("pgauth: marshal registration audit payload: %w", err) - } - _, err = tx.Exec(ctx, ` - INSERT INTO audit_events ( - id, actor_id, actor_type, event_type, schema_version, resource, resource_id, org_id, payload - ) VALUES ($1, $2, 'user', $3, 1, 'user', $4, $5, $6::jsonb)`, - business.NewID(), userID, string(business.EventUserRegistered), userID.String(), org, string(payload), - ) - if err != nil { - return fmt.Errorf("pgauth: insert registration audit event: %w", err) + if err := r.audit.RecordTx(ctx, business.AuditEntry{ + ActorID: userID.String(), + ActorType: business.ActorTypeUser, + EventType: business.EventUserRegistered, + Resource: "user", + ResourceID: userID.String(), + OrgID: org, + Payload: fields, + }); err != nil { + return fmt.Errorf("pgauth: record registration audit event: %w", err) } return nil } diff --git a/module/services/accounts/code/pkg/auth/pg/resolver_audit_test.go b/module/services/accounts/code/pkg/auth/pg/resolver_audit_test.go new file mode 100644 index 000000000..9c2df99ba --- /dev/null +++ b/module/services/accounts/code/pkg/auth/pg/resolver_audit_test.go @@ -0,0 +1,135 @@ +//go:build !pure + +package pgauth_test + +import ( + "context" + "encoding/json" + "testing" + + "github.com/google/uuid" + "github.com/stretchr/testify/require" + + "accounts/pkg/auth" + pgauth "accounts/pkg/auth/pg" + "accounts/pkg/business" + "accounts/pkg/infra" +) + +// newResolver is the production wiring: the resolver records its events +// through the audit emitter, in whichever mode opts select. +func newResolver(t *testing.T, opts ...business.DurableAuditEmitterOption) *pgauth.Resolver { + t.Helper() + recorder, err := business.NewDurableAuditEmitter(testStore, testStore, opts...) + require.NoError(t, err) + r := pgauth.NewResolver(testStore) + r.SetAuditRecorder(recorder) + return r +} + +func TestResolver_RefusesWithoutAnAuditRecorder(t *testing.T) { + r := pgauth.NewResolver(testStore) + _, err := r.Resolve(context.Background(), claims("nobody@test.local", "dev-nobody"), auth.SignupIntent{}) + require.ErrorContains(t, err, "audit recorder is required", + "a resolver that cannot record a registration must not provision one") +} + +type queuedRow struct { + eventType string + actorID string + resource string + resourceID string + orgID string + payload map[string]any +} + +// drainQueueFor reads and removes the queued events for one organization ("" is +// the platform's), as the relay — the job worker — does. Writers cannot read +// the queue back at all. +func drainQueueFor(t *testing.T, orgID string) []queuedRow { + t.Helper() + ctx := context.Background() + pool, err := infra.NewAuditRelayPool(ctx) + require.NoError(t, err) + defer pool.Close() + + rows, err := pool.Query(ctx, ` + SELECT event_type, COALESCE(actor_id::text, ''), resource, COALESCE(resource_id, ''), + COALESCE(org_id::text, ''), payload + FROM audit_event_queue + WHERE COALESCE(org_id::text, '') = $1 + ORDER BY seq`, orgID) + require.NoError(t, err) + var out []queuedRow + for rows.Next() { + var row queuedRow + var payload []byte + require.NoError(t, rows.Scan(&row.eventType, &row.actorID, &row.resource, &row.resourceID, &row.orgID, &payload)) + require.NoError(t, json.Unmarshal(payload, &row.payload)) + out = append(out, row) + } + require.NoError(t, rows.Err()) + _, err = pool.Exec(ctx, `DELETE FROM audit_event_queue WHERE COALESCE(org_id::text, '') = $1`, orgID) + require.NoError(t, err) + return out +} + +// Under a swap value the resolver's two events reach the queue, on the +// resolution transaction, and audit_events receives nothing — the raw SQL they +// were written with before would have kept them in audit_events, out of reach +// of the warehouse and the archive. +func TestResolver_UnderTheSwap_RecordsSsoProvisioningInTheQueue(t *testing.T) { + resetAuthTables(t) + ctx := context.Background() + r := newResolver(t, business.WithQueuedRecords()) + + orgID := seedOrg(t, seedUser(t), "Acme", "workos-acme") + setSsoProvisioning(t, orgID, "jit", "member", []string{"acme.test"}) + + identity, err := r.Resolve(ctx, claims("worker@acme.test", "sso-worker"), auth.SsoJitIntent{OrgID: orgID}) + require.NoError(t, err) + + require.Zero(t, registeredAuditCount(t, orgID), "audit_events receives no new rows under a swap value") + require.Zero(t, ssoJitAuditCount(t, orgID)) + + queued := drainQueueFor(t, orgID.String()) + require.Equal(t, []queuedRow{ + { + eventType: string(business.EventAuthSSOJitProvisioned), actorID: identity.UserID.String(), + resource: "organization", resourceID: orgID.String(), orgID: orgID.String(), + payload: map[string]any{"provider": "dev"}, + }, + { + eventType: string(business.EventUserRegistered), actorID: identity.UserID.String(), + resource: "user", resourceID: identity.UserID.String(), orgID: orgID.String(), + payload: map[string]any{"email": "worker@acme.test", "signup_method": "sso"}, + }, + }, queued, "the same two records the raw SQL wrote, in the order the resolution wrote them") +} + +// A registration with no organization is a platform event; under a swap value +// it reaches the queue too, as the actor-scoped row audit_events would have held. +func TestResolver_UnderTheSwap_RecordsAnOrglessRegistrationInTheQueue(t *testing.T) { + resetAuthTables(t) + ctx := context.Background() + r := newResolver(t, business.WithQueuedRecords()) + + identity, err := r.Resolve(ctx, claims("solo@test.local", "dev-solo"), auth.SignupIntent{}) + require.NoError(t, err) + require.Equal(t, uuid.Nil, identity.OrgID) + + var inAuditEvents int + scanControlPlane(t, &inAuditEvents, + `SELECT COUNT(*) FROM audit_events WHERE event_type = 'saas.user.registered' AND actor_id = $1`, identity.UserID) + require.Zero(t, inAuditEvents) + + var registrations []queuedRow + for _, row := range drainQueueFor(t, "") { + if row.actorID == identity.UserID.String() { + registrations = append(registrations, row) + } + } + require.Len(t, registrations, 1) + require.Equal(t, string(business.EventUserRegistered), registrations[0].eventType) + require.Equal(t, "solo@test.local", registrations[0].payload["email"]) +} diff --git a/module/services/accounts/code/pkg/auth/pg/resolver_sso_jit_test.go b/module/services/accounts/code/pkg/auth/pg/resolver_sso_jit_test.go index 9237dc190..52a438ae3 100644 --- a/module/services/accounts/code/pkg/auth/pg/resolver_sso_jit_test.go +++ b/module/services/accounts/code/pkg/auth/pg/resolver_sso_jit_test.go @@ -11,7 +11,6 @@ import ( "github.com/stretchr/testify/require" "accounts/pkg/auth" - pgauth "accounts/pkg/auth/pg" "accounts/pkg/infra/storetx" ) @@ -58,7 +57,7 @@ func registeredAuditCount(t *testing.T, orgID uuid.UUID) int { func TestResolver_Registration_RecordedOncePerIdentityInTheJoiningOrg(t *testing.T) { resetAuthTables(t) ctx := context.Background() - r := pgauth.NewResolver(testStore) + r := newResolver(t) orgID := seedOrg(t, seedUser(t), "Acme", "workos-acme") setSsoProvisioning(t, orgID, "jit", "member", []string{"acme.test"}) @@ -105,7 +104,7 @@ func TestResolver_Registration_RecordedOncePerIdentityInTheJoiningOrg(t *testing func TestResolver_Registration_RecordedForInviteOnlySsoProvisioning(t *testing.T) { resetAuthTables(t) ctx := context.Background() - r := pgauth.NewResolver(testStore) + r := newResolver(t) ownerID := seedUser(t) orgID := seedOrg(t, ownerID, "Acme", "workos-acme") @@ -122,7 +121,7 @@ func TestResolver_Registration_RecordedForInviteOnlySsoProvisioning(t *testing.T func TestResolver_SsoJit_FirstLoginProvisions_SecondIsPlainLogin(t *testing.T) { resetAuthTables(t) ctx := context.Background() - r := pgauth.NewResolver(testStore) + r := newResolver(t) orgID := seedOrg(t, seedUser(t), "Acme", "workos-acme") setSsoProvisioning(t, orgID, "jit", "member", []string{"acme.test"}) @@ -159,7 +158,7 @@ func TestResolver_SsoJit_FirstLoginProvisions_SecondIsPlainLogin(t *testing.T) { func TestResolver_SsoJit_EmailDomainNotAllowed_RejectedAndProvisionsNothing(t *testing.T) { resetAuthTables(t) ctx := context.Background() - r := pgauth.NewResolver(testStore) + r := newResolver(t) orgID := seedOrg(t, seedUser(t), "Acme", "workos-acme") setSsoProvisioning(t, orgID, "jit", "member", []string{"acme.test"}) @@ -178,7 +177,7 @@ func TestResolver_SsoJit_EmailDomainNotAllowed_RejectedAndProvisionsNothing(t *t func TestResolver_SsoJit_EmptyAllowlist_ReportsMisconfiguration(t *testing.T) { resetAuthTables(t) ctx := context.Background() - r := pgauth.NewResolver(testStore) + r := newResolver(t) orgID := seedOrg(t, seedUser(t), "Acme", "workos-acme") // jit enabled but no trusted domains named — a misconfiguration, distinct @@ -198,7 +197,7 @@ func TestResolver_SsoJit_EmptyAllowlist_ReportsMisconfiguration(t *testing.T) { func TestResolver_SsoJit_ReprovisionsIdentityRemovedFromOrg(t *testing.T) { resetAuthTables(t) ctx := context.Background() - r := pgauth.NewResolver(testStore) + r := newResolver(t) orgID := seedOrg(t, seedUser(t), "Acme", "workos-acme") setSsoProvisioning(t, orgID, "jit", "member", []string{"acme.test"}) @@ -231,7 +230,7 @@ func TestResolver_SsoJit_ReprovisionsIdentityRemovedFromOrg(t *testing.T) { func TestResolver_SsoJit_InviteOnly_RequiresPendingInvitation(t *testing.T) { resetAuthTables(t) ctx := context.Background() - r := pgauth.NewResolver(testStore) + r := newResolver(t) ownerID := seedUser(t) orgID := seedOrg(t, ownerID, "Acme", "workos-acme") @@ -269,7 +268,7 @@ func TestResolver_SsoJit_InviteOnly_RequiresPendingInvitation(t *testing.T) { func TestResolver_SsoJit_Disabled_RejectsFirstSeen_ExistingMemberLogsIn(t *testing.T) { resetAuthTables(t) ctx := context.Background() - r := pgauth.NewResolver(testStore) + r := newResolver(t) orgID := seedOrg(t, seedUser(t), "Acme", "workos-acme") setSsoProvisioning(t, orgID, "disabled", "member", []string{"acme.test"}) @@ -296,7 +295,7 @@ func TestResolver_SsoJit_Disabled_RejectsFirstSeen_ExistingMemberLogsIn(t *testi func TestResolver_SsoJit_ExistingMemberShortCircuits_NotReGatedByDomain(t *testing.T) { resetAuthTables(t) ctx := context.Background() - r := pgauth.NewResolver(testStore) + r := newResolver(t) orgID := seedOrg(t, seedUser(t), "Acme", "workos-acme") // Allowlist deliberately excludes the member's email domain. @@ -316,7 +315,7 @@ func TestResolver_SsoJit_ExistingMemberShortCircuits_NotReGatedByDomain(t *testi func TestResolver_SsoJit_NeverCreatesOrgOrTouchesAnotherOrg(t *testing.T) { resetAuthTables(t) ctx := context.Background() - r := pgauth.NewResolver(testStore) + r := newResolver(t) target := seedOrg(t, seedUser(t), "Target", "workos-target") setSsoProvisioning(t, target, "jit", "member", []string{"acme.test"}) @@ -346,7 +345,7 @@ func TestResolver_SsoJit_NeverCreatesOrgOrTouchesAnotherOrg(t *testing.T) { func TestResolver_ResolveOrgProvisioning(t *testing.T) { resetAuthTables(t) ctx := context.Background() - r := pgauth.NewResolver(testStore) + r := newResolver(t) withPolicy := seedOrg(t, seedUser(t), "WithPolicy", "workos-with") setSsoProvisioning(t, withPolicy, "jit", "member", []string{"acme.test"}) diff --git a/module/services/accounts/code/pkg/auth/pg/resolver_test.go b/module/services/accounts/code/pkg/auth/pg/resolver_test.go index c7ad20d68..5c991b786 100644 --- a/module/services/accounts/code/pkg/auth/pg/resolver_test.go +++ b/module/services/accounts/code/pkg/auth/pg/resolver_test.go @@ -164,7 +164,7 @@ func seedInvitation(t *testing.T, orgID, inviterID uuid.UUID, email, role, statu func TestResolver_Signup_NewUser_Provisioning(t *testing.T) { resetAuthTables(t) ctx := context.Background() - r := pgauth.NewResolver(testStore) + r := newResolver(t) id, err := r.Resolve(ctx, claims("alice@test.local", "dev-alice"), auth.SignupIntent{}) require.NoError(t, err) @@ -189,7 +189,7 @@ func TestResolver_Signup_NewUser_Provisioning(t *testing.T) { func TestResolver_Signup_MagicLinkProvider_Provisioning(t *testing.T) { resetAuthTables(t) ctx := context.Background() - r := pgauth.NewResolver(testStore) + r := newResolver(t) c := claims("linked@test.local", "linked@test.local") c.Provider = business.MagicLinkIdentityProvider @@ -212,7 +212,7 @@ func TestResolver_Signup_MagicLinkProvider_Provisioning(t *testing.T) { func TestResolver_CarriesPresentationalIdentity(t *testing.T) { resetAuthTables(t) ctx := context.Background() - r := pgauth.NewResolver(testStore) + r := newResolver(t) c := claims("Alice@Test.Local", "dev-alice") c.DisplayName = "Alice Example" @@ -228,7 +228,7 @@ func TestResolver_CarriesPresentationalIdentity(t *testing.T) { func TestResolver_Login_UnknownIdentity_ReturnsNoAccount(t *testing.T) { resetAuthTables(t) ctx := context.Background() - r := pgauth.NewResolver(testStore) + r := newResolver(t) _, err := r.Resolve(ctx, claims("stranger@test.local", "dev-stranger"), auth.LoginIntent{}) require.ErrorIs(t, err, auth.ErrNoAccount) @@ -244,7 +244,7 @@ func TestResolver_Login_UnknownIdentity_ReturnsNoAccount(t *testing.T) { func TestResolver_ExistingUser_Idempotent(t *testing.T) { resetAuthTables(t) ctx := context.Background() - r := pgauth.NewResolver(testStore) + r := newResolver(t) first, err := r.Resolve(ctx, claims("bob@test.local", "dev-bob"), auth.SignupIntent{}) require.NoError(t, err) @@ -262,7 +262,7 @@ func TestResolver_ExistingUser_Idempotent(t *testing.T) { func TestResolver_LastUsedIsInitializedCoalescedAndRefreshed(t *testing.T) { resetAuthTables(t) ctx := context.Background() - r := pgauth.NewResolver(testStore) + r := newResolver(t) loginClaims := claims("last-used@test.local", "dev-last-used") identity, err := r.Resolve(ctx, loginClaims, auth.SignupIntent{}) @@ -294,7 +294,7 @@ func TestResolver_LastUsedIsInitializedCoalescedAndRefreshed(t *testing.T) { func TestResolver_ConcurrentExistingLogin_AllSucceed(t *testing.T) { resetAuthTables(t) ctx := context.Background() - r := pgauth.NewResolver(testStore) + r := newResolver(t) first, err := r.Resolve(ctx, claims("concurrent@test.local", "dev-concurrent"), auth.SignupIntent{}) require.NoError(t, err) @@ -339,7 +339,7 @@ func identityLastUsed(t *testing.T, userID uuid.UUID) time.Time { func TestResolver_ExistingInactiveUserRejected(t *testing.T) { resetAuthTables(t) ctx := context.Background() - r := pgauth.NewResolver(testStore) + r := newResolver(t) identity, err := r.Resolve(ctx, claims("suspended@test.local", "dev-suspended"), auth.SignupIntent{}) require.NoError(t, err) @@ -356,7 +356,7 @@ func TestResolver_ExistingInactiveUserRejected(t *testing.T) { func TestResolver_Signup_CreatesOrg(t *testing.T) { resetAuthTables(t) ctx := context.Background() - r := pgauth.NewResolver(testStore) + r := newResolver(t) id, err := r.Resolve(ctx, claims("carol@test.local", "dev-carol"), auth.SignupIntent{OrganizationName: "Carol's Corp"}) require.NoError(t, err) @@ -380,7 +380,7 @@ func TestResolver_Signup_CreatesOrg(t *testing.T) { func TestResolver_Signup_NoOrgNameDoesNotCreateOrg(t *testing.T) { resetAuthTables(t) ctx := context.Background() - r := pgauth.NewResolver(testStore) + r := newResolver(t) id, err := r.Resolve(ctx, claims("dave@test.local", "dev-dave"), auth.SignupIntent{}) require.NoError(t, err) @@ -390,7 +390,7 @@ func TestResolver_Signup_NoOrgNameDoesNotCreateOrg(t *testing.T) { func TestResolver_ExistingOrgIsLoaded(t *testing.T) { resetAuthTables(t) ctx := context.Background() - r := pgauth.NewResolver(testStore) + r := newResolver(t) // Provision user + org first, err := r.Resolve(ctx, claims("erin@test.local", "dev-erin"), auth.SignupIntent{OrganizationName: "Erin Inc"}) @@ -406,7 +406,7 @@ func TestResolver_ExistingOrgIsLoaded(t *testing.T) { func TestResolver_Login_DefaultOrgWins_NotMostRecentMembership(t *testing.T) { resetAuthTables(t) ctx := context.Background() - r := pgauth.NewResolver(testStore) + r := newResolver(t) id, err := r.Resolve(ctx, claims("multi@test.local", "dev-multi"), auth.SignupIntent{}) require.NoError(t, err) @@ -426,7 +426,7 @@ func TestResolver_Login_DefaultOrgWins_NotMostRecentMembership(t *testing.T) { func TestResolver_Login_MultipleMembershipsNoDefault_Orgless(t *testing.T) { resetAuthTables(t) ctx := context.Background() - r := pgauth.NewResolver(testStore) + r := newResolver(t) id, err := r.Resolve(ctx, claims("ambig@test.local", "dev-ambig"), auth.SignupIntent{}) require.NoError(t, err) @@ -444,7 +444,7 @@ func TestResolver_Login_MultipleMembershipsNoDefault_Orgless(t *testing.T) { func TestResolver_Login_SSOProviderOrgWins_OverMoreRecentMembership(t *testing.T) { resetAuthTables(t) ctx := context.Background() - r := pgauth.NewResolver(testStore) + r := newResolver(t) id, err := r.Resolve(ctx, claims("sso@test.local", "dev-sso"), auth.SignupIntent{}) require.NoError(t, err) @@ -465,7 +465,7 @@ func TestResolver_Login_SSOProviderOrgWins_OverMoreRecentMembership(t *testing.T func TestResolver_Login_SSOProviderOrgNonMember_Rejected(t *testing.T) { resetAuthTables(t) ctx := context.Background() - r := pgauth.NewResolver(testStore) + r := newResolver(t) id, err := r.Resolve(ctx, claims("outsider@test.local", "dev-outsider"), auth.SignupIntent{}) require.NoError(t, err) @@ -513,7 +513,7 @@ func TestOrganizations_SSOOrganizationIDIsUnique(t *testing.T) { func TestResolver_Signup_SSOProviderOrgNonMember_Rejected(t *testing.T) { resetAuthTables(t) ctx := context.Background() - r := pgauth.NewResolver(testStore) + r := newResolver(t) // A provisioned tenant already carries the asserted WorkOS org id. owner := seedUser(t) @@ -538,7 +538,7 @@ func TestResolver_Signup_SSOProviderOrgNonMember_Rejected(t *testing.T) { func TestResolver_Login_SSOProviderOrgUnknown_FallsThroughToDefault(t *testing.T) { resetAuthTables(t) ctx := context.Background() - r := pgauth.NewResolver(testStore) + r := newResolver(t) id, err := r.Resolve(ctx, claims("firstsso@test.local", "dev-firstsso"), auth.SignupIntent{}) require.NoError(t, err) @@ -561,7 +561,7 @@ func TestResolver_Login_SSOProviderOrgUnknown_FallsThroughToDefault(t *testing.T func TestResolver_Login_SingleMembershipNoDefault_ResolvesToIt(t *testing.T) { resetAuthTables(t) ctx := context.Background() - r := pgauth.NewResolver(testStore) + r := newResolver(t) id, err := r.Resolve(ctx, claims("solo@test.local", "dev-solo"), auth.SignupIntent{}) require.NoError(t, err) @@ -577,7 +577,7 @@ func TestResolver_Login_SingleMembershipNoDefault_ResolvesToIt(t *testing.T) { func TestResolver_Login_ZeroMemberships_OrglessSuccess(t *testing.T) { resetAuthTables(t) ctx := context.Background() - r := pgauth.NewResolver(testStore) + r := newResolver(t) signup, err := r.Resolve(ctx, claims("lonely@test.local", "dev-lonely"), auth.SignupIntent{}) require.NoError(t, err) @@ -592,7 +592,7 @@ func TestResolver_Login_ZeroMemberships_OrglessSuccess(t *testing.T) { func TestResolver_Invite_ProvisionsUserAndMembershipAtomically(t *testing.T) { resetAuthTables(t) ctx := context.Background() - r := pgauth.NewResolver(testStore) + r := newResolver(t) inviterID, orgID := seedInviteOrg(t) token := seedInvitation(t, orgID, inviterID, "invitee@test.local", "admin", "pending", time.Now().Add(24*time.Hour)) @@ -622,7 +622,7 @@ func TestResolver_Invite_ProvisionsUserAndMembershipAtomically(t *testing.T) { func TestResolver_Invite_EmailMismatchRejectedAndProvisionsNothing(t *testing.T) { resetAuthTables(t) ctx := context.Background() - r := pgauth.NewResolver(testStore) + r := newResolver(t) inviterID, orgID := seedInviteOrg(t) token := seedInvitation(t, orgID, inviterID, "invitee@test.local", "member", "pending", time.Now().Add(24*time.Hour)) @@ -645,7 +645,7 @@ func TestResolver_Invite_EmailMismatchRejectedAndProvisionsNothing(t *testing.T) func TestResolver_Invite_UnverifiedEmailRejectedAndProvisionsNothing(t *testing.T) { resetAuthTables(t) ctx := context.Background() - r := pgauth.NewResolver(testStore) + r := newResolver(t) inviterID, orgID := seedInviteOrg(t) token := seedInvitation(t, orgID, inviterID, "invitee@test.local", "admin", "pending", time.Now().Add(24*time.Hour)) @@ -670,7 +670,7 @@ func TestResolver_Invite_UnverifiedEmailRejectedAndProvisionsNothing(t *testing. func TestResolver_Invite_VerifiedEmailIsAccepted(t *testing.T) { resetAuthTables(t) ctx := context.Background() - r := pgauth.NewResolver(testStore) + r := newResolver(t) inviterID, orgID := seedInviteOrg(t) token := seedInvitation(t, orgID, inviterID, "invitee@test.local", "admin", "pending", time.Now().Add(24*time.Hour)) @@ -686,7 +686,7 @@ func TestResolver_Invite_VerifiedEmailIsAccepted(t *testing.T) { func TestResolver_EmailVerifiedPersistsAndSurvivesRelogin(t *testing.T) { resetAuthTables(t) ctx := context.Background() - r := pgauth.NewResolver(testStore) + r := newResolver(t) // An unverified signup persists the provider's actual assertion, not a // hardcoded true. @@ -711,7 +711,7 @@ func TestResolver_EmailVerifiedPersistsAndSurvivesRelogin(t *testing.T) { func TestResolver_VerifiedUserNotDowngradedByLaterUnverifiedLogin(t *testing.T) { resetAuthTables(t) ctx := context.Background() - r := pgauth.NewResolver(testStore) + r := newResolver(t) verified, err := r.Resolve(ctx, claims("stable@test.local", "dev-stable"), auth.SignupIntent{}) require.NoError(t, err) @@ -728,7 +728,7 @@ func TestResolver_VerifiedUserNotDowngradedByLaterUnverifiedLogin(t *testing.T) func TestResolver_Invite_ExistingMemberRoleUpgraded(t *testing.T) { resetAuthTables(t) ctx := context.Background() - r := pgauth.NewResolver(testStore) + r := newResolver(t) inviterID, orgID := seedInviteOrg(t) @@ -761,7 +761,7 @@ func TestResolver_Invite_ExistingMemberRoleUpgraded(t *testing.T) { func TestResolver_Invite_IdempotentReacceptReportsCurrentRole(t *testing.T) { resetAuthTables(t) ctx := context.Background() - r := pgauth.NewResolver(testStore) + r := newResolver(t) inviterID, orgID := seedInviteOrg(t) token := seedInvitation(t, orgID, inviterID, "current@test.local", "member", "pending", time.Now().Add(24*time.Hour)) @@ -787,7 +787,7 @@ func TestResolver_Invite_IdempotentReacceptReportsCurrentRole(t *testing.T) { func TestResolver_Invite_NonPendingFailsClosed(t *testing.T) { ctx := context.Background() - r := pgauth.NewResolver(testStore) + r := newResolver(t) cases := []struct { name string @@ -818,7 +818,7 @@ func TestResolver_Invite_NonPendingFailsClosed(t *testing.T) { func TestResolver_Invite_UnknownTokenFailsClosed(t *testing.T) { resetAuthTables(t) ctx := context.Background() - r := pgauth.NewResolver(testStore) + r := newResolver(t) _, err := r.Resolve(ctx, claims("invitee@test.local", "dev-invitee"), auth.InviteIntent{Token: "not-a-real-token"}) require.ErrorIs(t, err, business.ErrInvitationUnavailable) @@ -827,7 +827,7 @@ func TestResolver_Invite_UnknownTokenFailsClosed(t *testing.T) { func TestResolver_Invite_ExistingUserJoinsAndReacceptIsIdempotent(t *testing.T) { resetAuthTables(t) ctx := context.Background() - r := pgauth.NewResolver(testStore) + r := newResolver(t) // The invitee already has an account (e.g. member of another org). existing, err := r.Resolve(ctx, claims("member@test.local", "dev-member"), auth.SignupIntent{}) @@ -858,7 +858,7 @@ func TestResolver_Bootstrap_FirstMatchGetsSuperAdmin(t *testing.T) { t.Setenv(pgauth.BootstrapAdminEmailEnv, "boss@test.local") - r := pgauth.NewResolver(testStore) + r := newResolver(t) id, err := r.Resolve(ctx, claims("boss@test.local", "dev-boss"), auth.SignupIntent{}) require.NoError(t, err) require.Equal(t, "super_admin", id.PlatformRole) @@ -881,7 +881,7 @@ func TestResolver_Bootstrap_SelfDisarms(t *testing.T) { ctx := context.Background() t.Setenv(pgauth.BootstrapAdminEmailEnv, "first@test.local") - r := pgauth.NewResolver(testStore) + r := newResolver(t) // First call grants id1, err := r.Resolve(ctx, claims("first@test.local", "dev-first"), auth.SignupIntent{}) @@ -908,7 +908,7 @@ func TestResolver_Bootstrap_UnverifiedEmailDenied(t *testing.T) { ctx := context.Background() t.Setenv(pgauth.BootstrapAdminEmailEnv, "boss@test.local") - r := pgauth.NewResolver(testStore) + r := newResolver(t) // An unverified claim matching the bootstrap address must not be granted // super_admin, and must leave the bootstrap slot unclaimed. @@ -937,7 +937,7 @@ func TestResolver_Bootstrap_NoEnvNoGrant(t *testing.T) { ctx := context.Background() _ = os.Unsetenv(pgauth.BootstrapAdminEmailEnv) - r := pgauth.NewResolver(testStore) + r := newResolver(t) id, err := r.Resolve(ctx, claims("anyone@test.local", "dev-anyone"), auth.SignupIntent{}) require.NoError(t, err) @@ -949,7 +949,7 @@ func TestResolver_Bootstrap_CaseInsensitiveEmail(t *testing.T) { ctx := context.Background() t.Setenv(pgauth.BootstrapAdminEmailEnv, " Boss@Test.LOCAL ") - r := pgauth.NewResolver(testStore) + r := newResolver(t) id, err := r.Resolve(ctx, claims("BOSS@test.local", "dev-boss-caps"), auth.SignupIntent{}) require.NoError(t, err) @@ -960,7 +960,7 @@ func TestResolver_Bootstrap_CaseInsensitiveEmail(t *testing.T) { func TestResolver_ConcurrentFirstSignup_OneUser(t *testing.T) { resetAuthTables(t) ctx := context.Background() - r := pgauth.NewResolver(testStore) + r := newResolver(t) // Fire 10 concurrent first signups for the same identity — must converge // on a single user row without deadlocking or duplicating rows. @@ -995,7 +995,7 @@ func TestResolver_ConcurrentFirstSignup_OneUser(t *testing.T) { func TestResolver_InvalidClaims_Rejected(t *testing.T) { ctx := context.Background() - r := pgauth.NewResolver(testStore) + r := newResolver(t) _, err := r.Resolve(ctx, &auth.Claims{Provider: "dev"}, auth.LoginIntent{}) require.Error(t, err) @@ -1007,8 +1007,9 @@ func TestResolver_InvalidClaims_Rejected(t *testing.T) { require.Error(t, err) } -func resolverWithSignupMode(mode auth.SignupMode) *pgauth.Resolver { - r := pgauth.NewResolver(testStore) +func resolverWithSignupMode(t *testing.T, mode auth.SignupMode) *pgauth.Resolver { + t.Helper() + r := newResolver(t) r.SetSignupMode(mode) return r } @@ -1043,7 +1044,7 @@ func execControlPlane(t *testing.T, query string, args ...any) { func TestResolver_SignupMode_Open_UnknownSignsUp(t *testing.T) { resetAuthTables(t) ctx := context.Background() - r := resolverWithSignupMode(auth.SignupModeOpen) + r := resolverWithSignupMode(t, auth.SignupModeOpen) id, err := r.Resolve(ctx, claims("open-unknown@test.local", "dev-open-unknown"), auth.SignupIntent{}) require.NoError(t, err) @@ -1057,7 +1058,7 @@ func TestResolver_SignupMode_Open_UnknownSignsUp(t *testing.T) { func TestResolver_SignupMode_Invite_StrangerRejectedAndProvisionsNothing(t *testing.T) { resetAuthTables(t) ctx := context.Background() - r := resolverWithSignupMode(auth.SignupModeInvite) + r := resolverWithSignupMode(t, auth.SignupModeInvite) _, err := r.Resolve(ctx, claims("stranger@test.local", "dev-invite-stranger"), auth.SignupIntent{}) require.ErrorIs(t, err, auth.ErrSignupNotAllowed) @@ -1075,7 +1076,7 @@ func TestResolver_SignupMode_Invite_InvitedSignsUpIntoOrg(t *testing.T) { inviterID, orgID := seedInviteOrg(t) seedInvitation(t, orgID, inviterID, "invitee@test.local", "admin", "pending", time.Now().Add(24*time.Hour)) - r := resolverWithSignupMode(auth.SignupModeInvite) + r := resolverWithSignupMode(t, auth.SignupModeInvite) id, err := r.Resolve(ctx, claims("invitee@test.local", "dev-invitee"), auth.SignupIntent{}) require.NoError(t, err) require.Equal(t, orgID, id.OrgID, "invited signup lands in the inviting org") @@ -1098,11 +1099,11 @@ func TestResolver_SignupMode_Invite_ExistingUserCanLogin(t *testing.T) { ctx := context.Background() // Provision the user while signup is open, then flip to invite mode. - open := resolverWithSignupMode(auth.SignupModeOpen) + open := resolverWithSignupMode(t, auth.SignupModeOpen) first, err := open.Resolve(ctx, claims("member@test.local", "dev-member"), auth.SignupIntent{}) require.NoError(t, err) - invite := resolverWithSignupMode(auth.SignupModeInvite) + invite := resolverWithSignupMode(t, auth.SignupModeInvite) second, err := invite.Resolve(ctx, claims("member@test.local", "dev-member"), auth.LoginIntent{}) require.NoError(t, err, "login is never gated by signup mode") require.Equal(t, first.UserID, second.UserID) @@ -1113,7 +1114,7 @@ func TestResolver_SignupMode_Waitlist_PendingRejected(t *testing.T) { ctx := context.Background() seedWaitlistEntry(t, "pending@test.local", "pending") - r := resolverWithSignupMode(auth.SignupModeWaitlist) + r := resolverWithSignupMode(t, auth.SignupModeWaitlist) _, err := r.Resolve(ctx, claims("pending@test.local", "dev-waitlist-pending"), auth.SignupIntent{}) require.ErrorIs(t, err, auth.ErrSignupNotAllowed) @@ -1127,7 +1128,7 @@ func TestResolver_SignupMode_Waitlist_ApprovedAccepted(t *testing.T) { ctx := context.Background() seedWaitlistEntry(t, "approved@test.local", "approved") - r := resolverWithSignupMode(auth.SignupModeWaitlist) + r := resolverWithSignupMode(t, auth.SignupModeWaitlist) id, err := r.Resolve(ctx, claims("approved@test.local", "dev-waitlist-approved"), auth.SignupIntent{}) require.NoError(t, err) require.NotEqual(t, uuid.Nil, id.UserID) @@ -1143,7 +1144,7 @@ func TestResolver_SignupMode_Invite_UnverifiedEmailRejectedAndProvisionsNothing( inviterID, orgID := seedInviteOrg(t) seedInvitation(t, orgID, inviterID, "invitee@test.local", "admin", "pending", time.Now().Add(24*time.Hour)) - r := resolverWithSignupMode(auth.SignupModeInvite) + r := resolverWithSignupMode(t, auth.SignupModeInvite) // The email matches a pending invitation, but the provider did not verify it. // Authorising on an unverified email would let a stranger inherit the org, so // the gate must reject it and leave every table untouched. @@ -1166,7 +1167,7 @@ func TestResolver_SignupMode_Waitlist_UnverifiedEmailRejectedAndProvisionsNothin ctx := context.Background() seedWaitlistEntry(t, "approved@test.local", "approved") - r := resolverWithSignupMode(auth.SignupModeWaitlist) + r := resolverWithSignupMode(t, auth.SignupModeWaitlist) // An approved waitlist entry matches by email, but an unverified claim has not // proven it controls that address and must not clear the gate. _, err := r.Resolve(ctx, claimsUnverified("approved@test.local", "dev-unverified-approved"), auth.SignupIntent{}) @@ -1195,7 +1196,7 @@ func TestResolver_SignupMode_Invite_NewerExpiredInvitationDoesNotShadowValidOne( execControlPlane(t, `UPDATE invitations SET created_at = NOW() - INTERVAL '2 hours' WHERE org_id = $1`, orgValid) execControlPlane(t, `UPDATE invitations SET created_at = NOW() - INTERVAL '1 hour' WHERE org_id = $1`, orgExpired) - r := resolverWithSignupMode(auth.SignupModeInvite) + r := resolverWithSignupMode(t, auth.SignupModeInvite) id, err := r.Resolve(ctx, claims("invitee@test.local", "dev-shadowed-invitee"), auth.SignupIntent{}) require.NoError(t, err, "a valid invitation must not be shadowed by a newer expired one") require.Equal(t, orgValid, id.OrgID, "signup lands in the org whose invitation is still valid") diff --git a/module/services/accounts/code/pkg/business/audit.go b/module/services/accounts/code/pkg/business/audit.go index 317b173d0..74b0c8e7b 100644 --- a/module/services/accounts/code/pkg/business/audit.go +++ b/module/services/accounts/code/pkg/business/audit.go @@ -136,6 +136,7 @@ type DurableAuditEmitter struct { producer jobs.Producer transport events.Transport teeExternal bool + queued bool } // DurableAuditEmitterOption tunes the emitter at construction. @@ -149,6 +150,15 @@ func WithExternalTee() DurableAuditEmitterOption { return func(e *DurableAuditEmitter) { e.teeExternal = true } } +// WithQueuedRecords writes each event's record into the transactional queue +// (audit_event_queue) instead of audit_events. It is the emitter under a swap +// value (ADR 0009): the queue row commits on the same transaction the +// audit_events row would have, beside the same domain event, and AuditRelay +// delivers it to the store of record afterwards. audit_events receives nothing. +func WithQueuedRecords() DurableAuditEmitterOption { + return func(e *DurableAuditEmitter) { e.queued = true } +} + // WithDomainEventTransport publishes an external-visibility domain event beside // each org-scoped audit row, in the same transaction. That event is what an // outbound webhook subscription is fanned out from; without a transport wired the @@ -168,6 +178,10 @@ func NewDurableAuditEmitter(store Store, producer jobs.Producer, opts ...Durable for _, opt := range opts { opt(emitter) } + if emitter.teeExternal && emitter.queued { + // The tee copies audit_events rows; under a swap there are none to copy. + return nil, errors.New("audit: the external tee and the queued store of record are exclusive") + } return emitter, nil } @@ -211,22 +225,8 @@ func (e *DurableAuditEmitter) write(ctx context.Context, entry AuditEntry) error if err != nil { return err } - if entry.IdempotencyKey != "" { - reserved, err := e.store.ReserveAuditIdempotency(ctx, entry.OrgID, string(entry.EventType), entry.IdempotencyKey) - if err != nil { - return err - } - if !reserved { - // A prior emit of this (org, event_type, idempotency_key) already wrote - // the event in a committed transaction. Skip the insert and the webhook - // fan-out and report success: the intended effect (exactly one audit row - // and one set of deliveries) is already in place. Reserving inside the - // caller's tx keeps the guard row and the audit row atomic, so a rolled - // back audit write also frees the key for a genuine retry. - return nil - } - } - if err := e.store.InsertAuditEvent(ctx, entry); err != nil { + written, err := e.record(ctx, entry) + if err != nil || !written { return err } if entry.OrgID == "" { @@ -247,6 +247,31 @@ func (e *DurableAuditEmitter) write(ctx context.Context, entry AuditEntry) error return nil } +// record writes the event's record — the audit_events row, or under a swap +// value the queue row — on the ambient transaction, after reserving its +// idempotency key. written is false for a duplicate emit, which writes nothing. +func (e *DurableAuditEmitter) record(ctx context.Context, entry AuditEntry) (bool, error) { + if entry.IdempotencyKey != "" { + reserved, err := e.store.ReserveAuditIdempotency(ctx, entry.OrgID, string(entry.EventType), entry.IdempotencyKey) + if err != nil { + return false, err + } + if !reserved { + // A prior emit of this (org, event_type, idempotency_key) already wrote + // the event in a committed transaction. Skip the insert and the webhook + // fan-out and report success: the intended effect (exactly one audit row + // and one set of deliveries) is already in place. Reserving inside the + // caller's tx keeps the guard row and the audit row atomic, so a rolled + // back audit write also frees the key for a genuine retry. + return false, nil + } + } + if e.queued { + return true, e.store.EnqueueAuditEvent(ctx, entry) + } + return true, e.store.InsertAuditEvent(ctx, entry) +} + func (e *DurableAuditEmitter) Emit(ctx context.Context, entry AuditEntry) { e.normalize(&entry) var err error @@ -277,6 +302,18 @@ func (e *DurableAuditEmitter) EmitTx(ctx context.Context, entry AuditEntry) erro return e.write(ctx, entry) } +// RecordTx writes only the event's record on the caller's transaction — no +// domain event, no tee job (AuditRecorder). Like EmitTx it MUST run inside an +// active transaction and its error MUST be propagated. +func (e *DurableAuditEmitter) RecordTx(ctx context.Context, entry AuditEntry) error { + e.normalize(&entry) + if _, err := e.resolve(ctx, &entry); err != nil { + return err + } + _, err := e.record(ctx, entry) + return err +} + func (e *DurableAuditEmitter) Close() {} // AuditQuery is the structured filter for the audit search path: per user, per diff --git a/module/services/accounts/code/pkg/business/audit_durability_gate_test.go b/module/services/accounts/code/pkg/business/audit_durability_gate_test.go index 4e9384fbb..9b0ad1809 100644 --- a/module/services/accounts/code/pkg/business/audit_durability_gate_test.go +++ b/module/services/accounts/code/pkg/business/audit_durability_gate_test.go @@ -296,8 +296,9 @@ func auditEventConstants(files []*ast.File) map[string]EventType { // TestAuditDurability_NoBypassOfTheClassifiedEmitPath closes the hole the // durability rules would otherwise have: a caller can write an audit row without // any emit site by reaching for the store directly, and every classification the -// gate above enforces would be silently irrelevant to it. InsertAuditEvent is the -// emitter's own call; nothing else in this package may make it. +// gate above enforces would be silently irrelevant to it. InsertAuditEvent and +// EnqueueAuditEvent — the record under the default and under a swap value — are +// the emitter's own calls; nothing else in this package may make them. func TestAuditDurability_NoBypassOfTheClassifiedEmitPath(t *testing.T) { fset := token.NewFileSet() entries, err := os.ReadDir(".") @@ -317,7 +318,7 @@ func TestAuditDurability_NoBypassOfTheClassifiedEmitPath(t *testing.T) { return true } sel, ok := call.Fun.(*ast.SelectorExpr) - if !ok || sel.Sel.Name != "InsertAuditEvent" { + if !ok || (sel.Sel.Name != "InsertAuditEvent" && sel.Sel.Name != "EnqueueAuditEvent") { return true } if name == "audit.go" { @@ -331,5 +332,5 @@ func TestAuditDurability_NoBypassOfTheClassifiedEmitPath(t *testing.T) { sort.Strings(callers) require.Empty(t, callers, "audit rows must be written through Service.emit / emitTx so their durability is classified and enforced; "+ - "direct InsertAuditEvent calls bypass that entirely:\n%s", strings.Join(callers, "\n")) + "direct InsertAuditEvent or EnqueueAuditEvent calls bypass that entirely:\n%s", strings.Join(callers, "\n")) } diff --git a/module/services/accounts/code/pkg/business/audit_queued_records_test.go b/module/services/accounts/code/pkg/business/audit_queued_records_test.go new file mode 100644 index 000000000..93ed5c577 --- /dev/null +++ b/module/services/accounts/code/pkg/business/audit_queued_records_test.go @@ -0,0 +1,163 @@ +package business + +import ( + "context" + "testing" + + "github.com/stretchr/testify/require" +) + +// Under a swap value (ADR 0009) the emitter's record is a queue row, written +// exactly where the audit_events row was — and nothing else about an emit +// changes. These hold the emitter to that on both of its write paths and on +// the record-only path the writers outside the Service use. + +// recordStore splits what the emitter writes by destination. +type recordStore struct { + Store + inserted []AuditEntry + enqueued []AuditEntry + reservations map[string]bool + controlPlane int +} + +func (s *recordStore) WithOrgTx(ctx context.Context, _ string, fn func(context.Context) error) error { + return fn(ctx) +} + +func (s *recordStore) WithControlPlane(ctx context.Context, fn func(context.Context) error) error { + s.controlPlane++ + return fn(ctx) +} + +func (s *recordStore) GetDeclaredAuditEventType(context.Context, EventType) (*DeclaredAuditEventType, error) { + return nil, nil +} + +func (s *recordStore) InsertAuditEvent(_ context.Context, entry AuditEntry) error { + s.inserted = append(s.inserted, entry) + return nil +} + +func (s *recordStore) EnqueueAuditEvent(_ context.Context, entry AuditEntry) error { + s.enqueued = append(s.enqueued, entry) + return nil +} + +func (s *recordStore) ReserveAuditIdempotency(_ context.Context, orgID, eventType, key string) (bool, error) { + if s.reservations == nil { + s.reservations = map[string]bool{} + } + id := orgID + "/" + eventType + "/" + key + if s.reservations[id] { + return false, nil + } + s.reservations[id] = true + return true, nil +} + +func newRecordEmitter(t *testing.T, store *recordStore, opts ...DurableAuditEmitterOption) (*DurableAuditEmitter, *recordingTransport) { + t.Helper() + transport := &recordingTransport{} + emitter, err := NewDurableAuditEmitter(store, silentProducer{}, append(opts, WithDomainEventTransport(transport))...) + require.NoError(t, err) + return emitter, transport +} + +// saas.session.revoked is published externally, so a webhook-visible emit is +// observable on the transport. +func externallyPublishedEntry() AuditEntry { + return AuditEntry{ + OrgID: teeOrgID, + ActorID: NewIDString(), + ActorType: ActorTypeUser, + EventType: EventSessionRevoked, + Resource: "session", + } +} + +func TestQueuedRecordsEmitTxWritesTheQueueBesideTheDomainEvent(t *testing.T) { + store := &recordStore{} + emitter, transport := newRecordEmitter(t, store, WithQueuedRecords()) + + require.NoError(t, emitter.EmitTx(context.Background(), externallyPublishedEntry())) + + require.Empty(t, store.inserted, "audit_events receives nothing under a swap value") + require.Len(t, store.enqueued, 1) + queued := store.enqueued[0] + require.NotEmpty(t, queued.ID, "the emitter assigns the id before the record is written") + require.False(t, queued.CreatedAt.IsZero()) + require.Equal(t, 1, queued.SchemaVersion, "the registered version is stamped as it is on an audit_events row") + require.Len(t, transport.published, 1, "webhook delivery still fans out from the same transaction") + require.Equal(t, queued.ID, transport.published[0].GetId(), "the domain event keeps the record's id") +} + +func TestDefaultRecordsEmitTxWritesAuditEvents(t *testing.T) { + store := &recordStore{} + emitter, transport := newRecordEmitter(t, store) + + require.NoError(t, emitter.EmitTx(context.Background(), externallyPublishedEntry())) + + require.Len(t, store.inserted, 1) + require.Empty(t, store.enqueued, "nothing writes the queue under postgres or both") + require.Len(t, transport.published, 1) +} + +func TestQueuedRecordsPlatformEventsGoThroughTheQueue(t *testing.T) { + store := &recordStore{} + emitter, transport := newRecordEmitter(t, store, WithQueuedRecords()) + + entry := externallyPublishedEntry() + entry.OrgID = "" + emitter.Emit(context.Background(), entry) + + require.Equal(t, 1, store.controlPlane, "a platform event is written on the control plane") + require.Len(t, store.enqueued, 1, "a platform event reaches the queue, not just organization events") + require.Empty(t, store.inserted) + require.Empty(t, transport.published, "a platform event has no tenant to publish to") +} + +func TestQueuedRecordsDuplicateEmitWritesNothing(t *testing.T) { + store := &recordStore{} + emitter, transport := newRecordEmitter(t, store, WithQueuedRecords()) + + entry := externallyPublishedEntry() + entry.IdempotencyKey = "op-1" + require.NoError(t, emitter.EmitTx(context.Background(), entry)) + require.NoError(t, emitter.EmitTx(context.Background(), entry)) + + require.Len(t, store.enqueued, 1, "the idempotency reservation stays in Postgres beside the queue row") + require.Len(t, transport.published, 1) +} + +func TestRecordTxWritesOnlyTheRecord(t *testing.T) { + for _, tc := range []struct { + name string + opts []DurableAuditEmitterOption + queued bool + }{ + {name: "postgres"}, + {name: "swap", opts: []DurableAuditEmitterOption{WithQueuedRecords()}, queued: true}, + } { + t.Run(tc.name, func(t *testing.T) { + store := &recordStore{} + emitter, transport := newRecordEmitter(t, store, tc.opts...) + + require.NoError(t, emitter.RecordTx(context.Background(), externallyPublishedEntry())) + + if tc.queued { + require.Len(t, store.enqueued, 1) + require.Empty(t, store.inserted) + } else { + require.Len(t, store.inserted, 1) + require.Empty(t, store.enqueued) + } + require.Empty(t, transport.published, "RecordTx publishes no domain event, as the raw-SQL writers it replaces never did") + }) + } +} + +func TestQueuedRecordsAndTheTeeAreExclusive(t *testing.T) { + _, err := NewDurableAuditEmitter(&recordStore{}, silentProducer{}, WithQueuedRecords(), WithExternalTee()) + require.ErrorContains(t, err, "exclusive") +} diff --git a/module/services/accounts/code/pkg/business/audit_registry.go b/module/services/accounts/code/pkg/business/audit_registry.go index 4bf2f8413..2fa443863 100644 --- a/module/services/accounts/code/pkg/business/audit_registry.go +++ b/module/services/accounts/code/pkg/business/audit_registry.go @@ -127,6 +127,31 @@ const ( DurabilityObservational AuditDurability = "observational" ) +// AuditRetentionClass declares how long an event type's full details must be +// kept once a warehouse is the audit store of record (ADR 0009). The envelope — +// who, which organization, what type, which target, when — and a hash of the +// details are kept for the compliance window whatever the class; the class +// decides where the details themselves go. +// +// It is its own field, deliberately not derived from Category: CategorySecurity +// names a narrower set (sign-in and MFA), and most security-class events — +// role changes, credential issuance, configuration — sit in other categories. +type AuditRetentionClass string + +const ( + // RetentionSecurity is the record of who could get in and what they were + // allowed to do: authentication (including its failures), credentials issued + // and revoked, permission, role, membership, sharing and delegation changes, + // administrative and operator actions, data leaving the platform (exports, + // replays), and configuration changes. Its full details are kept for the + // compliance window, in the events table and the locked archive alike. + RetentionSecurity AuditRetentionClass = "security" + // RetentionContent is everything else: what happened to a tenant's own + // content and operations. Its full details are kept for the shorter, + // configured content window; the archive holds only its envelope and hash. + RetentionContent AuditRetentionClass = "content" +) + // AuditEventDefinition is one registered event type. Namespace is the collision // key (always the leading segment of Type); Owner names the service that emits // it, which is a different axis entirely. Durability says how the record must @@ -142,13 +167,17 @@ const ( // (an observation about a whole solution; an event that is genuinely distinct on // every emit) go through the same two fields. type AuditEventDefinition struct { - Type EventType - Namespace string - Version int - Category AuditCategory - Owner string - Description string - Durability AuditDurability + Type EventType + Namespace string + Version int + Category AuditCategory + Owner string + Description string + Durability AuditDurability + // Retention is the type's retention class, required the way Durability is: + // every catalog entry is wrapped in securityRetained or contentRetained, and + // the index refuses one that is not. + Retention AuditRetentionClass RequiresEntry bool RequiresIdempotencyKey bool Fields []PayloadField @@ -245,6 +274,19 @@ func requiresIdempotencyKey(d AuditEventDefinition) AuditEventDefinition { return d } +// securityRetained and contentRetained declare a definition's retention class. +// Every catalog entry carries exactly one of them, outermost, so the class is +// read at the declaration and a new type cannot be added without choosing. +func securityRetained(d AuditEventDefinition) AuditEventDefinition { + d.Retention = RetentionSecurity + return d +} + +func contentRetained(d AuditEventDefinition) AuditEventDefinition { + d.Retention = RetentionContent + return d +} + func str(name string) PayloadField { return PayloadField{Name: name, Kind: FieldString} } func strs(name string) PayloadField { return PayloadField{Name: name, Kind: FieldStringArray} @@ -521,33 +563,33 @@ const ( ) var auditEventCatalog = []AuditEventDefinition{ - userJoined(mutation(EventUserRegistered, CategoryIdentity, "A new user account was registered.", - enum("signup_method", "password", "sso", "magic_link"), pii(str("email")))), - userJoined(mutation(EventUserCreated, CategoryIdentity, "A user was provisioned by an administrator.", pii(str("email")))), - mutation(EventUserUpdated, CategoryIdentity, "A user profile was updated."), - mutation(EventUserDeleted, CategoryIdentity, "A user account was deleted."), + securityRetained(userJoined(mutation(EventUserRegistered, CategoryIdentity, "A new user account was registered.", + enum("signup_method", "password", "sso", "magic_link"), pii(str("email"))))), + securityRetained(userJoined(mutation(EventUserCreated, CategoryIdentity, "A user was provisioned by an administrator.", pii(str("email"))))), + contentRetained(mutation(EventUserUpdated, CategoryIdentity, "A user profile was updated.")), + securityRetained(mutation(EventUserDeleted, CategoryIdentity, "A user account was deleted.")), // A suspension is allowed to leave an organization with no administrator — // containing a compromised account outranks that — so the organizations it // did leave that way are part of the record rather than a reason to refuse. - mutation(EventUserSuspended, CategoryIdentity, "A user account was suspended.", - strs("organizations_without_administrator")), - mutation(EventUserUnsuspended, CategoryIdentity, "A user account was reinstated."), - mutation(EventUserIdentityAdd, CategoryIdentity, "An external identity was linked to a user.", str("provider")), - observation(EventSettingsUpdated, CategoryIdentity, "A user's personal settings changed."), - mutation(EventConsentTerms, CategoryIdentity, "A user accepted the terms of service.", str("version")), - mutation(EventConsentPrefs, CategoryIdentity, "A user updated their consent preferences."), - - mutation(EventAPIKeyCreated, CategoryAccess, "An API key was minted.", uid("key_id"), PayloadField{Name: "scopes", Kind: FieldStringArray}), - mutation(EventModuleRegistrationMint, CategoryAccess, "A composed module was issued a gateway registration credential.", str("prefix")), - observation(EventModuleOrgAdminsNotified, CategorySystem, "A composed module notified a tenant's administrators; the host resolved the recipients.", + securityRetained(mutation(EventUserSuspended, CategoryIdentity, "A user account was suspended.", + strs("organizations_without_administrator"))), + securityRetained(mutation(EventUserUnsuspended, CategoryIdentity, "A user account was reinstated.")), + securityRetained(mutation(EventUserIdentityAdd, CategoryIdentity, "An external identity was linked to a user.", str("provider"))), + contentRetained(observation(EventSettingsUpdated, CategoryIdentity, "A user's personal settings changed.")), + contentRetained(mutation(EventConsentTerms, CategoryIdentity, "A user accepted the terms of service.", str("version"))), + contentRetained(mutation(EventConsentPrefs, CategoryIdentity, "A user updated their consent preferences.")), + + securityRetained(mutation(EventAPIKeyCreated, CategoryAccess, "An API key was minted.", uid("key_id"), PayloadField{Name: "scopes", Kind: FieldStringArray})), + securityRetained(mutation(EventModuleRegistrationMint, CategoryAccess, "A composed module was issued a gateway registration credential.", str("prefix"))), + contentRetained(observation(EventModuleOrgAdminsNotified, CategorySystem, "A composed module notified a tenant's administrators; the host resolved the recipients.", PayloadField{Name: "prefix", Kind: FieldString, Required: true}, str("category"), str("type"), PayloadField{Name: "recipients", Kind: FieldInt, Required: true}, PayloadField{Name: "delivered", Kind: FieldInt, Required: true}, - str("idempotency_key")), - mutation(EventModuleAuditTypesDeclared, CategorySystem, "A composed module declared audit event types of its own, or took a namespace over from the producer the operator unbound.", - PayloadField{Name: "prefix", Kind: FieldString, Required: true}, strs("event_types"), strs("namespaces_taken_over")), - mutation(EventModuleWorkContextMint, CategoryAccess, "A composed module was issued a Work Context for its service principal.", str("prefix"), str("tenant")), - mutation(EventModuleOperationContextMint, CategoryAccess, "A composed module was issued, with no person present, a Work Context for one of its installed operation audiences.", - str("prefix"), str("tenant"), str("binding_id"), str("audience"), strs("scopes")), + str("idempotency_key"))), + securityRetained(mutation(EventModuleAuditTypesDeclared, CategorySystem, "A composed module declared audit event types of its own, or took a namespace over from the producer the operator unbound.", + PayloadField{Name: "prefix", Kind: FieldString, Required: true}, strs("event_types"), strs("namespaces_taken_over"))), + securityRetained(mutation(EventModuleWorkContextMint, CategoryAccess, "A composed module was issued a Work Context for its service principal.", str("prefix"), str("tenant"))), + securityRetained(mutation(EventModuleOperationContextMint, CategoryAccess, "A composed module was issued, with no person present, a Work Context for one of its installed operation audiences.", + str("prefix"), str("tenant"), str("binding_id"), str("audience"), strs("scopes"))), // v2 adds `delegation_id` and stops requiring `owner_principal_id`. // // The exchange now has two arms. One presents a live parent capability, and @@ -562,7 +604,7 @@ var auditEventCatalog = []AuditEventDefinition{ // which is the one kind of refusal an authority surface most needs to keep. // `delegation_id` is what identifies them instead, so a probe against a // grant reference is legible even when no owner was ever resolved. - revised(observation(EventDelegatedAudienceExchange, CategoryAccess, "A composed module's installed delegated-audience exchange was issued or refused.", + securityRetained(revised(observation(EventDelegatedAudienceExchange, CategoryAccess, "A composed module's installed delegated-audience exchange was issued or refused.", PayloadField{Name: "owner_principal_id", Kind: FieldUUID}, uid("delegation_id"), PayloadField{Name: "actor_principal_id", Kind: FieldUUID, Required: true}, @@ -572,164 +614,164 @@ var auditEventCatalog = []AuditEventDefinition{ str("audience"), PayloadField{Name: "lookup", Kind: FieldBool, Required: true}, PayloadField{Name: "outcome", Kind: FieldEnum, Required: true, Enum: []string{DelegatedAudienceExchangeIssued, DelegatedAudienceExchangeRefused}}, - PayloadField{Name: "refusal_code", Kind: FieldEnum, Enum: []string{"InvalidArgument", "Unauthenticated", "PermissionDenied", "FailedPrecondition", "Unavailable", "Internal"}}), 2), - mutation(EventSolutionRegistrationMint, CategoryAccess, "A solution was issued a gateway and frontend registration credential.", str("solution_id")), - mutation(EventSolutionRegistrationUpdated, CategoryAccess, "A solution registered or replaced one half of its runtime registration.", str("solution_id"), str("publisher"), str("half"), PayloadField{Name: "revision", Kind: FieldInt}, strs("audit_namespaces_taken_over")), - mutation(EventSolutionRegistrationDeleted, CategoryAccess, "A solution registration was removed and tombstoned.", str("solution_id"), str("publisher"), PayloadField{Name: "revision", Kind: FieldInt}), - mutation(EventAPIKeyRevoked, CategoryAccess, "An API key was revoked.", uid("key_id")), - mutation(EventRoleCreated, CategoryAccess, "A role was created.", str("name")), - mutation(EventRoleUpdated, CategoryAccess, "A role's description and permission set were replaced.", str("name"), strs("permissions")), - mutation(EventRoleDeleted, CategoryAccess, "A role was deleted."), - mutation(EventRoleAssigned, CategoryAccess, "A role was assigned to a principal.", uid("role_id"), uid("subject_id")), - mutation(EventRoleRevoked, CategoryAccess, "A role assignment was revoked.", uid("role_id")), - mutation(EventSessionRevoked, CategoryAccess, "A session was revoked."), - mutation(EventInvitationCreated, CategoryAccess, "An organization invitation was created.", pii(str("email"))), - mutation(EventInvitationAccepted, CategoryAccess, "An organization invitation was accepted."), - mutation(EventInvitationRevoked, CategoryAccess, "An organization invitation was revoked."), - mutation(EventInvitationResent, CategoryAccess, "An organization invitation was resent."), - mutation(EventInvitationLinkIssued, CategoryAccess, "An organization invitation's accept link was issued to an administrator instead of emailed."), - mutation(EventDelegationRequested, CategoryAccess, "A delegation grant was requested."), - mutation(EventDelegationApproved, CategoryAccess, "A delegation grant was approved."), - mutation(EventDelegationDenied, CategoryAccess, "A delegation grant was denied."), - mutation(EventDelegationAutoApproved, CategoryAccess, "A delegation grant was auto-approved by policy."), - mutation(EventApprovalAsked, CategoryAccess, "An approval request was opened for a gated action.", str("resource"), str("action")), - mutation(EventApprovalApproved, CategoryAccess, "An approval request reached quorum and was approved.", str("resource"), str("action")), - mutation(EventApprovalDenied, CategoryAccess, "An approval request was denied."), - mutation(EventApprovalTimeout, CategoryAccess, "An approval request expired before reaching quorum."), - mutation(EventApprovalEscalated, CategoryAccess, "An approval request was escalated to a wider approver set."), - mutation(EventApprovalCancelled, CategoryAccess, "An approval request was cancelled before a decision.", str("reason")), - mutation(EventApprovalDecisionRecorded, CategoryAccess, "An approver recorded a decision on an approval request.", str("decision")), - mutation(EventPrincipalCreated, CategoryAccess, "An agent principal was created.", str("agent_identifier")), - mutation(EventPrincipalRevoked, CategoryAccess, "A principal was revoked.", str("reason")), - mutation(EventPrincipalDisabled, CategoryAccess, "An agent principal was disabled.", str("reason")), - mutation(EventPrincipalEnabled, CategoryAccess, "An agent principal was re-enabled."), - mutation(EventScopeNodeRegistered, CategoryAccess, "A scope node was registered.", str("scope_path"), str("kind")), - mutation(EventScopeGranted, CategoryAccess, "A role was granted at a scope node.", uid("role_id"), uid("subject_id"), str("scope_path")), - mutation(EventScopeRevoked, CategoryAccess, "A scope grant was revoked.", uid("role_id"), str("scope_path")), - mutation(EventInstallationCreated, CategoryAccess, "A solution was installed: an agent principal, solution scope node, standing grant, and installation row were composed.", - uid("agent_principal_id"), str("solution_identifier"), uid("role_id"), uid("owner_principal_id")), - mutation(EventInstallationRevoked, CategoryAccess, "A solution was uninstalled: its agent principal and standing grant were revoked and its scope node soft-deleted.", - str("solution_identifier")), - mutation(EventInstallationOwnershipTransferred, CategoryAccess, "An installation's owner of record was reassigned.", - uid("owner_principal_id")), - mutation(EventRecordShared, CategoryAccess, "A record was shared with a principal or team.", uid("role_id"), uid("subject_id")), - mutation(EventRecordShareRevoked, CategoryAccess, "A record share was revoked.", uid("role_id"), uid("subject_id")), - mutation(EventWorkContextTaskStarted, CategoryAccess, "A signed Work Context was issued for a new agent task and root session."), - mutation(EventWorkContextRootSession, CategoryAccess, "A new root agent session was started under an existing task."), - mutation(EventWorkContextChildSession, CategoryAccess, "An attenuated child agent session was started."), - mutation(EventWorkContextAudienceExch, CategoryAccess, "A Work Context task and session lineage was reissued for another audience."), - mutation(EventWorkContextRenewed, CategoryAccess, "A delegated actor renewed its Work Context past the signing TTL cap."), - - observation(EventAuthLogin, CategorySecurity, "A user authenticated.", str("method"), str("client_id")), - observation(EventAuthMagicLinkLogin, CategorySecurity, "A user authenticated via magic link."), - mutation(EventAuthSSOJitProvisioned, CategorySecurity, "A user was just-in-time provisioned via SSO.", str("provider")), - observation(EventAuthOrgSwitched, CategorySecurity, "A user switched active organization."), - observation(EventAuthMFAChallengeStart, CategorySecurity, "An MFA challenge was started."), - observation(EventAuthMFAChallengeDone, CategorySecurity, "An MFA challenge was completed.", enum("factor", "totp", "webauthn", "backup_code")), - observation(EventAuthClientAuthorized, CategorySecurity, "A person authorized a registered client to act for them.", str("client_id")), - mutation(EventMFATOTPSetupStarted, CategorySecurity, "TOTP enrollment was started."), - mutation(EventMFATOTPVerified, CategorySecurity, "A TOTP device was verified."), - mutation(EventMFAWebAuthnRegStarted, CategorySecurity, "WebAuthn registration was started."), - mutation(EventMFAWebAuthnRegistered, CategorySecurity, "A WebAuthn credential was registered."), - observation(EventMFAWebAuthnUsed, CategorySecurity, "A WebAuthn credential was used to authenticate."), - mutation(EventMFABackupGenerated, CategorySecurity, "MFA backup codes were generated."), - observation(EventMFABackupUsed, CategorySecurity, "An MFA backup code was consumed."), - mutation(EventMFADeviceRevoked, CategorySecurity, "An MFA device was revoked."), - mutation(EventPlatformRoleGranted, CategorySecurity, "A platform role was granted."), - mutation(EventPlatformRoleRevoked, CategorySecurity, "A platform role was revoked."), + PayloadField{Name: "refusal_code", Kind: FieldEnum, Enum: []string{"InvalidArgument", "Unauthenticated", "PermissionDenied", "FailedPrecondition", "Unavailable", "Internal"}}), 2)), + securityRetained(mutation(EventSolutionRegistrationMint, CategoryAccess, "A solution was issued a gateway and frontend registration credential.", str("solution_id"))), + securityRetained(mutation(EventSolutionRegistrationUpdated, CategoryAccess, "A solution registered or replaced one half of its runtime registration.", str("solution_id"), str("publisher"), str("half"), PayloadField{Name: "revision", Kind: FieldInt}, strs("audit_namespaces_taken_over"))), + securityRetained(mutation(EventSolutionRegistrationDeleted, CategoryAccess, "A solution registration was removed and tombstoned.", str("solution_id"), str("publisher"), PayloadField{Name: "revision", Kind: FieldInt})), + securityRetained(mutation(EventAPIKeyRevoked, CategoryAccess, "An API key was revoked.", uid("key_id"))), + securityRetained(mutation(EventRoleCreated, CategoryAccess, "A role was created.", str("name"))), + securityRetained(mutation(EventRoleUpdated, CategoryAccess, "A role's description and permission set were replaced.", str("name"), strs("permissions"))), + securityRetained(mutation(EventRoleDeleted, CategoryAccess, "A role was deleted.")), + securityRetained(mutation(EventRoleAssigned, CategoryAccess, "A role was assigned to a principal.", uid("role_id"), uid("subject_id"))), + securityRetained(mutation(EventRoleRevoked, CategoryAccess, "A role assignment was revoked.", uid("role_id"))), + securityRetained(mutation(EventSessionRevoked, CategoryAccess, "A session was revoked.")), + securityRetained(mutation(EventInvitationCreated, CategoryAccess, "An organization invitation was created.", pii(str("email")))), + securityRetained(mutation(EventInvitationAccepted, CategoryAccess, "An organization invitation was accepted.")), + securityRetained(mutation(EventInvitationRevoked, CategoryAccess, "An organization invitation was revoked.")), + securityRetained(mutation(EventInvitationResent, CategoryAccess, "An organization invitation was resent.")), + securityRetained(mutation(EventInvitationLinkIssued, CategoryAccess, "An organization invitation's accept link was issued to an administrator instead of emailed.")), + securityRetained(mutation(EventDelegationRequested, CategoryAccess, "A delegation grant was requested.")), + securityRetained(mutation(EventDelegationApproved, CategoryAccess, "A delegation grant was approved.")), + securityRetained(mutation(EventDelegationDenied, CategoryAccess, "A delegation grant was denied.")), + securityRetained(mutation(EventDelegationAutoApproved, CategoryAccess, "A delegation grant was auto-approved by policy.")), + securityRetained(mutation(EventApprovalAsked, CategoryAccess, "An approval request was opened for a gated action.", str("resource"), str("action"))), + securityRetained(mutation(EventApprovalApproved, CategoryAccess, "An approval request reached quorum and was approved.", str("resource"), str("action"))), + securityRetained(mutation(EventApprovalDenied, CategoryAccess, "An approval request was denied.")), + securityRetained(mutation(EventApprovalTimeout, CategoryAccess, "An approval request expired before reaching quorum.")), + securityRetained(mutation(EventApprovalEscalated, CategoryAccess, "An approval request was escalated to a wider approver set.")), + securityRetained(mutation(EventApprovalCancelled, CategoryAccess, "An approval request was cancelled before a decision.", str("reason"))), + securityRetained(mutation(EventApprovalDecisionRecorded, CategoryAccess, "An approver recorded a decision on an approval request.", str("decision"))), + securityRetained(mutation(EventPrincipalCreated, CategoryAccess, "An agent principal was created.", str("agent_identifier"))), + securityRetained(mutation(EventPrincipalRevoked, CategoryAccess, "A principal was revoked.", str("reason"))), + securityRetained(mutation(EventPrincipalDisabled, CategoryAccess, "An agent principal was disabled.", str("reason"))), + securityRetained(mutation(EventPrincipalEnabled, CategoryAccess, "An agent principal was re-enabled.")), + contentRetained(mutation(EventScopeNodeRegistered, CategoryAccess, "A scope node was registered.", str("scope_path"), str("kind"))), + securityRetained(mutation(EventScopeGranted, CategoryAccess, "A role was granted at a scope node.", uid("role_id"), uid("subject_id"), str("scope_path"))), + securityRetained(mutation(EventScopeRevoked, CategoryAccess, "A scope grant was revoked.", uid("role_id"), str("scope_path"))), + securityRetained(mutation(EventInstallationCreated, CategoryAccess, "A solution was installed: an agent principal, solution scope node, standing grant, and installation row were composed.", + uid("agent_principal_id"), str("solution_identifier"), uid("role_id"), uid("owner_principal_id"))), + securityRetained(mutation(EventInstallationRevoked, CategoryAccess, "A solution was uninstalled: its agent principal and standing grant were revoked and its scope node soft-deleted.", + str("solution_identifier"))), + securityRetained(mutation(EventInstallationOwnershipTransferred, CategoryAccess, "An installation's owner of record was reassigned.", + uid("owner_principal_id"))), + securityRetained(mutation(EventRecordShared, CategoryAccess, "A record was shared with a principal or team.", uid("role_id"), uid("subject_id"))), + securityRetained(mutation(EventRecordShareRevoked, CategoryAccess, "A record share was revoked.", uid("role_id"), uid("subject_id"))), + securityRetained(mutation(EventWorkContextTaskStarted, CategoryAccess, "A signed Work Context was issued for a new agent task and root session.")), + securityRetained(mutation(EventWorkContextRootSession, CategoryAccess, "A new root agent session was started under an existing task.")), + securityRetained(mutation(EventWorkContextChildSession, CategoryAccess, "An attenuated child agent session was started.")), + securityRetained(mutation(EventWorkContextAudienceExch, CategoryAccess, "A Work Context task and session lineage was reissued for another audience.")), + securityRetained(mutation(EventWorkContextRenewed, CategoryAccess, "A delegated actor renewed its Work Context past the signing TTL cap.")), + + securityRetained(observation(EventAuthLogin, CategorySecurity, "A user authenticated.", str("method"), str("client_id"))), + securityRetained(observation(EventAuthMagicLinkLogin, CategorySecurity, "A user authenticated via magic link.")), + securityRetained(mutation(EventAuthSSOJitProvisioned, CategorySecurity, "A user was just-in-time provisioned via SSO.", str("provider"))), + securityRetained(observation(EventAuthOrgSwitched, CategorySecurity, "A user switched active organization.")), + securityRetained(observation(EventAuthMFAChallengeStart, CategorySecurity, "An MFA challenge was started.")), + securityRetained(observation(EventAuthMFAChallengeDone, CategorySecurity, "An MFA challenge was completed.", enum("factor", "totp", "webauthn", "backup_code"))), + securityRetained(observation(EventAuthClientAuthorized, CategorySecurity, "A person authorized a registered client to act for them.", str("client_id"))), + securityRetained(mutation(EventMFATOTPSetupStarted, CategorySecurity, "TOTP enrollment was started.")), + securityRetained(mutation(EventMFATOTPVerified, CategorySecurity, "A TOTP device was verified.")), + securityRetained(mutation(EventMFAWebAuthnRegStarted, CategorySecurity, "WebAuthn registration was started.")), + securityRetained(mutation(EventMFAWebAuthnRegistered, CategorySecurity, "A WebAuthn credential was registered.")), + securityRetained(observation(EventMFAWebAuthnUsed, CategorySecurity, "A WebAuthn credential was used to authenticate.")), + securityRetained(mutation(EventMFABackupGenerated, CategorySecurity, "MFA backup codes were generated.")), + securityRetained(observation(EventMFABackupUsed, CategorySecurity, "An MFA backup code was consumed.")), + securityRetained(mutation(EventMFADeviceRevoked, CategorySecurity, "An MFA device was revoked.")), + securityRetained(mutation(EventPlatformRoleGranted, CategorySecurity, "A platform role was granted.")), + securityRetained(mutation(EventPlatformRoleRevoked, CategorySecurity, "A platform role was revoked.")), // The operator's justification is part of the record, not an optional // enrichment: who and whom are already implied by the actor/resource pair, // and why is the only thing this event can carry that the pair cannot. // session_id pairs this record with the user_impersonation_ended that closes // the same window, so the two reconcile to each other rather than by // timestamp proximity. - revised(mutation(EventPlatformImpersonated, CategorySecurity, "A platform admin impersonated a user.", - PayloadField{Name: "reason", Kind: FieldString, Required: true}, uid("session_id")), 3), + securityRetained(revised(mutation(EventPlatformImpersonated, CategorySecurity, "A platform admin impersonated a user.", + PayloadField{Name: "reason", Kind: FieldString, Required: true}, uid("session_id")), 3)), // access_token_revoked records whether the window's access token was actually // killed. Without a revocation store wired there is no mechanism to kill one // early, and a close record that did not say so would overstate what the stop // achieved. - mutation(EventPlatformImpersonationEnded, CategorySecurity, "A platform admin's impersonation session ended.", + securityRetained(mutation(EventPlatformImpersonationEnded, CategorySecurity, "A platform admin's impersonation session ended.", uid("session_id"), PayloadField{Name: "duration_seconds", Kind: FieldInt}, - PayloadField{Name: "access_token_revoked", Kind: FieldBool, Required: true}), - - observation(EventBillingCheckoutStarted, CategoryBilling, "A billing checkout session was started."), - observation(EventBillingPortalOpened, CategoryBilling, "The billing portal was opened."), - observation(EventBillingFreePlan, CategoryBilling, "The free plan was selected."), - mutation(EventEntitlementOverride, CategoryBilling, "An entitlement override was set.", str("key")), - - mutation(EventOrgCreated, CategoryOrganization, "An organization was created.", str("name")), - mutation(EventOrgMemberAdded, CategoryOrganization, "A member was added to an organization."), - mutation(EventOrgMemberRemoved, CategoryOrganization, "A member was removed from an organization."), - mutation(EventOrgMemberLeft, CategoryOrganization, "A member left an organization."), - mutation(EventOrgUpdated, CategoryOrganization, "An organization was renamed or its slug changed.", str("name"), str("slug")), - mutation(EventOrgDeleted, CategoryOrganization, "An organization was deleted: archived, its members removed and its credentials revoked.", str("slug")), - mutation(EventOrgSettingsUpdated, CategoryOrganization, "Organization branding settings were updated."), - mutation(EventOrgGenericSettingsUpdated, CategoryOrganization, "Organization generic (typed) settings were updated."), - mutation(EventTeamCreated, CategoryOrganization, "A team was created.", str("name")), - mutation(EventTeamUpdated, CategoryOrganization, "A team was updated."), - mutation(EventTeamDeleted, CategoryOrganization, "A team was deleted."), - mutation(EventTeamMemberAdded, CategoryOrganization, "A member was added to a team."), - mutation(EventTeamMemberRemoved, CategoryOrganization, "A member was removed from a team."), - mutation(EventSSOSetupStarted, CategoryOrganization, "SSO configuration was started."), - mutation(EventSSODisabled, CategoryOrganization, "SSO was disabled for an organization."), - observation(EventOnboardingStepDone, CategoryOrganization, "An onboarding step was completed.", str("step")), - observation(EventOnboardingStepSkip, CategoryOrganization, "An onboarding step was skipped.", str("step")), - observation(EventActivationAchieved, CategoryOrganization, "An organization reached activation."), - - observation(EventDashboardCreated, CategoryOrganization, "A dashboard was created."), - observation(EventDashboardUpdated, CategoryOrganization, "A dashboard was updated."), - observation(EventDashboardDeleted, CategoryOrganization, "A dashboard was deleted."), - observation(EventDashboardShared, CategoryOrganization, "A dashboard's visibility was changed."), - - observation(EventWaitlistJoined, CategoryLifecycle, "A prospect joined the waitlist.", pii(str("email"))), - observation(EventWaitlistPending, CategoryLifecycle, "A waitlist entry moved to pending."), - observation(EventWaitlistVerified, CategoryLifecycle, "A waitlist entry was verified."), - observation(EventWaitlistReviewed, CategoryLifecycle, "A waitlist entry was reviewed by an administrator."), - observation(EventWaitlistApproved, CategoryLifecycle, "A waitlist entry was approved."), - observation(EventWaitlistInvited, CategoryLifecycle, "A waitlist entry was invited."), - observation(EventWaitlistConverted, CategoryLifecycle, "A waitlist entry converted to a user."), - observation(EventWaitlistRejected, CategoryLifecycle, "A waitlist entry was rejected."), - mutation(EventGDPRExportReq, CategoryLifecycle, "A GDPR data export was requested."), - mutation(EventGDPRDeletionReq, CategoryLifecycle, "A GDPR deletion was requested."), - mutation(EventGDPRDeletionDone, CategoryLifecycle, "A GDPR deletion completed."), - - revised(mutation(EventWebhookCreated, CategorySystem, "A webhook subscription was created.", webhookAdminFields...), webhookAdminVersion), - revised(mutation(EventWebhookDeleted, CategorySystem, "A webhook subscription was deleted.", webhookAdminFields...), webhookAdminVersion), - revised(mutation(EventWebhookReplayed, CategorySystem, "A webhook delivery was replayed.", webhookAdminFields...), webhookAdminVersion), + PayloadField{Name: "access_token_revoked", Kind: FieldBool, Required: true})), + + contentRetained(observation(EventBillingCheckoutStarted, CategoryBilling, "A billing checkout session was started.")), + contentRetained(observation(EventBillingPortalOpened, CategoryBilling, "The billing portal was opened.")), + contentRetained(observation(EventBillingFreePlan, CategoryBilling, "The free plan was selected.")), + securityRetained(mutation(EventEntitlementOverride, CategoryBilling, "An entitlement override was set.", str("key"))), + + securityRetained(mutation(EventOrgCreated, CategoryOrganization, "An organization was created.", str("name"))), + securityRetained(mutation(EventOrgMemberAdded, CategoryOrganization, "A member was added to an organization.")), + securityRetained(mutation(EventOrgMemberRemoved, CategoryOrganization, "A member was removed from an organization.")), + securityRetained(mutation(EventOrgMemberLeft, CategoryOrganization, "A member left an organization.")), + securityRetained(mutation(EventOrgUpdated, CategoryOrganization, "An organization was renamed or its slug changed.", str("name"), str("slug"))), + securityRetained(mutation(EventOrgDeleted, CategoryOrganization, "An organization was deleted: archived, its members removed and its credentials revoked.", str("slug"))), + securityRetained(mutation(EventOrgSettingsUpdated, CategoryOrganization, "Organization branding settings were updated.")), + securityRetained(mutation(EventOrgGenericSettingsUpdated, CategoryOrganization, "Organization generic (typed) settings were updated.")), + securityRetained(mutation(EventTeamCreated, CategoryOrganization, "A team was created.", str("name"))), + securityRetained(mutation(EventTeamUpdated, CategoryOrganization, "A team was updated.")), + securityRetained(mutation(EventTeamDeleted, CategoryOrganization, "A team was deleted.")), + securityRetained(mutation(EventTeamMemberAdded, CategoryOrganization, "A member was added to a team.")), + securityRetained(mutation(EventTeamMemberRemoved, CategoryOrganization, "A member was removed from a team.")), + securityRetained(mutation(EventSSOSetupStarted, CategoryOrganization, "SSO configuration was started.")), + securityRetained(mutation(EventSSODisabled, CategoryOrganization, "SSO was disabled for an organization.")), + contentRetained(observation(EventOnboardingStepDone, CategoryOrganization, "An onboarding step was completed.", str("step"))), + contentRetained(observation(EventOnboardingStepSkip, CategoryOrganization, "An onboarding step was skipped.", str("step"))), + contentRetained(observation(EventActivationAchieved, CategoryOrganization, "An organization reached activation.")), + + contentRetained(observation(EventDashboardCreated, CategoryOrganization, "A dashboard was created.")), + contentRetained(observation(EventDashboardUpdated, CategoryOrganization, "A dashboard was updated.")), + contentRetained(observation(EventDashboardDeleted, CategoryOrganization, "A dashboard was deleted.")), + securityRetained(observation(EventDashboardShared, CategoryOrganization, "A dashboard's visibility was changed.")), + + contentRetained(observation(EventWaitlistJoined, CategoryLifecycle, "A prospect joined the waitlist.", pii(str("email")))), + contentRetained(observation(EventWaitlistPending, CategoryLifecycle, "A waitlist entry moved to pending.")), + contentRetained(observation(EventWaitlistVerified, CategoryLifecycle, "A waitlist entry was verified.")), + securityRetained(observation(EventWaitlistReviewed, CategoryLifecycle, "A waitlist entry was reviewed by an administrator.")), + securityRetained(observation(EventWaitlistApproved, CategoryLifecycle, "A waitlist entry was approved.")), + securityRetained(observation(EventWaitlistInvited, CategoryLifecycle, "A waitlist entry was invited.")), + contentRetained(observation(EventWaitlistConverted, CategoryLifecycle, "A waitlist entry converted to a user.")), + securityRetained(observation(EventWaitlistRejected, CategoryLifecycle, "A waitlist entry was rejected.")), + securityRetained(mutation(EventGDPRExportReq, CategoryLifecycle, "A GDPR data export was requested.")), + securityRetained(mutation(EventGDPRDeletionReq, CategoryLifecycle, "A GDPR deletion was requested.")), + securityRetained(mutation(EventGDPRDeletionDone, CategoryLifecycle, "A GDPR deletion completed.")), + + securityRetained(revised(mutation(EventWebhookCreated, CategorySystem, "A webhook subscription was created.", webhookAdminFields...), webhookAdminVersion)), + securityRetained(revised(mutation(EventWebhookDeleted, CategorySystem, "A webhook subscription was deleted.", webhookAdminFields...), webhookAdminVersion)), + securityRetained(revised(mutation(EventWebhookReplayed, CategorySystem, "A webhook delivery was replayed.", webhookAdminFields...), webhookAdminVersion)), // v2 records how a GitHub source authenticates — including `public`, a // source connected with no credential at all — and declares the `provider` // the provider-agnostic connect has always written, which v1 dropped. - revised(mutation(EventDatasourceSourceAdded, CategorySystem, "A datasource was connected.", - str("repo"), str("provider"), enum("credential_kind", "pat", "app", "public")), 2), - mutation(EventDatasourceGitHubAppSetupStarted, CategorySystem, "GitHub App setup was started for an organization."), - observation(EventDatasourceAccountLinkStarted, CategorySystem, "A person started linking a provider account.", str("connector")), - mutation(EventDatasourceAccountLinked, CategorySystem, "A person linked a provider account they signed in as.", - str("connector"), str("provider_account_id")), - mutation(EventDatasourceAccountUnlinked, CategorySystem, "A linked provider account was removed.", - str("connector"), str("provider_account_id"), str("user_id")), - mutation(EventDatasourceGroupBound, CategorySystem, "An administrator bound a provider group to a team.", - str("connector"), str("provider_group_id"), str("team_id")), - mutation(EventDatasourceGroupUnbound, CategorySystem, "A provider group binding was removed.", - str("connector"), str("provider_group_id"), str("team_id")), - mutation(EventDatasourceDomainClaimed, CategorySystem, "An administrator claimed a domain for the organization.", str("domain")), - mutation(EventDatasourceDomainVerified, CategorySystem, "A claimed domain was verified by its DNS TXT record.", str("domain")), - mutation(EventDatasourceDomainRemoved, CategorySystem, "A claimed domain was removed.", str("domain")), - mutation(EventDatasourceGitHubAppSetupCompleted, CategorySystem, "A GitHub App installation was verified and bound to an organization.", - str("installation_id")), - mutation(EventSourceDelegationCreated, CategoryAccess, "A person connecting a datasource delegated its sync to a module's installed operation binding.", - sourceDelegationFields...), + securityRetained(revised(mutation(EventDatasourceSourceAdded, CategorySystem, "A datasource was connected.", + str("repo"), str("provider"), enum("credential_kind", "pat", "app", "public")), 2)), + securityRetained(mutation(EventDatasourceGitHubAppSetupStarted, CategorySystem, "GitHub App setup was started for an organization.")), + contentRetained(observation(EventDatasourceAccountLinkStarted, CategorySystem, "A person started linking a provider account.", str("connector"))), + securityRetained(mutation(EventDatasourceAccountLinked, CategorySystem, "A person linked a provider account they signed in as.", + str("connector"), str("provider_account_id"))), + securityRetained(mutation(EventDatasourceAccountUnlinked, CategorySystem, "A linked provider account was removed.", + str("connector"), str("provider_account_id"), str("user_id"))), + securityRetained(mutation(EventDatasourceGroupBound, CategorySystem, "An administrator bound a provider group to a team.", + str("connector"), str("provider_group_id"), str("team_id"))), + securityRetained(mutation(EventDatasourceGroupUnbound, CategorySystem, "A provider group binding was removed.", + str("connector"), str("provider_group_id"), str("team_id"))), + securityRetained(mutation(EventDatasourceDomainClaimed, CategorySystem, "An administrator claimed a domain for the organization.", str("domain"))), + securityRetained(mutation(EventDatasourceDomainVerified, CategorySystem, "A claimed domain was verified by its DNS TXT record.", str("domain"))), + securityRetained(mutation(EventDatasourceDomainRemoved, CategorySystem, "A claimed domain was removed.", str("domain"))), + securityRetained(mutation(EventDatasourceGitHubAppSetupCompleted, CategorySystem, "A GitHub App installation was verified and bound to an organization.", + str("installation_id"))), + securityRetained(mutation(EventSourceDelegationCreated, CategoryAccess, "A person connecting a datasource delegated its sync to a module's installed operation binding.", + sourceDelegationFields...)), // v2 records `lookup`: a delegation now mints for one call at a time, and a // capability narrowed to recovering a receipt carries strictly less than // one that may produce the effect. A trail that cannot tell the two apart // cannot answer what a module was actually let do. - revised(mutation(EventSourceDelegationUsed, CategoryAccess, "A module was issued an operation context from a person's source delegation.", - append(slices.Clone(sourceDelegationFields), str("audience"), strs("scopes"), boolean("lookup"))...), 2), - mutation(EventSourceDelegationRevoked, CategoryAccess, "A source delegation was revoked.", - append(slices.Clone(sourceDelegationFields), enum("reason", SourceDelegationRevocationReasons...))...), - observation(EventDatasourceSourceSynced, CategorySystem, "A datasource sync was requested.", str("job_id"), str("repo")), - revised(mutation(EventDatasourceCredentialUpdated, CategorySystem, "A datasource credential was validated and replaced.", - str("repo"), enum("credential_kind", "pat", "app", "public")), 2), + securityRetained(revised(mutation(EventSourceDelegationUsed, CategoryAccess, "A module was issued an operation context from a person's source delegation.", + append(slices.Clone(sourceDelegationFields), str("audience"), strs("scopes"), boolean("lookup"))...), 2)), + securityRetained(mutation(EventSourceDelegationRevoked, CategoryAccess, "A source delegation was revoked.", + append(slices.Clone(sourceDelegationFields), enum("reason", SourceDelegationRevocationReasons...))...)), + contentRetained(observation(EventDatasourceSourceSynced, CategorySystem, "A datasource sync was requested.", str("job_id"), str("repo"))), + securityRetained(revised(mutation(EventDatasourceCredentialUpdated, CategorySystem, "A datasource credential was validated and replaced.", + str("repo"), enum("credential_kind", "pat", "app", "public")), 2)), // v2 names what was removed — v1 recorded an empty payload, so the trail // could not say which repository or which collection lost its source. // `boundary` is the collection (boundary node) the source fed, spelled as @@ -743,83 +785,83 @@ var auditEventCatalog = []AuditEventDefinition{ // would accept a provider that could never have been stored, and the // module-facing EmitAuditEvent — the one path where registry validation // rejects rather than warns — would pass it through. - revised(mutation(EventDatasourceSourceRemoved, CategorySystem, "A datasource was removed.", + securityRetained(revised(mutation(EventDatasourceSourceRemoved, CategorySystem, "A datasource was removed.", enum("provider", DatasourceProviderGitHub, DatasourceProviderAPI, DatasourceProviderCrawler, DatasourceProviderUpload), - str("repo"), str("boundary")), 2), - observation(EventDatasourceSyncCompleted, CategorySystem, "A datasource ingestion job completed.", sourceSyncFields...), - observation(EventDatasourceSyncFailed, CategorySystem, "A datasource ingestion attempt failed and may retry.", sourceSyncFields...), - observation(EventDatasourceChangeSetCompiled, CategorySystem, "A GitHub delivery was compiled into a change set.", - str("base"), str("head"), PayloadField{Name: "ops", Kind: FieldInt}, enum("mode", "compare", "snapshot"), str("delivery_id")), - observation(EventDatasourceForcePushReconciled, CategorySystem, "A GitHub force push or divergence was reconciled with a snapshot.", - str("head"), str("delivery_id")), - observation(EventDatasourceBranchDeleted, CategorySystem, "A GitHub branch-deletion delivery was acknowledged without removing documents.", - str("ref"), str("delivery_id")), - observation(EventDatasourceSnapshotTooLarge, CategorySystem, "A datasource snapshot manifest exceeded the ingest payload limit; the source was degraded pending operator reset.", - str("head"), PayloadField{Name: "bytes", Kind: FieldInt}, PayloadField{Name: "limit", Kind: FieldInt}, str("delivery_id")), - observation(EventDatasourceSourceRecovered, CategorySystem, "A degraded datasource source snapshotted within the ingest limit again and was returned to active.", - str("head"), str("delivery_id")), + str("repo"), str("boundary")), 2)), + contentRetained(observation(EventDatasourceSyncCompleted, CategorySystem, "A datasource ingestion job completed.", sourceSyncFields...)), + contentRetained(observation(EventDatasourceSyncFailed, CategorySystem, "A datasource ingestion attempt failed and may retry.", sourceSyncFields...)), + contentRetained(observation(EventDatasourceChangeSetCompiled, CategorySystem, "A GitHub delivery was compiled into a change set.", + str("base"), str("head"), PayloadField{Name: "ops", Kind: FieldInt}, enum("mode", "compare", "snapshot"), str("delivery_id"))), + contentRetained(observation(EventDatasourceForcePushReconciled, CategorySystem, "A GitHub force push or divergence was reconciled with a snapshot.", + str("head"), str("delivery_id"))), + contentRetained(observation(EventDatasourceBranchDeleted, CategorySystem, "A GitHub branch-deletion delivery was acknowledged without removing documents.", + str("ref"), str("delivery_id"))), + contentRetained(observation(EventDatasourceSnapshotTooLarge, CategorySystem, "A datasource snapshot manifest exceeded the ingest payload limit; the source was degraded pending operator reset.", + str("head"), PayloadField{Name: "bytes", Kind: FieldInt}, PayloadField{Name: "limit", Kind: FieldInt}, str("delivery_id"))), + contentRetained(observation(EventDatasourceSourceRecovered, CategorySystem, "A degraded datasource source snapshotted within the ingest limit again and was returned to active.", + str("head"), str("delivery_id"))), // v2 adds the third cause, public_repository_unreadable: a source connected // to a public repository that GitHub has stopped serving unauthenticated. // It has no installation, so installation_id is empty on those records — // which is also how a consumer tells the two families apart without reading // the reason. - revised(observation(EventDatasourceSourceAccessLost, CategorySystem, + contentRetained(revised(observation(EventDatasourceSourceAccessLost, CategorySystem, "A datasource source lost access to its repository — a GitHub App installation stopped granting it, or a public repository stopped being readable without a credential; the source was degraded until access returns.", str("repo"), str("installation_id"), enum("reason", DatasourceAccessLostRepositoryUnavailable, DatasourceAccessLostSuspended, - DatasourceAccessLostPublicRepositoryUnreadable)), 2), - revised(observation(EventDatasourceSourceAccessRestored, CategorySystem, + DatasourceAccessLostPublicRepositoryUnreadable)), 2)), + contentRetained(revised(observation(EventDatasourceSourceAccessRestored, CategorySystem, "A datasource source could read its repository again and was returned to active.", str("repo"), str("installation_id"), enum("restored_from", DatasourceAccessLostRepositoryUnavailable, DatasourceAccessLostSuspended, - DatasourceAccessLostPublicRepositoryUnreadable)), 2), - observation(EventDatasourceBlobFetched, CategorySystem, "A module fetched a datasource blob's bytes over FetchDatasourceBlob.", - str("repo"), str("blob_sha"), PayloadField{Name: "bytes", Kind: FieldInt}), - observation(EventDatasourceFilesFetched, CategorySystem, "A module fetched a batch of a datasource's files at one version over FetchDatasourceFiles.", - str("repo"), str("version"), PayloadField{Name: "files", Kind: FieldInt}, PayloadField{Name: "bytes", Kind: FieldInt}), - revised(mutation(EventWebhookSecretRotated, CategorySystem, "A webhook signing secret was rotated.", webhookAdminFields...), webhookAdminVersion), - observation(EventJobReplayed, CategorySystem, "A background job was replayed."), - mutation(EventFeatureFlagUpdated, CategorySystem, "A legacy feature flag was updated."), - mutation(EventEventSubscriptionCreated, CategorySystem, "A domain-event subscription was created.", - uid("subscription_id"), uid("subscriber_principal_id"), str("type_pattern"), str("queue")), - mutation(EventEventSubscriptionRevoked, CategorySystem, "A domain-event subscription was revoked.", uid("subscription_id")), - mutation(EventEventReplayed, CategorySystem, "Domain events were replayed to a subscriber.", - str("type"), PayloadField{Name: "redelivered", Kind: FieldInt}), - observation(EventDocumentRead, CategoryAccess, "A document read returned evidence or an explicit outcome.", documentReadFields...), - observation(EventDocumentSearch, CategoryAccess, "A collection search returned evidence or an explicit outcome.", documentReadFields...), - mutation(EventDocumentIngested, CategoryLifecycle, "A document was ingested into a solution.", documentFields...), - mutation(EventDocumentVersionMinted, CategoryLifecycle, "A new document version was minted.", documentFields...), - mutation(EventDocumentRenamed, CategoryLifecycle, "A document was renamed.", documentFields...), - mutation(EventDocumentDeleted, CategoryLifecycle, "A document was deleted.", documentFields...), - mutation(EventDocumentArchived, CategoryLifecycle, "A document was archived: taken out of the listing, every version kept.", documentFields...), - mutation(EventDocumentUnarchived, CategoryLifecycle, "A document was unarchived: listed again.", documentFields...), - mutation(EventDocumentQuarantined, CategoryLifecycle, "A document was quarantined.", documentFields...), + DatasourceAccessLostPublicRepositoryUnreadable)), 2)), + contentRetained(observation(EventDatasourceBlobFetched, CategorySystem, "A module fetched a datasource blob's bytes over FetchDatasourceBlob.", + str("repo"), str("blob_sha"), PayloadField{Name: "bytes", Kind: FieldInt})), + contentRetained(observation(EventDatasourceFilesFetched, CategorySystem, "A module fetched a batch of a datasource's files at one version over FetchDatasourceFiles.", + str("repo"), str("version"), PayloadField{Name: "files", Kind: FieldInt}, PayloadField{Name: "bytes", Kind: FieldInt})), + securityRetained(revised(mutation(EventWebhookSecretRotated, CategorySystem, "A webhook signing secret was rotated.", webhookAdminFields...), webhookAdminVersion)), + securityRetained(observation(EventJobReplayed, CategorySystem, "A background job was replayed.")), + securityRetained(mutation(EventFeatureFlagUpdated, CategorySystem, "A legacy feature flag was updated.")), + securityRetained(mutation(EventEventSubscriptionCreated, CategorySystem, "A domain-event subscription was created.", + uid("subscription_id"), uid("subscriber_principal_id"), str("type_pattern"), str("queue"))), + securityRetained(mutation(EventEventSubscriptionRevoked, CategorySystem, "A domain-event subscription was revoked.", uid("subscription_id"))), + securityRetained(mutation(EventEventReplayed, CategorySystem, "Domain events were replayed to a subscriber.", + str("type"), PayloadField{Name: "redelivered", Kind: FieldInt})), + contentRetained(observation(EventDocumentRead, CategoryAccess, "A document read returned evidence or an explicit outcome.", documentReadFields...)), + contentRetained(observation(EventDocumentSearch, CategoryAccess, "A collection search returned evidence or an explicit outcome.", documentReadFields...)), + contentRetained(mutation(EventDocumentIngested, CategoryLifecycle, "A document was ingested into a solution.", documentFields...)), + contentRetained(mutation(EventDocumentVersionMinted, CategoryLifecycle, "A new document version was minted.", documentFields...)), + contentRetained(mutation(EventDocumentRenamed, CategoryLifecycle, "A document was renamed.", documentFields...)), + contentRetained(mutation(EventDocumentDeleted, CategoryLifecycle, "A document was deleted.", documentFields...)), + contentRetained(mutation(EventDocumentArchived, CategoryLifecycle, "A document was archived: taken out of the listing, every version kept.", documentFields...)), + contentRetained(mutation(EventDocumentUnarchived, CategoryLifecycle, "A document was unarchived: listed again.", documentFields...)), + securityRetained(mutation(EventDocumentQuarantined, CategoryLifecycle, "A document was quarantined.", documentFields...)), // Version 2 of the release and the subscription pair: `outcome` became // required when a refusal stopped being its own type, so a v1 row (no // outcome) and a v2 row are told apart by schema_version, not guessed at. - revised(mutation(EventDocumentQuarantineReleased, CategoryLifecycle, "A document was released from quarantine, or the release refused (outcome failure).", + securityRetained(revised(mutation(EventDocumentQuarantineReleased, CategoryLifecycle, "A document was released from quarantine, or the release refused (outcome failure).", append(append([]PayloadField(nil), documentOutcomeFields...), - PayloadField{Name: "tenant_mismatch", Kind: FieldBool}, PayloadField{Name: "solution_mismatch", Kind: FieldBool})...), 2), - revised(mutation(EventDocumentSubscribed, CategoryLifecycle, "A subscription to a document was created, or refused (outcome failure).", documentOutcomeFields...), 2), - revised(mutation(EventDocumentUnsubscribed, CategoryLifecycle, "A subscription to a document was removed, or the removal refused (outcome failure).", documentOutcomeFields...), 2), - mutation(EventDocumentOwnershipTransferred, CategoryLifecycle, "A document's owner was reassigned, or the transfer refused (outcome failure).", - append(append([]PayloadField(nil), documentOutcomeFields...), str("new_owner_subject_id"))...), - mutation(EventDocumentFrozen, CategoryLifecycle, "A document was frozen into a boundary-governed record, or the freeze refused (outcome failure). A freeze lasts until a saas.document.unfrozen with outcome success.", documentOutcomeFields...), - mutation(EventDocumentUnfrozen, CategoryLifecycle, "A frozen document was released from its freeze, or the unfreeze refused (outcome failure).", documentOutcomeFields...), - observation(EventDocumentIngestSkippedStale, CategoryLifecycle, "A document ingest op was refused as behind the order already applied at its path; nothing was written.", - append(append([]PayloadField(nil), documentFields...), str("path"), PayloadField{Name: "ordinal", Kind: FieldInt})...), - observation(EventDocumentPayloadConflict, CategorySystem, "A producer re-ran and offered different bytes for an artifact already stored; the stored bytes were kept.", + PayloadField{Name: "tenant_mismatch", Kind: FieldBool}, PayloadField{Name: "solution_mismatch", Kind: FieldBool})...), 2)), + contentRetained(revised(mutation(EventDocumentSubscribed, CategoryLifecycle, "A subscription to a document was created, or refused (outcome failure).", documentOutcomeFields...), 2)), + contentRetained(revised(mutation(EventDocumentUnsubscribed, CategoryLifecycle, "A subscription to a document was removed, or the removal refused (outcome failure).", documentOutcomeFields...), 2)), + securityRetained(mutation(EventDocumentOwnershipTransferred, CategoryLifecycle, "A document's owner was reassigned, or the transfer refused (outcome failure).", + append(append([]PayloadField(nil), documentOutcomeFields...), str("new_owner_subject_id"))...)), + securityRetained(mutation(EventDocumentFrozen, CategoryLifecycle, "A document was frozen into a boundary-governed record, or the freeze refused (outcome failure). A freeze lasts until a saas.document.unfrozen with outcome success.", documentOutcomeFields...)), + securityRetained(mutation(EventDocumentUnfrozen, CategoryLifecycle, "A frozen document was released from its freeze, or the unfreeze refused (outcome failure).", documentOutcomeFields...)), + contentRetained(observation(EventDocumentIngestSkippedStale, CategoryLifecycle, "A document ingest op was refused as behind the order already applied at its path; nothing was written.", + append(append([]PayloadField(nil), documentFields...), str("path"), PayloadField{Name: "ordinal", Kind: FieldInt})...)), + contentRetained(observation(EventDocumentPayloadConflict, CategorySystem, "A producer re-ran and offered different bytes for an artifact already stored; the stored bytes were kept.", str("solution"), str("producer"), str("producer_version"), str("entry"), str("entry_version"), - PayloadField{Name: "stored_bytes", Kind: FieldInt}, PayloadField{Name: "offered_bytes", Kind: FieldInt}), - mutation(EventDocumentSnapshotCommitted, CategoryLifecycle, "A complete source listing was reconciled into a document scope as one effect.", documentSnapshotFields...), - observation(EventDocumentSnapshotSkippedStale, CategoryLifecycle, "A source listing was refused as older than the order its scope already holds; nothing was written.", documentSnapshotFields...), - mutation(EventDocumentEffectCommitted, CategoryLifecycle, "A batch of document changes committed as one effect.", - str("solution"), str("digest"), PayloadField{Name: "applied", Kind: FieldInt}, PayloadField{Name: "deleted", Kind: FieldInt}), - mutation(EventDocumentProductionCommitted, CategoryLifecycle, "A producer's derived artifact for a document version committed as one effect.", - str("solution"), str("effect_key"), str("task_id"), str("digest"), str("producer"), str("producer_version")), - mutation(EventDocumentKnowledgePublished, CategoryLifecycle, "A knowledge card was published into a collection as one effect.", - append(append([]PayloadField(nil), documentFields...), str("effect_key"), str("run_id"), str("digest"))...), + PayloadField{Name: "stored_bytes", Kind: FieldInt}, PayloadField{Name: "offered_bytes", Kind: FieldInt})), + contentRetained(mutation(EventDocumentSnapshotCommitted, CategoryLifecycle, "A complete source listing was reconciled into a document scope as one effect.", documentSnapshotFields...)), + contentRetained(observation(EventDocumentSnapshotSkippedStale, CategoryLifecycle, "A source listing was refused as older than the order its scope already holds; nothing was written.", documentSnapshotFields...)), + contentRetained(mutation(EventDocumentEffectCommitted, CategoryLifecycle, "A batch of document changes committed as one effect.", + str("solution"), str("digest"), PayloadField{Name: "applied", Kind: FieldInt}, PayloadField{Name: "deleted", Kind: FieldInt})), + contentRetained(mutation(EventDocumentProductionCommitted, CategoryLifecycle, "A producer's derived artifact for a document version committed as one effect.", + str("solution"), str("effect_key"), str("task_id"), str("digest"), str("producer"), str("producer_version"))), + contentRetained(mutation(EventDocumentKnowledgePublished, CategoryLifecycle, "A knowledge card was published into a collection as one effect.", + append(append([]PayloadField(nil), documentFields...), str("effect_key"), str("run_id"), str("digest"))...)), // Everything this record exists to say is required, because a redrive row // that names no version, no stage or no operation is indistinguishable from a // complete one while answering none of the questions it was written to @@ -842,12 +884,12 @@ var auditEventCatalog = []AuditEventDefinition{ // input. `producer` cannot be an enum — the host does not know a module's // stages — so it is bounded instead: short enough to name a stage, too short // to carry the failure text the spine deliberately does not hold. - requiresIdempotencyKey(requiresEntry(mutation(EventDocumentDeadLetterRedriven, CategoryLifecycle, "An operator re-queued a document's dead-lettered derivation.", + securityRetained(requiresIdempotencyKey(requiresEntry(mutation(EventDocumentDeadLetterRedriven, CategoryLifecycle, "An operator re-queued a document's dead-lettered derivation.", append(required(documentFields, "version"), PayloadField{Name: "correlation_id", Kind: FieldString, Required: true, MaxLen: 255}, PayloadField{Name: "producer", Kind: FieldString, Required: true, MaxLen: 128}, PayloadField{Name: "error_class", Kind: FieldEnum, Required: true, - Enum: []string{"permanent", "exhausted", "cancelled", "unknown"}})...))), + Enum: []string{"permanent", "exhausted", "cancelled", "unknown"}})...)))), } // webhookAdminVersion is version 2 of the webhook administration events: the @@ -936,6 +978,9 @@ var auditEventIndex = func() map[EventType]AuditEventDefinition { if d.Durability != DurabilityTransactional && d.Durability != DurabilityObservational { panic(fmt.Sprintf("audit registry: event type %q has no durability classification", d.Type)) } + if d.Retention != RetentionSecurity && d.Retention != RetentionContent { + panic(fmt.Sprintf("audit registry: event type %q has no retention class", d.Type)) + } m[d.Type] = d } return m diff --git a/module/services/accounts/code/pkg/business/audit_relay.go b/module/services/accounts/code/pkg/business/audit_relay.go new file mode 100644 index 000000000..c10dc4189 --- /dev/null +++ b/module/services/accounts/code/pkg/business/audit_relay.go @@ -0,0 +1,344 @@ +package business + +import ( + "context" + "errors" + "fmt" + "strconv" + "strings" + "sync" + "time" + + "github.com/codefly-dev/core/wool" + "github.com/google/uuid" +) + +// QueuedAuditEvent is one row of the transactional audit queue. +type QueuedAuditEvent struct { + // Seq is the queue's insertion order. + Seq int64 + Entry AuditEntry + // EnqueuedAt is when the transaction that queued the event began. + EnqueuedAt time.Time +} + +// AuditQueue is the relay's side of the transactional queue. +type AuditQueue interface { + // Drain hands deliver the oldest queued events whose transactions have + // committed — at most limit, in queue order — while holding a lease no other + // relay holds at the same time. When deliver reports true it deletes exactly + // those rows and returns how many; when deliver reports false or an error it + // deletes nothing. A relay that finds the lease held elsewhere returns 0 and + // no error without calling deliver. + // + // The queue never reports an event before every event queued ahead of it has + // committed or rolled back, so a batch cannot overtake an earlier event of + // the same organization that was still in flight. + Drain(ctx context.Context, limit int, deliver func(ctx context.Context, events []QueuedAuditEvent) (bool, error)) (int, error) +} + +// Relay defaults. A batch of 500 is BigQuery's recommended streaming request +// size; five seconds bounds how long a quiet deployment's events wait. +const ( + DefaultAuditRelayBatchSize = 500 + MaxAuditRelayBatchSize = 5000 + DefaultAuditRelayMaxWait = 5 * time.Second + // auditRelayAttemptTimeout bounds one delivery, which holds a queue + // transaction open across the archive and warehouse writes. + auditRelayAttemptTimeout = 2 * time.Minute + auditRelayMaxBackoff = time.Minute +) + +// AuditRelayConfig wires a relay. +type AuditRelayConfig struct { + Queue AuditQueue + Store AuditStoreWriter + Archive AuditArchive + // Types serves the solution-declared half of the registry; nil resolves the + // code catalog alone. + Types DeclaredAuditEventTypeReader + // DeploymentID stamps every record with the deployment it was written in. + DeploymentID string + // BatchSize is the most events one delivery carries. + BatchSize int + // MaxWait is how long a partial batch may wait for more events before it is + // delivered anyway, measured from its oldest event. + MaxWait time.Duration + + // Now and NewBatchID are seams for tests; nil uses the clock and a UUID. + Now func() time.Time + NewBatchID func() string +} + +// AuditRelay drains the transactional audit queue into the store of record and +// the archive (ADR 0009). Delivery is at-least-once: each batch is written to +// the archive as one object, then appended to the store, and its queue rows are +// deleted only after both writes were acknowledged. A crash anywhere before +// that delete leaves the rows queued and the batch is delivered again — the +// store keeps one record per event id, and the archive's readers deduplicate. +// +// The archive is written first so that a failure between the two writes +// leaves its duplicate in the archive, where readers already deduplicate, and +// not in the store. +// +// While it runs, the relay remembers which writes the batch at the head of the +// queue has already had acknowledged, so retrying it during an outage of one +// side does not write it to the other side again: a warehouse outage does not +// fill the locked archive with copies of one batch, and a lost delete is +// retried as a delete. Only a restart forgets, which is the at-least-once case +// both sides are built to absorb. +type AuditRelay struct { + queue AuditQueue + store AuditStoreWriter + archive AuditArchive + types DeclaredAuditEventTypeReader + deploymentID string + batchSize int + maxWait time.Duration + now func() time.Time + newBatchID func() string + + // drainMu serializes drains, which share head. + drainMu sync.Mutex + head batchProgress + + mu sync.Mutex + started bool + cancel context.CancelFunc + done chan struct{} +} + +// NewAuditRelay validates cfg and builds a relay. +func NewAuditRelay(cfg AuditRelayConfig) (*AuditRelay, error) { + if cfg.Queue == nil || cfg.Store == nil || cfg.Archive == nil { + return nil, errors.New("audit relay: queue, store and archive are required") + } + if strings.TrimSpace(cfg.DeploymentID) == "" { + return nil, errors.New("audit relay: deployment id is required") + } + if cfg.BatchSize < 1 || cfg.BatchSize > MaxAuditRelayBatchSize { + return nil, fmt.Errorf("audit relay: batch size must be between 1 and %d", MaxAuditRelayBatchSize) + } + if cfg.MaxWait <= 0 { + return nil, errors.New("audit relay: max wait must be positive") + } + relay := &AuditRelay{ + queue: cfg.Queue, + store: cfg.Store, + archive: cfg.Archive, + types: cfg.Types, + deploymentID: cfg.DeploymentID, + batchSize: cfg.BatchSize, + maxWait: cfg.MaxWait, + now: cfg.Now, + newBatchID: cfg.NewBatchID, + } + if relay.now == nil { + relay.now = time.Now + } + if relay.newBatchID == nil { + relay.newBatchID = func() string { return uuid.NewString() } + } + return relay, nil +} + +// DrainOnce delivers every batch that is due — each full batch, then a partial +// one whose oldest event has waited MaxWait — and returns how many events it +// delivered. It stops at the first failure, which leaves the failed batch and +// everything behind it queued, in order, for the next attempt. +func (r *AuditRelay) DrainOnce(ctx context.Context) (int, error) { + r.drainMu.Lock() + defer r.drainMu.Unlock() + delivered := 0 + for { + n, err := r.queue.Drain(ctx, r.batchSize, r.deliver) + delivered += n + if err != nil { + return delivered, err + } + if n > 0 { + // Its rows are gone: nothing about the head batch is worth remembering. + r.head = batchProgress{} + } + // Fewer than a full batch means the queue held no more that was due: + // nothing, a partial batch still waiting, a partial batch now delivered, + // or a lease held by another relay. + if n < r.batchSize { + return delivered, nil + } + } +} + +// batchProgress is what the relay knows about the batch at the head of the +// queue: which queued rows it holds, and which of the two writes acknowledged it. +type batchProgress struct { + rows string + batch AuditBatch + archived bool + appended bool +} + +// rowsKey identifies a run of queued rows by their sequence numbers. +func rowsKey(events []QueuedAuditEvent) string { + var key strings.Builder + for _, event := range events { + key.WriteString(strconv.FormatInt(event.Seq, 36)) + key.WriteByte(',') + } + return key.String() +} + +// deliver is the relay's half of one Drain: decide whether the events are due, +// compose them into a batch, and write it to the archive and then the store — +// each only if it has not already acknowledged these exact rows. +func (r *AuditRelay) deliver(ctx context.Context, events []QueuedAuditEvent) (bool, error) { + if len(events) == 0 { + return false, nil + } + if len(events) < r.batchSize && r.now().Sub(oldestEnqueued(events)) < r.maxWait { + return false, nil + } + if key := rowsKey(events); r.head.rows != key { + batch, err := r.compose(ctx, events) + if err != nil { + return false, err + } + r.head = batchProgress{rows: key, batch: batch} + } + batch := r.head.batch + if !r.head.archived { + if err := r.archive.WriteAuditBatch(ctx, batch); err != nil { + return false, fmt.Errorf("audit relay: archive batch %s: %w", batch.ID, err) + } + r.head.archived = true + } + if !r.head.appended { + if err := r.store.AppendAuditBatch(ctx, batch); err != nil { + return false, fmt.Errorf("audit relay: append batch %s: %w", batch.ID, err) + } + r.head.appended = true + } + return true, nil +} + +func oldestEnqueued(events []QueuedAuditEvent) time.Time { + oldest := events[0].EnqueuedAt + for _, event := range events[1:] { + if event.EnqueuedAt.Before(oldest) { + oldest = event.EnqueuedAt + } + } + return oldest +} + +// compose classifies and hashes each event. A registry that cannot be read is +// an error, retried with the batch — never mistaken for an unregistered type. +func (r *AuditRelay) compose(ctx context.Context, events []QueuedAuditEvent) (AuditBatch, error) { + resolver := NewAuditEventResolver(r.types) + batch := AuditBatch{ + ID: r.newBatchID(), + DeploymentID: r.deploymentID, + ComposedAt: r.now().UTC(), + Records: make([]AuditRecord, 0, len(events)), + } + for _, event := range events { + resolved, err := resolver.Resolve(ctx, event.Entry.EventType) + if err != nil { + return AuditBatch{}, err + } + record, err := NewAuditRecord(event.Entry, resolved.RetentionClass()) + if err != nil { + return AuditBatch{}, err + } + batch.Records = append(batch.Records, record) + } + return batch, nil +} + +// pollInterval is how often the loop looks for due events: a fifth of MaxWait, +// so a partial batch is delivered close to its deadline, within [100ms, 1s]. +func (r *AuditRelay) pollInterval() time.Duration { + interval := r.maxWait / 5 + if interval < 100*time.Millisecond { + interval = 100 * time.Millisecond + } + if interval > time.Second { + interval = time.Second + } + return interval +} + +// Start runs the relay loop until Shutdown. A failed delivery is logged and +// retried with a backoff that doubles up to a minute; the failed batch stays +// queued, so nothing behind it is delivered out of order. Start is idempotent. +func (r *AuditRelay) Start(parent context.Context) { + r.mu.Lock() + if r.started { + r.mu.Unlock() + return + } + r.started = true + ctx, cancel := context.WithCancel(parent) + r.cancel = cancel + r.done = make(chan struct{}) + done := r.done + r.mu.Unlock() + + go func() { + defer close(done) + w := wool.Get(ctx).In("audit.relay") + var backoff time.Duration + for { + // A delivery in flight is not cancelled by Shutdown: cancelling it + // between the store write and the queue delete is exactly what turns + // one delivery into two. It is bounded by its own timeout instead. + attempt, cancelAttempt := context.WithTimeout(context.WithoutCancel(ctx), auditRelayAttemptTimeout) + _, err := r.DrainOnce(attempt) + cancelAttempt() + wait := r.pollInterval() + if err != nil { + backoff = nextAuditRelayBackoff(backoff) + wait = backoff + w.Warn("audit relay delivery failed; the batch stays queued and is retried", + wool.Field("retry_in", backoff.String()), wool.ErrField(err)) + } else { + backoff = 0 + } + select { + case <-ctx.Done(): + return + case <-time.After(wait): + } + } + }() +} + +func nextAuditRelayBackoff(previous time.Duration) time.Duration { + if previous <= 0 { + return time.Second + } + if next := previous * 2; next < auditRelayMaxBackoff { + return next + } + return auditRelayMaxBackoff +} + +// Shutdown stops the loop and waits for a delivery in flight to finish, or for +// the caller's deadline. Events not yet delivered stay queued for the next start. +func (r *AuditRelay) Shutdown(ctx context.Context) error { + r.mu.Lock() + if !r.started { + r.mu.Unlock() + return nil + } + cancel, done := r.cancel, r.done + r.mu.Unlock() + + cancel() + select { + case <-done: + return nil + case <-ctx.Done(): + return ctx.Err() + } +} diff --git a/module/services/accounts/code/pkg/business/audit_relay_test.go b/module/services/accounts/code/pkg/business/audit_relay_test.go new file mode 100644 index 000000000..7eeb26abf --- /dev/null +++ b/module/services/accounts/code/pkg/business/audit_relay_test.go @@ -0,0 +1,544 @@ +package business + +import ( + "context" + "errors" + "fmt" + "slices" + "sync" + "testing" + "time" + + "github.com/stretchr/testify/require" +) + +// The relay's contract (ADR 0009): batches of a configured size, a partial batch +// delivered once its oldest event has waited the configured time, each batch +// written to the archive and the store before its queue rows are deleted, and +// every organization's events kept in queue order. These run it against +// in-memory fakes of the three sides. + +// memQueue is a fake AuditQueue with the Postgres queue's delete-on-success +// semantics. +type memQueue struct { + mu sync.Mutex + rows []QueuedAuditEvent + leased bool // another relay holds the lease + // crashes makes the next n successful deliveries fail before their delete + // commits, the way a crash or a lost connection between the writes and the + // delete would. + crashes int + drains int +} + +func (q *memQueue) Drain(ctx context.Context, limit int, deliver func(context.Context, []QueuedAuditEvent) (bool, error)) (int, error) { + q.mu.Lock() + q.drains++ + if q.leased { + q.mu.Unlock() + return 0, nil + } + events := slices.Clone(q.rows[:min(limit, len(q.rows))]) + q.mu.Unlock() + + delivered, err := deliver(ctx, events) + if err != nil || !delivered { + return 0, err + } + + q.mu.Lock() + defer q.mu.Unlock() + if q.crashes > 0 { + q.crashes-- + return 0, errors.New("connection lost before the delete committed") + } + q.rows = q.rows[len(events):] + return len(events), nil +} + +func (q *memQueue) remaining() []QueuedAuditEvent { + q.mu.Lock() + defer q.mu.Unlock() + return slices.Clone(q.rows) +} + +// memWarehouse is a fake store of record that keeps one record per event id, +// as BigQuery's insert-id deduplication does for a prompt redelivery. +type memWarehouse struct { + mu sync.Mutex + byID map[string]AuditRecord + appended []AuditRecord // every record in append order, duplicates included + fail func(batch AuditBatch) error +} + +func (w *memWarehouse) AppendAuditBatch(_ context.Context, batch AuditBatch) error { + w.mu.Lock() + defer w.mu.Unlock() + if w.fail != nil { + if err := w.fail(batch); err != nil { + return err + } + } + if w.byID == nil { + w.byID = map[string]AuditRecord{} + } + for _, record := range batch.Records { + w.appended = append(w.appended, record) + w.byID[record.Entry.ID] = record + } + return nil +} + +// memArchive is a fake archive: one object per write, never overwritten. +type memArchive struct { + mu sync.Mutex + objects map[string]AuditBatch + order []string + fail func(batch AuditBatch) error +} + +func (a *memArchive) WriteAuditBatch(_ context.Context, batch AuditBatch) error { + a.mu.Lock() + defer a.mu.Unlock() + if a.fail != nil { + if err := a.fail(batch); err != nil { + return err + } + } + if a.objects == nil { + a.objects = map[string]AuditBatch{} + } + if _, exists := a.objects[batch.ID]; exists { + return fmt.Errorf("object %s already exists", batch.ID) + } + a.objects[batch.ID] = batch + a.order = append(a.order, batch.ID) + return nil +} + +// eventIDs is every event id across the archive, duplicates included. +func (a *memArchive) eventIDs() []string { + a.mu.Lock() + defer a.mu.Unlock() + var ids []string + for _, name := range a.order { + for _, record := range a.objects[name].Records { + ids = append(ids, record.Entry.ID) + } + } + return ids +} + +// relayFixture wires a relay over the fakes with a controllable clock. +type relayFixture struct { + queue *memQueue + warehouse *memWarehouse + archive *memArchive + now time.Time + relay *AuditRelay + batches int +} + +func newRelayFixture(t *testing.T, batchSize int, maxWait time.Duration, types DeclaredAuditEventTypeReader) *relayFixture { + t.Helper() + f := &relayFixture{ + queue: &memQueue{}, + warehouse: &memWarehouse{}, + archive: &memArchive{}, + now: time.Date(2026, 10, 2, 12, 0, 0, 0, time.UTC), + } + f.relay = f.newRelay(t, batchSize, maxWait, types) + return f +} + +// newRelay builds a relay over the fixture's queue and stores: a second call +// is the same deployment after a restart, which remembers nothing. +func (f *relayFixture) newRelay(t *testing.T, batchSize int, maxWait time.Duration, types DeclaredAuditEventTypeReader) *AuditRelay { + t.Helper() + relay, err := NewAuditRelay(AuditRelayConfig{ + Queue: f.queue, + Store: f.warehouse, + Archive: f.archive, + Types: types, + DeploymentID: "deployment-1", + BatchSize: batchSize, + MaxWait: maxWait, + Now: func() time.Time { return f.now }, + NewBatchID: func() string { + f.batches++ + return fmt.Sprintf("batch-%d", f.batches) + }, + }) + require.NoError(t, err) + return relay +} + +// enqueue adds events for orgs in turn ("" is a platform event), queued age ago. +func (f *relayFixture) enqueue(age time.Duration, eventType EventType, orgs ...string) []string { + f.queue.mu.Lock() + defer f.queue.mu.Unlock() + var ids []string + for _, org := range orgs { + seq := int64(len(f.queue.rows) + 1) + if n := len(f.queue.rows); n > 0 { + seq = f.queue.rows[n-1].Seq + 1 + } + id := NewIDString() + ids = append(ids, id) + f.queue.rows = append(f.queue.rows, QueuedAuditEvent{ + Seq: seq, + Entry: AuditEntry{ + ID: id, + OrgID: org, + ActorID: NewIDString(), + ActorType: ActorTypeUser, + EventType: eventType, + SchemaVersion: 1, + Resource: "session", + ResourceID: fmt.Sprintf("r-%d", seq), + Payload: map[string]any{"seq": seq}, + CreatedAt: f.now.Add(-age), + }, + EnqueuedAt: f.now.Add(-age), + }) + } + return ids +} + +func repeatOrg(org string, n int) []string { + out := make([]string, n) + for i := range out { + out[i] = org + } + return out +} + +func TestAuditRelayDeliversFullBatchesThenTheDuePartialOne(t *testing.T) { + f := newRelayFixture(t, 500, 5*time.Second, nil) + ids := f.enqueue(time.Minute, EventSessionRevoked, repeatOrg(teeOrgID, 1200)...) + + delivered, err := f.relay.DrainOnce(context.Background()) + require.NoError(t, err) + require.Equal(t, 1200, delivered) + require.Empty(t, f.queue.remaining()) + + require.Len(t, f.archive.order, 3, "one archive object per batch") + var sizes []int + for _, name := range f.archive.order { + sizes = append(sizes, len(f.archive.objects[name].Records)) + } + require.Equal(t, []int{500, 500, 200}, sizes) + require.Len(t, f.warehouse.byID, 1200) + for i, record := range f.warehouse.appended { + require.Equal(t, ids[i], record.Entry.ID, "batches append in queue order") + } +} + +func TestAuditRelayHoldsAPartialBatchUntilMaxWait(t *testing.T) { + f := newRelayFixture(t, 500, 5*time.Second, nil) + f.enqueue(time.Second, EventSessionRevoked, repeatOrg(teeOrgID, 10)...) + + delivered, err := f.relay.DrainOnce(context.Background()) + require.NoError(t, err) + require.Zero(t, delivered, "a partial batch younger than max wait waits for more events") + require.Len(t, f.queue.remaining(), 10) + require.Empty(t, f.archive.order) + require.Empty(t, f.warehouse.appended) + + f.now = f.now.Add(4 * time.Second) // the oldest event has now waited 5s + delivered, err = f.relay.DrainOnce(context.Background()) + require.NoError(t, err) + require.Equal(t, 10, delivered) + require.Empty(t, f.queue.remaining()) +} + +func TestAuditRelayKeepsEachOrganizationsOrder(t *testing.T) { + const orgA, orgB = "00000000-0000-0000-0000-00000000000a", "00000000-0000-0000-0000-00000000000b" + f := newRelayFixture(t, 4, time.Second, nil) + // Interleaved organizations and platform events, as concurrent requests queue them. + f.enqueue(time.Minute, EventSessionRevoked, orgA, orgB, "", orgA, orgA, orgB, "", orgB, orgA, "") + + want := map[string][]int64{} + for _, row := range f.queue.remaining() { + want[row.Entry.OrgID] = append(want[row.Entry.OrgID], row.Seq) + } + + _, err := f.relay.DrainOnce(context.Background()) + require.NoError(t, err) + + gotStore := map[string][]int64{} + for _, record := range f.warehouse.appended { + gotStore[record.Entry.OrgID] = append(gotStore[record.Entry.OrgID], record.Entry.Payload["seq"].(int64)) + } + require.Equal(t, want, gotStore, "each organization's events, and the platform's, reach the store in queue order") + + gotArchive := map[string][]int64{} + for _, name := range f.archive.order { + for _, record := range f.archive.objects[name].Records { + gotArchive[record.Entry.OrgID] = append(gotArchive[record.Entry.OrgID], record.Entry.Payload["seq"].(int64)) + } + } + require.Equal(t, want, gotArchive, "and the archive in the same order") +} + +func TestAuditRelayDeletesOnlyAfterBothWritesAcknowledge(t *testing.T) { + t.Run("archive refuses", func(t *testing.T) { + f := newRelayFixture(t, 3, time.Second, nil) + f.enqueue(time.Minute, EventSessionRevoked, repeatOrg(teeOrgID, 3)...) + f.archive.fail = func(AuditBatch) error { return errors.New("archive unavailable") } + + delivered, err := f.relay.DrainOnce(context.Background()) + require.ErrorContains(t, err, "archive unavailable") + require.Zero(t, delivered) + require.Len(t, f.queue.remaining(), 3, "nothing is deleted before the archive acknowledges") + require.Empty(t, f.warehouse.appended, "the store is not written ahead of the archive") + }) + t.Run("store refuses", func(t *testing.T) { + f := newRelayFixture(t, 3, time.Second, nil) + f.enqueue(time.Minute, EventSessionRevoked, repeatOrg(teeOrgID, 3)...) + f.warehouse.fail = func(AuditBatch) error { return errors.New("warehouse unavailable") } + + delivered, err := f.relay.DrainOnce(context.Background()) + require.ErrorContains(t, err, "warehouse unavailable") + require.Zero(t, delivered) + require.Len(t, f.queue.remaining(), 3, "nothing is deleted before the store acknowledges") + require.Len(t, f.archive.order, 1) + + f.warehouse.fail = nil + delivered, err = f.relay.DrainOnce(context.Background()) + require.NoError(t, err) + require.Equal(t, 3, delivered) + require.Empty(t, f.queue.remaining(), "both acknowledged: the rows are deleted") + }) +} + +func TestAuditRelayCrashBetweenWriteAndDeleteLosesAndDuplicatesNothingInTheStore(t *testing.T) { + f := newRelayFixture(t, 5, time.Second, nil) + ids := f.enqueue(time.Minute, EventSessionRevoked, repeatOrg(teeOrgID, 5)...) + f.queue.crashes = 1 + + _, err := f.relay.DrainOnce(context.Background()) + require.Error(t, err) + require.Len(t, f.queue.remaining(), 5, "a crash before the delete commits leaves every event queued: nothing is lost") + require.Len(t, f.warehouse.byID, 5, "the store already holds the batch") + + // The process died with the delete: the relay that drains next is a new one. + restarted := f.newRelay(t, 5, time.Second, nil) + delivered, err := restarted.DrainOnce(context.Background()) + require.NoError(t, err) + require.Equal(t, 5, delivered) + require.Empty(t, f.queue.remaining()) + + require.Len(t, f.warehouse.appended, 10, "the batch was appended twice") + require.Len(t, f.warehouse.byID, 5, "and the store keeps one record per event id") + for _, record := range f.warehouse.appended { + require.Contains(t, ids, record.Entry.ID) + } + + require.Len(t, f.archive.order, 2, "a redelivered batch is a new archive object, never a rewrite") + archived := f.archive.eventIDs() + require.Len(t, archived, 10, "so the archive holds the events twice") + unique := slices.Compact(slices.Sorted(slices.Values(archived))) + require.ElementsMatch(t, ids, unique, "and deduplicating by event id recovers each event exactly once") + first, second := f.archive.objects[f.archive.order[0]], f.archive.objects[f.archive.order[1]] + for i := range first.Records { + require.Equal(t, first.Records[i].DetailsSHA256, second.Records[i].DetailsSHA256, + "duplicates carry the same hash, so they are provably the same event") + } +} + +// Retrying the head batch in the same process repeats only the write that has +// not been acknowledged: a warehouse outage does not fill the locked archive +// with copies of one batch, and a lost delete is retried as a delete. +func TestAuditRelayRetriesOnlyTheUnacknowledgedWrite(t *testing.T) { + t.Run("warehouse outage", func(t *testing.T) { + f := newRelayFixture(t, 3, time.Second, nil) + f.enqueue(time.Minute, EventSessionRevoked, repeatOrg(teeOrgID, 3)...) + outage := 4 + f.warehouse.fail = func(AuditBatch) error { + if outage > 0 { + outage-- + return errors.New("warehouse unavailable") + } + return nil + } + for range 4 { + _, err := f.relay.DrainOnce(context.Background()) + require.Error(t, err) + } + delivered, err := f.relay.DrainOnce(context.Background()) + require.NoError(t, err) + require.Equal(t, 3, delivered) + require.Len(t, f.archive.order, 1, "five attempts, one archive object") + require.Len(t, f.warehouse.appended, 3) + require.Equal(t, f.archive.order[0], "batch-1", "the retried batch keeps its identity") + }) + t.Run("lost delete", func(t *testing.T) { + f := newRelayFixture(t, 3, time.Second, nil) + f.enqueue(time.Minute, EventSessionRevoked, repeatOrg(teeOrgID, 3)...) + f.queue.crashes = 1 + _, err := f.relay.DrainOnce(context.Background()) + require.Error(t, err) + + delivered, err := f.relay.DrainOnce(context.Background()) + require.NoError(t, err) + require.Equal(t, 3, delivered) + require.Len(t, f.archive.order, 1) + require.Len(t, f.warehouse.appended, 3, "neither side is written twice") + }) + t.Run("a grown batch is a new batch", func(t *testing.T) { + f := newRelayFixture(t, 4, time.Second, nil) + f.enqueue(time.Minute, EventSessionRevoked, repeatOrg(teeOrgID, 2)...) + f.warehouse.fail = func(AuditBatch) error { return errors.New("warehouse unavailable") } + _, err := f.relay.DrainOnce(context.Background()) + require.Error(t, err) + + f.enqueue(time.Minute, EventSessionRevoked, repeatOrg(teeOrgID, 2)...) + f.warehouse.fail = nil + delivered, err := f.relay.DrainOnce(context.Background()) + require.NoError(t, err) + require.Equal(t, 4, delivered) + require.Len(t, f.archive.order, 2, "different rows are archived as their own object") + require.Len(t, f.archive.objects[f.archive.order[1]].Records, 4) + }) +} + +func TestAuditRelayStopsAtTheFirstFailedBatch(t *testing.T) { + f := newRelayFixture(t, 2, time.Second, nil) + ids := f.enqueue(time.Minute, EventSessionRevoked, repeatOrg(teeOrgID, 6)...) + appends := 0 + f.warehouse.fail = func(AuditBatch) error { + appends++ + if appends == 2 { + return errors.New("warehouse unavailable") + } + return nil + } + + delivered, err := f.relay.DrainOnce(context.Background()) + require.Error(t, err) + require.Equal(t, 2, delivered) + var remaining []string + for _, row := range f.queue.remaining() { + remaining = append(remaining, row.Entry.ID) + } + require.Equal(t, ids[2:], remaining, "the failed batch and everything behind it stay queued, in order") +} + +func TestAuditRelayLeaseHeldElsewhereDeliversNothing(t *testing.T) { + f := newRelayFixture(t, 2, time.Second, nil) + f.enqueue(time.Minute, EventSessionRevoked, repeatOrg(teeOrgID, 4)...) + f.queue.leased = true + + delivered, err := f.relay.DrainOnce(context.Background()) + require.NoError(t, err) + require.Zero(t, delivered) + require.Len(t, f.queue.remaining(), 4) + require.Empty(t, f.archive.order) +} + +// declaredTypes serves one declared type, or an outage. +type declaredTypes struct { + declared *DeclaredAuditEventType + err error +} + +func (d declaredTypes) GetDeclaredAuditEventType(_ context.Context, t EventType) (*DeclaredAuditEventType, error) { + if d.err != nil { + return nil, d.err + } + if d.declared != nil && d.declared.Type == t { + return d.declared, nil + } + return nil, nil +} + +func TestAuditRelayClassifiesEachRecordByItsType(t *testing.T) { + declared := declaredType("acme", AuditVisibilityTenant) + f := newRelayFixture(t, 4, time.Second, declaredTypes{declared: declared}) + login := f.enqueue(time.Minute, EventAuthLogin, teeOrgID)[0] + read := f.enqueue(time.Minute, EventDocumentRead, teeOrgID)[0] + solution := f.enqueue(time.Minute, declared.Type, teeOrgID)[0] + retired := f.enqueue(time.Minute, "saas.retired.event_type", teeOrgID)[0] + + _, err := f.relay.DrainOnce(context.Background()) + require.NoError(t, err) + + classOf := func(id string) AuditRetentionClass { return f.warehouse.byID[id].Retention } + require.Equal(t, RetentionSecurity, classOf(login), "authentication is security") + require.Equal(t, RetentionContent, classOf(read), "a content read is content") + require.Equal(t, RetentionContent, classOf(solution), "a declared type is content") + require.Equal(t, RetentionSecurity, classOf(retired), "a type the registry no longer knows keeps its details") + + for _, record := range f.warehouse.byID { + want, err := CanonicalAuditDetails(record.Entry.Payload) + require.NoError(t, err) + require.Equal(t, want, record.Details) + require.Equal(t, AuditDetailsSHA256(want), record.DetailsSHA256) + } + batch := f.archive.objects[f.archive.order[0]] + require.Equal(t, "deployment-1", batch.DeploymentID) + require.Equal(t, "batch-1", batch.ID) + require.Equal(t, f.now, batch.ComposedAt) +} + +func TestAuditRelayRegistryOutageFailsTheBatch(t *testing.T) { + declared := declaredType("acme", AuditVisibilityTenant) + f := newRelayFixture(t, 2, time.Second, declaredTypes{err: errors.New("database unavailable")}) + f.enqueue(time.Minute, declared.Type, teeOrgID, teeOrgID) + + _, err := f.relay.DrainOnce(context.Background()) + require.ErrorContains(t, err, "database unavailable") + require.Len(t, f.queue.remaining(), 2, "a type that cannot be resolved is never guessed at") + require.Empty(t, f.archive.order) +} + +func TestAuditRelayRunsUntilShutdown(t *testing.T) { + f := newRelayFixture(t, 3, 100*time.Millisecond, nil) + f.enqueue(time.Minute, EventSessionRevoked, repeatOrg(teeOrgID, 7)...) + + f.relay.Start(context.Background()) + require.Eventually(t, func() bool { return len(f.queue.remaining()) == 0 }, 5*time.Second, 10*time.Millisecond) + shutdown, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + require.NoError(t, f.relay.Shutdown(shutdown)) + require.Len(t, f.warehouse.byID, 7) +} + +func TestNewAuditRelayValidatesItsConfiguration(t *testing.T) { + valid := AuditRelayConfig{ + Queue: &memQueue{}, Store: &memWarehouse{}, Archive: &memArchive{}, + DeploymentID: "deployment-1", BatchSize: 10, MaxWait: time.Second, + } + _, err := NewAuditRelay(valid) + require.NoError(t, err) + + for name, mutate := range map[string]func(*AuditRelayConfig){ + "no queue": func(c *AuditRelayConfig) { c.Queue = nil }, + "no store": func(c *AuditRelayConfig) { c.Store = nil }, + "no archive": func(c *AuditRelayConfig) { c.Archive = nil }, + "no deployment": func(c *AuditRelayConfig) { c.DeploymentID = " " }, + "zero batch": func(c *AuditRelayConfig) { c.BatchSize = 0 }, + "oversized batch": func(c *AuditRelayConfig) { c.BatchSize = MaxAuditRelayBatchSize + 1 }, + "no max wait": func(c *AuditRelayConfig) { c.MaxWait = 0 }, + "negative waiting": func(c *AuditRelayConfig) { c.MaxWait = -time.Second }, + } { + config := valid + mutate(&config) + _, err := NewAuditRelay(config) + require.Error(t, err, name) + } +} + +func TestAuditRelayBackoffDoublesToAMinute(t *testing.T) { + var got []time.Duration + backoff := time.Duration(0) + for range 8 { + backoff = nextAuditRelayBackoff(backoff) + got = append(got, backoff) + } + require.Equal(t, []time.Duration{ + time.Second, 2 * time.Second, 4 * time.Second, 8 * time.Second, + 16 * time.Second, 32 * time.Second, time.Minute, time.Minute, + }, got) +} diff --git a/module/services/accounts/code/pkg/business/audit_retention_test.go b/module/services/accounts/code/pkg/business/audit_retention_test.go new file mode 100644 index 000000000..46d546b14 --- /dev/null +++ b/module/services/accounts/code/pkg/business/audit_retention_test.go @@ -0,0 +1,73 @@ +package business + +import ( + "testing" + + "github.com/stretchr/testify/require" +) + +// Every registered event type declares a retention class (ADR 0009), the way +// it declares a durability: a type without one cannot be registered. +func TestAuditCatalog_EveryTypeDeclaresARetentionClass(t *testing.T) { + for _, d := range auditEventCatalog { + require.Contains(t, []AuditRetentionClass{RetentionSecurity, RetentionContent}, d.Retention, + "event %q must be wrapped in securityRetained or contentRetained", d.Type) + } + for _, d := range AuditEventCatalog() { + resolved, err := NewAuditEventResolver(nil).Resolve(t.Context(), d.Type) + require.NoError(t, err) + require.Equal(t, d.Retention, resolved.RetentionClass(), "the resolver answers the declared class for %q", d.Type) + } +} + +// The security category is sign-in and MFA; every one of its types is +// security-retained, though the class reaches well beyond the category. +func TestAuditCatalog_SecurityCategoryIsSecurityRetained(t *testing.T) { + securityRetainedOutsideCategory := 0 + for _, d := range auditEventCatalog { + if d.Category == CategorySecurity { + require.Equal(t, RetentionSecurity, d.Retention, "event %q", d.Type) + } else if d.Retention == RetentionSecurity { + securityRetainedOutsideCategory++ + } + } + require.Positive(t, securityRetainedOutsideCategory, "the class is its own field, not the category renamed") +} + +// The ADR's examples of each class, held to their declared class. +func TestAuditCatalog_RetentionClassExamples(t *testing.T) { + for eventType, want := range map[EventType]AuditRetentionClass{ + EventAuthLogin: RetentionSecurity, // authentication + EventDelegatedAudienceExchange: RetentionSecurity, // includes refused exchanges + EventRoleAssigned: RetentionSecurity, // permission and role changes + EventPlatformImpersonated: RetentionSecurity, // admin actions + EventGDPRExportReq: RetentionSecurity, // data exports + EventWebhookSecretRotated: RetentionSecurity, // configuration changes + EventOrgGenericSettingsUpdated: RetentionSecurity, + EventUserUpdated: RetentionContent, + EventDocumentRead: RetentionContent, + EventDatasourceSyncCompleted: RetentionContent, + EventOnboardingStepDone: RetentionContent, + EventBillingPortalOpened: RetentionContent, + } { + d, ok := LookupAuditEvent(eventType) + require.True(t, ok, eventType) + require.Equal(t, want, d.Retention, eventType) + } +} + +func TestDeclaredAuditEventTypesAreContentRetained(t *testing.T) { + declared := declaredType("acme", AuditVisibilityExternal) + require.Equal(t, RetentionContent, declared.Definition().Retention) + + resolved, err := NewAuditEventResolver(declaredTypes{declared: declared}).Resolve(t.Context(), declared.Type) + require.NoError(t, err) + require.Equal(t, RetentionContent, resolved.RetentionClass()) +} + +func TestUnregisteredAuditEventTypesKeepTheirDetails(t *testing.T) { + resolved, err := NewAuditEventResolver(nil).Resolve(t.Context(), "saas.retired.event_type") + require.NoError(t, err) + require.False(t, resolved.Registered) + require.Equal(t, RetentionSecurity, resolved.RetentionClass()) +} diff --git a/module/services/accounts/code/pkg/business/audit_store.go b/module/services/accounts/code/pkg/business/audit_store.go new file mode 100644 index 000000000..a4541d5f2 --- /dev/null +++ b/module/services/accounts/code/pkg/business/audit_store.go @@ -0,0 +1,161 @@ +package business + +import ( + "bytes" + "context" + "crypto/sha256" + "encoding/hex" + "encoding/json" + "errors" + "fmt" + "strings" + "time" +) + +// The audit store swap (ADR 0009). Under AUDIT_SINK=postgres (the default) and +// AUDIT_SINK=both the audit store of record is the audit_events table, exactly +// as before. Under a swap value a warehouse is the store of record instead: +// the emitter writes each event into the audit_event_queue table on the +// caller's transaction, and AuditRelay drains that queue into the warehouse and +// a locked object-storage archive. + +// AuditSinkMode is the value of AUDIT_SINK. +type AuditSinkMode string + +const ( + // AuditSinkPostgres keeps every audit record in audit_events. The default. + AuditSinkPostgres AuditSinkMode = "postgres" + // AuditSinkBoth is ADR 0006's tee: audit_events stays the store of record + // and each organization's events are also posted to an HTTP endpoint. + AuditSinkBoth AuditSinkMode = "both" + // AuditSinkBigQuery is a swap value: BigQuery is the store of record and + // Postgres keeps only the transactional queue. + AuditSinkBigQuery AuditSinkMode = "bigquery" +) + +// ParseAuditSinkMode reads an AUDIT_SINK value. Empty is the default. +func ParseAuditSinkMode(raw string) (AuditSinkMode, error) { + switch strings.ToLower(strings.TrimSpace(raw)) { + case "", string(AuditSinkPostgres): + return AuditSinkPostgres, nil + case string(AuditSinkBoth): + return AuditSinkBoth, nil + case string(AuditSinkBigQuery): + return AuditSinkBigQuery, nil + case "external": + return "", errors.New("AUDIT_SINK=external is not permitted: no external destination can join the transaction " + + "a change commits in, so it cannot receive the change's record atomically; use AUDIT_SINK=both to copy " + + "records to an HTTP endpoint, or a swap value (bigquery), which commits each record to a Postgres queue first") + default: + return "", fmt.Errorf("AUDIT_SINK must be %s, %s or %s", AuditSinkPostgres, AuditSinkBoth, AuditSinkBigQuery) + } +} + +// Swaps reports whether the mode makes a warehouse the audit store of record, +// so the emitter writes the queue instead of audit_events. +func (m AuditSinkMode) Swaps() bool { + return m == AuditSinkBigQuery +} + +// AuditRecorder writes the record of one event on the caller's transaction and +// nothing more: the audit_events row, or under a swap value the queue row. It +// publishes no domain event and enqueues no tee job. It is the path for the +// writers that run outside the Service — the authentication resolver and the +// role catalog import — which recorded their events with no fan-out before the +// swap, and keep doing exactly that. DurableAuditEmitter implements it. +type AuditRecorder interface { + RecordTx(ctx context.Context, entry AuditEntry) error +} + +// AuditStoreWriter is the write half of AuditStore. +type AuditStoreWriter interface { + // AppendAuditBatch appends one relay batch, in its order. It is + // at-least-once: a batch that failed, or whose queue rows were not deleted + // afterwards, is delivered again, so the store must key each record by its + // event id and keep one record per id. + AppendAuditBatch(ctx context.Context, batch AuditBatch) error +} + +// AuditStore is the audit store of record under a swap value (ADR 0009): one +// interface carrying the write half, which the relay uses, and — once reads +// move — the read half (list, aggregate, export and the readable-source +// queries), embedded here beside AuditStoreWriter. The relay depends on +// AuditStoreWriter alone, so adding the read half changes no caller of it. +type AuditStore interface { + AuditStoreWriter +} + +// AuditArchive writes a relay batch to the locked object-storage archive as one +// object. An object is never overwritten: a batch delivered again is written as +// a new object, and archive readers deduplicate by event id. +type AuditArchive interface { + WriteAuditBatch(ctx context.Context, batch AuditBatch) error +} + +// AuditBatch is one delivery: the records of a run of queued events, in queue +// order — oldest first, which keeps every organization's events (and the +// platform's, which have none) in the order they were queued. +type AuditBatch struct { + // ID is unique to this delivery attempt; the archive names its object by it. + ID string + // DeploymentID names the deployment the events were written in. + DeploymentID string + // ComposedAt is when the relay composed the batch. + ComposedAt time.Time + Records []AuditRecord +} + +// AuditRecord is one event as a store of record keeps it. +type AuditRecord struct { + // Entry is the event's envelope as it was committed. Its Payload is not read + // downstream: Details is the one serialization of it. + Entry AuditEntry + // Retention decides where the full details go (AuditRetentionClass). + Retention AuditRetentionClass + // Details is the canonical JSON of the event's payload (CanonicalAuditDetails). + Details string + // DetailsSHA256 is the lowercase hex SHA-256 of Details. + DetailsSHA256 string +} + +// NewAuditRecord builds the record of entry under class. +func NewAuditRecord(entry AuditEntry, class AuditRetentionClass) (AuditRecord, error) { + details, err := CanonicalAuditDetails(entry.Payload) + if err != nil { + return AuditRecord{}, fmt.Errorf("audit: canonical details of event %s: %w", entry.ID, err) + } + return AuditRecord{ + Entry: entry, + Retention: class, + Details: details, + DetailsSHA256: AuditDetailsSHA256(details), + }, nil +} + +// CanonicalAuditDetails is the one serialization of an event's details that a +// store of record keeps and hashes: JSON with object keys sorted at every +// level, no insignificant whitespace, and no HTML escaping. An empty payload is +// "{}", as audit_events stores it. Numbers keep the text they were read with +// when the payload was decoded with json.Number, which the queue does. +// +// Every store keeps the string itself, never a re-encoding of it, so a reader +// verifies a record by hashing the stored string — it never has to reproduce +// this function. +func CanonicalAuditDetails(payload map[string]any) (string, error) { + if len(payload) == 0 { + return "{}", nil + } + var buf bytes.Buffer + encoder := json.NewEncoder(&buf) + encoder.SetEscapeHTML(false) + if err := encoder.Encode(payload); err != nil { + return "", err + } + return strings.TrimSuffix(buf.String(), "\n"), nil +} + +// AuditDetailsSHA256 is the lowercase hex SHA-256 of a canonical details string. +func AuditDetailsSHA256(details string) string { + sum := sha256.Sum256([]byte(details)) + return hex.EncodeToString(sum[:]) +} diff --git a/module/services/accounts/code/pkg/business/audit_store_test.go b/module/services/accounts/code/pkg/business/audit_store_test.go new file mode 100644 index 000000000..5defac587 --- /dev/null +++ b/module/services/accounts/code/pkg/business/audit_store_test.go @@ -0,0 +1,94 @@ +package business + +import ( + "bytes" + "crypto/sha256" + "encoding/hex" + "encoding/json" + "testing" + + "github.com/stretchr/testify/require" +) + +func TestParseAuditSinkMode(t *testing.T) { + for raw, want := range map[string]AuditSinkMode{ + "": AuditSinkPostgres, + "postgres": AuditSinkPostgres, + " Postgres ": AuditSinkPostgres, + "both": AuditSinkBoth, + "BigQuery": AuditSinkBigQuery, + } { + got, err := ParseAuditSinkMode(raw) + require.NoError(t, err, raw) + require.Equal(t, want, got, raw) + } + require.True(t, AuditSinkBigQuery.Swaps()) + require.False(t, AuditSinkPostgres.Swaps()) + require.False(t, AuditSinkBoth.Swaps(), "the tee keeps audit_events as the store of record") + + _, err := ParseAuditSinkMode("external") + require.ErrorContains(t, err, "not permitted") + _, err = ParseAuditSinkMode("clickhouse") + require.ErrorContains(t, err, "must be postgres, both or bigquery", "a swap value with no adapter yet is refused, not ignored") +} + +func TestCanonicalAuditDetails(t *testing.T) { + t.Run("empty payloads are an empty object", func(t *testing.T) { + for _, payload := range []map[string]any{nil, {}} { + details, err := CanonicalAuditDetails(payload) + require.NoError(t, err) + require.Equal(t, "{}", details) + } + }) + t.Run("keys sorted at every level, no whitespace, no HTML escaping", func(t *testing.T) { + details, err := CanonicalAuditDetails(map[string]any{ + "zeta": "a&c", + "alpha": map[string]any{"y": 2, "x": []any{"b", "a"}}, + "mid": true, + }) + require.NoError(t, err) + require.Equal(t, `{"alpha":{"x":["b","a"],"y":2},"mid":true,"zeta":"a&c"}`, details) + }) + t.Run("the same payload in any order is the same string", func(t *testing.T) { + first, err := CanonicalAuditDetails(map[string]any{"a": 1, "b": "two", "c": []string{"x"}}) + require.NoError(t, err) + second, err := CanonicalAuditDetails(map[string]any{"c": []string{"x"}, "b": "two", "a": 1}) + require.NoError(t, err) + require.Equal(t, first, second) + }) + t.Run("a payload read back from the queue keeps its number text", func(t *testing.T) { + decoder := json.NewDecoder(bytes.NewReader([]byte(`{"ratio": 1.50, "count": 9007199254740993}`))) + decoder.UseNumber() + var payload map[string]any + require.NoError(t, decoder.Decode(&payload)) + details, err := CanonicalAuditDetails(payload) + require.NoError(t, err) + require.Equal(t, `{"count":9007199254740993,"ratio":1.50}`, details) + }) + t.Run("an unencodable payload is an error, not a silently emptied record", func(t *testing.T) { + _, err := CanonicalAuditDetails(map[string]any{"bad": make(chan int)}) + require.Error(t, err) + }) +} + +func TestAuditDetailsSHA256IsTheHashOfTheStoredString(t *testing.T) { + // SHA-256 of "{}", so any reader can check the empty case by hand. + require.Equal(t, "44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a", AuditDetailsSHA256("{}")) + + details := `{"provider":"example"}` + sum := sha256.Sum256([]byte(details)) + require.Equal(t, hex.EncodeToString(sum[:]), AuditDetailsSHA256(details)) +} + +func TestNewAuditRecord(t *testing.T) { + entry := AuditEntry{ID: NewIDString(), EventType: EventAuthLogin, Payload: map[string]any{"method": "password"}} + record, err := NewAuditRecord(entry, RetentionSecurity) + require.NoError(t, err) + require.Equal(t, entry, record.Entry) + require.Equal(t, RetentionSecurity, record.Retention) + require.Equal(t, `{"method":"password"}`, record.Details) + require.Equal(t, AuditDetailsSHA256(record.Details), record.DetailsSHA256) + + _, err = NewAuditRecord(AuditEntry{Payload: map[string]any{"bad": func() {}}}, RetentionContent) + require.Error(t, err) +} diff --git a/module/services/accounts/code/pkg/business/audit_type_resolution.go b/module/services/accounts/code/pkg/business/audit_type_resolution.go index bc07f3096..a0d0d2306 100644 --- a/module/services/accounts/code/pkg/business/audit_type_resolution.go +++ b/module/services/accounts/code/pkg/business/audit_type_resolution.go @@ -40,6 +40,17 @@ func (r ResolvedAuditEvent) Category() string { return string(r.Definition.Category) } +// RetentionClass is the type's retention class. A type the registry does not +// know — one renamed or retired after its event was written — is held as +// security, so its full details are kept for the compliance window: a record +// whose meaning can no longer be looked up is not one to shorten. +func (r ResolvedAuditEvent) RetentionClass() AuditRetentionClass { + if !r.Registered || r.Definition.Retention == "" { + return RetentionSecurity + } + return r.Definition.Retention +} + // Redact returns a copy of payload with every field the definition marks PII // removed. Every path that sends an event outside the audit store uses it, so // downstream sinks never receive personally identifying fields. An unregistered diff --git a/module/services/accounts/code/pkg/business/service_test.go b/module/services/accounts/code/pkg/business/service_test.go index 800624ed8..8721eb999 100644 --- a/module/services/accounts/code/pkg/business/service_test.go +++ b/module/services/accounts/code/pkg/business/service_test.go @@ -184,6 +184,7 @@ func runBusinessTests(m *testing.M) int { } defer auditEmitter.Close() service.SetAuditEmitter(auditEmitter) + resolver.SetAuditRecorder(auditEmitter) entitlementChecker := business.NewDefaultEntitlementChecker(store) service.SetEntitlementChecker(entitlementChecker) diff --git a/module/services/accounts/code/pkg/business/solution_audit_events.go b/module/services/accounts/code/pkg/business/solution_audit_events.go index 1291d9eae..812309f54 100644 --- a/module/services/accounts/code/pkg/business/solution_audit_events.go +++ b/module/services/accounts/code/pkg/business/solution_audit_events.go @@ -249,6 +249,11 @@ func (d DeclaredAuditEventType) PayloadSchemaJSON() []byte { // Definition projects a declared type onto the definition shape the catalog // listing uses. It is observational: a module emission is its own operation, // written on the emitter's own transaction. +// +// Its retention class is content. A declaration does not yet carry a class of +// its own — the manifest, the module declaration RPC and the audit_event_types +// row would each have to grow one — so every declared type is held to the +// shorter details window, exactly as Durability is fixed above. func (d DeclaredAuditEventType) Definition() AuditEventDefinition { return AuditEventDefinition{ Type: d.Type, @@ -258,6 +263,7 @@ func (d DeclaredAuditEventType) Definition() AuditEventDefinition { Owner: SolutionAuditOwner(d.SolutionID), Description: d.Description, Durability: DurabilityObservational, + Retention: RetentionContent, Visibility: d.Visibility, // The whole payload the type carries, the host-stamped solution // included, so a declared type validates and redacts exactly as the diff --git a/module/services/accounts/code/pkg/business/store.go b/module/services/accounts/code/pkg/business/store.go index 4832a9009..d183b3f8d 100644 --- a/module/services/accounts/code/pkg/business/store.go +++ b/module/services/accounts/code/pkg/business/store.go @@ -530,6 +530,10 @@ type Store interface { // Audit InsertAuditEvent(ctx context.Context, entry AuditEntry) error + // EnqueueAuditEvent writes the event into the transactional audit queue on + // the ambient transaction — the record under a swap value (ADR 0009), which + // AuditRelay delivers to the store of record. Only the emitter calls it. + EnqueueAuditEvent(ctx context.Context, entry AuditEntry) error // ReserveAuditIdempotency records a (org_id, event_type, idempotency_key) // guard row so a retried emit collapses to one event. It returns true when the // row was newly inserted (write the event) and false when it already existed (a diff --git a/module/services/accounts/code/pkg/infra/postgres_audit.go b/module/services/accounts/code/pkg/infra/postgres_audit.go index ac2fc3417..514db89bf 100644 --- a/module/services/accounts/code/pkg/infra/postgres_audit.go +++ b/module/services/accounts/code/pkg/infra/postgres_audit.go @@ -50,7 +50,34 @@ func (s *PostgresStore) ReserveAuditIdempotency(ctx context.Context, orgID, even } func (s *PostgresStore) InsertAuditEvent(ctx context.Context, entry business.AuditEntry) error { - q := s.getQueryExecutor(ctx) + _, err := s.getQueryExecutor(ctx).Exec(ctx, ` + INSERT INTO audit_events ( + id, event_type, schema_version, actor_id, actor_type, + resource, resource_id, org_id, payload, ip_address, created_at, + impersonated_by, is_impersonated, client_id + ) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14)`, + auditEventColumns(entry)...) + return err +} + +// EnqueueAuditEvent writes the event into audit_event_queue on the ambient +// transaction: the record under a swap value (ADR 0009). The row carries +// exactly the columns an audit_events row would, normalized the same way, so +// the event the relay delivers is the event Postgres would have kept. +func (s *PostgresStore) EnqueueAuditEvent(ctx context.Context, entry business.AuditEntry) error { + _, err := s.getQueryExecutor(ctx).Exec(ctx, ` + INSERT INTO audit_event_queue ( + id, event_type, schema_version, actor_id, actor_type, + resource, resource_id, org_id, payload, ip_address, created_at, + impersonated_by, is_impersonated, client_id + ) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14)`, + auditEventColumns(entry)...) + return err +} + +// auditEventColumns normalizes an entry into the fourteen column values +// audit_events and audit_event_queue share, in their INSERT order. +func auditEventColumns(entry business.AuditEntry) []any { if entry.ID == "" { entry.ID = business.NewIDString() } @@ -66,12 +93,7 @@ func (s *PostgresStore) InsertAuditEvent(ctx context.Context, entry business.Aud payload = []byte("{}") } - _, err = q.Exec(ctx, ` - INSERT INTO audit_events ( - id, event_type, schema_version, actor_id, actor_type, - resource, resource_id, org_id, payload, ip_address, created_at, - impersonated_by, is_impersonated, client_id - ) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14)`, + return []any{ entry.ID, string(entry.EventType), entry.SchemaVersion, nilIfNotUUID(entry.ActorID), entry.ActorType, // resource_id is text: a resource is named by whatever id its owner mints @@ -80,8 +102,8 @@ func (s *PostgresStore) InsertAuditEvent(ctx context.Context, entry business.Aud entry.Resource, nilIfEmpty(entry.ResourceID), nilIfNotUUID(entry.OrgID), payload, nilIfEmpty(entry.IPAddress), entry.CreatedAt, nilIfNotUUID(entry.ImpersonatedBy), entry.IsImpersonated, - nilIfEmpty(entry.ClientID)) - return err + nilIfEmpty(entry.ClientID), + } } // encodeAuditCursor / decodeAuditCursor carry the keyset position for audit diff --git a/module/services/accounts/code/pkg/infra/postgres_audit_queue.go b/module/services/accounts/code/pkg/infra/postgres_audit_queue.go new file mode 100644 index 000000000..56df2ef26 --- /dev/null +++ b/module/services/accounts/code/pkg/infra/postgres_audit_queue.go @@ -0,0 +1,170 @@ +package infra + +import ( + "bytes" + "context" + "encoding/json" + "errors" + "fmt" + + "accounts/pkg/business" + + "github.com/codefly-dev/core/wool" + "github.com/jackc/pgx/v5/pgxpool" +) + +// auditRelayLeaseKey names the advisory lock one relay holds for the length of +// a delivery, so replicas take turns rather than deliver the same rows twice +// or interleave one organization's events. +const auditRelayLeaseKey = "audit-relay:audit_event_queue" + +// NewAuditRelayPool opens the audit relay's own pool, on the control-plane +// login assuming app_job_worker — the one role that may read and delete queued +// audit events. It is separate from the job workers' pool because a delivery +// holds its transaction open across the archive and warehouse writes, and a +// slow warehouse must not take a connection the job workers are waiting for. +func NewAuditRelayPool(ctx context.Context) (*pgxpool.Pool, error) { + w := wool.Get(ctx).In("NewAuditRelayPool") + connection, err := storeConnection(ctx, controlPlaneConnectionKey) + if err != nil { + return nil, w.Wrapf(err, "failed to get connection string") + } + return NewAuditRelayPoolFromURL(ctx, connection) +} + +// NewAuditRelayPoolFromURL is the explicit-URL variant for integration tests. +func NewAuditRelayPoolFromURL(ctx context.Context, connectionURL string) (*pgxpool.Pool, error) { + return newWorkerPoolFromURL(ctx, connectionURL, workerPoolConfig{ + role: jobWorkerDatabaseRole, + applicationName: "accounts-audit-relay", + logScope: "NewAuditRelayPool", + }) +} + +// PostgresAuditQueue is the relay's side of audit_event_queue +// (business.AuditQueue). +type PostgresAuditQueue struct { + pool *pgxpool.Pool +} + +// NewPostgresAuditQueue drains the queue over pool, whose connections must +// assume app_job_worker (NewAuditRelayPool). +func NewPostgresAuditQueue(pool *pgxpool.Pool) (*PostgresAuditQueue, error) { + if pool == nil { + return nil, errors.New("audit queue: pool is required") + } + return &PostgresAuditQueue{pool: pool}, nil +} + +// auditQueueSelectSQL reads the oldest queued events whose writing +// transactions are below the snapshot's xmin — every transaction that old has +// finished, so nothing still in flight can later commit a row ahead of these. +const auditQueueSelectSQL = ` + SELECT seq, id::text, event_type, schema_version, + COALESCE(actor_id::text, ''), actor_type, resource, + COALESCE(resource_id, ''), COALESCE(org_id::text, ''), payload, + COALESCE(ip_address, ''), created_at, + COALESCE(impersonated_by::text, ''), is_impersonated, + COALESCE(client_id, ''), enqueued_at + FROM public.audit_event_queue + WHERE xact_id < pg_snapshot_xmin(pg_current_snapshot()) + ORDER BY seq + LIMIT $1` + +// Drain implements business.AuditQueue. The whole delivery runs in one +// transaction holding the relay lease: read the batch, hand it to deliver, +// delete exactly its rows, commit. A delivery that fails, a crash, or a lost +// connection rolls the transaction back and leaves every row queued. +func (q *PostgresAuditQueue) Drain( + ctx context.Context, + limit int, + deliver func(ctx context.Context, events []business.QueuedAuditEvent) (bool, error), +) (int, error) { + if limit < 1 { + return 0, fmt.Errorf("audit queue: limit must be positive, got %d", limit) + } + tx, err := q.pool.Begin(ctx) + if err != nil { + return 0, err + } + defer tx.Rollback(context.WithoutCancel(ctx)) //nolint:errcheck // rollback after commit is a no-op + + var leased bool + if err := tx.QueryRow(ctx, `SELECT pg_try_advisory_xact_lock(hashtextextended($1, 0))`, auditRelayLeaseKey).Scan(&leased); err != nil { + return 0, fmt.Errorf("audit queue: take relay lease: %w", err) + } + if !leased { + return 0, nil + } + + rows, err := tx.Query(ctx, auditQueueSelectSQL, limit) + if err != nil { + return 0, fmt.Errorf("audit queue: read: %w", err) + } + var events []business.QueuedAuditEvent + for rows.Next() { + var ( + event business.QueuedAuditEvent + eventType string + payload []byte + ) + entry := &event.Entry + if err := rows.Scan( + &event.Seq, &entry.ID, &eventType, &entry.SchemaVersion, + &entry.ActorID, &entry.ActorType, &entry.Resource, + &entry.ResourceID, &entry.OrgID, &payload, + &entry.IPAddress, &entry.CreatedAt, + &entry.ImpersonatedBy, &entry.IsImpersonated, + &entry.ClientID, &event.EnqueuedAt, + ); err != nil { + rows.Close() + return 0, fmt.Errorf("audit queue: scan: %w", err) + } + entry.EventType = business.EventType(eventType) + entry.CreatedAt = entry.CreatedAt.UTC() + event.EnqueuedAt = event.EnqueuedAt.UTC() + if entry.Payload, err = decodeQueuedPayload(payload); err != nil { + rows.Close() + return 0, fmt.Errorf("audit queue: payload of event %s: %w", entry.ID, err) + } + events = append(events, event) + } + rows.Close() + if err := rows.Err(); err != nil { + return 0, fmt.Errorf("audit queue: read: %w", err) + } + + delivered, err := deliver(ctx, events) + if err != nil || !delivered { + return 0, err + } + + seqs := make([]int64, len(events)) + for i, event := range events { + seqs[i] = event.Seq + } + tag, err := tx.Exec(ctx, `DELETE FROM public.audit_event_queue WHERE seq = ANY($1)`, seqs) + if err != nil { + return 0, fmt.Errorf("audit queue: delete delivered rows: %w", err) + } + if tag.RowsAffected() != int64(len(seqs)) { + // Only the lease holder deletes, so this is a broken invariant, not a race. + return 0, fmt.Errorf("audit queue: deleted %d of %d delivered rows", tag.RowsAffected(), len(seqs)) + } + if err := tx.Commit(ctx); err != nil { + return 0, fmt.Errorf("audit queue: commit delivered batch: %w", err) + } + return len(events), nil +} + +// decodeQueuedPayload decodes a queued payload keeping each number's text +// (json.Number), so the canonical details carry exactly what Postgres stored. +func decodeQueuedPayload(raw []byte) (map[string]any, error) { + decoder := json.NewDecoder(bytes.NewReader(raw)) + decoder.UseNumber() + var payload map[string]any + if err := decoder.Decode(&payload); err != nil { + return nil, err + } + return payload, nil +} diff --git a/module/services/accounts/code/pkg/infra/postgres_audit_queue_test.go b/module/services/accounts/code/pkg/infra/postgres_audit_queue_test.go new file mode 100644 index 000000000..8fc40532b --- /dev/null +++ b/module/services/accounts/code/pkg/infra/postgres_audit_queue_test.go @@ -0,0 +1,330 @@ +//go:build !pure + +package infra_test + +import ( + "context" + "errors" + "sync" + "testing" + "time" + + "github.com/jackc/pgx/v5/pgxpool" + "github.com/stretchr/testify/require" + + "accounts/pkg/business" + "accounts/pkg/infra" + "accounts/pkg/infra/storetx" +) + +// The transactional audit queue under a swap value (ADR 0009), against the real +// schema: who may write it, who may read and delete it, and the guarantees the +// relay's side gives — committed rows only, in order, deleted only on success. + +// catalogAudit is the recorder the role catalog import requires, in the +// default (postgres) mode. +func catalogAudit(t *testing.T, opts ...business.DurableAuditEmitterOption) business.AuditRecorder { + t.Helper() + recorder, err := business.NewDurableAuditEmitter(testStore, testStore, opts...) + require.NoError(t, err) + return recorder +} + +func auditRelayPool(t *testing.T) *pgxpool.Pool { + t.Helper() + pool, err := infra.NewAuditRelayPool(testCtx) + require.NoError(t, err) + t.Cleanup(pool.Close) + return pool +} + +// emptyAuditQueue removes every queued event, as the relay would, so each test +// starts from an empty queue. +func emptyAuditQueue(t *testing.T, pool *pgxpool.Pool) { + t.Helper() + _, err := pool.Exec(testCtx, `DELETE FROM audit_event_queue`) + require.NoError(t, err) +} + +func queuedEntry(orgID string, eventType business.EventType) business.AuditEntry { + return business.AuditEntry{ + ID: business.NewIDString(), + OrgID: orgID, + ActorID: business.NewIDString(), + ActorType: business.ActorTypeUser, + EventType: eventType, + SchemaVersion: 1, + Resource: "session", + ResourceID: "session-1", + Payload: map[string]any{"ratio": 1.5, "label": ""}, + CreatedAt: time.Now().UTC().Truncate(time.Microsecond), + } +} + +func drainAll(t *testing.T, queue *infra.PostgresAuditQueue) []business.QueuedAuditEvent { + t.Helper() + var got []business.QueuedAuditEvent + _, err := queue.Drain(testCtx, 1000, func(_ context.Context, events []business.QueuedAuditEvent) (bool, error) { + got = events + return true, nil + }) + require.NoError(t, err) + return got +} + +func TestAuditQueueTenantWritesOnlyItsOwnOrganizationAndReadsNothing(t *testing.T) { + pool := auditRelayPool(t) + emptyAuditQueue(t, pool) + userID := seedUser(t) + orgID := seedOrg(t, userID) + otherOrgID := seedOrg(t, seedUser(t)) + + require.NoError(t, testStore.WithOrgTx(testCtx, orgID, func(ctx context.Context) error { + return testStore.EnqueueAuditEvent(ctx, queuedEntry(orgID, business.EventSessionRevoked)) + })) + + err := testStore.WithOrgTx(testCtx, orgID, func(ctx context.Context) error { + return testStore.EnqueueAuditEvent(ctx, queuedEntry(otherOrgID, business.EventSessionRevoked)) + }) + require.Error(t, err, "a tenant transaction cannot queue another organization's event") + + err = testStore.WithOrgTx(testCtx, orgID, func(ctx context.Context) error { + var n int + return storetx.Tx(ctx).QueryRow(ctx, `SELECT count(*) FROM audit_event_queue`).Scan(&n) + }) + require.Error(t, err, "a writer cannot read the queue back") + + err = testStore.WithControlPlane(testCtx, func(ctx context.Context) error { + _, err := storetx.Tx(ctx).Exec(ctx, `DELETE FROM audit_event_queue`) + return err + }) + require.Error(t, err, "the control plane writes the queue but cannot remove a queued event") + + queue, err := infra.NewPostgresAuditQueue(pool) + require.NoError(t, err) + events := drainAll(t, queue) + require.Len(t, events, 1) + require.Equal(t, orgID, events[0].Entry.OrgID) +} + +func TestAuditQueueCarriesTheEventAsAuditEventsWouldKeepIt(t *testing.T) { + pool := auditRelayPool(t) + emptyAuditQueue(t, pool) + orgID := seedOrg(t, seedUser(t)) + + entry := queuedEntry(orgID, business.EventSessionRevoked) + entry.IPAddress = "203.0.113.7" + entry.ClientID = "example-cli" + entry.ImpersonatedBy = business.NewIDString() + entry.IsImpersonated = true + require.NoError(t, testStore.WithOrgTx(testCtx, orgID, func(ctx context.Context) error { + return testStore.EnqueueAuditEvent(ctx, entry) + })) + platform := queuedEntry("", business.EventRoleCreated) + platform.ActorID = "" + platform.ActorType = business.ActorTypeSystem + require.NoError(t, testStore.WithControlPlane(testCtx, func(ctx context.Context) error { + return testStore.EnqueueAuditEvent(ctx, platform) + }), "a platform event is queued on the control plane") + + queue, err := infra.NewPostgresAuditQueue(pool) + require.NoError(t, err) + events := drainAll(t, queue) + require.Len(t, events, 2) + require.Less(t, events[0].Seq, events[1].Seq) + + got := events[0].Entry + require.Equal(t, entry.ID, got.ID) + require.Equal(t, entry.EventType, got.EventType) + require.Equal(t, entry.ActorID, got.ActorID) + require.Equal(t, entry.Resource, got.Resource) + require.Equal(t, entry.ResourceID, got.ResourceID) + require.Equal(t, entry.IPAddress, got.IPAddress) + require.Equal(t, entry.ClientID, got.ClientID) + require.Equal(t, entry.ImpersonatedBy, got.ImpersonatedBy) + require.True(t, got.IsImpersonated) + require.True(t, entry.CreatedAt.Equal(got.CreatedAt)) + details, err := business.CanonicalAuditDetails(got.Payload) + require.NoError(t, err) + require.Equal(t, `{"label":"","ratio":1.5}`, details, "the payload reads back as the canonical details of what was written") + + require.Empty(t, events[1].Entry.OrgID) + require.Empty(t, events[1].Entry.ActorID) + require.Equal(t, business.ActorTypeSystem, events[1].Entry.ActorType) +} + +func TestAuditQueueDeletesOnlyWhatWasDelivered(t *testing.T) { + pool := auditRelayPool(t) + emptyAuditQueue(t, pool) + orgID := seedOrg(t, seedUser(t)) + for range 5 { + require.NoError(t, testStore.WithOrgTx(testCtx, orgID, func(ctx context.Context) error { + return testStore.EnqueueAuditEvent(ctx, queuedEntry(orgID, business.EventSessionRevoked)) + })) + } + queue, err := infra.NewPostgresAuditQueue(pool) + require.NoError(t, err) + + _, err = queue.Drain(testCtx, 3, func(context.Context, []business.QueuedAuditEvent) (bool, error) { + return false, errors.New("warehouse unavailable") + }) + require.ErrorContains(t, err, "warehouse unavailable") + n, err := queue.Drain(testCtx, 3, func(context.Context, []business.QueuedAuditEvent) (bool, error) { + return false, nil + }) + require.NoError(t, err) + require.Zero(t, n, "a batch the relay is not ready to deliver is left queued") + + var first []business.QueuedAuditEvent + n, err = queue.Drain(testCtx, 3, func(_ context.Context, events []business.QueuedAuditEvent) (bool, error) { + first = events + return true, nil + }) + require.NoError(t, err) + require.Equal(t, 3, n) + + rest := drainAll(t, queue) + require.Len(t, rest, 2, "exactly the delivered rows were deleted") + require.Greater(t, rest[0].Seq, first[2].Seq, "and what remains is what came after them") +} + +// A row is not handed to the relay while a transaction that began before it +// is still running: when that transaction commits, its row is ordered first. +func TestAuditQueueNeverOvertakesAnEarlierTransactionStillInFlight(t *testing.T) { + pool := auditRelayPool(t) + emptyAuditQueue(t, pool) + orgID := seedOrg(t, seedUser(t)) + queue, err := infra.NewPostgresAuditQueue(pool) + require.NoError(t, err) + + early := queuedEntry(orgID, business.EventSessionRevoked) + late := queuedEntry(orgID, business.EventSessionRevoked) + earlyQueued, releaseEarly := make(chan struct{}), make(chan struct{}) + var wg sync.WaitGroup + var release sync.Once + finishEarly := func() { + release.Do(func() { close(releaseEarly) }) + wg.Wait() + } + // A failed assertion must still end the open transaction, or the package's + // pool never drains and the run hangs instead of failing. + t.Cleanup(finishEarly) + wg.Add(1) + go func() { + defer wg.Done() + _ = testStore.WithOrgTx(testCtx, orgID, func(ctx context.Context) error { + if err := testStore.EnqueueAuditEvent(ctx, early); err != nil { + return err + } + close(earlyQueued) + <-releaseEarly + return nil + }) + }() + <-earlyQueued + require.NoError(t, testStore.WithOrgTx(testCtx, orgID, func(ctx context.Context) error { + return testStore.EnqueueAuditEvent(ctx, late) + })) + + require.Empty(t, drainAll(t, queue), "the committed later row waits for the earlier transaction to finish") + + finishEarly() + events := drainAll(t, queue) + require.Len(t, events, 2) + require.Equal(t, early.ID, events[0].Entry.ID, "queue order, not commit order") + require.Equal(t, late.ID, events[1].Entry.ID) +} + +func TestAuditQueueLeaseAdmitsOneRelayAtATime(t *testing.T) { + pool := auditRelayPool(t) + emptyAuditQueue(t, pool) + orgID := seedOrg(t, seedUser(t)) + require.NoError(t, testStore.WithOrgTx(testCtx, orgID, func(ctx context.Context) error { + return testStore.EnqueueAuditEvent(ctx, queuedEntry(orgID, business.EventSessionRevoked)) + })) + first, err := infra.NewPostgresAuditQueue(pool) + require.NoError(t, err) + second, err := infra.NewPostgresAuditQueue(auditRelayPool(t)) + require.NoError(t, err) + + _, err = first.Drain(testCtx, 10, func(ctx context.Context, events []business.QueuedAuditEvent) (bool, error) { + require.Len(t, events, 1) + n, err := second.Drain(ctx, 10, func(context.Context, []business.QueuedAuditEvent) (bool, error) { + t.Fatal("a second relay must not be handed rows while the first holds the lease") + return false, nil + }) + require.NoError(t, err) + require.Zero(t, n) + return true, nil + }) + require.NoError(t, err) +} + +// Under a swap value the emitter's record is a queue row and audit_events +// receives nothing; the webhook outbox is written beside it as before. +func TestQueuedRecordsLeaveAuditEventsUntouched(t *testing.T) { + pool := auditRelayPool(t) + emptyAuditQueue(t, pool) + orgID := seedOrg(t, seedUser(t)) + emitter, err := business.NewDurableAuditEmitter(testStore, testStore, business.WithQueuedRecords()) + require.NoError(t, err) + + entry := queuedEntry(orgID, business.EventSessionRevoked) + require.NoError(t, testStore.WithOrgTx(testCtx, orgID, func(ctx context.Context) error { + return emitter.EmitTx(ctx, entry) + })) + + var inAuditEvents int + require.NoError(t, testStore.WithControlPlane(testCtx, func(ctx context.Context) error { + return storetx.Tx(ctx).QueryRow(ctx, `SELECT count(*) FROM audit_events WHERE id = $1`, entry.ID).Scan(&inAuditEvents) + })) + require.Zero(t, inAuditEvents) + + queue, err := infra.NewPostgresAuditQueue(pool) + require.NoError(t, err) + events := drainAll(t, queue) + require.Len(t, events, 1) + require.Equal(t, entry.ID, events[0].Entry.ID) +} + +// The role catalog import writes platform events outside the Service; under a +// swap value they reach the queue like every other record. +func TestImportRoleCatalogUnderTheSwapQueuesItsEvents(t *testing.T) { + pool := auditRelayPool(t) + emptyAuditQueue(t, pool) + resetCatalogRoles(t) + catalog := parseCatalog(t, `{"version":1,"roles":[ + {"name":"catalog-test:queued","permissions":[{"resource":"x","action":"read"}]}]}`) + + _, err := testStore.ImportRoleCatalog(testCtx, catalog, infra.ImportOptions{ + Audit: catalogAudit(t, business.WithQueuedRecords()), + }) + require.NoError(t, err) + role, found := readBuiltinRole(t, "catalog-test:queued") + require.True(t, found) + + var inAuditEvents int + require.NoError(t, testStore.WithControlPlane(testCtx, func(ctx context.Context) error { + return storetx.Tx(ctx).QueryRow(ctx, + `SELECT count(*) FROM audit_events WHERE resource = 'role' AND resource_id = $1`, role.id).Scan(&inAuditEvents) + })) + require.Zero(t, inAuditEvents) + + queue, err := infra.NewPostgresAuditQueue(pool) + require.NoError(t, err) + events := drainAll(t, queue) + require.Len(t, events, 1) + require.Equal(t, business.EventRoleCreated, events[0].Entry.EventType) + require.Equal(t, role.id, events[0].Entry.ResourceID) + require.Empty(t, events[0].Entry.OrgID, "built-in roles belong to no organization") +} + +func TestImportRoleCatalogRefusesToApplyUnrecorded(t *testing.T) { + resetCatalogRoles(t) + catalog := parseCatalog(t, `{"version":1,"roles":[ + {"name":"catalog-test:unrecorded","permissions":[{"resource":"x","action":"read"}]}]}`) + _, err := testStore.ImportRoleCatalog(testCtx, catalog, infra.ImportOptions{}) + require.ErrorContains(t, err, "audit recorder is required") + _, found := readBuiltinRole(t, "catalog-test:unrecorded") + require.False(t, found) +} diff --git a/module/services/accounts/code/pkg/infra/postgres_role_catalog.go b/module/services/accounts/code/pkg/infra/postgres_role_catalog.go index 3bfec1ad7..7e778a594 100644 --- a/module/services/accounts/code/pkg/infra/postgres_role_catalog.go +++ b/module/services/accounts/code/pkg/infra/postgres_role_catalog.go @@ -20,6 +20,10 @@ type ImportOptions struct { // Source is an optional provenance label (e.g. the catalog file path) // recorded on every audit event this import emits. Source string + // Audit records the import's events on its transaction: the audit_events + // row, or under a swap value (ADR 0009) the queue row. Required unless + // DryRun, so an import cannot change roles unrecorded. + Audit business.AuditRecorder } // ImportResult reports the outcome of an import. @@ -44,6 +48,9 @@ type ImportResult struct { func (s *PostgresStore) ImportRoleCatalog(ctx context.Context, cat *rolecatalog.Catalog, opts ImportOptions) (*ImportResult, error) { w := wool.Get(ctx).In("ImportRoleCatalog") result := &ImportResult{} + if opts.Audit == nil && !opts.DryRun { + return nil, w.NewError("an audit recorder is required to apply a role catalog") + } err := s.WithControlPlane(ctx, func(ctx context.Context) error { state, err := s.snapshotBuiltinRoles(ctx) @@ -61,7 +68,7 @@ func (s *PostgresStore) ImportRoleCatalog(ctx context.Context, cat *rolecatalog. result.RefusalReason = reason return nil } - if err := s.applyRoleCatalogPlan(ctx, plan, catalogProvenance(cat, opts.Source)); err != nil { + if err := s.applyRoleCatalogPlan(ctx, plan, opts.Audit, catalogProvenance(cat, opts.Source)); err != nil { return err } result.Applied = true @@ -215,7 +222,7 @@ func catalogProvenance(cat *rolecatalog.Catalog, source string) map[string]strin // applyRoleCatalogPlan writes the plan's changes and emits one system audit // event per changed role. Runs inside the control-plane transaction opened by // ImportRoleCatalog. -func (s *PostgresStore) applyRoleCatalogPlan(ctx context.Context, plan *rolecatalog.Plan, provenance map[string]string) error { +func (s *PostgresStore) applyRoleCatalogPlan(ctx context.Context, plan *rolecatalog.Plan, audit business.AuditRecorder, provenance map[string]string) error { w := wool.Get(ctx).In("applyRoleCatalogPlan") executor := s.getQueryExecutor(ctx) @@ -233,7 +240,7 @@ func (s *PostgresStore) applyRoleCatalogPlan(ctx context.Context, plan *rolecata return w.Wrapf(err, "failed to seed permission for role %q", create.Role.Name) } } - if err := s.emitCatalogAudit(ctx, business.EventRoleCreated, roleID, map[string]string{ + if err := emitCatalogAudit(ctx, audit, business.EventRoleCreated, roleID, map[string]string{ "name": create.Role.Name, "permissions_added": strconv.Itoa(len(create.Role.Permissions)), }, provenance); err != nil { @@ -264,7 +271,7 @@ func (s *PostgresStore) applyRoleCatalogPlan(ctx context.Context, plan *rolecata return w.Wrapf(err, "failed to add permission to role %q", update.Name) } } - if err := s.emitCatalogAudit(ctx, business.EventRoleUpdated, update.RoleID, map[string]string{ + if err := emitCatalogAudit(ctx, audit, business.EventRoleUpdated, update.RoleID, map[string]string{ "name": update.Name, "permissions_added": strconv.Itoa(len(update.AddPermissions)), "permissions_removed": strconv.Itoa(len(update.RemovePermissions)), @@ -278,7 +285,7 @@ func (s *PostgresStore) applyRoleCatalogPlan(ctx context.Context, plan *rolecata if _, err := executor.Exec(ctx, `DELETE FROM roles WHERE id = $1`, remove.RoleID); err != nil { return w.Wrapf(err, "failed to remove role %q", remove.Name) } - if err := s.emitCatalogAudit(ctx, business.EventRoleDeleted, remove.RoleID, map[string]string{ + if err := emitCatalogAudit(ctx, audit, business.EventRoleDeleted, remove.RoleID, map[string]string{ "name": remove.Name, "assignments_removed": strconv.Itoa(remove.AssignmentCount), }, provenance); err != nil { @@ -296,11 +303,11 @@ func insertRolePermission(ctx context.Context, executor QueryExecutor, roleID st return err } -// emitCatalogAudit writes one system-actor audit event with a NULL org — the -// catalog reconciles global built-in roles, which belong to no tenant. The -// enclosing control-plane transaction lets the polymorphic audit_events policy -// accept the NULL org_id. -func (s *PostgresStore) emitCatalogAudit(ctx context.Context, event business.EventType, roleID string, metadata, provenance map[string]string) error { +// emitCatalogAudit records one system-actor audit event with a NULL org — the +// catalog reconciles global built-in roles, which belong to no tenant — through +// the recorder, on the enclosing control-plane transaction, whose role the +// polymorphic audit_events policy (and the queue's) admits a NULL org_id for. +func emitCatalogAudit(ctx context.Context, audit business.AuditRecorder, event business.EventType, roleID string, metadata, provenance map[string]string) error { w := wool.Get(ctx).In("emitCatalogAudit") for k, v := range provenance { metadata[k] = v @@ -309,7 +316,7 @@ func (s *PostgresStore) emitCatalogAudit(ctx context.Context, event business.Eve for k, v := range metadata { payload[k] = v } - if err := s.InsertAuditEvent(ctx, business.AuditEntry{ + if err := audit.RecordTx(ctx, business.AuditEntry{ ActorType: "system", EventType: event, Resource: "role", diff --git a/module/services/accounts/code/pkg/infra/postgres_role_catalog_test.go b/module/services/accounts/code/pkg/infra/postgres_role_catalog_test.go index 5358c4435..795e2432c 100644 --- a/module/services/accounts/code/pkg/infra/postgres_role_catalog_test.go +++ b/module/services/accounts/code/pkg/infra/postgres_role_catalog_test.go @@ -130,7 +130,7 @@ func TestImportRoleCatalogAppliesExampleThenNoOp(t *testing.T) { catalog, err := rolecatalog.Parse(document) require.NoError(t, err) - result, err := testStore.ImportRoleCatalog(testCtx, catalog, infra.ImportOptions{}) + result, err := testStore.ImportRoleCatalog(testCtx, catalog, infra.ImportOptions{Audit: catalogAudit(t)}) require.NoError(t, err) require.True(t, result.Applied) require.Len(t, result.Plan.Creates, 1) @@ -152,7 +152,7 @@ func TestImportRoleCatalogAppliesExampleThenNoOp(t *testing.T) { } // Second run is a true no-op. - again, err := testStore.ImportRoleCatalog(testCtx, catalog, infra.ImportOptions{}) + again, err := testStore.ImportRoleCatalog(testCtx, catalog, infra.ImportOptions{Audit: catalogAudit(t)}) require.NoError(t, err) require.True(t, again.Plan.Empty()) } @@ -162,7 +162,7 @@ func TestImportRoleCatalogOnePermissionChangeIsOneRowOneAudit(t *testing.T) { base := parseCatalog(t, `{"version":1,"roles":[ {"name":"catalog-test:reader","description":"d","scope":"m", "permissions":[{"resource":"reports","action":"read"}]}]}`) - _, err := testStore.ImportRoleCatalog(testCtx, base, infra.ImportOptions{}) + _, err := testStore.ImportRoleCatalog(testCtx, base, infra.ImportOptions{Audit: catalogAudit(t)}) require.NoError(t, err) role, found := readBuiltinRole(t, "catalog-test:reader") @@ -175,7 +175,7 @@ func TestImportRoleCatalogOnePermissionChangeIsOneRowOneAudit(t *testing.T) { changed := parseCatalog(t, `{"version":1,"roles":[ {"name":"catalog-test:reader","description":"d","scope":"m", "permissions":[{"resource":"reports","action":"read"},{"resource":"reports","action":"write"}]}]}`) - result, err := testStore.ImportRoleCatalog(testCtx, changed, infra.ImportOptions{}) + result, err := testStore.ImportRoleCatalog(testCtx, changed, infra.ImportOptions{Audit: catalogAudit(t)}) require.NoError(t, err) require.True(t, result.Applied) require.Len(t, result.Plan.Updates, 1) @@ -190,7 +190,7 @@ func TestImportRoleCatalogOnePermissionChangeIsOneRowOneAudit(t *testing.T) { require.Equal(t, updatesBefore+1, countCatalogAudits(t, "saas.role.updated"), "exactly one update audit event") // Removing a single permission is likewise one row change and one event. - reverted, err := testStore.ImportRoleCatalog(testCtx, base, infra.ImportOptions{}) + reverted, err := testStore.ImportRoleCatalog(testCtx, base, infra.ImportOptions{Audit: catalogAudit(t)}) require.NoError(t, err) require.True(t, reverted.Applied) require.Len(t, reverted.Plan.Updates, 1) @@ -231,7 +231,7 @@ func TestImportRoleCatalogLeavesCustomRolesAndAssignmentsUntouched(t *testing.T) catalog := parseCatalog(t, `{"version":1,"roles":[ {"name":"catalog-test:writer","permissions":[{"resource":"docs","action":"write"}]}]}`) - _, err := testStore.ImportRoleCatalog(testCtx, catalog, infra.ImportOptions{}) + _, err := testStore.ImportRoleCatalog(testCtx, catalog, infra.ImportOptions{Audit: catalogAudit(t)}) require.NoError(t, err) // The custom role and its assignment are intact. @@ -259,7 +259,7 @@ func TestImportRoleCatalogRefusesOrphaningRemovalUnlessForced(t *testing.T) { seedCatalog := parseCatalog(t, `{"version":1,"roles":[ {"name":"catalog-test:keep","permissions":[{"resource":"y","action":"read"}]}, {"name":"catalog-test:temp","permissions":[{"resource":"x","action":"read"}]}]}`) - _, err := testStore.ImportRoleCatalog(testCtx, seedCatalog, infra.ImportOptions{}) + _, err := testStore.ImportRoleCatalog(testCtx, seedCatalog, infra.ImportOptions{Audit: catalogAudit(t)}) require.NoError(t, err) role, found := readBuiltinRole(t, "catalog-test:temp") require.True(t, found) @@ -276,7 +276,7 @@ func TestImportRoleCatalogRefusesOrphaningRemovalUnlessForced(t *testing.T) { // the empty-catalog guard) refuses it without force. dropTemp := parseCatalog(t, `{"version":1,"roles":[ {"name":"catalog-test:keep","permissions":[{"resource":"y","action":"read"}]}]}`) - refused, err := testStore.ImportRoleCatalog(testCtx, dropTemp, infra.ImportOptions{}) + refused, err := testStore.ImportRoleCatalog(testCtx, dropTemp, infra.ImportOptions{Audit: catalogAudit(t)}) require.NoError(t, err) require.True(t, refused.Refused) require.Contains(t, refused.RefusalReason, "orphan") @@ -285,7 +285,7 @@ func TestImportRoleCatalogRefusesOrphaningRemovalUnlessForced(t *testing.T) { require.True(t, stillThere, "refused import must not remove anything") // Force applies the removal; the assignment cascades away. - forced, err := testStore.ImportRoleCatalog(testCtx, dropTemp, infra.ImportOptions{Force: true}) + forced, err := testStore.ImportRoleCatalog(testCtx, dropTemp, infra.ImportOptions{Audit: catalogAudit(t), Force: true}) require.NoError(t, err) require.True(t, forced.Applied) _, gone := readBuiltinRole(t, "catalog-test:temp") @@ -309,11 +309,11 @@ func TestImportRoleCatalogRefusesEmptyCatalogWipeUnlessForced(t *testing.T) { // not protect it, so an empty catalog would previously delete it silently. seedCatalog := parseCatalog(t, `{"version":1,"roles":[ {"name":"catalog-test:unassigned","permissions":[{"resource":"x","action":"read"}]}]}`) - _, err := testStore.ImportRoleCatalog(testCtx, seedCatalog, infra.ImportOptions{}) + _, err := testStore.ImportRoleCatalog(testCtx, seedCatalog, infra.ImportOptions{Audit: catalogAudit(t)}) require.NoError(t, err) empty := parseCatalog(t, `{"version":1,"roles":[]}`) - refused, err := testStore.ImportRoleCatalog(testCtx, empty, infra.ImportOptions{}) + refused, err := testStore.ImportRoleCatalog(testCtx, empty, infra.ImportOptions{Audit: catalogAudit(t)}) require.NoError(t, err) require.True(t, refused.Refused, "empty catalog must not silently wipe catalog-managed roles") require.Contains(t, refused.RefusalReason, "no roles") @@ -322,7 +322,7 @@ func TestImportRoleCatalogRefusesEmptyCatalogWipeUnlessForced(t *testing.T) { require.True(t, stillThere) // Force makes the wipe a deliberate act. - forced, err := testStore.ImportRoleCatalog(testCtx, empty, infra.ImportOptions{Force: true}) + forced, err := testStore.ImportRoleCatalog(testCtx, empty, infra.ImportOptions{Audit: catalogAudit(t), Force: true}) require.NoError(t, err) require.True(t, forced.Applied) _, gone := readBuiltinRole(t, "catalog-test:unassigned") @@ -334,7 +334,7 @@ func TestImportRoleCatalogStampsProvenanceOnAudit(t *testing.T) { catalog := parseCatalog(t, `{"version":1,"roles":[ {"name":"catalog-test:provenance","permissions":[{"resource":"x","action":"read"}]}]}`) - _, err := testStore.ImportRoleCatalog(testCtx, catalog, infra.ImportOptions{Source: "roles.json"}) + _, err := testStore.ImportRoleCatalog(testCtx, catalog, infra.ImportOptions{Audit: catalogAudit(t), Source: "roles.json"}) require.NoError(t, err) role, found := readBuiltinRole(t, "catalog-test:provenance") @@ -362,7 +362,7 @@ func TestImportRoleCatalogDryRunWritesNothing(t *testing.T) { catalog := parseCatalog(t, `{"version":1,"roles":[ {"name":"catalog-test:dryrun","permissions":[{"resource":"x","action":"read"}]}]}`) - result, err := testStore.ImportRoleCatalog(testCtx, catalog, infra.ImportOptions{DryRun: true}) + result, err := testStore.ImportRoleCatalog(testCtx, catalog, infra.ImportOptions{Audit: catalogAudit(t), DryRun: true}) require.NoError(t, err) require.False(t, result.Applied) require.Len(t, result.Plan.Creates, 1) diff --git a/module/services/accounts/code/pkg/infra/postgres_role_hardening_test.go b/module/services/accounts/code/pkg/infra/postgres_role_hardening_test.go index d8eabf344..bc68d4ad9 100644 --- a/module/services/accounts/code/pkg/infra/postgres_role_hardening_test.go +++ b/module/services/accounts/code/pkg/infra/postgres_role_hardening_test.go @@ -62,6 +62,7 @@ var appTenantRelationPrivileges = map[string]relationPrivileges{ "approval_decisions": {selectRows: true, insertRows: true}, // append-only, like actor_chain_journal "approval_requests": {selectRows: true, insertRows: true, updateRows: true}, "audit_event_idempotency": {selectRows: true, insertRows: true}, // append-only guard, like audit_events + "audit_event_queue": {insertRows: true}, // a writer never reads, changes or removes a queued event "audit_events": {selectRows: true, insertRows: true}, "connector_credentials": {selectRows: true, insertRows: true, updateRows: true, deleteRows: true}, "dashboards": {selectRows: true, insertRows: true, updateRows: true, deleteRows: true}, @@ -360,6 +361,12 @@ func TestControlPlaneRelationGrantsAreExact(t *testing.T) { if relation == "audit_event_idempotency" { want = relationPrivileges{selectRows: true, insertRows: true} } + // audit_event_queue is written on the control plane for platform + // events and control-plane mutations; only the relay, as the job + // worker, reads and deletes queued events. + if relation == "audit_event_queue" { + want = relationPrivileges{insertRows: true} + } // actor_chain_journal / actor_chain_revocations are append-only // like audit_events: the control plane reads and inserts but never // updates or deletes (an immutable trigger rejects those anyway). diff --git a/module/services/accounts/code/pkg/relationcatalog/catalog.go b/module/services/accounts/code/pkg/relationcatalog/catalog.go index d614ff8db..d8cc76ac7 100644 --- a/module/services/accounts/code/pkg/relationcatalog/catalog.go +++ b/module/services/accounts/code/pkg/relationcatalog/catalog.go @@ -126,6 +126,10 @@ var authorities = map[string]Authority{ Scope: ScopeTenant, PolicyShape: ShapeDirect, ScopeColumn: "org_id", Notes: "Append-only emit-reservation guard; system emits reserve against the sentinel organization.", }, + "audit_event_queue": { + Scope: ScopeTenant, PolicyShape: ShapePolymorphic, ScopeColumn: "org_id", + Notes: "The transactional audit queue under a swap value (ADR 0009): writers insert only, under audit_events' tenant policy; the relay reads and deletes as the job worker.", + }, "audit_events": { Scope: ScopeTenant, PolicyShape: ShapePolymorphic, ScopeColumn: "org_id", Notes: "NULL org_id rows (system events) are visible only under the control-plane role. Monthly partitions inherit the partitioned parent's policy.", diff --git a/module/services/accounts/code/work.go b/module/services/accounts/code/work.go index 0a870243e..94349f4be 100644 --- a/module/services/accounts/code/work.go +++ b/module/services/accounts/code/work.go @@ -535,13 +535,19 @@ func doWork(ctx context.Context) (Clean, error) { // AUDIT_SINK=both additionally tees each org-scoped audit row to an external // warehouse, fed asynchronously from the durable outbox — Postgres stays the // atomic source of truth (an external destination cannot join its tx). - auditSinkMode, externalAuditSink, err := configuredAuditSink() + // AUDIT_SINK=bigquery swaps the store of record (ADR 0009): the record is a + // queue row on that same transaction, and the audit relay delivers it to + // BigQuery and the archive afterwards. + auditSink, err := configuredAuditSink() if err != nil { return nil, err } var auditEmitterOpts []business.DurableAuditEmitterOption - if auditSinkMode == auditSinkBoth { + switch { + case auditSink.mode == business.AuditSinkBoth: auditEmitterOpts = append(auditEmitterOpts, business.WithExternalTee()) + case auditSink.mode.Swaps(): + auditEmitterOpts = append(auditEmitterOpts, business.WithQueuedRecords()) } // Every org-scoped audit record publishes its external domain event in the // same transaction; that event is what the relay fans out to the endpoints @@ -552,6 +558,10 @@ func doWork(ctx context.Context) (Clean, error) { return nil, err } service.SetAuditEmitter(auditEmitter) + // The resolver records registrations and SSO provisioning on its own + // transaction, through the same emitter, so they reach whichever store of + // record the sink selects. + resolver.SetAuditRecorder(auditEmitter) // Refuse to serve behind an emitter that cannot write on the caller's // transaction: every security mutation commits its audit row and webhook // fan-out inside its own transaction, and an emitter without EmitTx would let @@ -560,15 +570,26 @@ func doWork(ctx context.Context) (Clean, error) { return nil, err } + var auditRelay *business.AuditRelay + closeAuditRelay := func() {} + if auditSink.bigQuery != nil { + w.Warn("AUDIT_SINK=bigquery: audit events are queued in Postgres and relayed to BigQuery and the archive; " + + "the activity list, aggregates, exports and readable-source queries still read audit_events, which receives no new rows, until reads move to the store (ADR 0009)") + auditRelay, closeAuditRelay, err = newBigQueryAuditRelay(ctx, store, auditSink.bigQuery) + if err != nil { + return nil, err + } + } + var auditExportWorker *jobs.Worker - if auditSinkMode == auditSinkBoth { + if auditSink.mode == business.AuditSinkBoth { // The tee carries org-scoped events only. Control-plane / platform-admin // audit events (NULL org) are Postgres-only: the job platform reserves // global-scope enqueue for the privileged worker pool, which opens its own // transaction and so cannot commit atomically with the audit row. Surface // this so operators don't assume the warehouse holds platform events. w.Warn("AUDIT_SINK=both tees only org-scoped audit events to the external sink; control-plane/platform-admin (NULL-org) events remain Postgres-only") - auditExportHandler, err := business.NewAuditExportJobHandler(externalAuditSink, store) + auditExportHandler, err := business.NewAuditExportJobHandler(auditSink.external, store) if err != nil { return nil, err } @@ -1120,6 +1141,9 @@ func doWork(ctx context.Context) (Clean, error) { if auditExportWorker != nil { auditExportWorker.Start(ctx) } + if auditRelay != nil { + auditRelay.Start(ctx) + } if emailWorker != nil { emailWorker.Start(ctx) } @@ -1157,6 +1181,15 @@ func doWork(ctx context.Context) (Clean, error) { } cancel() } + if auditRelay != nil { + sw.Info("stopping audit relay") + shutdownCtx, cancel := context.WithTimeout(context.Background(), 30*time.Second) + if err := auditRelay.Shutdown(shutdownCtx); err != nil { + sw.Warn("audit relay shutdown timed out", wool.ErrField(err)) + } + cancel() + } + closeAuditRelay() if jobOperationsMonitor != nil { sw.Info("stopping durable job metrics monitor") shutdownCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second) @@ -1283,48 +1316,6 @@ func configuredAnalyticsSink() (analytics.Destination, bool, error) { } } -type auditSinkMode int - -const ( - auditSinkPostgres auditSinkMode = iota - auditSinkBoth -) - -// configuredAuditSink selects the audit backend. "postgres" (default) keeps the -// durable emitter alone. "both" adds an asynchronous tee to an external -// warehouse while Postgres stays the atomic source of truth. "external" is -// rejected: an external destination cannot join the audit transaction, so it -// cannot replace the durable emitter without forfeiting audit/webhook -// atomicity — a deliberate posture we refuse to let a config value break. -func configuredAuditSink() (auditSinkMode, business.ExternalAuditSink, error) { - switch strings.ToLower(strings.TrimSpace(os.Getenv("AUDIT_SINK"))) { - case "", "postgres": - return auditSinkPostgres, nil, nil - case "both": - sink, err := configuredExternalAuditSink() - if err != nil { - return auditSinkPostgres, nil, err - } - return auditSinkBoth, sink, nil - case "external": - return auditSinkPostgres, nil, fmt.Errorf( - "AUDIT_SINK=external is not permitted: Postgres is the atomic source of truth and an external destination cannot join its transaction; use AUDIT_SINK=both to tee to the external sink") - default: - return auditSinkPostgres, nil, fmt.Errorf("AUDIT_SINK must be postgres or both") - } -} - -func configuredExternalAuditSink() (business.ExternalAuditSink, error) { - endpoint := strings.TrimSpace(os.Getenv("AUDIT_EXTERNAL_URL")) - if endpoint == "" { - return nil, fmt.Errorf("AUDIT_SINK=both requires AUDIT_EXTERNAL_URL for the external audit destination") - } - return business.NewHTTPAuditSink(business.HTTPAuditSinkConfig{ - Endpoint: endpoint, - Token: strings.TrimSpace(os.Getenv("AUDIT_EXTERNAL_TOKEN")), - }) -} - // anonymousEndpointsUnprotected reports whether the four anonymous endpoints // have NO app-layer throttle: abuse protection disabled AND no rate limiter // wired (no Redis). Either guard alone is a backstop; only the combination diff --git a/module/services/accounts/code/work_audit_test.go b/module/services/accounts/code/work_audit_test.go index 9aaa39814..02e313c66 100644 --- a/module/services/accounts/code/work_audit_test.go +++ b/module/services/accounts/code/work_audit_test.go @@ -2,6 +2,7 @@ package main import ( "testing" + "time" "accounts/pkg/business" @@ -10,39 +11,125 @@ import ( func TestConfiguredAuditSinkDefaultsToPostgres(t *testing.T) { t.Setenv("AUDIT_SINK", "") - mode, sink, err := configuredAuditSink() + sink, err := configuredAuditSink() require.NoError(t, err) - require.Equal(t, auditSinkPostgres, mode) - require.Nil(t, sink) + require.Equal(t, business.AuditSinkPostgres, sink.mode) + require.Nil(t, sink.external) + require.Nil(t, sink.bigQuery) t.Setenv("AUDIT_SINK", "postgres") - mode, sink, err = configuredAuditSink() + sink, err = configuredAuditSink() require.NoError(t, err) - require.Equal(t, auditSinkPostgres, mode) - require.Nil(t, sink) + require.Equal(t, business.AuditSinkPostgres, sink.mode) + require.False(t, sink.mode.Swaps()) } func TestConfiguredAuditSinkBothRequiresExternalURL(t *testing.T) { t.Setenv("AUDIT_SINK", "both") t.Setenv("AUDIT_EXTERNAL_URL", "") - _, _, err := configuredAuditSink() + _, err := configuredAuditSink() require.ErrorContains(t, err, "AUDIT_EXTERNAL_URL") t.Setenv("AUDIT_EXTERNAL_URL", "https://warehouse.example/audit") - mode, sink, err := configuredAuditSink() + sink, err := configuredAuditSink() require.NoError(t, err) - require.Equal(t, auditSinkBoth, mode) - require.IsType(t, &business.HTTPAuditSink{}, sink) + require.Equal(t, business.AuditSinkBoth, sink.mode) + require.False(t, sink.mode.Swaps()) + require.IsType(t, &business.HTTPAuditSink{}, sink.external) } func TestConfiguredAuditSinkRejectsExternalOnly(t *testing.T) { t.Setenv("AUDIT_SINK", "external") - _, _, err := configuredAuditSink() + _, err := configuredAuditSink() require.ErrorContains(t, err, "not permitted") + require.ErrorContains(t, err, "bigquery", "the refusal names the swap value that does what external cannot") } func TestConfiguredAuditSinkRejectsUnknownMode(t *testing.T) { - t.Setenv("AUDIT_SINK", "kafka") - _, _, err := configuredAuditSink() - require.ErrorContains(t, err, "must be postgres or both") + for _, value := range []string{"kafka", "clickhouse"} { + t.Setenv("AUDIT_SINK", value) + _, err := configuredAuditSink() + require.ErrorContains(t, err, "must be postgres, both or bigquery", value) + } +} + +// setBigQuerySwap sets every required bigquery setting to a valid value. +func setBigQuerySwap(t *testing.T) { + t.Helper() + t.Setenv("AUDIT_SINK", "bigquery") + t.Setenv("AUDIT_BIGQUERY_PROJECT", "example-project") + t.Setenv("AUDIT_BIGQUERY_DATASET", "audit") + t.Setenv("AUDIT_ARCHIVE_BUCKET", "example-audit-archive") + t.Setenv("AUDIT_DEPLOYMENT_ID", "acme-prod-1") + t.Setenv("AUDIT_CONTENT_DETAIL_RETENTION_DAYS", "90") + t.Setenv("AUDIT_RELAY_BATCH_SIZE", "") + t.Setenv("AUDIT_RELAY_MAX_WAIT", "") +} + +func TestConfiguredAuditSinkBigQuery(t *testing.T) { + setBigQuerySwap(t) + sink, err := configuredAuditSink() + require.NoError(t, err) + require.Equal(t, business.AuditSinkBigQuery, sink.mode) + require.True(t, sink.mode.Swaps()) + require.Nil(t, sink.external) + require.Equal(t, &bigQueryAuditSwap{ + project: "example-project", + dataset: "audit", + archiveBucket: "example-audit-archive", + deploymentID: "acme-prod-1", + contentDetailRetention: 90 * 24 * time.Hour, + batchSize: business.DefaultAuditRelayBatchSize, + maxWait: business.DefaultAuditRelayMaxWait, + }, sink.bigQuery) + + t.Setenv("AUDIT_RELAY_BATCH_SIZE", "200") + t.Setenv("AUDIT_RELAY_MAX_WAIT", "750ms") + sink, err = configuredAuditSink() + require.NoError(t, err) + require.Equal(t, 200, sink.bigQuery.batchSize) + require.Equal(t, 750*time.Millisecond, sink.bigQuery.maxWait) +} + +func TestConfiguredAuditSinkBigQueryNamesEveryMissingSetting(t *testing.T) { + t.Setenv("AUDIT_SINK", "bigquery") + for _, name := range []string{ + "AUDIT_BIGQUERY_PROJECT", "AUDIT_BIGQUERY_DATASET", "AUDIT_ARCHIVE_BUCKET", + "AUDIT_DEPLOYMENT_ID", "AUDIT_CONTENT_DETAIL_RETENTION_DAYS", + } { + t.Setenv(name, "") + } + _, err := configuredAuditSink() + require.EqualError(t, err, "AUDIT_SINK=bigquery requires AUDIT_BIGQUERY_PROJECT, AUDIT_BIGQUERY_DATASET, "+ + "AUDIT_ARCHIVE_BUCKET, AUDIT_DEPLOYMENT_ID, AUDIT_CONTENT_DETAIL_RETENTION_DAYS") + + setBigQuerySwap(t) + t.Setenv("AUDIT_ARCHIVE_BUCKET", " ") + _, err = configuredAuditSink() + require.EqualError(t, err, "AUDIT_SINK=bigquery requires AUDIT_ARCHIVE_BUCKET", "a blank value is a missing one") +} + +func TestConfiguredAuditSinkBigQueryRejectsInvalidSettings(t *testing.T) { + for _, tc := range []struct { + name, value, want string + }{ + {"AUDIT_CONTENT_DETAIL_RETENTION_DAYS", "0", "AUDIT_CONTENT_DETAIL_RETENTION_DAYS"}, + {"AUDIT_CONTENT_DETAIL_RETENTION_DAYS", "90d", "AUDIT_CONTENT_DETAIL_RETENTION_DAYS"}, + {"AUDIT_CONTENT_DETAIL_RETENTION_DAYS", "-1", "AUDIT_CONTENT_DETAIL_RETENTION_DAYS"}, + {"AUDIT_DEPLOYMENT_ID", "acme/prod", "AUDIT_DEPLOYMENT_ID"}, + {"AUDIT_DEPLOYMENT_ID", "..", "AUDIT_DEPLOYMENT_ID"}, + {"AUDIT_ARCHIVE_BUCKET", "gs://example-audit-archive", "AUDIT_ARCHIVE_BUCKET"}, + {"AUDIT_RELAY_BATCH_SIZE", "0", "AUDIT_RELAY_BATCH_SIZE"}, + {"AUDIT_RELAY_BATCH_SIZE", "5001", "AUDIT_RELAY_BATCH_SIZE"}, + {"AUDIT_RELAY_MAX_WAIT", "5", "AUDIT_RELAY_MAX_WAIT"}, + {"AUDIT_RELAY_MAX_WAIT", "-1s", "AUDIT_RELAY_MAX_WAIT"}, + {"AUDIT_RELAY_MAX_WAIT", "2h", "AUDIT_RELAY_MAX_WAIT"}, + } { + t.Run(tc.name+"="+tc.value, func(t *testing.T) { + setBigQuerySwap(t) + t.Setenv(tc.name, tc.value) + _, err := configuredAuditSink() + require.ErrorContains(t, err, tc.want) + }) + } } diff --git a/module/services/store/migrations/17_audit_event_queue.down.sql b/module/services/store/migrations/17_audit_event_queue.down.sql new file mode 100644 index 000000000..59b5f73ec --- /dev/null +++ b/module/services/store/migrations/17_audit_event_queue.down.sql @@ -0,0 +1,4 @@ +-- Removing the queue removes every event still in it. Under the default +-- (postgres) and the tee (both) nothing writes it; under a swap value, roll back +-- only once the relay has drained it. +DROP TABLE IF EXISTS public.audit_event_queue; diff --git a/module/services/store/migrations/17_audit_event_queue.up.sql b/module/services/store/migrations/17_audit_event_queue.up.sql new file mode 100644 index 000000000..403e03750 --- /dev/null +++ b/module/services/store/migrations/17_audit_event_queue.up.sql @@ -0,0 +1,88 @@ +-- The transactional audit queue (ADR 0009). Under a swap value of AUDIT_SINK a +-- warehouse is the audit store of record, and no external system can join the +-- transaction a change commits in. So the emitter writes each event here, on +-- that transaction — beside the domain event webhook delivery fans out from, +-- exactly where it would have written the audit_events row — and the accounts +-- relay delivers it to the warehouse and the locked archive afterwards. A row +-- lives here only until both have acknowledged it; then the relay deletes it. +-- +-- It is a table of its own so that audit_events stays exactly as it is: its +-- append-only triggers, its SELECT/INSERT-only grants and its policies are not +-- touched, and under a swap value it simply receives no new rows. Under the +-- default (postgres) and the tee (both) nothing writes this table. +CREATE TABLE public.audit_event_queue ( + -- Insertion order. The relay delivers in this order, which keeps every + -- organization's events — and the platform's, which have no organization — + -- in the order they were queued. + seq bigint NOT NULL, + -- The writing transaction. The relay reads a row only once every + -- transaction older than the oldest one still running has finished, so a + -- row committed late can never be overtaken by one queued after it. + xact_id xid8 DEFAULT pg_current_xact_id() NOT NULL, + -- The event, column for column as audit_events keeps it. + id uuid NOT NULL, + event_type text NOT NULL, + schema_version integer NOT NULL, + actor_id uuid, + actor_type text NOT NULL, + resource text NOT NULL, + resource_id text, + org_id uuid, + payload jsonb DEFAULT '{}'::jsonb NOT NULL, + ip_address text, + created_at timestamp with time zone NOT NULL, + impersonated_by uuid, + is_impersonated boolean DEFAULT false NOT NULL, + client_id text, + -- When the writing transaction began: how long a partial batch has waited. + enqueued_at timestamp with time zone DEFAULT now() NOT NULL, + CONSTRAINT audit_event_queue_actor_type_check CHECK ((actor_type = ANY (ARRAY['user'::text, 'api_key'::text, 'system'::text, 'agent'::text]))), + CONSTRAINT audit_event_queue_event_type_format CHECK ((event_type ~ '^[a-z][a-z0-9_]*(\.[a-z][a-z0-9_]*){2,}$'::text)), + CONSTRAINT audit_event_queue_impersonation_identity_complete CHECK (((is_impersonated AND (impersonated_by IS NOT NULL)) OR ((NOT is_impersonated) AND (impersonated_by IS NULL)))) +); + +ALTER TABLE public.audit_event_queue ALTER COLUMN seq ADD GENERATED ALWAYS AS IDENTITY ( + SEQUENCE NAME public.audit_event_queue_seq_seq + START WITH 1 + INCREMENT BY 1 + NO MINVALUE + NO MAXVALUE + CACHE 1 +); + +ALTER TABLE ONLY public.audit_event_queue + ADD CONSTRAINT audit_event_queue_pkey PRIMARY KEY (seq); + +-- A queued event names a registered type, as an audit_events row must. +ALTER TABLE ONLY public.audit_event_queue + ADD CONSTRAINT audit_event_queue_event_type_fkey FOREIGN KEY (event_type) REFERENCES public.audit_event_types(name); + +ALTER TABLE public.audit_event_queue ENABLE ROW LEVEL SECURITY; +ALTER TABLE ONLY public.audit_event_queue FORCE ROW LEVEL SECURITY; + +-- The tenant policy is audit_events_tenant's, expression for expression: a +-- request writes only its own organization's event, or an organization-less +-- one whose actor is the bound user. +CREATE POLICY audit_event_queue_tenant ON public.audit_event_queue + USING (((org_id IS NOT NULL) AND ((org_id)::text = current_setting('app.current_org_id'::text, true)))) + WITH CHECK ((((org_id IS NOT NULL) AND ((org_id)::text = current_setting('app.current_org_id'::text, true))) OR ((org_id IS NULL) AND (actor_id IS NOT NULL) AND ((actor_id)::text = current_setting('app.current_user_id'::text, true))))); + +-- The control plane writes platform events, which have no organization, and +-- every event of a mutation that runs on the control plane. +CREATE POLICY app_control_plane_explicit_rows ON public.audit_event_queue + TO app_control_plane + USING ((CURRENT_USER = 'app_control_plane'::name)) + WITH CHECK ((CURRENT_USER = 'app_control_plane'::name)); + +-- The relay drains every organization's rows, on the job worker's pool. +CREATE POLICY app_job_worker_explicit_rows ON public.audit_event_queue + TO app_job_worker + USING ((CURRENT_USER = 'app_job_worker'::name)) + WITH CHECK ((CURRENT_USER = 'app_job_worker'::name)); + +-- Writers only insert: nothing that writes an event can read the queue back, +-- change a queued event or remove one. Only the relay reads and deletes, and +-- it never updates. +GRANT INSERT ON TABLE public.audit_event_queue TO app_tenant; +GRANT INSERT ON TABLE public.audit_event_queue TO app_control_plane; +GRANT SELECT,DELETE ON TABLE public.audit_event_queue TO app_job_worker; From e19aaa3736d037863113f9641b6fdd05aa76ab44 Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Fri, 2 Oct 2026 19:18:07 -0400 Subject: [PATCH 05/69] feat(audit): take the deployment's archive URL, let declared types state their retention, and run the import under the service's sink Settles four points on the swap's write path (ADR 0009, #996). Configuration names match what a deployment emits: AUDIT_ARCHIVE_URL replaces AUDIT_ARCHIVE_BUCKET and must be gs:// (its scheme picks the archive writer; any other scheme is refused at startup until a writer for it exists), and AUDIT_CONTENT_RETENTION_DAYS replaces AUDIT_CONTENT_DETAIL_RETENTION_DAYS. Warehouse duplicates are handled on read. BigQuery collapses a repeated insert id only briefly, so a batch redelivered after a restart may write an event twice; every read returns each event once by event id, as archive readers already do. The interface, the relay and the insert-id site say so; committed write streams are not built. A solution- or module-declared audit type may state its retention class: manifest dashboard.events[].retention, or the new ModuleAuditEventTypeDeclaration.retention. Saying nothing is content; any other value is refused. Like pii the class only grows, so a re-declaration may raise it and is refused if it would lower it, omission included. Store migration 18 adds audit_event_types.retention_class (CHECKed, content by default), which admission writes and the startup projection fills from the catalog. The proto change is regenerated with the CLI CI pins (0.1.171) and the SDK tree with the one that produced it (0.1.170); every generator reproduces the committed tree, and @codefly-dev/saas-sdk moves to 0.3.16. The role-catalog-import deploy job declares service_environment: [AUDIT_SINK], surfaced in the bundle as serviceEnvironment, so the driver gives it the accounts service's own AUDIT_SINK and its events reach the queue under a swap value. Co-Authored-By: Claude Opus 5.5 --- AUTHZ.md | 3 +- gitops.go | 25 +- gitops_test.go | 27 +- module/FEATURES.md | 4 +- .../api/accounts/authority/contract.binpb | Bin 938963 -> 940443 bytes .../accounts/v1/module_capabilities.proto | 22 + .../api/accounts/connect/contract.binpb | Bin 938963 -> 940443 bytes .../accounts/v1/module_capabilities.proto | 22 + module/contracts/api/catalog.codefly.json | 4 +- module/deployment/README.md | 13 +- module/deployment/jobs.codefly.yaml | 9 +- module/module.package.codefly.yaml | 4 +- module/services/accounts/AGENTS.md | 7 +- module/services/accounts/code/audit_sink.go | 79 ++- .../module_audit_declarations_test.go | 33 ++ .../adapters/module_capabilities_server.go | 10 + .../auditstore/bigquerystore/bigquerystore.go | 13 +- .../accounts/code/pkg/business/audit_relay.go | 8 +- .../code/pkg/business/audit_relay_test.go | 11 +- .../code/pkg/business/audit_retention_test.go | 67 ++- .../accounts/code/pkg/business/audit_store.go | 6 +- .../business/audit_webhook_visibility_test.go | 1 + ...ule_audit_declarations_integration_test.go | 39 ++ .../pkg/business/solution_audit_events.go | 65 ++- .../business/solution_audit_events_test.go | 22 +- .../pkg/cataloggen/deployment_topology.go | 15 +- .../accounts/v1/module_capabilities.pb.go | 505 ++++++++++-------- .../code/pkg/infra/postgres_audit_registry.go | 11 +- .../postgres_declared_audit_relay_test.go | 2 + .../infra/postgres_solution_audit_events.go | 22 +- .../services/accounts/code/work_audit_test.go | 34 +- .../accounts/v1/module_capabilities.proto | 22 + .../services/frontend/code/package-lock.json | 4 +- module/services/frontend/code/package.json | 2 +- .../saas-plugin-manifest/src/data-graph.ts | 31 +- .../saas-plugin-manifest/src/index.ts | 1 + .../test/data-graph.test.ts | 20 + .../generated/contract/catalog.codefly.json | 2 +- .../generated/contract/contract.binpb | Bin 938963 -> 940443 bytes .../saas-sdk/generated/library.codefly.yaml | 2 +- .../accounts/v1/module_capabilities_pb.ts | 61 ++- .../code/packages/saas-sdk/package.json | 2 +- .../code/packages/saas-sdk/src/schema.ts | 6 + .../accounts/v1/module_capabilities_pb.ts | 61 ++- ..._audit_event_type_retention_class.down.sql | 2 + ...18_audit_event_type_retention_class.up.sql | 18 + 46 files changed, 985 insertions(+), 332 deletions(-) create mode 100644 module/services/store/migrations/18_audit_event_type_retention_class.down.sql create mode 100644 module/services/store/migrations/18_audit_event_type_retention_class.up.sql diff --git a/AUTHZ.md b/AUTHZ.md index ca49e94a8..5c921afaf 100644 --- a/AUTHZ.md +++ b/AUTHZ.md @@ -912,7 +912,8 @@ The importer records one audit event per changed role through the same emitter the service uses, so it must run under the deployment's `AUDIT_SINK` (`-audit-sink`, defaulting to `$AUDIT_SINK`): under a swap value (ADR 0009) its events go to the transactional queue the accounts relay delivers, not to -`audit_events`. +`audit_events`. The deploy step below declares it (`service_environment: +[AUDIT_SINK]`), so the driver gives the Job the accounts service's own value. Domain core is `pkg/rolecatalog` (parse + diff + deterministic plan, no DB); `pkg/infra` snapshots current state and applies the plan in one transaction. diff --git a/gitops.go b/gitops.go index 8403a4988..7f04b0bd3 100644 --- a/gitops.go +++ b/gitops.go @@ -113,6 +113,11 @@ type bundleDeployJob struct { Force bool `json:"force,omitempty"` Writes bundleDeployTarget `json:"writes"` After []string `json:"after,omitempty"` + // ServiceEnvironment names environment variables the driver sets on the Job + // to the values the running service has in the same environment, so the + // step behaves as the service would — the import, for one, records its audit + // events wherever the service's AUDIT_SINK sends them. + ServiceEnvironment []string `json:"serviceEnvironment,omitempty"` } type bundleDeployTarget struct { @@ -317,6 +322,9 @@ type topologyDeployJob struct { Force bool `yaml:"force,omitempty"` Writes topologyDeployJobTarget `yaml:"writes"` After []string `yaml:"after,omitempty"` + // ServiceEnvironment names the running service's environment variables the + // Job receives with the service's own values (bundleDeployJob). + ServiceEnvironment []string `yaml:"service_environment,omitempty"` } type topologyDeployJobTarget struct { @@ -453,7 +461,8 @@ func planDeployJobs(topology deploymentTopology, plan environmentPlan) []bundleD Endpoint: job.Writes.Endpoint, Port: endpoint.Port, }, - After: append([]string(nil), job.After...), + After: append([]string(nil), job.After...), + ServiceEnvironment: append([]string(nil), job.ServiceEnvironment...), }) } if len(jobs) == 0 { @@ -1833,10 +1842,24 @@ func validateDeployJobs(moduleDir string, topology deploymentTopology, services return fmt.Errorf("deployment topology deploy Job %q runs after undeclared service %q", job.Name, after) } } + inherited := make(map[string]struct{}, len(job.ServiceEnvironment)) + for _, name := range job.ServiceEnvironment { + if !environmentVariableName.MatchString(name) { + return fmt.Errorf("deployment topology deploy Job %q service_environment entry %q is not an environment variable name", job.Name, name) + } + if _, repeated := inherited[name]; repeated { + return fmt.Errorf("deployment topology deploy Job %q service_environment repeats %q", job.Name, name) + } + inherited[name] = struct{}{} + } } return nil } +// environmentVariableName is the shape of a name a deploy Job may inherit from +// its running service: upper-case letters, digits and underscores. +var environmentVariableName = regexp.MustCompile(`^[A-Z][A-Z0-9_]*$`) + func serviceDependsOn(service topologyService, dependency, endpoint string) bool { for _, declared := range service.Dependencies { if declared.Service == dependency && slices.Contains(declared.Endpoints, endpoint) { diff --git a/gitops_test.go b/gitops_test.go index 2a76635bd..845adc2cf 100644 --- a/gitops_test.go +++ b/gitops_test.go @@ -597,6 +597,26 @@ func TestGenerateBundleRejectsHostileContracts(t *testing.T) { }, want: "references undeclared service", }, + { + name: "deploy job inheriting a malformed environment name", + mutate: func(t *testing.T, moduleDir string, _ *workspaceManifest) { + t.Helper() + writeCatalogArtifact(t, moduleDir) + appendDeployJob(t, moduleDir, "store", "http") + replaceInDeployJobs(t, moduleDir, " - AUDIT_SINK\n", " - audit-sink\n") + }, + want: "is not an environment variable name", + }, + { + name: "deploy job inheriting an environment name twice", + mutate: func(t *testing.T, moduleDir string, _ *workspaceManifest) { + t.Helper() + writeCatalogArtifact(t, moduleDir) + appendDeployJob(t, moduleDir, "store", "http") + replaceInDeployJobs(t, moduleDir, " - AUDIT_SINK\n", " - AUDIT_SINK\n - AUDIT_SINK\n") + }, + want: "service_environment repeats", + }, { name: "deploy job to a migration-bearing target without ordering", mutate: func(t *testing.T, moduleDir string, _ *workspaceManifest) { @@ -809,6 +829,9 @@ func TestDeployJobWiresCatalogImport(t *testing.T) { if !slices.Equal(job.After, []string{"store"}) { t.Errorf("%s deploy job ordering = %v", env.Name, job.After) } + if !slices.Equal(job.ServiceEnvironment, []string{"AUDIT_SINK"}) { + t.Errorf("%s deploy job service environment = %v, want the running service's AUDIT_SINK", env.Name, job.ServiceEnvironment) + } } overlays := filepath.Join(moduleDir, filepath.FromSlash(bundleRelativeDir), "overlays") @@ -2348,7 +2371,9 @@ func appendDeployJob(t *testing.T, moduleDir, writesService, writesEndpoint stri " service: "+writesService+"\n"+ " endpoint: "+writesEndpoint+"\n"+ " after:\n"+ - " - store\n") + " - store\n"+ + " service_environment:\n"+ + " - AUDIT_SINK\n") } // serviceManifestFixture renders an authored service.codefly.yaml: one endpoint, diff --git a/module/FEATURES.md b/module/FEATURES.md index deabe5fcd..380715eaa 100644 --- a/module/FEATURES.md +++ b/module/FEATURES.md @@ -656,9 +656,9 @@ Environment variables consumed by the api: | `POSTHOG_API_HOST` | Separate PostHog management/deletion origin | | `AUDIT_SINK` | `postgres` (default), `both` (tee to `AUDIT_EXTERNAL_URL`, optional `AUDIT_EXTERNAL_TOKEN`), or `bigquery` (swap the audit store of record, ADR 0009); `external` is refused | | `AUDIT_BIGQUERY_PROJECT`, `AUDIT_BIGQUERY_DATASET` | Required under `bigquery`: the store of record. The dataset must exist; the `audit_events` and `audit_event_details` tables are created if missing. Credentials are Application Default Credentials only | -| `AUDIT_ARCHIVE_BUCKET` | Required under `bigquery`: the GCS bucket, under a retention lock the deployment sets, that receives one object per relay batch | +| `AUDIT_ARCHIVE_URL` | Required under `bigquery`: the locked archive, as `gs://` (the scheme picks the writer; GCS is the one built). The bucket carries a retention lock the deployment sets and receives one object per relay batch | | `AUDIT_DEPLOYMENT_ID` | Required under `bigquery`: stamped on every record and the archive path | -| `AUDIT_CONTENT_DETAIL_RETENTION_DAYS` | Required under `bigquery`: days a content-class event's full details are kept (the details table's partition expiration) | +| `AUDIT_CONTENT_RETENTION_DAYS` | Required under `bigquery`: days a content-class event's full details are kept (the details table's partition expiration) | | `AUDIT_RELAY_BATCH_SIZE`, `AUDIT_RELAY_MAX_WAIT` | Optional under `bigquery`: events per delivery (default 500, at most 5000) and how long a partial batch waits (default `5s`) | | `ERROR_TRACKING_MODE` | Explicit `disabled` or `sentry`; rejects partial config | | `SENTRY_DSN` | Server Sentry DSN, required in Sentry mode | diff --git a/module/contracts/api/accounts/authority/contract.binpb b/module/contracts/api/accounts/authority/contract.binpb index f4b26e939e72e4235bd2b1586c70ea84caab7867..ab689635876dc824d1e4fdbc1dd44e398a19c7a8 100644 GIT binary patch delta 20185 zcmZ{sd6*SN*5>ccjEvj>A)zR$DB!XqP{=BuZQJdJg6#_0ZR^)dHIz_eS>;gB>eus4 zf}$v^+Q?M_1zbRoeYq&^sHi9^iU^9J*b0KQvMDGs?>P~fQB(8GAKee{$%vCDezBd1 zyyXWA&YHFAtZ@^v6RZjL1Tn#xASbvJyb1p5VA-wK)=h;Kw+7EQ%kD4KP6~>%Euvw8 zKe}>EWzCqX+L~5QF59h1-~8u=pd{aJN)U7$nmjt+dv>sMd)8{29cs#VcqJQj%wIVr zXj^z{agwEx z$}tsH)pfbr8*&v8!>cPsSB}Vyx$Wl4u64Oe3cW4|Q|`vfo5$p8Ylc_mDuUcCu#Bo5 zTUVK@s$*9b)X-erm|B#Cr_q(eYe$cWbKTCnaA>Y~@7`yf-Ya)C-dI`l_sZ(pn=5l) zjGK|WWmIiX{9o1ZQ9-VD^vH^us{g7W{$#8e!y)MBxiJ-^M^=s@RawJzD{4j{+))*E zqv&(%8&x}cOy%gFx#6`nH&l%rOPz~<(ookh6oKm6;Wt)}2+-BT zM^*j3GB>KWdISbVP4#UYblPzQo^TWTG|(&p{lz-HyAG5X-ELc5q=lmk(F^GG1<1f;B#;x9ZAS(c7E{jU{^fZ z_8TV!Z7SHbS#X-F=TX6Enk|AT@Tj6U(QiVw=F@CDXcqX14Pi~OMU(m_xSDEPPdC&CDJ2VGDv&RQV?9C^g><25|OyqwgT-P z(Lhxs4<*U4OCs^Ed+n~}iM$Z5?zMaONi<$|uif{r`ehbg7O}s6f6*mu_?u~4Wx)p) zw*~cQ7OH*o6I(PN_Bw^w=?aY*K4;n+%|>VnRfYgx`cq+S+>l{;3QRn`q^ePLQ^jq$(G^v7 zSXlIukQ-B*OICD@NQ}|i8oeakQaiSK1P1yoaFP_^wXdpXcr}(=EaHE=4Kty#;_vit zEFE>Zu~>D+RSvJ9#hRBIHJImSLkP^Wk9CliOE}@> zdv20^DcW-A+#)gZnJI`yVxTp}Ehp zdt4hV5ST@Tp*KywLVf;}^h= z_bs%e<~(H}F=ZCoEuPj>h73Gq7TQHm>nWpc=>4aXDWgqrwTPw+O)PD~pjd1RH8@*Z zSqzFLwiO3K6AmgHNyz24ql3w!_!2wh1rZ)-S}n1A^iH0_{}Q`je;sNTp)RopEKftF zi4~FU%;jmQY=fJnCa$bktmecrTQmzk%K~C?B-h%RXC62i?u^&kUQd2uv6Vr%g=AVn zvf`c+P*@QPcGxT_i{ZM%0M%!lc1rsb~{ZB$S_g6=eiAz7EH6$%pG->|W*z2_j{ zl1P4JXSNrnJK?)l{E_Xc-YwyPw@?o z@Qg&0N+Y?~&U}N7Ymv<5=PgGXThhpFT=Qwc-(IFTU~-6o*?HmTU(>oCDchfR?`&*_&;I$ zY6?KY{|UQA5zkLZ%+C{c%aeM3l7W|!6L#@QJwLS#W9_G8DbXgl`kAKeYNX<#ERqvM z=5CQKeZtCJR9SenY1kthPY|A(ZDma1PGPw`JRu_irNV+jly?fR5jP4FuJ075sg1qN zqBh=Ts-i4Zu`Wvuyi0frAj+cZT|xuApvAjMwJ0-OEsz<~cxW4Aj7i9X?gu=_BC=Y9Wl3?wwOX`Ms{@|G*=o_b3wH}7;#w`b z7SgylJ0x4TLK+v_;ARa4K|49sQyb>mrb09i7Sfb#7s(ey=C#7y)?wFd{DSaQ1nsz6 zUQD_LG7?Z~K0(3ni%GXYBD@!qZfVERUouru7E0yRc*s(xEs&`ClF%DM+>WQ>%cd^4 zLa7BE4^j7J;j1|YiMlU~vNp`Don>PdzhbK5A(Wc7c!;X6BpnUOPch-BciLG3RW}GL zR>uNpmPuVo?KYUYl-X^dL89t9sQapUtx{_7XlFU7`f4)0AW`*InqEV3F=0pY4UyS` z)$qdHg*bcF)#i)u3r@=qYa8~voZ@{$IJ~nvHUocCV5i2uO)?TtoKKMgr8h;dGbkWP zBD_i9O#VkMODg|i+whuBc!ZLhgg#N*%yYBAMmCyD;^F4}W$nV6&Ui#ZsM831QaJAm zt5-0aPPEXF`Q7cp2aEA<6n$Ux=^M91pjh!ji%D+#WpKnZK_?_^0xn z!d3GGWymT@6Av`ZcF`VnEmy%P+m&zKF}$vPm+;hXp5t@FE>WW5&sj9OKBv)j9akZh zpWiY3&)nxiA5&2VZ~0u{n94bD`$&E*GT-Lgc0v;TbuyvbGylE9YRM2GBLSrdL!qgA zMX}0ZAmMSZ=%iBQ_KbL6GQ=T~(FjXzrcf;P(H#<1_X!-`sq5nQ%=8;m7r>!Z>+leD zze%PCB)c}9TA+qhSv;Pi z?g7zCjUY(WJs>)C435wlqC+GPiOdg}mjBA;@|#xR5_D*n@Wc3!@Y?h0-ND0deppz? zu+&nZ(C~Lycl>W>#8UV#o2N< z(ScDP6$nBXM%|;LSrlxcdrlF_QzCPRQ+TIqcxg7KHlD(Oqq2_R*N$`4>?0WoDDG}j z=#_C!JDvuR)T84abP_E#kgU{71ckEW9W|fuke=S|Slm{UtT5$s{8pt~mjwrCY3LZq zNls>}lP%q8+h3)vzE|ucl^CaoVXR2~XjDilZ-1AmM$A z({pIh6gN;LC;BC1DZdKdEYT^F)1Ax%PL^&-MJKqL?&##c6E|uGHEJ1MsX#{p^4!w7 z7T+)zTK5dQWY3@B=nNbM@a`E7GH{9v67QbjwEQh^<0MtfW;n&a3APg1&d3*?%)-JQ z<>A%Yc($Vtlb!j^bLh>Kc0fiFYM!IgP-w(C$I-bhBwjnm(b*p)(u_Gy={!fJp=99n zXO7c)o}mpJ;qBJRS= z?^0724S`BxqJ}I@x*ugxcPVv0jk+$p{4O(9@eoR#-0=`qmnFRe$xqQcswyqN%N;9L z{JKy+1*!N!q3-g8A0+B7Cw{7~gSsmmeMQlQwqz)E8wQ1{D;#}80g0+B9NbXQ6zqxw z*vYKbQ?M&aUb))w)I!^p7v(h$(omZFkVuRjz5PL<@EWH`EkclRyT-xx_XoJ`7RhIv z%zCU^L$GMA4O-ar%%H3jOkl0j2U&C?st53fdq-V`yHJ!mQw;H4kWV# zl9gJNpjhfd1|&QkaH1qAT80idI?XS~-ht8wsRcvOk)*}J{K>P!-<2P9{G^UVor9*1 zi$dQKps(k0@PYUIhhk%b#z-uV0fQBsm`0H?BHa_Zv zYA1sgp&oTQ@<&uE1RqBo9M)(gms>twdrYT#Bv~cz^HKGf(^{>dc!;XUoDQ9W=PBKE zB6-rujF;KcrC1B_5uH9CURYH(yt=lou=F=!$81cLPC5>^*l~EU{X$76rInEQW~wI* zDU6U`XqF%@NE%(I9Gvk~S*19IriUnd%27EpB#owVQe|}{Ss|fr&7PEQ-yzs&W5V=` zHM&3JVol<3WZQ-iqxBHFP)(o8%=-RaU*s~jY%PM7+g zqnAbd^1XyaRmBj8Qu_oRqUya;@5_*=dapdaN%vp_jTITmyv)ogJbyv>n`{S(vU%z7 zV&F(-`;bIe55Ta1jKo9vHiN=Go$+sAc}VJxmOLnCOPu2Fqe}>AT$<+V28C0L@oyBJ zExUFLrqH*FGa~tv%&aaHUldNZ;-{qEs?XrvZjr=_K#hRJhc1aj5y63i!$nfhYDhF< zk<_#L3_d(8PTCKN4`R|%=Qb#oW$K~|5>*#V3_5aLbO!hK66&(w^J@|7lKh!N!fVTy zNS*Ve4C*YAI_Ez_WnWaKKUcxSW%>0(!ZXX4NqudMGGtA9=nOveu8>$maPfoYpmSH` z+g=i0AFq%_>N_NisIo%#RNpSn;7(m>8ih`UN*|(8D`lDbl7X_Qx>EK~U%Sq*Bnq!e zh9M*?m4-s0>ME(fQ?-Of-Bl8IC%E_$y(76+W}eA6`CWKbyjJSZbiH{B7NoxYhs2jR ziEll`5DFd((o<_WBpenJT98=$3bN&3KAn-ANUK#r7GKPnm9`+_f*f#meN)WE&bdgx zg)WoUC5`G$oeVW7dB{@FEj&ckby80)NQAIXwz^Pz=j?(@lw5$(W{EzLd{$t)}5x-d22xugkw zs0mOiy@O(@vr$O9IXy=1{U)UoeIxmf%zRn+_od-rJANmbHhsBgHYYwIaW^P+S_cK6 zo25RLLn55bQlH8pfn~GQr*e`it=la1sk|>QpIgZJUi!ifjT}i@ALZ9x8GapaN!Bvh z;dl%2rV}6}^0_Ut)kjjDPRYRe+!k5#kyNKsZNqf?KT&p~ik+ zp0V2`WfGKZLPrvEhjf@SG#b54>Rb~N4!232YeFKnZ8F-C#zr^q_`%9H*?dPD8`}`u z$0jy5!PRyt)Zs7g%L~CLhCQA_{Z@xT%l#)(ACgg)pXy^6B-noQTtQc_*56*iyF$e-jz0!e%63yek#j$rOjj; znz_?7lTC2-8FhgQgc7XXCJzP6B4y|OQf4B5C?(mmL+@AKpYjw zZdtk~4TNn73NvCggO)wAsm(^!*mF0a>7zNRENM8 zgm74%bwO&1)d)Q-2VJH8P$GWBgn$ZAS0oP+!jWX@>Cdz9NV4=mB7`Hd#pOB#H3wlS zx=g<>nSn=Sf7Lzx&*yJsG)a!gb1&r~F(8u1WacM5BnIG($C52;0Qb?4lu!L91_Wdz ze#~ETZTRQ%AEm2qu3&_>AElm@19-VRK@Fl^8WP{y^INYCuWx@M=}s6?01Y1)$%oy{({8r(gCIAovbu8kn7YF8>%yy@_+i%@ zi1*rJpc~0qZuU`^mV6fyJ3p>DwTZ^!K;|?z2^$jEkO?O!ggrOm1c?adCY&G<_S}RM zNyRCin}E_buBAZ@@5)14?w{)><>9}|=!7}!gYn{&Xi^@#&TO(z5k+)zimFR_~Zf1VS z36CVqOI@^jA$ED0K_|!+ZuTYDD$QqegKni?rRL`h4`;@bP~U_EV2x|X zWkK^GOp3W#43>f`2H=OML25+hb*|`LzcPC5S=a7U--N^Tde`n&-xMI*a_#Q*O<^RS zckLdjsRV6k6p0sHyLYftp+Z9Rnw$MFe??{ZY5bb&@D~aMLua1X+-`r+cxWKK=Kift z1n)TZ4j4@O;pqU=vXMyB-R*5sL2&3bp~o_ohW`0*XTh1)G54utC8lpg7W?B2XNq zQXABmK#QE8UF&Wy-}A=s<@je;^7@FjKXLrEm8ihc`ldhQPC8I(p z6erzqqL+*cwgJU21_hho`IJGyCZHJSnNh(uLNVS;M+Mt};&v|`6>I{EJG^vMun8z8 zcYn*idU1_YY`Vn&0C0Pz5o`U8W&CDcOz@kIWjn(+7WL!RUl7FsS5#zS6HzP=|3 z5D$5M25S%!ARhL_<0%NKKs@Y)kEbBm1`x9h2sXjxB#)x6Dv~?t^%>p3s}> ze5W^qPV_4Pw3_bA*LW&_B8kY?c=$PkehMaQydLLjCb)u^Yd!H&%EVJl)_UPfDHFEA zYr!yK6FjdoOuU$?0?IR$MGD`S}`z4-l_(=P2jMx-Yj7ofq27!U>iWZY1+dkfOyM*U=u)WG9cIl5N|iA z2oUd3sVkX75Xtv3OLpd~>cT7H_szN)AdL6Tx)}h(`)1t?0Ah+QTM*_}GA86F_V?AlL*DI~r63h)<|g z4TA_Hxf>w?GfN*2(;@{~rD+{@ws-gfPB0KpFwa_XbEK06FA| zpHm=>6p%w+_;U(`Z2)rE0AUk6A2C4K1RzK2jSRLCh+}4Cuni!7FzsOzK>TPxun8cJ z8xU*)h!YJe0>n>LYAAzfOusnutjT_U)-B`J z4PneOAWj2_Sq8*u05RJaPo^MFQy^yh;gcx{wgJQ(1AvCV+U%fM63qJl>!pKs-UE@YnA&B%0zdw#c`h$xr)7I454@Oa978 z4-m&9zs zb*aHNKD96RagC@hHIxyRSNQ1~$~NFxWm?H5;8|_(unBn97(8qOp0)KWwo{iH1u8a6 z@o5&x=X~o8-!7empV|J;>|37{KKtJx{)%_AFZtItXgmGs?`FT0KM0aU zu$%qg@`p*1AhX%O=+AmcMfmdjt}oi)qmHDfqP^DY-VBfBd%oC_f zZ1FVuy7ayw$K~K|iy_A*1oMHPCar8Da330QYy-Hhru}RJxQ`4tHUZo=1#Tc+*0Bxb zK5kGGiGpWa^GNRTt$$-Q9kcV7b_&Z16UK-C%Eo(q$v<|4r{?gr$8UY6hSVHU z@A1#$pXt$4Al%~*uF!+4IT(HEi-Ren=8Dmmet0lt#5N%MmqElPyyz>1D5fT`4R8C} zFk%xi+G`lG4UG1gma++qzA=p01V;PK;9?UH9cWMyh+2G0#lBNBu0TPPGq{BV4b?AB}pilYVvo8bAF z87*uAi682%NNnR3>A0U}b;^j!C(Me(Ht_k`G?Pu>bJFl(6Zrh%%W}Pivk81o`Lb7P z()Wnt0`NJZW*`1D26qQmzA)~t@V0DxcOd!K9O%Fz1af!KE?I|)z~}B@NOBEb1U~l! zVrI&xNb$KR2xq2z*akk6f;5|96Fg52(rkuJ;4>wlY(}lvY-2vt0Eu`T+5HO!2{AsLu|p#f7H#gdOlB z^B`TPTOyFzLD_%k@zWA~W(ODiRga&R;4>!>i&8!<6`wglxG3erHt>1G@L>}?KWg}} z34G?(3WhfUz~q~XIR@L6E^unBw?8a^!(K2H&!=kYBC z!vuePzA~_0E)1C*K9`MG1~R!cC`KSFgZAfZKE>d(GPo?+@r%J{RUlqU`4lTYtAg;Q zln>j$XSLzOCU{Ox) z;Pa8;!zS?AX85oPd_FdOT8UOFBic@U)UgtOU%xZ3_T`)2k5lE&Aib3mty~G{d=`kkDV-8U=d&Q(o6=z$=MX@3fcUCGMQl`GQz><`j=+ERcx5Qt+bh*AaOKoA~FL9h)VzBM4&1kVQz2sQ!4zw3cu8-e)F zfM6Ryd~e#rCV)6(K(GlQ4jT|`0*E6GDgwk&Ds?H(7yP#Zj^i(RdNc>_*unij zhWX|*!)xM+q2%AQVy!4c923JEy{V9nuMdI!-ygwzuHjua{ zOt)q>!Skdr-J0115|cx7(ZDt$F(phV2irhms%a3LKw_F9!6uNHZb+~RB<^ic5hU)T zQuu)h+Kd0*zyo$@&C6Hi@w?6k!gQ5rgD@Tl(^aAkKs*qpt3(@sm>G(>DTp=-#LO_9 zn}T2)K;#VwHo^0Q1_YY`;-OIHFL*F)_Sc6^ixkWK|FZ+)x%b;QzdDRd>yuoRf2Wp00Yk>Q_s@UHuKU zOD^0p@4{gt@*}Jf_6RY;86iiwBfJs*(%_{(S>sj~7X2xBy=8uKVMt9-k}vEW6KpLl z+&nkWfD3T|w+^8ft9X#0aq~&!Dng)(2 zvT{x&zPBtd_$VI;(vgHbZ6&p*gT3+hmftijXj8!!EraH&o<{|v)0POLz@v)ZXulbG ztxj8Z&@%8-&x*tkmT1<{LZ}}t(LzVgqX}oI2~D^Kc~)1`#TYU2to_z$Vbncim6WE^ zBUyHpj-IDRaEbCObqPyE;;dx_IyzbdH7LC(NrqigiF=*3x>l$1N`yLV_3o2u-0ZB? z_t*Mn7H$@CynesZC2YhSXC4?PmpL@3s zYa?t6gdJ&D-xbUg7`x&~K4Ir-?R@zZQI+f)6Lc&bv%=H)4o}#g3R2ADIKc+QIvPjl zNJ36dS~%hCc!KRW=bORDy(idFD;`Ej4C4g5^<-NOBN;r56YQeNwi-t5p+!&GdKk5Z zP!nwoBMr8;(EmxcP{4| zB$S0pZ$k@UP8z982~YVscBG~YBtp%h>GCN}hSEsBZ0D8~uJOX(S@FxZ*OqTl$}?aQ z%>Y_Pkdc7;d3q6dTx9!gQi+g@?6&Pvi5a-auDX^-5>;sqV7wRE?FZ@-G*XLgtBnqh zXiz1qlBKl(cobJJU4jNn*$(43)4PdWws**vAf)0f^4Ch(#O50N- zU&d`;m9`x+5>P+mdqd%FtI`<^iRi0n2GhME(e_n#>x;SVB+>R_vWkAHOAy}GCOD!& zT^n4l>eK$R+V;!S!ic`wu2fLVEKEAcb^~>E-d_-XwOw^huw0^d<&oTM=e8D(H487W z2W?IlWI4~BE$N6uMgmIhEKtC?#SYu^O(D^SEp~@)sYDyL*geh-#?T$wMDk-h_bK>o z5>=BI2j$89%CLE1Nfhq1;*V{w1J`Q94ELwqgp35#W$8uSVZZH1sl*-j+eO8xL@)Q- zC6{t9QB^bCZX03)l?(&%57@q143LO_z;0c{OBWJL_kdk| zNH1M7c$Ya~mmJbdS9_RspQO8twh-zdt>&c`U{pl%3p@8UV9d3ut{$B9UK|t`<`joT z`S=UlQ;V;HQG97zF3(oTNIjxkE-0>M^0PLnG)Y;j6|&BIqd5sx@PW1oly)=rN%-cQP3JC{g^F zP@B8<(A3dFZ|;?`Du-B)3hbD)Oe&d6jcEub0MxjwAw;G|=-#0;D%S|~j*`vEgcd_#GjT92_#5N6GbyL2{L#{ritPinuPX1qBaGgErgm*AZSF| zT9nw#6nHq%l587ex=F}7JtVk}J!Yl|E7Ia<*Gy5Vb_-lZu$iKB7ak!m5T9>tp zlN<87)n)Bs4`F6eBj}+|_0+R5EohM3;}H`6|{PjZQmDpz1)Lm%m3e;UllSI{ZQ1?Z1Tcy-K(#~>F^~H3hL89u5w9;-3EC*9yvB<5!;<&Qv z%0WX1Jv=07(kZ1!y{Ija1b3ORx(ENp zWl1H|JB7D*!X=bkCaQbtl5~@Lfk)g!ToM=SlcLVygPkq>k1nY^8Zn97Yr^UoOh>Lt z8Zw#RIh<00|3=Z*M4!IFWSUD=kz6fu?-cIrLOc9w;i-|S;s@v&fkc~@IAkQCRKf&> z_-oRs4vF|{XsQ$4Do#q*(z;PqQ5H&NLbwRnYlTkgAo*&f*vHTawN~itpo*8^Iui;_ zhf)tATtujKCKTaWCnA-B;VL4n6FLK{;+eKySTeYq`-6*673DWn{btA?9{lZ~hyMJl zhXyD2cMGqtUN2m=W>ATXm8T(T%54yM>iv>qV8(4o_IC^Ks@@l5iC0}@HV387yp zNU9{@gwU@P9W8ntI!R9vnmdrJZVO^{YwGMt(N1kgxQgf}g?`1;cXJm{{ZbIR@e&H|j z@hC^XU3TU>k9Kq-3K>bLF^)<^q0!FKj?Vueaof?3&i^2hU5s|h$2ckxC4;k`(N3E& zj!Hzehrxc-(TQm1Sd~Mp#~ht;=wiTM*4Y3|KZR1jC+Y?y9x^0w&jd!f( zJX7!}WTl>BP-xkBr-eGqf<(0O4yN|MVI;alQcQAk)03+6!e7Rd98c|PU3erWQ*UYR zK}G^fC52FkIN9+-zB?o$PIj89!=Ek;Dlt`27D{cyxX991k1t46O&ooI5qII8c8aNs zjzFbtqK-^ShaY88cM1(Z&AKkU(@r&2aS=+rzvCjRPEAJ#lCNTPR8`t(r#V)v_;sN) z3R3ZdLfvU8KSx9>-AXybKCeYaM-L0g0-$4vs8n33f%^>*QuT zY6*5l$?Il1p4wi!@>V>{LH0@fA(0O|`bh_c!n2$rwFyBY>?{XQx;qiJTO^-$a&wbQ zF9`3r|7l0Rt#;!9eI~^N840LsxD!wa_>AMLDFTUr&!k<1#Cm?lsl1xkGfB0cpK;n= z75oLaJO^3yU#S0v-3-_&iB%gP33w7VRBl`1>J`C>8XBICwIyHhsg6pWnP(T6a z3r?umL!#;nPFuwj5)FI7LE2bP+enW{E^%_NI{ESiR@Dtdewz%P60|Hd{YhAvkC!-l zkM6-eUz+wDG7?Z~kA}hxmO6TmhQtk)Iy$X~M9-HxdXFZldcM@rdvp(;?8_)Py&pkF z5~|+OFKRFlf0?5*3rNIY=IHZnNNfwsoTy%J3uN$?wajT%uXhFQVI{nh?h4vMr~>T@ zG!J|5i`epX9`?AK_5etg8{sONvD~RpGX@fMm(z@)iFj@#S2($qda%w#$rX-%7M#oQ z-gJ<^3`YMTBk^W3{HNjXI=|_7YH`6tv^SmNl3*d7BvnW9Jty}OM&(Xbqq0|SLLod}SqyCt0n)x5cEHC1sDDs3@sE?Z5DX>-|XT1*M^ zHf%0xE3Kx@1yZGeP^i1j@zpsIBbD3;MB45|X`^Vb z+3x7Hy&4Y)l-@x-xE2FRTI@*HT^3$fy~FX-Iudnum^v;BeMp3E<0>vxa`ENi?L9wq ziqw8yZMmrOq0{Xgo`uzxhX(JZ1}~%30UENf^78Qfe7x5Q)#C~-n!4BN$nSJi2r>3L zcpIacTy6Qd?LOM|Y3qYzmEO-s)qPGIwN2t8s_t_-bP8UlOj1Phkdr%#srMo_guz1x z{T8pMSN{Itdmk9`aN&$S< zWLi*CxasO}LH@39(#JthUNOhh`3D&ZDD_GPg_|9BeD(GP2`0xK{r1(9SIkMO+66cv zkp()*tpmfZ@yYaA3Kw-wrq9xzyi!h?+oAbTYL?(4LYzvcJ0z-}N~e2IUMb(2s<;TH z9xS+ss^2=k+BjlpzKorNM!qK>XME@2{gg%?5?QIEPePzj_dCZ``yC{ze&^`ZjGh)f zhQCj@KS);Uu>pmu-=~jZNL2maIj33oU=7V0Cz6lI+~~r{Yr~)AJ5bq2q{G_+jwXoh zaEU=5fC&N#m3;E^uuo_FH?RzsddMA~9-}1Q&&Jbv12j%alZ&nk$Cco}QFN5-+ASDM z-zs`V@+p~{S=f7BIL3;flKL6liywXyCH4mDKP0|AqALBl3NB7fhTjyPS3Ongb7GVsSLzY!#jmrq61xe`c+edTZf$b#rtt2# zRu-x41174}%5&8Rj9xsb(@m!^s8HEObZWY+P@gkU7FDOq9_njXFH55E3~Hc4iP-_A zvQAt?)frNMrz(a<-5C>q!*2&_5{4z#zH0@h;vg8_0skDO@*U14FHMC+bcJbBvE_7}* zC+$>k8f2)e(u*wh+QLOtog?+qfWgKAf`6l{#gEj09Bw^djKSm7$t!xQM!QWhXw`6Og#eT-mpuE=*l`Htj+m>H?I? z_MlkmG!)XtE9bFvDf%cnH zs8!IH*X}Aw*##w_(2;~(D;?$yjZUwUI{$=3z*SP`pO9$VDjBWK+D0es_!MU41wN&N^!anycG;t{BZK%kp%$u=9ZZVJ`Sw`9eJ ztefnio9j(C*+Qs)&=9DGP|~&0Gz8BMs7+Gm`+fOQvr+046cV*JN~EBwAr2a{QI>7W z8seyiY?S4jvWBpShP-VW!WKfkLk*$j+7C&#%zdDjYrlN^NW3TY8&5x82HUBT2Pq4K zMk-jy-xc1MzkGYTKf^`9?dkpui9y&di_{>HR109cEWS8cK!=L`Be_%N_9oBY9nOz; zO8ujd{(QGx>A*uq0_qOt1%-jxl@1If?zxKwhG;_Kp1Wk5+j(OoNwW`5#V>d&Xa_8I z$&1vlZ?r`l<1Ts4Z8{F^D!WY+a8IaTq!(H0A?eS{b2kl%Y65MHyJgvJx(RA=!YRL1 zH-Q~AVYj^Sip+}D4BaiSx<$vKlw*%+0xCe=lwL#=_M}@+e_n-q(ya#)P1qw_->93Q z)*zgs8}$9s6}U(CSHsi)a{eYpi)4?y^m?8W10uOk=04F=VgT;AFMW6o;4%7GBD4B; zObEzGe4Ir0g!fc`EM0YE1ryOemU>YR;O*`Jb%>tVkoeY~%(y4KyZwQ5IANm70UFL1 zXo8%FgIt;WOeST&3ZIJ)OHXC==keW-$hazuI2=VX4kly6C_$t<7!Tj zXm*XOIgwPH;u=?j(jM2+pw8zp9%F8ZJ3y&*gNrORCrDHs<7!TjxZxPL%WVvWq*@VU z+^)AW=kxi|bmLez?{E$VjT2EfuTmr^L?7!)_2LVO=wn^|;(Ifm*Yoj|mpKfAtabCxxmJ0Sue$0__uej2|x94{~4~` zmfD|Mh&%Ma8d`YE zp6lA(8d?F8FW2tg&Y#Lod1A?$o4cV!h=4=3z|=KVp+FyVmO&sk~rPFT4Kh87cNaYLOwu z79uY;q}T$fC9a)3^gtNJ%M2X8=A!&71`d0GQ*Yp~1vtwM9JT=GRRf1DzUnB_>2i*ZTY7_z$54qx4hC(P5hurX3hJrny_|%|a3y}{S6l?*-kp?K(BNRss z3ig2FGt(osfZ}t5f-Ru|5Lq}kB$j(E5y z`6UZImuSXtuLU1ak_3q1UY~&)gan8Yo_IV1Ar*)bUif$hf<1s3>1F!_TZlZ$%k~Mj z0AjQ!ll?=(mhq!rwm0xKbpA0f+Z)(I=WD!dZ(s|ZALF50>RHAXIzP5iMRa}~l^V#M zN1im^vu1d9`2|%22M>B^=x+uO8k#)(hw#F9yeIjS9lA~fGTv*&$2}xLV!U_$Et-T2 z5)(WzJtN^N5)-^|dPaghka)_FU<;8a8WL=Q#3aw`nCwvtCL1d3fl6XJ#}=qeF;v(B zm8pgbTc9$nQAJRxrBW9&734LuFeRQ(7XC5(X*|nQi4%J6p&7G094xD}gU-@tc|9)G zAaEWpXM5tg41}jZ%=W_PG7#(mM4bV_79!6vAlL$kryGD^k3c+QK(Gf8b4`EP0*Ge~ z2(|!Xo&muYK+JDc5g-;&scRU7AIXIPu{=5QKjGE!LQgi;J@?U!gTf-Qhp)Bpr~1Y)rP!5%;?G5ui+AeI^s zYyreF1A;Apc%@NAfGAL@n;Aq9$=3j4ebVR8;Z5;tUbb5XXvS+^wp#`O@tT+ImH|Ml z@Wfjgh(Li@;e~HyAlL(l*9{1^5cv%Qf-Qh}vjGV92*gSQf<1s(W%|PwK&&<(*aC<( z1_WCGv9?h~fLKSR9%K+;c3(1_WD(yvcxI3n1RbZn?RSjN7$9r`$leAsgFOPV&&&+=0OBLl zAGQGEV*`RMfY@(9umunY8dU^{PpH(b45BIhvdptic*(tg3vZ4Ods%AL6zw?dWvNwD zkT~oOxI#~jrXX>|6UQ?WO%;hFUU)ns!5&B)H6++Ve+?199W zrbBFj#8-v{TOe`FkYEcWzHU?zB)*|i_}q)Rk2BpxsYnX=>F>f%w)7|L&)$6?*{jodLlXB7bi{umuo5GyuUK zf%wsYU=JWpoBpr`5N8YswgBR+0l^kP4D++Jge`!0#0LnKV>ZW;i|t$ElhZMB%#nTu zqB)u|($7FN2Z)h=2BJAYjPk{k8HnZz#3(;}G6TULK#caY)QT-ce$>xWE4BdQF<&NI zh9R}8@w3#5uc7l}Odr@n=g0b4YQ+{hKaRSkQY*I5`NtboMCYHNQZv=NMvF*J^sT3T zyL?=}>dL|QJ}?MB_}wAH=E*~kgpb4%eaT-NF)!$E11I{Gm+E=Z0yrl6*A3D0q6KhF z^2MACM+=2xk{`~=aIgm)lMN2G5IHe8*aD6z!12)E@s2*t$GM?8z~E~%KWhDK?_>{% zW|*F`1w=CqBDR2NmO;c85Y28>5s2!jRIPqTZyCvFeQR+N3=gNp&-#+TvV%xVwBuR7 zf`2$f5=5T$uYEuhX$d0pe6c7a(ozwb=ZA|jBJ6?4d_#mSL|$NsumvK|fk?*@m>|!a z3BuRV`xi_n*h23Yn%=X8-oI#i&lYV4(ado4bpty#LjTXGywueLvX@$0K@-O3G;n7uq+vE=%q^DFXka^b^J2En@6q$GZa7RXlJ%IVA z0mBw<^qv9}Q^VQA-QG83*dj8U4H@=8W{c?`TOhO5kYNjCwwdY07Qk$8R1siWe?Y~) zQmd3(U9GK$MQep(j~^b* zP_PFSdkqS<5P6@OBW!`hM?SsJsnb*T`1EwYpCxX}MCAi!^I;Es4w`PV1wMxiAGW~f zQ(so={h2NBIqb`xnblUEo*p4S->db9KcN5Gw;l=X^21j0_>Dn%;df)hyYlhZzT{tT zU?_{wq_6#U={{5hQeXSmrsvE>AoYzeh6U*pyGW7x#t(-D=@XkhkUDNiv4zMd3@Nri z>SO~_>@lfteXpP1rj?1x-x*Tufz%I%6k8zmqano>NS!vM*aE3DhE$O#Q>4xksizew z{JHw5z?xLp@p#w)zabB@eYzM885LAqpl4Ds_>2m!__dx%#o#kK5EC;##fs19Ae@-- zVGn#B4YCJ0TZsHvkUhxR0-u_|%p~@h&)6WFNyo6buh5IXkwGJ!oUHrj# zpZl=X#KiCoD}KeGDn*-K3Dlpb18FHxy%OB^7d@3qfvOOQwHc~Xg{lyQYco{r0ad+0 z#TFtjH>lVGs#hDJVvkX+FjGmHsQkJ?#U4<-X;85RR4WZCwt#AtLB$qOtv0AiMX6e9 zYY3Hkoy4EkuMey($&LhXlk0=*2(ApxSRZ6ZaAlygKFE&X%0TBIf!LhUDN}U*5rmsF zI_!bY21AD}MBZrVumw7AGaZ#0vPVy^cY-W?W)C1X84zp%#JdIrTLAG->Y>V>*#e07 z8db#8>wPMvULEj9_}lP%&5x4vrsCCMTacZ~mZKTlg6veb93Zv@*{N(fKx_}hz6?aU z0^_hlg11Bed{2(}P;hXKJBK>Vu#2=)lXhXw?D0I}2bhb@5EWk9e65W5Wswg6&J zqly5rmr7mFO9%f}z<&Id^p`>M#I*3|@%|utt!{&M><_Zn>NX&;KN#?Hy}%G3<ccjEvj>A)zR$DB!XqP{=BuZQJdJg6#_0ZR^)dHIz_eS>;gB>eus4 zf}$v^+Q?M_1zbRoeYq&^sHi9^iU^9J*b0KQvMDGs?>P~fQB(8GAKee{$%vCDezBd1 zyyXWA&YHFAtZ@^v6RZjL1Tn#xASbvJyb1p5VA-wK)=h;Kw+7EQ%kD4KP6~>%Euvw8 zKe}>EWzCqX+L~5QF59h1-~8u=pd{aJN)U7$nmjt+dv>sMd)8{29cs#VcqJQj%wIVr zXj^z{agwEx z$}tsH)pfbr8*&v8!>cPsSB}Vyx$Wl4u64Oe3cW4|Q|`vfo5$p8Ylc_mDuUcCu#Bo5 zTUVK@s$*9b)X-erm|B#Cr_q(eYe$cWbKTCnaA>Y~@7`yf-Ya)C-dI`l_sZ(pn=5l) zjGK|WWmIiX{9o1ZQ9-VD^vH^us{g7W{$#8e!y)MBxiJ-^M^=s@RawJzD{4j{+))*E zqv&(%8&x}cOy%gFx#6`nH&l%rOPz~<(ookh6oKm6;Wt)}2+-BT zM^*j3GB>KWdISbVP4#UYblPzQo^TWTG|(&p{lz-HyAG5X-ELc5q=lmk(F^GG1<1f;B#;x9ZAS(c7E{jU{^fZ z_8TV!Z7SHbS#X-F=TX6Enk|AT@Tj6U(QiVw=F@CDXcqX14Pi~OMU(m_xSDEPPdC&CDJ2VGDv&RQV?9C^g><25|OyqwgT-P z(Lhxs4<*U4OCs^Ed+n~}iM$Z5?zMaONi<$|uif{r`ehbg7O}s6f6*mu_?u~4Wx)p) zw*~cQ7OH*o6I(PN_Bw^w=?aY*K4;n+%|>VnRfYgx`cq+S+>l{;3QRn`q^ePLQ^jq$(G^v7 zSXlIukQ-B*OICD@NQ}|i8oeakQaiSK1P1yoaFP_^wXdpXcr}(=EaHE=4Kty#;_vit zEFE>Zu~>D+RSvJ9#hRBIHJImSLkP^Wk9CliOE}@> zdv20^DcW-A+#)gZnJI`yVxTp}Ehp zdt4hV5ST@Tp*KywLVf;}^h= z_bs%e<~(H}F=ZCoEuPj>h73Gq7TQHm>nWpc=>4aXDWgqrwTPw+O)PD~pjd1RH8@*Z zSqzFLwiO3K6AmgHNyz24ql3w!_!2wh1rZ)-S}n1A^iH0_{}Q`je;sNTp)RopEKftF zi4~FU%;jmQY=fJnCa$bktmecrTQmzk%K~C?B-h%RXC62i?u^&kUQd2uv6Vr%g=AVn zvf`c+P*@QPcGxT_i{ZM%0M%!lc1rsb~{ZB$S_g6=eiAz7EH6$%pG->|W*z2_j{ zl1P4JXSNrnJK?)l{E_Xc-YwyPw@?o z@Qg&0N+Y?~&U}N7Ymv<5=PgGXThhpFT=Qwc-(IFTU~-6o*?HmTU(>oCDchfR?`&*_&;I$ zY6?KY{|UQA5zkLZ%+C{c%aeM3l7W|!6L#@QJwLS#W9_G8DbXgl`kAKeYNX<#ERqvM z=5CQKeZtCJR9SenY1kthPY|A(ZDma1PGPw`JRu_irNV+jly?fR5jP4FuJ075sg1qN zqBh=Ts-i4Zu`Wvuyi0frAj+cZT|xuApvAjMwJ0-OEsz<~cxW4Aj7i9X?gu=_BC=Y9Wl3?wwOX`Ms{@|G*=o_b3wH}7;#w`b z7SgylJ0x4TLK+v_;ARa4K|49sQyb>mrb09i7Sfb#7s(ey=C#7y)?wFd{DSaQ1nsz6 zUQD_LG7?Z~K0(3ni%GXYBD@!qZfVERUouru7E0yRc*s(xEs&`ClF%DM+>WQ>%cd^4 zLa7BE4^j7J;j1|YiMlU~vNp`Don>PdzhbK5A(Wc7c!;X6BpnUOPch-BciLG3RW}GL zR>uNpmPuVo?KYUYl-X^dL89t9sQapUtx{_7XlFU7`f4)0AW`*InqEV3F=0pY4UyS` z)$qdHg*bcF)#i)u3r@=qYa8~voZ@{$IJ~nvHUocCV5i2uO)?TtoKKMgr8h;dGbkWP zBD_i9O#VkMODg|i+whuBc!ZLhgg#N*%yYBAMmCyD;^F4}W$nV6&Ui#ZsM831QaJAm zt5-0aPPEXF`Q7cp2aEA<6n$Ux=^M91pjh!ji%D+#WpKnZK_?_^0xn z!d3GGWymT@6Av`ZcF`VnEmy%P+m&zKF}$vPm+;hXp5t@FE>WW5&sj9OKBv)j9akZh zpWiY3&)nxiA5&2VZ~0u{n94bD`$&E*GT-Lgc0v;TbuyvbGylE9YRM2GBLSrdL!qgA zMX}0ZAmMSZ=%iBQ_KbL6GQ=T~(FjXzrcf;P(H#<1_X!-`sq5nQ%=8;m7r>!Z>+leD zze%PCB)c}9TA+qhSv;Pi z?g7zCjUY(WJs>)C435wlqC+GPiOdg}mjBA;@|#xR5_D*n@Wc3!@Y?h0-ND0deppz? zu+&nZ(C~Lycl>W>#8UV#o2N< z(ScDP6$nBXM%|;LSrlxcdrlF_QzCPRQ+TIqcxg7KHlD(Oqq2_R*N$`4>?0WoDDG}j z=#_C!JDvuR)T84abP_E#kgU{71ckEW9W|fuke=S|Slm{UtT5$s{8pt~mjwrCY3LZq zNls>}lP%q8+h3)vzE|ucl^CaoVXR2~XjDilZ-1AmM$A z({pIh6gN;LC;BC1DZdKdEYT^F)1Ax%PL^&-MJKqL?&##c6E|uGHEJ1MsX#{p^4!w7 z7T+)zTK5dQWY3@B=nNbM@a`E7GH{9v67QbjwEQh^<0MtfW;n&a3APg1&d3*?%)-JQ z<>A%Yc($Vtlb!j^bLh>Kc0fiFYM!IgP-w(C$I-bhBwjnm(b*p)(u_Gy={!fJp=99n zXO7c)o}mpJ;qBJRS= z?^0724S`BxqJ}I@x*ugxcPVv0jk+$p{4O(9@eoR#-0=`qmnFRe$xqQcswyqN%N;9L z{JKy+1*!N!q3-g8A0+B7Cw{7~gSsmmeMQlQwqz)E8wQ1{D;#}80g0+B9NbXQ6zqxw z*vYKbQ?M&aUb))w)I!^p7v(h$(omZFkVuRjz5PL<@EWH`EkclRyT-xx_XoJ`7RhIv z%zCU^L$GMA4O-ar%%H3jOkl0j2U&C?st53fdq-V`yHJ!mQw;H4kWV# zl9gJNpjhfd1|&QkaH1qAT80idI?XS~-ht8wsRcvOk)*}J{K>P!-<2P9{G^UVor9*1 zi$dQKps(k0@PYUIhhk%b#z-uV0fQBsm`0H?BHa_Zv zYA1sgp&oTQ@<&uE1RqBo9M)(gms>twdrYT#Bv~cz^HKGf(^{>dc!;XUoDQ9W=PBKE zB6-rujF;KcrC1B_5uH9CURYH(yt=lou=F=!$81cLPC5>^*l~EU{X$76rInEQW~wI* zDU6U`XqF%@NE%(I9Gvk~S*19IriUnd%27EpB#owVQe|}{Ss|fr&7PEQ-yzs&W5V=` zHM&3JVol<3WZQ-iqxBHFP)(o8%=-RaU*s~jY%PM7+g zqnAbd^1XyaRmBj8Qu_oRqUya;@5_*=dapdaN%vp_jTITmyv)ogJbyv>n`{S(vU%z7 zV&F(-`;bIe55Ta1jKo9vHiN=Go$+sAc}VJxmOLnCOPu2Fqe}>AT$<+V28C0L@oyBJ zExUFLrqH*FGa~tv%&aaHUldNZ;-{qEs?XrvZjr=_K#hRJhc1aj5y63i!$nfhYDhF< zk<_#L3_d(8PTCKN4`R|%=Qb#oW$K~|5>*#V3_5aLbO!hK66&(w^J@|7lKh!N!fVTy zNS*Ve4C*YAI_Ez_WnWaKKUcxSW%>0(!ZXX4NqudMGGtA9=nOveu8>$maPfoYpmSH` z+g=i0AFq%_>N_NisIo%#RNpSn;7(m>8ih`UN*|(8D`lDbl7X_Qx>EK~U%Sq*Bnq!e zh9M*?m4-s0>ME(fQ?-Of-Bl8IC%E_$y(76+W}eA6`CWKbyjJSZbiH{B7NoxYhs2jR ziEll`5DFd((o<_WBpenJT98=$3bN&3KAn-ANUK#r7GKPnm9`+_f*f#meN)WE&bdgx zg)WoUC5`G$oeVW7dB{@FEj&ckby80)NQAIXwz^Pz=j?(@lw5$(W{EzLd{$t)}5x-d22xugkw zs0mOiy@O(@vr$O9IXy=1{U)UoeIxmf%zRn+_od-rJANmbHhsBgHYYwIaW^P+S_cK6 zo25RLLn55bQlH8pfn~GQr*e`it=la1sk|>QpIgZJUi!ifjT}i@ALZ9x8GapaN!Bvh z;dl%2rV}6}^0_Ut)kjjDPRYRe+!k5#kyNKsZNqf?KT&p~ik+ zp0V2`WfGKZLPrvEhjf@SG#b54>Rb~N4!232YeFKnZ8F-C#zr^q_`%9H*?dPD8`}`u z$0jy5!PRyt)Zs7g%L~CLhCQA_{Z@xT%l#)(ACgg)pXy^6B-noQTtQc_*56*iyF$e-jz0!e%63yek#j$rOjj; znz_?7lTC2-8FhgQgc7XXCJzP6B4y|OQf4B5C?(mmL+@AKpYjw zZdtk~4TNn73NvCggO)wAsm(^!*mF0a>7zNRENM8 zgm74%bwO&1)d)Q-2VJH8P$GWBgn$ZAS0oP+!jWX@>Cdz9NV4=mB7`Hd#pOB#H3wlS zx=g<>nSn=Sf7Lzx&*yJsG)a!gb1&r~F(8u1WacM5BnIG($C52;0Qb?4lu!L91_Wdz ze#~ETZTRQ%AEm2qu3&_>AElm@19-VRK@Fl^8WP{y^INYCuWx@M=}s6?01Y1)$%oy{({8r(gCIAovbu8kn7YF8>%yy@_+i%@ zi1*rJpc~0qZuU`^mV6fyJ3p>DwTZ^!K;|?z2^$jEkO?O!ggrOm1c?adCY&G<_S}RM zNyRCin}E_buBAZ@@5)14?w{)><>9}|=!7}!gYn{&Xi^@#&TO(z5k+)zimFR_~Zf1VS z36CVqOI@^jA$ED0K_|!+ZuTYDD$QqegKni?rRL`h4`;@bP~U_EV2x|X zWkK^GOp3W#43>f`2H=OML25+hb*|`LzcPC5S=a7U--N^Tde`n&-xMI*a_#Q*O<^RS zckLdjsRV6k6p0sHyLYftp+Z9Rnw$MFe??{ZY5bb&@D~aMLua1X+-`r+cxWKK=Kift z1n)TZ4j4@O;pqU=vXMyB-R*5sL2&3bp~o_ohW`0*XTh1)G54utC8lpg7W?B2XNq zQXABmK#QE8UF&Wy-}A=s<@je;^7@FjKXLrEm8ihc`ldhQPC8I(p z6erzqqL+*cwgJU21_hho`IJGyCZHJSnNh(uLNVS;M+Mt};&v|`6>I{EJG^vMun8z8 zcYn*idU1_YY`Vn&0C0Pz5o`U8W&CDcOz@kIWjn(+7WL!RUl7FsS5#zS6HzP=|3 z5D$5M25S%!ARhL_<0%NKKs@Y)kEbBm1`x9h2sXjxB#)x6Dv~?t^%>p3s}> ze5W^qPV_4Pw3_bA*LW&_B8kY?c=$PkehMaQydLLjCb)u^Yd!H&%EVJl)_UPfDHFEA zYr!yK6FjdoOuU$?0?IR$MGD`S}`z4-l_(=P2jMx-Yj7ofq27!U>iWZY1+dkfOyM*U=u)WG9cIl5N|iA z2oUd3sVkX75Xtv3OLpd~>cT7H_szN)AdL6Tx)}h(`)1t?0Ah+QTM*_}GA86F_V?AlL*DI~r63h)<|g z4TA_Hxf>w?GfN*2(;@{~rD+{@ws-gfPB0KpFwa_XbEK06FA| zpHm=>6p%w+_;U(`Z2)rE0AUk6A2C4K1RzK2jSRLCh+}4Cuni!7FzsOzK>TPxun8cJ z8xU*)h!YJe0>n>LYAAzfOusnutjT_U)-B`J z4PneOAWj2_Sq8*u05RJaPo^MFQy^yh;gcx{wgJQ(1AvCV+U%fM63qJl>!pKs-UE@YnA&B%0zdw#c`h$xr)7I454@Oa978 z4-m&9zs zb*aHNKD96RagC@hHIxyRSNQ1~$~NFxWm?H5;8|_(unBn97(8qOp0)KWwo{iH1u8a6 z@o5&x=X~o8-!7empV|J;>|37{KKtJx{)%_AFZtItXgmGs?`FT0KM0aU zu$%qg@`p*1AhX%O=+AmcMfmdjt}oi)qmHDfqP^DY-VBfBd%oC_f zZ1FVuy7ayw$K~K|iy_A*1oMHPCar8Da330QYy-Hhru}RJxQ`4tHUZo=1#Tc+*0Bxb zK5kGGiGpWa^GNRTt$$-Q9kcV7b_&Z16UK-C%Eo(q$v<|4r{?gr$8UY6hSVHU z@A1#$pXt$4Al%~*uF!+4IT(HEi-Ren=8Dmmet0lt#5N%MmqElPyyz>1D5fT`4R8C} zFk%xi+G`lG4UG1gma++qzA=p01V;PK;9?UH9cWMyh+2G0#lBNBu0TPPGq{BV4b?AB}pilYVvo8bAF z87*uAi682%NNnR3>A0U}b;^j!C(Me(Ht_k`G?Pu>bJFl(6Zrh%%W}Pivk81o`Lb7P z()Wnt0`NJZW*`1D26qQmzA)~t@V0DxcOd!K9O%Fz1af!KE?I|)z~}B@NOBEb1U~l! zVrI&xNb$KR2xq2z*akk6f;5|96Fg52(rkuJ;4>wlY(}lvY-2vt0Eu`T+5HO!2{AsLu|p#f7H#gdOlB z^B`TPTOyFzLD_%k@zWA~W(ODiRga&R;4>!>i&8!<6`wglxG3erHt>1G@L>}?KWg}} z34G?(3WhfUz~q~XIR@L6E^unBw?8a^!(K2H&!=kYBC z!vuePzA~_0E)1C*K9`MG1~R!cC`KSFgZAfZKE>d(GPo?+@r%J{RUlqU`4lTYtAg;Q zln>j$XSLzOCU{Ox) z;Pa8;!zS?AX85oPd_FdOT8UOFBic@U)UgtOU%xZ3_T`)2k5lE&Aib3mty~G{d=`kkDV-8U=d&Q(o6=z$=MX@3fcUCGMQl`GQz><`j=+ERcx5Qt+bh*AaOKoA~FL9h)VzBM4&1kVQz2sQ!4zw3cu8-e)F zfM6Ryd~e#rCV)6(K(GlQ4jT|`0*E6GDgwk&Ds?H(7yP#Zj^i(RdNc>_*unij zhWX|*!)xM+q2%AQVy!4c923JEy{V9nuMdI!-ygwzuHjua{ zOt)q>!Skdr-J0115|cx7(ZDt$F(phV2irhms%a3LKw_F9!6uNHZb+~RB<^ic5hU)T zQuu)h+Kd0*zyo$@&C6Hi@w?6k!gQ5rgD@Tl(^aAkKs*qpt3(@sm>G(>DTp=-#LO_9 zn}T2)K;#VwHo^0Q1_YY`;-OIHFL*F)_Sc6^ixkWK|FZ+)x%b;QzdDRd>yuoRf2Wp00Yk>Q_s@UHuKU zOD^0p@4{gt@*}Jf_6RY;86iiwBfJs*(%_{(S>sj~7X2xBy=8uKVMt9-k}vEW6KpLl z+&nkWfD3T|w+^8ft9X#0aq~&!Dng)(2 zvT{x&zPBtd_$VI;(vgHbZ6&p*gT3+hmftijXj8!!EraH&o<{|v)0POLz@v)ZXulbG ztxj8Z&@%8-&x*tkmT1<{LZ}}t(LzVgqX}oI2~D^Kc~)1`#TYU2to_z$Vbncim6WE^ zBUyHpj-IDRaEbCObqPyE;;dx_IyzbdH7LC(NrqigiF=*3x>l$1N`yLV_3o2u-0ZB? z_t*Mn7H$@CynesZC2YhSXC4?PmpL@3s zYa?t6gdJ&D-xbUg7`x&~K4Ir-?R@zZQI+f)6Lc&bv%=H)4o}#g3R2ADIKc+QIvPjl zNJ36dS~%hCc!KRW=bORDy(idFD;`Ej4C4g5^<-NOBN;r56YQeNwi-t5p+!&GdKk5Z zP!nwoBMr8;(EmxcP{4| zB$S0pZ$k@UP8z982~YVscBG~YBtp%h>GCN}hSEsBZ0D8~uJOX(S@FxZ*OqTl$}?aQ z%>Y_Pkdc7;d3q6dTx9!gQi+g@?6&Pvi5a-auDX^-5>;sqV7wRE?FZ@-G*XLgtBnqh zXiz1qlBKl(cobJJU4jNn*$(43)4PdWws**vAf)0f^4Ch(#O50N- zU&d`;m9`x+5>P+mdqd%FtI`<^iRi0n2GhME(e_n#>x;SVB+>R_vWkAHOAy}GCOD!& zT^n4l>eK$R+V;!S!ic`wu2fLVEKEAcb^~>E-d_-XwOw^huw0^d<&oTM=e8D(H487W z2W?IlWI4~BE$N6uMgmIhEKtC?#SYu^O(D^SEp~@)sYDyL*geh-#?T$wMDk-h_bK>o z5>=BI2j$89%CLE1Nfhq1;*V{w1J`Q94ELwqgp35#W$8uSVZZH1sl*-j+eO8xL@)Q- zC6{t9QB^bCZX03)l?(&%57@q143LO_z;0c{OBWJL_kdk| zNH1M7c$Ya~mmJbdS9_RspQO8twh-zdt>&c`U{pl%3p@8UV9d3ut{$B9UK|t`<`joT z`S=UlQ;V;HQG97zF3(oTNIjxkE-0>M^0PLnG)Y;j6|&BIqd5sx@PW1oly)=rN%-cQP3JC{g^F zP@B8<(A3dFZ|;?`Du-B)3hbD)Oe&d6jcEub0MxjwAw;G|=-#0;D%S|~j*`vEgcd_#GjT92_#5N6GbyL2{L#{ritPinuPX1qBaGgErgm*AZSF| zT9nw#6nHq%l587ex=F}7JtVk}J!Yl|E7Ia<*Gy5Vb_-lZu$iKB7ak!m5T9>tp zlN<87)n)Bs4`F6eBj}+|_0+R5EohM3;}H`6|{PjZQmDpz1)Lm%m3e;UllSI{ZQ1?Z1Tcy-K(#~>F^~H3hL89u5w9;-3EC*9yvB<5!;<&Qv z%0WX1Jv=07(kZ1!y{Ija1b3ORx(ENp zWl1H|JB7D*!X=bkCaQbtl5~@Lfk)g!ToM=SlcLVygPkq>k1nY^8Zn97Yr^UoOh>Lt z8Zw#RIh<00|3=Z*M4!IFWSUD=kz6fu?-cIrLOc9w;i-|S;s@v&fkc~@IAkQCRKf&> z_-oRs4vF|{XsQ$4Do#q*(z;PqQ5H&NLbwRnYlTkgAo*&f*vHTawN~itpo*8^Iui;_ zhf)tATtujKCKTaWCnA-B;VL4n6FLK{;+eKySTeYq`-6*673DWn{btA?9{lZ~hyMJl zhXyD2cMGqtUN2m=W>ATXm8T(T%54yM>iv>qV8(4o_IC^Ks@@l5iC0}@HV387yp zNU9{@gwU@P9W8ntI!R9vnmdrJZVO^{YwGMt(N1kgxQgf}g?`1;cXJm{{ZbIR@e&H|j z@hC^XU3TU>k9Kq-3K>bLF^)<^q0!FKj?Vueaof?3&i^2hU5s|h$2ckxC4;k`(N3E& zj!Hzehrxc-(TQm1Sd~Mp#~ht;=wiTM*4Y3|KZR1jC+Y?y9x^0w&jd!f( zJX7!}WTl>BP-xkBr-eGqf<(0O4yN|MVI;alQcQAk)03+6!e7Rd98c|PU3erWQ*UYR zK}G^fC52FkIN9+-zB?o$PIj89!=Ek;Dlt`27D{cyxX991k1t46O&ooI5qII8c8aNs zjzFbtqK-^ShaY88cM1(Z&AKkU(@r&2aS=+rzvCjRPEAJ#lCNTPR8`t(r#V)v_;sN) z3R3ZdLfvU8KSx9>-AXybKCeYaM-L0g0-$4vs8n33f%^>*QuT zY6*5l$?Il1p4wi!@>V>{LH0@fA(0O|`bh_c!n2$rwFyBY>?{XQx;qiJTO^-$a&wbQ zF9`3r|7l0Rt#;!9eI~^N840LsxD!wa_>AMLDFTUr&!k<1#Cm?lsl1xkGfB0cpK;n= z75oLaJO^3yU#S0v-3-_&iB%gP33w7VRBl`1>J`C>8XBICwIyHhsg6pWnP(T6a z3r?umL!#;nPFuwj5)FI7LE2bP+enW{E^%_NI{ESiR@Dtdewz%P60|Hd{YhAvkC!-l zkM6-eUz+wDG7?Z~kA}hxmO6TmhQtk)Iy$X~M9-HxdXFZldcM@rdvp(;?8_)Py&pkF z5~|+OFKRFlf0?5*3rNIY=IHZnNNfwsoTy%J3uN$?wajT%uXhFQVI{nh?h4vMr~>T@ zG!J|5i`epX9`?AK_5etg8{sONvD~RpGX@fMm(z@)iFj@#S2($qda%w#$rX-%7M#oQ z-gJ<^3`YMTBk^W3{HNjXI=|_7YH`6tv^SmNl3*d7BvnW9Jty}OM&(Xbqq0|SLLod}SqyCt0n)x5cEHC1sDDs3@sE?Z5DX>-|XT1*M^ zHf%0xE3Kx@1yZGeP^i1j@zpsIBbD3;MB45|X`^Vb z+3x7Hy&4Y)l-@x-xE2FRTI@*HT^3$fy~FX-Iudnum^v;BeMp3E<0>vxa`ENi?L9wq ziqw8yZMmrOq0{Xgo`uzxhX(JZ1}~%30UENf^78Qfe7x5Q)#C~-n!4BN$nSJi2r>3L zcpIacTy6Qd?LOM|Y3qYzmEO-s)qPGIwN2t8s_t_-bP8UlOj1Phkdr%#srMo_guz1x z{T8pMSN{Itdmk9`aN&$S< zWLi*CxasO}LH@39(#JthUNOhh`3D&ZDD_GPg_|9BeD(GP2`0xK{r1(9SIkMO+66cv zkp()*tpmfZ@yYaA3Kw-wrq9xzyi!h?+oAbTYL?(4LYzvcJ0z-}N~e2IUMb(2s<;TH z9xS+ss^2=k+BjlpzKorNM!qK>XME@2{gg%?5?QIEPePzj_dCZ``yC{ze&^`ZjGh)f zhQCj@KS);Uu>pmu-=~jZNL2maIj33oU=7V0Cz6lI+~~r{Yr~)AJ5bq2q{G_+jwXoh zaEU=5fC&N#m3;E^uuo_FH?RzsddMA~9-}1Q&&Jbv12j%alZ&nk$Cco}QFN5-+ASDM z-zs`V@+p~{S=f7BIL3;flKL6liywXyCH4mDKP0|AqALBl3NB7fhTjyPS3Ongb7GVsSLzY!#jmrq61xe`c+edTZf$b#rtt2# zRu-x41174}%5&8Rj9xsb(@m!^s8HEObZWY+P@gkU7FDOq9_njXFH55E3~Hc4iP-_A zvQAt?)frNMrz(a<-5C>q!*2&_5{4z#zH0@h;vg8_0skDO@*U14FHMC+bcJbBvE_7}* zC+$>k8f2)e(u*wh+QLOtog?+qfWgKAf`6l{#gEj09Bw^djKSm7$t!xQM!QWhXw`6Og#eT-mpuE=*l`Htj+m>H?I? z_MlkmG!)XtE9bFvDf%cnH zs8!IH*X}Aw*##w_(2;~(D;?$yjZUwUI{$=3z*SP`pO9$VDjBWK+D0es_!MU41wN&N^!anycG;t{BZK%kp%$u=9ZZVJ`Sw`9eJ ztefnio9j(C*+Qs)&=9DGP|~&0Gz8BMs7+Gm`+fOQvr+046cV*JN~EBwAr2a{QI>7W z8seyiY?S4jvWBpShP-VW!WKfkLk*$j+7C&#%zdDjYrlN^NW3TY8&5x82HUBT2Pq4K zMk-jy-xc1MzkGYTKf^`9?dkpui9y&di_{>HR109cEWS8cK!=L`Be_%N_9oBY9nOz; zO8ujd{(QGx>A*uq0_qOt1%-jxl@1If?zxKwhG;_Kp1Wk5+j(OoNwW`5#V>d&Xa_8I z$&1vlZ?r`l<1Ts4Z8{F^D!WY+a8IaTq!(H0A?eS{b2kl%Y65MHyJgvJx(RA=!YRL1 zH-Q~AVYj^Sip+}D4BaiSx<$vKlw*%+0xCe=lwL#=_M}@+e_n-q(ya#)P1qw_->93Q z)*zgs8}$9s6}U(CSHsi)a{eYpi)4?y^m?8W10uOk=04F=VgT;AFMW6o;4%7GBD4B; zObEzGe4Ir0g!fc`EM0YE1ryOemU>YR;O*`Jb%>tVkoeY~%(y4KyZwQ5IANm70UFL1 zXo8%FgIt;WOeST&3ZIJ)OHXC==keW-$hazuI2=VX4kly6C_$t<7!Tj zXm*XOIgwPH;u=?j(jM2+pw8zp9%F8ZJ3y&*gNrORCrDHs<7!TjxZxPL%WVvWq*@VU z+^)AW=kxi|bmLez?{E$VjT2EfuTmr^L?7!)_2LVO=wn^|;(Ifm*Yoj|mpKfAtabCxxmJ0Sue$0__uej2|x94{~4~` zmfD|Mh&%Ma8d`YE zp6lA(8d?F8FW2tg&Y#Lod1A?$o4cV!h=4=3z|=KVp+FyVmO&sk~rPFT4Kh87cNaYLOwu z79uY;q}T$fC9a)3^gtNJ%M2X8=A!&71`d0GQ*Yp~1vtwM9JT=GRRf1DzUnB_>2i*ZTY7_z$54qx4hC(P5hurX3hJrny_|%|a3y}{S6l?*-kp?K(BNRss z3ig2FGt(osfZ}t5f-Ru|5Lq}kB$j(E5y z`6UZImuSXtuLU1ak_3q1UY~&)gan8Yo_IV1Ar*)bUif$hf<1s3>1F!_TZlZ$%k~Mj z0AjQ!ll?=(mhq!rwm0xKbpA0f+Z)(I=WD!dZ(s|ZALF50>RHAXIzP5iMRa}~l^V#M zN1im^vu1d9`2|%22M>B^=x+uO8k#)(hw#F9yeIjS9lA~fGTv*&$2}xLV!U_$Et-T2 z5)(WzJtN^N5)-^|dPaghka)_FU<;8a8WL=Q#3aw`nCwvtCL1d3fl6XJ#}=qeF;v(B zm8pgbTc9$nQAJRxrBW9&734LuFeRQ(7XC5(X*|nQi4%J6p&7G094xD}gU-@tc|9)G zAaEWpXM5tg41}jZ%=W_PG7#(mM4bV_79!6vAlL$kryGD^k3c+QK(Gf8b4`EP0*Ge~ z2(|!Xo&muYK+JDc5g-;&scRU7AIXIPu{=5QKjGE!LQgi;J@?U!gTf-Qhp)Bpr~1Y)rP!5%;?G5ui+AeI^s zYyreF1A;Apc%@NAfGAL@n;Aq9$=3j4ebVR8;Z5;tUbb5XXvS+^wp#`O@tT+ImH|Ml z@Wfjgh(Li@;e~HyAlL(l*9{1^5cv%Qf-Qh}vjGV92*gSQf<1s(W%|PwK&&<(*aC<( z1_WCGv9?h~fLKSR9%K+;c3(1_WD(yvcxI3n1RbZn?RSjN7$9r`$leAsgFOPV&&&+=0OBLl zAGQGEV*`RMfY@(9umunY8dU^{PpH(b45BIhvdptic*(tg3vZ4Ods%AL6zw?dWvNwD zkT~oOxI#~jrXX>|6UQ?WO%;hFUU)ns!5&B)H6++Ve+?199W zrbBFj#8-v{TOe`FkYEcWzHU?zB)*|i_}q)Rk2BpxsYnX=>F>f%w)7|L&)$6?*{jodLlXB7bi{umuo5GyuUK zf%wsYU=JWpoBpr`5N8YswgBR+0l^kP4D++Jge`!0#0LnKV>ZW;i|t$ElhZMB%#nTu zqB)u|($7FN2Z)h=2BJAYjPk{k8HnZz#3(;}G6TULK#caY)QT-ce$>xWE4BdQF<&NI zh9R}8@w3#5uc7l}Odr@n=g0b4YQ+{hKaRSkQY*I5`NtboMCYHNQZv=NMvF*J^sT3T zyL?=}>dL|QJ}?MB_}wAH=E*~kgpb4%eaT-NF)!$E11I{Gm+E=Z0yrl6*A3D0q6KhF z^2MACM+=2xk{`~=aIgm)lMN2G5IHe8*aD6z!12)E@s2*t$GM?8z~E~%KWhDK?_>{% zW|*F`1w=CqBDR2NmO;c85Y28>5s2!jRIPqTZyCvFeQR+N3=gNp&-#+TvV%xVwBuR7 zf`2$f5=5T$uYEuhX$d0pe6c7a(ozwb=ZA|jBJ6?4d_#mSL|$NsumvK|fk?*@m>|!a z3BuRV`xi_n*h23Yn%=X8-oI#i&lYV4(ado4bpty#LjTXGywueLvX@$0K@-O3G;n7uq+vE=%q^DFXka^b^J2En@6q$GZa7RXlJ%IVA z0mBw<^qv9}Q^VQA-QG83*dj8U4H@=8W{c?`TOhO5kYNjCwwdY07Qk$8R1siWe?Y~) zQmd3(U9GK$MQep(j~^b* zP_PFSdkqS<5P6@OBW!`hM?SsJsnb*T`1EwYpCxX}MCAi!^I;Es4w`PV1wMxiAGW~f zQ(so={h2NBIqb`xnblUEo*p4S->db9KcN5Gw;l=X^21j0_>Dn%;df)hyYlhZzT{tT zU?_{wq_6#U={{5hQeXSmrsvE>AoYzeh6U*pyGW7x#t(-D=@XkhkUDNiv4zMd3@Nri z>SO~_>@lfteXpP1rj?1x-x*Tufz%I%6k8zmqano>NS!vM*aE3DhE$O#Q>4xksizew z{JHw5z?xLp@p#w)zabB@eYzM885LAqpl4Ds_>2m!__dx%#o#kK5EC;##fs19Ae@-- zVGn#B4YCJ0TZsHvkUhxR0-u_|%p~@h&)6WFNyo6buh5IXkwGJ!oUHrj# zpZl=X#KiCoD}KeGDn*-K3Dlpb18FHxy%OB^7d@3qfvOOQwHc~Xg{lyQYco{r0ad+0 z#TFtjH>lVGs#hDJVvkX+FjGmHsQkJ?#U4<-X;85RR4WZCwt#AtLB$qOtv0AiMX6e9 zYY3Hkoy4EkuMey($&LhXlk0=*2(ApxSRZ6ZaAlygKFE&X%0TBIf!LhUDN}U*5rmsF zI_!bY21AD}MBZrVumw7AGaZ#0vPVy^cY-W?W)C1X84zp%#JdIrTLAG->Y>V>*#e07 z8db#8>wPMvULEj9_}lP%&5x4vrsCCMTacZ~mZKTlg6veb93Zv@*{N(fKx_}hz6?aU z0^_hlg11Bed{2(}P;hXKJBK>Vu#2=)lXhXw?D0I}2bhb@5EWk9e65W5Wswg6&J zqly5rmr7mFO9%f}z<&Id^p`>M#I*3|@%|utt!{&M><_Zn>NX&;KN#?Hy}%G3<, nothing more. +// Another scheme names an archive no writer exists for yet, which is refused +// at startup rather than left to fail at the first delivery. +func parseAuditArchiveURL(raw string) (auditArchiveLocation, error) { + parsed, err := url.Parse(raw) + if err != nil || parsed.Scheme == "" { + return auditArchiveLocation{}, fmt.Errorf("%s must be a URL like gs://; got %q", envAuditArchiveURL, raw) + } + if parsed.Scheme != auditArchiveSchemeGCS { + return auditArchiveLocation{}, fmt.Errorf("%s scheme %q has no archive writer; the supported scheme is %s:// (got %q)", + envAuditArchiveURL, parsed.Scheme, auditArchiveSchemeGCS, raw) + } + if parsed.User != nil || parsed.Port() != "" || (parsed.Path != "" && parsed.Path != "/") || parsed.RawQuery != "" || parsed.Fragment != "" { + return auditArchiveLocation{}, fmt.Errorf("%s names a bucket and nothing else, like gs://; got %q", envAuditArchiveURL, raw) + } + if !gcsBucketNamePattern.MatchString(parsed.Host) { + return auditArchiveLocation{}, fmt.Errorf("%s bucket %q is not a valid bucket name", envAuditArchiveURL, parsed.Host) + } + return auditArchiveLocation{scheme: parsed.Scheme, bucket: parsed.Host}, nil +} + // auditDeploymentIDPattern keeps the deployment id usable as an archive path // segment and a warehouse value without escaping. var auditDeploymentIDPattern = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$`) @@ -104,21 +140,21 @@ var auditDeploymentIDPattern = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9._-]{0, // setting, reporting all the missing ones at once. func configuredBigQueryAuditSwap() (*bigQueryAuditSwap, error) { swap := &bigQueryAuditSwap{ - project: strings.TrimSpace(os.Getenv(envAuditBigQueryProject)), - dataset: strings.TrimSpace(os.Getenv(envAuditBigQueryDataset)), - archiveBucket: strings.TrimSpace(os.Getenv(envAuditArchiveBucket)), - deploymentID: strings.TrimSpace(os.Getenv(envAuditDeploymentID)), - batchSize: business.DefaultAuditRelayBatchSize, - maxWait: business.DefaultAuditRelayMaxWait, - } - retentionDays := strings.TrimSpace(os.Getenv(envAuditContentDetailRetentionDays)) + project: strings.TrimSpace(os.Getenv(envAuditBigQueryProject)), + dataset: strings.TrimSpace(os.Getenv(envAuditBigQueryDataset)), + deploymentID: strings.TrimSpace(os.Getenv(envAuditDeploymentID)), + batchSize: business.DefaultAuditRelayBatchSize, + maxWait: business.DefaultAuditRelayMaxWait, + } + archiveURL := strings.TrimSpace(os.Getenv(envAuditArchiveURL)) + retentionDays := strings.TrimSpace(os.Getenv(envAuditContentRetentionDays)) var missing []string for _, setting := range []struct{ name, value string }{ {envAuditBigQueryProject, swap.project}, {envAuditBigQueryDataset, swap.dataset}, - {envAuditArchiveBucket, swap.archiveBucket}, + {envAuditArchiveURL, archiveURL}, {envAuditDeploymentID, swap.deploymentID}, - {envAuditContentDetailRetentionDays, retentionDays}, + {envAuditContentRetentionDays, retentionDays}, } { if setting.value == "" { missing = append(missing, setting.name) @@ -130,15 +166,15 @@ func configuredBigQueryAuditSwap() (*bigQueryAuditSwap, error) { days, err := strconv.Atoi(retentionDays) if err != nil || days < 1 { - return nil, fmt.Errorf("%s must be a whole number of days, at least 1; got %q", envAuditContentDetailRetentionDays, retentionDays) + return nil, fmt.Errorf("%s must be a whole number of days, at least 1; got %q", envAuditContentRetentionDays, retentionDays) } swap.contentDetailRetention = time.Duration(days) * 24 * time.Hour if !auditDeploymentIDPattern.MatchString(swap.deploymentID) { return nil, fmt.Errorf("%s must be letters, digits, '.', '_' or '-' (at most 128, starting with a letter or digit); got %q", envAuditDeploymentID, swap.deploymentID) } - if strings.Contains(swap.archiveBucket, "/") { - return nil, fmt.Errorf("%s is a bucket name, not a URL or path; got %q", envAuditArchiveBucket, swap.archiveBucket) + if swap.archive, err = parseAuditArchiveURL(archiveURL); err != nil { + return nil, err } if raw := strings.TrimSpace(os.Getenv(envAuditRelayBatchSize)); raw != "" { size, err := strconv.Atoi(raw) @@ -192,12 +228,15 @@ func newBigQueryAuditRelay(ctx context.Context, types business.DeclaredAuditEven return fail(err) } + if swap.archive.scheme != auditArchiveSchemeGCS { + return fail(fmt.Errorf("audit archive: no writer for scheme %q", swap.archive.scheme)) + } storageClient, err := storage.NewClient(ctx) if err != nil { return fail(fmt.Errorf("audit archive: storage client: %w", err)) } closers = append(closers, func() { _ = storageClient.Close() }) - archive, err := gcsarchive.New(storageClient, swap.archiveBucket) + archive, err := gcsarchive.New(storageClient, swap.archive.bucket) if err != nil { return fail(err) } diff --git a/module/services/accounts/code/pkg/adapters/module_audit_declarations_test.go b/module/services/accounts/code/pkg/adapters/module_audit_declarations_test.go index f6cb96dab..e13dfe5ae 100644 --- a/module/services/accounts/code/pkg/adapters/module_audit_declarations_test.go +++ b/module/services/accounts/code/pkg/adapters/module_audit_declarations_test.go @@ -50,3 +50,36 @@ func TestModuleAuditDeclarations_CarryEveryFieldAttribute(t *testing.T) { t.Errorf("withheld_turns = %#v", f) } } + +// Every wire retention class but UNSPECIFIED maps to the registry's, and +// UNSPECIFIED reads as the registry's default, content. +func TestModuleAuditRetentions_CoverEveryWireClass(t *testing.T) { + for value, name := range gen.ModuleAuditEventRetention_name { + retention := gen.ModuleAuditEventRetention(value) + if retention == gen.ModuleAuditEventRetention_MODULE_AUDIT_EVENT_RETENTION_UNSPECIFIED { + continue + } + if _, ok := moduleAuditRetentions[retention]; !ok { + t.Errorf("%s has no registry class", name) + } + } + out := moduleAuditDeclarations([]*gen.ModuleAuditEventTypeDeclaration{ + {Type: "acme.access.granted", Retention: gen.ModuleAuditEventRetention_MODULE_AUDIT_EVENT_RETENTION_SECURITY}, + {Type: "acme.page.viewed", Retention: gen.ModuleAuditEventRetention_MODULE_AUDIT_EVENT_RETENTION_CONTENT}, + {Type: "acme.page.opened"}, + }) + declared, err := business.ValidateAuditEventTypeDeclarations("acme", out) + if err != nil { + t.Fatalf("validate: %v", err) + } + want := map[business.EventType]business.AuditRetentionClass{ + "acme.access.granted": business.RetentionSecurity, + "acme.page.viewed": business.RetentionContent, + "acme.page.opened": business.RetentionContent, + } + for _, d := range declared { + if d.Retention != want[d.Type] { + t.Errorf("%s retention = %q, want %q", d.Type, d.Retention, want[d.Type]) + } + } +} diff --git a/module/services/accounts/code/pkg/adapters/module_capabilities_server.go b/module/services/accounts/code/pkg/adapters/module_capabilities_server.go index 016c6ba95..4d77ad3c6 100644 --- a/module/services/accounts/code/pkg/adapters/module_capabilities_server.go +++ b/module/services/accounts/code/pkg/adapters/module_capabilities_server.go @@ -409,6 +409,15 @@ var moduleAuditVisibilities = map[gen.ModuleAuditEventVisibility]string{ gen.ModuleAuditEventVisibility_MODULE_AUDIT_EVENT_VISIBILITY_EXTERNAL: business.AuditVisibilityExternal, } +// moduleAuditRetentions maps the wire retention class of a declared type to +// the registry's. UNSPECIFIED has no entry and yields the empty string, which +// the registry resolves to its default (content) — the reading a manifest +// declaration that omits `retention` gets as well. +var moduleAuditRetentions = map[gen.ModuleAuditEventRetention]string{ + gen.ModuleAuditEventRetention_MODULE_AUDIT_EVENT_RETENTION_CONTENT: string(business.RetentionContent), + gen.ModuleAuditEventRetention_MODULE_AUDIT_EVENT_RETENTION_SECURITY: string(business.RetentionSecurity), +} + // moduleAuditDeclarations converts the wire declarations to the registry's // declaration shape, the one a solution's manifest is decoded into as well. func moduleAuditDeclarations(types []*gen.ModuleAuditEventTypeDeclaration) []business.AuditEventTypeDeclaration { @@ -418,6 +427,7 @@ func moduleAuditDeclarations(types []*gen.ModuleAuditEventTypeDeclaration) []bus Type: t.GetType(), Description: t.GetDescription(), Visibility: moduleAuditVisibilities[t.GetVisibility()], + Retention: moduleAuditRetentions[t.GetRetention()], } for _, f := range t.GetFields() { field := business.AuditFieldDeclaration{ diff --git a/module/services/accounts/code/pkg/auditstore/bigquerystore/bigquerystore.go b/module/services/accounts/code/pkg/auditstore/bigquerystore/bigquerystore.go index d109e5c5b..5116b6e36 100644 --- a/module/services/accounts/code/pkg/auditstore/bigquerystore/bigquerystore.go +++ b/module/services/accounts/code/pkg/auditstore/bigquerystore/bigquerystore.go @@ -230,8 +230,7 @@ func conforms(spec tableSpec, existing *bigquery.TableMetadata) error { // AppendAuditBatch implements business.AuditStoreWriter. Every record goes to // the events table; a content-class record's details go to the details table -// instead of the events row. Each row's insert id is its event id, so a batch -// delivered again is collapsed by BigQuery's streaming deduplication. +// instead of the events row. func (s *Store) AppendAuditBatch(ctx context.Context, batch business.AuditBatch) error { var events, details []bigquery.ValueSaver for _, record := range batch.Records { @@ -281,6 +280,13 @@ func (r Row) Save() (map[string]bigquery.Value, string, error) { } // EventRow is a record's events-table row. +// +// Its insert id is the event id. BigQuery collapses a repeated insert id only +// within a short window, so a batch redelivered after a restart can write an +// event twice; that is accepted (ADR 0009), not prevented here: every read of +// these tables returns each event once by event id, as archive readers do. +// Committed write streams with offsets would prevent it, at the cost of +// stream state the relay would have to carry — machinery for no compliance gain. func EventRow(deploymentID string, record business.AuditRecord) Row { entry := record.Entry values := map[string]bigquery.Value{ @@ -308,7 +314,8 @@ func EventRow(deploymentID string, record business.AuditRecord) Row { return Row{InsertID: entry.ID, Values: values} } -// DetailRow is a content-class record's details-table row. +// DetailRow is a content-class record's details-table row, keyed and +// deduplicated on read exactly as EventRow is. func DetailRow(deploymentID string, record business.AuditRecord) Row { entry := record.Entry return Row{InsertID: entry.ID, Values: map[string]bigquery.Value{ diff --git a/module/services/accounts/code/pkg/business/audit_relay.go b/module/services/accounts/code/pkg/business/audit_relay.go index c10dc4189..0408cc719 100644 --- a/module/services/accounts/code/pkg/business/audit_relay.go +++ b/module/services/accounts/code/pkg/business/audit_relay.go @@ -74,12 +74,12 @@ type AuditRelayConfig struct { // the archive (ADR 0009). Delivery is at-least-once: each batch is written to // the archive as one object, then appended to the store, and its queue rows are // deleted only after both writes were acknowledged. A crash anywhere before -// that delete leaves the rows queued and the batch is delivered again — the -// store keeps one record per event id, and the archive's readers deduplicate. +// that delete leaves the rows queued and the batch is delivered again, so both +// the store and the archive may hold an event twice; every read of either +// returns each event once by event id. // // The archive is written first so that a failure between the two writes -// leaves its duplicate in the archive, where readers already deduplicate, and -// not in the store. +// repeats the archive write, not the store's. // // While it runs, the relay remembers which writes the batch at the head of the // queue has already had acknowledged, so retrying it during an outage of one diff --git a/module/services/accounts/code/pkg/business/audit_relay_test.go b/module/services/accounts/code/pkg/business/audit_relay_test.go index 7eeb26abf..f0b33e38b 100644 --- a/module/services/accounts/code/pkg/business/audit_relay_test.go +++ b/module/services/accounts/code/pkg/business/audit_relay_test.go @@ -62,8 +62,9 @@ func (q *memQueue) remaining() []QueuedAuditEvent { return slices.Clone(q.rows) } -// memWarehouse is a fake store of record that keeps one record per event id, -// as BigQuery's insert-id deduplication does for a prompt redelivery. +// memWarehouse is a fake store of record. It keeps every append, duplicates +// included, as BigQuery may after a redelivery; byID is what a read returns, +// one record per event id. type memWarehouse struct { mu sync.Mutex byID map[string]AuditRecord @@ -312,7 +313,7 @@ func TestAuditRelayDeletesOnlyAfterBothWritesAcknowledge(t *testing.T) { }) } -func TestAuditRelayCrashBetweenWriteAndDeleteLosesAndDuplicatesNothingInTheStore(t *testing.T) { +func TestAuditRelayCrashBetweenWriteAndDeleteLosesNothingAndReadsEachEventOnce(t *testing.T) { f := newRelayFixture(t, 5, time.Second, nil) ids := f.enqueue(time.Minute, EventSessionRevoked, repeatOrg(teeOrgID, 5)...) f.queue.crashes = 1 @@ -329,8 +330,8 @@ func TestAuditRelayCrashBetweenWriteAndDeleteLosesAndDuplicatesNothingInTheStore require.Equal(t, 5, delivered) require.Empty(t, f.queue.remaining()) - require.Len(t, f.warehouse.appended, 10, "the batch was appended twice") - require.Len(t, f.warehouse.byID, 5, "and the store keeps one record per event id") + require.Len(t, f.warehouse.appended, 10, "the batch was appended twice: the store may hold an event more than once") + require.Len(t, f.warehouse.byID, 5, "and a read by event id returns each event once") for _, record := range f.warehouse.appended { require.Contains(t, ids, record.Entry.ID) } diff --git a/module/services/accounts/code/pkg/business/audit_retention_test.go b/module/services/accounts/code/pkg/business/audit_retention_test.go index 46d546b14..d236adb33 100644 --- a/module/services/accounts/code/pkg/business/audit_retention_test.go +++ b/module/services/accounts/code/pkg/business/audit_retention_test.go @@ -1,6 +1,7 @@ package business import ( + "context" "testing" "github.com/stretchr/testify/require" @@ -56,7 +57,7 @@ func TestAuditCatalog_RetentionClassExamples(t *testing.T) { } } -func TestDeclaredAuditEventTypesAreContentRetained(t *testing.T) { +func TestDeclaredAuditEventTypesThatSayNothingAreContent(t *testing.T) { declared := declaredType("acme", AuditVisibilityExternal) require.Equal(t, RetentionContent, declared.Definition().Retention) @@ -71,3 +72,67 @@ func TestUnregisteredAuditEventTypesKeepTheirDetails(t *testing.T) { require.False(t, resolved.Registered) require.Equal(t, RetentionSecurity, resolved.RetentionClass()) } + +// A solution's manifest and a module's declaration may each state a retention +// class; saying nothing is content, and anything outside the vocabulary is +// refused rather than defaulted. +func TestDeclaredAuditEventTypesDeclareTheirRetentionClass(t *testing.T) { + declared, err := ParseDeclaredAuditEventTypes("acme", declaringManifest("acme", + `{"name":"granted","type":"acme.access.granted","retention":"security","fields":[]},`+ + `{"name":"viewed","type":"acme.page.viewed","retention":"content","fields":[]},`+ + `{"name":"opened","type":"acme.page.opened","fields":[]}`)) + require.NoError(t, err) + classes := map[EventType]AuditRetentionClass{} + for _, d := range declared { + classes[d.Type] = d.Retention + require.Equal(t, d.Retention, d.Definition().Retention) + } + require.Equal(t, map[EventType]AuditRetentionClass{ + "acme.access.granted": RetentionSecurity, + "acme.page.viewed": RetentionContent, + "acme.page.opened": RetentionContent, + }, classes) + + for _, retention := range []string{"Security", "permanent", "compliance"} { + _, err := ParseDeclaredAuditEventTypes("acme", declaringManifest("acme", + `{"name":"granted","type":"acme.access.granted","retention":"`+retention+`","fields":[]}`)) + require.ErrorIs(t, err, ErrSolutionAuditDeclarationRejected, retention) + require.ErrorContains(t, err, "retention", retention) + } + + module, err := ValidateAuditEventTypeDeclarations("acme", []AuditEventTypeDeclaration{ + {Type: "acme.access.granted", Retention: "security"}, + {Type: "acme.page.viewed"}, + }) + require.NoError(t, err) + require.Equal(t, RetentionSecurity, module[0].Retention) + require.Equal(t, RetentionContent, module[1].Retention) +} + +// The class only grows, like pii: a re-declaration may raise content to +// security, and may not lower security — not even by leaving the class out. +func TestDeclaredAuditRetentionClassOnlyGrows(t *testing.T) { + store := newDeclaredAuditStore() + svc, _ := newModuleDeclaringService(t, store, moduleGrants("acme", "acme")) + caller := ModuleCaller{PrincipalID: ModulePrincipalID("acme")} + declare := func(retention string) error { + _, _, err := svc.ModuleDeclareAuditEventTypes(context.Background(), caller, "acme", + []AuditEventTypeDeclaration{{Type: "acme.access.granted", Retention: retention}}) + return err + } + stored := func() AuditRetentionClass { return store.rows["acme.access.granted"].declared.Retention } + + require.NoError(t, declare("")) + require.Equal(t, RetentionContent, stored()) + + require.NoError(t, declare("security"), "raising the class is allowed") + require.Equal(t, RetentionSecurity, stored()) + + require.Error(t, declare("content"), "lowering it is refused") + require.Error(t, declare(""), "and so is lowering it by omission") + require.Equal(t, RetentionSecurity, stored()) + + puts := store.puts + require.NoError(t, declare("security")) + require.Equal(t, puts, store.puts, "re-declaring the admitted class writes nothing") +} diff --git a/module/services/accounts/code/pkg/business/audit_store.go b/module/services/accounts/code/pkg/business/audit_store.go index a4541d5f2..62fba4e4c 100644 --- a/module/services/accounts/code/pkg/business/audit_store.go +++ b/module/services/accounts/code/pkg/business/audit_store.go @@ -71,8 +71,10 @@ type AuditRecorder interface { type AuditStoreWriter interface { // AppendAuditBatch appends one relay batch, in its order. It is // at-least-once: a batch that failed, or whose queue rows were not deleted - // afterwards, is delivered again, so the store must key each record by its - // event id and keep one record per id. + // afterwards, is delivered again. The store keys every record by its event + // id and may hold an event more than once after a redelivery; every read of + // the store returns each event once by event id, exactly as every reader of + // the archive does. AppendAuditBatch(ctx context.Context, batch AuditBatch) error } diff --git a/module/services/accounts/code/pkg/business/audit_webhook_visibility_test.go b/module/services/accounts/code/pkg/business/audit_webhook_visibility_test.go index 6dfaa34b4..563f9faf8 100644 --- a/module/services/accounts/code/pkg/business/audit_webhook_visibility_test.go +++ b/module/services/accounts/code/pkg/business/audit_webhook_visibility_test.go @@ -76,6 +76,7 @@ func declaredType(namespace, visibility string) *DeclaredAuditEventType { Namespace: namespace, SolutionID: namespace, Visibility: visibility, + Retention: RetentionContent, Fields: []PayloadField{{Name: "count", Kind: FieldInt}}, } } diff --git a/module/services/accounts/code/pkg/business/module_audit_declarations_integration_test.go b/module/services/accounts/code/pkg/business/module_audit_declarations_integration_test.go index d00ccaf3e..21cffc6c8 100644 --- a/module/services/accounts/code/pkg/business/module_audit_declarations_integration_test.go +++ b/module/services/accounts/code/pkg/business/module_audit_declarations_integration_test.go @@ -15,6 +15,7 @@ import ( "google.golang.org/protobuf/types/known/structpb" "accounts/pkg/business" + "accounts/pkg/infra/storetx" ) // Module-declared audit types against Postgres, under the codefly harness: the @@ -173,3 +174,41 @@ func exportPayload(t *testing.T, row map[string]any) map[string]any { t.Fatal(errors.New("export row carries no payload")) return nil } + +// The retention class a module declares is stored on the type's row and read +// back by the one type lookup the relay classifies with; it only grows; and a +// code-owned type's row carries the catalog's class. +func TestModuleAuditDeclarations_RetentionClassIsStoredAndOnlyGrows(t *testing.T) { + clearData(t) + namespace := freshAuditNamespace(t) + prefix := testSolutionID(t) + svc := moduleDeclaringService(t, map[string][]string{prefix: {namespace}}) + caller := business.ModuleCaller{PrincipalID: business.ModulePrincipalID(prefix)} + eventType := business.EventType(namespace + ".access.granted") + declare := func(retention string) error { + _, _, err := svc.ModuleDeclareAuditEventTypes(testCtx, caller, prefix, + []business.AuditEventTypeDeclaration{{Type: string(eventType), Retention: retention}}) + return err + } + stored := func() business.AuditRetentionClass { + t.Helper() + resolved, err := business.NewAuditEventResolver(testStore).Resolve(testCtx, eventType) + require.NoError(t, err) + require.True(t, resolved.Registered) + return resolved.RetentionClass() + } + + require.NoError(t, declare("")) + require.Equal(t, business.RetentionContent, stored(), "saying nothing is content") + require.NoError(t, declare("security")) + require.Equal(t, business.RetentionSecurity, stored(), "the class may rise") + require.Error(t, declare(""), "and may not fall, not even by omission") + require.Equal(t, business.RetentionSecurity, stored()) + + var catalogClass string + require.NoError(t, testStore.WithControlPlane(testCtx, func(ctx context.Context) error { + return storetx.Tx(ctx).QueryRow(ctx, + `SELECT retention_class FROM audit_event_types WHERE name = $1`, string(business.EventAuthLogin)).Scan(&catalogClass) + })) + require.Equal(t, string(business.RetentionSecurity), catalogClass, "the startup projection writes the catalog's class") +} diff --git a/module/services/accounts/code/pkg/business/solution_audit_events.go b/module/services/accounts/code/pkg/business/solution_audit_events.go index 812309f54..4ffb8d6c5 100644 --- a/module/services/accounts/code/pkg/business/solution_audit_events.go +++ b/module/services/accounts/code/pkg/business/solution_audit_events.go @@ -85,6 +85,13 @@ import ( // subscribed, and widening it would start sending outside the platform // facts a tenant's endpoint never agreed to receive under that name. A // producer that needs the other visibility declares another type. +// - A type may declare its retention class (ADR 0009): `content` (the +// default) or `security`, which keeps its full details for the compliance +// window once a warehouse is the audit store of record. Like `pii`, the +// class only ever grows: a re-declaration may raise `content` to +// `security`, and is refused if it would lower `security` to `content` — +// including by omitting the class — because that would shorten how long +// the type's evidence is kept without anyone deciding so. // SolutionAuditOwnerPrefix prefixes the owner of every solution-declared audit // event type, followed by the solution id — the same subject form the @@ -178,7 +185,10 @@ type DeclaredAuditEventType struct { // never empty on a validated declaration, so a zero value reaching a gate // is a type that skipped validation rather than one that opted out. Visibility string - Fields []PayloadField + // Retention is RetentionContent or RetentionSecurity; never empty on a + // validated declaration. + Retention AuditRetentionClass + Fields []PayloadField } // ExternallyDeliverable reports whether an event of this type may be delivered @@ -250,10 +260,7 @@ func (d DeclaredAuditEventType) PayloadSchemaJSON() []byte { // listing uses. It is observational: a module emission is its own operation, // written on the emitter's own transaction. // -// Its retention class is content. A declaration does not yet carry a class of -// its own — the manifest, the module declaration RPC and the audit_event_types -// row would each have to grow one — so every declared type is held to the -// shorter details window, exactly as Durability is fixed above. +// Its retention class is the one the producer declared, content by default. func (d DeclaredAuditEventType) Definition() AuditEventDefinition { return AuditEventDefinition{ Type: d.Type, @@ -263,7 +270,7 @@ func (d DeclaredAuditEventType) Definition() AuditEventDefinition { Owner: SolutionAuditOwner(d.SolutionID), Description: d.Description, Durability: DurabilityObservational, - Retention: RetentionContent, + Retention: d.Retention, Visibility: d.Visibility, // The whole payload the type carries, the host-stamped solution // included, so a declared type validates and redacts exactly as the @@ -290,7 +297,7 @@ type storedDeclaredAuditSchema struct { // DeclaredAuditEventTypeFromSchema reads a declared type back from its stored // row. It inverts PayloadSchemaJSON exactly and refuses anything else, so a row // that was not written by admission never validates a payload. -func DeclaredAuditEventTypeFromSchema(eventType EventType, namespace, owner, visibility string, schema []byte) (DeclaredAuditEventType, error) { +func DeclaredAuditEventTypeFromSchema(eventType EventType, namespace, owner, visibility, retention string, schema []byte) (DeclaredAuditEventType, error) { solutionID, ok := SolutionIDFromAuditOwner(owner) if !ok { return DeclaredAuditEventType{}, fmt.Errorf("audit: event type %q is not owned by a solution", eventType) @@ -302,6 +309,11 @@ func DeclaredAuditEventTypeFromSchema(eventType EventType, namespace, owner, vis if visibility != AuditVisibilityTenant && visibility != AuditVisibilityExternal { return DeclaredAuditEventType{}, fmt.Errorf("audit: event type %q has unreadable visibility %q", eventType, visibility) } + // The same for the retention class: the relay routes the type's details by + // it, and neither default is a scan's to pick. + if retention != string(RetentionContent) && retention != string(RetentionSecurity) { + return DeclaredAuditEventType{}, fmt.Errorf("audit: event type %q has unreadable retention class %q", eventType, retention) + } var stored storedDeclaredAuditSchema if err := json.Unmarshal(schema, &stored); err != nil { return DeclaredAuditEventType{}, fmt.Errorf("audit: event type %q has an unreadable payload schema: %w", eventType, err) @@ -312,6 +324,7 @@ func DeclaredAuditEventTypeFromSchema(eventType EventType, namespace, owner, vis SolutionID: solutionID, Description: stored.Description, Visibility: visibility, + Retention: AuditRetentionClass(retention), } for name, property := range stored.Properties { if name == hostStampedAuditField { @@ -358,6 +371,7 @@ type declaredEventJSON struct { Type string `json:"type"` Description string `json:"description"` Visibility string `json:"visibility"` + Retention string `json:"retention"` Fields json.RawMessage `json:"fields"` } @@ -401,7 +415,7 @@ func ParseDeclaredAuditEventTypes(solutionID, manifest string) ([]DeclaredAuditE } inputs = append(inputs, AuditEventTypeDeclaration{ Type: event.Type, Description: event.Description, - Visibility: event.Visibility, Fields: fields, + Visibility: event.Visibility, Retention: event.Retention, Fields: fields, }) } return ValidateAuditEventTypeDeclarations(solutionID, inputs) @@ -418,7 +432,9 @@ type AuditEventTypeDeclaration struct { // The operator's grant is checked at admission, which is the only place // the declaring principal's MODULE_PRINCIPALS entry is in hand. Visibility string - Fields []AuditFieldDeclaration + // Retention is "content", "security", or empty for the default (content). + Retention string + Fields []AuditFieldDeclaration } // AuditFieldDeclaration is one declared payload field. Kind is a FieldKind's @@ -464,6 +480,10 @@ func ValidateAuditEventTypeDeclarations(solutionID string, inputs []AuditEventTy if err != nil { return nil, err } + retention, err := validateDeclaredAuditRetention(event.Type, event.Retention) + if err != nil { + return nil, err + } fields, err := validateDeclaredAuditFields(event.Type, event.Fields) if err != nil { return nil, err @@ -474,6 +494,7 @@ func ValidateAuditEventTypeDeclarations(solutionID string, inputs []AuditEventTy SolutionID: solutionID, Description: strings.TrimSpace(event.Description), Visibility: visibility, + Retention: retention, Fields: fields, }) } @@ -503,6 +524,22 @@ func validateDeclaredAuditVisibility(eventType, visibility string) (string, erro } } +// validateDeclaredAuditRetention resolves a declaration's retention class. +// Empty is the default and means content; anything else the vocabulary does +// not name is refused rather than defaulted, so a misspelt "Security" is a +// legible refusal instead of a type whose evidence silently expires early. +func validateDeclaredAuditRetention(eventType, retention string) (AuditRetentionClass, error) { + switch AuditRetentionClass(strings.TrimSpace(retention)) { + case "", RetentionContent: + return RetentionContent, nil + case RetentionSecurity: + return RetentionSecurity, nil + default: + return "", declarationRejected("event type %q declares retention %q; it must be %q or %q", + eventType, retention, RetentionContent, RetentionSecurity) + } +} + // decodeDeclaredAuditFields reads a manifest's `fields` array strictly: an // unknown key on a field is refused, not ignored. func decodeDeclaredAuditFields(eventType string, raw json.RawMessage) ([]AuditFieldDeclaration, error) { @@ -633,7 +670,7 @@ func checkAdditiveAuditFieldChange(admitted, declared DeclaredAuditEventType) er } func sameDeclaredAuditEventType(a, b DeclaredAuditEventType) bool { - return bytes.Equal(a.PayloadSchemaJSON(), b.PayloadSchemaJSON()) + return a.Retention == b.Retention && bytes.Equal(a.PayloadSchemaJSON(), b.PayloadSchemaJSON()) } // admitDeclaredAuditEventTypes records a solution's declared types and @@ -743,6 +780,14 @@ func (s *Service) admitDeclaredAuditEventTypesWritten(ctx context.Context, solut "event type %q is admitted with %q visibility and may not become %q; declare another type instead", d.Type, admitted.Visibility, d.Visibility) } + // The retention class only grows, like pii: lowering it would shorten + // how long this type's evidence is kept, and a re-declaration that + // simply omits it must not do that by default. + if admitted.Retention == RetentionSecurity && d.Retention != RetentionSecurity { + return nil, nil, declarationRejected( + "event type %q is admitted with %q retention and may not become %q", + d.Type, RetentionSecurity, d.Retention) + } if err := checkAdditiveAuditFieldChange(*admitted, d); err != nil { return nil, nil, err } diff --git a/module/services/accounts/code/pkg/business/solution_audit_events_test.go b/module/services/accounts/code/pkg/business/solution_audit_events_test.go index ec0938aaf..df1fe728e 100644 --- a/module/services/accounts/code/pkg/business/solution_audit_events_test.go +++ b/module/services/accounts/code/pkg/business/solution_audit_events_test.go @@ -51,11 +51,12 @@ func TestParseDeclaredAuditEventTypes_ReadsTypedFields(t *testing.T) { } want := []DeclaredAuditEventType{ // A declaration that names no visibility gets the default: the event - // stays inside the platform. - {Type: "acme.item.closed", Namespace: "acme", SolutionID: "acme", Visibility: AuditVisibilityTenant, Fields: []PayloadField{}}, + // stays inside the platform. One that names no retention class is + // content. + {Type: "acme.item.closed", Namespace: "acme", SolutionID: "acme", Visibility: AuditVisibilityTenant, Retention: RetentionContent, Fields: []PayloadField{}}, { Type: "acme.item.created", Namespace: "acme", SolutionID: "acme", Description: "An item was created.", - Visibility: AuditVisibilityTenant, + Visibility: AuditVisibilityTenant, Retention: RetentionContent, Fields: []PayloadField{ {Name: "count", Kind: FieldInt}, {Name: "score", Kind: FieldNumber}, @@ -120,7 +121,7 @@ func manyFields(n int) string { func TestDeclaredAuditEventType_SchemaRoundTrip(t *testing.T) { declared := DeclaredAuditEventType{ Type: "acme.item.created", Namespace: "acme", SolutionID: "acme", Description: "An item was created.", - Visibility: AuditVisibilityExternal, + Visibility: AuditVisibilityExternal, Retention: RetentionSecurity, Fields: []PayloadField{ {Name: "count", Kind: FieldInt}, {Name: "enabled", Kind: FieldBool}, @@ -133,14 +134,14 @@ func TestDeclaredAuditEventType_SchemaRoundTrip(t *testing.T) { }, } back, err := DeclaredAuditEventTypeFromSchema(declared.Type, declared.Namespace, - SolutionAuditOwner("acme"), declared.Visibility, declared.PayloadSchemaJSON()) + SolutionAuditOwner("acme"), declared.Visibility, string(declared.Retention), declared.PayloadSchemaJSON()) if err != nil { t.Fatalf("read back: %v", err) } if !reflect.DeepEqual(back, declared) { t.Fatalf("round trip = %#v\nwant %#v", back, declared) } - if _, err := DeclaredAuditEventTypeFromSchema(declared.Type, declared.Namespace, "accounts", declared.Visibility, declared.PayloadSchemaJSON()); err == nil { + if _, err := DeclaredAuditEventTypeFromSchema(declared.Type, declared.Namespace, "accounts", declared.Visibility, string(declared.Retention), declared.PayloadSchemaJSON()); err == nil { t.Fatal("a code-owned row must not read back as a declared type") } // The column is the delivery gate's input, so a row carrying anything the @@ -149,10 +150,17 @@ func TestDeclaredAuditEventType_SchemaRoundTrip(t *testing.T) { // leave the platform. for _, visibility := range []string{"", "internal", "External"} { if _, err := DeclaredAuditEventTypeFromSchema(declared.Type, declared.Namespace, - SolutionAuditOwner("acme"), visibility, declared.PayloadSchemaJSON()); err == nil { + SolutionAuditOwner("acme"), visibility, string(declared.Retention), declared.PayloadSchemaJSON()); err == nil { t.Fatalf("visibility %q must not read back", visibility) } } + // So is the retention class, which routes the type's details. + for _, retention := range []string{"", "Security", "permanent"} { + if _, err := DeclaredAuditEventTypeFromSchema(declared.Type, declared.Namespace, + SolutionAuditOwner("acme"), declared.Visibility, retention, declared.PayloadSchemaJSON()); err == nil { + t.Fatalf("retention %q must not read back", retention) + } + } } func TestCheckAdditiveAuditFieldChange(t *testing.T) { diff --git a/module/services/accounts/code/pkg/cataloggen/deployment_topology.go b/module/services/accounts/code/pkg/cataloggen/deployment_topology.go index fd83f8b7c..d6d52a3e1 100644 --- a/module/services/accounts/code/pkg/cataloggen/deployment_topology.go +++ b/module/services/accounts/code/pkg/cataloggen/deployment_topology.go @@ -46,13 +46,14 @@ type deploymentBindings struct { } type deploymentDeployJobBinding struct { - Name string `yaml:"name"` - Service string `yaml:"service"` - Command string `yaml:"command"` - Catalog string `yaml:"catalog"` - Force bool `yaml:"force,omitempty"` - Writes deploymentDeployJobWrite `yaml:"writes"` - After []string `yaml:"after,omitempty"` + Name string `yaml:"name"` + Service string `yaml:"service"` + Command string `yaml:"command"` + Catalog string `yaml:"catalog"` + Force bool `yaml:"force,omitempty"` + Writes deploymentDeployJobWrite `yaml:"writes"` + After []string `yaml:"after,omitempty"` + ServiceEnvironment []string `yaml:"service_environment,omitempty"` } type deploymentDeployJobWrite struct { diff --git a/module/services/accounts/code/pkg/gen/saas/accounts/v1/module_capabilities.pb.go b/module/services/accounts/code/pkg/gen/saas/accounts/v1/module_capabilities.pb.go index 31668aeb0..f4bf2660c 100644 --- a/module/services/accounts/code/pkg/gen/saas/accounts/v1/module_capabilities.pb.go +++ b/module/services/accounts/code/pkg/gen/saas/accounts/v1/module_capabilities.pb.go @@ -163,6 +163,66 @@ func (ModuleAuditEventVisibility) EnumDescriptor() ([]byte, []int) { return file_saas_accounts_v1_module_capabilities_proto_rawDescGZIP(), []int{1} } +// How long the full details of a declared type's events are kept once a +// warehouse is the audit store of record (ADR 0009). The envelope — who, which +// organization, what type, which target, when — and a hash of the details are +// kept for the compliance window either way; the class decides where the +// details go. +type ModuleAuditEventRetention int32 + +const ( + // Unset means CONTENT, the class of a type that says nothing. + ModuleAuditEventRetention_MODULE_AUDIT_EVENT_RETENTION_UNSPECIFIED ModuleAuditEventRetention = 0 + // Full details kept for the shorter, configured content window; the locked + // archive holds only the envelope and the hash. + ModuleAuditEventRetention_MODULE_AUDIT_EVENT_RETENTION_CONTENT ModuleAuditEventRetention = 1 + // Full details kept for the compliance window, in the warehouse and the + // locked archive alike: authentication, permission and role changes, admin + // actions, data exports, configuration changes. + ModuleAuditEventRetention_MODULE_AUDIT_EVENT_RETENTION_SECURITY ModuleAuditEventRetention = 2 +) + +// Enum value maps for ModuleAuditEventRetention. +var ( + ModuleAuditEventRetention_name = map[int32]string{ + 0: "MODULE_AUDIT_EVENT_RETENTION_UNSPECIFIED", + 1: "MODULE_AUDIT_EVENT_RETENTION_CONTENT", + 2: "MODULE_AUDIT_EVENT_RETENTION_SECURITY", + } + ModuleAuditEventRetention_value = map[string]int32{ + "MODULE_AUDIT_EVENT_RETENTION_UNSPECIFIED": 0, + "MODULE_AUDIT_EVENT_RETENTION_CONTENT": 1, + "MODULE_AUDIT_EVENT_RETENTION_SECURITY": 2, + } +) + +func (x ModuleAuditEventRetention) Enum() *ModuleAuditEventRetention { + p := new(ModuleAuditEventRetention) + *p = x + return p +} + +func (x ModuleAuditEventRetention) String() string { + return protoimpl.X.EnumStringOf(x.Descriptor(), protoreflect.EnumNumber(x)) +} + +func (ModuleAuditEventRetention) Descriptor() protoreflect.EnumDescriptor { + return file_saas_accounts_v1_module_capabilities_proto_enumTypes[2].Descriptor() +} + +func (ModuleAuditEventRetention) Type() protoreflect.EnumType { + return &file_saas_accounts_v1_module_capabilities_proto_enumTypes[2] +} + +func (x ModuleAuditEventRetention) Number() protoreflect.EnumNumber { + return protoreflect.EnumNumber(x) +} + +// Deprecated: Use ModuleAuditEventRetention.Descriptor instead. +func (ModuleAuditEventRetention) EnumDescriptor() ([]byte, []int) { + return file_saas_accounts_v1_module_capabilities_proto_rawDescGZIP(), []int{2} +} + // DatasourceItemReadersBasis says where an item's readers came from. type DatasourceItemReadersBasis int32 @@ -205,11 +265,11 @@ func (x DatasourceItemReadersBasis) String() string { } func (DatasourceItemReadersBasis) Descriptor() protoreflect.EnumDescriptor { - return file_saas_accounts_v1_module_capabilities_proto_enumTypes[2].Descriptor() + return file_saas_accounts_v1_module_capabilities_proto_enumTypes[3].Descriptor() } func (DatasourceItemReadersBasis) Type() protoreflect.EnumType { - return &file_saas_accounts_v1_module_capabilities_proto_enumTypes[2] + return &file_saas_accounts_v1_module_capabilities_proto_enumTypes[3] } func (x DatasourceItemReadersBasis) Number() protoreflect.EnumNumber { @@ -218,7 +278,7 @@ func (x DatasourceItemReadersBasis) Number() protoreflect.EnumNumber { // Deprecated: Use DatasourceItemReadersBasis.Descriptor instead. func (DatasourceItemReadersBasis) EnumDescriptor() ([]byte, []int) { - return file_saas_accounts_v1_module_capabilities_proto_rawDescGZIP(), []int{2} + return file_saas_accounts_v1_module_capabilities_proto_rawDescGZIP(), []int{3} } // EventDelivery is the ordering guarantee a subscription requests, mirroring @@ -257,11 +317,11 @@ func (x EventDelivery) String() string { } func (EventDelivery) Descriptor() protoreflect.EnumDescriptor { - return file_saas_accounts_v1_module_capabilities_proto_enumTypes[3].Descriptor() + return file_saas_accounts_v1_module_capabilities_proto_enumTypes[4].Descriptor() } func (EventDelivery) Type() protoreflect.EnumType { - return &file_saas_accounts_v1_module_capabilities_proto_enumTypes[3] + return &file_saas_accounts_v1_module_capabilities_proto_enumTypes[4] } func (x EventDelivery) Number() protoreflect.EnumNumber { @@ -270,7 +330,7 @@ func (x EventDelivery) Number() protoreflect.EnumNumber { // Deprecated: Use EventDelivery.Descriptor instead. func (EventDelivery) EnumDescriptor() ([]byte, []int) { - return file_saas_accounts_v1_module_capabilities_proto_rawDescGZIP(), []int{3} + return file_saas_accounts_v1_module_capabilities_proto_rawDescGZIP(), []int{4} } // MetadataDisclosure separates "the host withheld this from you" from "there is @@ -310,11 +370,11 @@ func (x MetadataDisclosure) String() string { } func (MetadataDisclosure) Descriptor() protoreflect.EnumDescriptor { - return file_saas_accounts_v1_module_capabilities_proto_enumTypes[4].Descriptor() + return file_saas_accounts_v1_module_capabilities_proto_enumTypes[5].Descriptor() } func (MetadataDisclosure) Type() protoreflect.EnumType { - return &file_saas_accounts_v1_module_capabilities_proto_enumTypes[4] + return &file_saas_accounts_v1_module_capabilities_proto_enumTypes[5] } func (x MetadataDisclosure) Number() protoreflect.EnumNumber { @@ -323,7 +383,7 @@ func (x MetadataDisclosure) Number() protoreflect.EnumNumber { // Deprecated: Use MetadataDisclosure.Descriptor instead. func (MetadataDisclosure) EnumDescriptor() ([]byte, []int) { - return file_saas_accounts_v1_module_capabilities_proto_rawDescGZIP(), []int{4} + return file_saas_accounts_v1_module_capabilities_proto_rawDescGZIP(), []int{5} } // SourceSyncStage names which occurrence CollectionSyncProvenance.at records. @@ -363,11 +423,11 @@ func (x SourceSyncStage) String() string { } func (SourceSyncStage) Descriptor() protoreflect.EnumDescriptor { - return file_saas_accounts_v1_module_capabilities_proto_enumTypes[5].Descriptor() + return file_saas_accounts_v1_module_capabilities_proto_enumTypes[6].Descriptor() } func (SourceSyncStage) Type() protoreflect.EnumType { - return &file_saas_accounts_v1_module_capabilities_proto_enumTypes[5] + return &file_saas_accounts_v1_module_capabilities_proto_enumTypes[6] } func (x SourceSyncStage) Number() protoreflect.EnumNumber { @@ -376,7 +436,7 @@ func (x SourceSyncStage) Number() protoreflect.EnumNumber { // Deprecated: Use SourceSyncStage.Descriptor instead. func (SourceSyncStage) EnumDescriptor() ([]byte, []int) { - return file_saas_accounts_v1_module_capabilities_proto_rawDescGZIP(), []int{5} + return file_saas_accounts_v1_module_capabilities_proto_rawDescGZIP(), []int{6} } // ModuleEnqueueJobRequest names the tenant the work belongs to and carries the @@ -1856,7 +1916,12 @@ type ModuleAuditEventTypeDeclaration struct { // narrowing it would silently stop deliveries to endpoints already // subscribed and widening it would start sending out facts a tenant's // endpoint never agreed to receive under this name. - Visibility ModuleAuditEventVisibility `protobuf:"varint,4,opt,name=visibility,proto3,enum=saas.accounts.v1.ModuleAuditEventVisibility" json:"visibility,omitempty"` + Visibility ModuleAuditEventVisibility `protobuf:"varint,4,opt,name=visibility,proto3,enum=saas.accounts.v1.ModuleAuditEventVisibility" json:"visibility,omitempty"` + // The type's retention class. It only grows: a later declaration may raise + // CONTENT to SECURITY, and one that would lower SECURITY — including by + // leaving this unset — is refused, because that would shorten how long the + // type's evidence is kept without anyone deciding so. + Retention ModuleAuditEventRetention `protobuf:"varint,5,opt,name=retention,proto3,enum=saas.accounts.v1.ModuleAuditEventRetention" json:"retention,omitempty"` unknownFields protoimpl.UnknownFields sizeCache protoimpl.SizeCache } @@ -1919,6 +1984,13 @@ func (x *ModuleAuditEventTypeDeclaration) GetVisibility() ModuleAuditEventVisibi return ModuleAuditEventVisibility_MODULE_AUDIT_EVENT_VISIBILITY_UNSPECIFIED } +func (x *ModuleAuditEventTypeDeclaration) GetRetention() ModuleAuditEventRetention { + if x != nil { + return x.Retention + } + return ModuleAuditEventRetention_MODULE_AUDIT_EVENT_RETENTION_UNSPECIFIED +} + // ModuleDeclareAuditEventTypesRequest declares the audit event types a composed // module owns, so its EmitAuditEvent calls for them are accepted and checked // against the declared fields. It is the module-side form of a solution's @@ -4316,7 +4388,7 @@ const file_saas_accounts_v1_module_capabilities_proto_rawDesc = "" + "\x04kind\x18\x02 \x01(\x0e2&.saas.accounts.v1.ModuleAuditFieldKindB\n" + "\xbaH\a\x82\x01\x04\x10\x01 \x00R\x04kind\x12 \n" + "\x06values\x18\x03 \x03(\tB\b\xbaH\x05\x92\x01\x02\x10@R\x06values\x12\x10\n" + - "\x03pii\x18\x04 \x01(\bR\x03pii\"\x96\x02\n" + + "\x03pii\x18\x04 \x01(\bR\x03pii\"\xeb\x02\n" + "\x1fModuleAuditEventTypeDeclaration\x12\x1e\n" + "\x04type\x18\x01 \x01(\tB\n" + "\xbaH\ar\x05\x10\x01\x18\x80\x01R\x04type\x12*\n" + @@ -4324,7 +4396,8 @@ const file_saas_accounts_v1_module_capabilities_proto_rawDesc = "" + "\x06fields\x18\x03 \x03(\v2-.saas.accounts.v1.ModuleAuditFieldDeclarationB\b\xbaH\x05\x92\x01\x02\x10 R\x06fields\x12V\n" + "\n" + "visibility\x18\x04 \x01(\x0e2,.saas.accounts.v1.ModuleAuditEventVisibilityB\b\xbaH\x05\x82\x01\x02\x10\x01R\n" + - "visibility\"\xc2\x01\n" + + "visibility\x12S\n" + + "\tretention\x18\x05 \x01(\x0e2+.saas.accounts.v1.ModuleAuditEventRetentionB\b\xbaH\x05\x82\x01\x02\x10\x01R\tretention\"\xc2\x01\n" + "#ModuleDeclareAuditEventTypesRequest\x12F\n" + "\x06prefix\x18\x01 \x01(\tB.\xbaH+r)\x10\x01\x18\x80\x012\"^[a-z0-9](?:[a-z0-9_-]*[a-z0-9])?$R\x06prefix\x12S\n" + "\x05types\x18\x02 \x03(\v21.saas.accounts.v1.ModuleAuditEventTypeDeclarationB\n" + @@ -4506,7 +4579,11 @@ const file_saas_accounts_v1_module_capabilities_proto_rawDesc = "" + "\x1aModuleAuditEventVisibility\x12-\n" + ")MODULE_AUDIT_EVENT_VISIBILITY_UNSPECIFIED\x10\x00\x12(\n" + "$MODULE_AUDIT_EVENT_VISIBILITY_TENANT\x10\x01\x12*\n" + - "&MODULE_AUDIT_EVENT_VISIBILITY_EXTERNAL\x10\x02*\xdc\x01\n" + + "&MODULE_AUDIT_EVENT_VISIBILITY_EXTERNAL\x10\x02*\x9e\x01\n" + + "\x19ModuleAuditEventRetention\x12,\n" + + "(MODULE_AUDIT_EVENT_RETENTION_UNSPECIFIED\x10\x00\x12(\n" + + "$MODULE_AUDIT_EVENT_RETENTION_CONTENT\x10\x01\x12)\n" + + "%MODULE_AUDIT_EVENT_RETENTION_SECURITY\x10\x02*\xdc\x01\n" + "\x1aDatasourceItemReadersBasis\x12-\n" + ")DATASOURCE_ITEM_READERS_BASIS_UNSPECIFIED\x10\x00\x12,\n" + "(DATASOURCE_ITEM_READERS_BASIS_TRANSLATED\x10\x01\x12/\n" + @@ -4579,210 +4656,212 @@ func file_saas_accounts_v1_module_capabilities_proto_rawDescGZIP() []byte { return file_saas_accounts_v1_module_capabilities_proto_rawDescData } -var file_saas_accounts_v1_module_capabilities_proto_enumTypes = make([]protoimpl.EnumInfo, 6) +var file_saas_accounts_v1_module_capabilities_proto_enumTypes = make([]protoimpl.EnumInfo, 7) var file_saas_accounts_v1_module_capabilities_proto_msgTypes = make([]protoimpl.MessageInfo, 56) var file_saas_accounts_v1_module_capabilities_proto_goTypes = []any{ (ModuleAuditFieldKind)(0), // 0: saas.accounts.v1.ModuleAuditFieldKind (ModuleAuditEventVisibility)(0), // 1: saas.accounts.v1.ModuleAuditEventVisibility - (DatasourceItemReadersBasis)(0), // 2: saas.accounts.v1.DatasourceItemReadersBasis - (EventDelivery)(0), // 3: saas.accounts.v1.EventDelivery - (MetadataDisclosure)(0), // 4: saas.accounts.v1.MetadataDisclosure - (SourceSyncStage)(0), // 5: saas.accounts.v1.SourceSyncStage - (*ModuleEnqueueJobRequest)(nil), // 6: saas.accounts.v1.ModuleEnqueueJobRequest - (*ModuleEnqueueJobResponse)(nil), // 7: saas.accounts.v1.ModuleEnqueueJobResponse - (*ModuleClaimJobsRequest)(nil), // 8: saas.accounts.v1.ModuleClaimJobsRequest - (*ModuleClaimJobsResponse)(nil), // 9: saas.accounts.v1.ModuleClaimJobsResponse - (*ModuleHeartbeatJobRequest)(nil), // 10: saas.accounts.v1.ModuleHeartbeatJobRequest - (*ModuleHeartbeatJobResponse)(nil), // 11: saas.accounts.v1.ModuleHeartbeatJobResponse - (*ModuleAckJobRequest)(nil), // 12: saas.accounts.v1.ModuleAckJobRequest - (*ModuleNackJobRequest)(nil), // 13: saas.accounts.v1.ModuleNackJobRequest - (*ModuleNotifyUserRequest)(nil), // 14: saas.accounts.v1.ModuleNotifyUserRequest - (*ModuleNotifyUserResponse)(nil), // 15: saas.accounts.v1.ModuleNotifyUserResponse - (*ModuleNotifyOrgAdminsRequest)(nil), // 16: saas.accounts.v1.ModuleNotifyOrgAdminsRequest - (*ModuleNotifyOrgAdminsResponse)(nil), // 17: saas.accounts.v1.ModuleNotifyOrgAdminsResponse - (*ModuleApprovalPolicy)(nil), // 18: saas.accounts.v1.ModuleApprovalPolicy - (*ModuleResumeRef)(nil), // 19: saas.accounts.v1.ModuleResumeRef - (*ModuleRequestApprovalRequest)(nil), // 20: saas.accounts.v1.ModuleRequestApprovalRequest - (*ModuleRequestApprovalResponse)(nil), // 21: saas.accounts.v1.ModuleRequestApprovalResponse - (*ModuleGetApprovalRequest)(nil), // 22: saas.accounts.v1.ModuleGetApprovalRequest - (*ModuleApproval)(nil), // 23: saas.accounts.v1.ModuleApproval - (*ModuleCancelApprovalRequest)(nil), // 24: saas.accounts.v1.ModuleCancelApprovalRequest - (*ModuleEmitAuditEventRequest)(nil), // 25: saas.accounts.v1.ModuleEmitAuditEventRequest - (*ModuleAuditFieldDeclaration)(nil), // 26: saas.accounts.v1.ModuleAuditFieldDeclaration - (*ModuleAuditEventTypeDeclaration)(nil), // 27: saas.accounts.v1.ModuleAuditEventTypeDeclaration - (*ModuleDeclareAuditEventTypesRequest)(nil), // 28: saas.accounts.v1.ModuleDeclareAuditEventTypesRequest - (*ModuleDeclareAuditEventTypesResponse)(nil), // 29: saas.accounts.v1.ModuleDeclareAuditEventTypesResponse - (*SubjectVisibilityGrant)(nil), // 30: saas.accounts.v1.SubjectVisibilityGrant - (*ModuleListSubjectVisibilityRequest)(nil), // 31: saas.accounts.v1.ModuleListSubjectVisibilityRequest - (*ModuleListSubjectVisibilityResponse)(nil), // 32: saas.accounts.v1.ModuleListSubjectVisibilityResponse - (*FetchDatasourceBlobRequest)(nil), // 33: saas.accounts.v1.FetchDatasourceBlobRequest - (*FetchDatasourceBlobChunk)(nil), // 34: saas.accounts.v1.FetchDatasourceBlobChunk - (*DatasourceFileRef)(nil), // 35: saas.accounts.v1.DatasourceFileRef - (*FetchDatasourceFilesRequest)(nil), // 36: saas.accounts.v1.FetchDatasourceFilesRequest - (*DatasourceProvenance)(nil), // 37: saas.accounts.v1.DatasourceProvenance - (*DatasourceItemReaders)(nil), // 38: saas.accounts.v1.DatasourceItemReaders - (*DatasourceFileHeader)(nil), // 39: saas.accounts.v1.DatasourceFileHeader - (*FetchDatasourceFilesFrame)(nil), // 40: saas.accounts.v1.FetchDatasourceFilesFrame - (*ModulePublishEventRequest)(nil), // 41: saas.accounts.v1.ModulePublishEventRequest - (*ModulePublishEventResponse)(nil), // 42: saas.accounts.v1.ModulePublishEventResponse - (*ModuleSubscribeRequest)(nil), // 43: saas.accounts.v1.ModuleSubscribeRequest - (*ModuleSubscription)(nil), // 44: saas.accounts.v1.ModuleSubscription - (*ModuleSubscribeResponse)(nil), // 45: saas.accounts.v1.ModuleSubscribeResponse - (*ModuleUnsubscribeRequest)(nil), // 46: saas.accounts.v1.ModuleUnsubscribeRequest - (*ModuleListSubscriptionsRequest)(nil), // 47: saas.accounts.v1.ModuleListSubscriptionsRequest - (*ModuleListSubscriptionsResponse)(nil), // 48: saas.accounts.v1.ModuleListSubscriptionsResponse - (*ModuleReplayEventsRequest)(nil), // 49: saas.accounts.v1.ModuleReplayEventsRequest - (*ModuleReplayEventsResponse)(nil), // 50: saas.accounts.v1.ModuleReplayEventsResponse - (*ReadableSourceCollection)(nil), // 51: saas.accounts.v1.ReadableSourceCollection - (*ReadableCollectionGrant)(nil), // 52: saas.accounts.v1.ReadableCollectionGrant - (*CollectionSyncProvenance)(nil), // 53: saas.accounts.v1.CollectionSyncProvenance - (*ListReadableSourceCollectionsRequest)(nil), // 54: saas.accounts.v1.ListReadableSourceCollectionsRequest - (*ListReadableSourceCollectionsResponse)(nil), // 55: saas.accounts.v1.ListReadableSourceCollectionsResponse - (*ModulePlaceRecordRequest)(nil), // 56: saas.accounts.v1.ModulePlaceRecordRequest - (*ModulePlaceRecordResponse)(nil), // 57: saas.accounts.v1.ModulePlaceRecordResponse - (*CheckWorkContextRecordAccessRequest)(nil), // 58: saas.accounts.v1.CheckWorkContextRecordAccessRequest - (*CheckWorkContextRecordAccessResponse)(nil), // 59: saas.accounts.v1.CheckWorkContextRecordAccessResponse - (*ModuleExchangeDelegatedReadAudienceRequest)(nil), // 60: saas.accounts.v1.ModuleExchangeDelegatedReadAudienceRequest - (*ModuleExchangeDelegatedOperationAudienceRequest)(nil), // 61: saas.accounts.v1.ModuleExchangeDelegatedOperationAudienceRequest - (*v1.NewJob)(nil), // 62: saas.jobs.v1.NewJob - (v1.JobEnqueueDisposition)(0), // 63: saas.jobs.v1.JobEnqueueDisposition - (*durationpb.Duration)(nil), // 64: google.protobuf.Duration - (*v1.JobEnvelope)(nil), // 65: saas.jobs.v1.JobEnvelope - (*v1.JobLeaseReference)(nil), // 66: saas.jobs.v1.JobLeaseReference - (*v1.JobLease)(nil), // 67: saas.jobs.v1.JobLease - (*v1.JobFailure)(nil), // 68: saas.jobs.v1.JobFailure - (*timestamppb.Timestamp)(nil), // 69: google.protobuf.Timestamp - (*structpb.Struct)(nil), // 70: google.protobuf.Struct - (*v11.EventEnvelope)(nil), // 71: saas.events.v1.EventEnvelope - (*ModuleMintRegistrationRequest)(nil), // 72: saas.accounts.v1.ModuleMintRegistrationRequest - (*SolutionMintRegistrationRequest)(nil), // 73: saas.accounts.v1.SolutionMintRegistrationRequest - (*ModuleMintWorkContextRequest)(nil), // 74: saas.accounts.v1.ModuleMintWorkContextRequest - (*ModuleMintOperationContextRequest)(nil), // 75: saas.accounts.v1.ModuleMintOperationContextRequest - (*ModuleMintSourceOperationContextRequest)(nil), // 76: saas.accounts.v1.ModuleMintSourceOperationContextRequest - (*IssuedWorkContext)(nil), // 77: saas.accounts.v1.IssuedWorkContext - (*emptypb.Empty)(nil), // 78: google.protobuf.Empty - (*ModuleMintRegistrationResponse)(nil), // 79: saas.accounts.v1.ModuleMintRegistrationResponse - (*SolutionMintRegistrationResponse)(nil), // 80: saas.accounts.v1.SolutionMintRegistrationResponse - (*ModuleMintWorkContextResponse)(nil), // 81: saas.accounts.v1.ModuleMintWorkContextResponse - (*ModuleMintOperationContextResponse)(nil), // 82: saas.accounts.v1.ModuleMintOperationContextResponse - (*ModuleMintSourceOperationContextResponse)(nil), // 83: saas.accounts.v1.ModuleMintSourceOperationContextResponse + (ModuleAuditEventRetention)(0), // 2: saas.accounts.v1.ModuleAuditEventRetention + (DatasourceItemReadersBasis)(0), // 3: saas.accounts.v1.DatasourceItemReadersBasis + (EventDelivery)(0), // 4: saas.accounts.v1.EventDelivery + (MetadataDisclosure)(0), // 5: saas.accounts.v1.MetadataDisclosure + (SourceSyncStage)(0), // 6: saas.accounts.v1.SourceSyncStage + (*ModuleEnqueueJobRequest)(nil), // 7: saas.accounts.v1.ModuleEnqueueJobRequest + (*ModuleEnqueueJobResponse)(nil), // 8: saas.accounts.v1.ModuleEnqueueJobResponse + (*ModuleClaimJobsRequest)(nil), // 9: saas.accounts.v1.ModuleClaimJobsRequest + (*ModuleClaimJobsResponse)(nil), // 10: saas.accounts.v1.ModuleClaimJobsResponse + (*ModuleHeartbeatJobRequest)(nil), // 11: saas.accounts.v1.ModuleHeartbeatJobRequest + (*ModuleHeartbeatJobResponse)(nil), // 12: saas.accounts.v1.ModuleHeartbeatJobResponse + (*ModuleAckJobRequest)(nil), // 13: saas.accounts.v1.ModuleAckJobRequest + (*ModuleNackJobRequest)(nil), // 14: saas.accounts.v1.ModuleNackJobRequest + (*ModuleNotifyUserRequest)(nil), // 15: saas.accounts.v1.ModuleNotifyUserRequest + (*ModuleNotifyUserResponse)(nil), // 16: saas.accounts.v1.ModuleNotifyUserResponse + (*ModuleNotifyOrgAdminsRequest)(nil), // 17: saas.accounts.v1.ModuleNotifyOrgAdminsRequest + (*ModuleNotifyOrgAdminsResponse)(nil), // 18: saas.accounts.v1.ModuleNotifyOrgAdminsResponse + (*ModuleApprovalPolicy)(nil), // 19: saas.accounts.v1.ModuleApprovalPolicy + (*ModuleResumeRef)(nil), // 20: saas.accounts.v1.ModuleResumeRef + (*ModuleRequestApprovalRequest)(nil), // 21: saas.accounts.v1.ModuleRequestApprovalRequest + (*ModuleRequestApprovalResponse)(nil), // 22: saas.accounts.v1.ModuleRequestApprovalResponse + (*ModuleGetApprovalRequest)(nil), // 23: saas.accounts.v1.ModuleGetApprovalRequest + (*ModuleApproval)(nil), // 24: saas.accounts.v1.ModuleApproval + (*ModuleCancelApprovalRequest)(nil), // 25: saas.accounts.v1.ModuleCancelApprovalRequest + (*ModuleEmitAuditEventRequest)(nil), // 26: saas.accounts.v1.ModuleEmitAuditEventRequest + (*ModuleAuditFieldDeclaration)(nil), // 27: saas.accounts.v1.ModuleAuditFieldDeclaration + (*ModuleAuditEventTypeDeclaration)(nil), // 28: saas.accounts.v1.ModuleAuditEventTypeDeclaration + (*ModuleDeclareAuditEventTypesRequest)(nil), // 29: saas.accounts.v1.ModuleDeclareAuditEventTypesRequest + (*ModuleDeclareAuditEventTypesResponse)(nil), // 30: saas.accounts.v1.ModuleDeclareAuditEventTypesResponse + (*SubjectVisibilityGrant)(nil), // 31: saas.accounts.v1.SubjectVisibilityGrant + (*ModuleListSubjectVisibilityRequest)(nil), // 32: saas.accounts.v1.ModuleListSubjectVisibilityRequest + (*ModuleListSubjectVisibilityResponse)(nil), // 33: saas.accounts.v1.ModuleListSubjectVisibilityResponse + (*FetchDatasourceBlobRequest)(nil), // 34: saas.accounts.v1.FetchDatasourceBlobRequest + (*FetchDatasourceBlobChunk)(nil), // 35: saas.accounts.v1.FetchDatasourceBlobChunk + (*DatasourceFileRef)(nil), // 36: saas.accounts.v1.DatasourceFileRef + (*FetchDatasourceFilesRequest)(nil), // 37: saas.accounts.v1.FetchDatasourceFilesRequest + (*DatasourceProvenance)(nil), // 38: saas.accounts.v1.DatasourceProvenance + (*DatasourceItemReaders)(nil), // 39: saas.accounts.v1.DatasourceItemReaders + (*DatasourceFileHeader)(nil), // 40: saas.accounts.v1.DatasourceFileHeader + (*FetchDatasourceFilesFrame)(nil), // 41: saas.accounts.v1.FetchDatasourceFilesFrame + (*ModulePublishEventRequest)(nil), // 42: saas.accounts.v1.ModulePublishEventRequest + (*ModulePublishEventResponse)(nil), // 43: saas.accounts.v1.ModulePublishEventResponse + (*ModuleSubscribeRequest)(nil), // 44: saas.accounts.v1.ModuleSubscribeRequest + (*ModuleSubscription)(nil), // 45: saas.accounts.v1.ModuleSubscription + (*ModuleSubscribeResponse)(nil), // 46: saas.accounts.v1.ModuleSubscribeResponse + (*ModuleUnsubscribeRequest)(nil), // 47: saas.accounts.v1.ModuleUnsubscribeRequest + (*ModuleListSubscriptionsRequest)(nil), // 48: saas.accounts.v1.ModuleListSubscriptionsRequest + (*ModuleListSubscriptionsResponse)(nil), // 49: saas.accounts.v1.ModuleListSubscriptionsResponse + (*ModuleReplayEventsRequest)(nil), // 50: saas.accounts.v1.ModuleReplayEventsRequest + (*ModuleReplayEventsResponse)(nil), // 51: saas.accounts.v1.ModuleReplayEventsResponse + (*ReadableSourceCollection)(nil), // 52: saas.accounts.v1.ReadableSourceCollection + (*ReadableCollectionGrant)(nil), // 53: saas.accounts.v1.ReadableCollectionGrant + (*CollectionSyncProvenance)(nil), // 54: saas.accounts.v1.CollectionSyncProvenance + (*ListReadableSourceCollectionsRequest)(nil), // 55: saas.accounts.v1.ListReadableSourceCollectionsRequest + (*ListReadableSourceCollectionsResponse)(nil), // 56: saas.accounts.v1.ListReadableSourceCollectionsResponse + (*ModulePlaceRecordRequest)(nil), // 57: saas.accounts.v1.ModulePlaceRecordRequest + (*ModulePlaceRecordResponse)(nil), // 58: saas.accounts.v1.ModulePlaceRecordResponse + (*CheckWorkContextRecordAccessRequest)(nil), // 59: saas.accounts.v1.CheckWorkContextRecordAccessRequest + (*CheckWorkContextRecordAccessResponse)(nil), // 60: saas.accounts.v1.CheckWorkContextRecordAccessResponse + (*ModuleExchangeDelegatedReadAudienceRequest)(nil), // 61: saas.accounts.v1.ModuleExchangeDelegatedReadAudienceRequest + (*ModuleExchangeDelegatedOperationAudienceRequest)(nil), // 62: saas.accounts.v1.ModuleExchangeDelegatedOperationAudienceRequest + (*v1.NewJob)(nil), // 63: saas.jobs.v1.NewJob + (v1.JobEnqueueDisposition)(0), // 64: saas.jobs.v1.JobEnqueueDisposition + (*durationpb.Duration)(nil), // 65: google.protobuf.Duration + (*v1.JobEnvelope)(nil), // 66: saas.jobs.v1.JobEnvelope + (*v1.JobLeaseReference)(nil), // 67: saas.jobs.v1.JobLeaseReference + (*v1.JobLease)(nil), // 68: saas.jobs.v1.JobLease + (*v1.JobFailure)(nil), // 69: saas.jobs.v1.JobFailure + (*timestamppb.Timestamp)(nil), // 70: google.protobuf.Timestamp + (*structpb.Struct)(nil), // 71: google.protobuf.Struct + (*v11.EventEnvelope)(nil), // 72: saas.events.v1.EventEnvelope + (*ModuleMintRegistrationRequest)(nil), // 73: saas.accounts.v1.ModuleMintRegistrationRequest + (*SolutionMintRegistrationRequest)(nil), // 74: saas.accounts.v1.SolutionMintRegistrationRequest + (*ModuleMintWorkContextRequest)(nil), // 75: saas.accounts.v1.ModuleMintWorkContextRequest + (*ModuleMintOperationContextRequest)(nil), // 76: saas.accounts.v1.ModuleMintOperationContextRequest + (*ModuleMintSourceOperationContextRequest)(nil), // 77: saas.accounts.v1.ModuleMintSourceOperationContextRequest + (*IssuedWorkContext)(nil), // 78: saas.accounts.v1.IssuedWorkContext + (*emptypb.Empty)(nil), // 79: google.protobuf.Empty + (*ModuleMintRegistrationResponse)(nil), // 80: saas.accounts.v1.ModuleMintRegistrationResponse + (*SolutionMintRegistrationResponse)(nil), // 81: saas.accounts.v1.SolutionMintRegistrationResponse + (*ModuleMintWorkContextResponse)(nil), // 82: saas.accounts.v1.ModuleMintWorkContextResponse + (*ModuleMintOperationContextResponse)(nil), // 83: saas.accounts.v1.ModuleMintOperationContextResponse + (*ModuleMintSourceOperationContextResponse)(nil), // 84: saas.accounts.v1.ModuleMintSourceOperationContextResponse } var file_saas_accounts_v1_module_capabilities_proto_depIdxs = []int32{ - 62, // 0: saas.accounts.v1.ModuleEnqueueJobRequest.job:type_name -> saas.jobs.v1.NewJob - 63, // 1: saas.accounts.v1.ModuleEnqueueJobResponse.disposition:type_name -> saas.jobs.v1.JobEnqueueDisposition - 64, // 2: saas.accounts.v1.ModuleClaimJobsRequest.lease_duration:type_name -> google.protobuf.Duration - 65, // 3: saas.accounts.v1.ModuleClaimJobsResponse.jobs:type_name -> saas.jobs.v1.JobEnvelope - 66, // 4: saas.accounts.v1.ModuleHeartbeatJobRequest.lease:type_name -> saas.jobs.v1.JobLeaseReference - 64, // 5: saas.accounts.v1.ModuleHeartbeatJobRequest.extension:type_name -> google.protobuf.Duration - 67, // 6: saas.accounts.v1.ModuleHeartbeatJobResponse.lease:type_name -> saas.jobs.v1.JobLease - 66, // 7: saas.accounts.v1.ModuleAckJobRequest.lease:type_name -> saas.jobs.v1.JobLeaseReference - 66, // 8: saas.accounts.v1.ModuleNackJobRequest.lease:type_name -> saas.jobs.v1.JobLeaseReference - 68, // 9: saas.accounts.v1.ModuleNackJobRequest.failure:type_name -> saas.jobs.v1.JobFailure - 69, // 10: saas.accounts.v1.ModuleNackJobRequest.retry_at:type_name -> google.protobuf.Timestamp - 70, // 11: saas.accounts.v1.ModuleResumeRef.payload:type_name -> google.protobuf.Struct - 70, // 12: saas.accounts.v1.ModuleRequestApprovalRequest.subject:type_name -> google.protobuf.Struct - 18, // 13: saas.accounts.v1.ModuleRequestApprovalRequest.policy:type_name -> saas.accounts.v1.ModuleApprovalPolicy - 19, // 14: saas.accounts.v1.ModuleRequestApprovalRequest.resume_ref:type_name -> saas.accounts.v1.ModuleResumeRef - 69, // 15: saas.accounts.v1.ModuleRequestApprovalRequest.expires_at:type_name -> google.protobuf.Timestamp - 69, // 16: saas.accounts.v1.ModuleRequestApprovalRequest.escalate_at:type_name -> google.protobuf.Timestamp - 70, // 17: saas.accounts.v1.ModuleApproval.subject:type_name -> google.protobuf.Struct - 19, // 18: saas.accounts.v1.ModuleApproval.resume_ref:type_name -> saas.accounts.v1.ModuleResumeRef - 69, // 19: saas.accounts.v1.ModuleApproval.expires_at:type_name -> google.protobuf.Timestamp - 69, // 20: saas.accounts.v1.ModuleApproval.escalate_at:type_name -> google.protobuf.Timestamp - 69, // 21: saas.accounts.v1.ModuleApproval.created_at:type_name -> google.protobuf.Timestamp - 70, // 22: saas.accounts.v1.ModuleEmitAuditEventRequest.fields:type_name -> google.protobuf.Struct + 63, // 0: saas.accounts.v1.ModuleEnqueueJobRequest.job:type_name -> saas.jobs.v1.NewJob + 64, // 1: saas.accounts.v1.ModuleEnqueueJobResponse.disposition:type_name -> saas.jobs.v1.JobEnqueueDisposition + 65, // 2: saas.accounts.v1.ModuleClaimJobsRequest.lease_duration:type_name -> google.protobuf.Duration + 66, // 3: saas.accounts.v1.ModuleClaimJobsResponse.jobs:type_name -> saas.jobs.v1.JobEnvelope + 67, // 4: saas.accounts.v1.ModuleHeartbeatJobRequest.lease:type_name -> saas.jobs.v1.JobLeaseReference + 65, // 5: saas.accounts.v1.ModuleHeartbeatJobRequest.extension:type_name -> google.protobuf.Duration + 68, // 6: saas.accounts.v1.ModuleHeartbeatJobResponse.lease:type_name -> saas.jobs.v1.JobLease + 67, // 7: saas.accounts.v1.ModuleAckJobRequest.lease:type_name -> saas.jobs.v1.JobLeaseReference + 67, // 8: saas.accounts.v1.ModuleNackJobRequest.lease:type_name -> saas.jobs.v1.JobLeaseReference + 69, // 9: saas.accounts.v1.ModuleNackJobRequest.failure:type_name -> saas.jobs.v1.JobFailure + 70, // 10: saas.accounts.v1.ModuleNackJobRequest.retry_at:type_name -> google.protobuf.Timestamp + 71, // 11: saas.accounts.v1.ModuleResumeRef.payload:type_name -> google.protobuf.Struct + 71, // 12: saas.accounts.v1.ModuleRequestApprovalRequest.subject:type_name -> google.protobuf.Struct + 19, // 13: saas.accounts.v1.ModuleRequestApprovalRequest.policy:type_name -> saas.accounts.v1.ModuleApprovalPolicy + 20, // 14: saas.accounts.v1.ModuleRequestApprovalRequest.resume_ref:type_name -> saas.accounts.v1.ModuleResumeRef + 70, // 15: saas.accounts.v1.ModuleRequestApprovalRequest.expires_at:type_name -> google.protobuf.Timestamp + 70, // 16: saas.accounts.v1.ModuleRequestApprovalRequest.escalate_at:type_name -> google.protobuf.Timestamp + 71, // 17: saas.accounts.v1.ModuleApproval.subject:type_name -> google.protobuf.Struct + 20, // 18: saas.accounts.v1.ModuleApproval.resume_ref:type_name -> saas.accounts.v1.ModuleResumeRef + 70, // 19: saas.accounts.v1.ModuleApproval.expires_at:type_name -> google.protobuf.Timestamp + 70, // 20: saas.accounts.v1.ModuleApproval.escalate_at:type_name -> google.protobuf.Timestamp + 70, // 21: saas.accounts.v1.ModuleApproval.created_at:type_name -> google.protobuf.Timestamp + 71, // 22: saas.accounts.v1.ModuleEmitAuditEventRequest.fields:type_name -> google.protobuf.Struct 0, // 23: saas.accounts.v1.ModuleAuditFieldDeclaration.kind:type_name -> saas.accounts.v1.ModuleAuditFieldKind - 26, // 24: saas.accounts.v1.ModuleAuditEventTypeDeclaration.fields:type_name -> saas.accounts.v1.ModuleAuditFieldDeclaration + 27, // 24: saas.accounts.v1.ModuleAuditEventTypeDeclaration.fields:type_name -> saas.accounts.v1.ModuleAuditFieldDeclaration 1, // 25: saas.accounts.v1.ModuleAuditEventTypeDeclaration.visibility:type_name -> saas.accounts.v1.ModuleAuditEventVisibility - 27, // 26: saas.accounts.v1.ModuleDeclareAuditEventTypesRequest.types:type_name -> saas.accounts.v1.ModuleAuditEventTypeDeclaration - 69, // 27: saas.accounts.v1.SubjectVisibilityGrant.expires_at:type_name -> google.protobuf.Timestamp - 30, // 28: saas.accounts.v1.ModuleListSubjectVisibilityResponse.grants:type_name -> saas.accounts.v1.SubjectVisibilityGrant - 35, // 29: saas.accounts.v1.FetchDatasourceFilesRequest.files:type_name -> saas.accounts.v1.DatasourceFileRef - 2, // 30: saas.accounts.v1.DatasourceItemReaders.basis:type_name -> saas.accounts.v1.DatasourceItemReadersBasis - 37, // 31: saas.accounts.v1.DatasourceFileHeader.provenance:type_name -> saas.accounts.v1.DatasourceProvenance - 38, // 32: saas.accounts.v1.DatasourceFileHeader.readers:type_name -> saas.accounts.v1.DatasourceItemReaders - 39, // 33: saas.accounts.v1.FetchDatasourceFilesFrame.header:type_name -> saas.accounts.v1.DatasourceFileHeader - 71, // 34: saas.accounts.v1.ModulePublishEventRequest.envelope:type_name -> saas.events.v1.EventEnvelope - 3, // 35: saas.accounts.v1.ModuleSubscribeRequest.delivery:type_name -> saas.accounts.v1.EventDelivery - 3, // 36: saas.accounts.v1.ModuleSubscription.delivery:type_name -> saas.accounts.v1.EventDelivery - 69, // 37: saas.accounts.v1.ModuleSubscription.created_at:type_name -> google.protobuf.Timestamp - 44, // 38: saas.accounts.v1.ModuleSubscribeResponse.subscription:type_name -> saas.accounts.v1.ModuleSubscription - 44, // 39: saas.accounts.v1.ModuleListSubscriptionsResponse.subscriptions:type_name -> saas.accounts.v1.ModuleSubscription - 69, // 40: saas.accounts.v1.ModuleReplayEventsRequest.since:type_name -> google.protobuf.Timestamp - 4, // 41: saas.accounts.v1.ReadableSourceCollection.grant_disclosure:type_name -> saas.accounts.v1.MetadataDisclosure - 52, // 42: saas.accounts.v1.ReadableSourceCollection.read_grants:type_name -> saas.accounts.v1.ReadableCollectionGrant - 53, // 43: saas.accounts.v1.ReadableSourceCollection.sync:type_name -> saas.accounts.v1.CollectionSyncProvenance - 69, // 44: saas.accounts.v1.ReadableCollectionGrant.expires_at:type_name -> google.protobuf.Timestamp - 5, // 45: saas.accounts.v1.CollectionSyncProvenance.stage:type_name -> saas.accounts.v1.SourceSyncStage - 69, // 46: saas.accounts.v1.CollectionSyncProvenance.at:type_name -> google.protobuf.Timestamp - 4, // 47: saas.accounts.v1.CollectionSyncProvenance.requester_disclosure:type_name -> saas.accounts.v1.MetadataDisclosure - 69, // 48: saas.accounts.v1.CollectionSyncProvenance.requested_at:type_name -> google.protobuf.Timestamp - 51, // 49: saas.accounts.v1.ListReadableSourceCollectionsResponse.collections:type_name -> saas.accounts.v1.ReadableSourceCollection - 60, // 50: saas.accounts.v1.ModuleCapabilitiesService.ExchangeDelegatedReadAudience:input_type -> saas.accounts.v1.ModuleExchangeDelegatedReadAudienceRequest - 61, // 51: saas.accounts.v1.ModuleCapabilitiesService.ExchangeDelegatedOperationAudience:input_type -> saas.accounts.v1.ModuleExchangeDelegatedOperationAudienceRequest - 58, // 52: saas.accounts.v1.ModuleCapabilitiesService.CheckWorkContextRecordAccess:input_type -> saas.accounts.v1.CheckWorkContextRecordAccessRequest - 54, // 53: saas.accounts.v1.ModuleCapabilitiesService.ListReadableSourceCollections:input_type -> saas.accounts.v1.ListReadableSourceCollectionsRequest - 56, // 54: saas.accounts.v1.ModuleCapabilitiesService.PlaceRecord:input_type -> saas.accounts.v1.ModulePlaceRecordRequest - 6, // 55: saas.accounts.v1.ModuleCapabilitiesService.EnqueueJob:input_type -> saas.accounts.v1.ModuleEnqueueJobRequest - 8, // 56: saas.accounts.v1.ModuleCapabilitiesService.ClaimJobs:input_type -> saas.accounts.v1.ModuleClaimJobsRequest - 10, // 57: saas.accounts.v1.ModuleCapabilitiesService.HeartbeatJob:input_type -> saas.accounts.v1.ModuleHeartbeatJobRequest - 12, // 58: saas.accounts.v1.ModuleCapabilitiesService.AckJob:input_type -> saas.accounts.v1.ModuleAckJobRequest - 13, // 59: saas.accounts.v1.ModuleCapabilitiesService.NackJob:input_type -> saas.accounts.v1.ModuleNackJobRequest - 14, // 60: saas.accounts.v1.ModuleCapabilitiesService.NotifyUser:input_type -> saas.accounts.v1.ModuleNotifyUserRequest - 16, // 61: saas.accounts.v1.ModuleCapabilitiesService.NotifyOrgAdmins:input_type -> saas.accounts.v1.ModuleNotifyOrgAdminsRequest - 20, // 62: saas.accounts.v1.ModuleCapabilitiesService.RequestApproval:input_type -> saas.accounts.v1.ModuleRequestApprovalRequest - 22, // 63: saas.accounts.v1.ModuleCapabilitiesService.GetApproval:input_type -> saas.accounts.v1.ModuleGetApprovalRequest - 24, // 64: saas.accounts.v1.ModuleCapabilitiesService.CancelApproval:input_type -> saas.accounts.v1.ModuleCancelApprovalRequest - 25, // 65: saas.accounts.v1.ModuleCapabilitiesService.EmitAuditEvent:input_type -> saas.accounts.v1.ModuleEmitAuditEventRequest - 28, // 66: saas.accounts.v1.ModuleCapabilitiesService.DeclareAuditEventTypes:input_type -> saas.accounts.v1.ModuleDeclareAuditEventTypesRequest - 31, // 67: saas.accounts.v1.ModuleCapabilitiesService.ListSubjectVisibility:input_type -> saas.accounts.v1.ModuleListSubjectVisibilityRequest - 33, // 68: saas.accounts.v1.ModuleCapabilitiesService.FetchDatasourceBlob:input_type -> saas.accounts.v1.FetchDatasourceBlobRequest - 36, // 69: saas.accounts.v1.ModuleCapabilitiesService.FetchDatasourceFiles:input_type -> saas.accounts.v1.FetchDatasourceFilesRequest - 72, // 70: saas.accounts.v1.ModuleCapabilitiesService.MintModuleRegistration:input_type -> saas.accounts.v1.ModuleMintRegistrationRequest - 73, // 71: saas.accounts.v1.ModuleCapabilitiesService.MintSolutionRegistration:input_type -> saas.accounts.v1.SolutionMintRegistrationRequest - 74, // 72: saas.accounts.v1.ModuleCapabilitiesService.MintModuleWorkContext:input_type -> saas.accounts.v1.ModuleMintWorkContextRequest - 75, // 73: saas.accounts.v1.ModuleCapabilitiesService.MintModuleOperationContext:input_type -> saas.accounts.v1.ModuleMintOperationContextRequest - 76, // 74: saas.accounts.v1.ModuleCapabilitiesService.MintSourceOperationContext:input_type -> saas.accounts.v1.ModuleMintSourceOperationContextRequest - 41, // 75: saas.accounts.v1.ModuleCapabilitiesService.PublishEvent:input_type -> saas.accounts.v1.ModulePublishEventRequest - 43, // 76: saas.accounts.v1.ModuleCapabilitiesService.Subscribe:input_type -> saas.accounts.v1.ModuleSubscribeRequest - 46, // 77: saas.accounts.v1.ModuleCapabilitiesService.Unsubscribe:input_type -> saas.accounts.v1.ModuleUnsubscribeRequest - 47, // 78: saas.accounts.v1.ModuleCapabilitiesService.ListSubscriptions:input_type -> saas.accounts.v1.ModuleListSubscriptionsRequest - 49, // 79: saas.accounts.v1.ModuleCapabilitiesService.ReplayEvents:input_type -> saas.accounts.v1.ModuleReplayEventsRequest - 77, // 80: saas.accounts.v1.ModuleCapabilitiesService.ExchangeDelegatedReadAudience:output_type -> saas.accounts.v1.IssuedWorkContext - 77, // 81: saas.accounts.v1.ModuleCapabilitiesService.ExchangeDelegatedOperationAudience:output_type -> saas.accounts.v1.IssuedWorkContext - 59, // 82: saas.accounts.v1.ModuleCapabilitiesService.CheckWorkContextRecordAccess:output_type -> saas.accounts.v1.CheckWorkContextRecordAccessResponse - 55, // 83: saas.accounts.v1.ModuleCapabilitiesService.ListReadableSourceCollections:output_type -> saas.accounts.v1.ListReadableSourceCollectionsResponse - 57, // 84: saas.accounts.v1.ModuleCapabilitiesService.PlaceRecord:output_type -> saas.accounts.v1.ModulePlaceRecordResponse - 7, // 85: saas.accounts.v1.ModuleCapabilitiesService.EnqueueJob:output_type -> saas.accounts.v1.ModuleEnqueueJobResponse - 9, // 86: saas.accounts.v1.ModuleCapabilitiesService.ClaimJobs:output_type -> saas.accounts.v1.ModuleClaimJobsResponse - 11, // 87: saas.accounts.v1.ModuleCapabilitiesService.HeartbeatJob:output_type -> saas.accounts.v1.ModuleHeartbeatJobResponse - 78, // 88: saas.accounts.v1.ModuleCapabilitiesService.AckJob:output_type -> google.protobuf.Empty - 78, // 89: saas.accounts.v1.ModuleCapabilitiesService.NackJob:output_type -> google.protobuf.Empty - 15, // 90: saas.accounts.v1.ModuleCapabilitiesService.NotifyUser:output_type -> saas.accounts.v1.ModuleNotifyUserResponse - 17, // 91: saas.accounts.v1.ModuleCapabilitiesService.NotifyOrgAdmins:output_type -> saas.accounts.v1.ModuleNotifyOrgAdminsResponse - 21, // 92: saas.accounts.v1.ModuleCapabilitiesService.RequestApproval:output_type -> saas.accounts.v1.ModuleRequestApprovalResponse - 23, // 93: saas.accounts.v1.ModuleCapabilitiesService.GetApproval:output_type -> saas.accounts.v1.ModuleApproval - 78, // 94: saas.accounts.v1.ModuleCapabilitiesService.CancelApproval:output_type -> google.protobuf.Empty - 78, // 95: saas.accounts.v1.ModuleCapabilitiesService.EmitAuditEvent:output_type -> google.protobuf.Empty - 29, // 96: saas.accounts.v1.ModuleCapabilitiesService.DeclareAuditEventTypes:output_type -> saas.accounts.v1.ModuleDeclareAuditEventTypesResponse - 32, // 97: saas.accounts.v1.ModuleCapabilitiesService.ListSubjectVisibility:output_type -> saas.accounts.v1.ModuleListSubjectVisibilityResponse - 34, // 98: saas.accounts.v1.ModuleCapabilitiesService.FetchDatasourceBlob:output_type -> saas.accounts.v1.FetchDatasourceBlobChunk - 40, // 99: saas.accounts.v1.ModuleCapabilitiesService.FetchDatasourceFiles:output_type -> saas.accounts.v1.FetchDatasourceFilesFrame - 79, // 100: saas.accounts.v1.ModuleCapabilitiesService.MintModuleRegistration:output_type -> saas.accounts.v1.ModuleMintRegistrationResponse - 80, // 101: saas.accounts.v1.ModuleCapabilitiesService.MintSolutionRegistration:output_type -> saas.accounts.v1.SolutionMintRegistrationResponse - 81, // 102: saas.accounts.v1.ModuleCapabilitiesService.MintModuleWorkContext:output_type -> saas.accounts.v1.ModuleMintWorkContextResponse - 82, // 103: saas.accounts.v1.ModuleCapabilitiesService.MintModuleOperationContext:output_type -> saas.accounts.v1.ModuleMintOperationContextResponse - 83, // 104: saas.accounts.v1.ModuleCapabilitiesService.MintSourceOperationContext:output_type -> saas.accounts.v1.ModuleMintSourceOperationContextResponse - 42, // 105: saas.accounts.v1.ModuleCapabilitiesService.PublishEvent:output_type -> saas.accounts.v1.ModulePublishEventResponse - 45, // 106: saas.accounts.v1.ModuleCapabilitiesService.Subscribe:output_type -> saas.accounts.v1.ModuleSubscribeResponse - 78, // 107: saas.accounts.v1.ModuleCapabilitiesService.Unsubscribe:output_type -> google.protobuf.Empty - 48, // 108: saas.accounts.v1.ModuleCapabilitiesService.ListSubscriptions:output_type -> saas.accounts.v1.ModuleListSubscriptionsResponse - 50, // 109: saas.accounts.v1.ModuleCapabilitiesService.ReplayEvents:output_type -> saas.accounts.v1.ModuleReplayEventsResponse - 80, // [80:110] is the sub-list for method output_type - 50, // [50:80] is the sub-list for method input_type - 50, // [50:50] is the sub-list for extension type_name - 50, // [50:50] is the sub-list for extension extendee - 0, // [0:50] is the sub-list for field type_name + 2, // 26: saas.accounts.v1.ModuleAuditEventTypeDeclaration.retention:type_name -> saas.accounts.v1.ModuleAuditEventRetention + 28, // 27: saas.accounts.v1.ModuleDeclareAuditEventTypesRequest.types:type_name -> saas.accounts.v1.ModuleAuditEventTypeDeclaration + 70, // 28: saas.accounts.v1.SubjectVisibilityGrant.expires_at:type_name -> google.protobuf.Timestamp + 31, // 29: saas.accounts.v1.ModuleListSubjectVisibilityResponse.grants:type_name -> saas.accounts.v1.SubjectVisibilityGrant + 36, // 30: saas.accounts.v1.FetchDatasourceFilesRequest.files:type_name -> saas.accounts.v1.DatasourceFileRef + 3, // 31: saas.accounts.v1.DatasourceItemReaders.basis:type_name -> saas.accounts.v1.DatasourceItemReadersBasis + 38, // 32: saas.accounts.v1.DatasourceFileHeader.provenance:type_name -> saas.accounts.v1.DatasourceProvenance + 39, // 33: saas.accounts.v1.DatasourceFileHeader.readers:type_name -> saas.accounts.v1.DatasourceItemReaders + 40, // 34: saas.accounts.v1.FetchDatasourceFilesFrame.header:type_name -> saas.accounts.v1.DatasourceFileHeader + 72, // 35: saas.accounts.v1.ModulePublishEventRequest.envelope:type_name -> saas.events.v1.EventEnvelope + 4, // 36: saas.accounts.v1.ModuleSubscribeRequest.delivery:type_name -> saas.accounts.v1.EventDelivery + 4, // 37: saas.accounts.v1.ModuleSubscription.delivery:type_name -> saas.accounts.v1.EventDelivery + 70, // 38: saas.accounts.v1.ModuleSubscription.created_at:type_name -> google.protobuf.Timestamp + 45, // 39: saas.accounts.v1.ModuleSubscribeResponse.subscription:type_name -> saas.accounts.v1.ModuleSubscription + 45, // 40: saas.accounts.v1.ModuleListSubscriptionsResponse.subscriptions:type_name -> saas.accounts.v1.ModuleSubscription + 70, // 41: saas.accounts.v1.ModuleReplayEventsRequest.since:type_name -> google.protobuf.Timestamp + 5, // 42: saas.accounts.v1.ReadableSourceCollection.grant_disclosure:type_name -> saas.accounts.v1.MetadataDisclosure + 53, // 43: saas.accounts.v1.ReadableSourceCollection.read_grants:type_name -> saas.accounts.v1.ReadableCollectionGrant + 54, // 44: saas.accounts.v1.ReadableSourceCollection.sync:type_name -> saas.accounts.v1.CollectionSyncProvenance + 70, // 45: saas.accounts.v1.ReadableCollectionGrant.expires_at:type_name -> google.protobuf.Timestamp + 6, // 46: saas.accounts.v1.CollectionSyncProvenance.stage:type_name -> saas.accounts.v1.SourceSyncStage + 70, // 47: saas.accounts.v1.CollectionSyncProvenance.at:type_name -> google.protobuf.Timestamp + 5, // 48: saas.accounts.v1.CollectionSyncProvenance.requester_disclosure:type_name -> saas.accounts.v1.MetadataDisclosure + 70, // 49: saas.accounts.v1.CollectionSyncProvenance.requested_at:type_name -> google.protobuf.Timestamp + 52, // 50: saas.accounts.v1.ListReadableSourceCollectionsResponse.collections:type_name -> saas.accounts.v1.ReadableSourceCollection + 61, // 51: saas.accounts.v1.ModuleCapabilitiesService.ExchangeDelegatedReadAudience:input_type -> saas.accounts.v1.ModuleExchangeDelegatedReadAudienceRequest + 62, // 52: saas.accounts.v1.ModuleCapabilitiesService.ExchangeDelegatedOperationAudience:input_type -> saas.accounts.v1.ModuleExchangeDelegatedOperationAudienceRequest + 59, // 53: saas.accounts.v1.ModuleCapabilitiesService.CheckWorkContextRecordAccess:input_type -> saas.accounts.v1.CheckWorkContextRecordAccessRequest + 55, // 54: saas.accounts.v1.ModuleCapabilitiesService.ListReadableSourceCollections:input_type -> saas.accounts.v1.ListReadableSourceCollectionsRequest + 57, // 55: saas.accounts.v1.ModuleCapabilitiesService.PlaceRecord:input_type -> saas.accounts.v1.ModulePlaceRecordRequest + 7, // 56: saas.accounts.v1.ModuleCapabilitiesService.EnqueueJob:input_type -> saas.accounts.v1.ModuleEnqueueJobRequest + 9, // 57: saas.accounts.v1.ModuleCapabilitiesService.ClaimJobs:input_type -> saas.accounts.v1.ModuleClaimJobsRequest + 11, // 58: saas.accounts.v1.ModuleCapabilitiesService.HeartbeatJob:input_type -> saas.accounts.v1.ModuleHeartbeatJobRequest + 13, // 59: saas.accounts.v1.ModuleCapabilitiesService.AckJob:input_type -> saas.accounts.v1.ModuleAckJobRequest + 14, // 60: saas.accounts.v1.ModuleCapabilitiesService.NackJob:input_type -> saas.accounts.v1.ModuleNackJobRequest + 15, // 61: saas.accounts.v1.ModuleCapabilitiesService.NotifyUser:input_type -> saas.accounts.v1.ModuleNotifyUserRequest + 17, // 62: saas.accounts.v1.ModuleCapabilitiesService.NotifyOrgAdmins:input_type -> saas.accounts.v1.ModuleNotifyOrgAdminsRequest + 21, // 63: saas.accounts.v1.ModuleCapabilitiesService.RequestApproval:input_type -> saas.accounts.v1.ModuleRequestApprovalRequest + 23, // 64: saas.accounts.v1.ModuleCapabilitiesService.GetApproval:input_type -> saas.accounts.v1.ModuleGetApprovalRequest + 25, // 65: saas.accounts.v1.ModuleCapabilitiesService.CancelApproval:input_type -> saas.accounts.v1.ModuleCancelApprovalRequest + 26, // 66: saas.accounts.v1.ModuleCapabilitiesService.EmitAuditEvent:input_type -> saas.accounts.v1.ModuleEmitAuditEventRequest + 29, // 67: saas.accounts.v1.ModuleCapabilitiesService.DeclareAuditEventTypes:input_type -> saas.accounts.v1.ModuleDeclareAuditEventTypesRequest + 32, // 68: saas.accounts.v1.ModuleCapabilitiesService.ListSubjectVisibility:input_type -> saas.accounts.v1.ModuleListSubjectVisibilityRequest + 34, // 69: saas.accounts.v1.ModuleCapabilitiesService.FetchDatasourceBlob:input_type -> saas.accounts.v1.FetchDatasourceBlobRequest + 37, // 70: saas.accounts.v1.ModuleCapabilitiesService.FetchDatasourceFiles:input_type -> saas.accounts.v1.FetchDatasourceFilesRequest + 73, // 71: saas.accounts.v1.ModuleCapabilitiesService.MintModuleRegistration:input_type -> saas.accounts.v1.ModuleMintRegistrationRequest + 74, // 72: saas.accounts.v1.ModuleCapabilitiesService.MintSolutionRegistration:input_type -> saas.accounts.v1.SolutionMintRegistrationRequest + 75, // 73: saas.accounts.v1.ModuleCapabilitiesService.MintModuleWorkContext:input_type -> saas.accounts.v1.ModuleMintWorkContextRequest + 76, // 74: saas.accounts.v1.ModuleCapabilitiesService.MintModuleOperationContext:input_type -> saas.accounts.v1.ModuleMintOperationContextRequest + 77, // 75: saas.accounts.v1.ModuleCapabilitiesService.MintSourceOperationContext:input_type -> saas.accounts.v1.ModuleMintSourceOperationContextRequest + 42, // 76: saas.accounts.v1.ModuleCapabilitiesService.PublishEvent:input_type -> saas.accounts.v1.ModulePublishEventRequest + 44, // 77: saas.accounts.v1.ModuleCapabilitiesService.Subscribe:input_type -> saas.accounts.v1.ModuleSubscribeRequest + 47, // 78: saas.accounts.v1.ModuleCapabilitiesService.Unsubscribe:input_type -> saas.accounts.v1.ModuleUnsubscribeRequest + 48, // 79: saas.accounts.v1.ModuleCapabilitiesService.ListSubscriptions:input_type -> saas.accounts.v1.ModuleListSubscriptionsRequest + 50, // 80: saas.accounts.v1.ModuleCapabilitiesService.ReplayEvents:input_type -> saas.accounts.v1.ModuleReplayEventsRequest + 78, // 81: saas.accounts.v1.ModuleCapabilitiesService.ExchangeDelegatedReadAudience:output_type -> saas.accounts.v1.IssuedWorkContext + 78, // 82: saas.accounts.v1.ModuleCapabilitiesService.ExchangeDelegatedOperationAudience:output_type -> saas.accounts.v1.IssuedWorkContext + 60, // 83: saas.accounts.v1.ModuleCapabilitiesService.CheckWorkContextRecordAccess:output_type -> saas.accounts.v1.CheckWorkContextRecordAccessResponse + 56, // 84: saas.accounts.v1.ModuleCapabilitiesService.ListReadableSourceCollections:output_type -> saas.accounts.v1.ListReadableSourceCollectionsResponse + 58, // 85: saas.accounts.v1.ModuleCapabilitiesService.PlaceRecord:output_type -> saas.accounts.v1.ModulePlaceRecordResponse + 8, // 86: saas.accounts.v1.ModuleCapabilitiesService.EnqueueJob:output_type -> saas.accounts.v1.ModuleEnqueueJobResponse + 10, // 87: saas.accounts.v1.ModuleCapabilitiesService.ClaimJobs:output_type -> saas.accounts.v1.ModuleClaimJobsResponse + 12, // 88: saas.accounts.v1.ModuleCapabilitiesService.HeartbeatJob:output_type -> saas.accounts.v1.ModuleHeartbeatJobResponse + 79, // 89: saas.accounts.v1.ModuleCapabilitiesService.AckJob:output_type -> google.protobuf.Empty + 79, // 90: saas.accounts.v1.ModuleCapabilitiesService.NackJob:output_type -> google.protobuf.Empty + 16, // 91: saas.accounts.v1.ModuleCapabilitiesService.NotifyUser:output_type -> saas.accounts.v1.ModuleNotifyUserResponse + 18, // 92: saas.accounts.v1.ModuleCapabilitiesService.NotifyOrgAdmins:output_type -> saas.accounts.v1.ModuleNotifyOrgAdminsResponse + 22, // 93: saas.accounts.v1.ModuleCapabilitiesService.RequestApproval:output_type -> saas.accounts.v1.ModuleRequestApprovalResponse + 24, // 94: saas.accounts.v1.ModuleCapabilitiesService.GetApproval:output_type -> saas.accounts.v1.ModuleApproval + 79, // 95: saas.accounts.v1.ModuleCapabilitiesService.CancelApproval:output_type -> google.protobuf.Empty + 79, // 96: saas.accounts.v1.ModuleCapabilitiesService.EmitAuditEvent:output_type -> google.protobuf.Empty + 30, // 97: saas.accounts.v1.ModuleCapabilitiesService.DeclareAuditEventTypes:output_type -> saas.accounts.v1.ModuleDeclareAuditEventTypesResponse + 33, // 98: saas.accounts.v1.ModuleCapabilitiesService.ListSubjectVisibility:output_type -> saas.accounts.v1.ModuleListSubjectVisibilityResponse + 35, // 99: saas.accounts.v1.ModuleCapabilitiesService.FetchDatasourceBlob:output_type -> saas.accounts.v1.FetchDatasourceBlobChunk + 41, // 100: saas.accounts.v1.ModuleCapabilitiesService.FetchDatasourceFiles:output_type -> saas.accounts.v1.FetchDatasourceFilesFrame + 80, // 101: saas.accounts.v1.ModuleCapabilitiesService.MintModuleRegistration:output_type -> saas.accounts.v1.ModuleMintRegistrationResponse + 81, // 102: saas.accounts.v1.ModuleCapabilitiesService.MintSolutionRegistration:output_type -> saas.accounts.v1.SolutionMintRegistrationResponse + 82, // 103: saas.accounts.v1.ModuleCapabilitiesService.MintModuleWorkContext:output_type -> saas.accounts.v1.ModuleMintWorkContextResponse + 83, // 104: saas.accounts.v1.ModuleCapabilitiesService.MintModuleOperationContext:output_type -> saas.accounts.v1.ModuleMintOperationContextResponse + 84, // 105: saas.accounts.v1.ModuleCapabilitiesService.MintSourceOperationContext:output_type -> saas.accounts.v1.ModuleMintSourceOperationContextResponse + 43, // 106: saas.accounts.v1.ModuleCapabilitiesService.PublishEvent:output_type -> saas.accounts.v1.ModulePublishEventResponse + 46, // 107: saas.accounts.v1.ModuleCapabilitiesService.Subscribe:output_type -> saas.accounts.v1.ModuleSubscribeResponse + 79, // 108: saas.accounts.v1.ModuleCapabilitiesService.Unsubscribe:output_type -> google.protobuf.Empty + 49, // 109: saas.accounts.v1.ModuleCapabilitiesService.ListSubscriptions:output_type -> saas.accounts.v1.ModuleListSubscriptionsResponse + 51, // 110: saas.accounts.v1.ModuleCapabilitiesService.ReplayEvents:output_type -> saas.accounts.v1.ModuleReplayEventsResponse + 81, // [81:111] is the sub-list for method output_type + 51, // [51:81] is the sub-list for method input_type + 51, // [51:51] is the sub-list for extension type_name + 51, // [51:51] is the sub-list for extension extendee + 0, // [0:51] is the sub-list for field type_name } func init() { file_saas_accounts_v1_module_capabilities_proto_init() } @@ -4801,7 +4880,7 @@ func file_saas_accounts_v1_module_capabilities_proto_init() { File: protoimpl.DescBuilder{ GoPackagePath: reflect.TypeOf(x{}).PkgPath(), RawDescriptor: unsafe.Slice(unsafe.StringData(file_saas_accounts_v1_module_capabilities_proto_rawDesc), len(file_saas_accounts_v1_module_capabilities_proto_rawDesc)), - NumEnums: 6, + NumEnums: 7, NumMessages: 56, NumExtensions: 0, NumServices: 1, diff --git a/module/services/accounts/code/pkg/infra/postgres_audit_registry.go b/module/services/accounts/code/pkg/infra/postgres_audit_registry.go index de4bf1b63..6c659816c 100644 --- a/module/services/accounts/code/pkg/infra/postgres_audit_registry.go +++ b/module/services/accounts/code/pkg/infra/postgres_audit_registry.go @@ -54,22 +54,23 @@ func (s *PostgresStore) SyncAuditEventTypes(ctx context.Context, defs []business // reads it from the composed event catalog for a code-owned type. Writing // it here is what lets one table answer "may this leave the platform" for // the whole registry, instead of a reader having to know which half a - // name came from. + // name came from. The retention class is projected for the same reason. tag, err := q.Exec(ctx, ` - INSERT INTO audit_event_types (name, namespace, version, category, owner, visibility, payload_schema, deprecated, updated_at) - VALUES ($1, $2, $3, $4, $5, $6, $7, FALSE, NOW()) + INSERT INTO audit_event_types (name, namespace, version, category, owner, visibility, retention_class, payload_schema, deprecated, updated_at) + VALUES ($1, $2, $3, $4, $5, $6, $7, $8, FALSE, NOW()) ON CONFLICT (name) DO UPDATE SET namespace = EXCLUDED.namespace, version = EXCLUDED.version, category = EXCLUDED.category, owner = EXCLUDED.owner, visibility = EXCLUDED.visibility, + retention_class = EXCLUDED.retention_class, payload_schema = EXCLUDED.payload_schema, deprecated = FALSE, updated_at = NOW() - WHERE NOT starts_with(audit_event_types.owner, $8)`, + WHERE NOT starts_with(audit_event_types.owner, $9)`, string(d.Type), d.Namespace, d.Version, string(d.Category), d.Owner, - d.EffectiveVisibility(), d.PayloadSchemaJSON(), + d.EffectiveVisibility(), string(d.Retention), d.PayloadSchemaJSON(), business.SolutionAuditOwnerPrefix) if err != nil { return err diff --git a/module/services/accounts/code/pkg/infra/postgres_declared_audit_relay_test.go b/module/services/accounts/code/pkg/infra/postgres_declared_audit_relay_test.go index 491407a4e..ae2de21a3 100644 --- a/module/services/accounts/code/pkg/infra/postgres_declared_audit_relay_test.go +++ b/module/services/accounts/code/pkg/infra/postgres_declared_audit_relay_test.go @@ -35,6 +35,7 @@ func admitDeclaredType(t *testing.T, visibility string) business.EventType { Namespace: namespace, SolutionID: namespace, Visibility: visibility, + Retention: business.RetentionSecurity, Fields: []business.PayloadField{{Name: "count", Kind: business.FieldInt}}, } require.NoError(t, testStore.WithControlPlane(testCtx, func(ctx context.Context) error { @@ -51,6 +52,7 @@ func admitDeclaredType(t *testing.T, visibility string) business.EventType { })) require.NotNil(t, admitted) require.Equal(t, visibility, admitted.Visibility) + require.Equal(t, business.RetentionSecurity, admitted.Retention, "and the retention class it was declared with") return declared.Type } diff --git a/module/services/accounts/code/pkg/infra/postgres_solution_audit_events.go b/module/services/accounts/code/pkg/infra/postgres_solution_audit_events.go index 803d5bfe5..91aa91a2c 100644 --- a/module/services/accounts/code/pkg/infra/postgres_solution_audit_events.go +++ b/module/services/accounts/code/pkg/infra/postgres_solution_audit_events.go @@ -55,13 +55,13 @@ func (s *PostgresStore) ListAuditEventNamespaceOwners(ctx context.Context, names func scanDeclaredAuditEventType(row pgx.Row) (*business.DeclaredAuditEventType, error) { var ( - name, namespace, owner, visibility string - schema []byte + name, namespace, owner, visibility, retention string + schema []byte ) - if err := row.Scan(&name, &namespace, &owner, &visibility, &schema); err != nil { + if err := row.Scan(&name, &namespace, &owner, &visibility, &retention, &schema); err != nil { return nil, err } - declared, err := business.DeclaredAuditEventTypeFromSchema(business.EventType(name), namespace, owner, visibility, schema) + declared, err := business.DeclaredAuditEventTypeFromSchema(business.EventType(name), namespace, owner, visibility, retention, schema) if err != nil { return nil, err } @@ -89,7 +89,7 @@ func (s *PostgresStore) GetDeclaredAuditEventType(ctx context.Context, eventType return declared, err } declared, err := scanDeclaredAuditEventType(s.getQueryExecutor(ctx).QueryRow(ctx, - `SELECT name, namespace, owner, visibility, payload_schema FROM audit_event_types + `SELECT name, namespace, owner, visibility, retention_class, payload_schema FROM audit_event_types WHERE name = $1 AND starts_with(owner, $2)`, string(eventType), business.SolutionAuditOwnerPrefix)) if errors.Is(err, pgx.ErrNoRows) { @@ -102,7 +102,7 @@ func (s *PostgresStore) GetDeclaredAuditEventType(ctx context.Context, eventType // type. func (s *PostgresStore) ListDeclaredAuditEventTypes(ctx context.Context) ([]business.DeclaredAuditEventType, error) { rows, err := s.getQueryExecutor(ctx).Query(ctx, - `SELECT name, namespace, owner, visibility, payload_schema FROM audit_event_types + `SELECT name, namespace, owner, visibility, retention_class, payload_schema FROM audit_event_types WHERE starts_with(owner, $1) ORDER BY name`, business.SolutionAuditOwnerPrefix) if err != nil { return nil, err @@ -122,19 +122,21 @@ func (s *PostgresStore) ListDeclaredAuditEventTypes(ctx context.Context) ([]busi // PutDeclaredAuditEventType inserts a declared type or replaces the one its // owner already holds. The conflict update is conditioned on the owner, so a // row another producer holds is left alone and the write is refused rather -// than silently transferring the type. +// than silently transferring the type. Visibility is never rewritten (it is +// fixed at admission); the retention class is, since admission lets it rise. func (s *PostgresStore) PutDeclaredAuditEventType(ctx context.Context, declared business.DeclaredAuditEventType) error { tag, err := s.getQueryExecutor(ctx).Exec(ctx, ` - INSERT INTO audit_event_types (name, namespace, version, category, owner, visibility, payload_schema, deprecated, updated_at) - VALUES ($1, $2, $3, $4, $5, $6, $7, FALSE, NOW()) + INSERT INTO audit_event_types (name, namespace, version, category, owner, visibility, retention_class, payload_schema, deprecated, updated_at) + VALUES ($1, $2, $3, $4, $5, $6, $7, $8, FALSE, NOW()) ON CONFLICT (name) DO UPDATE SET + retention_class = EXCLUDED.retention_class, payload_schema = EXCLUDED.payload_schema, deprecated = FALSE, updated_at = NOW() WHERE audit_event_types.owner = EXCLUDED.owner`, string(declared.Type), declared.Namespace, business.DeclaredAuditEventVersion, string(business.CategorySolution), business.SolutionAuditOwner(declared.SolutionID), - declared.Visibility, declared.PayloadSchemaJSON()) + declared.Visibility, string(declared.Retention), declared.PayloadSchemaJSON()) if err != nil { return err } diff --git a/module/services/accounts/code/work_audit_test.go b/module/services/accounts/code/work_audit_test.go index 02e313c66..f5c54b36f 100644 --- a/module/services/accounts/code/work_audit_test.go +++ b/module/services/accounts/code/work_audit_test.go @@ -59,9 +59,9 @@ func setBigQuerySwap(t *testing.T) { t.Setenv("AUDIT_SINK", "bigquery") t.Setenv("AUDIT_BIGQUERY_PROJECT", "example-project") t.Setenv("AUDIT_BIGQUERY_DATASET", "audit") - t.Setenv("AUDIT_ARCHIVE_BUCKET", "example-audit-archive") + t.Setenv("AUDIT_ARCHIVE_URL", "gs://example-audit-archive") t.Setenv("AUDIT_DEPLOYMENT_ID", "acme-prod-1") - t.Setenv("AUDIT_CONTENT_DETAIL_RETENTION_DAYS", "90") + t.Setenv("AUDIT_CONTENT_RETENTION_DAYS", "90") t.Setenv("AUDIT_RELAY_BATCH_SIZE", "") t.Setenv("AUDIT_RELAY_MAX_WAIT", "") } @@ -76,13 +76,18 @@ func TestConfiguredAuditSinkBigQuery(t *testing.T) { require.Equal(t, &bigQueryAuditSwap{ project: "example-project", dataset: "audit", - archiveBucket: "example-audit-archive", + archive: auditArchiveLocation{scheme: "gs", bucket: "example-audit-archive"}, deploymentID: "acme-prod-1", contentDetailRetention: 90 * 24 * time.Hour, batchSize: business.DefaultAuditRelayBatchSize, maxWait: business.DefaultAuditRelayMaxWait, }, sink.bigQuery) + t.Setenv("AUDIT_ARCHIVE_URL", "gs://example-audit-archive/") + sink, err = configuredAuditSink() + require.NoError(t, err, "a trailing slash still names only the bucket") + require.Equal(t, "example-audit-archive", sink.bigQuery.archive.bucket) + t.Setenv("AUDIT_RELAY_BATCH_SIZE", "200") t.Setenv("AUDIT_RELAY_MAX_WAIT", "750ms") sink, err = configuredAuditSink() @@ -94,31 +99,36 @@ func TestConfiguredAuditSinkBigQuery(t *testing.T) { func TestConfiguredAuditSinkBigQueryNamesEveryMissingSetting(t *testing.T) { t.Setenv("AUDIT_SINK", "bigquery") for _, name := range []string{ - "AUDIT_BIGQUERY_PROJECT", "AUDIT_BIGQUERY_DATASET", "AUDIT_ARCHIVE_BUCKET", - "AUDIT_DEPLOYMENT_ID", "AUDIT_CONTENT_DETAIL_RETENTION_DAYS", + "AUDIT_BIGQUERY_PROJECT", "AUDIT_BIGQUERY_DATASET", "AUDIT_ARCHIVE_URL", + "AUDIT_DEPLOYMENT_ID", "AUDIT_CONTENT_RETENTION_DAYS", } { t.Setenv(name, "") } _, err := configuredAuditSink() require.EqualError(t, err, "AUDIT_SINK=bigquery requires AUDIT_BIGQUERY_PROJECT, AUDIT_BIGQUERY_DATASET, "+ - "AUDIT_ARCHIVE_BUCKET, AUDIT_DEPLOYMENT_ID, AUDIT_CONTENT_DETAIL_RETENTION_DAYS") + "AUDIT_ARCHIVE_URL, AUDIT_DEPLOYMENT_ID, AUDIT_CONTENT_RETENTION_DAYS") setBigQuerySwap(t) - t.Setenv("AUDIT_ARCHIVE_BUCKET", " ") + t.Setenv("AUDIT_ARCHIVE_URL", " ") _, err = configuredAuditSink() - require.EqualError(t, err, "AUDIT_SINK=bigquery requires AUDIT_ARCHIVE_BUCKET", "a blank value is a missing one") + require.EqualError(t, err, "AUDIT_SINK=bigquery requires AUDIT_ARCHIVE_URL", "a blank value is a missing one") } func TestConfiguredAuditSinkBigQueryRejectsInvalidSettings(t *testing.T) { for _, tc := range []struct { name, value, want string }{ - {"AUDIT_CONTENT_DETAIL_RETENTION_DAYS", "0", "AUDIT_CONTENT_DETAIL_RETENTION_DAYS"}, - {"AUDIT_CONTENT_DETAIL_RETENTION_DAYS", "90d", "AUDIT_CONTENT_DETAIL_RETENTION_DAYS"}, - {"AUDIT_CONTENT_DETAIL_RETENTION_DAYS", "-1", "AUDIT_CONTENT_DETAIL_RETENTION_DAYS"}, + {"AUDIT_CONTENT_RETENTION_DAYS", "0", "AUDIT_CONTENT_RETENTION_DAYS"}, + {"AUDIT_CONTENT_RETENTION_DAYS", "90d", "AUDIT_CONTENT_RETENTION_DAYS"}, + {"AUDIT_CONTENT_RETENTION_DAYS", "-1", "AUDIT_CONTENT_RETENTION_DAYS"}, {"AUDIT_DEPLOYMENT_ID", "acme/prod", "AUDIT_DEPLOYMENT_ID"}, {"AUDIT_DEPLOYMENT_ID", "..", "AUDIT_DEPLOYMENT_ID"}, - {"AUDIT_ARCHIVE_BUCKET", "gs://example-audit-archive", "AUDIT_ARCHIVE_BUCKET"}, + {"AUDIT_ARCHIVE_URL", "example-audit-archive", "like gs://"}, + {"AUDIT_ARCHIVE_URL", "s3://example-audit-archive", `scheme "s3" has no archive writer`}, + {"AUDIT_ARCHIVE_URL", "https://storage.example/archive", `scheme "https" has no archive writer`}, + {"AUDIT_ARCHIVE_URL", "gs://example-audit-archive/audit", "names a bucket and nothing else"}, + {"AUDIT_ARCHIVE_URL", "gs://user@example-audit-archive", "names a bucket and nothing else"}, + {"AUDIT_ARCHIVE_URL", "gs://Example_Archive", "not a valid bucket name"}, {"AUDIT_RELAY_BATCH_SIZE", "0", "AUDIT_RELAY_BATCH_SIZE"}, {"AUDIT_RELAY_BATCH_SIZE", "5001", "AUDIT_RELAY_BATCH_SIZE"}, {"AUDIT_RELAY_MAX_WAIT", "5", "AUDIT_RELAY_MAX_WAIT"}, diff --git a/module/services/accounts/proto/saas/accounts/v1/module_capabilities.proto b/module/services/accounts/proto/saas/accounts/v1/module_capabilities.proto index f6d799cb7..64b530e33 100644 --- a/module/services/accounts/proto/saas/accounts/v1/module_capabilities.proto +++ b/module/services/accounts/proto/saas/accounts/v1/module_capabilities.proto @@ -370,6 +370,23 @@ enum ModuleAuditEventVisibility { MODULE_AUDIT_EVENT_VISIBILITY_EXTERNAL = 2; } +// How long the full details of a declared type's events are kept once a +// warehouse is the audit store of record (ADR 0009). The envelope — who, which +// organization, what type, which target, when — and a hash of the details are +// kept for the compliance window either way; the class decides where the +// details go. +enum ModuleAuditEventRetention { + // Unset means CONTENT, the class of a type that says nothing. + MODULE_AUDIT_EVENT_RETENTION_UNSPECIFIED = 0; + // Full details kept for the shorter, configured content window; the locked + // archive holds only the envelope and the hash. + MODULE_AUDIT_EVENT_RETENTION_CONTENT = 1; + // Full details kept for the compliance window, in the warehouse and the + // locked archive alike: authentication, permission and role changes, admin + // actions, data exports, configuration changes. + MODULE_AUDIT_EVENT_RETENTION_SECURITY = 2; +} + // ModuleAuditEventTypeDeclaration is one audit event type a composed module // declares for itself. message ModuleAuditEventTypeDeclaration { @@ -388,6 +405,11 @@ message ModuleAuditEventTypeDeclaration { // subscribed and widening it would start sending out facts a tenant's // endpoint never agreed to receive under this name. ModuleAuditEventVisibility visibility = 4 [(buf.validate.field).enum.defined_only = true]; + // The type's retention class. It only grows: a later declaration may raise + // CONTENT to SECURITY, and one that would lower SECURITY — including by + // leaving this unset — is refused, because that would shorten how long the + // type's evidence is kept without anyone deciding so. + ModuleAuditEventRetention retention = 5 [(buf.validate.field).enum.defined_only = true]; } // ModuleDeclareAuditEventTypesRequest declares the audit event types a composed diff --git a/module/services/frontend/code/package-lock.json b/module/services/frontend/code/package-lock.json index 606968f29..5e5dcd2af 100644 --- a/module/services/frontend/code/package-lock.json +++ b/module/services/frontend/code/package-lock.json @@ -13,7 +13,7 @@ "dependencies": { "@base-ui/react": "^1.3.0", "@bufbuild/protobuf": "^2.11.0", - "@codefly-dev/saas-sdk": "0.3.15", + "@codefly-dev/saas-sdk": "0.3.16", "@codefly-dev/saas-ui": "0.11.4", "@codefly-dev/ui": "0.11.4", "@codefly/saas-plugin-contract": "2.3.0", @@ -15384,7 +15384,7 @@ }, "packages/saas-sdk": { "name": "@codefly-dev/saas-sdk", - "version": "0.3.15", + "version": "0.3.16", "license": "Apache-2.0", "dependencies": { "@bufbuild/protobuf": "^2.11.0", diff --git a/module/services/frontend/code/package.json b/module/services/frontend/code/package.json index e64b5826e..0afd71679 100644 --- a/module/services/frontend/code/package.json +++ b/module/services/frontend/code/package.json @@ -36,7 +36,7 @@ "dependencies": { "@base-ui/react": "^1.3.0", "@bufbuild/protobuf": "^2.11.0", - "@codefly-dev/saas-sdk": "0.3.15", + "@codefly-dev/saas-sdk": "0.3.16", "@codefly-dev/saas-ui": "0.11.4", "@codefly-dev/ui": "0.11.4", "@codefly/saas-plugin-contract": "2.3.0", diff --git a/module/services/frontend/code/packages/saas-plugin-manifest/src/data-graph.ts b/module/services/frontend/code/packages/saas-plugin-manifest/src/data-graph.ts index f311aa96c..1a8fac8a5 100644 --- a/module/services/frontend/code/packages/saas-plugin-manifest/src/data-graph.ts +++ b/module/services/frontend/code/packages/saas-plugin-manifest/src/data-graph.ts @@ -89,6 +89,16 @@ export interface EventFieldDeclaration { pii?: boolean; } +/** + * How long the host keeps a declared type's full details once a warehouse is + * its audit store of record: `security` for the compliance window, `content` + * (the default) for the shorter content window. The class only ever grows: a + * type admitted as `security` stays so. + */ +export type EventRetention = "security" | "content"; + +const EVENT_RETENTIONS: readonly EventRetention[] = ["security", "content"]; + /** * A named audit event a metric can filter on. `name` is graph-local (metrics * reference it); `type` is the audit event type it binds to, e.g. @@ -108,6 +118,8 @@ export interface EventDeclaration { name: string; type: string; description?: string; + /** The declared type's retention class; only with `fields`. */ + retention?: EventRetention; fields?: readonly EventFieldDeclaration[]; } @@ -356,7 +368,11 @@ function validateEventField( function validateEvent(value: unknown): asserts value is EventDeclaration { assertGraph(isObject(value), "event must be an object"); - assertExactKeys(value, ["name", "type", "description", "fields"], "event"); + assertExactKeys( + value, + ["name", "type", "description", "retention", "fields"], + "event", + ); assertLogicalId(value.name, "event name"); assertGraph( typeof value.type === "string" && EVENT_TYPE.test(value.type), @@ -366,8 +382,19 @@ function validateEvent(value: unknown): asserts value is EventDeclaration { value.description, `event '${String(value.name)}' description`, ); - if (value.fields === undefined) return; + if (value.fields === undefined) { + assertGraph( + value.retention === undefined, + `event '${String(value.name)}' declares a retention class but no fields; only a declared type has one`, + ); + return; + } const context = `event '${value.name}'`; + assertGraph( + value.retention === undefined || + EVENT_RETENTIONS.includes(value.retention as EventRetention), + `${context} retention '${String(value.retention)}' must be one of ${EVENT_RETENTIONS.join(", ")}`, + ); const type = value.type as string; assertGraph( DECLARED_EVENT_TYPE.test(type) && diff --git a/module/services/frontend/code/packages/saas-plugin-manifest/src/index.ts b/module/services/frontend/code/packages/saas-plugin-manifest/src/index.ts index 08916d74c..dfeb9d8d8 100644 --- a/module/services/frontend/code/packages/saas-plugin-manifest/src/index.ts +++ b/module/services/frontend/code/packages/saas-plugin-manifest/src/index.ts @@ -37,6 +37,7 @@ export type { EventDeclaration, EventFieldDeclaration, EventFieldKind, + EventRetention, Metric, MetricAggregation, MetricBucket, diff --git a/module/services/frontend/code/packages/saas-plugin-manifest/test/data-graph.test.ts b/module/services/frontend/code/packages/saas-plugin-manifest/test/data-graph.test.ts index 8b1becc9c..6a0d8eb85 100644 --- a/module/services/frontend/code/packages/saas-plugin-manifest/test/data-graph.test.ts +++ b/module/services/frontend/code/packages/saas-plugin-manifest/test/data-graph.test.ts @@ -363,6 +363,14 @@ describe("assertDataGraph — declared event types", () => { ).not.toThrow(); }); + it("accepts a declared retention class", () => { + for (const retention of ["security", "content"]) { + expect(() => + assertDataGraph(declaring({ ...itemCreated, retention })), + ).not.toThrow(); + } + }); + it("keeps an event without fields exactly as before", () => { expect(() => assertDataGraph(declaring({ name: "viewed", type: "acme.viewed" })), @@ -437,6 +445,18 @@ describe("assertDataGraph — declared event types", () => { }, /pii must be a boolean/, ], + "an unknown retention class": [ + { ...itemCreated, retention: "permanent" }, + /retention 'permanent' must be one of security, content/, + ], + "a misspelt retention class": [ + { ...itemCreated, retention: "Security" }, + /retention 'Security' must be one of/, + ], + "a retention class on an event that only binds": [ + { name: "viewed", type: "acme.page.viewed", retention: "security" }, + /declares a retention class but no fields/, + ], "an unknown key on a field": [ { ...itemCreated, diff --git a/module/services/frontend/code/packages/saas-sdk/generated/contract/catalog.codefly.json b/module/services/frontend/code/packages/saas-sdk/generated/contract/catalog.codefly.json index 4ef3b0fcc..d12667441 100644 --- a/module/services/frontend/code/packages/saas-sdk/generated/contract/catalog.codefly.json +++ b/module/services/frontend/code/packages/saas-sdk/generated/contract/catalog.codefly.json @@ -10,7 +10,7 @@ "kind": "protobuf", "package": "saas.accounts.v1", "path": "contract.binpb", - "digest": "sha256:91c5b2d9cc104f254ad5de886ebd1204323051d63e007c8992c64f97124f1086", + "digest": "sha256:75f36458044f80ca65eaccf1588f093e67f5e4cea48d534444077a65940d1645", "services": [ { "name": "APIKeyService", diff --git a/module/services/frontend/code/packages/saas-sdk/generated/contract/contract.binpb b/module/services/frontend/code/packages/saas-sdk/generated/contract/contract.binpb index f4b26e939e72e4235bd2b1586c70ea84caab7867..ab689635876dc824d1e4fdbc1dd44e398a19c7a8 100644 GIT binary patch delta 20185 zcmZ{sd6*SN*5>ccjEvj>A)zR$DB!XqP{=BuZQJdJg6#_0ZR^)dHIz_eS>;gB>eus4 zf}$v^+Q?M_1zbRoeYq&^sHi9^iU^9J*b0KQvMDGs?>P~fQB(8GAKee{$%vCDezBd1 zyyXWA&YHFAtZ@^v6RZjL1Tn#xASbvJyb1p5VA-wK)=h;Kw+7EQ%kD4KP6~>%Euvw8 zKe}>EWzCqX+L~5QF59h1-~8u=pd{aJN)U7$nmjt+dv>sMd)8{29cs#VcqJQj%wIVr zXj^z{agwEx z$}tsH)pfbr8*&v8!>cPsSB}Vyx$Wl4u64Oe3cW4|Q|`vfo5$p8Ylc_mDuUcCu#Bo5 zTUVK@s$*9b)X-erm|B#Cr_q(eYe$cWbKTCnaA>Y~@7`yf-Ya)C-dI`l_sZ(pn=5l) zjGK|WWmIiX{9o1ZQ9-VD^vH^us{g7W{$#8e!y)MBxiJ-^M^=s@RawJzD{4j{+))*E zqv&(%8&x}cOy%gFx#6`nH&l%rOPz~<(ookh6oKm6;Wt)}2+-BT zM^*j3GB>KWdISbVP4#UYblPzQo^TWTG|(&p{lz-HyAG5X-ELc5q=lmk(F^GG1<1f;B#;x9ZAS(c7E{jU{^fZ z_8TV!Z7SHbS#X-F=TX6Enk|AT@Tj6U(QiVw=F@CDXcqX14Pi~OMU(m_xSDEPPdC&CDJ2VGDv&RQV?9C^g><25|OyqwgT-P z(Lhxs4<*U4OCs^Ed+n~}iM$Z5?zMaONi<$|uif{r`ehbg7O}s6f6*mu_?u~4Wx)p) zw*~cQ7OH*o6I(PN_Bw^w=?aY*K4;n+%|>VnRfYgx`cq+S+>l{;3QRn`q^ePLQ^jq$(G^v7 zSXlIukQ-B*OICD@NQ}|i8oeakQaiSK1P1yoaFP_^wXdpXcr}(=EaHE=4Kty#;_vit zEFE>Zu~>D+RSvJ9#hRBIHJImSLkP^Wk9CliOE}@> zdv20^DcW-A+#)gZnJI`yVxTp}Ehp zdt4hV5ST@Tp*KywLVf;}^h= z_bs%e<~(H}F=ZCoEuPj>h73Gq7TQHm>nWpc=>4aXDWgqrwTPw+O)PD~pjd1RH8@*Z zSqzFLwiO3K6AmgHNyz24ql3w!_!2wh1rZ)-S}n1A^iH0_{}Q`je;sNTp)RopEKftF zi4~FU%;jmQY=fJnCa$bktmecrTQmzk%K~C?B-h%RXC62i?u^&kUQd2uv6Vr%g=AVn zvf`c+P*@QPcGxT_i{ZM%0M%!lc1rsb~{ZB$S_g6=eiAz7EH6$%pG->|W*z2_j{ zl1P4JXSNrnJK?)l{E_Xc-YwyPw@?o z@Qg&0N+Y?~&U}N7Ymv<5=PgGXThhpFT=Qwc-(IFTU~-6o*?HmTU(>oCDchfR?`&*_&;I$ zY6?KY{|UQA5zkLZ%+C{c%aeM3l7W|!6L#@QJwLS#W9_G8DbXgl`kAKeYNX<#ERqvM z=5CQKeZtCJR9SenY1kthPY|A(ZDma1PGPw`JRu_irNV+jly?fR5jP4FuJ075sg1qN zqBh=Ts-i4Zu`Wvuyi0frAj+cZT|xuApvAjMwJ0-OEsz<~cxW4Aj7i9X?gu=_BC=Y9Wl3?wwOX`Ms{@|G*=o_b3wH}7;#w`b z7SgylJ0x4TLK+v_;ARa4K|49sQyb>mrb09i7Sfb#7s(ey=C#7y)?wFd{DSaQ1nsz6 zUQD_LG7?Z~K0(3ni%GXYBD@!qZfVERUouru7E0yRc*s(xEs&`ClF%DM+>WQ>%cd^4 zLa7BE4^j7J;j1|YiMlU~vNp`Don>PdzhbK5A(Wc7c!;X6BpnUOPch-BciLG3RW}GL zR>uNpmPuVo?KYUYl-X^dL89t9sQapUtx{_7XlFU7`f4)0AW`*InqEV3F=0pY4UyS` z)$qdHg*bcF)#i)u3r@=qYa8~voZ@{$IJ~nvHUocCV5i2uO)?TtoKKMgr8h;dGbkWP zBD_i9O#VkMODg|i+whuBc!ZLhgg#N*%yYBAMmCyD;^F4}W$nV6&Ui#ZsM831QaJAm zt5-0aPPEXF`Q7cp2aEA<6n$Ux=^M91pjh!ji%D+#WpKnZK_?_^0xn z!d3GGWymT@6Av`ZcF`VnEmy%P+m&zKF}$vPm+;hXp5t@FE>WW5&sj9OKBv)j9akZh zpWiY3&)nxiA5&2VZ~0u{n94bD`$&E*GT-Lgc0v;TbuyvbGylE9YRM2GBLSrdL!qgA zMX}0ZAmMSZ=%iBQ_KbL6GQ=T~(FjXzrcf;P(H#<1_X!-`sq5nQ%=8;m7r>!Z>+leD zze%PCB)c}9TA+qhSv;Pi z?g7zCjUY(WJs>)C435wlqC+GPiOdg}mjBA;@|#xR5_D*n@Wc3!@Y?h0-ND0deppz? zu+&nZ(C~Lycl>W>#8UV#o2N< z(ScDP6$nBXM%|;LSrlxcdrlF_QzCPRQ+TIqcxg7KHlD(Oqq2_R*N$`4>?0WoDDG}j z=#_C!JDvuR)T84abP_E#kgU{71ckEW9W|fuke=S|Slm{UtT5$s{8pt~mjwrCY3LZq zNls>}lP%q8+h3)vzE|ucl^CaoVXR2~XjDilZ-1AmM$A z({pIh6gN;LC;BC1DZdKdEYT^F)1Ax%PL^&-MJKqL?&##c6E|uGHEJ1MsX#{p^4!w7 z7T+)zTK5dQWY3@B=nNbM@a`E7GH{9v67QbjwEQh^<0MtfW;n&a3APg1&d3*?%)-JQ z<>A%Yc($Vtlb!j^bLh>Kc0fiFYM!IgP-w(C$I-bhBwjnm(b*p)(u_Gy={!fJp=99n zXO7c)o}mpJ;qBJRS= z?^0724S`BxqJ}I@x*ugxcPVv0jk+$p{4O(9@eoR#-0=`qmnFRe$xqQcswyqN%N;9L z{JKy+1*!N!q3-g8A0+B7Cw{7~gSsmmeMQlQwqz)E8wQ1{D;#}80g0+B9NbXQ6zqxw z*vYKbQ?M&aUb))w)I!^p7v(h$(omZFkVuRjz5PL<@EWH`EkclRyT-xx_XoJ`7RhIv z%zCU^L$GMA4O-ar%%H3jOkl0j2U&C?st53fdq-V`yHJ!mQw;H4kWV# zl9gJNpjhfd1|&QkaH1qAT80idI?XS~-ht8wsRcvOk)*}J{K>P!-<2P9{G^UVor9*1 zi$dQKps(k0@PYUIhhk%b#z-uV0fQBsm`0H?BHa_Zv zYA1sgp&oTQ@<&uE1RqBo9M)(gms>twdrYT#Bv~cz^HKGf(^{>dc!;XUoDQ9W=PBKE zB6-rujF;KcrC1B_5uH9CURYH(yt=lou=F=!$81cLPC5>^*l~EU{X$76rInEQW~wI* zDU6U`XqF%@NE%(I9Gvk~S*19IriUnd%27EpB#owVQe|}{Ss|fr&7PEQ-yzs&W5V=` zHM&3JVol<3WZQ-iqxBHFP)(o8%=-RaU*s~jY%PM7+g zqnAbd^1XyaRmBj8Qu_oRqUya;@5_*=dapdaN%vp_jTITmyv)ogJbyv>n`{S(vU%z7 zV&F(-`;bIe55Ta1jKo9vHiN=Go$+sAc}VJxmOLnCOPu2Fqe}>AT$<+V28C0L@oyBJ zExUFLrqH*FGa~tv%&aaHUldNZ;-{qEs?XrvZjr=_K#hRJhc1aj5y63i!$nfhYDhF< zk<_#L3_d(8PTCKN4`R|%=Qb#oW$K~|5>*#V3_5aLbO!hK66&(w^J@|7lKh!N!fVTy zNS*Ve4C*YAI_Ez_WnWaKKUcxSW%>0(!ZXX4NqudMGGtA9=nOveu8>$maPfoYpmSH` z+g=i0AFq%_>N_NisIo%#RNpSn;7(m>8ih`UN*|(8D`lDbl7X_Qx>EK~U%Sq*Bnq!e zh9M*?m4-s0>ME(fQ?-Of-Bl8IC%E_$y(76+W}eA6`CWKbyjJSZbiH{B7NoxYhs2jR ziEll`5DFd((o<_WBpenJT98=$3bN&3KAn-ANUK#r7GKPnm9`+_f*f#meN)WE&bdgx zg)WoUC5`G$oeVW7dB{@FEj&ckby80)NQAIXwz^Pz=j?(@lw5$(W{EzLd{$t)}5x-d22xugkw zs0mOiy@O(@vr$O9IXy=1{U)UoeIxmf%zRn+_od-rJANmbHhsBgHYYwIaW^P+S_cK6 zo25RLLn55bQlH8pfn~GQr*e`it=la1sk|>QpIgZJUi!ifjT}i@ALZ9x8GapaN!Bvh z;dl%2rV}6}^0_Ut)kjjDPRYRe+!k5#kyNKsZNqf?KT&p~ik+ zp0V2`WfGKZLPrvEhjf@SG#b54>Rb~N4!232YeFKnZ8F-C#zr^q_`%9H*?dPD8`}`u z$0jy5!PRyt)Zs7g%L~CLhCQA_{Z@xT%l#)(ACgg)pXy^6B-noQTtQc_*56*iyF$e-jz0!e%63yek#j$rOjj; znz_?7lTC2-8FhgQgc7XXCJzP6B4y|OQf4B5C?(mmL+@AKpYjw zZdtk~4TNn73NvCggO)wAsm(^!*mF0a>7zNRENM8 zgm74%bwO&1)d)Q-2VJH8P$GWBgn$ZAS0oP+!jWX@>Cdz9NV4=mB7`Hd#pOB#H3wlS zx=g<>nSn=Sf7Lzx&*yJsG)a!gb1&r~F(8u1WacM5BnIG($C52;0Qb?4lu!L91_Wdz ze#~ETZTRQ%AEm2qu3&_>AElm@19-VRK@Fl^8WP{y^INYCuWx@M=}s6?01Y1)$%oy{({8r(gCIAovbu8kn7YF8>%yy@_+i%@ zi1*rJpc~0qZuU`^mV6fyJ3p>DwTZ^!K;|?z2^$jEkO?O!ggrOm1c?adCY&G<_S}RM zNyRCin}E_buBAZ@@5)14?w{)><>9}|=!7}!gYn{&Xi^@#&TO(z5k+)zimFR_~Zf1VS z36CVqOI@^jA$ED0K_|!+ZuTYDD$QqegKni?rRL`h4`;@bP~U_EV2x|X zWkK^GOp3W#43>f`2H=OML25+hb*|`LzcPC5S=a7U--N^Tde`n&-xMI*a_#Q*O<^RS zckLdjsRV6k6p0sHyLYftp+Z9Rnw$MFe??{ZY5bb&@D~aMLua1X+-`r+cxWKK=Kift z1n)TZ4j4@O;pqU=vXMyB-R*5sL2&3bp~o_ohW`0*XTh1)G54utC8lpg7W?B2XNq zQXABmK#QE8UF&Wy-}A=s<@je;^7@FjKXLrEm8ihc`ldhQPC8I(p z6erzqqL+*cwgJU21_hho`IJGyCZHJSnNh(uLNVS;M+Mt};&v|`6>I{EJG^vMun8z8 zcYn*idU1_YY`Vn&0C0Pz5o`U8W&CDcOz@kIWjn(+7WL!RUl7FsS5#zS6HzP=|3 z5D$5M25S%!ARhL_<0%NKKs@Y)kEbBm1`x9h2sXjxB#)x6Dv~?t^%>p3s}> ze5W^qPV_4Pw3_bA*LW&_B8kY?c=$PkehMaQydLLjCb)u^Yd!H&%EVJl)_UPfDHFEA zYr!yK6FjdoOuU$?0?IR$MGD`S}`z4-l_(=P2jMx-Yj7ofq27!U>iWZY1+dkfOyM*U=u)WG9cIl5N|iA z2oUd3sVkX75Xtv3OLpd~>cT7H_szN)AdL6Tx)}h(`)1t?0Ah+QTM*_}GA86F_V?AlL*DI~r63h)<|g z4TA_Hxf>w?GfN*2(;@{~rD+{@ws-gfPB0KpFwa_XbEK06FA| zpHm=>6p%w+_;U(`Z2)rE0AUk6A2C4K1RzK2jSRLCh+}4Cuni!7FzsOzK>TPxun8cJ z8xU*)h!YJe0>n>LYAAzfOusnutjT_U)-B`J z4PneOAWj2_Sq8*u05RJaPo^MFQy^yh;gcx{wgJQ(1AvCV+U%fM63qJl>!pKs-UE@YnA&B%0zdw#c`h$xr)7I454@Oa978 z4-m&9zs zb*aHNKD96RagC@hHIxyRSNQ1~$~NFxWm?H5;8|_(unBn97(8qOp0)KWwo{iH1u8a6 z@o5&x=X~o8-!7empV|J;>|37{KKtJx{)%_AFZtItXgmGs?`FT0KM0aU zu$%qg@`p*1AhX%O=+AmcMfmdjt}oi)qmHDfqP^DY-VBfBd%oC_f zZ1FVuy7ayw$K~K|iy_A*1oMHPCar8Da330QYy-Hhru}RJxQ`4tHUZo=1#Tc+*0Bxb zK5kGGiGpWa^GNRTt$$-Q9kcV7b_&Z16UK-C%Eo(q$v<|4r{?gr$8UY6hSVHU z@A1#$pXt$4Al%~*uF!+4IT(HEi-Ren=8Dmmet0lt#5N%MmqElPyyz>1D5fT`4R8C} zFk%xi+G`lG4UG1gma++qzA=p01V;PK;9?UH9cWMyh+2G0#lBNBu0TPPGq{BV4b?AB}pilYVvo8bAF z87*uAi682%NNnR3>A0U}b;^j!C(Me(Ht_k`G?Pu>bJFl(6Zrh%%W}Pivk81o`Lb7P z()Wnt0`NJZW*`1D26qQmzA)~t@V0DxcOd!K9O%Fz1af!KE?I|)z~}B@NOBEb1U~l! zVrI&xNb$KR2xq2z*akk6f;5|96Fg52(rkuJ;4>wlY(}lvY-2vt0Eu`T+5HO!2{AsLu|p#f7H#gdOlB z^B`TPTOyFzLD_%k@zWA~W(ODiRga&R;4>!>i&8!<6`wglxG3erHt>1G@L>}?KWg}} z34G?(3WhfUz~q~XIR@L6E^unBw?8a^!(K2H&!=kYBC z!vuePzA~_0E)1C*K9`MG1~R!cC`KSFgZAfZKE>d(GPo?+@r%J{RUlqU`4lTYtAg;Q zln>j$XSLzOCU{Ox) z;Pa8;!zS?AX85oPd_FdOT8UOFBic@U)UgtOU%xZ3_T`)2k5lE&Aib3mty~G{d=`kkDV-8U=d&Q(o6=z$=MX@3fcUCGMQl`GQz><`j=+ERcx5Qt+bh*AaOKoA~FL9h)VzBM4&1kVQz2sQ!4zw3cu8-e)F zfM6Ryd~e#rCV)6(K(GlQ4jT|`0*E6GDgwk&Ds?H(7yP#Zj^i(RdNc>_*unij zhWX|*!)xM+q2%AQVy!4c923JEy{V9nuMdI!-ygwzuHjua{ zOt)q>!Skdr-J0115|cx7(ZDt$F(phV2irhms%a3LKw_F9!6uNHZb+~RB<^ic5hU)T zQuu)h+Kd0*zyo$@&C6Hi@w?6k!gQ5rgD@Tl(^aAkKs*qpt3(@sm>G(>DTp=-#LO_9 zn}T2)K;#VwHo^0Q1_YY`;-OIHFL*F)_Sc6^ixkWK|FZ+)x%b;QzdDRd>yuoRf2Wp00Yk>Q_s@UHuKU zOD^0p@4{gt@*}Jf_6RY;86iiwBfJs*(%_{(S>sj~7X2xBy=8uKVMt9-k}vEW6KpLl z+&nkWfD3T|w+^8ft9X#0aq~&!Dng)(2 zvT{x&zPBtd_$VI;(vgHbZ6&p*gT3+hmftijXj8!!EraH&o<{|v)0POLz@v)ZXulbG ztxj8Z&@%8-&x*tkmT1<{LZ}}t(LzVgqX}oI2~D^Kc~)1`#TYU2to_z$Vbncim6WE^ zBUyHpj-IDRaEbCObqPyE;;dx_IyzbdH7LC(NrqigiF=*3x>l$1N`yLV_3o2u-0ZB? z_t*Mn7H$@CynesZC2YhSXC4?PmpL@3s zYa?t6gdJ&D-xbUg7`x&~K4Ir-?R@zZQI+f)6Lc&bv%=H)4o}#g3R2ADIKc+QIvPjl zNJ36dS~%hCc!KRW=bORDy(idFD;`Ej4C4g5^<-NOBN;r56YQeNwi-t5p+!&GdKk5Z zP!nwoBMr8;(EmxcP{4| zB$S0pZ$k@UP8z982~YVscBG~YBtp%h>GCN}hSEsBZ0D8~uJOX(S@FxZ*OqTl$}?aQ z%>Y_Pkdc7;d3q6dTx9!gQi+g@?6&Pvi5a-auDX^-5>;sqV7wRE?FZ@-G*XLgtBnqh zXiz1qlBKl(cobJJU4jNn*$(43)4PdWws**vAf)0f^4Ch(#O50N- zU&d`;m9`x+5>P+mdqd%FtI`<^iRi0n2GhME(e_n#>x;SVB+>R_vWkAHOAy}GCOD!& zT^n4l>eK$R+V;!S!ic`wu2fLVEKEAcb^~>E-d_-XwOw^huw0^d<&oTM=e8D(H487W z2W?IlWI4~BE$N6uMgmIhEKtC?#SYu^O(D^SEp~@)sYDyL*geh-#?T$wMDk-h_bK>o z5>=BI2j$89%CLE1Nfhq1;*V{w1J`Q94ELwqgp35#W$8uSVZZH1sl*-j+eO8xL@)Q- zC6{t9QB^bCZX03)l?(&%57@q143LO_z;0c{OBWJL_kdk| zNH1M7c$Ya~mmJbdS9_RspQO8twh-zdt>&c`U{pl%3p@8UV9d3ut{$B9UK|t`<`joT z`S=UlQ;V;HQG97zF3(oTNIjxkE-0>M^0PLnG)Y;j6|&BIqd5sx@PW1oly)=rN%-cQP3JC{g^F zP@B8<(A3dFZ|;?`Du-B)3hbD)Oe&d6jcEub0MxjwAw;G|=-#0;D%S|~j*`vEgcd_#GjT92_#5N6GbyL2{L#{ritPinuPX1qBaGgErgm*AZSF| zT9nw#6nHq%l587ex=F}7JtVk}J!Yl|E7Ia<*Gy5Vb_-lZu$iKB7ak!m5T9>tp zlN<87)n)Bs4`F6eBj}+|_0+R5EohM3;}H`6|{PjZQmDpz1)Lm%m3e;UllSI{ZQ1?Z1Tcy-K(#~>F^~H3hL89u5w9;-3EC*9yvB<5!;<&Qv z%0WX1Jv=07(kZ1!y{Ija1b3ORx(ENp zWl1H|JB7D*!X=bkCaQbtl5~@Lfk)g!ToM=SlcLVygPkq>k1nY^8Zn97Yr^UoOh>Lt z8Zw#RIh<00|3=Z*M4!IFWSUD=kz6fu?-cIrLOc9w;i-|S;s@v&fkc~@IAkQCRKf&> z_-oRs4vF|{XsQ$4Do#q*(z;PqQ5H&NLbwRnYlTkgAo*&f*vHTawN~itpo*8^Iui;_ zhf)tATtujKCKTaWCnA-B;VL4n6FLK{;+eKySTeYq`-6*673DWn{btA?9{lZ~hyMJl zhXyD2cMGqtUN2m=W>ATXm8T(T%54yM>iv>qV8(4o_IC^Ks@@l5iC0}@HV387yp zNU9{@gwU@P9W8ntI!R9vnmdrJZVO^{YwGMt(N1kgxQgf}g?`1;cXJm{{ZbIR@e&H|j z@hC^XU3TU>k9Kq-3K>bLF^)<^q0!FKj?Vueaof?3&i^2hU5s|h$2ckxC4;k`(N3E& zj!Hzehrxc-(TQm1Sd~Mp#~ht;=wiTM*4Y3|KZR1jC+Y?y9x^0w&jd!f( zJX7!}WTl>BP-xkBr-eGqf<(0O4yN|MVI;alQcQAk)03+6!e7Rd98c|PU3erWQ*UYR zK}G^fC52FkIN9+-zB?o$PIj89!=Ek;Dlt`27D{cyxX991k1t46O&ooI5qII8c8aNs zjzFbtqK-^ShaY88cM1(Z&AKkU(@r&2aS=+rzvCjRPEAJ#lCNTPR8`t(r#V)v_;sN) z3R3ZdLfvU8KSx9>-AXybKCeYaM-L0g0-$4vs8n33f%^>*QuT zY6*5l$?Il1p4wi!@>V>{LH0@fA(0O|`bh_c!n2$rwFyBY>?{XQx;qiJTO^-$a&wbQ zF9`3r|7l0Rt#;!9eI~^N840LsxD!wa_>AMLDFTUr&!k<1#Cm?lsl1xkGfB0cpK;n= z75oLaJO^3yU#S0v-3-_&iB%gP33w7VRBl`1>J`C>8XBICwIyHhsg6pWnP(T6a z3r?umL!#;nPFuwj5)FI7LE2bP+enW{E^%_NI{ESiR@Dtdewz%P60|Hd{YhAvkC!-l zkM6-eUz+wDG7?Z~kA}hxmO6TmhQtk)Iy$X~M9-HxdXFZldcM@rdvp(;?8_)Py&pkF z5~|+OFKRFlf0?5*3rNIY=IHZnNNfwsoTy%J3uN$?wajT%uXhFQVI{nh?h4vMr~>T@ zG!J|5i`epX9`?AK_5etg8{sONvD~RpGX@fMm(z@)iFj@#S2($qda%w#$rX-%7M#oQ z-gJ<^3`YMTBk^W3{HNjXI=|_7YH`6tv^SmNl3*d7BvnW9Jty}OM&(Xbqq0|SLLod}SqyCt0n)x5cEHC1sDDs3@sE?Z5DX>-|XT1*M^ zHf%0xE3Kx@1yZGeP^i1j@zpsIBbD3;MB45|X`^Vb z+3x7Hy&4Y)l-@x-xE2FRTI@*HT^3$fy~FX-Iudnum^v;BeMp3E<0>vxa`ENi?L9wq ziqw8yZMmrOq0{Xgo`uzxhX(JZ1}~%30UENf^78Qfe7x5Q)#C~-n!4BN$nSJi2r>3L zcpIacTy6Qd?LOM|Y3qYzmEO-s)qPGIwN2t8s_t_-bP8UlOj1Phkdr%#srMo_guz1x z{T8pMSN{Itdmk9`aN&$S< zWLi*CxasO}LH@39(#JthUNOhh`3D&ZDD_GPg_|9BeD(GP2`0xK{r1(9SIkMO+66cv zkp()*tpmfZ@yYaA3Kw-wrq9xzyi!h?+oAbTYL?(4LYzvcJ0z-}N~e2IUMb(2s<;TH z9xS+ss^2=k+BjlpzKorNM!qK>XME@2{gg%?5?QIEPePzj_dCZ``yC{ze&^`ZjGh)f zhQCj@KS);Uu>pmu-=~jZNL2maIj33oU=7V0Cz6lI+~~r{Yr~)AJ5bq2q{G_+jwXoh zaEU=5fC&N#m3;E^uuo_FH?RzsddMA~9-}1Q&&Jbv12j%alZ&nk$Cco}QFN5-+ASDM z-zs`V@+p~{S=f7BIL3;flKL6liywXyCH4mDKP0|AqALBl3NB7fhTjyPS3Ongb7GVsSLzY!#jmrq61xe`c+edTZf$b#rtt2# zRu-x41174}%5&8Rj9xsb(@m!^s8HEObZWY+P@gkU7FDOq9_njXFH55E3~Hc4iP-_A zvQAt?)frNMrz(a<-5C>q!*2&_5{4z#zH0@h;vg8_0skDO@*U14FHMC+bcJbBvE_7}* zC+$>k8f2)e(u*wh+QLOtog?+qfWgKAf`6l{#gEj09Bw^djKSm7$t!xQM!QWhXw`6Og#eT-mpuE=*l`Htj+m>H?I? z_MlkmG!)XtE9bFvDf%cnH zs8!IH*X}Aw*##w_(2;~(D;?$yjZUwUI{$=3z*SP`pO9$VDjBWK+D0es_!MU41wN&N^!anycG;t{BZK%kp%$u=9ZZVJ`Sw`9eJ ztefnio9j(C*+Qs)&=9DGP|~&0Gz8BMs7+Gm`+fOQvr+046cV*JN~EBwAr2a{QI>7W z8seyiY?S4jvWBpShP-VW!WKfkLk*$j+7C&#%zdDjYrlN^NW3TY8&5x82HUBT2Pq4K zMk-jy-xc1MzkGYTKf^`9?dkpui9y&di_{>HR109cEWS8cK!=L`Be_%N_9oBY9nOz; zO8ujd{(QGx>A*uq0_qOt1%-jxl@1If?zxKwhG;_Kp1Wk5+j(OoNwW`5#V>d&Xa_8I z$&1vlZ?r`l<1Ts4Z8{F^D!WY+a8IaTq!(H0A?eS{b2kl%Y65MHyJgvJx(RA=!YRL1 zH-Q~AVYj^Sip+}D4BaiSx<$vKlw*%+0xCe=lwL#=_M}@+e_n-q(ya#)P1qw_->93Q z)*zgs8}$9s6}U(CSHsi)a{eYpi)4?y^m?8W10uOk=04F=VgT;AFMW6o;4%7GBD4B; zObEzGe4Ir0g!fc`EM0YE1ryOemU>YR;O*`Jb%>tVkoeY~%(y4KyZwQ5IANm70UFL1 zXo8%FgIt;WOeST&3ZIJ)OHXC==keW-$hazuI2=VX4kly6C_$t<7!Tj zXm*XOIgwPH;u=?j(jM2+pw8zp9%F8ZJ3y&*gNrORCrDHs<7!TjxZxPL%WVvWq*@VU z+^)AW=kxi|bmLez?{E$VjT2EfuTmr^L?7!)_2LVO=wn^|;(Ifm*Yoj|mpKfAtabCxxmJ0Sue$0__uej2|x94{~4~` zmfD|Mh&%Ma8d`YE zp6lA(8d?F8FW2tg&Y#Lod1A?$o4cV!h=4=3z|=KVp+FyVmO&sk~rPFT4Kh87cNaYLOwu z79uY;q}T$fC9a)3^gtNJ%M2X8=A!&71`d0GQ*Yp~1vtwM9JT=GRRf1DzUnB_>2i*ZTY7_z$54qx4hC(P5hurX3hJrny_|%|a3y}{S6l?*-kp?K(BNRss z3ig2FGt(osfZ}t5f-Ru|5Lq}kB$j(E5y z`6UZImuSXtuLU1ak_3q1UY~&)gan8Yo_IV1Ar*)bUif$hf<1s3>1F!_TZlZ$%k~Mj z0AjQ!ll?=(mhq!rwm0xKbpA0f+Z)(I=WD!dZ(s|ZALF50>RHAXIzP5iMRa}~l^V#M zN1im^vu1d9`2|%22M>B^=x+uO8k#)(hw#F9yeIjS9lA~fGTv*&$2}xLV!U_$Et-T2 z5)(WzJtN^N5)-^|dPaghka)_FU<;8a8WL=Q#3aw`nCwvtCL1d3fl6XJ#}=qeF;v(B zm8pgbTc9$nQAJRxrBW9&734LuFeRQ(7XC5(X*|nQi4%J6p&7G094xD}gU-@tc|9)G zAaEWpXM5tg41}jZ%=W_PG7#(mM4bV_79!6vAlL$kryGD^k3c+QK(Gf8b4`EP0*Ge~ z2(|!Xo&muYK+JDc5g-;&scRU7AIXIPu{=5QKjGE!LQgi;J@?U!gTf-Qhp)Bpr~1Y)rP!5%;?G5ui+AeI^s zYyreF1A;Apc%@NAfGAL@n;Aq9$=3j4ebVR8;Z5;tUbb5XXvS+^wp#`O@tT+ImH|Ml z@Wfjgh(Li@;e~HyAlL(l*9{1^5cv%Qf-Qh}vjGV92*gSQf<1s(W%|PwK&&<(*aC<( z1_WCGv9?h~fLKSR9%K+;c3(1_WD(yvcxI3n1RbZn?RSjN7$9r`$leAsgFOPV&&&+=0OBLl zAGQGEV*`RMfY@(9umunY8dU^{PpH(b45BIhvdptic*(tg3vZ4Ods%AL6zw?dWvNwD zkT~oOxI#~jrXX>|6UQ?WO%;hFUU)ns!5&B)H6++Ve+?199W zrbBFj#8-v{TOe`FkYEcWzHU?zB)*|i_}q)Rk2BpxsYnX=>F>f%w)7|L&)$6?*{jodLlXB7bi{umuo5GyuUK zf%wsYU=JWpoBpr`5N8YswgBR+0l^kP4D++Jge`!0#0LnKV>ZW;i|t$ElhZMB%#nTu zqB)u|($7FN2Z)h=2BJAYjPk{k8HnZz#3(;}G6TULK#caY)QT-ce$>xWE4BdQF<&NI zh9R}8@w3#5uc7l}Odr@n=g0b4YQ+{hKaRSkQY*I5`NtboMCYHNQZv=NMvF*J^sT3T zyL?=}>dL|QJ}?MB_}wAH=E*~kgpb4%eaT-NF)!$E11I{Gm+E=Z0yrl6*A3D0q6KhF z^2MACM+=2xk{`~=aIgm)lMN2G5IHe8*aD6z!12)E@s2*t$GM?8z~E~%KWhDK?_>{% zW|*F`1w=CqBDR2NmO;c85Y28>5s2!jRIPqTZyCvFeQR+N3=gNp&-#+TvV%xVwBuR7 zf`2$f5=5T$uYEuhX$d0pe6c7a(ozwb=ZA|jBJ6?4d_#mSL|$NsumvK|fk?*@m>|!a z3BuRV`xi_n*h23Yn%=X8-oI#i&lYV4(ado4bpty#LjTXGywueLvX@$0K@-O3G;n7uq+vE=%q^DFXka^b^J2En@6q$GZa7RXlJ%IVA z0mBw<^qv9}Q^VQA-QG83*dj8U4H@=8W{c?`TOhO5kYNjCwwdY07Qk$8R1siWe?Y~) zQmd3(U9GK$MQep(j~^b* zP_PFSdkqS<5P6@OBW!`hM?SsJsnb*T`1EwYpCxX}MCAi!^I;Es4w`PV1wMxiAGW~f zQ(so={h2NBIqb`xnblUEo*p4S->db9KcN5Gw;l=X^21j0_>Dn%;df)hyYlhZzT{tT zU?_{wq_6#U={{5hQeXSmrsvE>AoYzeh6U*pyGW7x#t(-D=@XkhkUDNiv4zMd3@Nri z>SO~_>@lfteXpP1rj?1x-x*Tufz%I%6k8zmqano>NS!vM*aE3DhE$O#Q>4xksizew z{JHw5z?xLp@p#w)zabB@eYzM885LAqpl4Ds_>2m!__dx%#o#kK5EC;##fs19Ae@-- zVGn#B4YCJ0TZsHvkUhxR0-u_|%p~@h&)6WFNyo6buh5IXkwGJ!oUHrj# zpZl=X#KiCoD}KeGDn*-K3Dlpb18FHxy%OB^7d@3qfvOOQwHc~Xg{lyQYco{r0ad+0 z#TFtjH>lVGs#hDJVvkX+FjGmHsQkJ?#U4<-X;85RR4WZCwt#AtLB$qOtv0AiMX6e9 zYY3Hkoy4EkuMey($&LhXlk0=*2(ApxSRZ6ZaAlygKFE&X%0TBIf!LhUDN}U*5rmsF zI_!bY21AD}MBZrVumw7AGaZ#0vPVy^cY-W?W)C1X84zp%#JdIrTLAG->Y>V>*#e07 z8db#8>wPMvULEj9_}lP%&5x4vrsCCMTacZ~mZKTlg6veb93Zv@*{N(fKx_}hz6?aU z0^_hlg11Bed{2(}P;hXKJBK>Vu#2=)lXhXw?D0I}2bhb@5EWk9e65W5Wswg6&J zqly5rmr7mFO9%f}z<&Id^p`>M#I*3|@%|utt!{&M><_Zn>NX&;KN#?Hy}%G3< = /*@__PURE__*/ enumDesc(file_saas_accounts_v1_module_capabilities, 1); +/** + * How long the full details of a declared type's events are kept once a + * warehouse is the audit store of record (ADR 0009). The envelope — who, which + * organization, what type, which target, when — and a hash of the details are + * kept for the compliance window either way; the class decides where the + * details go. + * + * @generated from enum saas.accounts.v1.ModuleAuditEventRetention + */ +export enum ModuleAuditEventRetention { + /** + * Unset means CONTENT, the class of a type that says nothing. + * + * @generated from enum value: MODULE_AUDIT_EVENT_RETENTION_UNSPECIFIED = 0; + */ + UNSPECIFIED = 0, + + /** + * Full details kept for the shorter, configured content window; the locked + * archive holds only the envelope and the hash. + * + * @generated from enum value: MODULE_AUDIT_EVENT_RETENTION_CONTENT = 1; + */ + CONTENT = 1, + + /** + * Full details kept for the compliance window, in the warehouse and the + * locked archive alike: authentication, permission and role changes, admin + * actions, data exports, configuration changes. + * + * @generated from enum value: MODULE_AUDIT_EVENT_RETENTION_SECURITY = 2; + */ + SECURITY = 2, +} + +/** + * Describes the enum saas.accounts.v1.ModuleAuditEventRetention. + */ +export const ModuleAuditEventRetentionSchema: GenEnum = /*@__PURE__*/ + enumDesc(file_saas_accounts_v1_module_capabilities, 2); + /** * DatasourceItemReadersBasis says where an item's readers came from. * @@ -2114,7 +2165,7 @@ export enum DatasourceItemReadersBasis { * Describes the enum saas.accounts.v1.DatasourceItemReadersBasis. */ export const DatasourceItemReadersBasisSchema: GenEnum = /*@__PURE__*/ - enumDesc(file_saas_accounts_v1_module_capabilities, 2); + enumDesc(file_saas_accounts_v1_module_capabilities, 3); /** * EventDelivery is the ordering guarantee a subscription requests, mirroring @@ -2144,7 +2195,7 @@ export enum EventDelivery { * Describes the enum saas.accounts.v1.EventDelivery. */ export const EventDeliverySchema: GenEnum = /*@__PURE__*/ - enumDesc(file_saas_accounts_v1_module_capabilities, 3); + enumDesc(file_saas_accounts_v1_module_capabilities, 4); /** * MetadataDisclosure separates "the host withheld this from you" from "there is @@ -2175,7 +2226,7 @@ export enum MetadataDisclosure { * Describes the enum saas.accounts.v1.MetadataDisclosure. */ export const MetadataDisclosureSchema: GenEnum = /*@__PURE__*/ - enumDesc(file_saas_accounts_v1_module_capabilities, 4); + enumDesc(file_saas_accounts_v1_module_capabilities, 5); /** * SourceSyncStage names which occurrence CollectionSyncProvenance.at records. @@ -2206,7 +2257,7 @@ export enum SourceSyncStage { * Describes the enum saas.accounts.v1.SourceSyncStage. */ export const SourceSyncStageSchema: GenEnum = /*@__PURE__*/ - enumDesc(file_saas_accounts_v1_module_capabilities, 5); + enumDesc(file_saas_accounts_v1_module_capabilities, 6); /** * @generated from service saas.accounts.v1.ModuleCapabilitiesService diff --git a/module/services/frontend/code/packages/saas-sdk/package.json b/module/services/frontend/code/packages/saas-sdk/package.json index 5f9ade62c..55dd02b7f 100644 --- a/module/services/frontend/code/packages/saas-sdk/package.json +++ b/module/services/frontend/code/packages/saas-sdk/package.json @@ -1,6 +1,6 @@ { "name": "@codefly-dev/saas-sdk", - "version": "0.3.15", + "version": "0.3.16", "description": "Typed Connect client, gateway-bound facades, and audit data-graph tooling for the saas-starter public API.", "repository": { "type": "git", diff --git a/module/services/frontend/code/packages/saas-sdk/src/schema.ts b/module/services/frontend/code/packages/saas-sdk/src/schema.ts index 0a3928aa7..c39d855de 100644 --- a/module/services/frontend/code/packages/saas-sdk/src/schema.ts +++ b/module/services/frontend/code/packages/saas-sdk/src/schema.ts @@ -84,6 +84,12 @@ export interface EventDeclaration { name: string; type: string; description?: string; + /** + * The declared type's retention class: `security` keeps its full details + * for the compliance window, `content` (the default) for the shorter content + * window. Only with `fields`; it only ever grows. + */ + retention?: "security" | "content"; fields?: readonly EventFieldDeclaration[]; } diff --git a/module/services/frontend/code/src/gen/saas/accounts/v1/module_capabilities_pb.ts b/module/services/frontend/code/src/gen/saas/accounts/v1/module_capabilities_pb.ts index e5fad3b34..3dd0c6d8e 100644 --- a/module/services/frontend/code/src/gen/saas/accounts/v1/module_capabilities_pb.ts +++ b/module/services/frontend/code/src/gen/saas/accounts/v1/module_capabilities_pb.ts @@ -22,7 +22,7 @@ import type { JsonObject, Message } from "@bufbuild/protobuf"; * Describes the file saas/accounts/v1/module_capabilities.proto. */ export const file_saas_accounts_v1_module_capabilities: GenFile = /*@__PURE__*/ - fileDesc("CipzYWFzL2FjY291bnRzL3YxL21vZHVsZV9jYXBhYmlsaXRpZXMucHJvdG8SEHNhYXMuYWNjb3VudHMudjEiXgoXTW9kdWxlRW5xdWV1ZUpvYlJlcXVlc3QSGAoGdGVuYW50GAEgASgJQgi6SAVyA7ABARIpCgNqb2IYAiABKAsyFC5zYWFzLmpvYnMudjEuTmV3Sm9iQga6SAPIAQEidgoYTW9kdWxlRW5xdWV1ZUpvYlJlc3BvbnNlEhgKBmpvYl9pZBgBIAEoCUIIukgFcgOwAQESQAoLZGlzcG9zaXRpb24YAiABKA4yIy5zYWFzLmpvYnMudjEuSm9iRW5xdWV1ZURpc3Bvc2l0aW9uQga6SAPIAQEiygEKFk1vZHVsZUNsYWltSm9ic1JlcXVlc3QSLgoFcXVldWUYASABKAlCH7pIHHIaEAEYgAEyE15bYS16XVthLXowLTlfLi1dKiQSHQoJd29ya2VyX2lkGAIgASgJQgq6SAdyBRABGP8BEhgKBWxpbWl0GAMgASgNQgm6SAYqBBhkKAESRwoObGVhc2VfZHVyYXRpb24YBCABKAsyGS5nb29nbGUucHJvdG9idWYuRHVyYXRpb25CFLpIEcgBAaoBCyIDCJAcMgQQwIQ9IkIKF01vZHVsZUNsYWltSm9ic1Jlc3BvbnNlEicKBGpvYnMYASADKAsyGS5zYWFzLmpvYnMudjEuSm9iRW52ZWxvcGUilwEKGU1vZHVsZUhlYXJ0YmVhdEpvYlJlcXVlc3QSNgoFbGVhc2UYASABKAsyHy5zYWFzLmpvYnMudjEuSm9iTGVhc2VSZWZlcmVuY2VCBrpIA8gBARJCCglleHRlbnNpb24YAiABKAsyGS5nb29nbGUucHJvdG9idWYuRHVyYXRpb25CFLpIEcgBAaoBCyIDCJAcMgQQwIQ9IksKGk1vZHVsZUhlYXJ0YmVhdEpvYlJlc3BvbnNlEi0KBWxlYXNlGAEgASgLMhYuc2Fhcy5qb2JzLnYxLkpvYkxlYXNlQga6SAPIAQEipgEKE01vZHVsZUFja0pvYlJlcXVlc3QSNgoFbGVhc2UYASABKAsyHy5zYWFzLmpvYnMudjEuSm9iTGVhc2VSZWZlcmVuY2VCBrpIA8gBARI3Cg5leGVjdXRpb25fa2luZBgCIAEoCUIfukgcchoYQDIWXiR8XlthLXpdW2EtejAtOV8uLV0qJBIeCgxleGVjdXRpb25faWQYAyABKAlCCLpIBXIDGP8BIsIBChRNb2R1bGVOYWNrSm9iUmVxdWVzdBI2CgVsZWFzZRgBIAEoCzIfLnNhYXMuam9icy52MS5Kb2JMZWFzZVJlZmVyZW5jZUIGukgDyAEBEjEKB2ZhaWx1cmUYAiABKAsyGC5zYWFzLmpvYnMudjEuSm9iRmFpbHVyZUIGukgDyAEBEhEKCXJldHJ5YWJsZRgDIAEoCBIsCghyZXRyeV9hdBgEIAEoCzIaLmdvb2dsZS5wcm90b2J1Zi5UaW1lc3RhbXAi9gEKF01vZHVsZU5vdGlmeVVzZXJSZXF1ZXN0EhgKBnRlbmFudBgBIAEoCUIIukgFcgOwAQESGQoHdXNlcl9pZBgCIAEoCUIIukgFcgOwAQESGQoFdGl0bGUYAyABKAlCCrpIB3IFEAEY5gcSFgoEYm9keRgEIAEoCUIIukgFcgMYgEASFQoEdHlwZRgFIAEoCUIHukgEcgIYQBIcCgphY3Rpb25fdXJsGAYgASgJQgi6SAVyAxiAEBIbCghjYXRlZ29yeRgHIAEoCUIJukgGcgQQARggEiEKD2lkZW1wb3RlbmN5X2tleRgIIAEoCUIIukgFcgMY/wEiRgoYTW9kdWxlTm90aWZ5VXNlclJlc3BvbnNlEhcKD25vdGlmaWNhdGlvbl9pZBgBIAEoCRIRCglkZWxpdmVyZWQYAiABKAgi4AEKHE1vZHVsZU5vdGlmeU9yZ0FkbWluc1JlcXVlc3QSGAoGdGVuYW50GAEgASgJQgi6SAVyA7ABARIZCgV0aXRsZRgCIAEoCUIKukgHcgUQARjmBxIWCgRib2R5GAMgASgJQgi6SAVyAxiAQBIVCgR0eXBlGAQgASgJQge6SARyAhhAEhwKCmFjdGlvbl91cmwYBSABKAlCCLpIBXIDGIAQEhsKCGNhdGVnb3J5GAYgASgJQgm6SAZyBBABGCASIQoPaWRlbXBvdGVuY3lfa2V5GAcgASgJQgi6SAVyAxj/ASIyCh1Nb2R1bGVOb3RpZnlPcmdBZG1pbnNSZXNwb25zZRIRCglkZWxpdmVyZWQYASABKAgiYwoUTW9kdWxlQXBwcm92YWxQb2xpY3kSFwoGcXVvcnVtGAEgASgNQge6SAQqAhhkEh4KDGFwcHJvdmVyX3NldBgCIAMoCUIIukgFkgECEGQSEgoKYWxsb3dfc2VsZhgDIAEoCCKbAQoPTW9kdWxlUmVzdW1lUmVmEi4KBXF1ZXVlGAEgASgJQh+6SBxyGhABGIABMhNeW2Etel1bYS16MC05Xy4tXSokEi4KBXRvcGljGAIgASgJQh+6SBxyGhABGP8BMhNeW2Etel1bYS16MC05Xy4tXSokEigKB3BheWxvYWQYAyABKAsyFy5nb29nbGUucHJvdG9idWYuU3RydWN0IpYDChxNb2R1bGVSZXF1ZXN0QXBwcm92YWxSZXF1ZXN0EhgKBnRlbmFudBgBIAEoCUIIukgFcgOwAQESHAoIcmVzb3VyY2UYAiABKAlCCrpIB3IFEAEYgAESGgoGYWN0aW9uGAMgASgJQgq6SAdyBRABGIABEigKB3N1YmplY3QYBCABKAsyFy5nb29nbGUucHJvdG9idWYuU3RydWN0EiAKDHJlcXVlc3RlZF9ieRgFIAEoCUIKukgHcgUQARj/ARI2CgZwb2xpY3kYBiABKAsyJi5zYWFzLmFjY291bnRzLnYxLk1vZHVsZUFwcHJvdmFsUG9saWN5Ej0KCnJlc3VtZV9yZWYYByABKAsyIS5zYWFzLmFjY291bnRzLnYxLk1vZHVsZVJlc3VtZVJlZkIGukgDyAEBEi4KCmV4cGlyZXNfYXQYCCABKAsyGi5nb29nbGUucHJvdG9idWYuVGltZXN0YW1wEi8KC2VzY2FsYXRlX2F0GAkgASgLMhouZ29vZ2xlLnByb3RvYnVmLlRpbWVzdGFtcCI+Ch1Nb2R1bGVSZXF1ZXN0QXBwcm92YWxSZXNwb25zZRIdCgthcHByb3ZhbF9pZBgBIAEoCUIIukgFcgOwAQEiUwoYTW9kdWxlR2V0QXBwcm92YWxSZXF1ZXN0EhgKBnRlbmFudBgBIAEoCUIIukgFcgOwAQESHQoLYXBwcm92YWxfaWQYAiABKAlCCLpIBXIDsAEBIvUCCg5Nb2R1bGVBcHByb3ZhbBIKCgJpZBgBIAEoCRIOCgZ0ZW5hbnQYAiABKAkSEAoIcmVzb3VyY2UYAyABKAkSDgoGYWN0aW9uGAQgASgJEigKB3N1YmplY3QYBSABKAsyFy5nb29nbGUucHJvdG9idWYuU3RydWN0EhQKDHJlcXVlc3RlZF9ieRgGIAEoCRIOCgZxdW9ydW0YByABKA0SDQoFc3RhdGUYCCABKAkSNQoKcmVzdW1lX3JlZhgJIAEoCzIhLnNhYXMuYWNjb3VudHMudjEuTW9kdWxlUmVzdW1lUmVmEi4KCmV4cGlyZXNfYXQYCiABKAsyGi5nb29nbGUucHJvdG9idWYuVGltZXN0YW1wEi8KC2VzY2FsYXRlX2F0GAsgASgLMhouZ29vZ2xlLnByb3RvYnVmLlRpbWVzdGFtcBIuCgpjcmVhdGVkX2F0GAwgASgLMhouZ29vZ2xlLnByb3RvYnVmLlRpbWVzdGFtcCJwChtNb2R1bGVDYW5jZWxBcHByb3ZhbFJlcXVlc3QSGAoGdGVuYW50GAEgASgJQgi6SAVyA7ABARIdCgthcHByb3ZhbF9pZBgCIAEoCUIIukgFcgOwAQESGAoGcmVhc29uGAMgASgJQgi6SAVyAxiACCLuAQobTW9kdWxlRW1pdEF1ZGl0RXZlbnRSZXF1ZXN0Eg4KBnRlbmFudBgBIAEoCRIeCgpldmVudF90eXBlGAIgASgJQgq6SAdyBRABGIABEhkKBWFjdG9yGAMgASgJQgq6SAdyBRABGP8BEhwKCHNvbHV0aW9uGAQgASgJQgq6SAdyBRABGIABEhoKCGVudHJ5X2lkGAUgASgJQgi6SAVyAxj/ARInCgZmaWVsZHMYBiABKAsyFy5nb29nbGUucHJvdG9idWYuU3RydWN0EiEKD2lkZW1wb3RlbmN5X2tleRgHIAEoCUIIukgFcgMY/wEinwEKG01vZHVsZUF1ZGl0RmllbGREZWNsYXJhdGlvbhIXCgRuYW1lGAEgASgJQgm6SAZyBBABGEASQAoEa2luZBgCIAEoDjImLnNhYXMuYWNjb3VudHMudjEuTW9kdWxlQXVkaXRGaWVsZEtpbmRCCrpIB4IBBBABIAASGAoGdmFsdWVzGAMgAygJQgi6SAWSAQIQQBILCgNwaWkYBCABKAgi7wEKH01vZHVsZUF1ZGl0RXZlbnRUeXBlRGVjbGFyYXRpb24SGAoEdHlwZRgBIAEoCUIKukgHcgUQARiAARIdCgtkZXNjcmlwdGlvbhgCIAEoCUIIukgFcgMYgAgSRwoGZmllbGRzGAMgAygLMi0uc2Fhcy5hY2NvdW50cy52MS5Nb2R1bGVBdWRpdEZpZWxkRGVjbGFyYXRpb25CCLpIBZIBAhAgEkoKCnZpc2liaWxpdHkYBCABKA4yLC5zYWFzLmFjY291bnRzLnYxLk1vZHVsZUF1ZGl0RXZlbnRWaXNpYmlsaXR5Qgi6SAWCAQIQASKzAQojTW9kdWxlRGVjbGFyZUF1ZGl0RXZlbnRUeXBlc1JlcXVlc3QSPgoGcHJlZml4GAEgASgJQi66SCtyKRABGIABMiJeW2EtejAtOV0oPzpbYS16MC05Xy1dKlthLXowLTldKT8kEkwKBXR5cGVzGAIgAygLMjEuc2Fhcy5hY2NvdW50cy52MS5Nb2R1bGVBdWRpdEV2ZW50VHlwZURlY2xhcmF0aW9uQgq6SAeSAQQIARBAIl0KJE1vZHVsZURlY2xhcmVBdWRpdEV2ZW50VHlwZXNSZXNwb25zZRIWCg5hZG1pdHRlZF90eXBlcxgBIAMoCRIdChVuYW1lc3BhY2VzX3Rha2VuX292ZXIYAiADKAkibQoWU3ViamVjdFZpc2liaWxpdHlHcmFudBIjChJ2aXNpYmxlX3N1YmplY3RfaWQYASABKAlCB7pIBHICEAESLgoKZXhwaXJlc19hdBgCIAEoCzIaLmdvb2dsZS5wcm90b2J1Zi5UaW1lc3RhbXAiYwoiTW9kdWxlTGlzdFN1YmplY3RWaXNpYmlsaXR5UmVxdWVzdBIYCgZ0ZW5hbnQYASABKAlCCLpIBXIDsAEBEiMKEXZpZXdlcl9zdWJqZWN0X2lkGAIgASgJQgi6SAVyA7ABASJfCiNNb2R1bGVMaXN0U3ViamVjdFZpc2liaWxpdHlSZXNwb25zZRI4CgZncmFudHMYASADKAsyKC5zYWFzLmFjY291bnRzLnYxLlN1YmplY3RWaXNpYmlsaXR5R3JhbnQiVwoaRmV0Y2hEYXRhc291cmNlQmxvYlJlcXVlc3QSGwoJc291cmNlX2lkGAEgASgJQgi6SAVyA7ABARIcCghibG9iX3NoYRgCIAEoCUIKukgHcgUQARj/ASJSChhGZXRjaERhdGFzb3VyY2VCbG9iQ2h1bmsSDAoEZGF0YRgBIAEoDBISCgp0b3RhbF9zaXplGAIgASgDEhQKDGNvbnRlbnRfdHlwZRgDIAEoCSJSChFEYXRhc291cmNlRmlsZVJlZhIbCgdpdGVtX2lkGAEgASgJQgq6SAdyBRABGIAgEiAKDGl0ZW1fdmVyc2lvbhgCIAEoCUIKukgHcgUQARj/ASKYAQobRmV0Y2hEYXRhc291cmNlRmlsZXNSZXF1ZXN0EhsKCXNvdXJjZV9pZBgBIAEoCUIIukgFcgOwAQESGwoHdmVyc2lvbhgCIAEoCUIKukgHcgUQARj/ARI/CgVmaWxlcxgDIAMoCzIjLnNhYXMuYWNjb3VudHMudjEuRGF0YXNvdXJjZUZpbGVSZWZCC7pICJIBBQgBEOgHIosBChREYXRhc291cmNlUHJvdmVuYW5jZRIRCglzb3VyY2VfaWQYASABKAkSDgoGb3JnX2lkGAIgASgJEhgKEGJvdW5kYXJ5X25vZGVfaWQYAyABKAkSDwoHdmVyc2lvbhgEIAEoCRIPCgdpdGVtX2lkGAUgASgJEhQKDGl0ZW1fdmVyc2lvbhgGIAEoCSLAAQoVRGF0YXNvdXJjZUl0ZW1SZWFkZXJzEjsKBWJhc2lzGAEgASgOMiwuc2Fhcy5hY2NvdW50cy52MS5EYXRhc291cmNlSXRlbVJlYWRlcnNCYXNpcxIQCgh1c2VyX2lkcxgCIAMoCRIRCglncm91cF9pZHMYAyADKAkSGAoQYm91bmRhcnlfcmVhZGVycxgEIAEoCBIWCg51bm1hcHBlZF9jb3VudBgFIAEoBRITCgthY2xfdmVyc2lvbhgGIAEoCSK+AQoURGF0YXNvdXJjZUZpbGVIZWFkZXISOgoKcHJvdmVuYW5jZRgBIAEoCzImLnNhYXMuYWNjb3VudHMudjEuRGF0YXNvdXJjZVByb3ZlbmFuY2USDAoEcGF0aBgCIAEoCRIUCgxjb250ZW50X3R5cGUYAyABKAkSDAoEc2l6ZRgEIAEoAxI4CgdyZWFkZXJzGAUgASgLMicuc2Fhcy5hY2NvdW50cy52MS5EYXRhc291cmNlSXRlbVJlYWRlcnMibgoZRmV0Y2hEYXRhc291cmNlRmlsZXNGcmFtZRI4CgZoZWFkZXIYASABKAsyJi5zYWFzLmFjY291bnRzLnYxLkRhdGFzb3VyY2VGaWxlSGVhZGVySAASDgoEZGF0YRgCIAEoDEgAQgcKBWZyYW1lIm4KGU1vZHVsZVB1Ymxpc2hFdmVudFJlcXVlc3QSGAoGdGVuYW50GAEgASgJQgi6SAVyA7ABARI3CghlbnZlbG9wZRgCIAEoCzIdLnNhYXMuZXZlbnRzLnYxLkV2ZW50RW52ZWxvcGVCBrpIA8gBASI4ChpNb2R1bGVQdWJsaXNoRXZlbnRSZXNwb25zZRIaCghldmVudF9pZBgBIAEoCUIIukgFcgOwAQEiyAEKFk1vZHVsZVN1YnNjcmliZVJlcXVlc3QSSwoMdHlwZV9wYXR0ZXJuGAEgASgJQjW6SDJyMBABGIABMileW2Etel1bYS16MC05XSooPzpcLlthLXowLTldKykqKD86XC5cKik/JBIuCgVxdWV1ZRgCIAEoCUIfukgcchoQARiAATITXlthLXpdW2EtejAtOV8uLV0qJBIxCghkZWxpdmVyeRgDIAEoDjIfLnNhYXMuYWNjb3VudHMudjEuRXZlbnREZWxpdmVyeSLJAQoSTW9kdWxlU3Vic2NyaXB0aW9uEgoKAmlkGAEgASgJEh8KF3N1YnNjcmliZXJfcHJpbmNpcGFsX2lkGAIgASgJEhQKDHR5cGVfcGF0dGVybhgDIAEoCRINCgVxdWV1ZRgEIAEoCRIxCghkZWxpdmVyeRgFIAEoDjIfLnNhYXMuYWNjb3VudHMudjEuRXZlbnREZWxpdmVyeRIuCgpjcmVhdGVkX2F0GAYgASgLMhouZ29vZ2xlLnByb3RvYnVmLlRpbWVzdGFtcCJVChdNb2R1bGVTdWJzY3JpYmVSZXNwb25zZRI6CgxzdWJzY3JpcHRpb24YASABKAsyJC5zYWFzLmFjY291bnRzLnYxLk1vZHVsZVN1YnNjcmlwdGlvbiI9ChhNb2R1bGVVbnN1YnNjcmliZVJlcXVlc3QSIQoPc3Vic2NyaXB0aW9uX2lkGAEgASgJQgi6SAVyA7ABASIgCh5Nb2R1bGVMaXN0U3Vic2NyaXB0aW9uc1JlcXVlc3QiXgofTW9kdWxlTGlzdFN1YnNjcmlwdGlvbnNSZXNwb25zZRI7Cg1zdWJzY3JpcHRpb25zGAEgAygLMiQuc2Fhcy5hY2NvdW50cy52MS5Nb2R1bGVTdWJzY3JpcHRpb24iegoZTW9kdWxlUmVwbGF5RXZlbnRzUmVxdWVzdBIYCgZ0ZW5hbnQYASABKAlCCLpIBXIDsAEBEhgKBHR5cGUYAiABKAlCCrpIB3IFEAEYgAESKQoFc2luY2UYAyABKAsyGi5nb29nbGUucHJvdG9idWYuVGltZXN0YW1wIjEKGk1vZHVsZVJlcGxheUV2ZW50c1Jlc3BvbnNlEhMKC3JlZGVsaXZlcmVkGAEgASgFItMCChhSZWFkYWJsZVNvdXJjZUNvbGxlY3Rpb24SEQoJc291cmNlX2lkGAEgASgJEhMKC2JvdW5kYXJ5X2lkGAIgASgJEg4KBm9yaWdpbhgDIAEoCRIRCgljb250YWluZXIYBCABKAkSCwoDcmVmGAUgASgJEg0KBXBhdGhzGAYgAygJEhYKDmJvdW5kYXJ5X2xhYmVsGAcgASgJEj4KEGdyYW50X2Rpc2Nsb3N1cmUYCCABKA4yJC5zYWFzLmFjY291bnRzLnYxLk1ldGFkYXRhRGlzY2xvc3VyZRI+CgtyZWFkX2dyYW50cxgJIAMoCzIpLnNhYXMuYWNjb3VudHMudjEuUmVhZGFibGVDb2xsZWN0aW9uR3JhbnQSOAoEc3luYxgKIAEoCzIqLnNhYXMuYWNjb3VudHMudjEuQ29sbGVjdGlvblN5bmNQcm92ZW5hbmNlIrABChdSZWFkYWJsZUNvbGxlY3Rpb25HcmFudBIVCg1zdWJqZWN0X2xhYmVsGAEgASgJEhQKDHN1YmplY3Rfa2luZBgCIAEoCRIRCglyb2xlX25hbWUYAyABKAkSEgoKc2NvcGVfcGF0aBgEIAEoCRIRCglpbmhlcml0ZWQYBSABKAgSLgoKZXhwaXJlc19hdBgGIAEoCzIaLmdvb2dsZS5wcm90b2J1Zi5UaW1lc3RhbXAiqQIKGENvbGxlY3Rpb25TeW5jUHJvdmVuYW5jZRIwCgVzdGFnZRgBIAEoDjIhLnNhYXMuYWNjb3VudHMudjEuU291cmNlU3luY1N0YWdlEiYKAmF0GAIgASgLMhouZ29vZ2xlLnByb3RvYnVmLlRpbWVzdGFtcBIQCghyZXZpc2lvbhgDIAEoCRIPCgd0cmlnZ2VyGAQgASgJEkIKFHJlcXVlc3Rlcl9kaXNjbG9zdXJlGAUgASgOMiQuc2Fhcy5hY2NvdW50cy52MS5NZXRhZGF0YURpc2Nsb3N1cmUSMAoMcmVxdWVzdGVkX2F0GAYgASgLMhouZ29vZ2xlLnByb3RvYnVmLlRpbWVzdGFtcBIaChJyZXF1ZXN0ZWRfYnlfbGFiZWwYByABKAkiYwokTGlzdFJlYWRhYmxlU291cmNlQ29sbGVjdGlvbnNSZXF1ZXN0Eh0KCXBhZ2Vfc2l6ZRgBIAEoBUIKukgHGgUY6AcoABIcCgpwYWdlX3Rva2VuGAIgASgJQgi6SAVyAxiAECKBAQolTGlzdFJlYWRhYmxlU291cmNlQ29sbGVjdGlvbnNSZXNwb25zZRI/Cgtjb2xsZWN0aW9ucxgBIAMoCzIqLnNhYXMuYWNjb3VudHMudjEuUmVhZGFibGVTb3VyY2VDb2xsZWN0aW9uEhcKD25leHRfcGFnZV90b2tlbhgCIAEoCSLpAQoYTW9kdWxlUGxhY2VSZWNvcmRSZXF1ZXN0EhgKBnRlbmFudBgBIAEoCUIIukgFcgOwAQESPQoKc2NvcGVfcGF0aBgCIAEoCUIpukgmciQQARiABDIdXlthLXowLTlfXSsoPzpcLlthLXowLTlfXSspKiQSFwoEa2luZBgDIAEoCUIJukgGcgQQARhAEhcKBWxhYmVsGAQgASgJQgi6SAVyAxj/ARIhCg1yZXNvdXJjZV90eXBlGAUgASgJQgq6SAdyBRABGIABEh8KC3Jlc291cmNlX2lkGAYgASgJQgq6SAdyBRABGIAEIjYKGU1vZHVsZVBsYWNlUmVjb3JkUmVzcG9uc2USGQoHbm9kZV9pZBgBIAEoCUIIukgFcgOwAQEihQEKI0NoZWNrV29ya0NvbnRleHRSZWNvcmRBY2Nlc3NSZXF1ZXN0EiEKDXJlc291cmNlX3R5cGUYASABKAlCCrpIB3IFEAEYgAISHwoLcmVzb3VyY2VfaWQYAiABKAlCCrpIB3IFEAEYgAQSGgoGYWN0aW9uGAMgASgJQgq6SAdyBRABGIACIk4KJENoZWNrV29ya0NvbnRleHRSZWNvcmRBY2Nlc3NSZXNwb25zZRIPCgdhbGxvd2VkGAEgASgIEhUKDXNjb3BlX25vZGVfaWQYAiABKAkifAoqTW9kdWxlRXhjaGFuZ2VEZWxlZ2F0ZWRSZWFkQXVkaWVuY2VSZXF1ZXN0Eh4KCmJpbmRpbmdfaWQYASABKAlCCrpIB3IFEAEYgAESLgoZcGFyZW50X3dvcmtfY29udGV4dF90b2tlbhgCIAEoCUILukgIcgYQARiAgAIigAMKL01vZHVsZUV4Y2hhbmdlRGVsZWdhdGVkT3BlcmF0aW9uQXVkaWVuY2VSZXF1ZXN0Eh4KCmJpbmRpbmdfaWQYASABKAlCCrpIB3IFEAEYgAESLAoZcGFyZW50X3dvcmtfY29udGV4dF90b2tlbhgCIAEoCUIJukgGcgQYgIACEg4KBmxvb2t1cBgDIAEoCBIiCg1kZWxlZ2F0aW9uX2lkGAQgASgJQgu6SAjYAQFyA7ABATrKAbpIxgEawwEKM2V4Y2hhbmdlX2RlbGVnYXRlZF9vcGVyYXRpb25fYXVkaWVuY2Uub25lX2F1dGhvcml0eRJGZXhhY3RseSBvbmUgb2YgcGFyZW50X3dvcmtfY29udGV4dF90b2tlbiBhbmQgZGVsZWdhdGlvbl9pZCBpcyByZXF1aXJlZBpEKHRoaXMucGFyZW50X3dvcmtfY29udGV4dF90b2tlbiAhPSAnJykgIT0gKHRoaXMuZGVsZWdhdGlvbl9pZCAhPSAnJykquAIKFE1vZHVsZUF1ZGl0RmllbGRLaW5kEicKI01PRFVMRV9BVURJVF9GSUVMRF9LSU5EX1VOU1BFQ0lGSUVEEAASIgoeTU9EVUxFX0FVRElUX0ZJRUxEX0tJTkRfU1RSSU5HEAESIAocTU9EVUxFX0FVRElUX0ZJRUxEX0tJTkRfVVVJRBACEh8KG01PRFVMRV9BVURJVF9GSUVMRF9LSU5EX0lOVBADEiIKHk1PRFVMRV9BVURJVF9GSUVMRF9LSU5EX05VTUJFUhAEEiAKHE1PRFVMRV9BVURJVF9GSUVMRF9LSU5EX0JPT0wQBRIgChxNT0RVTEVfQVVESVRfRklFTERfS0lORF9FTlVNEAYSKAokTU9EVUxFX0FVRElUX0ZJRUxEX0tJTkRfU1RSSU5HX0FSUkFZEAcqoQEKGk1vZHVsZUF1ZGl0RXZlbnRWaXNpYmlsaXR5Ei0KKU1PRFVMRV9BVURJVF9FVkVOVF9WSVNJQklMSVRZX1VOU1BFQ0lGSUVEEAASKAokTU9EVUxFX0FVRElUX0VWRU5UX1ZJU0lCSUxJVFlfVEVOQU5UEAESKgomTU9EVUxFX0FVRElUX0VWRU5UX1ZJU0lCSUxJVFlfRVhURVJOQUwQAircAQoaRGF0YXNvdXJjZUl0ZW1SZWFkZXJzQmFzaXMSLQopREFUQVNPVVJDRV9JVEVNX1JFQURFUlNfQkFTSVNfVU5TUEVDSUZJRUQQABIsCihEQVRBU09VUkNFX0lURU1fUkVBREVSU19CQVNJU19UUkFOU0xBVEVEEAESLworREFUQVNPVVJDRV9JVEVNX1JFQURFUlNfQkFTSVNfU09VUkNFX1NDT1BFRBACEjAKLERBVEFTT1VSQ0VfSVRFTV9SRUFERVJTX0JBU0lTX1VOVFJBTlNMQVRBQkxFEAMqaQoNRXZlbnREZWxpdmVyeRIeChpFVkVOVF9ERUxJVkVSWV9VTlNQRUNJRklFRBAAEhwKGEVWRU5UX0RFTElWRVJZX1VOT1JERVJFRBABEhoKFkVWRU5UX0RFTElWRVJZX09SREVSRUQQAip+ChJNZXRhZGF0YURpc2Nsb3N1cmUSIwofTUVUQURBVEFfRElTQ0xPU1VSRV9VTlNQRUNJRklFRBAAEiEKHU1FVEFEQVRBX0RJU0NMT1NVUkVfRElTQ0xPU0VEEAESIAocTUVUQURBVEFfRElTQ0xPU1VSRV9XSVRISEVMRBACKlwKD1NvdXJjZVN5bmNTdGFnZRIhCh1TT1VSQ0VfU1lOQ19TVEFHRV9VTlNQRUNJRklFRBAAEiYKIlNPVVJDRV9TWU5DX1NUQUdFX0NIQU5HRVNfRU5RVUVVRUQQATLRIwoZTW9kdWxlQ2FwYWJpbGl0aWVzU2VydmljZRLFAQodRXhjaGFuZ2VEZWxlZ2F0ZWRSZWFkQXVkaWVuY2USPC5zYWFzLmFjY291bnRzLnYxLk1vZHVsZUV4Y2hhbmdlRGVsZWdhdGVkUmVhZEF1ZGllbmNlUmVxdWVzdBojLnNhYXMuYWNjb3VudHMudjEuSXNzdWVkV29ya0NvbnRleHQiQcLzGD0IAxABMAE6Kwonc2Fhcy5tb2R1bGUuZGVsZWdhdGVkX2F1ZGllbmNlX2V4Y2hhbmdlEARAAUgHUARYBGABEs8BCiJFeGNoYW5nZURlbGVnYXRlZE9wZXJhdGlvbkF1ZGllbmNlEkEuc2Fhcy5hY2NvdW50cy52MS5Nb2R1bGVFeGNoYW5nZURlbGVnYXRlZE9wZXJhdGlvbkF1ZGllbmNlUmVxdWVzdBojLnNhYXMuYWNjb3VudHMudjEuSXNzdWVkV29ya0NvbnRleHQiQcLzGD0IAxABMAE6Kwonc2Fhcy5tb2R1bGUuZGVsZWdhdGVkX2F1ZGllbmNlX2V4Y2hhbmdlEARAAUgHUARYBGABEqcBChxDaGVja1dvcmtDb250ZXh0UmVjb3JkQWNjZXNzEjUuc2Fhcy5hY2NvdW50cy52MS5DaGVja1dvcmtDb250ZXh0UmVjb3JkQWNjZXNzUmVxdWVzdBo2LnNhYXMuYWNjb3VudHMudjEuQ2hlY2tXb3JrQ29udGV4dFJlY29yZEFjY2Vzc1Jlc3BvbnNlIhjC8xgUCAMQATABOgIQAUABSAdQA1gDYAESqgEKHUxpc3RSZWFkYWJsZVNvdXJjZUNvbGxlY3Rpb25zEjYuc2Fhcy5hY2NvdW50cy52MS5MaXN0UmVhZGFibGVTb3VyY2VDb2xsZWN0aW9uc1JlcXVlc3QaNy5zYWFzLmFjY291bnRzLnYxLkxpc3RSZWFkYWJsZVNvdXJjZUNvbGxlY3Rpb25zUmVzcG9uc2UiGMLzGBQIAxABMAE6AhABQAFIB1ADWANgARKAAQoLUGxhY2VSZWNvcmQSKi5zYWFzLmFjY291bnRzLnYxLk1vZHVsZVBsYWNlUmVjb3JkUmVxdWVzdBorLnNhYXMuYWNjb3VudHMudjEuTW9kdWxlUGxhY2VSZWNvcmRSZXNwb25zZSIYwvMYFAgDEAEwAToCEAFAAUgHUANYA2ABEn0KCkVucXVldWVKb2ISKS5zYWFzLmFjY291bnRzLnYxLk1vZHVsZUVucXVldWVKb2JSZXF1ZXN0Giouc2Fhcy5hY2NvdW50cy52MS5Nb2R1bGVFbnF1ZXVlSm9iUmVzcG9uc2UiGMLzGBQIAxABMAE6AhABQAFIB1ADWANgARJ6CglDbGFpbUpvYnMSKC5zYWFzLmFjY291bnRzLnYxLk1vZHVsZUNsYWltSm9ic1JlcXVlc3QaKS5zYWFzLmFjY291bnRzLnYxLk1vZHVsZUNsYWltSm9ic1Jlc3BvbnNlIhjC8xgUCAMQATABOgIQAUABSAdQA1gDYAESgwEKDEhlYXJ0YmVhdEpvYhIrLnNhYXMuYWNjb3VudHMudjEuTW9kdWxlSGVhcnRiZWF0Sm9iUmVxdWVzdBosLnNhYXMuYWNjb3VudHMudjEuTW9kdWxlSGVhcnRiZWF0Sm9iUmVzcG9uc2UiGMLzGBQIAxABMAE6AhABQAFIB1ADWANgARJhCgZBY2tKb2ISJS5zYWFzLmFjY291bnRzLnYxLk1vZHVsZUFja0pvYlJlcXVlc3QaFi5nb29nbGUucHJvdG9idWYuRW1wdHkiGMLzGBQIAxABMAE6AhABQAFIB1ADWANgARJjCgdOYWNrSm9iEiYuc2Fhcy5hY2NvdW50cy52MS5Nb2R1bGVOYWNrSm9iUmVxdWVzdBoWLmdvb2dsZS5wcm90b2J1Zi5FbXB0eSIYwvMYFAgDEAEwAToCEAFAAUgHUANYA2ABEn0KCk5vdGlmeVVzZXISKS5zYWFzLmFjY291bnRzLnYxLk1vZHVsZU5vdGlmeVVzZXJSZXF1ZXN0Giouc2Fhcy5hY2NvdW50cy52MS5Nb2R1bGVOb3RpZnlVc2VyUmVzcG9uc2UiGMLzGBQIAxABMAE6AhABQAFIB1ADWANgARKtAQoPTm90aWZ5T3JnQWRtaW5zEi4uc2Fhcy5hY2NvdW50cy52MS5Nb2R1bGVOb3RpZnlPcmdBZG1pbnNSZXF1ZXN0Gi8uc2Fhcy5hY2NvdW50cy52MS5Nb2R1bGVOb3RpZnlPcmdBZG1pbnNSZXNwb25zZSI5wvMYNQgDEAEwATojCh9zYWFzLm1vZHVsZS5vcmdfYWRtaW5zX25vdGlmaWVkEAJAAUgHUANYA2ABEowBCg9SZXF1ZXN0QXBwcm92YWwSLi5zYWFzLmFjY291bnRzLnYxLk1vZHVsZVJlcXVlc3RBcHByb3ZhbFJlcXVlc3QaLy5zYWFzLmFjY291bnRzLnYxLk1vZHVsZVJlcXVlc3RBcHByb3ZhbFJlc3BvbnNlIhjC8xgUCAMQATABOgIQAUABSAdQA1gDYAESdQoLR2V0QXBwcm92YWwSKi5zYWFzLmFjY291bnRzLnYxLk1vZHVsZUdldEFwcHJvdmFsUmVxdWVzdBogLnNhYXMuYWNjb3VudHMudjEuTW9kdWxlQXBwcm92YWwiGMLzGBQIAxABMAE6AhABQAFIB1ADWANgARJxCg5DYW5jZWxBcHByb3ZhbBItLnNhYXMuYWNjb3VudHMudjEuTW9kdWxlQ2FuY2VsQXBwcm92YWxSZXF1ZXN0GhYuZ29vZ2xlLnByb3RvYnVmLkVtcHR5IhjC8xgUCAMQATABOgIQAUABSAdQA1gDYAEScQoORW1pdEF1ZGl0RXZlbnQSLS5zYWFzLmFjY291bnRzLnYxLk1vZHVsZUVtaXRBdWRpdEV2ZW50UmVxdWVzdBoWLmdvb2dsZS5wcm90b2J1Zi5FbXB0eSIYwvMYFAgDEAEwAToCEAFAAUgHUANYA2ABEsMBChZEZWNsYXJlQXVkaXRFdmVudFR5cGVzEjUuc2Fhcy5hY2NvdW50cy52MS5Nb2R1bGVEZWNsYXJlQXVkaXRFdmVudFR5cGVzUmVxdWVzdBo2LnNhYXMuYWNjb3VudHMudjEuTW9kdWxlRGVjbGFyZUF1ZGl0RXZlbnRUeXBlc1Jlc3BvbnNlIjrC8xg2CAMQATABOiQKIHNhYXMubW9kdWxlLmF1ZGl0X3R5cGVzX2RlY2xhcmVkEAJAAUgHUANYA2ABEp4BChVMaXN0U3ViamVjdFZpc2liaWxpdHkSNC5zYWFzLmFjY291bnRzLnYxLk1vZHVsZUxpc3RTdWJqZWN0VmlzaWJpbGl0eVJlcXVlc3QaNS5zYWFzLmFjY291bnRzLnYxLk1vZHVsZUxpc3RTdWJqZWN0VmlzaWJpbGl0eVJlc3BvbnNlIhjC8xgUCAMQATABOgIQAUABSAdQA1gDYAESjgEKE0ZldGNoRGF0YXNvdXJjZUJsb2ISLC5zYWFzLmFjY291bnRzLnYxLkZldGNoRGF0YXNvdXJjZUJsb2JSZXF1ZXN0Giouc2Fhcy5hY2NvdW50cy52MS5GZXRjaERhdGFzb3VyY2VCbG9iQ2h1bmsiG4gCAcLzGBQIAxABMAE6AhABQAFIB1ADWANgATABEo4BChRGZXRjaERhdGFzb3VyY2VGaWxlcxItLnNhYXMuYWNjb3VudHMudjEuRmV0Y2hEYXRhc291cmNlRmlsZXNSZXF1ZXN0Gisuc2Fhcy5hY2NvdW50cy52MS5GZXRjaERhdGFzb3VyY2VGaWxlc0ZyYW1lIhjC8xgUCAMQATABOgIQAUABSAdQA1gDYAEwARK2AQoWTWludE1vZHVsZVJlZ2lzdHJhdGlvbhIvLnNhYXMuYWNjb3VudHMudjEuTW9kdWxlTWludFJlZ2lzdHJhdGlvblJlcXVlc3QaMC5zYWFzLmFjY291bnRzLnYxLk1vZHVsZU1pbnRSZWdpc3RyYXRpb25SZXNwb25zZSI5wvMYNQgDEAEwATojCh9zYWFzLm1vZHVsZS5yZWdpc3RyYXRpb25fbWludGVkEAJAAUgHUARYBGABEr4BChhNaW50U29sdXRpb25SZWdpc3RyYXRpb24SMS5zYWFzLmFjY291bnRzLnYxLlNvbHV0aW9uTWludFJlZ2lzdHJhdGlvblJlcXVlc3QaMi5zYWFzLmFjY291bnRzLnYxLlNvbHV0aW9uTWludFJlZ2lzdHJhdGlvblJlc3BvbnNlIjvC8xg3CAMQATABOiUKIXNhYXMuc29sdXRpb24ucmVnaXN0cmF0aW9uX21pbnRlZBACQAFIB1AEWARgARKzAQoVTWludE1vZHVsZVdvcmtDb250ZXh0Ei4uc2Fhcy5hY2NvdW50cy52MS5Nb2R1bGVNaW50V29ya0NvbnRleHRSZXF1ZXN0Gi8uc2Fhcy5hY2NvdW50cy52MS5Nb2R1bGVNaW50V29ya0NvbnRleHRSZXNwb25zZSI5wvMYNQgDEAEwATojCh9zYWFzLm1vZHVsZS53b3JrX2NvbnRleHRfbWludGVkEAJAAUgHUARYBGABEscBChpNaW50TW9kdWxlT3BlcmF0aW9uQ29udGV4dBIzLnNhYXMuYWNjb3VudHMudjEuTW9kdWxlTWludE9wZXJhdGlvbkNvbnRleHRSZXF1ZXN0GjQuc2Fhcy5hY2NvdW50cy52MS5Nb2R1bGVNaW50T3BlcmF0aW9uQ29udGV4dFJlc3BvbnNlIj7C8xg6CAMQATABOigKJHNhYXMubW9kdWxlLm9wZXJhdGlvbl9jb250ZXh0X21pbnRlZBACQAFIB1AEWARgARLOAQoaTWludFNvdXJjZU9wZXJhdGlvbkNvbnRleHQSOS5zYWFzLmFjY291bnRzLnYxLk1vZHVsZU1pbnRTb3VyY2VPcGVyYXRpb25Db250ZXh0UmVxdWVzdBo6LnNhYXMuYWNjb3VudHMudjEuTW9kdWxlTWludFNvdXJjZU9wZXJhdGlvbkNvbnRleHRSZXNwb25zZSI5wvMYNQgDEAEwATojCh9zYWFzLmRhdGFzb3VyY2UuZGVsZWdhdGlvbi51c2VkEAJAAUgHUARYBGABEoMBCgxQdWJsaXNoRXZlbnQSKy5zYWFzLmFjY291bnRzLnYxLk1vZHVsZVB1Ymxpc2hFdmVudFJlcXVlc3QaLC5zYWFzLmFjY291bnRzLnYxLk1vZHVsZVB1Ymxpc2hFdmVudFJlc3BvbnNlIhjC8xgUCAMQATABOgIQAUABSAdQA1gDYAESegoJU3Vic2NyaWJlEiguc2Fhcy5hY2NvdW50cy52MS5Nb2R1bGVTdWJzY3JpYmVSZXF1ZXN0Gikuc2Fhcy5hY2NvdW50cy52MS5Nb2R1bGVTdWJzY3JpYmVSZXNwb25zZSIYwvMYFAgDEAEwAToCEAFAAUgHUANYA2ABEmsKC1Vuc3Vic2NyaWJlEiouc2Fhcy5hY2NvdW50cy52MS5Nb2R1bGVVbnN1YnNjcmliZVJlcXVlc3QaFi5nb29nbGUucHJvdG9idWYuRW1wdHkiGMLzGBQIAxABMAE6AhABQAFIB1ADWANgARKSAQoRTGlzdFN1YnNjcmlwdGlvbnMSMC5zYWFzLmFjY291bnRzLnYxLk1vZHVsZUxpc3RTdWJzY3JpcHRpb25zUmVxdWVzdBoxLnNhYXMuYWNjb3VudHMudjEuTW9kdWxlTGlzdFN1YnNjcmlwdGlvbnNSZXNwb25zZSIYwvMYFAgDEAEwAToCEAFAAUgHUANYA2ABEoMBCgxSZXBsYXlFdmVudHMSKy5zYWFzLmFjY291bnRzLnYxLk1vZHVsZVJlcGxheUV2ZW50c1JlcXVlc3QaLC5zYWFzLmFjY291bnRzLnYxLk1vZHVsZVJlcGxheUV2ZW50c1Jlc3BvbnNlIhjC8xgUCAMQATABOgIQAUABSAdQA1gDYAFCvwEKFGNvbS5zYWFzLmFjY291bnRzLnYxQhdNb2R1bGVDYXBhYmlsaXRpZXNQcm90b1ABWixhY2NvdW50cy9wa2cvZ2VuL3NhYXMvYWNjb3VudHMvdjE7YWNjb3VudHN2MaICA1NBWKoCEFNhYXMuQWNjb3VudHMuVjHKAhBTYWFzXEFjY291bnRzXFYx4gIcU2Fhc1xBY2NvdW50c1xWMVxHUEJNZXRhZGF0YeoCElNhYXM6OkFjY291bnRzOjpWMWIGcHJvdG8z", [file_saas_accounts_v1_work_contexts, file_buf_validate_validate, file_google_protobuf_empty, file_google_protobuf_duration, file_google_protobuf_struct, file_google_protobuf_timestamp, file_saas_accounts_v1_module_registration, file_saas_events_v1_events, file_saas_jobs_v1_jobs, file_saas_policy_v1_options]); + fileDesc("CipzYWFzL2FjY291bnRzL3YxL21vZHVsZV9jYXBhYmlsaXRpZXMucHJvdG8SEHNhYXMuYWNjb3VudHMudjEiXgoXTW9kdWxlRW5xdWV1ZUpvYlJlcXVlc3QSGAoGdGVuYW50GAEgASgJQgi6SAVyA7ABARIpCgNqb2IYAiABKAsyFC5zYWFzLmpvYnMudjEuTmV3Sm9iQga6SAPIAQEidgoYTW9kdWxlRW5xdWV1ZUpvYlJlc3BvbnNlEhgKBmpvYl9pZBgBIAEoCUIIukgFcgOwAQESQAoLZGlzcG9zaXRpb24YAiABKA4yIy5zYWFzLmpvYnMudjEuSm9iRW5xdWV1ZURpc3Bvc2l0aW9uQga6SAPIAQEiygEKFk1vZHVsZUNsYWltSm9ic1JlcXVlc3QSLgoFcXVldWUYASABKAlCH7pIHHIaEAEYgAEyE15bYS16XVthLXowLTlfLi1dKiQSHQoJd29ya2VyX2lkGAIgASgJQgq6SAdyBRABGP8BEhgKBWxpbWl0GAMgASgNQgm6SAYqBBhkKAESRwoObGVhc2VfZHVyYXRpb24YBCABKAsyGS5nb29nbGUucHJvdG9idWYuRHVyYXRpb25CFLpIEcgBAaoBCyIDCJAcMgQQwIQ9IkIKF01vZHVsZUNsYWltSm9ic1Jlc3BvbnNlEicKBGpvYnMYASADKAsyGS5zYWFzLmpvYnMudjEuSm9iRW52ZWxvcGUilwEKGU1vZHVsZUhlYXJ0YmVhdEpvYlJlcXVlc3QSNgoFbGVhc2UYASABKAsyHy5zYWFzLmpvYnMudjEuSm9iTGVhc2VSZWZlcmVuY2VCBrpIA8gBARJCCglleHRlbnNpb24YAiABKAsyGS5nb29nbGUucHJvdG9idWYuRHVyYXRpb25CFLpIEcgBAaoBCyIDCJAcMgQQwIQ9IksKGk1vZHVsZUhlYXJ0YmVhdEpvYlJlc3BvbnNlEi0KBWxlYXNlGAEgASgLMhYuc2Fhcy5qb2JzLnYxLkpvYkxlYXNlQga6SAPIAQEipgEKE01vZHVsZUFja0pvYlJlcXVlc3QSNgoFbGVhc2UYASABKAsyHy5zYWFzLmpvYnMudjEuSm9iTGVhc2VSZWZlcmVuY2VCBrpIA8gBARI3Cg5leGVjdXRpb25fa2luZBgCIAEoCUIfukgcchoYQDIWXiR8XlthLXpdW2EtejAtOV8uLV0qJBIeCgxleGVjdXRpb25faWQYAyABKAlCCLpIBXIDGP8BIsIBChRNb2R1bGVOYWNrSm9iUmVxdWVzdBI2CgVsZWFzZRgBIAEoCzIfLnNhYXMuam9icy52MS5Kb2JMZWFzZVJlZmVyZW5jZUIGukgDyAEBEjEKB2ZhaWx1cmUYAiABKAsyGC5zYWFzLmpvYnMudjEuSm9iRmFpbHVyZUIGukgDyAEBEhEKCXJldHJ5YWJsZRgDIAEoCBIsCghyZXRyeV9hdBgEIAEoCzIaLmdvb2dsZS5wcm90b2J1Zi5UaW1lc3RhbXAi9gEKF01vZHVsZU5vdGlmeVVzZXJSZXF1ZXN0EhgKBnRlbmFudBgBIAEoCUIIukgFcgOwAQESGQoHdXNlcl9pZBgCIAEoCUIIukgFcgOwAQESGQoFdGl0bGUYAyABKAlCCrpIB3IFEAEY5gcSFgoEYm9keRgEIAEoCUIIukgFcgMYgEASFQoEdHlwZRgFIAEoCUIHukgEcgIYQBIcCgphY3Rpb25fdXJsGAYgASgJQgi6SAVyAxiAEBIbCghjYXRlZ29yeRgHIAEoCUIJukgGcgQQARggEiEKD2lkZW1wb3RlbmN5X2tleRgIIAEoCUIIukgFcgMY/wEiRgoYTW9kdWxlTm90aWZ5VXNlclJlc3BvbnNlEhcKD25vdGlmaWNhdGlvbl9pZBgBIAEoCRIRCglkZWxpdmVyZWQYAiABKAgi4AEKHE1vZHVsZU5vdGlmeU9yZ0FkbWluc1JlcXVlc3QSGAoGdGVuYW50GAEgASgJQgi6SAVyA7ABARIZCgV0aXRsZRgCIAEoCUIKukgHcgUQARjmBxIWCgRib2R5GAMgASgJQgi6SAVyAxiAQBIVCgR0eXBlGAQgASgJQge6SARyAhhAEhwKCmFjdGlvbl91cmwYBSABKAlCCLpIBXIDGIAQEhsKCGNhdGVnb3J5GAYgASgJQgm6SAZyBBABGCASIQoPaWRlbXBvdGVuY3lfa2V5GAcgASgJQgi6SAVyAxj/ASIyCh1Nb2R1bGVOb3RpZnlPcmdBZG1pbnNSZXNwb25zZRIRCglkZWxpdmVyZWQYASABKAgiYwoUTW9kdWxlQXBwcm92YWxQb2xpY3kSFwoGcXVvcnVtGAEgASgNQge6SAQqAhhkEh4KDGFwcHJvdmVyX3NldBgCIAMoCUIIukgFkgECEGQSEgoKYWxsb3dfc2VsZhgDIAEoCCKbAQoPTW9kdWxlUmVzdW1lUmVmEi4KBXF1ZXVlGAEgASgJQh+6SBxyGhABGIABMhNeW2Etel1bYS16MC05Xy4tXSokEi4KBXRvcGljGAIgASgJQh+6SBxyGhABGP8BMhNeW2Etel1bYS16MC05Xy4tXSokEigKB3BheWxvYWQYAyABKAsyFy5nb29nbGUucHJvdG9idWYuU3RydWN0IpYDChxNb2R1bGVSZXF1ZXN0QXBwcm92YWxSZXF1ZXN0EhgKBnRlbmFudBgBIAEoCUIIukgFcgOwAQESHAoIcmVzb3VyY2UYAiABKAlCCrpIB3IFEAEYgAESGgoGYWN0aW9uGAMgASgJQgq6SAdyBRABGIABEigKB3N1YmplY3QYBCABKAsyFy5nb29nbGUucHJvdG9idWYuU3RydWN0EiAKDHJlcXVlc3RlZF9ieRgFIAEoCUIKukgHcgUQARj/ARI2CgZwb2xpY3kYBiABKAsyJi5zYWFzLmFjY291bnRzLnYxLk1vZHVsZUFwcHJvdmFsUG9saWN5Ej0KCnJlc3VtZV9yZWYYByABKAsyIS5zYWFzLmFjY291bnRzLnYxLk1vZHVsZVJlc3VtZVJlZkIGukgDyAEBEi4KCmV4cGlyZXNfYXQYCCABKAsyGi5nb29nbGUucHJvdG9idWYuVGltZXN0YW1wEi8KC2VzY2FsYXRlX2F0GAkgASgLMhouZ29vZ2xlLnByb3RvYnVmLlRpbWVzdGFtcCI+Ch1Nb2R1bGVSZXF1ZXN0QXBwcm92YWxSZXNwb25zZRIdCgthcHByb3ZhbF9pZBgBIAEoCUIIukgFcgOwAQEiUwoYTW9kdWxlR2V0QXBwcm92YWxSZXF1ZXN0EhgKBnRlbmFudBgBIAEoCUIIukgFcgOwAQESHQoLYXBwcm92YWxfaWQYAiABKAlCCLpIBXIDsAEBIvUCCg5Nb2R1bGVBcHByb3ZhbBIKCgJpZBgBIAEoCRIOCgZ0ZW5hbnQYAiABKAkSEAoIcmVzb3VyY2UYAyABKAkSDgoGYWN0aW9uGAQgASgJEigKB3N1YmplY3QYBSABKAsyFy5nb29nbGUucHJvdG9idWYuU3RydWN0EhQKDHJlcXVlc3RlZF9ieRgGIAEoCRIOCgZxdW9ydW0YByABKA0SDQoFc3RhdGUYCCABKAkSNQoKcmVzdW1lX3JlZhgJIAEoCzIhLnNhYXMuYWNjb3VudHMudjEuTW9kdWxlUmVzdW1lUmVmEi4KCmV4cGlyZXNfYXQYCiABKAsyGi5nb29nbGUucHJvdG9idWYuVGltZXN0YW1wEi8KC2VzY2FsYXRlX2F0GAsgASgLMhouZ29vZ2xlLnByb3RvYnVmLlRpbWVzdGFtcBIuCgpjcmVhdGVkX2F0GAwgASgLMhouZ29vZ2xlLnByb3RvYnVmLlRpbWVzdGFtcCJwChtNb2R1bGVDYW5jZWxBcHByb3ZhbFJlcXVlc3QSGAoGdGVuYW50GAEgASgJQgi6SAVyA7ABARIdCgthcHByb3ZhbF9pZBgCIAEoCUIIukgFcgOwAQESGAoGcmVhc29uGAMgASgJQgi6SAVyAxiACCLuAQobTW9kdWxlRW1pdEF1ZGl0RXZlbnRSZXF1ZXN0Eg4KBnRlbmFudBgBIAEoCRIeCgpldmVudF90eXBlGAIgASgJQgq6SAdyBRABGIABEhkKBWFjdG9yGAMgASgJQgq6SAdyBRABGP8BEhwKCHNvbHV0aW9uGAQgASgJQgq6SAdyBRABGIABEhoKCGVudHJ5X2lkGAUgASgJQgi6SAVyAxj/ARInCgZmaWVsZHMYBiABKAsyFy5nb29nbGUucHJvdG9idWYuU3RydWN0EiEKD2lkZW1wb3RlbmN5X2tleRgHIAEoCUIIukgFcgMY/wEinwEKG01vZHVsZUF1ZGl0RmllbGREZWNsYXJhdGlvbhIXCgRuYW1lGAEgASgJQgm6SAZyBBABGEASQAoEa2luZBgCIAEoDjImLnNhYXMuYWNjb3VudHMudjEuTW9kdWxlQXVkaXRGaWVsZEtpbmRCCrpIB4IBBBABIAASGAoGdmFsdWVzGAMgAygJQgi6SAWSAQIQQBILCgNwaWkYBCABKAgiuQIKH01vZHVsZUF1ZGl0RXZlbnRUeXBlRGVjbGFyYXRpb24SGAoEdHlwZRgBIAEoCUIKukgHcgUQARiAARIdCgtkZXNjcmlwdGlvbhgCIAEoCUIIukgFcgMYgAgSRwoGZmllbGRzGAMgAygLMi0uc2Fhcy5hY2NvdW50cy52MS5Nb2R1bGVBdWRpdEZpZWxkRGVjbGFyYXRpb25CCLpIBZIBAhAgEkoKCnZpc2liaWxpdHkYBCABKA4yLC5zYWFzLmFjY291bnRzLnYxLk1vZHVsZUF1ZGl0RXZlbnRWaXNpYmlsaXR5Qgi6SAWCAQIQARJICglyZXRlbnRpb24YBSABKA4yKy5zYWFzLmFjY291bnRzLnYxLk1vZHVsZUF1ZGl0RXZlbnRSZXRlbnRpb25CCLpIBYIBAhABIrMBCiNNb2R1bGVEZWNsYXJlQXVkaXRFdmVudFR5cGVzUmVxdWVzdBI+CgZwcmVmaXgYASABKAlCLrpIK3IpEAEYgAEyIl5bYS16MC05XSg/OlthLXowLTlfLV0qW2EtejAtOV0pPyQSTAoFdHlwZXMYAiADKAsyMS5zYWFzLmFjY291bnRzLnYxLk1vZHVsZUF1ZGl0RXZlbnRUeXBlRGVjbGFyYXRpb25CCrpIB5IBBAgBEEAiXQokTW9kdWxlRGVjbGFyZUF1ZGl0RXZlbnRUeXBlc1Jlc3BvbnNlEhYKDmFkbWl0dGVkX3R5cGVzGAEgAygJEh0KFW5hbWVzcGFjZXNfdGFrZW5fb3ZlchgCIAMoCSJtChZTdWJqZWN0VmlzaWJpbGl0eUdyYW50EiMKEnZpc2libGVfc3ViamVjdF9pZBgBIAEoCUIHukgEcgIQARIuCgpleHBpcmVzX2F0GAIgASgLMhouZ29vZ2xlLnByb3RvYnVmLlRpbWVzdGFtcCJjCiJNb2R1bGVMaXN0U3ViamVjdFZpc2liaWxpdHlSZXF1ZXN0EhgKBnRlbmFudBgBIAEoCUIIukgFcgOwAQESIwoRdmlld2VyX3N1YmplY3RfaWQYAiABKAlCCLpIBXIDsAEBIl8KI01vZHVsZUxpc3RTdWJqZWN0VmlzaWJpbGl0eVJlc3BvbnNlEjgKBmdyYW50cxgBIAMoCzIoLnNhYXMuYWNjb3VudHMudjEuU3ViamVjdFZpc2liaWxpdHlHcmFudCJXChpGZXRjaERhdGFzb3VyY2VCbG9iUmVxdWVzdBIbCglzb3VyY2VfaWQYASABKAlCCLpIBXIDsAEBEhwKCGJsb2Jfc2hhGAIgASgJQgq6SAdyBRABGP8BIlIKGEZldGNoRGF0YXNvdXJjZUJsb2JDaHVuaxIMCgRkYXRhGAEgASgMEhIKCnRvdGFsX3NpemUYAiABKAMSFAoMY29udGVudF90eXBlGAMgASgJIlIKEURhdGFzb3VyY2VGaWxlUmVmEhsKB2l0ZW1faWQYASABKAlCCrpIB3IFEAEYgCASIAoMaXRlbV92ZXJzaW9uGAIgASgJQgq6SAdyBRABGP8BIpgBChtGZXRjaERhdGFzb3VyY2VGaWxlc1JlcXVlc3QSGwoJc291cmNlX2lkGAEgASgJQgi6SAVyA7ABARIbCgd2ZXJzaW9uGAIgASgJQgq6SAdyBRABGP8BEj8KBWZpbGVzGAMgAygLMiMuc2Fhcy5hY2NvdW50cy52MS5EYXRhc291cmNlRmlsZVJlZkILukgIkgEFCAEQ6AciiwEKFERhdGFzb3VyY2VQcm92ZW5hbmNlEhEKCXNvdXJjZV9pZBgBIAEoCRIOCgZvcmdfaWQYAiABKAkSGAoQYm91bmRhcnlfbm9kZV9pZBgDIAEoCRIPCgd2ZXJzaW9uGAQgASgJEg8KB2l0ZW1faWQYBSABKAkSFAoMaXRlbV92ZXJzaW9uGAYgASgJIsABChVEYXRhc291cmNlSXRlbVJlYWRlcnMSOwoFYmFzaXMYASABKA4yLC5zYWFzLmFjY291bnRzLnYxLkRhdGFzb3VyY2VJdGVtUmVhZGVyc0Jhc2lzEhAKCHVzZXJfaWRzGAIgAygJEhEKCWdyb3VwX2lkcxgDIAMoCRIYChBib3VuZGFyeV9yZWFkZXJzGAQgASgIEhYKDnVubWFwcGVkX2NvdW50GAUgASgFEhMKC2FjbF92ZXJzaW9uGAYgASgJIr4BChREYXRhc291cmNlRmlsZUhlYWRlchI6Cgpwcm92ZW5hbmNlGAEgASgLMiYuc2Fhcy5hY2NvdW50cy52MS5EYXRhc291cmNlUHJvdmVuYW5jZRIMCgRwYXRoGAIgASgJEhQKDGNvbnRlbnRfdHlwZRgDIAEoCRIMCgRzaXplGAQgASgDEjgKB3JlYWRlcnMYBSABKAsyJy5zYWFzLmFjY291bnRzLnYxLkRhdGFzb3VyY2VJdGVtUmVhZGVycyJuChlGZXRjaERhdGFzb3VyY2VGaWxlc0ZyYW1lEjgKBmhlYWRlchgBIAEoCzImLnNhYXMuYWNjb3VudHMudjEuRGF0YXNvdXJjZUZpbGVIZWFkZXJIABIOCgRkYXRhGAIgASgMSABCBwoFZnJhbWUibgoZTW9kdWxlUHVibGlzaEV2ZW50UmVxdWVzdBIYCgZ0ZW5hbnQYASABKAlCCLpIBXIDsAEBEjcKCGVudmVsb3BlGAIgASgLMh0uc2Fhcy5ldmVudHMudjEuRXZlbnRFbnZlbG9wZUIGukgDyAEBIjgKGk1vZHVsZVB1Ymxpc2hFdmVudFJlc3BvbnNlEhoKCGV2ZW50X2lkGAEgASgJQgi6SAVyA7ABASLIAQoWTW9kdWxlU3Vic2NyaWJlUmVxdWVzdBJLCgx0eXBlX3BhdHRlcm4YASABKAlCNbpIMnIwEAEYgAEyKV5bYS16XVthLXowLTldKig/OlwuW2EtejAtOV0rKSooPzpcLlwqKT8kEi4KBXF1ZXVlGAIgASgJQh+6SBxyGhABGIABMhNeW2Etel1bYS16MC05Xy4tXSokEjEKCGRlbGl2ZXJ5GAMgASgOMh8uc2Fhcy5hY2NvdW50cy52MS5FdmVudERlbGl2ZXJ5IskBChJNb2R1bGVTdWJzY3JpcHRpb24SCgoCaWQYASABKAkSHwoXc3Vic2NyaWJlcl9wcmluY2lwYWxfaWQYAiABKAkSFAoMdHlwZV9wYXR0ZXJuGAMgASgJEg0KBXF1ZXVlGAQgASgJEjEKCGRlbGl2ZXJ5GAUgASgOMh8uc2Fhcy5hY2NvdW50cy52MS5FdmVudERlbGl2ZXJ5Ei4KCmNyZWF0ZWRfYXQYBiABKAsyGi5nb29nbGUucHJvdG9idWYuVGltZXN0YW1wIlUKF01vZHVsZVN1YnNjcmliZVJlc3BvbnNlEjoKDHN1YnNjcmlwdGlvbhgBIAEoCzIkLnNhYXMuYWNjb3VudHMudjEuTW9kdWxlU3Vic2NyaXB0aW9uIj0KGE1vZHVsZVVuc3Vic2NyaWJlUmVxdWVzdBIhCg9zdWJzY3JpcHRpb25faWQYASABKAlCCLpIBXIDsAEBIiAKHk1vZHVsZUxpc3RTdWJzY3JpcHRpb25zUmVxdWVzdCJeCh9Nb2R1bGVMaXN0U3Vic2NyaXB0aW9uc1Jlc3BvbnNlEjsKDXN1YnNjcmlwdGlvbnMYASADKAsyJC5zYWFzLmFjY291bnRzLnYxLk1vZHVsZVN1YnNjcmlwdGlvbiJ6ChlNb2R1bGVSZXBsYXlFdmVudHNSZXF1ZXN0EhgKBnRlbmFudBgBIAEoCUIIukgFcgOwAQESGAoEdHlwZRgCIAEoCUIKukgHcgUQARiAARIpCgVzaW5jZRgDIAEoCzIaLmdvb2dsZS5wcm90b2J1Zi5UaW1lc3RhbXAiMQoaTW9kdWxlUmVwbGF5RXZlbnRzUmVzcG9uc2USEwoLcmVkZWxpdmVyZWQYASABKAUi0wIKGFJlYWRhYmxlU291cmNlQ29sbGVjdGlvbhIRCglzb3VyY2VfaWQYASABKAkSEwoLYm91bmRhcnlfaWQYAiABKAkSDgoGb3JpZ2luGAMgASgJEhEKCWNvbnRhaW5lchgEIAEoCRILCgNyZWYYBSABKAkSDQoFcGF0aHMYBiADKAkSFgoOYm91bmRhcnlfbGFiZWwYByABKAkSPgoQZ3JhbnRfZGlzY2xvc3VyZRgIIAEoDjIkLnNhYXMuYWNjb3VudHMudjEuTWV0YWRhdGFEaXNjbG9zdXJlEj4KC3JlYWRfZ3JhbnRzGAkgAygLMikuc2Fhcy5hY2NvdW50cy52MS5SZWFkYWJsZUNvbGxlY3Rpb25HcmFudBI4CgRzeW5jGAogASgLMiouc2Fhcy5hY2NvdW50cy52MS5Db2xsZWN0aW9uU3luY1Byb3ZlbmFuY2UisAEKF1JlYWRhYmxlQ29sbGVjdGlvbkdyYW50EhUKDXN1YmplY3RfbGFiZWwYASABKAkSFAoMc3ViamVjdF9raW5kGAIgASgJEhEKCXJvbGVfbmFtZRgDIAEoCRISCgpzY29wZV9wYXRoGAQgASgJEhEKCWluaGVyaXRlZBgFIAEoCBIuCgpleHBpcmVzX2F0GAYgASgLMhouZ29vZ2xlLnByb3RvYnVmLlRpbWVzdGFtcCKpAgoYQ29sbGVjdGlvblN5bmNQcm92ZW5hbmNlEjAKBXN0YWdlGAEgASgOMiEuc2Fhcy5hY2NvdW50cy52MS5Tb3VyY2VTeW5jU3RhZ2USJgoCYXQYAiABKAsyGi5nb29nbGUucHJvdG9idWYuVGltZXN0YW1wEhAKCHJldmlzaW9uGAMgASgJEg8KB3RyaWdnZXIYBCABKAkSQgoUcmVxdWVzdGVyX2Rpc2Nsb3N1cmUYBSABKA4yJC5zYWFzLmFjY291bnRzLnYxLk1ldGFkYXRhRGlzY2xvc3VyZRIwCgxyZXF1ZXN0ZWRfYXQYBiABKAsyGi5nb29nbGUucHJvdG9idWYuVGltZXN0YW1wEhoKEnJlcXVlc3RlZF9ieV9sYWJlbBgHIAEoCSJjCiRMaXN0UmVhZGFibGVTb3VyY2VDb2xsZWN0aW9uc1JlcXVlc3QSHQoJcGFnZV9zaXplGAEgASgFQgq6SAcaBRjoBygAEhwKCnBhZ2VfdG9rZW4YAiABKAlCCLpIBXIDGIAQIoEBCiVMaXN0UmVhZGFibGVTb3VyY2VDb2xsZWN0aW9uc1Jlc3BvbnNlEj8KC2NvbGxlY3Rpb25zGAEgAygLMiouc2Fhcy5hY2NvdW50cy52MS5SZWFkYWJsZVNvdXJjZUNvbGxlY3Rpb24SFwoPbmV4dF9wYWdlX3Rva2VuGAIgASgJIukBChhNb2R1bGVQbGFjZVJlY29yZFJlcXVlc3QSGAoGdGVuYW50GAEgASgJQgi6SAVyA7ABARI9CgpzY29wZV9wYXRoGAIgASgJQim6SCZyJBABGIAEMh1eW2EtejAtOV9dKyg/OlwuW2EtejAtOV9dKykqJBIXCgRraW5kGAMgASgJQgm6SAZyBBABGEASFwoFbGFiZWwYBCABKAlCCLpIBXIDGP8BEiEKDXJlc291cmNlX3R5cGUYBSABKAlCCrpIB3IFEAEYgAESHwoLcmVzb3VyY2VfaWQYBiABKAlCCrpIB3IFEAEYgAQiNgoZTW9kdWxlUGxhY2VSZWNvcmRSZXNwb25zZRIZCgdub2RlX2lkGAEgASgJQgi6SAVyA7ABASKFAQojQ2hlY2tXb3JrQ29udGV4dFJlY29yZEFjY2Vzc1JlcXVlc3QSIQoNcmVzb3VyY2VfdHlwZRgBIAEoCUIKukgHcgUQARiAAhIfCgtyZXNvdXJjZV9pZBgCIAEoCUIKukgHcgUQARiABBIaCgZhY3Rpb24YAyABKAlCCrpIB3IFEAEYgAIiTgokQ2hlY2tXb3JrQ29udGV4dFJlY29yZEFjY2Vzc1Jlc3BvbnNlEg8KB2FsbG93ZWQYASABKAgSFQoNc2NvcGVfbm9kZV9pZBgCIAEoCSJ8CipNb2R1bGVFeGNoYW5nZURlbGVnYXRlZFJlYWRBdWRpZW5jZVJlcXVlc3QSHgoKYmluZGluZ19pZBgBIAEoCUIKukgHcgUQARiAARIuChlwYXJlbnRfd29ya19jb250ZXh0X3Rva2VuGAIgASgJQgu6SAhyBhABGICAAiKAAwovTW9kdWxlRXhjaGFuZ2VEZWxlZ2F0ZWRPcGVyYXRpb25BdWRpZW5jZVJlcXVlc3QSHgoKYmluZGluZ19pZBgBIAEoCUIKukgHcgUQARiAARIsChlwYXJlbnRfd29ya19jb250ZXh0X3Rva2VuGAIgASgJQgm6SAZyBBiAgAISDgoGbG9va3VwGAMgASgIEiIKDWRlbGVnYXRpb25faWQYBCABKAlCC7pICNgBAXIDsAEBOsoBukjGARrDAQozZXhjaGFuZ2VfZGVsZWdhdGVkX29wZXJhdGlvbl9hdWRpZW5jZS5vbmVfYXV0aG9yaXR5EkZleGFjdGx5IG9uZSBvZiBwYXJlbnRfd29ya19jb250ZXh0X3Rva2VuIGFuZCBkZWxlZ2F0aW9uX2lkIGlzIHJlcXVpcmVkGkQodGhpcy5wYXJlbnRfd29ya19jb250ZXh0X3Rva2VuICE9ICcnKSAhPSAodGhpcy5kZWxlZ2F0aW9uX2lkICE9ICcnKSq4AgoUTW9kdWxlQXVkaXRGaWVsZEtpbmQSJwojTU9EVUxFX0FVRElUX0ZJRUxEX0tJTkRfVU5TUEVDSUZJRUQQABIiCh5NT0RVTEVfQVVESVRfRklFTERfS0lORF9TVFJJTkcQARIgChxNT0RVTEVfQVVESVRfRklFTERfS0lORF9VVUlEEAISHwobTU9EVUxFX0FVRElUX0ZJRUxEX0tJTkRfSU5UEAMSIgoeTU9EVUxFX0FVRElUX0ZJRUxEX0tJTkRfTlVNQkVSEAQSIAocTU9EVUxFX0FVRElUX0ZJRUxEX0tJTkRfQk9PTBAFEiAKHE1PRFVMRV9BVURJVF9GSUVMRF9LSU5EX0VOVU0QBhIoCiRNT0RVTEVfQVVESVRfRklFTERfS0lORF9TVFJJTkdfQVJSQVkQByqhAQoaTW9kdWxlQXVkaXRFdmVudFZpc2liaWxpdHkSLQopTU9EVUxFX0FVRElUX0VWRU5UX1ZJU0lCSUxJVFlfVU5TUEVDSUZJRUQQABIoCiRNT0RVTEVfQVVESVRfRVZFTlRfVklTSUJJTElUWV9URU5BTlQQARIqCiZNT0RVTEVfQVVESVRfRVZFTlRfVklTSUJJTElUWV9FWFRFUk5BTBACKp4BChlNb2R1bGVBdWRpdEV2ZW50UmV0ZW50aW9uEiwKKE1PRFVMRV9BVURJVF9FVkVOVF9SRVRFTlRJT05fVU5TUEVDSUZJRUQQABIoCiRNT0RVTEVfQVVESVRfRVZFTlRfUkVURU5USU9OX0NPTlRFTlQQARIpCiVNT0RVTEVfQVVESVRfRVZFTlRfUkVURU5USU9OX1NFQ1VSSVRZEAIq3AEKGkRhdGFzb3VyY2VJdGVtUmVhZGVyc0Jhc2lzEi0KKURBVEFTT1VSQ0VfSVRFTV9SRUFERVJTX0JBU0lTX1VOU1BFQ0lGSUVEEAASLAooREFUQVNPVVJDRV9JVEVNX1JFQURFUlNfQkFTSVNfVFJBTlNMQVRFRBABEi8KK0RBVEFTT1VSQ0VfSVRFTV9SRUFERVJTX0JBU0lTX1NPVVJDRV9TQ09QRUQQAhIwCixEQVRBU09VUkNFX0lURU1fUkVBREVSU19CQVNJU19VTlRSQU5TTEFUQUJMRRADKmkKDUV2ZW50RGVsaXZlcnkSHgoaRVZFTlRfREVMSVZFUllfVU5TUEVDSUZJRUQQABIcChhFVkVOVF9ERUxJVkVSWV9VTk9SREVSRUQQARIaChZFVkVOVF9ERUxJVkVSWV9PUkRFUkVEEAIqfgoSTWV0YWRhdGFEaXNjbG9zdXJlEiMKH01FVEFEQVRBX0RJU0NMT1NVUkVfVU5TUEVDSUZJRUQQABIhCh1NRVRBREFUQV9ESVNDTE9TVVJFX0RJU0NMT1NFRBABEiAKHE1FVEFEQVRBX0RJU0NMT1NVUkVfV0lUSEhFTEQQAipcCg9Tb3VyY2VTeW5jU3RhZ2USIQodU09VUkNFX1NZTkNfU1RBR0VfVU5TUEVDSUZJRUQQABImCiJTT1VSQ0VfU1lOQ19TVEFHRV9DSEFOR0VTX0VOUVVFVUVEEAEy0SMKGU1vZHVsZUNhcGFiaWxpdGllc1NlcnZpY2USxQEKHUV4Y2hhbmdlRGVsZWdhdGVkUmVhZEF1ZGllbmNlEjwuc2Fhcy5hY2NvdW50cy52MS5Nb2R1bGVFeGNoYW5nZURlbGVnYXRlZFJlYWRBdWRpZW5jZVJlcXVlc3QaIy5zYWFzLmFjY291bnRzLnYxLklzc3VlZFdvcmtDb250ZXh0IkHC8xg9CAMQATABOisKJ3NhYXMubW9kdWxlLmRlbGVnYXRlZF9hdWRpZW5jZV9leGNoYW5nZRAEQAFIB1AEWARgARLPAQoiRXhjaGFuZ2VEZWxlZ2F0ZWRPcGVyYXRpb25BdWRpZW5jZRJBLnNhYXMuYWNjb3VudHMudjEuTW9kdWxlRXhjaGFuZ2VEZWxlZ2F0ZWRPcGVyYXRpb25BdWRpZW5jZVJlcXVlc3QaIy5zYWFzLmFjY291bnRzLnYxLklzc3VlZFdvcmtDb250ZXh0IkHC8xg9CAMQATABOisKJ3NhYXMubW9kdWxlLmRlbGVnYXRlZF9hdWRpZW5jZV9leGNoYW5nZRAEQAFIB1AEWARgARKnAQocQ2hlY2tXb3JrQ29udGV4dFJlY29yZEFjY2VzcxI1LnNhYXMuYWNjb3VudHMudjEuQ2hlY2tXb3JrQ29udGV4dFJlY29yZEFjY2Vzc1JlcXVlc3QaNi5zYWFzLmFjY291bnRzLnYxLkNoZWNrV29ya0NvbnRleHRSZWNvcmRBY2Nlc3NSZXNwb25zZSIYwvMYFAgDEAEwAToCEAFAAUgHUANYA2ABEqoBCh1MaXN0UmVhZGFibGVTb3VyY2VDb2xsZWN0aW9ucxI2LnNhYXMuYWNjb3VudHMudjEuTGlzdFJlYWRhYmxlU291cmNlQ29sbGVjdGlvbnNSZXF1ZXN0Gjcuc2Fhcy5hY2NvdW50cy52MS5MaXN0UmVhZGFibGVTb3VyY2VDb2xsZWN0aW9uc1Jlc3BvbnNlIhjC8xgUCAMQATABOgIQAUABSAdQA1gDYAESgAEKC1BsYWNlUmVjb3JkEiouc2Fhcy5hY2NvdW50cy52MS5Nb2R1bGVQbGFjZVJlY29yZFJlcXVlc3QaKy5zYWFzLmFjY291bnRzLnYxLk1vZHVsZVBsYWNlUmVjb3JkUmVzcG9uc2UiGMLzGBQIAxABMAE6AhABQAFIB1ADWANgARJ9CgpFbnF1ZXVlSm9iEikuc2Fhcy5hY2NvdW50cy52MS5Nb2R1bGVFbnF1ZXVlSm9iUmVxdWVzdBoqLnNhYXMuYWNjb3VudHMudjEuTW9kdWxlRW5xdWV1ZUpvYlJlc3BvbnNlIhjC8xgUCAMQATABOgIQAUABSAdQA1gDYAESegoJQ2xhaW1Kb2JzEiguc2Fhcy5hY2NvdW50cy52MS5Nb2R1bGVDbGFpbUpvYnNSZXF1ZXN0Gikuc2Fhcy5hY2NvdW50cy52MS5Nb2R1bGVDbGFpbUpvYnNSZXNwb25zZSIYwvMYFAgDEAEwAToCEAFAAUgHUANYA2ABEoMBCgxIZWFydGJlYXRKb2ISKy5zYWFzLmFjY291bnRzLnYxLk1vZHVsZUhlYXJ0YmVhdEpvYlJlcXVlc3QaLC5zYWFzLmFjY291bnRzLnYxLk1vZHVsZUhlYXJ0YmVhdEpvYlJlc3BvbnNlIhjC8xgUCAMQATABOgIQAUABSAdQA1gDYAESYQoGQWNrSm9iEiUuc2Fhcy5hY2NvdW50cy52MS5Nb2R1bGVBY2tKb2JSZXF1ZXN0GhYuZ29vZ2xlLnByb3RvYnVmLkVtcHR5IhjC8xgUCAMQATABOgIQAUABSAdQA1gDYAESYwoHTmFja0pvYhImLnNhYXMuYWNjb3VudHMudjEuTW9kdWxlTmFja0pvYlJlcXVlc3QaFi5nb29nbGUucHJvdG9idWYuRW1wdHkiGMLzGBQIAxABMAE6AhABQAFIB1ADWANgARJ9CgpOb3RpZnlVc2VyEikuc2Fhcy5hY2NvdW50cy52MS5Nb2R1bGVOb3RpZnlVc2VyUmVxdWVzdBoqLnNhYXMuYWNjb3VudHMudjEuTW9kdWxlTm90aWZ5VXNlclJlc3BvbnNlIhjC8xgUCAMQATABOgIQAUABSAdQA1gDYAESrQEKD05vdGlmeU9yZ0FkbWlucxIuLnNhYXMuYWNjb3VudHMudjEuTW9kdWxlTm90aWZ5T3JnQWRtaW5zUmVxdWVzdBovLnNhYXMuYWNjb3VudHMudjEuTW9kdWxlTm90aWZ5T3JnQWRtaW5zUmVzcG9uc2UiOcLzGDUIAxABMAE6Iwofc2Fhcy5tb2R1bGUub3JnX2FkbWluc19ub3RpZmllZBACQAFIB1ADWANgARKMAQoPUmVxdWVzdEFwcHJvdmFsEi4uc2Fhcy5hY2NvdW50cy52MS5Nb2R1bGVSZXF1ZXN0QXBwcm92YWxSZXF1ZXN0Gi8uc2Fhcy5hY2NvdW50cy52MS5Nb2R1bGVSZXF1ZXN0QXBwcm92YWxSZXNwb25zZSIYwvMYFAgDEAEwAToCEAFAAUgHUANYA2ABEnUKC0dldEFwcHJvdmFsEiouc2Fhcy5hY2NvdW50cy52MS5Nb2R1bGVHZXRBcHByb3ZhbFJlcXVlc3QaIC5zYWFzLmFjY291bnRzLnYxLk1vZHVsZUFwcHJvdmFsIhjC8xgUCAMQATABOgIQAUABSAdQA1gDYAEScQoOQ2FuY2VsQXBwcm92YWwSLS5zYWFzLmFjY291bnRzLnYxLk1vZHVsZUNhbmNlbEFwcHJvdmFsUmVxdWVzdBoWLmdvb2dsZS5wcm90b2J1Zi5FbXB0eSIYwvMYFAgDEAEwAToCEAFAAUgHUANYA2ABEnEKDkVtaXRBdWRpdEV2ZW50Ei0uc2Fhcy5hY2NvdW50cy52MS5Nb2R1bGVFbWl0QXVkaXRFdmVudFJlcXVlc3QaFi5nb29nbGUucHJvdG9idWYuRW1wdHkiGMLzGBQIAxABMAE6AhABQAFIB1ADWANgARLDAQoWRGVjbGFyZUF1ZGl0RXZlbnRUeXBlcxI1LnNhYXMuYWNjb3VudHMudjEuTW9kdWxlRGVjbGFyZUF1ZGl0RXZlbnRUeXBlc1JlcXVlc3QaNi5zYWFzLmFjY291bnRzLnYxLk1vZHVsZURlY2xhcmVBdWRpdEV2ZW50VHlwZXNSZXNwb25zZSI6wvMYNggDEAEwATokCiBzYWFzLm1vZHVsZS5hdWRpdF90eXBlc19kZWNsYXJlZBACQAFIB1ADWANgARKeAQoVTGlzdFN1YmplY3RWaXNpYmlsaXR5EjQuc2Fhcy5hY2NvdW50cy52MS5Nb2R1bGVMaXN0U3ViamVjdFZpc2liaWxpdHlSZXF1ZXN0GjUuc2Fhcy5hY2NvdW50cy52MS5Nb2R1bGVMaXN0U3ViamVjdFZpc2liaWxpdHlSZXNwb25zZSIYwvMYFAgDEAEwAToCEAFAAUgHUANYA2ABEo4BChNGZXRjaERhdGFzb3VyY2VCbG9iEiwuc2Fhcy5hY2NvdW50cy52MS5GZXRjaERhdGFzb3VyY2VCbG9iUmVxdWVzdBoqLnNhYXMuYWNjb3VudHMudjEuRmV0Y2hEYXRhc291cmNlQmxvYkNodW5rIhuIAgHC8xgUCAMQATABOgIQAUABSAdQA1gDYAEwARKOAQoURmV0Y2hEYXRhc291cmNlRmlsZXMSLS5zYWFzLmFjY291bnRzLnYxLkZldGNoRGF0YXNvdXJjZUZpbGVzUmVxdWVzdBorLnNhYXMuYWNjb3VudHMudjEuRmV0Y2hEYXRhc291cmNlRmlsZXNGcmFtZSIYwvMYFAgDEAEwAToCEAFAAUgHUANYA2ABMAEStgEKFk1pbnRNb2R1bGVSZWdpc3RyYXRpb24SLy5zYWFzLmFjY291bnRzLnYxLk1vZHVsZU1pbnRSZWdpc3RyYXRpb25SZXF1ZXN0GjAuc2Fhcy5hY2NvdW50cy52MS5Nb2R1bGVNaW50UmVnaXN0cmF0aW9uUmVzcG9uc2UiOcLzGDUIAxABMAE6Iwofc2Fhcy5tb2R1bGUucmVnaXN0cmF0aW9uX21pbnRlZBACQAFIB1AEWARgARK+AQoYTWludFNvbHV0aW9uUmVnaXN0cmF0aW9uEjEuc2Fhcy5hY2NvdW50cy52MS5Tb2x1dGlvbk1pbnRSZWdpc3RyYXRpb25SZXF1ZXN0GjIuc2Fhcy5hY2NvdW50cy52MS5Tb2x1dGlvbk1pbnRSZWdpc3RyYXRpb25SZXNwb25zZSI7wvMYNwgDEAEwATolCiFzYWFzLnNvbHV0aW9uLnJlZ2lzdHJhdGlvbl9taW50ZWQQAkABSAdQBFgEYAESswEKFU1pbnRNb2R1bGVXb3JrQ29udGV4dBIuLnNhYXMuYWNjb3VudHMudjEuTW9kdWxlTWludFdvcmtDb250ZXh0UmVxdWVzdBovLnNhYXMuYWNjb3VudHMudjEuTW9kdWxlTWludFdvcmtDb250ZXh0UmVzcG9uc2UiOcLzGDUIAxABMAE6Iwofc2Fhcy5tb2R1bGUud29ya19jb250ZXh0X21pbnRlZBACQAFIB1AEWARgARLHAQoaTWludE1vZHVsZU9wZXJhdGlvbkNvbnRleHQSMy5zYWFzLmFjY291bnRzLnYxLk1vZHVsZU1pbnRPcGVyYXRpb25Db250ZXh0UmVxdWVzdBo0LnNhYXMuYWNjb3VudHMudjEuTW9kdWxlTWludE9wZXJhdGlvbkNvbnRleHRSZXNwb25zZSI+wvMYOggDEAEwATooCiRzYWFzLm1vZHVsZS5vcGVyYXRpb25fY29udGV4dF9taW50ZWQQAkABSAdQBFgEYAESzgEKGk1pbnRTb3VyY2VPcGVyYXRpb25Db250ZXh0Ejkuc2Fhcy5hY2NvdW50cy52MS5Nb2R1bGVNaW50U291cmNlT3BlcmF0aW9uQ29udGV4dFJlcXVlc3QaOi5zYWFzLmFjY291bnRzLnYxLk1vZHVsZU1pbnRTb3VyY2VPcGVyYXRpb25Db250ZXh0UmVzcG9uc2UiOcLzGDUIAxABMAE6Iwofc2Fhcy5kYXRhc291cmNlLmRlbGVnYXRpb24udXNlZBACQAFIB1AEWARgARKDAQoMUHVibGlzaEV2ZW50Eisuc2Fhcy5hY2NvdW50cy52MS5Nb2R1bGVQdWJsaXNoRXZlbnRSZXF1ZXN0Giwuc2Fhcy5hY2NvdW50cy52MS5Nb2R1bGVQdWJsaXNoRXZlbnRSZXNwb25zZSIYwvMYFAgDEAEwAToCEAFAAUgHUANYA2ABEnoKCVN1YnNjcmliZRIoLnNhYXMuYWNjb3VudHMudjEuTW9kdWxlU3Vic2NyaWJlUmVxdWVzdBopLnNhYXMuYWNjb3VudHMudjEuTW9kdWxlU3Vic2NyaWJlUmVzcG9uc2UiGMLzGBQIAxABMAE6AhABQAFIB1ADWANgARJrCgtVbnN1YnNjcmliZRIqLnNhYXMuYWNjb3VudHMudjEuTW9kdWxlVW5zdWJzY3JpYmVSZXF1ZXN0GhYuZ29vZ2xlLnByb3RvYnVmLkVtcHR5IhjC8xgUCAMQATABOgIQAUABSAdQA1gDYAESkgEKEUxpc3RTdWJzY3JpcHRpb25zEjAuc2Fhcy5hY2NvdW50cy52MS5Nb2R1bGVMaXN0U3Vic2NyaXB0aW9uc1JlcXVlc3QaMS5zYWFzLmFjY291bnRzLnYxLk1vZHVsZUxpc3RTdWJzY3JpcHRpb25zUmVzcG9uc2UiGMLzGBQIAxABMAE6AhABQAFIB1ADWANgARKDAQoMUmVwbGF5RXZlbnRzEisuc2Fhcy5hY2NvdW50cy52MS5Nb2R1bGVSZXBsYXlFdmVudHNSZXF1ZXN0Giwuc2Fhcy5hY2NvdW50cy52MS5Nb2R1bGVSZXBsYXlFdmVudHNSZXNwb25zZSIYwvMYFAgDEAEwAToCEAFAAUgHUANYA2ABQr8BChRjb20uc2Fhcy5hY2NvdW50cy52MUIXTW9kdWxlQ2FwYWJpbGl0aWVzUHJvdG9QAVosYWNjb3VudHMvcGtnL2dlbi9zYWFzL2FjY291bnRzL3YxO2FjY291bnRzdjGiAgNTQViqAhBTYWFzLkFjY291bnRzLlYxygIQU2Fhc1xBY2NvdW50c1xWMeICHFNhYXNcQWNjb3VudHNcVjFcR1BCTWV0YWRhdGHqAhJTYWFzOjpBY2NvdW50czo6VjFiBnByb3RvMw", [file_saas_accounts_v1_work_contexts, file_buf_validate_validate, file_google_protobuf_empty, file_google_protobuf_duration, file_google_protobuf_struct, file_google_protobuf_timestamp, file_saas_accounts_v1_module_registration, file_saas_events_v1_events, file_saas_jobs_v1_jobs, file_saas_policy_v1_options]); /** * ModuleEnqueueJobRequest names the tenant the work belongs to and carries the @@ -794,6 +794,16 @@ export type ModuleAuditEventTypeDeclaration = Message<"saas.accounts.v1.ModuleAu * @generated from field: saas.accounts.v1.ModuleAuditEventVisibility visibility = 4; */ visibility: ModuleAuditEventVisibility; + + /** + * The type's retention class. It only grows: a later declaration may raise + * CONTENT to SECURITY, and one that would lower SECURITY — including by + * leaving this unset — is refused, because that would shorten how long the + * type's evidence is kept without anyone deciding so. + * + * @generated from field: saas.accounts.v1.ModuleAuditEventRetention retention = 5; + */ + retention: ModuleAuditEventRetention; }; /** @@ -2075,6 +2085,47 @@ export enum ModuleAuditEventVisibility { export const ModuleAuditEventVisibilitySchema: GenEnum = /*@__PURE__*/ enumDesc(file_saas_accounts_v1_module_capabilities, 1); +/** + * How long the full details of a declared type's events are kept once a + * warehouse is the audit store of record (ADR 0009). The envelope — who, which + * organization, what type, which target, when — and a hash of the details are + * kept for the compliance window either way; the class decides where the + * details go. + * + * @generated from enum saas.accounts.v1.ModuleAuditEventRetention + */ +export enum ModuleAuditEventRetention { + /** + * Unset means CONTENT, the class of a type that says nothing. + * + * @generated from enum value: MODULE_AUDIT_EVENT_RETENTION_UNSPECIFIED = 0; + */ + UNSPECIFIED = 0, + + /** + * Full details kept for the shorter, configured content window; the locked + * archive holds only the envelope and the hash. + * + * @generated from enum value: MODULE_AUDIT_EVENT_RETENTION_CONTENT = 1; + */ + CONTENT = 1, + + /** + * Full details kept for the compliance window, in the warehouse and the + * locked archive alike: authentication, permission and role changes, admin + * actions, data exports, configuration changes. + * + * @generated from enum value: MODULE_AUDIT_EVENT_RETENTION_SECURITY = 2; + */ + SECURITY = 2, +} + +/** + * Describes the enum saas.accounts.v1.ModuleAuditEventRetention. + */ +export const ModuleAuditEventRetentionSchema: GenEnum = /*@__PURE__*/ + enumDesc(file_saas_accounts_v1_module_capabilities, 2); + /** * DatasourceItemReadersBasis says where an item's readers came from. * @@ -2114,7 +2165,7 @@ export enum DatasourceItemReadersBasis { * Describes the enum saas.accounts.v1.DatasourceItemReadersBasis. */ export const DatasourceItemReadersBasisSchema: GenEnum = /*@__PURE__*/ - enumDesc(file_saas_accounts_v1_module_capabilities, 2); + enumDesc(file_saas_accounts_v1_module_capabilities, 3); /** * EventDelivery is the ordering guarantee a subscription requests, mirroring @@ -2144,7 +2195,7 @@ export enum EventDelivery { * Describes the enum saas.accounts.v1.EventDelivery. */ export const EventDeliverySchema: GenEnum = /*@__PURE__*/ - enumDesc(file_saas_accounts_v1_module_capabilities, 3); + enumDesc(file_saas_accounts_v1_module_capabilities, 4); /** * MetadataDisclosure separates "the host withheld this from you" from "there is @@ -2175,7 +2226,7 @@ export enum MetadataDisclosure { * Describes the enum saas.accounts.v1.MetadataDisclosure. */ export const MetadataDisclosureSchema: GenEnum = /*@__PURE__*/ - enumDesc(file_saas_accounts_v1_module_capabilities, 4); + enumDesc(file_saas_accounts_v1_module_capabilities, 5); /** * SourceSyncStage names which occurrence CollectionSyncProvenance.at records. @@ -2206,7 +2257,7 @@ export enum SourceSyncStage { * Describes the enum saas.accounts.v1.SourceSyncStage. */ export const SourceSyncStageSchema: GenEnum = /*@__PURE__*/ - enumDesc(file_saas_accounts_v1_module_capabilities, 5); + enumDesc(file_saas_accounts_v1_module_capabilities, 6); /** * @generated from service saas.accounts.v1.ModuleCapabilitiesService diff --git a/module/services/store/migrations/18_audit_event_type_retention_class.down.sql b/module/services/store/migrations/18_audit_event_type_retention_class.down.sql new file mode 100644 index 000000000..bf29b55a7 --- /dev/null +++ b/module/services/store/migrations/18_audit_event_type_retention_class.down.sql @@ -0,0 +1,2 @@ +ALTER TABLE public.audit_event_types DROP CONSTRAINT audit_event_types_retention_class; +ALTER TABLE public.audit_event_types DROP COLUMN retention_class; diff --git a/module/services/store/migrations/18_audit_event_type_retention_class.up.sql b/module/services/store/migrations/18_audit_event_type_retention_class.up.sql new file mode 100644 index 000000000..27ff7aca7 --- /dev/null +++ b/module/services/store/migrations/18_audit_event_type_retention_class.up.sql @@ -0,0 +1,18 @@ +-- An audit event type's retention class (ADR 0009): whether a warehouse store +-- of record keeps the type's full details for the compliance window +-- ('security') or for the shorter content window ('content'). The code-owned +-- catalog declares one per type and the startup projection writes it here, as +-- it writes visibility; a solution- or module-declared type states its own, or +-- gets 'content' by saying nothing. +-- +-- 'content' is the default and the backfill: a declared row written before this +-- migration was declared under no class at all, which is what a declaration +-- that says nothing means; every code-owned row is rewritten from the catalog +-- at the next startup. +ALTER TABLE public.audit_event_types + ADD COLUMN retention_class text NOT NULL DEFAULT 'content'; + +-- The relay routes a type's details by this column, so a value outside the +-- vocabulary is refused at the table rather than read as either class. +ALTER TABLE public.audit_event_types + ADD CONSTRAINT audit_event_types_retention_class CHECK (retention_class IN ('security', 'content')); From 510b7ae82017f229b768b76f3109595131ce13f2 Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Fri, 2 Oct 2026 19:20:20 -0400 Subject: [PATCH 06/69] docs(adr): settle warehouse duplicates, declared retention classes and swap settings in ADR 0009 The warehouse may hold an event more than once after a redelivery: BigQuery collapses a repeated insert id only briefly, and a relay that restarts between its writes and its delete delivers the batch again. Every read returns each event once by event id, as archive readers already do, and the conformance suite tests that instead of append idempotency. Committed write streams are rejected: stream state to carry across restarts for no compliance gain. A solution- or module-declared type states its retention class in its declaration; one that states none is content, any other value is refused, and a declared class only grows. A code-owned type still cannot register without one. A deployment selecting a swap value provides the warehouse settings, AUDIT_ARCHIVE_URL (its scheme picks the archive writer, gs:// for GCS), AUDIT_CONTENT_RETENTION_DAYS and AUDIT_DEPLOYMENT_ID. Co-Authored-By: Claude Opus 5.5 --- module/docs/adr/0009-audit-store-swap.md | 46 +++++++++++++++++++----- 1 file changed, 37 insertions(+), 9 deletions(-) diff --git a/module/docs/adr/0009-audit-store-swap.md b/module/docs/adr/0009-audit-store-swap.md index d0950f7d8..dc34378ea 100644 --- a/module/docs/adr/0009-audit-store-swap.md +++ b/module/docs/adr/0009-audit-store-swap.md @@ -56,6 +56,16 @@ Postgres transaction. default for every deployment that does not opt in. `both` — ADR 0006's tee — keeps its current behaviour, for compatibility. `external` stays refused. + A deployment that selects a swap value sets, beside `AUDIT_SINK`: the + warehouse (`AUDIT_BIGQUERY_PROJECT` and `AUDIT_BIGQUERY_DATASET` for + `bigquery`), `AUDIT_ARCHIVE_URL` — the locked archive as a URL whose scheme + picks the archive writer, `gs://` for GCS, any scheme without a + writer refused at startup — `AUDIT_CONTENT_RETENTION_DAYS` (the details + window of item 5) and `AUDIT_DEPLOYMENT_ID`, stamped on every record. A + missing setting fails startup. Credentials are the platform's ambient + identity (on Kubernetes, the pod's workload identity), never a key in + configuration. + 2. **Postgres keeps only the transactional queue.** `EmitTx` still writes inside the caller's transaction (and `Emit` inside its own, for the observational types of ADR 0003's durability amendment), but what it writes is a row in a @@ -74,8 +84,9 @@ Postgres transaction. A relay drains the queue: - in **batches**; - - **at-least-once, idempotent by event id** — a redelivered event leaves one - record in the warehouse; + - **at-least-once, keyed by event id** — a batch redelivered after a crash + may leave an event in the warehouse twice, as it may in the archive, and + every read returns each event once by event id (item 4); - **ordered per organization** (platform-level events form one more ordering key); - each batch is **appended to the warehouse and written as one object to the @@ -108,13 +119,27 @@ Postgres transaction. Adapter-native row policies (BigQuery row-level access policies, ClickHouse row policies) are defence in depth, not the boundary. -5. **Retention tiers.** Every registered event type — code-owned, or declared - by a solution or a composed module — declares a **retention class** in the - typed registry (ADR 0003): + Every read returns each event once by event id. The warehouse may hold an + event more than once after a redelivery (item 2): BigQuery collapses a + repeated streaming insert id only within a short window, and a relay that + restarts between its writes and its delete delivers the batch again. The + read half deduplicates exactly as every archive reader does (item 5), and + the per-event hash makes the copies provably identical. + +5. **Retention tiers.** Every registered event type has a **retention class** + in the typed registry (ADR 0003): - **security** — authentication, failed authentication, permission and role changes, admin actions, data exports, configuration changes; - **content** — everything else. + A code-owned type declares its class beside its durability, and cannot be + registered without one. A type declared by a solution (its manifest) or a + composed module (its declaration call) states its class in that + declaration; one that states none is **content**, and a value outside the + two is refused. A declared class only grows, as a `pii` mark does: a + re-declaration may raise `content` to `security` and is refused if it would + lower it, omission included. + The retention class is its own field. `Category`'s `security` value names a narrower set and is not reused for it. @@ -160,9 +185,9 @@ Postgres transaction. deleted and the append-only triggers stay as they are. From then on `audit_events` receives no new rows, and Postgres holds only the queue. -8. **Conformance.** Every adapter passes one shared test suite: append - idempotency (a redelivered batch leaves one record per event id), batch - append, list and aggregate parity with the Postgres implementation over the +8. **Conformance.** Every adapter passes one shared test suite: read + deduplication (after a batch is appended twice, every read returns each + event once by event id), batch append, list and aggregate parity with the Postgres implementation over the same fixture, refusal of a query without tenant scope, and retention-class routing (a content-class event's full details land only in the details table, and only its content-free form in the archive). @@ -180,6 +205,7 @@ The queue between the transaction and the warehouse (item 2): | Queue rows in `audit_events`, with the no-delete trigger relaxed for them | Weakens the append-only guarantee on the table every `postgres` and `both` deployment keeps its history in, to serve a mode those deployments do not use. | | Fill the archive from the warehouse later, on a schedule | Opens a window in which the compliance copy lags the store of record, and adds a second job to watch. | | Conditional create-if-absent archive object names, so a retry cannot duplicate | Needs the same batch composition on every retry — more moving parts for no compliance gain. | +| Committed write streams with offsets, so a redelivery cannot duplicate in the warehouse | Stream state the relay must carry across restarts and recover on every one — more machinery for no compliance gain, since every read already deduplicates by event id, as archive readers do. | Where the adapters live (item 3): @@ -222,7 +248,9 @@ The sink shape itself: Under a swap value it becomes a warehouse query for the latest actor ids, then a join to display names in the service from Postgres — outside that snapshot. - `pkg/business/audit_registry.go` — `AuditEventDefinition` gains the retention - class, required the way `Durability` is. + class, required the way `Durability` is. A declared type's class comes from + its declaration — the manifest event, or the module declaration message — + and is stored on its `audit_event_types` row, `content` when it states none. - `pkg/business/retention.go` — partition drop keeps running for `audit_events`, and is also how the history migration (item 7) removes copied partitions. Under a swap value, tier expiry belongs to the warehouse (TTL or table From fe18a78a6f4d391041dc5bd11b6fea1cd5370608 Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Sat, 3 Oct 2026 20:14:46 -0400 Subject: [PATCH 07/69] feat(audit): read the BigQuery store of record through the Storage Read API ADR 0009 (#996) step 3: under AUDIT_SINK=bigquery the activity list, aggregation, exports and the readable-source query now read the warehouse. postgres and both read audit_events exactly as before. The service that reads also holds the append grant (bigquery.tables.updateData), and append plus bigquery.jobs.create would authorise DELETE, UPDATE and MERGE against the store of record. So reads run no query job: each one is a Storage Read API session whose row restriction carries this deployment, the caller's organization (unless the read is the explicit platform read), the time window, so partitions prune, and the event types when filtered. The service then sorts, pages, deduplicates by event id, joins content details within their window and aggregates (pkg/auditstore/auditeval), with the Postgres reads' semantics: jsonb containment and ->> text, the numeric reading of payload keys, UTC time buckets, percentile_cont, NULL-aware sample counts, and the same ordering and page token. The list reads the newest day first, then windows growing eightfold, and stops once the page and the proof of a next one are in hand. A stream broken by a transient error resumes at the row reached. business.AuditStore now carries the read half beside the writer: AuditReader (list, aggregate, export) and AuditSourceSyncReader. Every read names an AuditReadScope, one organization or the explicit platform read, and every store refuses a read without one. The Postgres reads become postgresAuditReader, the same transactions and SQL as before. AggregateAuditLogForReader under a swap value checks the reader's grant in the tenant transaction, then reads the warehouse. LatestSourceSyncRequests under a swap value reads each source's newest request from the warehouse, then joins principals' display names inside the source-read snapshot (Store.SourceSyncRequesterLabels). The control-plane call-site list moves the platform reads from Service.QueryAuditLog and Service.AggregateAuditLog to postgresAuditReader.within. Parity: one fixture is written through the emitter into audit_events and through the queue and the relay into a fake Storage Read API that evaluates the reader's row restrictions; the warehouse also holds events delivered twice, another deployment's and another organization's. List pages, aggregations, both export formats and the readable-source query read back identically, against Postgres 16. Co-Authored-By: Claude Opus 5.5 --- module/FEATURES.md | 4 +- module/services/accounts/AGENTS.md | 12 + module/services/accounts/code/audit_sink.go | 35 +- module/services/accounts/code/go.mod | 4 +- .../pkg/auditstore/auditeval/aggregate.go | 315 +++++++ .../code/pkg/auditstore/auditeval/eval.go | 303 +++++++ .../pkg/auditstore/auditeval/eval_test.go | 174 ++++ .../code/pkg/auditstore/auditeval/jsonb.go | 259 ++++++ .../code/pkg/auditstore/auditeval/page.go | 129 +++ .../auditstore/bigqueryfake/bigqueryfake.go | 690 +++++++++++++++ .../auditstore/bigquerystore/bigquerystore.go | 84 +- .../pkg/auditstore/bigquerystore/reader.go | 813 ++++++++++++++++++ .../auditstore/bigquerystore/reader_test.go | 274 ++++++ .../auditstore/bigquerystore/restriction.go | 88 ++ .../accounts/code/pkg/business/audit.go | 158 ++-- .../code/pkg/business/audit_export.go | 33 +- .../accounts/code/pkg/business/audit_store.go | 11 +- .../pkg/business/audit_store_parity_test.go | 424 +++++++++ .../code/pkg/business/audit_store_read.go | 281 ++++++ .../code/pkg/business/readable_sources.go | 42 +- .../accounts/code/pkg/business/service.go | 18 + .../accounts/code/pkg/business/store.go | 6 + .../accounts/code/pkg/infra/postgres_audit.go | 28 +- .../pkg/infra/postgres_readable_sources.go | 39 + .../testdata/control_plane_call_sites.txt | 3 +- module/services/accounts/code/work.go | 9 +- 26 files changed, 4092 insertions(+), 144 deletions(-) create mode 100644 module/services/accounts/code/pkg/auditstore/auditeval/aggregate.go create mode 100644 module/services/accounts/code/pkg/auditstore/auditeval/eval.go create mode 100644 module/services/accounts/code/pkg/auditstore/auditeval/eval_test.go create mode 100644 module/services/accounts/code/pkg/auditstore/auditeval/jsonb.go create mode 100644 module/services/accounts/code/pkg/auditstore/auditeval/page.go create mode 100644 module/services/accounts/code/pkg/auditstore/bigqueryfake/bigqueryfake.go create mode 100644 module/services/accounts/code/pkg/auditstore/bigquerystore/reader.go create mode 100644 module/services/accounts/code/pkg/auditstore/bigquerystore/reader_test.go create mode 100644 module/services/accounts/code/pkg/auditstore/bigquerystore/restriction.go create mode 100644 module/services/accounts/code/pkg/business/audit_store_parity_test.go create mode 100644 module/services/accounts/code/pkg/business/audit_store_read.go diff --git a/module/FEATURES.md b/module/FEATURES.md index 380715eaa..d88a70603 100644 --- a/module/FEATURES.md +++ b/module/FEATURES.md @@ -398,7 +398,7 @@ see `JOBS.md` for the exact boundary and sequencing. | Export (JSON/CSV) | ✅ | `audit_export.go` | | Impersonation tracking | ✅ | Records both real actor + viewed-as user | | Replay / event sourcing| ❌ | Audit log is read-only history; not used to reconstruct state | -| Warehouse store of record | 🟡 | `AUDIT_SINK=bigquery` (ADR 0009): each record commits to a Postgres queue on the mutation's transaction and a relay delivers it to BigQuery and a locked GCS archive. Writes only — the activity list, aggregates and exports still read Postgres until reads move; ClickHouse is not built | +| Warehouse store of record | 🟡 | `AUDIT_SINK=bigquery` (ADR 0009): each record commits to a Postgres queue on the mutation's transaction and a relay delivers it to BigQuery and a locked GCS archive. The activity list, aggregates, exports and the readable-source query read BigQuery through the Storage Read API — no query jobs — scoped to the caller's organization and the deployment, with each event returned once; Postgres history is not yet copied over; ClickHouse is not built | ### Admin panel @@ -655,7 +655,7 @@ Environment variables consumed by the api: | `POSTHOG_HOST` | Explicit HTTPS or local capture origin | | `POSTHOG_API_HOST` | Separate PostHog management/deletion origin | | `AUDIT_SINK` | `postgres` (default), `both` (tee to `AUDIT_EXTERNAL_URL`, optional `AUDIT_EXTERNAL_TOKEN`), or `bigquery` (swap the audit store of record, ADR 0009); `external` is refused | -| `AUDIT_BIGQUERY_PROJECT`, `AUDIT_BIGQUERY_DATASET` | Required under `bigquery`: the store of record. The dataset must exist; the `audit_events` and `audit_event_details` tables are created if missing. Credentials are Application Default Credentials only | +| `AUDIT_BIGQUERY_PROJECT`, `AUDIT_BIGQUERY_DATASET` | Required under `bigquery`: the store of record. The dataset must exist; the `audit_events` and `audit_event_details` tables are created if missing. Credentials are Application Default Credentials only. The service's identity holds no `bigquery.jobs.create`: it writes with streaming inserts and reads through the Storage Read API (`bigquery.tables.getData`, `bigquery.readsessions.create`/`getData`/`update`), because append plus job creation would permit DML | | `AUDIT_ARCHIVE_URL` | Required under `bigquery`: the locked archive, as `gs://` (the scheme picks the writer; GCS is the one built). The bucket carries a retention lock the deployment sets and receives one object per relay batch | | `AUDIT_DEPLOYMENT_ID` | Required under `bigquery`: stamped on every record and the archive path | | `AUDIT_CONTENT_RETENTION_DAYS` | Required under `bigquery`: days a content-class event's full details are kept (the details table's partition expiration) | diff --git a/module/services/accounts/AGENTS.md b/module/services/accounts/AGENTS.md index 965defc32..7267ecd06 100644 --- a/module/services/accounts/AGENTS.md +++ b/module/services/accounts/AGENTS.md @@ -71,6 +71,18 @@ survives a restart and reaches every replica, and is only served when it is warehouse store of record keeps its full details. The class only grows, like `pii`: a re-declaration may raise it to `security` and is refused if it would lower it, omission included. +- **Audit reads go through the Service**, never straight to `audit_events`: + `QueryAuditLog`, `AggregateAuditLog(ForReader)`, `ExportAuditLog` and + `LatestSourceSyncRequests` read the audit store of record — `audit_events` + under `postgres`/`both`, the warehouse under a swap value (ADR 0009), where + `audit_events` receives no new rows. Each read names its scope + (`AuditReadScope`: one organization, or the explicit platform read) and the + store refuses one without it. The BigQuery reads run no query job — the + service also holds the append grant, and append plus job creation would + allow DML — so they use the Storage Read API with row restrictions and are + evaluated in the service (`pkg/auditstore/auditeval`) with the Postgres + reads' semantics; `pkg/business/audit_store_parity_test.go` holds the two to + one fixture. - A composed **module** with no frontend half declares its own audit event types through `ModuleCapabilitiesService.DeclareAuditEventTypes` (`pkg/business/module_audit_declarations.go`): the same validator diff --git a/module/services/accounts/code/audit_sink.go b/module/services/accounts/code/audit_sink.go index a38d8c123..9142b7e99 100644 --- a/module/services/accounts/code/audit_sink.go +++ b/module/services/accounts/code/audit_sink.go @@ -15,6 +15,7 @@ import ( "time" "cloud.google.com/go/bigquery" + bqstorage "cloud.google.com/go/bigquery/storage/apiv1" "cloud.google.com/go/storage" ) @@ -193,20 +194,21 @@ func configuredBigQueryAuditSwap() (*bigQueryAuditSwap, error) { return swap, nil } -// newBigQueryAuditRelay builds the relay of the bigquery swap value: the -// BigQuery store of record (its tables created if missing), the GCS archive, -// and the queue on the relay's own pool. Both clients use Application Default -// Credentials. The returned close releases them all. -func newBigQueryAuditRelay(ctx context.Context, types business.DeclaredAuditEventTypeReader, swap *bigQueryAuditSwap) (*business.AuditRelay, func(), error) { +// newBigQueryAuditSwap builds the bigquery swap value: the BigQuery store of +// record (its tables created if missing) with its read half on the Storage +// Read API, the GCS archive, and the relay that drains the queue on the +// relay's own pool. Every client uses Application Default Credentials. The +// service reads the returned store; the returned close releases them all. +func newBigQueryAuditSwap(ctx context.Context, types business.DeclaredAuditEventTypeReader, swap *bigQueryAuditSwap) (*business.AuditRelay, business.AuditStore, func(), error) { var closers []func() closeAll := func() { for i := len(closers) - 1; i >= 0; i-- { closers[i]() } } - fail := func(err error) (*business.AuditRelay, func(), error) { + fail := func(err error) (*business.AuditRelay, business.AuditStore, func(), error) { closeAll() - return nil, nil, err + return nil, nil, nil, err } bigQueryClient, err := bigquery.NewClient(ctx, swap.project) @@ -214,9 +216,26 @@ func newBigQueryAuditRelay(ctx context.Context, types business.DeclaredAuditEven return fail(fmt.Errorf("audit store: bigquery client: %w", err)) } closers = append(closers, func() { _ = bigQueryClient.Close() }) + // Reads run no query job: the service holds the append grant, and append + // plus job creation would let it run DML against the store of record. + readClient, err := bqstorage.NewBigQueryReadClient(ctx) + if err != nil { + return fail(fmt.Errorf("audit store: bigquery storage read client: %w", err)) + } + closers = append(closers, func() { _ = readClient.Close() }) + reader, err := bigquerystore.NewReader(bigquerystore.ReadConfig{ + Client: readClient, + Project: swap.project, + Dataset: swap.dataset, + DeploymentID: swap.deploymentID, + }) + if err != nil { + return fail(err) + } warehouse, err := bigquerystore.New(bigQueryClient, bigquerystore.Config{ Dataset: swap.dataset, ContentDetailRetention: swap.contentDetailRetention, + Reader: reader, }) if err != nil { return fail(err) @@ -263,5 +282,5 @@ func newBigQueryAuditRelay(ctx context.Context, types business.DeclaredAuditEven if err != nil { return fail(err) } - return relay, closeAll, nil + return relay, warehouse, closeAll, nil } diff --git a/module/services/accounts/code/go.mod b/module/services/accounts/code/go.mod index 989d40d10..472ab909d 100644 --- a/module/services/accounts/code/go.mod +++ b/module/services/accounts/code/go.mod @@ -9,6 +9,7 @@ require ( cloud.google.com/go/storage v1.69.0 connectrpc.com/connect v1.21.0 connectrpc.com/otelconnect v0.9.0 + github.com/apache/arrow/go/v15 v15.0.2 github.com/codefly-dev/core v0.5.11-0.20260926231217-d481a2d49683 github.com/codefly-dev/interface-cache/go/cache v0.2.2-0.20260926205419-327a6e1d9aba github.com/codefly-dev/sdk-go v0.2.1-0.20260926212413-b91ce91eb23e @@ -18,6 +19,7 @@ require ( github.com/go-webauthn/webauthn v0.18.0 github.com/golang-jwt/jwt/v5 v5.3.1 github.com/google/uuid v1.6.0 + github.com/googleapis/gax-go/v2 v2.26.2 github.com/grpc-ecosystem/grpc-gateway/v2 v2.30.0 github.com/jackc/pgx/v5 v5.10.0 github.com/prometheus/client_golang v1.24.1 @@ -65,7 +67,6 @@ require ( github.com/Masterminds/semver/v3 v3.5.0 // indirect github.com/VividCortex/ewma v1.2.0 // indirect github.com/antlr4-go/antlr/v4 v4.13.1 // indirect - github.com/apache/arrow/go/v15 v15.0.2 // indirect github.com/beorn7/perks v1.0.1 // indirect github.com/blang/semver v3.5.1+incompatible // indirect github.com/cenkalti/backoff/v5 v5.0.3 // indirect @@ -107,7 +108,6 @@ require ( github.com/google/go-tpm v0.9.8 // indirect github.com/google/s2a-go v0.1.9 // indirect github.com/googleapis/enterprise-certificate-proxy v0.3.17 // indirect - github.com/googleapis/gax-go/v2 v2.26.2 // indirect github.com/hashicorp/errwrap v1.1.0 // indirect github.com/hashicorp/go-multierror v1.1.1 // indirect github.com/jackc/pgpassfile v1.0.0 // indirect diff --git a/module/services/accounts/code/pkg/auditstore/auditeval/aggregate.go b/module/services/accounts/code/pkg/auditstore/auditeval/aggregate.go new file mode 100644 index 000000000..c3d4da7dc --- /dev/null +++ b/module/services/accounts/code/pkg/auditstore/auditeval/aggregate.go @@ -0,0 +1,315 @@ +package auditeval + +import ( + "errors" + "fmt" + "math" + "sort" + "strings" + "time" + + "accounts/pkg/business" +) + +// Aggregator computes AggregateAuditLog over the events offered to it, with +// the Postgres aggregation's semantics: +// +// - dimensions: event_type; category from the event-type index ("unknown" +// for a type it does not name); actor (the actor id, or ""); time, the UTC +// start of the event's day, ISO week or month, printed as Postgres prints +// it under the UTC session time zone ("2026-10-05T00:00:00+00"); and +// payload:, the key's ->> text or ""; +// - metrics: count; count_distinct over a column or payload key's text, +// ignoring NULLs; sum, avg, min, max and percentile (percentile_cont) over +// a payload key read as a number, ignoring what is not one; +// - every metric's sample count is the number of non-NULL inputs, and a +// metric with none is absent rather than 0; a derived ratio is absent when +// an operand is absent or the denominator is 0; +// - order: a sole time dimension ascending; otherwise by count descending, +// then by the dimensions ascending, compared bytewise. +// +// An event without details (a content-class event past the content window) +// has no payload: its payload dimensions are "" and its payload metrics NULL. +type Aggregator struct { + spec business.AuditAggregationSpec + dims []string + types business.AuditEventTypeIndex + aliases []string + groups map[string]*group + order []*group +} + +type group struct { + keys []string + count int64 + metrics []*accumulator +} + +type accumulator struct { + samples int64 + sum float64 + min, max float64 + values []float64 + distinct map[string]struct{} +} + +// NewAggregator prepares spec, which the caller has validated. It refuses a +// spec that groups or counts by category without the event-type index, and +// any dimension or metric the Postgres aggregation would not run either. +func NewAggregator(spec business.AuditAggregationSpec, types business.AuditEventTypeIndex) (*Aggregator, error) { + dims := spec.GroupBy + if len(dims) == 0 { + dims = []string{"event_type"} + } + needsTypes := false + for _, dim := range dims { + if _, ok := payloadKey(dim); ok { + continue + } + switch dim { + case "event_type", "actor", "time": + case "category": + needsTypes = true + default: + return nil, fmt.Errorf("audit: unhandled group dimension %q", dim) + } + } + aliases := make([]string, len(spec.Metrics)) + for i, metric := range spec.Metrics { + switch metric.Op { + case "", "count": + case "count_distinct": + if _, ok := payloadKey(metric.Field); !ok { + switch metric.Field { + case "actor_id", "resource_id", "event_type", "resource": + case "category": + needsTypes = true + default: + return nil, fmt.Errorf("audit: unhandled count_distinct field %q", metric.Field) + } + } + case "sum", "avg", "min", "max", "percentile": + default: + return nil, fmt.Errorf("audit: unhandled metric op %q", metric.Op) + } + aliases[i] = metric.ResolvedAlias() + } + if needsTypes && types == nil { + return nil, errors.New("audit: a category dimension or count needs the event-type index") + } + return &Aggregator{ + spec: spec, + dims: dims, + types: types, + aliases: aliases, + groups: map[string]*group{}, + }, nil +} + +func payloadKey(field string) (string, bool) { + key, ok := strings.CutPrefix(field, "payload:") + return key, ok && key != "" +} + +// Add counts one matching event. The caller offers each event once. +func (a *Aggregator) Add(event *Event) error { + payload, err := event.payload() + if err != nil { + return fmt.Errorf("audit read: details of event %s: %w", event.Entry.ID, err) + } + keys := make([]string, len(a.dims)) + for i, dim := range a.dims { + keys[i] = a.dimension(dim, event.Entry, payload) + } + id := groupID(keys) + g, ok := a.groups[id] + if !ok { + g = &group{keys: keys, metrics: make([]*accumulator, len(a.spec.Metrics))} + for i := range g.metrics { + g.metrics[i] = &accumulator{min: math.Inf(1), max: math.Inf(-1)} + } + a.groups[id] = g + a.order = append(a.order, g) + } + g.count++ + for i, metric := range a.spec.Metrics { + acc := g.metrics[i] + switch metric.Op { + case "", "count": + acc.samples++ + case "count_distinct": + value, ok := a.distinctValue(metric.Field, event.Entry, payload) + if !ok { + continue + } + acc.samples++ + if acc.distinct == nil { + acc.distinct = map[string]struct{}{} + } + acc.distinct[value] = struct{}{} + default: + key, _ := payloadKey(metric.Field) + value, ok := jsonbNumeric(payload, key) + if !ok { + continue + } + acc.samples++ + acc.sum += value + acc.min = math.Min(acc.min, value) + acc.max = math.Max(acc.max, value) + if metric.Op == "percentile" { + acc.values = append(acc.values, value) + } + } + } + return nil +} + +func groupID(keys []string) string { + var id strings.Builder + for _, key := range keys { + id.WriteString(fmt.Sprintf("%d:", len(key))) + id.WriteString(key) + } + return id.String() +} + +func (a *Aggregator) dimension(dim string, entry business.AuditEntry, payload map[string]any) string { + if key, ok := payloadKey(dim); ok { + text, _ := jsonbText(payload, key) + return text + } + switch dim { + case "category": + if facts, ok := a.types[string(entry.EventType)]; ok { + return facts.Category + } + return "unknown" + case "actor": + return entry.ActorID + case "time": + return TimeBucket(entry.CreatedAt, a.spec.Bucket) + default: + return string(entry.EventType) + } +} + +// TimeBucket is the time dimension's key: the UTC start of the event's day +// (the default), ISO week or month, as Postgres's +// to_char(date_trunc(grain, created_at), 'YYYY-MM-DD"T"HH24:MI:SSOF') prints it +// under the UTC session time zone. +func TimeBucket(at time.Time, bucket string) string { + at = at.UTC() + day := time.Date(at.Year(), at.Month(), at.Day(), 0, 0, 0, 0, time.UTC) + switch bucket { + case "week": + day = day.AddDate(0, 0, -((int(day.Weekday()) + 6) % 7)) + case "month": + day = time.Date(at.Year(), at.Month(), 1, 0, 0, 0, 0, time.UTC) + } + return day.Format("2006-01-02T15:04:05") + "+00" +} + +func (a *Aggregator) distinctValue(field string, entry business.AuditEntry, payload map[string]any) (string, bool) { + if key, ok := payloadKey(field); ok { + return jsonbText(payload, key) + } + switch field { + case "actor_id": + return entry.ActorID, entry.ActorID != "" + case "resource_id": + return entry.ResourceID, entry.ResourceID != "" + case "event_type": + return string(entry.EventType), true + case "resource": + return entry.Resource, true + case "category": + facts, ok := a.types[string(entry.EventType)] + return facts.Category, ok + default: + return "", false + } +} + +// Buckets is the aggregation result, ordered as Postgres orders it; nil when no +// event matched. +func (a *Aggregator) Buckets() []business.AuditAggregateBucket { + groups := append([]*group(nil), a.order...) + soleTime := len(a.dims) == 1 && a.dims[0] == "time" + sort.SliceStable(groups, func(i, j int) bool { + gi, gj := groups[i], groups[j] + if !soleTime && gi.count != gj.count { + return gi.count > gj.count + } + for k := range gi.keys { + if gi.keys[k] != gj.keys[k] { + return gi.keys[k] < gj.keys[k] + } + } + return false + }) + var out []business.AuditAggregateBucket + for _, g := range groups { + bucket := business.AuditAggregateBucket{ + Keys: append([]string(nil), g.keys...), + Count: g.count, + Samples: make(map[string]int64, len(a.aliases)), + Metrics: make(map[string]float64, len(a.aliases)+len(a.spec.Derived)), + } + bucket.Key = bucket.Keys[0] + for i, metric := range a.spec.Metrics { + alias := a.aliases[i] + acc := g.metrics[i] + bucket.Samples[alias] = acc.samples + if acc.samples == 0 { + continue + } + switch metric.Op { + case "", "count": + bucket.Metrics[alias] = float64(g.count) + case "count_distinct": + bucket.Metrics[alias] = float64(len(acc.distinct)) + case "sum": + bucket.Metrics[alias] = acc.sum + case "avg": + bucket.Metrics[alias] = acc.sum / float64(acc.samples) + case "min": + bucket.Metrics[alias] = acc.min + case "max": + bucket.Metrics[alias] = acc.max + case "percentile": + bucket.Metrics[alias] = percentileCont(acc.values, metric.Percentile) + } + } + for _, derived := range a.spec.Derived { + numerator, numeratorOK := bucket.Metrics[derived.Numerator] + denominator, denominatorOK := bucket.Metrics[derived.Denominator] + if numeratorOK && denominatorOK && denominator != 0 { + bucket.Metrics[derived.Alias] = numerator / denominator + } + } + out = append(out, bucket) + } + return out +} + +// percentileCont is Postgres's percentile_cont: the value at fraction p of the +// sorted inputs, interpolated linearly between the two nearest. +// +// The interpolation is written as Postgres's float8_lerp writes it, lo + pct +// * (hi - lo), and left for the compiler to treat as C compilers treat it: Go +// and GCC both fuse that multiply-add into one rounding on arm64 and neither +// does on baseline amd64, so the result matches the Postgres of the same +// architecture to the last bit. Forcing either rounding would match only one. +func percentileCont(values []float64, p float64) float64 { + sorted := append([]float64(nil), values...) + sort.Float64s(sorted) + position := p * float64(len(sorted)-1) + first := math.Floor(position) + second := math.Ceil(position) + low, high := sorted[int(first)], sorted[int(second)] + if first == second { + return low + } + return low + (position-first)*(high-low) +} diff --git a/module/services/accounts/code/pkg/auditstore/auditeval/eval.go b/module/services/accounts/code/pkg/auditstore/auditeval/eval.go new file mode 100644 index 000000000..ef765572d --- /dev/null +++ b/module/services/accounts/code/pkg/auditstore/auditeval/eval.go @@ -0,0 +1,303 @@ +// Package auditeval evaluates an audit read in the service, over the events a +// store of record returned: the predicates of AuditQuery, the scope, newest- +// first paging, export, and AggregateAuditLog's aggregation — each with the +// semantics the Postgres reads give it, so a store that cannot run them itself +// answers exactly as Postgres does. +// +// A store pushes down what it can (scope, time window, event type) to read +// less; the evaluation here is still the authority on every predicate, +// including the scope, so a store that returned too much still answers +// correctly. Every event is counted once by event id: a store of record may +// hold an event more than once after a redelivery (ADR 0009). +package auditeval + +import ( + "encoding/json" + "errors" + "fmt" + "time" + + "accounts/pkg/business" + + "github.com/google/uuid" +) + +// Event is one stored event as a store returned it: the envelope, and the +// canonical details when the store had them. A content-class event's details +// are kept for the content window only; past it the event has no details, and +// every payload predicate, dimension and metric reads it as having no payload. +type Event struct { + // Entry is the envelope. Its Payload is ignored: Details is the payload. + Entry business.AuditEntry + Details string + HasDetails bool + // DetailsSHA256 is the hash the event's details carry wherever they are + // kept; a store joins details to the event by it as well as by id. + DetailsSHA256 string + + decoded map[string]any + decodeErr error + decodedSet bool +} + +// payload is the details decoded with json.Number, or nil when there are none. +func (e *Event) payload() (map[string]any, error) { + if !e.HasDetails { + return nil, nil + } + if !e.decodedSet { + e.decoded, e.decodeErr = decodeDetails(e.Details) + e.decodedSet = true + } + return e.decoded, e.decodeErr +} + +// Result is the entry a read returns for the event: the envelope with its +// payload decoded exactly as the Postgres reads decode a stored payload +// (encoding/json into map[string]any). An event without details has no payload. +func (e *Event) Result() business.AuditEntry { + entry := e.Entry + entry.Payload = nil + if e.HasDetails { + var payload map[string]any + if json.Unmarshal([]byte(e.Details), &payload) == nil { + entry.Payload = payload + } + } + return entry +} + +// newer reports whether a sorts before b in the activity list: by occurrence, +// then by event id, both descending. +func newer(a, b business.AuditEntry) bool { + if !a.CreatedAt.Equal(b.CreatedAt) { + return a.CreatedAt.After(b.CreatedAt) + } + return a.ID > b.ID +} + +// Matcher decides whether an event is one a read returns. +type Matcher struct { + scope business.AuditReadScope + actorID string + eventType string + eventTypes map[string]bool + category string + namespace string + types business.AuditEventTypeIndex + resource string + resourceID string + clientID string + payload any + from, to *time.Time +} + +// NewMatcher compiles a read. It refuses a read without a scope, a query that +// reaches past its scope, a category or namespace filter without the +// event-type index, and an organization or actor id that is not a uuid — +// which Postgres refuses too, as invalid input for a uuid column. +func NewMatcher(read business.AuditRead) (*Matcher, error) { + if err := read.Validate(); err != nil { + return nil, err + } + q := read.Query + m := &Matcher{ + scope: read.Scope, + eventType: q.EventType, + category: q.Category, + namespace: q.Namespace, + types: read.Types, + resource: q.Resource, + resourceID: q.ResourceID, + clientID: q.ClientID, + } + if org := read.Scope.OrgID(); org != "" { + if _, err := uuid.Parse(org); err != nil { + return nil, fmt.Errorf("audit read: organization %q is not a uuid", org) + } + } + if q.ActorID != "" { + actor, err := uuid.Parse(q.ActorID) + if err != nil { + return nil, fmt.Errorf("audit read: actor %q is not a uuid", q.ActorID) + } + m.actorID = actor.String() + } + if len(q.EventTypes) > 0 { + m.eventTypes = make(map[string]bool, len(q.EventTypes)) + for _, eventType := range q.EventTypes { + m.eventTypes[eventType] = true + } + } + if (q.Category != "" || q.Namespace != "") && read.Types == nil { + return nil, errors.New("audit read: a category or namespace filter needs the event-type index") + } + if len(q.PayloadContains) > 0 { + payload, err := normalizeJSON(q.PayloadContains) + if err != nil { + // A filter that cannot be bound is never "no filter": dropping it would + // widen the result past what the caller was cleared to read. + return nil, fmt.Errorf("audit: cannot bind payload filter: %w", err) + } + m.payload = payload + } + m.from = MicrosecondBound(q.From) + m.to = MicrosecondBound(q.To) + return m, nil +} + +// MicrosecondBound is a query's time bound at the precision Postgres compares +// it at: a bound parameter is sent in whole microseconds, truncated. +func MicrosecondBound(t *time.Time) *time.Time { + if t == nil { + return nil + } + bound := t.Truncate(time.Microsecond) + return &bound +} + +// Scope is the read's scope. +func (m *Matcher) Scope() business.AuditReadScope { return m.scope } + +// ActorID is the actor filter as stored: a canonical lowercase uuid, or "". +func (m *Matcher) ActorID() string { return m.actorID } + +// From and To are the read's inclusive time bounds, at microsecond precision. +func (m *Matcher) From() *time.Time { return m.from } + +// To is the read's inclusive upper bound, or nil. +func (m *Matcher) To() *time.Time { return m.to } + +// EventTypes is the set of event types the read can match, and whether the +// read restricts the type at all — the scalar filter, the set filter and the +// category and namespace filters intersected. A store pushes it down. +func (m *Matcher) EventTypes() ([]string, bool) { + var candidates map[string]bool + restrict := func(allowed func(string) bool, from map[string]bool) { + next := map[string]bool{} + for name := range from { + if allowed(name) { + next[name] = true + } + } + candidates = next + } + switch { + case m.eventType != "": + candidates = map[string]bool{m.eventType: true} + case m.eventTypes != nil: + candidates = map[string]bool{} + for name := range m.eventTypes { + candidates[name] = true + } + case m.category != "" || m.namespace != "": + candidates = map[string]bool{} + for name := range m.types { + candidates[name] = true + } + default: + return nil, false + } + if m.eventTypes != nil { + restrict(func(name string) bool { return m.eventTypes[name] }, candidates) + } + if m.category != "" { + restrict(func(name string) bool { facts, ok := m.types[name]; return ok && facts.Category == m.category }, candidates) + } + if m.namespace != "" { + restrict(func(name string) bool { facts, ok := m.types[name]; return ok && facts.Namespace == m.namespace }, candidates) + } + out := make([]string, 0, len(candidates)) + for name := range candidates { + out = append(out, name) + } + sortStrings(out) + return out, true +} + +// NeedsPayload reports whether the read filters on the payload, so a store +// must join every candidate's details before Match can decide. +func (m *Matcher) NeedsPayload() bool { return m.payload != nil } + +// InScope reports whether an event belongs to the read's scope: its +// organization's, or for the platform scope any event at all. +func (m *Matcher) InScope(entry business.AuditEntry) bool { + return m.scope.Platform() || entry.OrgID == m.scope.OrgID() +} + +// Match reports whether an event is in the read's scope and matches every +// predicate of its query. +func (m *Matcher) Match(event *Event) (bool, error) { + entry := event.Entry + if !m.InScope(entry) { + return false, nil + } + if m.actorID != "" && entry.ActorID != m.actorID { + return false, nil + } + eventType := string(entry.EventType) + if m.eventType != "" && eventType != m.eventType { + return false, nil + } + if m.eventTypes != nil && !m.eventTypes[eventType] { + return false, nil + } + if m.category != "" || m.namespace != "" { + facts, ok := m.types[eventType] + if !ok || (m.category != "" && facts.Category != m.category) || (m.namespace != "" && facts.Namespace != m.namespace) { + return false, nil + } + } + if m.resource != "" && entry.Resource != m.resource { + return false, nil + } + if m.resourceID != "" && entry.ResourceID != m.resourceID { + return false, nil + } + if m.clientID != "" && entry.ClientID != m.clientID { + return false, nil + } + if m.from != nil && entry.CreatedAt.Before(*m.from) { + return false, nil + } + if m.to != nil && entry.CreatedAt.After(*m.to) { + return false, nil + } + if m.payload != nil { + payload, err := event.payload() + if err != nil { + return false, fmt.Errorf("audit read: details of event %s: %w", entry.ID, err) + } + if payload == nil || !jsonbContains(payload, m.payload) { + return false, nil + } + } + return true, nil +} + +// Dedupe remembers which event ids a read has already counted. +type Dedupe struct { + uuids map[uuid.UUID]struct{} + other map[string]struct{} +} + +// NewDedupe is an empty set. +func NewDedupe() *Dedupe { + return &Dedupe{uuids: map[uuid.UUID]struct{}{}, other: map[string]struct{}{}} +} + +// First reports whether id is seen for the first time, and remembers it. +func (d *Dedupe) First(id string) bool { + if parsed, err := uuid.Parse(id); err == nil && parsed.String() == id { + if _, seen := d.uuids[parsed]; seen { + return false + } + d.uuids[parsed] = struct{}{} + return true + } + if _, seen := d.other[id]; seen { + return false + } + d.other[id] = struct{}{} + return true +} diff --git a/module/services/accounts/code/pkg/auditstore/auditeval/eval_test.go b/module/services/accounts/code/pkg/auditstore/auditeval/eval_test.go new file mode 100644 index 000000000..ca6e7e6c9 --- /dev/null +++ b/module/services/accounts/code/pkg/auditstore/auditeval/eval_test.go @@ -0,0 +1,174 @@ +package auditeval + +import ( + "fmt" + "testing" + "time" + + "accounts/pkg/business" + + "github.com/stretchr/testify/require" +) + +// The semantics the Postgres reads give each predicate, dimension and metric, +// as Postgres documents them. The parity test in pkg/business holds the whole +// evaluation to a real Postgres over one fixture; these pin the edges a +// fixture does not reach. + +func decoded(t *testing.T, details string) map[string]any { + t.Helper() + payload, err := decodeDetails(details) + require.NoError(t, err) + return payload +} + +func TestJSONBContainment(t *testing.T) { + payload := decoded(t, `{"a":1,"b":"x","c":[1,"y",{"k":true}],"d":{"e":null,"f":[2,3]},"n":1.50}`) + for filter, want := range map[string]bool{ + `{}`: true, + `{"a":1}`: true, + `{"a":1.0}`: true, // numbers compare numerically + `{"n":1.5}`: true, + `{"a":"1"}`: false, // a string never equals a number + `{"b":"x"}`: true, + `{"b":"X"}`: false, + `{"c":[1]}`: true, + `{"c":["y",1]}`: true, // order and multiplicity do not matter + `{"c":[{"k":true}]}`: true, + `{"c":[{}]}`: true, + `{"c":[2]}`: false, + `{"c":1}`: false, // an array contains a scalar only at the top level + `{"d":{"e":null}}`: true, + `{"d":{"f":[3]}}`: true, + `{"d":{}}`: true, + `{"d":[]}`: false, // an object never contains an array + `{"missing":null}`: false, + `{"a":1,"b":"y"}`: false, + `{"d":{"e":null,"g":1}}`: false, + } { + rhs, err := normalizeJSON(decoded(t, filter)) + require.NoError(t, err) + require.Equal(t, want, jsonbContains(payload, rhs), filter) + } +} + +func TestJSONBText(t *testing.T) { + payload := decoded(t, `{"s":"plain \"q\"","n":1.50,"t":true,"z":null,"o":{"bb":[1,"x"],"a":{"c":"\né<"}}}`) + for key, want := range map[string]string{ + "s": `plain "q"`, + "n": "1.50", + "t": "true", + // jsonb orders keys by length, then bytes, and prints ", " and ": ". + "o": `{"a": {"c": "\n` + "é" + `<"}, "bb": [1, "x"]}`, + } { + got, ok := jsonbText(payload, key) + require.True(t, ok, key) + require.Equal(t, want, got, key) + } + _, ok := jsonbText(payload, "z") + require.False(t, ok, "JSON null is SQL NULL") + _, ok = jsonbText(payload, "missing") + require.False(t, ok) + _, ok = jsonbText(nil, "s") + require.False(t, ok, "no payload reads as NULL") +} + +func TestJSONBNumeric(t *testing.T) { + payload := decoded(t, `{"n":12.5,"i":-3,"s":"7.25","neg":"-1","exp":"1e3","word":"fast","lead":" 1","b":true,"o":{}}`) + for key, want := range map[string]float64{"n": 12.5, "i": -3, "s": 7.25, "neg": -1} { + got, ok := jsonbNumeric(payload, key) + require.True(t, ok, key) + require.Equal(t, want, got, key) + } + for _, key := range []string{"exp", "word", "lead", "b", "o", "missing"} { + _, ok := jsonbNumeric(payload, key) + require.False(t, ok, key) + } +} + +func TestTimeBucket(t *testing.T) { + at := time.Date(2026, 10, 3, 23, 30, 0, 0, time.FixedZone("ahead", 3*3600)) // 20:30 UTC, a Saturday + require.Equal(t, "2026-10-03T00:00:00+00", TimeBucket(at, "")) + require.Equal(t, "2026-10-03T00:00:00+00", TimeBucket(at, "day")) + require.Equal(t, "2026-09-28T00:00:00+00", TimeBucket(at, "week"), "ISO weeks start on Monday") + require.Equal(t, "2026-10-01T00:00:00+00", TimeBucket(at, "month")) + sunday := time.Date(2026, 10, 4, 1, 0, 0, 0, time.UTC) + require.Equal(t, "2026-09-28T00:00:00+00", TimeBucket(sunday, "week")) + monday := time.Date(2026, 10, 5, 0, 0, 0, 0, time.UTC) + require.Equal(t, "2026-10-05T00:00:00+00", TimeBucket(monday, "week")) +} + +func TestPercentileCont(t *testing.T) { + require.Equal(t, 5.0, percentileCont([]float64{5}, 0.9)) + require.Equal(t, 2.5, percentileCont([]float64{4, 1, 3, 2}, 0.5)) + require.Equal(t, 4.0, percentileCont([]float64{4, 1, 3, 2}, 1)) + require.Equal(t, 5.0, percentileCont([]float64{20, 0, 10}, 0.25)) + require.Equal(t, 0.5+1.75/512, percentileCont([]float64{100, 0.5, 2.25}, 1.0/1024)) +} + +func entryAt(id int, at time.Time) business.AuditEntry { + return business.AuditEntry{ID: fmt.Sprintf("00000000-0000-4000-8000-%012d", id), CreatedAt: at} +} + +func TestPageKeepsTheNewestOnceEachInAnyOfferOrder(t *testing.T) { + base := time.Date(2026, 10, 3, 0, 0, 0, 0, time.UTC) + var events []*Event + for i := 1; i <= 6; i++ { + events = append(events, &Event{Entry: entryAt(i, base.Add(time.Duration(i)*time.Minute))}) + } + tie := &Event{Entry: entryAt(9, base.Add(3*time.Minute))} // ties event 3; the larger id is newer + duplicate := &Event{Entry: events[5].Entry} // event 6 again + page, err := NewPage(business.AuditQuery{PageSize: 3}) + require.NoError(t, err) + for _, i := range []int{2, 5, 0, 3, 1, 4} { + page.Offer(events[i]) + } + page.Offer(tie) + page.Offer(duplicate) + got, next := page.Result() + require.Equal(t, []string{events[5].Entry.ID, events[4].Entry.ID, events[3].Entry.ID}, []string{got[0].Entry.ID, got[1].Entry.ID, got[2].Entry.ID}) + require.NotEmpty(t, next) + + page, err = NewPage(business.AuditQuery{PageSize: 3, PageToken: next}) + require.NoError(t, err) + for _, event := range append(events, tie, duplicate) { + page.Offer(event) + } + got, next = page.Result() + require.Equal(t, []string{tie.Entry.ID, events[2].Entry.ID, events[1].Entry.ID}, []string{got[0].Entry.ID, got[1].Entry.ID, got[2].Entry.ID}) + require.NotEmpty(t, next) + + _, err = NewPage(business.AuditQuery{PageToken: "not a token!"}) + require.ErrorContains(t, err, "invalid page token") +} + +func TestDedupe(t *testing.T) { + seen := NewDedupe() + require.True(t, seen.First("aaaaaaaa-0000-4000-8000-000000000001")) + require.False(t, seen.First("aaaaaaaa-0000-4000-8000-000000000001")) + require.True(t, seen.First("AAAAAAAA-0000-4000-8000-000000000001"), "ids compare as stored, exactly") + require.True(t, seen.First("not-a-uuid")) + require.False(t, seen.First("not-a-uuid")) +} + +func TestMatcherRefusals(t *testing.T) { + org := "aaaaaaaa-0000-4000-8000-000000000001" + _, err := NewMatcher(business.AuditRead{}) + require.ErrorIs(t, err, business.ErrAuditReadUnscoped) + _, err = NewMatcher(business.AuditRead{Scope: business.OrganizationAuditScope("org-1"), Query: business.AuditQuery{OrgID: "org-1"}}) + require.ErrorContains(t, err, "not a uuid") + _, err = NewMatcher(business.AuditRead{Scope: business.OrganizationAuditScope(org), Query: business.AuditQuery{OrgID: org, ActorID: "someone"}}) + require.ErrorContains(t, err, "not a uuid") + _, err = NewMatcher(business.AuditRead{Scope: business.OrganizationAuditScope(org), Query: business.AuditQuery{OrgID: org, Category: "access"}}) + require.ErrorContains(t, err, "event-type index") + _, err = NewMatcher(business.AuditRead{Scope: business.OrganizationAuditScope(org), Query: business.AuditQuery{OrgID: org, CollectionID: "c"}}) + require.ErrorContains(t, err, "uncompiled collection filter") + _, err = NewAggregator(business.AuditAggregationSpec{GroupBy: []string{"category"}}, nil) + require.ErrorContains(t, err, "event-type index") + + m, err := NewMatcher(business.AuditRead{Scope: business.OrganizationAuditScope(org), Query: business.AuditQuery{ + OrgID: org, ActorID: "BBBBBBBB-0000-4000-8000-000000000002", + }}) + require.NoError(t, err) + require.Equal(t, "bbbbbbbb-0000-4000-8000-000000000002", m.ActorID(), "an actor id compares as the uuid it names") +} diff --git a/module/services/accounts/code/pkg/auditstore/auditeval/jsonb.go b/module/services/accounts/code/pkg/auditstore/auditeval/jsonb.go new file mode 100644 index 000000000..256a17f05 --- /dev/null +++ b/module/services/accounts/code/pkg/auditstore/auditeval/jsonb.go @@ -0,0 +1,259 @@ +package auditeval + +import ( + "bytes" + "encoding/json" + "math/big" + "regexp" + "sort" + "strconv" + "strings" +) + +// The payload semantics of the Postgres reads, reproduced over a decoded +// payload: jsonb containment (@>), the text of a key (->>), and the numeric +// reading of a key that AggregateAuditLog's metrics use. Payloads are decoded +// with json.Number (decodeDetails), so a number keeps the text Postgres +// printed for it, and a store's canonical details — which were serialized +// from that text — read exactly as the jsonb they came from. + +// decodeDetails decodes a canonical details string keeping each number's text. +func decodeDetails(details string) (map[string]any, error) { + decoder := json.NewDecoder(strings.NewReader(details)) + decoder.UseNumber() + var payload map[string]any + if err := decoder.Decode(&payload); err != nil { + return nil, err + } + return payload, nil +} + +// normalizeJSON is value as Postgres receives it when it binds the value as +// jsonb: marshalled, then read back with json.Number. +func normalizeJSON(value any) (any, error) { + raw, err := json.Marshal(value) + if err != nil { + return nil, err + } + decoder := json.NewDecoder(bytes.NewReader(raw)) + decoder.UseNumber() + var out any + if err := decoder.Decode(&out); err != nil { + return nil, err + } + return out, nil +} + +// jsonbContains is lhs @> rhs for two jsonb values: every key of an object on +// the right is in the object on the left with a value that contains it; every +// element of an array on the right is contained by some element of the array +// on the left; scalars are equal (numbers numerically). Containers of +// different kinds never contain one another. +func jsonbContains(lhs, rhs any) bool { + switch right := rhs.(type) { + case map[string]any: + left, ok := lhs.(map[string]any) + if !ok { + return false + } + for key, rightValue := range right { + leftValue, ok := left[key] + if !ok { + return false + } + if isContainer(rightValue) { + if !jsonbContains(leftValue, rightValue) { + return false + } + continue + } + if !jsonbScalarEqual(leftValue, rightValue) { + return false + } + } + return true + case []any: + left, ok := lhs.([]any) + if !ok { + return false + } + for _, rightElement := range right { + found := false + for _, leftElement := range left { + if isContainer(rightElement) { + found = jsonbContains(leftElement, rightElement) + } else { + found = jsonbScalarEqual(leftElement, rightElement) + } + if found { + break + } + } + if !found { + return false + } + } + return true + default: + return jsonbScalarEqual(lhs, rhs) + } +} + +func isContainer(value any) bool { + switch value.(type) { + case map[string]any, []any: + return true + default: + return false + } +} + +// jsonbScalarEqual is jsonb scalar equality: the same kind, and equal — two +// numbers when they are numerically equal, whatever their text. +func jsonbScalarEqual(lhs, rhs any) bool { + switch right := rhs.(type) { + case nil: + return lhs == nil + case string: + left, ok := lhs.(string) + return ok && left == right + case bool: + left, ok := lhs.(bool) + return ok && left == right + case json.Number: + left, ok := lhs.(json.Number) + if !ok { + return false + } + l, lok := new(big.Rat).SetString(string(left)) + r, rok := new(big.Rat).SetString(string(right)) + if !lok || !rok { + return string(left) == string(right) + } + return l.Cmp(r) == 0 + default: + return false + } +} + +// jsonbText is payload ->> key: a string's own text, a number's text, true or +// false, and for an object or array the jsonb text of it. A missing key or a +// JSON null is SQL NULL (ok false). +func jsonbText(payload map[string]any, key string) (string, bool) { + value, ok := payload[key] + if !ok || value == nil { + return "", false + } + switch v := value.(type) { + case string: + return v, true + default: + var buf strings.Builder + writeJSONBText(&buf, v) + return buf.String(), true + } +} + +// writeJSONBText renders a value the way jsonb prints it: object keys ordered +// by length and then bytewise, ", " and ": " separators, and strings escaped +// only where JSON requires it. +func writeJSONBText(buf *strings.Builder, value any) { + switch v := value.(type) { + case nil: + buf.WriteString("null") + case bool: + buf.WriteString(strconv.FormatBool(v)) + case json.Number: + buf.WriteString(string(v)) + case string: + writeJSONBString(buf, v) + case []any: + buf.WriteByte('[') + for i, element := range v { + if i > 0 { + buf.WriteString(", ") + } + writeJSONBText(buf, element) + } + buf.WriteByte(']') + case map[string]any: + keys := make([]string, 0, len(v)) + for key := range v { + keys = append(keys, key) + } + sort.Slice(keys, func(i, j int) bool { + if len(keys[i]) != len(keys[j]) { + return len(keys[i]) < len(keys[j]) + } + return keys[i] < keys[j] + }) + buf.WriteByte('{') + for i, key := range keys { + if i > 0 { + buf.WriteString(", ") + } + writeJSONBString(buf, key) + buf.WriteString(": ") + writeJSONBText(buf, v[key]) + } + buf.WriteByte('}') + default: + // Decoded payloads hold nothing else; render defensively as JSON. + raw, _ := json.Marshal(v) + buf.Write(raw) + } +} + +// writeJSONBString is Postgres's escape_json. +func writeJSONBString(buf *strings.Builder, s string) { + buf.WriteByte('"') + for _, r := range s { + switch r { + case '\b': + buf.WriteString(`\b`) + case '\f': + buf.WriteString(`\f`) + case '\n': + buf.WriteString(`\n`) + case '\r': + buf.WriteString(`\r`) + case '\t': + buf.WriteString(`\t`) + case '"': + buf.WriteString(`\"`) + case '\\': + buf.WriteString(`\\`) + default: + if r < 0x20 { + buf.WriteString(`\u00`) + buf.WriteByte("0123456789abcdef"[r>>4]) + buf.WriteByte("0123456789abcdef"[r&0xf]) + continue + } + buf.WriteRune(r) + } + } + buf.WriteByte('"') +} + +// plainNumber is the text a JSON string must have for the numeric reading to +// take it: an optionally negative decimal, as the Postgres expression's regex. +var plainNumber = regexp.MustCompile(`^-?[0-9]+(\.[0-9]+)?$`) + +// jsonbNumeric is the numeric reading of payload[key]: a JSON number, or a JSON +// string holding a plain decimal, as a double; anything else is NULL. +func jsonbNumeric(payload map[string]any, key string) (float64, bool) { + switch v := payload[key].(type) { + case json.Number: + f, err := strconv.ParseFloat(string(v), 64) + return f, err == nil + case string: + if !plainNumber.MatchString(v) { + return 0, false + } + f, err := strconv.ParseFloat(v, 64) + return f, err == nil + default: + return 0, false + } +} diff --git a/module/services/accounts/code/pkg/auditstore/auditeval/page.go b/module/services/accounts/code/pkg/auditstore/auditeval/page.go new file mode 100644 index 000000000..3d6e9ae35 --- /dev/null +++ b/module/services/accounts/code/pkg/auditstore/auditeval/page.go @@ -0,0 +1,129 @@ +package auditeval + +import ( + "container/heap" + "fmt" + "sort" + "time" + + "accounts/pkg/business" +) + +// DefaultPageSize is the activity list's page size when the query names none, +// as it is in Postgres. +const DefaultPageSize = 50 + +// Page collects one page of the activity list from events offered in any +// order: the newest PageSize events — by occurrence, then event id, both +// descending — strictly after the query's page token, each once. It keeps one +// more than a page, which is how it knows a further page exists, and never +// more, so a store can offer it every event of a long history in bounded +// memory. +type Page struct { + size int + after *cursor + kept eventHeap + keptIDs map[string]bool +} + +type cursor struct { + at time.Time + id string +} + +// NewPage starts a page of q: its size (DefaultPageSize when zero) and the +// position its page token names. +func NewPage(q business.AuditQuery) (*Page, error) { + size := int(q.PageSize) + if size == 0 { + size = DefaultPageSize + } + if size < 0 { + return nil, fmt.Errorf("invalid page size %d", q.PageSize) + } + p := &Page{size: size, keptIDs: map[string]bool{}} + if q.PageToken != "" { + at, id, err := business.DecodeAuditPageToken(q.PageToken) + if err != nil { + return nil, fmt.Errorf("invalid page token: %w", err) + } + p.after = &cursor{at: at, id: id} + } + return p, nil +} + +// Before is the occurrence the page token names, if any: no event at or after +// it beyond the token's own tie-break can be on this page. +func (p *Page) Before() *time.Time { + if p.after == nil { + return nil + } + at := p.after.at + return &at +} + +// Offer considers an event that matched the read. +func (p *Page) Offer(event *Event) { + entry := event.Entry + if p.after != nil && !newer(business.AuditEntry{CreatedAt: p.after.at, ID: p.after.id}, entry) { + return + } + if p.keptIDs[entry.ID] { + return + } + if len(p.kept) == p.size+1 { + if !newer(entry, p.kept[0].Entry) { + return + } + evicted := heap.Pop(&p.kept).(*Event) + delete(p.keptIDs, evicted.Entry.ID) + } + heap.Push(&p.kept, event) + p.keptIDs[entry.ID] = true +} + +// Full reports whether the page holds a whole page and the proof of another: +// every event older than all of them can no longer change it. +func (p *Page) Full() bool { return len(p.kept) == p.size+1 } + +// Events are the kept events, newest first: the page and, when Full, the one +// past it. +func (p *Page) Events() []*Event { + out := append([]*Event(nil), p.kept...) + sort.Slice(out, func(i, j int) bool { return newer(out[i].Entry, out[j].Entry) }) + return out +} + +// Result is the page and its next token, empty after the last page. +func (p *Page) Result() ([]*Event, string) { + events := p.Events() + if len(events) <= p.size { + return events, "" + } + events = events[:p.size] + last := events[len(events)-1].Entry + return events, business.EncodeAuditPageToken(last.CreatedAt, last.ID) +} + +// eventHeap is a min-heap by list order: its root is the oldest kept event, +// the first to go when a newer one arrives. +type eventHeap []*Event + +func (h eventHeap) Len() int { return len(h) } +func (h eventHeap) Less(i, j int) bool { return newer(h[j].Entry, h[i].Entry) } +func (h eventHeap) Swap(i, j int) { h[i], h[j] = h[j], h[i] } +func (h *eventHeap) Push(x any) { *h = append(*h, x.(*Event)) } +func (h *eventHeap) Pop() any { + old := *h + n := len(old) + item := old[n-1] + *h = old[:n-1] + return item +} + +// SortNewestFirst orders events as the activity list does. +func SortNewestFirst(events []*Event) { + sort.Slice(events, func(i, j int) bool { return newer(events[i].Entry, events[j].Entry) }) +} + +func sortStrings(values []string) { sort.Strings(values) } diff --git a/module/services/accounts/code/pkg/auditstore/bigqueryfake/bigqueryfake.go b/module/services/accounts/code/pkg/auditstore/bigqueryfake/bigqueryfake.go new file mode 100644 index 000000000..5d548b9b9 --- /dev/null +++ b/module/services/accounts/code/pkg/auditstore/bigqueryfake/bigqueryfake.go @@ -0,0 +1,690 @@ +// Package bigqueryfake is an in-memory BigQuery Storage Read API for tests: +// tables of rows, read sessions that evaluate their row restriction and +// project their selected fields, and Arrow record batches split across +// streams, as the real service returns them. +// +// It understands the row restrictions the audit reader writes and refuses any +// other: a conjunction (AND) of `field = "literal"`, `field IN ("a", "b")` and +// `field op CAST("timestamp" AS TIMESTAMP)` for op one of >=, >, <=, <. A +// restriction it cannot parse fails the session, so a reader change that +// writes something BigQuery might read differently fails its tests loudly +// rather than reading everything. +// +// Rows are the values a streaming insert carries (bigquery.Value maps, as +// bigquerystore.EventRow builds them), so a test fills the fake with exactly +// what the writer sends. +package bigqueryfake + +import ( + "bytes" + "context" + "errors" + "fmt" + "io" + "slices" + "strings" + "sync" + "time" + + "cloud.google.com/go/bigquery" + "cloud.google.com/go/bigquery/storage/apiv1/storagepb" + "github.com/apache/arrow/go/v15/arrow" + "github.com/apache/arrow/go/v15/arrow/array" + "github.com/apache/arrow/go/v15/arrow/ipc" + "github.com/apache/arrow/go/v15/arrow/memory" + "github.com/googleapis/gax-go/v2" + "google.golang.org/grpc" + "google.golang.org/grpc/codes" + "google.golang.org/grpc/status" +) + +// Server is the fake. Its exported fields tune it before the first read. +type Server struct { + // StreamsPerSession caps how many streams a session's rows are spread over + // (default 3), further capped by the request's max stream count. + StreamsPerSession int + // RowsPerBatch is the rows of one record batch (default 2), so a stream + // carries several. + RowsPerBatch int + // IgnoreRestrictions returns every row of the table, as a store that + // over-returns would: what the reader answers must not change. + IgnoreRestrictions bool + // FailReadRows, when set, is asked before every response of a stream; an + // error it returns ends the stream with that error. + FailReadRows func(stream string, offset int64) error + + mu sync.Mutex + tables map[string]*table + streams map[string][]*storagepb.ReadRowsResponse + sessions []*storagepb.CreateReadSessionRequest + next int +} + +type table struct { + schema bigquery.Schema + rows []map[string]bigquery.Value +} + +// New is an empty fake. +func New() *Server { + return &Server{tables: map[string]*table{}, streams: map[string][]*storagepb.ReadRowsResponse{}} +} + +// TablePath names a table as a read session does. +func TablePath(project, dataset, name string) string { + return fmt.Sprintf("projects/%s/datasets/%s/tables/%s", project, dataset, name) +} + +// CreateTable adds an empty table. +func (s *Server) CreateTable(path string, schema bigquery.Schema) { + s.mu.Lock() + defer s.mu.Unlock() + s.tables[path] = &table{schema: schema} +} + +// Insert appends rows to a table, checked against its schema. +func (s *Server) Insert(path string, rows ...map[string]bigquery.Value) error { + s.mu.Lock() + defer s.mu.Unlock() + t, ok := s.tables[path] + if !ok { + return fmt.Errorf("no table %s", path) + } + for _, row := range rows { + typed := make(map[string]bigquery.Value, len(row)) + for _, field := range t.schema { + value, err := typedValue(field, row[field.Name]) + if err != nil { + return err + } + typed[field.Name] = value + } + for name := range row { + if !slices.ContainsFunc(t.schema, func(f *bigquery.FieldSchema) bool { return f.Name == name }) { + return fmt.Errorf("table %s has no column %s", path, name) + } + } + t.rows = append(t.rows, typed) + } + return nil +} + +// Rows is a copy of a table's rows. +func (s *Server) Rows(path string) []map[string]bigquery.Value { + s.mu.Lock() + defer s.mu.Unlock() + t, ok := s.tables[path] + if !ok { + return nil + } + return slices.Clone(t.rows) +} + +// Sessions is every read session request made so far, in order. +func (s *Server) Sessions() []*storagepb.CreateReadSessionRequest { + s.mu.Lock() + defer s.mu.Unlock() + return slices.Clone(s.sessions) +} + +// typedValue reads an insert value as the column's type: STRING as string, +// INTEGER as int64, BOOLEAN as bool, TIMESTAMP as a UTC time; nil is NULL. +func typedValue(field *bigquery.FieldSchema, value bigquery.Value) (bigquery.Value, error) { + if value == nil { + if field.Required { + return nil, fmt.Errorf("column %s is required", field.Name) + } + return nil, nil + } + switch field.Type { + case bigquery.StringFieldType: + if v, ok := value.(string); ok { + return v, nil + } + case bigquery.IntegerFieldType: + switch v := value.(type) { + case int: + return int64(v), nil + case int64: + return v, nil + } + case bigquery.BooleanFieldType: + if v, ok := value.(bool); ok { + return v, nil + } + case bigquery.TimestampFieldType: + switch v := value.(type) { + case time.Time: + return v.UTC().Truncate(time.Microsecond), nil + case string: + at, err := time.Parse(time.RFC3339Nano, v) + if err != nil { + return nil, fmt.Errorf("column %s: %w", field.Name, err) + } + return at.UTC(), nil + } + } + return nil, fmt.Errorf("column %s (%s) cannot hold %T", field.Name, field.Type, value) +} + +// CreateReadSession evaluates the restriction over the table and spreads the +// selected columns of the admitted rows over the session's streams. +func (s *Server) CreateReadSession(_ context.Context, req *storagepb.CreateReadSessionRequest, _ ...gax.CallOption) (*storagepb.ReadSession, error) { + s.mu.Lock() + defer s.mu.Unlock() + s.sessions = append(s.sessions, req) + requested := req.GetReadSession() + t, ok := s.tables[requested.GetTable()] + if !ok { + return nil, status.Errorf(codes.NotFound, "table %s not found", requested.GetTable()) + } + if !strings.HasPrefix(requested.GetTable(), req.GetParent()+"/") { + return nil, status.Errorf(codes.InvalidArgument, "parent %s does not own %s", req.GetParent(), requested.GetTable()) + } + if requested.GetDataFormat() != storagepb.DataFormat_ARROW { + return nil, status.Error(codes.InvalidArgument, "the fake serves ARROW only") + } + options := requested.GetReadOptions() + fields, err := selected(t.schema, options.GetSelectedFields()) + if err != nil { + return nil, status.Error(codes.InvalidArgument, err.Error()) + } + where, err := parseRestriction(options.GetRowRestriction()) + if err != nil { + return nil, status.Errorf(codes.InvalidArgument, "row restriction %q: %v", options.GetRowRestriction(), err) + } + for _, term := range where { + if !slices.ContainsFunc(t.schema, func(f *bigquery.FieldSchema) bool { return f.Name == term.field }) { + return nil, status.Errorf(codes.InvalidArgument, "row restriction names unknown column %s", term.field) + } + } + var admitted []map[string]bigquery.Value + for _, row := range t.rows { + if s.IgnoreRestrictions || where.admits(row) { + admitted = append(admitted, row) + } + } + + schema := arrowSchema(fields) + serializedSchema, err := serializeSchema(schema) + if err != nil { + return nil, status.Error(codes.Internal, err.Error()) + } + s.next++ + name := fmt.Sprintf("projects/fake/locations/us/sessions/s%d", s.next) + session := &storagepb.ReadSession{ + Name: name, + Table: requested.GetTable(), + DataFormat: storagepb.DataFormat_ARROW, + Schema: &storagepb.ReadSession_ArrowSchema{ArrowSchema: &storagepb.ArrowSchema{SerializedSchema: serializedSchema}}, + } + if len(admitted) == 0 { + return session, nil + } + streams := s.StreamsPerSession + if streams <= 0 { + streams = 3 + } + if limit := int(req.GetMaxStreamCount()); limit > 0 && streams > limit { + streams = limit + } + streams = min(streams, len(admitted)) + perBatch := s.RowsPerBatch + if perBatch <= 0 { + perBatch = 2 + } + for i := 0; i < streams; i++ { + var share []map[string]bigquery.Value + for j := i; j < len(admitted); j += streams { + share = append(share, admitted[j]) + } + streamName := fmt.Sprintf("%s/streams/%d", name, i) + var responses []*storagepb.ReadRowsResponse + for start := 0; start < len(share); start += perBatch { + rows := share[start:min(start+perBatch, len(share))] + batch, err := serializeBatch(schema, len(serializedSchema), fields, rows) + if err != nil { + return nil, status.Error(codes.Internal, err.Error()) + } + responses = append(responses, &storagepb.ReadRowsResponse{ + RowCount: int64(len(rows)), + Rows: &storagepb.ReadRowsResponse_ArrowRecordBatch{ArrowRecordBatch: &storagepb.ArrowRecordBatch{SerializedRecordBatch: batch}}, + }) + } + s.streams[streamName] = responses + session.Streams = append(session.Streams, &storagepb.ReadStream{Name: streamName}) + } + return session, nil +} + +// ReadRows serves a stream from the row offset the request names, which must +// fall on a batch boundary, as every offset the reader resumes from does. +func (s *Server) ReadRows(ctx context.Context, req *storagepb.ReadRowsRequest, _ ...gax.CallOption) (storagepb.BigQueryRead_ReadRowsClient, error) { + s.mu.Lock() + responses, ok := s.streams[req.GetReadStream()] + fail := s.FailReadRows + s.mu.Unlock() + if !ok { + return nil, status.Errorf(codes.NotFound, "stream %s not found", req.GetReadStream()) + } + var skipped int64 + start := 0 + for start < len(responses) && skipped < req.GetOffset() { + skipped += responses[start].GetRowCount() + start++ + } + if skipped != req.GetOffset() { + return nil, status.Errorf(codes.OutOfRange, "offset %d is not a batch boundary", req.GetOffset()) + } + return &rowStream{ctx: ctx, stream: req.GetReadStream(), responses: responses[start:], offset: skipped, fail: fail}, nil +} + +type rowStream struct { + grpc.ClientStream + ctx context.Context + stream string + responses []*storagepb.ReadRowsResponse + offset int64 + fail func(string, int64) error +} + +func (r *rowStream) Recv() (*storagepb.ReadRowsResponse, error) { + if err := r.ctx.Err(); err != nil { + return nil, err + } + if len(r.responses) == 0 { + return nil, io.EOF + } + if r.fail != nil { + if err := r.fail(r.stream, r.offset); err != nil { + return nil, err + } + } + response := r.responses[0] + r.responses = r.responses[1:] + r.offset += response.GetRowCount() + return response, nil +} + +func (r *rowStream) Context() context.Context { return r.ctx } + +// selected is the selected fields in table order, as BigQuery orders them; +// none selected is every field. +func selected(schema bigquery.Schema, names []string) (bigquery.Schema, error) { + if len(names) == 0 { + return schema, nil + } + for _, name := range names { + if !slices.ContainsFunc(schema, func(f *bigquery.FieldSchema) bool { return f.Name == name }) { + return nil, fmt.Errorf("selected field %s is not in the table", name) + } + } + var out bigquery.Schema + for _, field := range schema { + if slices.Contains(names, field.Name) { + out = append(out, field) + } + } + return out, nil +} + +func arrowSchema(fields bigquery.Schema) *arrow.Schema { + out := make([]arrow.Field, len(fields)) + for i, field := range fields { + var dataType arrow.DataType + switch field.Type { + case bigquery.IntegerFieldType: + dataType = arrow.PrimitiveTypes.Int64 + case bigquery.BooleanFieldType: + dataType = arrow.FixedWidthTypes.Boolean + case bigquery.TimestampFieldType: + dataType = &arrow.TimestampType{Unit: arrow.Microsecond, TimeZone: "UTC"} + default: + dataType = arrow.BinaryTypes.String + } + out[i] = arrow.Field{Name: field.Name, Type: dataType, Nullable: !field.Required} + } + return arrow.NewSchema(out, nil) +} + +// serializeSchema is the IPC schema message alone, as a read session carries +// it. A stream writer emits the schema before its first batch; writing one +// batch twice tells the two apart, since both copies of the batch are the same +// length. +func serializeSchema(schema *arrow.Schema) ([]byte, error) { + record := array.NewRecord(schema, emptyColumns(schema), 0) + defer record.Release() + var buf bytes.Buffer + writer := ipc.NewWriter(&buf, ipc.WithSchema(schema)) + if err := writer.Write(record); err != nil { + return nil, err + } + once := buf.Len() + if err := writer.Write(record); err != nil { + return nil, err + } + batchLength := buf.Len() - once + return slices.Clone(buf.Bytes()[:once-batchLength]), nil +} + +func emptyColumns(schema *arrow.Schema) []arrow.Array { + columns := make([]arrow.Array, schema.NumFields()) + for i, field := range schema.Fields() { + builder := array.NewBuilder(memory.DefaultAllocator, field.Type) + columns[i] = builder.NewArray() + builder.Release() + } + return columns +} + +// serializeBatch is one IPC record batch message, without the schema that +// precedes it in a stream. +func serializeBatch(schema *arrow.Schema, schemaLength int, fields bigquery.Schema, rows []map[string]bigquery.Value) ([]byte, error) { + builder := array.NewRecordBuilder(memory.DefaultAllocator, schema) + defer builder.Release() + for _, row := range rows { + for i, field := range fields { + value := row[field.Name] + column := builder.Field(i) + if value == nil { + column.AppendNull() + continue + } + switch b := column.(type) { + case *array.StringBuilder: + b.Append(value.(string)) + case *array.Int64Builder: + b.Append(value.(int64)) + case *array.BooleanBuilder: + b.Append(value.(bool)) + case *array.TimestampBuilder: + b.Append(arrow.Timestamp(value.(time.Time).UnixMicro())) + default: + return nil, fmt.Errorf("no builder for column %s", field.Name) + } + } + } + record := builder.NewRecord() + defer record.Release() + var buf bytes.Buffer + writer := ipc.NewWriter(&buf, ipc.WithSchema(schema)) + if err := writer.Write(record); err != nil { + return nil, err + } + if buf.Len() < schemaLength { + return nil, errors.New("record batch shorter than its schema") + } + return slices.Clone(buf.Bytes()[schemaLength:]), nil +} + +// term is one comparison of a restriction. +type term struct { + field string + op string + values []any // string or time.Time +} + +type conjunction []term + +func (c conjunction) admits(row map[string]bigquery.Value) bool { + for _, t := range c { + if !t.admits(row[t.field]) { + return false + } + } + return true +} + +func (t term) admits(value bigquery.Value) bool { + if value == nil { + return false + } + switch t.op { + case "IN": + for _, candidate := range t.values { + if equal(value, candidate) { + return true + } + } + return false + case "=": + return equal(value, t.values[0]) + } + at, ok := value.(time.Time) + bound, boundOK := t.values[0].(time.Time) + if !ok || !boundOK { + return false + } + switch t.op { + case ">=": + return !at.Before(bound) + case ">": + return at.After(bound) + case "<=": + return !at.After(bound) + case "<": + return at.Before(bound) + } + return false +} + +func equal(value bigquery.Value, literal any) bool { + switch v := value.(type) { + case string: + s, ok := literal.(string) + return ok && v == s + case time.Time: + at, ok := literal.(time.Time) + return ok && v.Equal(at) + } + return false +} + +// parseRestriction parses the restriction grammar the package comment names. +func parseRestriction(text string) (conjunction, error) { + if strings.TrimSpace(text) == "" { + return nil, nil + } + tokens, err := tokenize(text) + if err != nil { + return nil, err + } + p := &parser{tokens: tokens} + var out conjunction + for { + t, err := p.term() + if err != nil { + return nil, err + } + out = append(out, t) + if p.done() { + return out, nil + } + if err := p.expect("AND"); err != nil { + return nil, err + } + } +} + +type token struct { + kind string // ident, op, string, punct + text string +} + +func tokenize(text string) ([]token, error) { + var tokens []token + for i := 0; i < len(text); { + c := text[i] + switch { + case c == ' ': + i++ + case c == '(' || c == ')' || c == ',': + tokens = append(tokens, token{kind: "punct", text: string(c)}) + i++ + case c == '=': + tokens = append(tokens, token{kind: "op", text: "="}) + i++ + case c == '<' || c == '>': + op := string(c) + if i+1 < len(text) && text[i+1] == '=' { + op += "=" + } + tokens = append(tokens, token{kind: "op", text: op}) + i += len(op) + case c == '"': + value, n, err := unquote(text[i:]) + if err != nil { + return nil, err + } + tokens = append(tokens, token{kind: "string", text: value}) + i += n + case c == '_' || c >= 'a' && c <= 'z' || c >= 'A' && c <= 'Z': + j := i + for j < len(text) && (text[j] == '_' || text[j] >= 'a' && text[j] <= 'z' || text[j] >= 'A' && text[j] <= 'Z' || text[j] >= '0' && text[j] <= '9') { + j++ + } + tokens = append(tokens, token{kind: "ident", text: text[i:j]}) + i = j + default: + return nil, fmt.Errorf("unexpected %q at %d", c, i) + } + } + return tokens, nil +} + +// unquote reads a double-quoted GoogleSQL string literal with the escapes the +// reader writes: \" \\ and \Uhhhhhhhh. +func unquote(text string) (string, int, error) { + var b strings.Builder + for i := 1; i < len(text); i++ { + switch c := text[i]; c { + case '"': + return b.String(), i + 1, nil + case '\\': + if i+1 >= len(text) { + return "", 0, errors.New("unterminated escape") + } + switch text[i+1] { + case '"', '\\': + b.WriteByte(text[i+1]) + i++ + case 'U': + if i+10 > len(text) { + return "", 0, errors.New("short \\U escape") + } + var r rune + if _, err := fmt.Sscanf(text[i+2:i+10], "%08X", &r); err != nil { + return "", 0, fmt.Errorf("bad \\U escape: %w", err) + } + b.WriteRune(r) + i += 9 + default: + return "", 0, fmt.Errorf("escape \\%c is not one the reader writes", text[i+1]) + } + default: + if c < 0x20 || c >= 0x7f { + return "", 0, fmt.Errorf("raw byte %#x in a literal", c) + } + b.WriteByte(c) + } + } + return "", 0, errors.New("unterminated string literal") +} + +type parser struct { + tokens []token + at int +} + +func (p *parser) done() bool { return p.at == len(p.tokens) } + +func (p *parser) peek() token { + if p.done() { + return token{} + } + return p.tokens[p.at] +} + +func (p *parser) take() token { + t := p.peek() + p.at++ + return t +} + +func (p *parser) expect(text string) error { + if t := p.take(); t.text != text || t.kind == "string" { + return fmt.Errorf("expected %s, got %q", text, t.text) + } + return nil +} + +func (p *parser) term() (term, error) { + field := p.take() + if field.kind != "ident" { + return term{}, fmt.Errorf("expected a column, got %q", field.text) + } + op := p.take() + switch { + case op.kind == "ident" && op.text == "IN": + if err := p.expect("("); err != nil { + return term{}, err + } + t := term{field: field.text, op: "IN"} + for { + value := p.take() + if value.kind != "string" { + return term{}, fmt.Errorf("IN takes string literals, got %q", value.text) + } + t.values = append(t.values, value.text) + next := p.take() + if next.text == ")" { + return t, nil + } + if next.text != "," { + return term{}, fmt.Errorf("expected , or ), got %q", next.text) + } + } + case op.kind == "op": + value, err := p.value() + if err != nil { + return term{}, err + } + if _, isTime := value.(time.Time); op.text != "=" && !isTime { + return term{}, fmt.Errorf("%s compares timestamps only", op.text) + } + return term{field: field.text, op: op.text, values: []any{value}}, nil + default: + return term{}, fmt.Errorf("expected an operator, got %q", op.text) + } +} + +func (p *parser) value() (any, error) { + t := p.take() + if t.kind == "string" { + return t.text, nil + } + if t.kind != "ident" || t.text != "CAST" { + return nil, fmt.Errorf("expected a literal, got %q", t.text) + } + if err := p.expect("("); err != nil { + return nil, err + } + literal := p.take() + if literal.kind != "string" { + return nil, fmt.Errorf("CAST takes a string literal, got %q", literal.text) + } + if err := p.expect("AS"); err != nil { + return nil, err + } + if err := p.expect("TIMESTAMP"); err != nil { + return nil, err + } + if err := p.expect(")"); err != nil { + return nil, err + } + at, err := time.Parse("2006-01-02 15:04:05.000000-07:00", literal.text) + if err != nil { + return nil, fmt.Errorf("timestamp literal: %w", err) + } + return at.UTC(), nil +} diff --git a/module/services/accounts/code/pkg/auditstore/bigquerystore/bigquerystore.go b/module/services/accounts/code/pkg/auditstore/bigquerystore/bigquerystore.go index 5116b6e36..61049e0c9 100644 --- a/module/services/accounts/code/pkg/auditstore/bigquerystore/bigquerystore.go +++ b/module/services/accounts/code/pkg/auditstore/bigquerystore/bigquerystore.go @@ -1,13 +1,15 @@ // Package bigquerystore is the BigQuery adapter of the audit store swap -// (ADR 0009): the write half of business.AuditStore. +// (ADR 0009): both halves of business.AuditStore. // -// The writer's authority is deliberately minimal — bigquery.datasets.get, -// bigquery.tables.create, bigquery.tables.get and bigquery.tables.updateData, -// and never bigquery.jobs.create — so it runs no load, query or DML job. Rows -// are appended with streaming inserts (tabledata.insertAll) and tables are -// created, never altered, at startup. Credentials are Application Default -// Credentials only: the caller builds the client, and on Kubernetes that is the -// pod's workload identity. +// The service's authority is deliberately minimal and never includes +// bigquery.jobs.create, so it runs no load, query or DML job. The write half +// holds bigquery.datasets.get, bigquery.tables.create, bigquery.tables.get and +// bigquery.tables.updateData: rows are appended with streaming inserts +// (tabledata.insertAll) and tables are created, never altered, at startup. The +// read half (reader.go) holds bigquery.tables.getData, bigquery.tables.list and +// the read-session permissions, and reads through the Storage Read API. +// Credentials are Application Default Credentials only: the caller builds the +// clients, and on Kubernetes that is the pod's workload identity. package bigquerystore import ( @@ -46,6 +48,9 @@ type Config struct { // ContentDetailRetention is the details table's partition expiration: how // long a content-class event's full details are kept. ContentDetailRetention time.Duration + // Reader is the read half. Without one the store is write-only, as the + // relay needs it, and every read is refused. + Reader *Reader } // rowInserter is the streaming-insert seam; *bigquery.Inserter satisfies it. @@ -60,6 +65,7 @@ type Store struct { retention time.Duration events rowInserter details rowInserter + reader *Reader } // New builds a store over client, which the caller constructs with @@ -87,9 +93,47 @@ func New(client *bigquery.Client, cfg Config) (*Store, error) { retention: cfg.ContentDetailRetention, events: events, details: details, + reader: cfg.Reader, }, nil } +// errWriteOnly refuses a read of a store built without a reader. +var errWriteOnly = errors.New("bigquery audit store: built without a reader") + +// ListAuditEvents implements business.AuditReader. +func (s *Store) ListAuditEvents(ctx context.Context, read business.AuditRead) ([]business.AuditEntry, string, error) { + if s.reader == nil { + return nil, "", errWriteOnly + } + return s.reader.ListAuditEvents(ctx, read) +} + +// AggregateAuditEvents implements business.AuditReader. +func (s *Store) AggregateAuditEvents(ctx context.Context, read business.AuditRead, spec business.AuditAggregationSpec) ([]business.AuditAggregateBucket, error) { + if s.reader == nil { + return nil, errWriteOnly + } + return s.reader.AggregateAuditEvents(ctx, read, spec) +} + +// ExportAuditEvents implements business.AuditReader. +func (s *Store) ExportAuditEvents(ctx context.Context, read business.AuditRead) ([]business.AuditEntry, error) { + if s.reader == nil { + return nil, errWriteOnly + } + return s.reader.ExportAuditEvents(ctx, read) +} + +// LatestSourceSyncEvents implements business.AuditSourceSyncReader. +func (s *Store) LatestSourceSyncEvents(ctx context.Context, scope business.AuditReadScope, sources []string) (map[string]business.AuditSourceSyncEvent, error) { + if s.reader == nil { + return nil, errWriteOnly + } + return s.reader.LatestSourceSyncEvents(ctx, scope, sources) +} + +var _ business.AuditStore = (*Store)(nil) + // EventsSchema is the events table: every event's envelope, the SHA-256 of its // canonical details, and the details themselves for security-class events. func EventsSchema() bigquery.Schema { @@ -228,17 +272,31 @@ func conforms(spec tableSpec, existing *bigquery.TableMetadata) error { return nil } -// AppendAuditBatch implements business.AuditStoreWriter. Every record goes to -// the events table; a content-class record's details go to the details table -// instead of the events row. -func (s *Store) AppendAuditBatch(ctx context.Context, batch business.AuditBatch) error { - var events, details []bigquery.ValueSaver +// BatchRows is the rows a batch appends: one events-table row per record, and +// a details-table row for each content-class record, whose details go there +// instead of its events row. +func BatchRows(batch business.AuditBatch) (events, details []Row) { for _, record := range batch.Records { events = append(events, EventRow(batch.DeploymentID, record)) if record.Retention != business.RetentionSecurity { details = append(details, DetailRow(batch.DeploymentID, record)) } } + return events, details +} + +// AppendAuditBatch implements business.AuditStoreWriter: the batch's rows +// (BatchRows) streamed to the events and details tables. +func (s *Store) AppendAuditBatch(ctx context.Context, batch business.AuditBatch) error { + eventRows, detailRows := BatchRows(batch) + events := make([]bigquery.ValueSaver, len(eventRows)) + for i, row := range eventRows { + events[i] = row + } + details := make([]bigquery.ValueSaver, len(detailRows)) + for i, row := range detailRows { + details[i] = row + } if err := putInChunks(ctx, s.events, events); err != nil { return fmt.Errorf("bigquery audit store: append to %s: %w", EventsTable, err) } diff --git a/module/services/accounts/code/pkg/auditstore/bigquerystore/reader.go b/module/services/accounts/code/pkg/auditstore/bigquerystore/reader.go new file mode 100644 index 000000000..7f79d9280 --- /dev/null +++ b/module/services/accounts/code/pkg/auditstore/bigquerystore/reader.go @@ -0,0 +1,813 @@ +package bigquerystore + +import ( + "bytes" + "context" + "errors" + "fmt" + "io" + "strings" + "sync" + "time" + + "accounts/pkg/auditstore/auditeval" + "accounts/pkg/business" + + "cloud.google.com/go/bigquery/storage/apiv1/storagepb" + "github.com/apache/arrow/go/v15/arrow" + "github.com/apache/arrow/go/v15/arrow/array" + "github.com/apache/arrow/go/v15/arrow/ipc" + "github.com/apache/arrow/go/v15/arrow/memory" + "github.com/googleapis/gax-go/v2" + "golang.org/x/sync/errgroup" + "google.golang.org/grpc/codes" + "google.golang.org/grpc/status" +) + +// The read half of the BigQuery store (ADR 0009). +// +// The service that reads is the service that appends, and it holds +// bigquery.tables.updateData for the relay. With bigquery.jobs.create beside +// it, that identity could run DELETE, UPDATE and MERGE against the store of +// record, which append-only exists to prevent; so the deployment grants it no +// job permission at all. Reads therefore run no query: they go through the +// BigQuery Storage Read API, whose grants are bigquery.readsessions.create, +// .getData and .update and bigquery.tables.getData. Each read session carries a +// row restriction — this deployment, the caller's organization, the time window +// (so partitions outside it are pruned) and the event types when the read +// names them — and the service sorts, pages, deduplicates by event id, joins +// content details and aggregates the rows it streams back (auditeval). + +// ReadSessionClient is the Storage Read API surface the reader uses. +// *storage.BigQueryReadClient (cloud.google.com/go/bigquery/storage/apiv1) +// satisfies it; the caller builds it with Application Default Credentials. +type ReadSessionClient interface { + CreateReadSession(ctx context.Context, req *storagepb.CreateReadSessionRequest, opts ...gax.CallOption) (*storagepb.ReadSession, error) + ReadRows(ctx context.Context, req *storagepb.ReadRowsRequest, opts ...gax.CallOption) (storagepb.BigQueryRead_ReadRowsClient, error) +} + +// ReadConfig names what the reader reads and for whom. +type ReadConfig struct { + Client ReadSessionClient + // Project is the BigQuery project: the dataset's, and the one read sessions + // are created (and billed) in. + Project string + Dataset string + // DeploymentID confines every read to this deployment's events. + DeploymentID string + // MaxStreams bounds the streams one read session is split into and read in + // parallel. Zero is DefaultReadStreams. + MaxStreams int + // Now is a seam for tests; nil is the clock. + Now func() time.Time +} + +// DefaultReadStreams is the parallelism of one read session. +const DefaultReadStreams = 4 + +// Newest-first windows: the activity list and the readable-source query read +// the newest day first, then windows growing eightfold (8, 64 and 512 days +// before it), and stop once the answer is complete; past those, one last read +// takes everything older. Most pages fill from the first window. +const ( + firstReadWindow = 24 * time.Hour + readWindowGrowth = 8 + boundedReadWindows = 4 + readRowsRetryAttempts = 5 +) + +// Reader reads the events and details tables. +type Reader struct { + client ReadSessionClient + project string + dataset string + deploymentID string + maxStreams int + now func() time.Time +} + +// NewReader validates cfg and builds a reader. +func NewReader(cfg ReadConfig) (*Reader, error) { + if cfg.Client == nil { + return nil, errors.New("bigquery audit store: read client is required") + } + for name, value := range map[string]string{"project": cfg.Project, "dataset": cfg.Dataset, "deployment id": cfg.DeploymentID} { + if strings.TrimSpace(value) == "" { + return nil, fmt.Errorf("bigquery audit store: %s is required to read", name) + } + } + reader := &Reader{ + client: cfg.Client, + project: cfg.Project, + dataset: cfg.Dataset, + deploymentID: cfg.DeploymentID, + maxStreams: cfg.MaxStreams, + now: cfg.Now, + } + if reader.maxStreams <= 0 { + reader.maxStreams = DefaultReadStreams + } + if reader.now == nil { + reader.now = time.Now + } + return reader, nil +} + +var ( + eventFields = []string{ + "event_id", "deployment_id", "org_id", "actor_id", "actor_type", "event_type", "schema_version", + "resource", "resource_id", "occurred_at", "ip_address", "impersonated_by", "is_impersonated", + "client_id", "retention_class", "details_sha256", "details", + } + detailFields = []string{"event_id", "deployment_id", "org_id", "occurred_at", "details_sha256", "details"} + sourceSyncFields = []string{"event_id", "deployment_id", "org_id", "actor_id", "event_type", "resource", "resource_id", "occurred_at"} +) + +// window is a span of occurred_at: [lo, hi], or [lo, hi) when hiExclusive. A +// nil bound is open. +type window struct { + lo, hi *time.Time + hiExclusive bool +} + +// newestFirst yields the windows a newest-first read walks, from upper +// (inclusive, or open when nil) down to floor (inclusive, or open when nil), +// until visit says the answer is complete. +func (r *Reader) newestFirst(upper, floor *time.Time, visit func(window) (bool, error)) error { + anchor := r.now() + if upper != nil { + anchor = *upper + } + current := window{hi: upper} + span := firstReadWindow + lo := anchor.Add(-span) + for step := 1; ; step++ { + last := false + switch { + case floor != nil && !lo.After(*floor): + current.lo, last = floor, true + case step >= boundedReadWindows+1: + current.lo, last = floor, true + default: + bound := lo + current.lo = &bound + } + done, err := visit(current) + if err != nil || done || last { + return err + } + next := *current.lo + current = window{hi: &next, hiExclusive: true} + span *= readWindowGrowth + lo = next.Add(-span) + } +} + +// scope adds the read's scope to a restriction: this deployment always, and +// the organization unless the read is the explicit platform read. +func (r *Reader) scope(scope business.AuditReadScope) *restriction { + where := &restriction{} + where.eq("deployment_id", r.deploymentID) + if !scope.Platform() { + where.eq("org_id", scope.OrgID()) + } + return where +} + +func (w window) restrict(where *restriction) { + if w.lo != nil { + where.timeBound("occurred_at", ">=", *w.lo) + } + if w.hi != nil { + op := "<=" + if w.hiExclusive { + op = "<" + } + where.timeBound("occurred_at", op, *w.hi) + } +} + +// candidates is a read's restriction over a window: the scope, the window, +// and every envelope predicate of the query that a restriction can carry. ok +// is false when the read can match no event type at all. +func (r *Reader) candidates(m *auditeval.Matcher, q business.AuditQuery, w window) (*restriction, bool) { + where := r.scope(m.Scope()) + w.restrict(where) + if types, restricted := m.EventTypes(); restricted { + if len(types) == 0 { + return nil, false + } + if len(types) == 1 { + where.eq("event_type", types[0]) + } else { + where.in("event_type", types) + } + } + if m.ActorID() != "" { + where.eq("actor_id", m.ActorID()) + } + if q.Resource != "" { + where.eq("resource", q.Resource) + } + if q.ResourceID != "" { + where.eq("resource_id", q.ResourceID) + } + if q.ClientID != "" { + where.eq("client_id", q.ClientID) + } + return where, true +} + +// readEvents streams the events in a window that the read could match, each +// joined to its content details when withDetails, and hands each to visit +// (serially). A row of another deployment is never handed on, whatever the +// session returned. +func (r *Reader) readEvents(ctx context.Context, m *auditeval.Matcher, q business.AuditQuery, w window, withDetails bool, visit func(*auditeval.Event) error) error { + where, ok := r.candidates(m, q, w) + if !ok { + return nil + } + var details map[string]storedDetail + if withDetails { + detailsWhere := r.scope(m.Scope()) + w.restrict(detailsWhere) + if types, restricted := m.EventTypes(); restricted { + if len(types) == 1 { + detailsWhere.eq("event_type", types[0]) + } else { + detailsWhere.in("event_type", types) + } + } + var err error + if details, err = r.readDetails(ctx, detailsWhere); err != nil { + return err + } + } + return r.scan(ctx, EventsTable, eventFields, where, func(row arrowRow) error { + stored, err := decodeEventRow(row) + if err != nil { + return err + } + if stored.deploymentID != r.deploymentID { + return nil + } + event := stored.event() + if withDetails && stored.retentionClass != string(business.RetentionSecurity) { + if detail, ok := details[event.Entry.ID]; ok && detail.sha256 == stored.detailsSHA256 { + event.Details, event.HasDetails = detail.details, true + } + } + return visit(event) + }) +} + +// storedDetail is one details-table row: the canonical details of a +// content-class event and their hash. +type storedDetail struct { + sha256 string + details string +} + +func (r *Reader) readDetails(ctx context.Context, where *restriction) (map[string]storedDetail, error) { + details := map[string]storedDetail{} + err := r.scan(ctx, DetailsTable, detailFields, where, func(row arrowRow) error { + eventID, err := row.str("event_id") + if err != nil { + return err + } + deploymentID, err := row.str("deployment_id") + if err != nil { + return err + } + if deploymentID != r.deploymentID { + return nil + } + sha, err := row.str("details_sha256") + if err != nil { + return err + } + text, err := row.str("details") + if err != nil { + return err + } + details[eventID] = storedDetail{sha256: sha, details: text} + return nil + }) + return details, err +} + +// joinDetails reads the details of the given events that have none yet — the +// content-class events of a list page that did not need them to match — in +// one session bounded by their occurrences and ids. An event whose details are +// past the content window keeps none. +func (r *Reader) joinDetails(ctx context.Context, scope business.AuditReadScope, events []*auditeval.Event) error { + var ids []string + var lo, hi time.Time + for _, event := range events { + if event.HasDetails { + continue + } + at := event.Entry.CreatedAt + if len(ids) == 0 || at.Before(lo) { + lo = at + } + if len(ids) == 0 || at.After(hi) { + hi = at + } + ids = append(ids, event.Entry.ID) + } + if len(ids) == 0 { + return nil + } + where := r.scope(scope) + window{lo: &lo, hi: &hi}.restrict(where) + where.in("event_id", ids) + details, err := r.readDetails(ctx, where) + if err != nil { + return err + } + for _, event := range events { + if event.HasDetails { + continue + } + if detail, ok := details[event.Entry.ID]; ok && detail.sha256 == event.DetailsSHA256 { + event.Details, event.HasDetails = detail.details, true + } + } + return nil +} + +// ListAuditEvents implements business.AuditReader: the newest windows first, +// stopping once the page and the proof of a next one are in hand. +func (r *Reader) ListAuditEvents(ctx context.Context, read business.AuditRead) ([]business.AuditEntry, string, error) { + m, err := auditeval.NewMatcher(read) + if err != nil { + return nil, "", err + } + page, err := auditeval.NewPage(read.Query) + if err != nil { + return nil, "", err + } + upper := m.To() + if before := page.Before(); before != nil && (upper == nil || before.Before(*upper)) { + upper = before + } + // A payload filter needs every candidate's details to decide; otherwise + // only the settled page's content details are read, afterwards. + withDetails := m.NeedsPayload() + err = r.newestFirst(upper, m.From(), func(w window) (bool, error) { + err := r.readEvents(ctx, m, read.Query, w, withDetails, func(event *auditeval.Event) error { + ok, err := m.Match(event) + if err != nil || !ok { + return err + } + page.Offer(event) + return nil + }) + return page.Full(), err + }) + if err != nil { + return nil, "", err + } + events, next := page.Result() + if !withDetails { + if err := r.joinDetails(ctx, m.Scope(), events); err != nil { + return nil, "", err + } + } + return results(events), next, nil +} + +func results(events []*auditeval.Event) []business.AuditEntry { + if len(events) == 0 { + return nil + } + out := make([]business.AuditEntry, len(events)) + for i, event := range events { + out[i] = event.Result() + } + return out +} + +// needsPayload reports whether an aggregation reads payloads: a payload filter, +// a payload dimension, or a metric over a payload key. +func needsPayload(m *auditeval.Matcher, spec business.AuditAggregationSpec) bool { + if m.NeedsPayload() { + return true + } + for _, dimension := range spec.GroupBy { + if strings.HasPrefix(dimension, "payload:") { + return true + } + } + for _, metric := range spec.Metrics { + if strings.HasPrefix(metric.Field, "payload:") { + return true + } + } + return false +} + +// AggregateAuditEvents implements business.AuditReader: one read of the +// query's window, deduplicated and aggregated in the service. +func (r *Reader) AggregateAuditEvents(ctx context.Context, read business.AuditRead, spec business.AuditAggregationSpec) ([]business.AuditAggregateBucket, error) { + m, err := auditeval.NewMatcher(read) + if err != nil { + return nil, err + } + aggregator, err := auditeval.NewAggregator(spec, read.Types) + if err != nil { + return nil, err + } + seen := auditeval.NewDedupe() + w := window{lo: m.From(), hi: m.To()} + err = r.readEvents(ctx, m, read.Query, w, needsPayload(m, spec), func(event *auditeval.Event) error { + ok, err := m.Match(event) + if err != nil || !ok || !seen.First(event.Entry.ID) { + return err + } + return aggregator.Add(event) + }) + if err != nil { + return nil, err + } + return aggregator.Buckets(), nil +} + +// ExportAuditEvents implements business.AuditReader: every matching event of +// the query's window, once each, newest first, with its payload. +func (r *Reader) ExportAuditEvents(ctx context.Context, read business.AuditRead) ([]business.AuditEntry, error) { + m, err := auditeval.NewMatcher(read) + if err != nil { + return nil, err + } + seen := auditeval.NewDedupe() + var events []*auditeval.Event + w := window{lo: m.From(), hi: m.To()} + err = r.readEvents(ctx, m, read.Query, w, true, func(event *auditeval.Event) error { + ok, err := m.Match(event) + if err != nil || !ok || !seen.First(event.Entry.ID) { + return err + } + events = append(events, event) + return nil + }) + if err != nil { + return nil, err + } + auditeval.SortNewestFirst(events) + return results(events), nil +} + +// LatestSourceSyncEvents implements business.AuditSourceSyncReader: the newest +// windows first, until every source has its newest request or the history +// runs out. +func (r *Reader) LatestSourceSyncEvents(ctx context.Context, scope business.AuditReadScope, sources []string) (map[string]business.AuditSourceSyncEvent, error) { + if err := scope.Validate(); err != nil { + return nil, err + } + if scope.Platform() { + return nil, errors.New("bigquery audit store: the readable-source query reads one organization") + } + out := map[string]business.AuditSourceSyncEvent{} + wanted := map[string]bool{} + var ids []string + for _, source := range sources { + if !wanted[source] { + wanted[source] = true + ids = append(ids, source) + } + } + if len(ids) == 0 { + return out, nil + } + newestIDs := map[string]string{} + eventType := string(business.EventDatasourceSourceSynced) + err := r.newestFirst(nil, nil, func(w window) (bool, error) { + where := r.scope(scope) + w.restrict(where) + where.eq("event_type", eventType) + where.eq("resource", "datasource") + where.in("resource_id", ids) + err := r.scan(ctx, EventsTable, sourceSyncFields, where, func(row arrowRow) error { + stored, err := decodeEventRow(row) + if err != nil { + return err + } + entry := stored.event().Entry + if stored.deploymentID != r.deploymentID || entry.OrgID != scope.OrgID() || string(entry.EventType) != eventType || + entry.Resource != "datasource" || !wanted[entry.ResourceID] { + return nil + } + current, found := out[entry.ResourceID] + if !found || entry.CreatedAt.After(current.RequestedAt) || + (entry.CreatedAt.Equal(current.RequestedAt) && entry.ID > newestIDs[entry.ResourceID]) { + out[entry.ResourceID] = business.AuditSourceSyncEvent{RequestedAt: entry.CreatedAt, ActorID: entry.ActorID} + newestIDs[entry.ResourceID] = entry.ID + } + return nil + }) + return len(out) == len(ids), err + }) + if err != nil { + return nil, err + } + return out, nil +} + +// scan creates one read session over table and hands every row it returns to +// visit, serially. The session's streams are read in parallel; a stream broken +// by a transient error is reopened at the row it reached. +func (r *Reader) scan(ctx context.Context, table string, fields []string, where *restriction, visit func(arrowRow) error) error { + if where.err != nil { + return where.err + } + session, err := r.client.CreateReadSession(ctx, &storagepb.CreateReadSessionRequest{ + Parent: "projects/" + r.project, + ReadSession: &storagepb.ReadSession{ + Table: fmt.Sprintf("projects/%s/datasets/%s/tables/%s", r.project, r.dataset, table), + DataFormat: storagepb.DataFormat_ARROW, + ReadOptions: &storagepb.ReadSession_TableReadOptions{ + SelectedFields: fields, + RowRestriction: where.String(), + }, + }, + MaxStreamCount: int32(r.maxStreams), + }) + if err != nil { + return fmt.Errorf("bigquery audit store: create read session on %s: %w", table, err) + } + if len(session.GetStreams()) == 0 { + return nil + } + schema := session.GetArrowSchema().GetSerializedSchema() + if len(schema) == 0 { + return fmt.Errorf("bigquery audit store: read session on %s returned no Arrow schema", table) + } + var mu sync.Mutex + group, ctx := errgroup.WithContext(ctx) + for _, stream := range session.GetStreams() { + name := stream.GetName() + group.Go(func() error { + return r.readStream(ctx, table, name, schema, func(row arrowRow) error { + mu.Lock() + defer mu.Unlock() + return visit(row) + }) + }) + } + return group.Wait() +} + +// readStream reads one stream to its end, reopening it at the offset reached +// after a transient failure. +func (r *Reader) readStream(ctx context.Context, table, stream string, schema []byte, visit func(arrowRow) error) error { + var offset int64 + backoff := gax.Backoff{Initial: 100 * time.Millisecond, Max: 5 * time.Second} + failures := 0 + for { + rows, err := r.client.ReadRows(ctx, &storagepb.ReadRowsRequest{ReadStream: stream, Offset: offset}) + if err == nil { + offset, err = r.consume(rows, schema, offset, visit) + if err == nil { + return nil + } + } + var decodeErr decodeError + if errors.As(err, &decodeErr) || !retryableRead(err) || ctx.Err() != nil { + return fmt.Errorf("bigquery audit store: read %s stream %s: %w", table, stream, err) + } + failures++ + if failures >= readRowsRetryAttempts { + return fmt.Errorf("bigquery audit store: read %s stream %s after %d attempts: %w", table, stream, failures, err) + } + if err := gax.Sleep(ctx, backoff.Pause()); err != nil { + return err + } + } +} + +// consume reads responses until the stream ends, returning the offset reached. +func (r *Reader) consume(rows storagepb.BigQueryRead_ReadRowsClient, schema []byte, offset int64, visit func(arrowRow) error) (int64, error) { + for { + response, err := rows.Recv() + if errors.Is(err, io.EOF) { + return offset, nil + } + if err != nil { + return offset, err + } + batch := response.GetArrowRecordBatch() + if batch == nil || response.GetRowCount() == 0 { + continue + } + if err := decodeBatch(schema, batch.GetSerializedRecordBatch(), visit); err != nil { + return offset, err + } + offset += response.GetRowCount() + } +} + +func retryableRead(err error) bool { + switch status.Code(err) { + case codes.Aborted, codes.DeadlineExceeded, codes.Internal, codes.Unavailable, codes.ResourceExhausted: + return true + default: + return false + } +} + +// decodeError is a row the reader could not decode: a schema the store did +// not write, never worth retrying. +type decodeError struct{ err error } + +func (e decodeError) Error() string { return e.err.Error() } +func (e decodeError) Unwrap() error { return e.err } + +// decodeBatch decodes one serialized Arrow record batch against the session's +// serialized schema and hands each row to visit. A visit error is returned as +// it is; a decoding failure as a decodeError. +func decodeBatch(schema, batch []byte, visit func(arrowRow) error) error { + reader, err := ipc.NewReader(io.MultiReader(bytes.NewReader(schema), bytes.NewReader(batch)), ipc.WithAllocator(memory.DefaultAllocator)) + if err != nil { + return decodeError{fmt.Errorf("decode arrow batch: %w", err)} + } + defer reader.Release() + for reader.Next() { + record := reader.Record() + columns := make(map[string]arrow.Array, record.NumCols()) + for i, field := range record.Schema().Fields() { + columns[field.Name] = record.Column(i) + } + for i := 0; i < int(record.NumRows()); i++ { + if err := visit(arrowRow{columns: columns, index: i}); err != nil { + return err + } + } + } + if err := reader.Err(); err != nil { + return decodeError{fmt.Errorf("decode arrow batch: %w", err)} + } + return nil +} + +// arrowRow is one row of a decoded record batch. +type arrowRow struct { + columns map[string]arrow.Array + index int +} + +func (r arrowRow) column(name string) (arrow.Array, error) { + column, ok := r.columns[name] + if !ok { + return nil, decodeError{fmt.Errorf("column %s is missing", name)} + } + return column, nil +} + +// str is a STRING column's value; NULL is "". +func (r arrowRow) str(name string) (string, error) { + column, err := r.column(name) + if err != nil { + return "", err + } + if column.IsNull(r.index) { + return "", nil + } + switch values := column.(type) { + case *array.String: + return values.Value(r.index), nil + case *array.LargeString: + return values.Value(r.index), nil + default: + return "", decodeError{fmt.Errorf("column %s is %s, want a string", name, column.DataType())} + } +} + +// optional is a nullable STRING column's value and whether it is set. +func (r arrowRow) optional(name string) (string, bool, error) { + column, err := r.column(name) + if err != nil { + return "", false, err + } + if column.IsNull(r.index) { + return "", false, nil + } + value, err := r.str(name) + return value, err == nil, err +} + +func (r arrowRow) integer(name string) (int64, error) { + column, err := r.column(name) + if err != nil { + return 0, err + } + values, ok := column.(*array.Int64) + if !ok { + return 0, decodeError{fmt.Errorf("column %s is %s, want int64", name, column.DataType())} + } + return values.Value(r.index), nil +} + +func (r arrowRow) boolean(name string) (bool, error) { + column, err := r.column(name) + if err != nil { + return false, err + } + values, ok := column.(*array.Boolean) + if !ok { + return false, decodeError{fmt.Errorf("column %s is %s, want bool", name, column.DataType())} + } + return !values.IsNull(r.index) && values.Value(r.index), nil +} + +// timestamp is a TIMESTAMP column's value in the process's local zone — the +// zone a Postgres read hands back — at the microsecond precision both keep. +func (r arrowRow) timestamp(name string) (time.Time, error) { + column, err := r.column(name) + if err != nil { + return time.Time{}, err + } + values, ok := column.(*array.Timestamp) + if !ok { + return time.Time{}, decodeError{fmt.Errorf("column %s is %s, want a timestamp", name, column.DataType())} + } + value := int64(values.Value(r.index)) + switch values.DataType().(*arrow.TimestampType).Unit { + case arrow.Second: + return time.Unix(value, 0), nil + case arrow.Millisecond: + return time.UnixMilli(value), nil + case arrow.Microsecond: + return time.UnixMicro(value), nil + default: + return time.Unix(0, value), nil + } +} + +// storedEvent is one events-table row. +type storedEvent struct { + entry business.AuditEntry + deploymentID string + retentionClass string + detailsSHA256 string + details string + hasDetails bool +} + +func (s storedEvent) event() *auditeval.Event { + return &auditeval.Event{Entry: s.entry, Details: s.details, HasDetails: s.hasDetails, DetailsSHA256: s.detailsSHA256} +} + +// decodeEventRow reads the selected columns of an events-table row; a column +// the read did not select is left zero. +func decodeEventRow(row arrowRow) (storedEvent, error) { + var stored storedEvent + entry := &stored.entry + text := func(name string, into *string) error { + if _, selected := row.columns[name]; !selected { + return nil + } + value, err := row.str(name) + *into = value + return err + } + var eventType string + for name, into := range map[string]*string{ + "event_id": &entry.ID, "deployment_id": &stored.deploymentID, "org_id": &entry.OrgID, + "actor_id": &entry.ActorID, "actor_type": &entry.ActorType, "event_type": &eventType, + "resource": &entry.Resource, "resource_id": &entry.ResourceID, "ip_address": &entry.IPAddress, + "impersonated_by": &entry.ImpersonatedBy, "client_id": &entry.ClientID, + "retention_class": &stored.retentionClass, "details_sha256": &stored.detailsSHA256, + } { + if err := text(name, into); err != nil { + return storedEvent{}, err + } + } + entry.EventType = business.EventType(eventType) + if _, selected := row.columns["schema_version"]; selected { + version, err := row.integer("schema_version") + if err != nil { + return storedEvent{}, err + } + entry.SchemaVersion = int(version) + } + if _, selected := row.columns["is_impersonated"]; selected { + impersonated, err := row.boolean("is_impersonated") + if err != nil { + return storedEvent{}, err + } + entry.IsImpersonated = impersonated + } + occurredAt, err := row.timestamp("occurred_at") + if err != nil { + return storedEvent{}, err + } + entry.CreatedAt = occurredAt + if _, selected := row.columns["details"]; selected { + if stored.details, stored.hasDetails, err = row.optional("details"); err != nil { + return storedEvent{}, err + } + } + return stored, nil +} diff --git a/module/services/accounts/code/pkg/auditstore/bigquerystore/reader_test.go b/module/services/accounts/code/pkg/auditstore/bigquerystore/reader_test.go new file mode 100644 index 000000000..bc53ff3d9 --- /dev/null +++ b/module/services/accounts/code/pkg/auditstore/bigquerystore/reader_test.go @@ -0,0 +1,274 @@ +package bigquerystore_test + +import ( + "context" + "fmt" + "strings" + "sync" + "testing" + "time" + + "accounts/pkg/auditstore/bigqueryfake" + "accounts/pkg/auditstore/bigquerystore" + "accounts/pkg/business" + + "github.com/stretchr/testify/require" + "google.golang.org/grpc/codes" + "google.golang.org/grpc/status" +) + +// The reader against the fake Storage Read API: what each read pushes down, +// and that the answer does not depend on it. + +const ( + project = "test-project" + dataset = "audit" + deployment = "deployment-1" + orgA = "aaaaaaaa-0000-4000-8000-000000000001" + orgB = "bbbbbbbb-0000-4000-8000-000000000002" + actor = "cccccccc-0000-4000-8000-000000000003" +) + +var now = time.Date(2026, 10, 3, 12, 0, 0, 0, time.UTC) + +type warehouse struct { + fake *bigqueryfake.Server + reader *bigquerystore.Reader +} + +func newWarehouse(t *testing.T) *warehouse { + t.Helper() + fake := bigqueryfake.New() + fake.CreateTable(bigqueryfake.TablePath(project, dataset, bigquerystore.EventsTable), bigquerystore.EventsSchema()) + fake.CreateTable(bigqueryfake.TablePath(project, dataset, bigquerystore.DetailsTable), bigquerystore.DetailsSchema()) + reader, err := bigquerystore.NewReader(bigquerystore.ReadConfig{ + Client: fake, Project: project, Dataset: dataset, DeploymentID: deployment, + Now: func() time.Time { return now }, + }) + require.NoError(t, err) + return &warehouse{fake: fake, reader: reader} +} + +// append writes records as the relay's batch would, under deploymentID. +func (w *warehouse) append(t *testing.T, deploymentID string, records ...business.AuditRecord) { + t.Helper() + events, details := bigquerystore.BatchRows(business.AuditBatch{ID: "batch", DeploymentID: deploymentID, Records: records}) + for _, row := range events { + require.NoError(t, w.fake.Insert(bigqueryfake.TablePath(project, dataset, bigquerystore.EventsTable), row.Values)) + } + for _, row := range details { + require.NoError(t, w.fake.Insert(bigqueryfake.TablePath(project, dataset, bigquerystore.DetailsTable), row.Values)) + } +} + +func event(t *testing.T, id int, org string, eventType business.EventType, class business.AuditRetentionClass, at time.Time, payload map[string]any) business.AuditRecord { + t.Helper() + record, err := business.NewAuditRecord(business.AuditEntry{ + ID: fmt.Sprintf("00000000-0000-4000-8000-%012d", id), + OrgID: org, + ActorID: actor, + ActorType: business.ActorTypeUser, + EventType: eventType, + SchemaVersion: 1, + Resource: "datasource", + ResourceID: fmt.Sprintf("source-%d", id%3), + Payload: payload, + CreatedAt: at, + }, class) + require.NoError(t, err) + return record +} + +func ids(entries []business.AuditEntry) []string { + out := make([]string, len(entries)) + for i, entry := range entries { + out[i] = entry.ID[len(entry.ID)-3:] + } + return out +} + +func orgRead(q business.AuditQuery) business.AuditRead { + q.OrgID = orgA + return business.AuditRead{Scope: business.OrganizationAuditScope(orgA), Query: q} +} + +func TestListReadsNewestWindowsFirstAndStopsWhenThePageIsFull(t *testing.T) { + w := newWarehouse(t) + // Two events today, one 3 days ago, one 100 days ago, one 2 years ago. + ages := []time.Duration{time.Hour, 2 * time.Hour, 72 * time.Hour, 100 * 24 * time.Hour, 730 * 24 * time.Hour} + for i, age := range ages { + w.append(t, deployment, event(t, i+1, orgA, business.EventAuthLogin, business.RetentionSecurity, now.Add(-age), map[string]any{"n": i})) + } + + entries, next, err := w.reader.ListAuditEvents(context.Background(), orgRead(business.AuditQuery{PageSize: 1})) + require.NoError(t, err) + require.Equal(t, []string{"001"}, ids(entries)) + require.NotEmpty(t, next) + require.Len(t, w.fake.Sessions(), 1, "the newest day held a page and the proof of another") + + entries, next, err = w.reader.ListAuditEvents(context.Background(), orgRead(business.AuditQuery{PageSize: 3, PageToken: next})) + require.NoError(t, err) + require.Equal(t, []string{"002", "003", "004"}, ids(entries)) + require.NotEmpty(t, next) + + entries, next, err = w.reader.ListAuditEvents(context.Background(), orgRead(business.AuditQuery{PageSize: 3, PageToken: next})) + require.NoError(t, err) + require.Equal(t, []string{"005"}, ids(entries), "the last read takes everything older than the bounded windows") + require.Empty(t, next) + + for _, session := range w.fake.Sessions() { + restriction := session.GetReadSession().GetReadOptions().GetRowRestriction() + require.Contains(t, restriction, `deployment_id = "deployment-1"`) + require.Contains(t, restriction, `org_id = "`+orgA+`"`) + } +} + +func TestEveryEventsReadIsBoundedInTimeWhenTheQueryIs(t *testing.T) { + w := newWarehouse(t) + from, to := now.Add(-48*time.Hour), now.Add(-time.Hour) + fromBound := `occurred_at >= CAST("2026-10-01 12:00:00.000000+00:00" AS TIMESTAMP)` + toBound := `occurred_at <= CAST("2026-10-03 11:00:00.000000+00:00" AS TIMESTAMP)` + _, err := w.reader.AggregateAuditEvents(context.Background(), orgRead(business.AuditQuery{From: &from, To: &to}), business.AuditAggregationSpec{}) + require.NoError(t, err) + _, err = w.reader.ExportAuditEvents(context.Background(), orgRead(business.AuditQuery{From: &from, To: &to})) + require.NoError(t, err) + for _, session := range w.fake.Sessions() { + restriction := session.GetReadSession().GetReadOptions().GetRowRestriction() + require.Contains(t, restriction, fromBound, "partitions before the window are pruned") + require.Contains(t, restriction, toBound, "partitions after the window are pruned") + } + + sessions := len(w.fake.Sessions()) + _, _, err = w.reader.ListAuditEvents(context.Background(), orgRead(business.AuditQuery{From: &from, To: &to})) + require.NoError(t, err) + list := w.fake.Sessions()[sessions:] + require.Len(t, list, 2, "the newest day, then the rest of the window") + require.Contains(t, list[0].GetReadSession().GetReadOptions().GetRowRestriction(), toBound) + require.Contains(t, list[1].GetReadSession().GetReadOptions().GetRowRestriction(), fromBound) + for _, session := range list { + restriction := session.GetReadSession().GetReadOptions().GetRowRestriction() + require.Contains(t, restriction, "occurred_at >=") + require.Contains(t, restriction, "occurred_at <") + } +} + +func TestEventTypeFiltersArePushedDown(t *testing.T) { + w := newWarehouse(t) + read := orgRead(business.AuditQuery{Category: "auth"}) + read.Types = business.AuditEventTypeIndex{ + "saas.auth.login": {Category: "auth"}, + "saas.auth.logout": {Category: "auth"}, + "saas.org.created": {Category: "org"}, + } + _, _, err := w.reader.ListAuditEvents(context.Background(), read) + require.NoError(t, err) + require.Contains(t, w.fake.Sessions()[0].GetReadSession().GetReadOptions().GetRowRestriction(), + `event_type IN ("saas.auth.login", "saas.auth.logout")`) + + sessions := len(w.fake.Sessions()) + read.Query.EventType = "saas.org.created" + entries, _, err := w.reader.ListAuditEvents(context.Background(), read) + require.NoError(t, err) + require.Empty(t, entries) + require.Len(t, w.fake.Sessions(), sessions, "a read no type can satisfy reads nothing") +} + +func TestReadsRefuseAMissingScopeAndAQueryOutsideIt(t *testing.T) { + w := newWarehouse(t) + ctx := context.Background() + _, _, err := w.reader.ListAuditEvents(ctx, business.AuditRead{Query: business.AuditQuery{}}) + require.ErrorIs(t, err, business.ErrAuditReadUnscoped) + _, err = w.reader.AggregateAuditEvents(ctx, business.AuditRead{}, business.AuditAggregationSpec{}) + require.ErrorIs(t, err, business.ErrAuditReadUnscoped) + _, err = w.reader.ExportAuditEvents(ctx, business.AuditRead{}) + require.ErrorIs(t, err, business.ErrAuditReadUnscoped) + _, err = w.reader.LatestSourceSyncEvents(ctx, business.AuditReadScope{}, []string{"source-1"}) + require.ErrorIs(t, err, business.ErrAuditReadUnscoped) + + _, _, err = w.reader.ListAuditEvents(ctx, business.AuditRead{ + Scope: business.OrganizationAuditScope(orgA), Query: business.AuditQuery{OrgID: orgB}, + }) + require.ErrorContains(t, err, "outside the organization") + _, _, err = w.reader.ListAuditEvents(ctx, business.AuditRead{ + Scope: business.PlatformAuditScope(), Query: business.AuditQuery{OrgID: orgB}, + }) + require.ErrorContains(t, err, "outside the platform scope") + require.Empty(t, w.fake.Sessions(), "a refused read reads nothing") +} + +func TestScopeHoldsWhenTheStoreReturnsEveryRow(t *testing.T) { + w := newWarehouse(t) + w.fake.IgnoreRestrictions = true + w.append(t, deployment, event(t, 1, orgA, business.EventAuthLogin, business.RetentionSecurity, now.Add(-time.Hour), nil)) + w.append(t, deployment, event(t, 2, orgB, business.EventAuthLogin, business.RetentionSecurity, now.Add(-time.Hour), nil)) + w.append(t, "deployment-2", event(t, 3, orgA, business.EventAuthLogin, business.RetentionSecurity, now.Add(-time.Hour), nil)) + w.append(t, deployment, event(t, 4, "", business.EventAuthLogin, business.RetentionSecurity, now.Add(-time.Hour), nil)) + + entries, _, err := w.reader.ListAuditEvents(context.Background(), orgRead(business.AuditQuery{})) + require.NoError(t, err) + require.Equal(t, []string{"001"}, ids(entries), "another organization, another deployment and the platform are out of scope") + + entries, _, err = w.reader.ListAuditEvents(context.Background(), business.AuditRead{Scope: business.PlatformAuditScope()}) + require.NoError(t, err) + require.Equal(t, []string{"004", "002", "001"}, ids(entries), "the platform read spans organizations, never deployments") +} + +func TestContentDetailsAreJoinedWithinTheirWindowOnly(t *testing.T) { + w := newWarehouse(t) + recent := event(t, 1, orgA, business.EventDatasourceSourceSynced, business.RetentionContent, now.Add(-time.Hour), map[string]any{"n": 1}) + expired := event(t, 2, orgA, business.EventDatasourceSourceSynced, business.RetentionContent, now.Add(-2*time.Hour), map[string]any{"n": 2}) + w.append(t, deployment, recent, expired) + // The details table expires partitions at the content window; drop the row + // that BigQuery would have expired. + detailsPath := bigqueryfake.TablePath(project, dataset, bigquerystore.DetailsTable) + kept := w.fake.Rows(detailsPath)[:1] + w.fake.CreateTable(detailsPath, bigquerystore.DetailsSchema()) + require.NoError(t, w.fake.Insert(detailsPath, kept...)) + + entries, _, err := w.reader.ListAuditEvents(context.Background(), orgRead(business.AuditQuery{})) + require.NoError(t, err) + require.Len(t, entries, 2) + require.Equal(t, map[string]any{"n": float64(1)}, entries[0].Payload) + require.Nil(t, entries[1].Payload, "an event past the content window keeps its envelope, not its details") + + entries, _, err = w.reader.ListAuditEvents(context.Background(), orgRead(business.AuditQuery{PayloadContains: map[string]any{"n": 2}})) + require.NoError(t, err) + require.Empty(t, entries, "a payload filter cannot match details that are gone") +} + +func TestATransientStreamFailureResumesAtTheOffsetReached(t *testing.T) { + w := newWarehouse(t) + w.fake.StreamsPerSession, w.fake.RowsPerBatch = 1, 1 + for i := 1; i <= 4; i++ { + w.append(t, deployment, event(t, i, orgA, business.EventAuthLogin, business.RetentionSecurity, now.Add(-time.Duration(i)*time.Minute), nil)) + } + var mu sync.Mutex + failed := map[int64]bool{} + w.fake.FailReadRows = func(_ string, offset int64) error { + mu.Lock() + defer mu.Unlock() + if offset == 2 && !failed[offset] { + failed[offset] = true + return status.Error(codes.Unavailable, "transient") + } + return nil + } + entries, _, err := w.reader.ListAuditEvents(context.Background(), orgRead(business.AuditQuery{})) + require.NoError(t, err) + require.Equal(t, []string{"001", "002", "003", "004"}, ids(entries), "nothing lost, nothing read twice") + + w.fake.FailReadRows = func(string, int64) error { return status.Error(codes.PermissionDenied, "no getData") } + _, _, err = w.reader.ListAuditEvents(context.Background(), orgRead(business.AuditQuery{})) + require.Error(t, err) + require.Equal(t, codes.PermissionDenied, status.Code(err), "a permanent failure is not retried: %v", err) +} + +func TestRestrictionLiteralsCannotEscapeTheirQuotes(t *testing.T) { + w := newWarehouse(t) + hostile := `x" OR TRUE OR "` + "\né\\" + _, _, err := w.reader.ListAuditEvents(context.Background(), orgRead(business.AuditQuery{ResourceID: hostile})) + require.NoError(t, err, "the fake parses the restriction back, so a broken literal fails here") + restriction := w.fake.Sessions()[0].GetReadSession().GetReadOptions().GetRowRestriction() + require.Contains(t, restriction, `resource_id = "x\" OR TRUE OR \"\U0000000A\U000000E9\\"`) + require.False(t, strings.Contains(restriction, "\n")) +} diff --git a/module/services/accounts/code/pkg/auditstore/bigquerystore/restriction.go b/module/services/accounts/code/pkg/auditstore/bigquerystore/restriction.go new file mode 100644 index 000000000..c851a4e48 --- /dev/null +++ b/module/services/accounts/code/pkg/auditstore/bigquerystore/restriction.go @@ -0,0 +1,88 @@ +package bigquerystore + +import ( + "fmt" + "strings" + "time" + "unicode/utf8" +) + +// restriction is a Storage Read API row restriction: a conjunction of simple +// comparisons the read session evaluates server-side, so BigQuery prunes the +// partitions outside its occurred_at bounds and the blocks outside its org_id +// cluster, and streams back only the rows it admits. It is an optimization and +// a narrowing, never the authority: the service evaluates every predicate +// again on every row it reads (auditeval), the scope included. +type restriction struct { + terms []string + err error +} + +func (r *restriction) add(term string) { r.terms = append(r.terms, term) } + +// eq adds field = value. +func (r *restriction) eq(field, value string) { + literal, err := stringLiteral(value) + if err != nil { + r.fail(field, err) + return + } + r.add(field + " = " + literal) +} + +// in adds field IN (values...). The caller never passes an empty set: a read +// that can match no type reads nothing at all. +func (r *restriction) in(field string, values []string) { + literals := make([]string, len(values)) + for i, value := range values { + literal, err := stringLiteral(value) + if err != nil { + r.fail(field, err) + return + } + literals[i] = literal + } + r.add(field + " IN (" + strings.Join(literals, ", ") + ")") +} + +// timeBound adds field op timestamp, for op one of >=, <, <=. +func (r *restriction) timeBound(field, op string, at time.Time) { + r.add(fmt.Sprintf("%s %s CAST(%q AS TIMESTAMP)", field, op, at.UTC().Format(restrictionTimestampLayout))) +} + +func (r *restriction) fail(field string, err error) { + if r.err == nil { + r.err = fmt.Errorf("bigquery audit store: cannot restrict %s: %w", field, err) + } +} + +// String is the restriction's text, or "" for none. +func (r *restriction) String() string { return strings.Join(r.terms, " AND ") } + +// restrictionTimestampLayout is a GoogleSQL timestamp string at microsecond +// precision, in UTC. +const restrictionTimestampLayout = "2006-01-02 15:04:05.000000+00:00" + +// stringLiteral quotes value as a GoogleSQL string literal: printable ASCII +// as itself, a quote and a backslash escaped, and every other code point as a +// \U escape, so no value can end the literal or carry a raw control character. +func stringLiteral(value string) (string, error) { + if !utf8.ValidString(value) { + return "", fmt.Errorf("%q is not valid UTF-8", value) + } + var b strings.Builder + b.WriteByte('"') + for _, r := range value { + switch { + case r == '"' || r == '\\': + b.WriteByte('\\') + b.WriteRune(r) + case r >= 0x20 && r < 0x7f: + b.WriteRune(r) + default: + fmt.Fprintf(&b, `\U%08X`, r) + } + } + b.WriteByte('"') + return b.String(), nil +} diff --git a/module/services/accounts/code/pkg/business/audit.go b/module/services/accounts/code/pkg/business/audit.go index 74b0c8e7b..dd142ce73 100644 --- a/module/services/accounts/code/pkg/business/audit.go +++ b/module/services/accounts/code/pkg/business/audit.go @@ -472,35 +472,54 @@ type AuditAggregateBucket struct { Metrics map[string]float64 } -// QueryAuditLog delegates to the store, scoping the read to the -// requested org under WithOrgTx so RLS lets the rows through. When -// OrgID is empty the caller is platform-admin (handler authz -// already enforced this in adapters/rpcs.go AuditServer.QueryAuditLog) -// and we use WithControlPlane to span tenants. +// QueryAuditLog reads one page of the activity list from the audit store of +// record. Under postgres and both that is audit_events, read under WithOrgTx +// for the requested org so RLS lets its rows through — or, when OrgID is +// empty, under WithControlPlane to span tenants: the caller is then a platform +// admin (handler authz already enforced this in adapters/rpcs.go +// AuditServer.QueryAuditLog). Under a swap value it is the store the swap +// selected, read under the same scope, named explicitly (AuditReadScope). func (s *Service) QueryAuditLog(ctx context.Context, q AuditQuery) ([]AuditEntry, string, int32, error) { if q.CollectionID != "" { return nil, "", 0, status.Error(codes.InvalidArgument, "collection analytics requires reader authorization") } - var entries []AuditEntry - var nextToken string - var total int32 - wrap := func(ctx context.Context) error { - ev, nt, tot, err := s.store.QueryAuditLog(ctx, q) - entries, nextToken, total = ev, nt, tot - return err + read, err := s.auditRead(ctx, q, nil) + if err != nil { + return nil, "", 0, err } - var err error - if q.OrgID == "" { - err = s.store.WithControlPlane(ctx, wrap) - } else { - err = s.store.WithOrgTx(ctx, q.OrgID, wrap) + entries, nextToken, err := s.auditReads().ListAuditEvents(ctx, read) + return entries, nextToken, int32(len(entries)), err +} + +// auditRead is the read of q in its scope (auditReadScopeFor). Under a swap +// value it also carries the event-type facts the query or the aggregation +// needs, read from audit_event_types under that same scope — the rows the +// Postgres reads would have joined in SQL. +func (s *Service) auditRead(ctx context.Context, q AuditQuery, spec *AuditAggregationSpec) (AuditRead, error) { + read := AuditRead{Scope: auditReadScopeFor(q), Query: q} + if s.auditStore == nil || !auditReadNeedsTypes(q, spec) { + return read, nil + } + err := postgresAuditReader{store: s.store}.within(ctx, read.Scope, func(ctx context.Context) error { + rows, err := s.store.ListAuditEventTypes(ctx) + if err != nil { + return err + } + read.Types = NewAuditEventTypeIndex(rows) + return nil + }) + if err != nil { + return AuditRead{}, fmt.Errorf("audit read: event types: %w", err) } - return entries, nextToken, total, err + return read, nil } // AggregateAuditLogForReader additionally gates exact-resource analytics on the -// reader's current resource grant. The check and query share the tenant tx. -// Organization-wide audit reads retain their existing audit authority contract. +// reader's current resource grant. Under postgres and both the check and query +// share the tenant tx. Under a swap value the check runs in that tenant tx and +// the aggregate is then read from the store of record — two steps, no shared +// snapshot. Organization-wide audit reads retain their existing audit +// authority contract. func (s *Service) AggregateAuditLogForReader(ctx context.Context, reader string, q AuditQuery, spec AuditAggregationSpec) ([]AuditAggregateBucket, error) { if q.From != nil && q.To != nil && q.From.After(*q.To) { return nil, status.Error(codes.InvalidArgument, "audit window is reversed") @@ -537,46 +556,67 @@ func (s *Service) AggregateAuditLogForReader(ctx context.Context, reader string, payload["boundary"] = q.CollectionID q.PayloadContains = payload } - var out []AuditAggregateBucket - err := s.store.WithOrgTx(ctx, q.OrgID, func(ctx context.Context) error { - // Each scope proves itself; there is no permissive default to fall - // through to. `proven` records that at least one check actually ran, so - // a future caller reaching here with neither scope set is denied rather - // than served on the strength of an untested assumption. - denied := status.Error(codes.PermissionDenied, "resource read access required") - proven := false - if q.CollectionID != "" { - allowed, err := s.canReadAuditCollection(ctx, reader, q.OrgID, q.CollectionID) - if err != nil { - return err - } - if !allowed { - return denied - } - proven = true + compiled := q + compiled.CollectionID = "" // Authorized and compiled into the boundary predicate above. + if s.auditStore != nil { + if err := s.store.WithOrgTx(ctx, q.OrgID, func(ctx context.Context) error { + return s.authorizeAuditResourceRead(ctx, reader, q) + }); err != nil { + return nil, err } - if q.ResourceID != "" { - allowed, err := s.canReadAuditResource(ctx, reader, q) - if err != nil { - return err - } - if !allowed { - return denied - } - proven = true + read, err := s.auditRead(ctx, compiled, &spec) + if err != nil { + return nil, err } - if !proven { - return denied + return s.auditStore.AggregateAuditEvents(ctx, read, spec) + } + var out []AuditAggregateBucket + err := s.store.WithOrgTx(ctx, q.OrgID, func(ctx context.Context) error { + if err := s.authorizeAuditResourceRead(ctx, reader, q); err != nil { + return err } var err error - compiled := q - compiled.CollectionID = "" // Authorized and compiled into the boundary predicate above. out, err = s.store.AggregateAuditLog(ctx, compiled, spec) return err }) return out, err } +// authorizeAuditResourceRead proves the reader may read the resource or +// collection q names, inside the caller's tenant transaction. Each scope proves +// itself; there is no permissive default to fall through to. `proven` records +// that at least one check actually ran, so a future caller reaching here with +// neither scope set is denied rather than served on the strength of an +// untested assumption. +func (s *Service) authorizeAuditResourceRead(ctx context.Context, reader string, q AuditQuery) error { + denied := status.Error(codes.PermissionDenied, "resource read access required") + proven := false + if q.CollectionID != "" { + allowed, err := s.canReadAuditCollection(ctx, reader, q.OrgID, q.CollectionID) + if err != nil { + return err + } + if !allowed { + return denied + } + proven = true + } + if q.ResourceID != "" { + allowed, err := s.canReadAuditResource(ctx, reader, q) + if err != nil { + return err + } + if !allowed { + return denied + } + proven = true + } + if !proven { + return denied + } + return nil +} + // isCollectionBoundaryEvent reports whether eventType is a registered event // that records a collection node under the payload key `boundary` — the shape // for which a payload `boundary` value names a collection node. Both the @@ -632,24 +672,16 @@ func (s *Service) canReadAuditCollection(ctx context.Context, reader, orgID, bou // The spec selects the group dimensions (event type, category, actor, time // bucket, or a payload field), the aggregations (count, distinct-count, sum, // avg, min, max, percentile over payload fields), and any derived ratios, -// filtered by the same predicates as QueryAuditLog. +// filtered by the same predicates as QueryAuditLog, and read in the same scope. func (s *Service) AggregateAuditLog(ctx context.Context, q AuditQuery, spec AuditAggregationSpec) ([]AuditAggregateBucket, error) { if q.CollectionID != "" { return nil, status.Error(codes.InvalidArgument, "collection analytics requires reader authorization") } - var out []AuditAggregateBucket - wrap := func(ctx context.Context) error { - var err error - out, err = s.store.AggregateAuditLog(ctx, q, spec) - return err - } - var err error - if q.OrgID == "" { - err = s.store.WithControlPlane(ctx, wrap) - } else { - err = s.store.WithOrgTx(ctx, q.OrgID, wrap) + read, err := s.auditRead(ctx, q, &spec) + if err != nil { + return nil, err } - return out, err + return s.auditReads().AggregateAuditEvents(ctx, read, spec) } // buildAuditEntry assembles an AuditEntry. actorID is the effective subject the diff --git a/module/services/accounts/code/pkg/business/audit_export.go b/module/services/accounts/code/pkg/business/audit_export.go index c3af8cd1a..ea9d09771 100644 --- a/module/services/accounts/code/pkg/business/audit_export.go +++ b/module/services/accounts/code/pkg/business/audit_export.go @@ -25,26 +25,19 @@ func (s *Service) ExportAuditLog(ctx context.Context, orgID, format, actorID, ev return nil, "", "", w.NewError("unsupported format: %s (use csv or json)", format) } - // Fetch all matching events (paginate internally to avoid huge single queries) - var all []AuditEntry - pageToken := "" - for { - // Service.QueryAuditLog wraps in WithOrgTx (or WithControlPlane when - // orgID is empty for platform-admin export) so RLS lets the - // rows through. Don't use s.store.QueryAuditLog directly here - // — that bypasses the wrap and returns zero rows. - entries, nextToken, _, err := s.QueryAuditLog(ctx, AuditQuery{ - OrgID: orgID, ActorID: actorID, EventType: eventType, EventTypes: eventTypes, - PageSize: 100, PageToken: pageToken, - }) - if err != nil { - return nil, "", "", w.Wrapf(err, "query audit log for export") - } - all = append(all, entries...) - if nextToken == "" || len(entries) == 0 { - break - } - pageToken = nextToken + // Every matching event, newest first, from the audit store of record. In + // Postgres the export pages the list 100 rows at a time, each page under + // WithOrgTx (or WithControlPlane when orgID is empty for platform-admin + // export) so RLS lets the rows through. + read, err := s.auditRead(ctx, AuditQuery{ + OrgID: orgID, ActorID: actorID, EventType: eventType, EventTypes: eventTypes, + }, nil) + if err != nil { + return nil, "", "", w.Wrapf(err, "query audit log for export") + } + all, err := s.auditReads().ExportAuditEvents(ctx, read) + if err != nil { + return nil, "", "", w.Wrapf(err, "query audit log for export") } timestamp := time.Now().UTC().Format("20060102-150405") diff --git a/module/services/accounts/code/pkg/business/audit_store.go b/module/services/accounts/code/pkg/business/audit_store.go index 62fba4e4c..6a1aba522 100644 --- a/module/services/accounts/code/pkg/business/audit_store.go +++ b/module/services/accounts/code/pkg/business/audit_store.go @@ -79,12 +79,15 @@ type AuditStoreWriter interface { } // AuditStore is the audit store of record under a swap value (ADR 0009): one -// interface carrying the write half, which the relay uses, and — once reads -// move — the read half (list, aggregate, export and the readable-source -// queries), embedded here beside AuditStoreWriter. The relay depends on -// AuditStoreWriter alone, so adding the read half changes no caller of it. +// interface carrying the write half, which the relay uses, and the read half — +// the activity list, aggregation and export (AuditReader) and the +// readable-source query (AuditSourceSyncReader) — which the service uses. The +// relay depends on AuditStoreWriter alone and the service on the read halves +// alone. type AuditStore interface { AuditStoreWriter + AuditReader + AuditSourceSyncReader } // AuditArchive writes a relay batch to the locked object-storage archive as one diff --git a/module/services/accounts/code/pkg/business/audit_store_parity_test.go b/module/services/accounts/code/pkg/business/audit_store_parity_test.go new file mode 100644 index 000000000..fedac8e4f --- /dev/null +++ b/module/services/accounts/code/pkg/business/audit_store_parity_test.go @@ -0,0 +1,424 @@ +//go:build !pure + +package business_test + +import ( + "context" + "fmt" + "testing" + "time" + + "accounts/pkg/auditstore/bigqueryfake" + "accounts/pkg/auditstore/bigquerystore" + "accounts/pkg/auth" + "accounts/pkg/business" + gen "accounts/pkg/gen/saas/accounts/v1" + "accounts/pkg/infra" + + "cloud.google.com/go/bigquery" + "github.com/google/uuid" + "github.com/stretchr/testify/require" + "google.golang.org/api/option" + "google.golang.org/grpc/codes" + "google.golang.org/grpc/status" +) + +// Read parity of the audit store swap (ADR 0009): one fixture, written once +// through the emitter as a postgres deployment writes it (audit_events) and +// once as a bigquery deployment does (the queue, drained by the relay into the +// warehouse), must read back identically through the service — the activity +// list page by page, every aggregation, both export formats and the +// readable-source query. The warehouse is the fake Storage Read API, which +// evaluates the reader's row restrictions; it also holds what Postgres does +// not: events delivered twice, and another deployment's and another +// organization's events, none of which may change an answer. + +const ( + parityProject = "parity-project" + parityDataset = "audit" + parityDeployment = "deployment-parity" +) + +// parityWarehouse is the relay's store of record: each batch's rows appended +// to the fake, as the BigQuery writer appends them. +type parityWarehouse struct { + fake *bigqueryfake.Server + batches []business.AuditBatch +} + +func (w *parityWarehouse) AppendAuditBatch(_ context.Context, batch business.AuditBatch) error { + events, details := bigquerystore.BatchRows(batch) + for _, row := range events { + if err := w.fake.Insert(bigqueryfake.TablePath(parityProject, parityDataset, bigquerystore.EventsTable), row.Values); err != nil { + return err + } + } + for _, row := range details { + if err := w.fake.Insert(bigqueryfake.TablePath(parityProject, parityDataset, bigquerystore.DetailsTable), row.Values); err != nil { + return err + } + } + w.batches = append(w.batches, batch) + return nil +} + +type discardArchive struct{} + +func (discardArchive) WriteAuditBatch(context.Context, business.AuditBatch) error { return nil } + +type parityFixture struct { + warehouse *parityWarehouse + postgres *business.Service + swapped *business.Service + store *bigquerystore.Store + org, otherOrg string + actor, other string + sourceOrg string + sourceReader string + sources []string + members map[string]string + unseenUser string + newestSyncedBy map[string]string +} + +func parityEntry(org, actor string, eventType business.EventType, resourceID string, at time.Time, payload map[string]any) business.AuditEntry { + return business.AuditEntry{ + ID: uuid.NewString(), + OrgID: org, + ActorID: actor, + ActorType: business.ActorTypeUser, + EventType: eventType, + SchemaVersion: 1, + Resource: "datasource", + ResourceID: resourceID, + Payload: payload, + CreatedAt: at, + } +} + +func newParityFixture(t *testing.T) *parityFixture { + t.Helper() + clearData(t) + ctx := testCtx + require.NoError(t, testStore.WithControlPlane(ctx, func(ctx context.Context) error { + return testStore.EnsureAuditPartitions(ctx, 1) + })) + + f := &parityFixture{ + org: uuid.NewString(), otherOrg: uuid.NewString(), + actor: uuid.NewString(), other: uuid.NewString(), + members: map[string]string{}, newestSyncedBy: map[string]string{}, + } + + // The readable-source half needs real principals: a member of the source's + // organization (labelled by display name), a registered user who is not a + // member (invisible under row-level security, so labelled by id), and an id + // with no principal at all. + member, err := testService.RegisterUser(ctx, &gen.RegisterUserRequest{ + PrimaryEmail: "parity-member@test.com", + Identity: &gen.UserIdentity{Provider: "email", ProviderId: "parity-member", ProviderEmail: "parity-member@test.com", EmailVerified: true}, + }) + require.NoError(t, err) + outsider, err := testService.RegisterUser(ctx, &gen.RegisterUserRequest{ + PrimaryEmail: "parity-outsider@test.com", + Identity: &gen.UserIdentity{Provider: "email", ProviderId: "parity-outsider", ProviderEmail: "parity-outsider@test.com", EmailVerified: true}, + }) + require.NoError(t, err) + created, err := testService.CreateOrganization(ctx, member.User.Uuid, &gen.CreateOrganizationRequest{Name: "Parity Organization", Slug: "parity-organization"}) + require.NoError(t, err) + f.sourceOrg, f.sourceReader, f.unseenUser = created.Organization.Id, member.User.Uuid, outsider.User.Uuid + f.sources = []string{uuid.NewString(), uuid.NewString(), uuid.NewString(), uuid.NewString()} + + // Every instant falls in a partition EnsureAuditPartitions provisions: this + // month or the one before it. + base := time.Now().UTC().Truncate(time.Microsecond).Add(-time.Minute) + day := 24 * time.Hour + shared := base.Add(-5 * time.Hour) // two events share this instant; the id breaks the tie + readPayload := func(boundary string, duration any, count any, outcome string) map[string]any { + payload := map[string]any{"boundary": boundary, "duration_ms": duration, "outcome": outcome} + if count != nil { + payload["result_count"] = count + } + return payload + } + entries := []business.AuditEntry{ + parityEntry(f.org, f.actor, business.EventAuthLogin, "", base, map[string]any{"method": "password", "client_id": "cli"}), + parityEntry(f.org, f.actor, business.EventDocumentRead, "doc-1", base.Add(-time.Hour), readPayload("b-1", 12.5, 3, "ok")), + parityEntry(f.org, f.other, business.EventDocumentRead, "doc-1", base.Add(-2*time.Hour), readPayload("b-1", "7.25", 1, "ok")), + parityEntry(f.org, f.other, business.EventDocumentRead, "doc-2", shared, readPayload("b-2", "fast", nil, "denied")), + parityEntry(f.org, f.actor, business.EventDocumentRead, "doc-2", shared, readPayload("b-2", 0.5, 4, "ok")), + parityEntry(f.org, "", business.EventOrgUpdated, "", base.Add(-26*time.Hour), map[string]any{"name": "Renamed", "slug": "renamed"}), + parityEntry(f.org, f.actor, business.EventDocumentRead, "doc-3", base.Add(-3*day), map[string]any{ + "boundary": "b-1", "duration_ms": 100, "outcome": "ok", "result_count": 2, + "tags": []any{"x", map[string]any{"k": 1}}, "nested": map[string]any{"zz": 1, "a": "q"}, "label": "&é", + }), + parityEntry(f.org, f.other, business.EventAuthLogin, "", base.Add(-9*day), map[string]any{"method": "passkey"}), + parityEntry(f.org, f.actor, business.EventDocumentRead, "doc-1", base.Add(-12*day), readPayload("b-3", 2.25, 8, "ok")), + parityEntry(f.org, f.actor, business.EventOrgUpdated, "", base.Add(-15*day), map[string]any{"name": "First"}), + parityEntry(f.org, f.other, business.EventDocumentRead, "doc-4", base.Add(-20*day), map[string]any{}), + // Another organization, and the platform, which only the platform read sees. + parityEntry(f.otherOrg, f.actor, business.EventDocumentRead, "doc-1", base.Add(-time.Hour), readPayload("b-1", 1, 1, "ok")), + parityEntry(f.otherOrg, f.actor, business.EventAuthLogin, "", base.Add(-4*day), nil), + parityEntry("", f.actor, business.EventAuthLogin, "", base.Add(-30*time.Minute), map[string]any{"method": "password"}), + } + impersonated := parityEntry(f.org, f.actor, business.EventOrgUpdated, "", base.Add(-6*day), map[string]any{"name": "Acting"}) + impersonated.IsImpersonated, impersonated.ImpersonatedBy, impersonated.ClientID, impersonated.IPAddress = true, f.other, "client-1", "203.0.113.7" + entries = append(entries, impersonated) + + // The readable-source fixture: per source, several requests by different + // actors; the newest one names who the projection reports. + synced := func(source, actor string, at time.Time) business.AuditEntry { + entry := parityEntry(f.sourceOrg, actor, business.EventDatasourceSourceSynced, source, at, map[string]any{"job_id": uuid.NewString(), "repo": "acme/handbook"}) + if actor == "" { + entry.ActorType = business.ActorTypeSystem + } + return entry + } + entries = append(entries, + synced(f.sources[0], f.unseenUser, base.Add(-3*day)), + synced(f.sources[0], f.sourceReader, base.Add(-2*time.Hour)), + synced(f.sources[1], f.sourceReader, base.Add(-25*day)), + synced(f.sources[1], f.other, base.Add(-10*day)), + synced(f.sources[2], f.sourceReader, base.Add(-time.Hour)), + synced(f.sources[2], "", base.Add(-time.Minute)), + synced(f.sources[3], f.unseenUser, base.Add(-90*time.Minute)), + ) + f.newestSyncedBy = map[string]string{f.sources[0]: f.sourceReader, f.sources[1]: f.other, f.sources[2]: "", f.sources[3]: f.unseenUser} + + // Written as a postgres deployment writes them, and as a bigquery one does. + postgresEmitter, err := business.NewDurableAuditEmitter(testStore, testStore) + require.NoError(t, err) + queuedEmitter, err := business.NewDurableAuditEmitter(testStore, testStore, business.WithQueuedRecords()) + require.NoError(t, err) + for _, entry := range entries { + for _, emitter := range []*business.DurableAuditEmitter{postgresEmitter, queuedEmitter} { + write := func(ctx context.Context) error { return emitter.EmitTx(ctx, entry) } + if entry.OrgID == "" { + require.NoError(t, testStore.WithControlPlane(ctx, write)) + } else { + require.NoError(t, testStore.WithOrgTx(ctx, entry.OrgID, write)) + } + } + } + + fake := bigqueryfake.New() + fake.CreateTable(bigqueryfake.TablePath(parityProject, parityDataset, bigquerystore.EventsTable), bigquerystore.EventsSchema()) + fake.CreateTable(bigqueryfake.TablePath(parityProject, parityDataset, bigquerystore.DetailsTable), bigquerystore.DetailsSchema()) + f.warehouse = &parityWarehouse{fake: fake} + pool, err := infra.NewAuditRelayPool(ctx) + require.NoError(t, err) + t.Cleanup(pool.Close) + queue, err := infra.NewPostgresAuditQueue(pool) + require.NoError(t, err) + relay, err := business.NewAuditRelay(business.AuditRelayConfig{ + Queue: queue, Store: f.warehouse, Archive: discardArchive{}, Types: testStore, + DeploymentID: parityDeployment, BatchSize: 4, MaxWait: time.Nanosecond, + }) + require.NoError(t, err) + delivered, err := relay.DrainOnce(ctx) + require.NoError(t, err) + require.GreaterOrEqual(t, delivered, len(entries)) + + // A relay that restarted between its writes and its queue delete delivers a + // batch again; the warehouse then holds those events twice. + require.GreaterOrEqual(t, len(f.warehouse.batches), 3) + for _, again := range []business.AuditBatch{f.warehouse.batches[0], f.warehouse.batches[len(f.warehouse.batches)/2]} { + again.ID = uuid.NewString() + require.NoError(t, f.warehouse.AppendAuditBatch(ctx, again)) + } + // Another deployment writing the same organization's events into the same + // tables is outside every read of this one. + foreign := f.warehouse.batches[1] + foreign.ID, foreign.DeploymentID = uuid.NewString(), "deployment-other" + require.NoError(t, f.warehouse.AppendAuditBatch(ctx, foreign)) + + client, err := bigquery.NewClient(ctx, parityProject, option.WithoutAuthentication(), option.WithEndpoint("http://127.0.0.1:1")) + require.NoError(t, err) + t.Cleanup(func() { _ = client.Close() }) + reader, err := bigquerystore.NewReader(bigquerystore.ReadConfig{ + Client: fake, Project: parityProject, Dataset: parityDataset, DeploymentID: parityDeployment, + }) + require.NoError(t, err) + f.store, err = bigquerystore.New(client, bigquerystore.Config{Dataset: parityDataset, ContentDetailRetention: 30 * 24 * time.Hour, Reader: reader}) + require.NoError(t, err) + + f.postgres, err = business.NewService(testStore) + require.NoError(t, err) + f.swapped, err = business.NewService(testStore) + require.NoError(t, err) + f.swapped.SetAuditStore(f.store) + return f +} + +func TestAuditStoreParityPostgresAndBigQuery(t *testing.T) { + f := newParityFixture(t) + ctx := testCtx + eventsPath := bigqueryfake.TablePath(parityProject, parityDataset, bigquerystore.EventsTable) + stored := map[string]int{} + for _, row := range f.warehouse.fake.Rows(eventsPath) { + stored[row["event_id"].(string)]++ + } + duplicated := 0 + for _, n := range stored { + if n > 1 { + duplicated++ + } + } + require.Positive(t, duplicated, "the warehouse holds events more than once") + + week := time.Now().Add(-7 * 24 * time.Hour) + recent := time.Now().Add(-3 * time.Hour) + queries := map[string]business.AuditQuery{ + "organization": {OrgID: f.org}, + "actor": {OrgID: f.org, ActorID: f.actor}, + "event type": {OrgID: f.org, EventType: string(business.EventDocumentRead)}, + "event type set": {OrgID: f.org, EventTypes: []string{string(business.EventAuthLogin), string(business.EventOrgUpdated)}}, + "type and set": {OrgID: f.org, EventType: string(business.EventAuthLogin), EventTypes: []string{string(business.EventOrgUpdated)}}, + "category": {OrgID: f.org, Category: "organization"}, + "namespace": {OrgID: f.org, Namespace: "saas"}, + "resource": {OrgID: f.org, Resource: "datasource", ResourceID: "doc-1"}, + "client": {OrgID: f.org, ClientID: "client-1"}, + "payload": {OrgID: f.org, PayloadContains: map[string]any{"boundary": "b-1"}}, + "payload number": {OrgID: f.org, PayloadContains: map[string]any{"result_count": 3.0}}, + "payload nested": {OrgID: f.org, PayloadContains: map[string]any{"nested": map[string]any{"a": "q"}, "tags": []any{"x"}}}, + "window": {OrgID: f.org, From: &week, To: &recent}, + "platform for actor": {ActorID: f.actor}, + "other organization": {OrgID: f.otherOrg}, + } + + t.Run("list", func(t *testing.T) { + // The fixture is what the comparisons compare: not two empty answers. + all, _, _, err := f.swapped.QueryAuditLog(ctx, business.AuditQuery{OrgID: f.org, PageSize: 100}) + require.NoError(t, err) + require.Len(t, all, 12) + for name, q := range queries { + for _, size := range []int32{0, 1, 3} { + q := q + q.PageSize = size + var pages int + for { + want, wantNext, wantTotal, err := f.postgres.QueryAuditLog(ctx, q) + require.NoError(t, err, name) + got, gotNext, gotTotal, err := f.swapped.QueryAuditLog(ctx, q) + require.NoError(t, err, name) + require.Equal(t, want, got, "%s, page size %d, page %d", name, size, pages) + require.Equal(t, wantNext, gotNext, name) + require.Equal(t, wantTotal, gotTotal, name) + pages++ + if wantNext == "" { + break + } + q.PageToken = wantNext + } + } + } + }) + + t.Run("aggregate", func(t *testing.T) { + specs := map[string]business.AuditAggregationSpec{ + "default": {}, + "category": {GroupBy: []string{"category"}}, + "actor": {GroupBy: []string{"actor"}, Metrics: []business.AuditMetric{{Op: "count"}, {Op: "count_distinct", Field: "resource_id"}}}, + "day": {GroupBy: []string{"time"}, Bucket: "day"}, + "week": {GroupBy: []string{"time"}, Bucket: "week", Metrics: []business.AuditMetric{{Op: "count_distinct", Field: "actor_id"}}}, + "month": {GroupBy: []string{"time"}, Bucket: "month"}, + "payload dimensions": {GroupBy: []string{"event_type", "payload:outcome"}, Metrics: []business.AuditMetric{ + {Op: "count_distinct", Field: "payload:boundary"}, + {Op: "count_distinct", Field: "category"}, + {Op: "count_distinct", Field: "event_type"}, + {Op: "count_distinct", Field: "resource"}, + }}, + "nested payload dimension": {GroupBy: []string{"payload:tags", "payload:nested", "payload:label"}}, + "metrics": {GroupBy: []string{"payload:boundary"}, Metrics: []business.AuditMetric{ + {Op: "count", Alias: "reads"}, + {Op: "sum", Field: "payload:duration_ms"}, + {Op: "avg", Field: "payload:duration_ms"}, + {Op: "min", Field: "payload:duration_ms"}, + {Op: "max", Field: "payload:duration_ms"}, + {Op: "percentile", Field: "payload:duration_ms", Percentile: 0.5, Alias: "p50"}, + {Op: "percentile", Field: "payload:duration_ms", Percentile: 0.9, Alias: "p90"}, + {Op: "sum", Field: "payload:result_count", Alias: "results"}, + {Op: "max", Field: "payload:missing", Alias: "absent"}, + }, Derived: []business.AuditDerivedMetric{ + {Alias: "results_per_read", Numerator: "results", Denominator: "reads"}, + {Alias: "absent_ratio", Numerator: "absent", Denominator: "reads"}, + }}, + } + for queryName, q := range queries { + for specName, spec := range specs { + want, err := f.postgres.AggregateAuditLog(ctx, q, spec) + require.NoError(t, err, "%s / %s", queryName, specName) + if queryName == "organization" { + require.NotEmpty(t, want, specName) + } + got, err := f.swapped.AggregateAuditLog(ctx, q, spec) + require.NoError(t, err, "%s / %s", queryName, specName) + require.Equal(t, want, got, "%s / %s", queryName, specName) + } + } + }) + + t.Run("export", func(t *testing.T) { + for _, format := range []string{"json", "csv"} { + for _, export := range []struct { + org, actor, eventType string + eventTypes []string + }{ + {org: f.org}, + {org: f.org, eventType: string(business.EventDocumentRead)}, + {org: f.org, eventTypes: []string{string(business.EventAuthLogin)}}, + {actor: f.actor}, + } { + want, wantType, _, err := f.postgres.ExportAuditLog(ctx, export.org, format, export.actor, export.eventType, export.eventTypes) + require.NoError(t, err) + got, gotType, _, err := f.swapped.ExportAuditLog(ctx, export.org, format, export.actor, export.eventType, export.eventTypes) + require.NoError(t, err) + require.Equal(t, wantType, gotType) + require.Greater(t, len(want), 200, "%s export of %+v holds events", format, export) + require.Equal(t, string(want), string(got), "%s export of %+v", format, export) + } + } + }) + + t.Run("readable sources", func(t *testing.T) { + verified := auth.WithVerifiedDatabaseIdentity(ctx, f.sourceReader, f.sourceOrg) + asked := append([]string{uuid.NewString()}, f.sources...) + require.NoError(t, testStore.WithSourceReadSnapshot(verified, f.sourceOrg, func(snapshot context.Context) error { + want, err := f.postgres.LatestSourceSyncRequests(snapshot, f.sourceOrg, asked) + require.NoError(t, err) + got, err := f.swapped.LatestSourceSyncRequests(snapshot, f.sourceOrg, asked) + require.NoError(t, err) + require.Equal(t, want, got) + require.Len(t, got, len(f.sources), "a source never synced has no request") + require.Equal(t, "parity-member@test.com", got[f.sources[0]].RequestedBy, "a visible principal is named") + require.Equal(t, f.other, got[f.sources[1]].RequestedBy, "an id with no principal names itself") + require.Equal(t, "", got[f.sources[2]].RequestedBy, "a request with no actor names no one") + require.Equal(t, f.unseenUser, got[f.sources[3]].RequestedBy, "a principal outside the organization is not disclosed") + return nil + })) + }) + + t.Run("scope", func(t *testing.T) { + _, _, err := f.store.ListAuditEvents(ctx, business.AuditRead{Query: business.AuditQuery{OrgID: f.org}}) + require.ErrorIs(t, err, business.ErrAuditReadUnscoped) + _, err = f.store.AggregateAuditEvents(ctx, business.AuditRead{}, business.AuditAggregationSpec{}) + require.ErrorIs(t, err, business.ErrAuditReadUnscoped) + _, err = f.store.ExportAuditEvents(ctx, business.AuditRead{}) + require.ErrorIs(t, err, business.ErrAuditReadUnscoped) + _, err = f.store.LatestSourceSyncEvents(ctx, business.AuditReadScope{}, f.sources) + require.ErrorIs(t, err, business.ErrAuditReadUnscoped) + _, _, err = f.store.ListAuditEvents(ctx, business.AuditRead{ + Scope: business.OrganizationAuditScope(f.org), Query: business.AuditQuery{OrgID: f.otherOrg}, + }) + require.Error(t, err, "a query naming another organization than its scope is refused") + + // A resource read is authorized in Postgres before the warehouse is read; + // a reader without the grant reads nothing from it. + before := len(f.warehouse.fake.Sessions()) + _, err = f.swapped.AggregateAuditLogForReader(ctx, f.actor, business.AuditQuery{ + OrgID: f.org, Resource: "datasource", ResourceID: uuid.NewString(), EventType: string(business.EventDatasourceSourceSynced), + }, business.AuditAggregationSpec{}) + require.Equal(t, codes.PermissionDenied, status.Code(err), fmt.Sprint(err)) + require.Len(t, f.warehouse.fake.Sessions(), before, "a refused read never reaches the warehouse") + }) +} diff --git a/module/services/accounts/code/pkg/business/audit_store_read.go b/module/services/accounts/code/pkg/business/audit_store_read.go new file mode 100644 index 000000000..3a9f13693 --- /dev/null +++ b/module/services/accounts/code/pkg/business/audit_store_read.go @@ -0,0 +1,281 @@ +package business + +import ( + "context" + "encoding/base64" + "errors" + "fmt" + "strings" + "time" +) + +// The read half of the audit store (ADR 0009). Under AUDIT_SINK=postgres and +// AUDIT_SINK=both the activity list, aggregation, export and readable-source +// queries read audit_events under row-level security, exactly as before. Under +// a swap value they read the store of record instead, which row-level security +// does not reach: every read names its scope explicitly (AuditReadScope), and +// the store refuses one that names none. + +// ErrAuditReadUnscoped is the refusal of a read that names no scope. +var ErrAuditReadUnscoped = errors.New("audit read: no scope; a read names its organization, or is explicitly a platform read") + +// AuditReadScope is whose events a read may return. It is either one +// organization's events, or — explicitly — the platform's view, which spans +// every organization and the platform's own events (no organization). The zero +// value is no scope, and every store refuses it. +// +// The platform scope is exactly what Postgres serves under WithControlPlane +// today: a read whose query names no organization, which the transport admits +// only for a platform admin. +type AuditReadScope struct { + orgID string + platform bool +} + +// OrganizationAuditScope scopes a read to one organization's events. +func OrganizationAuditScope(orgID string) AuditReadScope { + return AuditReadScope{orgID: orgID} +} + +// PlatformAuditScope is the explicit platform read: every organization's events +// and the platform's own. Only the platform-admin path takes it. +func PlatformAuditScope() AuditReadScope { + return AuditReadScope{platform: true} +} + +// OrgID is the organization an organization-scoped read is confined to; empty +// for the platform scope. +func (s AuditReadScope) OrgID() string { return s.orgID } + +// Platform reports whether the read is the explicit platform read. +func (s AuditReadScope) Platform() bool { return s.platform } + +// Validate refuses the zero scope. +func (s AuditReadScope) Validate() error { + if s.platform == (s.orgID != "") { + return ErrAuditReadUnscoped + } + return nil +} + +// String names the scope for errors and logs. +func (s AuditReadScope) String() string { + switch { + case s.platform: + return "platform" + case s.orgID != "": + return "organization " + s.orgID + default: + return "unscoped" + } +} + +// auditReadScopeFor is the scope of a Service read of q: its organization, or +// the platform scope when it names none — the read Postgres serves under +// WithControlPlane, which the transport admits only for a platform admin. +func auditReadScopeFor(q AuditQuery) AuditReadScope { + if q.OrgID == "" { + return PlatformAuditScope() + } + return OrganizationAuditScope(q.OrgID) +} + +// AuditEventTypeFacts is what a read needs to know about an event type it does +// not hold itself: the category and namespace its audit_event_types row names. +type AuditEventTypeFacts struct { + Category string + Namespace string +} + +// AuditEventTypeIndex is the audit_event_types projection by type name — the +// registry's code-owned and declared types alike. In Postgres the category and +// namespace filters and the category dimension read that table in SQL; a +// store of record outside Postgres takes the same facts from this index, which +// the service reads from the same table. +type AuditEventTypeIndex map[string]AuditEventTypeFacts + +// NewAuditEventTypeIndex indexes audit_event_types rows. +func NewAuditEventTypeIndex(rows []AuditEventTypeRow) AuditEventTypeIndex { + index := make(AuditEventTypeIndex, len(rows)) + for _, row := range rows { + index[row.Name] = AuditEventTypeFacts{Category: row.Category, Namespace: row.Namespace} + } + return index +} + +// AuditRead is one read of the store of record: whose events, which ones, and +// the event-type facts the query needs. +type AuditRead struct { + Scope AuditReadScope + Query AuditQuery + // Types is set when the query filters on a category or namespace, or the + // aggregation groups or counts by category. A store that needs it and finds + // it nil refuses the read rather than treating every type as unknown. + Types AuditEventTypeIndex +} + +// Validate checks the read's scope and that the query does not reach past it: +// an organization-scoped read's query names that organization, and a platform +// read's query names none. +func (r AuditRead) Validate() error { + if err := r.Scope.Validate(); err != nil { + return err + } + if r.Query.OrgID != r.Scope.orgID { + return fmt.Errorf("audit read: query organization %q is outside the %s scope", r.Query.OrgID, r.Scope) + } + if r.Query.CollectionID != "" { + return errors.New("uncompiled collection filter: reader authorization required") + } + return nil +} + +// AuditReader is the read half of AuditStore: the activity list, aggregation and +// export. Every read returns each event once by event id, whatever the store +// holds after a redelivery. +type AuditReader interface { + // ListAuditEvents returns one page, newest first — by occurrence, then by + // event id, both descending — and the token of the next page, empty after + // the last. A zero page size is 50, as it is in Postgres. + ListAuditEvents(ctx context.Context, read AuditRead) ([]AuditEntry, string, error) + // AggregateAuditEvents groups the matching events per spec, which the caller + // has validated. + AggregateAuditEvents(ctx context.Context, read AuditRead, spec AuditAggregationSpec) ([]AuditAggregateBucket, error) + // ExportAuditEvents returns every matching event, newest first. + ExportAuditEvents(ctx context.Context, read AuditRead) ([]AuditEntry, error) +} + +// AuditSourceSyncEvent is the newest "a sync was requested" event of one +// source: when, and the actor id the event names (empty when it names none). +type AuditSourceSyncEvent struct { + RequestedAt time.Time + ActorID string +} + +// AuditSourceSyncReader is the readable-source query of the store of record +// (ADR 0008 keeps a sync's actor in the audit trail alone). Postgres answers it +// in one query joined to principals inside the source-read snapshot +// (Store.LatestSourceSyncRequests); a store outside Postgres returns the actor +// ids, and the service joins their display names from Postgres. +type AuditSourceSyncReader interface { + // LatestSourceSyncEvents returns, per source that has one, its newest + // EventDatasourceSourceSynced event in the organization the scope names. + LatestSourceSyncEvents(ctx context.Context, scope AuditReadScope, sources []string) (map[string]AuditSourceSyncEvent, error) +} + +// EncodeAuditPageToken / DecodeAuditPageToken carry the keyset position of the +// activity list: the last returned event's (occurrence, id). The token is +// opaque to callers — base64 of "|" — and every store reads +// the same token, so a page boundary means the same thing in all of them. +func EncodeAuditPageToken(occurredAt time.Time, id string) string { + raw := occurredAt.UTC().Format(time.RFC3339Nano) + "|" + id + return base64.RawURLEncoding.EncodeToString([]byte(raw)) +} + +// DecodeAuditPageToken reads a token EncodeAuditPageToken wrote. +func DecodeAuditPageToken(token string) (time.Time, string, error) { + raw, err := base64.RawURLEncoding.DecodeString(token) + if err != nil { + return time.Time{}, "", err + } + parts := strings.SplitN(string(raw), "|", 2) + if len(parts) != 2 { + return time.Time{}, "", fmt.Errorf("malformed cursor") + } + occurredAt, err := time.Parse(time.RFC3339Nano, parts[0]) + if err != nil { + return time.Time{}, "", err + } + return occurredAt, parts[1], nil +} + +// auditReadNeedsTypes reports whether a read of q, aggregated per spec when it +// is not nil, needs the event-type index. +func auditReadNeedsTypes(q AuditQuery, spec *AuditAggregationSpec) bool { + if q.Category != "" || q.Namespace != "" { + return true + } + if spec == nil { + return false + } + for _, dimension := range spec.GroupBy { + if dimension == "category" { + return true + } + } + for _, metric := range spec.Metrics { + if metric.Op == "count_distinct" && metric.Field == "category" { + return true + } + } + return false +} + +// postgresAuditReader is the Postgres implementation of AuditReader — the reads +// every postgres and both deployment has always made, and the reference the +// other stores are measured against. Each read runs in its own transaction: the +// organization's (row-level security admits its rows) or, for the platform +// scope, the control plane's (which spans tenants). Postgres reads the +// event-type facts itself, in SQL, so it ignores AuditRead.Types. +type postgresAuditReader struct { + store Store +} + +// NewPostgresAuditReader is the Postgres reference reader over store. +func NewPostgresAuditReader(store Store) AuditReader { + return postgresAuditReader{store: store} +} + +func (r postgresAuditReader) within(ctx context.Context, scope AuditReadScope, fn func(context.Context) error) error { + if scope.Platform() { + return r.store.WithControlPlane(ctx, fn) + } + return r.store.WithOrgTx(ctx, scope.OrgID(), fn) +} + +func (r postgresAuditReader) ListAuditEvents(ctx context.Context, read AuditRead) ([]AuditEntry, string, error) { + if err := read.Validate(); err != nil { + return nil, "", err + } + var entries []AuditEntry + var nextToken string + err := r.within(ctx, read.Scope, func(ctx context.Context) error { + ev, nt, _, err := r.store.QueryAuditLog(ctx, read.Query) + entries, nextToken = ev, nt + return err + }) + return entries, nextToken, err +} + +func (r postgresAuditReader) AggregateAuditEvents(ctx context.Context, read AuditRead, spec AuditAggregationSpec) ([]AuditAggregateBucket, error) { + if err := read.Validate(); err != nil { + return nil, err + } + var out []AuditAggregateBucket + err := r.within(ctx, read.Scope, func(ctx context.Context) error { + var err error + out, err = r.store.AggregateAuditLog(ctx, read.Query, spec) + return err + }) + return out, err +} + +// ExportAuditEvents pages the list 100 events at a time, each page in its own +// transaction, until it runs out. +func (r postgresAuditReader) ExportAuditEvents(ctx context.Context, read AuditRead) ([]AuditEntry, error) { + var all []AuditEntry + page := read + page.Query.PageSize = 100 + page.Query.PageToken = "" + for { + entries, nextToken, err := r.ListAuditEvents(ctx, page) + if err != nil { + return nil, err + } + all = append(all, entries...) + if nextToken == "" || len(entries) == 0 { + return all, nil + } + page.Query.PageToken = nextToken + } +} diff --git a/module/services/accounts/code/pkg/business/readable_sources.go b/module/services/accounts/code/pkg/business/readable_sources.go index 90c29e400..e58223c8c 100644 --- a/module/services/accounts/code/pkg/business/readable_sources.go +++ b/module/services/accounts/code/pkg/business/readable_sources.go @@ -164,7 +164,7 @@ func (s *Service) discloseCollectionMetadata(ctx context.Context, org string, re } } if disclosure.SyncRequester { - requests, err := s.store.LatestSourceSyncRequests(ctx, org, sources) + requests, err := s.LatestSourceSyncRequests(ctx, org, sources) if err != nil { return err } @@ -179,3 +179,43 @@ func (s *Service) discloseCollectionMetadata(ctx context.Context, org string, re } return nil } + +// LatestSourceSyncRequests returns, per source, when a sync was last requested +// and by whom — read from the audit store of record, inside the caller's +// source-read snapshot. Under postgres and both it is one query over +// audit_events joined to principals within the snapshot. Under a swap value +// (ADR 0009) the store of record returns each source's newest request and its +// actor id, then the actors' display names are joined from principals within +// the snapshot — the audit read itself is outside it. Either way an actor with +// no visible principal is labelled by its id, and an event with no actor by "". +func (s *Service) LatestSourceSyncRequests(ctx context.Context, org string, sources []string) (map[string]SourceSyncRequest, error) { + if s.auditStore == nil { + return s.store.LatestSourceSyncRequests(ctx, org, sources) + } + events, err := s.auditStore.LatestSourceSyncEvents(ctx, OrganizationAuditScope(org), sources) + if err != nil { + return nil, err + } + var actors []string + for _, event := range events { + if event.ActorID != "" && !slices.Contains(actors, event.ActorID) { + actors = append(actors, event.ActorID) + } + } + slices.Sort(actors) + labels := map[string]string{} + if len(actors) > 0 { + if labels, err = s.store.SourceSyncRequesterLabels(ctx, org, actors); err != nil { + return nil, err + } + } + out := make(map[string]SourceSyncRequest, len(events)) + for source, event := range events { + label, ok := labels[event.ActorID] + if !ok { + label = event.ActorID + } + out[source] = SourceSyncRequest{RequestedAt: event.RequestedAt, RequestedBy: label} + } + return out, nil +} diff --git a/module/services/accounts/code/pkg/business/service.go b/module/services/accounts/code/pkg/business/service.go index 14d1f1e53..92d41fd00 100644 --- a/module/services/accounts/code/pkg/business/service.go +++ b/module/services/accounts/code/pkg/business/service.go @@ -35,6 +35,7 @@ type Service struct { orgCreationPolicy OrganizationCreationPolicy audit AuditEmitter auditTx TxAuditEmitter // set when audit also writes on the caller's tx; required in production + auditStore AuditStore // the store of record's reads under a swap value (ADR 0009); nil reads audit_events entitlements EntitlementChecker membership MembershipInvalidator slack *SlackNotifier // optional: sends critical notifications to Slack @@ -387,6 +388,23 @@ func (s *Service) SetAuditEmitter(a AuditEmitter) { s.auditTx, _ = a.(TxAuditEmitter) } +// SetAuditStore routes every audit read — the activity list, aggregation, +// export and the readable-source query — to the store of record of a swap value +// (ADR 0009). Without it, reads go to audit_events, as they do under postgres +// and both. +func (s *Service) SetAuditStore(store AuditStore) { + s.auditStore = store +} + +// auditReads is the reader of the audit store of record: the swap value's +// store when one is wired, Postgres otherwise. +func (s *Service) auditReads() AuditReader { + if s.auditStore != nil { + return s.auditStore + } + return postgresAuditReader{store: s.store} +} + // VerifyAuditWiring fails startup when the wired audit emitter cannot write on // the caller's transaction. Every security mutation commits its audit row and // webhook fan-out inside its own transaction; an emitter without EmitTx would diff --git a/module/services/accounts/code/pkg/business/store.go b/module/services/accounts/code/pkg/business/store.go index d183b3f8d..5c53d9543 100644 --- a/module/services/accounts/code/pkg/business/store.go +++ b/module/services/accounts/code/pkg/business/store.go @@ -104,6 +104,12 @@ type Store interface { ListReadableSourcesPage(context.Context, string, []string, []string, string, int) ([]*gen.ReadableSourceCollection, error) ReadableCollectionGrants(context.Context, string, []string, []string) (map[string][]*gen.ReadableCollectionGrant, error) LatestSourceSyncRequests(context.Context, string, []string) (map[string]SourceSyncRequest, error) + // SourceSyncRequesterLabels names, inside the same source-read snapshot, the + // actors a swap value's store of record returned for LatestSourceSyncEvents: + // each visible principal's display name by id. An id without a visible + // principal is absent, and the caller labels it by the id itself, exactly as + // LatestSourceSyncRequests does. + SourceSyncRequesterLabels(ctx context.Context, org string, actorIDs []string) (map[string]string, error) ListDatasourceSources(ctx context.Context, orgID string) ([]*DatasourceSource, error) GetDatasourceSource(ctx context.Context, orgID, id string) (*DatasourceSource, error) // DeleteDatasourceSource removes the Source and returns what it removed, diff --git a/module/services/accounts/code/pkg/infra/postgres_audit.go b/module/services/accounts/code/pkg/infra/postgres_audit.go index 514db89bf..5453d3af7 100644 --- a/module/services/accounts/code/pkg/infra/postgres_audit.go +++ b/module/services/accounts/code/pkg/infra/postgres_audit.go @@ -2,7 +2,6 @@ package infra import ( "context" - "encoding/base64" "encoding/json" "fmt" "strings" @@ -106,29 +105,6 @@ func auditEventColumns(entry business.AuditEntry) []any { } } -// encodeAuditCursor / decodeAuditCursor carry the keyset position for audit -// pagination. The token is opaque to callers: base64 of "|". -func encodeAuditCursor(ct time.Time, id string) string { - raw := ct.UTC().Format(time.RFC3339Nano) + "|" + id - return base64.RawURLEncoding.EncodeToString([]byte(raw)) -} - -func decodeAuditCursor(token string) (time.Time, string, error) { - raw, err := base64.RawURLEncoding.DecodeString(token) - if err != nil { - return time.Time{}, "", err - } - parts := strings.SplitN(string(raw), "|", 2) - if len(parts) != 2 { - return time.Time{}, "", fmt.Errorf("malformed cursor") - } - ct, err := time.Parse(time.RFC3339Nano, parts[0]) - if err != nil { - return time.Time{}, "", err - } - return ct, parts[1], nil -} - // auditWhere builds the shared WHERE clause for the search and aggregate paths // from an AuditQuery, returning the SQL fragment and its ordered args. func auditWhere(q business.AuditQuery, startArg int) (string, []any, error) { @@ -218,7 +194,7 @@ func (s *PostgresStore) QueryAuditLog(ctx context.Context, q business.AuditQuery // (created_at, id); the compound comparison is stable under the DESC // ordering even when many rows share a created_at. if q.PageToken != "" { - ct, id, err := decodeAuditCursor(q.PageToken) + ct, id, err := business.DecodeAuditPageToken(q.PageToken) if err != nil { return nil, "", 0, fmt.Errorf("invalid page token: %w", err) } @@ -291,7 +267,7 @@ func (s *PostgresStore) QueryAuditLog(ctx context.Context, q business.AuditQuery if len(events) > int(pageSize) { events = events[:pageSize] last := events[len(events)-1] - nextToken = encodeAuditCursor(last.CreatedAt, last.ID) + nextToken = business.EncodeAuditPageToken(last.CreatedAt, last.ID) } return events, nextToken, int32(len(events)), nil diff --git a/module/services/accounts/code/pkg/infra/postgres_readable_sources.go b/module/services/accounts/code/pkg/infra/postgres_readable_sources.go index c20ce3bf4..51091a394 100644 --- a/module/services/accounts/code/pkg/infra/postgres_readable_sources.go +++ b/module/services/accounts/code/pkg/infra/postgres_readable_sources.go @@ -285,3 +285,42 @@ func (s *PostgresStore) LatestSourceSyncRequests(ctx context.Context, org string } return out, rows.Err() } + +// SourceSyncRequesterLabels implements business.Store: the display name of each +// actor id whose principal this snapshot can see. It is the principals half of +// LatestSourceSyncRequests' join, for a store of record outside Postgres. +func (s *PostgresStore) SourceSyncRequesterLabels(ctx context.Context, org string, actorIDs []string) (map[string]string, error) { + snapshot, ok := ctx.Value(sourceReadSnapshotKey{}).(sourceReadSnapshot) + if !ok { + return nil, fmt.Errorf("source read snapshot required") + } + if snapshot.orgID != org { + return nil, fmt.Errorf("source read snapshot is bound to another organization") + } + tx := snapshot.tx + // The ids come from the store of record, which keeps them as text; one that + // is not a uuid names no principal, and is labelled by itself. + var ids []string + for _, id := range actorIDs { + if nilIfNotUUID(id) != nil { + ids = append(ids, id) + } + } + labels := make(map[string]string, len(ids)) + if len(ids) == 0 { + return labels, nil + } + rows, err := tx.Query(ctx, `SELECT id::text, display_name FROM principals WHERE id = ANY($1::uuid[])`, ids) + if err != nil { + return nil, err + } + defer rows.Close() + for rows.Next() { + var id, label string + if err := rows.Scan(&id, &label); err != nil { + return nil, err + } + labels[id] = label + } + return labels, rows.Err() +} diff --git a/module/services/accounts/code/pkg/infra/testdata/control_plane_call_sites.txt b/module/services/accounts/code/pkg/infra/testdata/control_plane_call_sites.txt index f0df8f0fc..506b51bbc 100644 --- a/module/services/accounts/code/pkg/infra/testdata/control_plane_call_sites.txt +++ b/module/services/accounts/code/pkg/infra/testdata/control_plane_call_sites.txt @@ -27,8 +27,7 @@ pkg/auth/pg/session_store.go: (*SessionStore).FindByRefreshHash: WithControlPlan pkg/auth/pg/session_store.go: (*SessionStore).RevokeFamily: WithControlPlane pkg/auth/pg/session_store.go: (*SessionStore).RotateRefresh: WithControlPlane pkg/business/audit.go: (*DurableAuditEmitter).Emit: WithControlPlane -pkg/business/audit.go: (*Service).AggregateAuditLog: WithControlPlane -pkg/business/audit.go: (*Service).QueryAuditLog: WithControlPlane +pkg/business/audit_store_read.go: (postgresAuditReader).within: WithControlPlane pkg/business/connector_credentials.go: (*Service).SourceSigningSecret: WithControlPlane pkg/business/datasource_budget.go: (datasourceScheduler).Acquire: WithControlPlane pkg/business/datasource_budget.go: (datasourceScheduler).Blocked: WithControlPlane diff --git a/module/services/accounts/code/work.go b/module/services/accounts/code/work.go index 94349f4be..cc27ecc48 100644 --- a/module/services/accounts/code/work.go +++ b/module/services/accounts/code/work.go @@ -573,12 +573,15 @@ func doWork(ctx context.Context) (Clean, error) { var auditRelay *business.AuditRelay closeAuditRelay := func() {} if auditSink.bigQuery != nil { - w.Warn("AUDIT_SINK=bigquery: audit events are queued in Postgres and relayed to BigQuery and the archive; " + - "the activity list, aggregates, exports and readable-source queries still read audit_events, which receives no new rows, until reads move to the store (ADR 0009)") - auditRelay, closeAuditRelay, err = newBigQueryAuditRelay(ctx, store, auditSink.bigQuery) + // Under the swap BigQuery is the store of record for writes and reads + // alike: the activity list, aggregates, exports and the readable-source + // query read it, scoped to the caller's organization and this deployment. + var auditStore business.AuditStore + auditRelay, auditStore, closeAuditRelay, err = newBigQueryAuditSwap(ctx, store, auditSink.bigQuery) if err != nil { return nil, err } + service.SetAuditStore(auditStore) } var auditExportWorker *jobs.Worker From 21feb102c3bba8c492463df12b30f397d8f5fc65 Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Sat, 3 Oct 2026 20:16:37 -0400 Subject: [PATCH 08/69] docs(adr): settle how ADR 0009 reads BigQuery The service that reads the store of record also appends to it, so it holds bigquery.tables.updateData; with bigquery.jobs.create beside it, that identity could run DELETE, UPDATE and MERGE. ADR 0009 now decides that BigQuery reads run no query job: they use the Storage Read API with a server-side row restriction (deployment, organization, time window, event types), and the service sorts, pages, deduplicates and aggregates. It names the read grants, and rejects a second, query-capable identity for reads. Co-Authored-By: Claude Opus 5.5 --- module/docs/adr/0009-audit-store-swap.md | 27 ++++++++++++++++++++++-- 1 file changed, 25 insertions(+), 2 deletions(-) diff --git a/module/docs/adr/0009-audit-store-swap.md b/module/docs/adr/0009-audit-store-swap.md index dc34378ea..614fa89b3 100644 --- a/module/docs/adr/0009-audit-store-swap.md +++ b/module/docs/adr/0009-audit-store-swap.md @@ -126,6 +126,22 @@ Postgres transaction. read half deduplicates exactly as every archive reader does (item 5), and the per-event hash makes the copies provably identical. + BigQuery reads run no query job. They go through the BigQuery Storage Read + API. Each read session carries a server-side row restriction: the + deployment, the caller's organization (unless the read is the platform + read), the time window, so partitions outside it are pruned, and the event + types when the read names them. The service then sorts, pages, + deduplicates, joins content details and aggregates the rows it streams back, + with the semantics of the Postgres reads. + + The reason is the grants. The service that reads is the service that + appends, so it holds `bigquery.tables.updateData`. With + `bigquery.jobs.create` beside it, that identity could run DELETE, UPDATE and + MERGE against the store of record. So the identity is granted + `bigquery.datasets.get`, `bigquery.tables.get`, `bigquery.tables.getData`, + `bigquery.tables.list`, and `bigquery.readsessions.create`, `.getData` and + `.update`, and no job permission at all. + 5. **Retention tiers.** Every registered event type has a **retention class** in the typed registry (ADR 0003): - **security** — authentication, failed authentication, permission and role @@ -214,6 +230,12 @@ Where the adapters live (item 3): | A separate warehouse service the kit calls | One more deployable per environment, a second queue in front of it, a second permission check, and every read becomes a network hop. Worth it only if producers or readers outside the kit needed the store directly; none do. | | One managed warehouse per cloud (BigQuery, Redshift, Synapse) | One adapter per cloud to build and hold at parity. ClickHouse covers every non-Google environment, on-premises included, with one adapter. | +How BigQuery is read (item 4): + +| Alternative | Rejected because | +| --- | --- | +| A second, query-capable identity for reads | Adds a second principal to grant, rotate and audit. And the process that appends would then hold job creation through it too: the DML risk the append-only invariant exists to prevent. | + The sink shape itself: | Alternative | Rejected because | @@ -245,8 +267,9 @@ The sink shape itself: registry in the service instead. - `pkg/infra/postgres_readable_sources.go` — `LatestSourceSyncRequests` joins `audit_events` to `principals` in one query, inside the source-read snapshot. - Under a swap value it becomes a warehouse query for the latest actor ids, then - a join to display names in the service from Postgres — outside that snapshot. + Under a swap value it becomes a warehouse read of each source's latest actor + id — outside that snapshot — then a join to display names from Postgres, + inside it. - `pkg/business/audit_registry.go` — `AuditEventDefinition` gains the retention class, required the way `Durability` is. A declared type's class comes from its declaration — the manifest event, or the module declaration message — From 58ceb093922d3d009158918bf910fa886b7644e7 Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Sat, 3 Oct 2026 20:24:42 -0400 Subject: [PATCH 09/69] refactor(audit): share the sink settings, read back a store's window, and run parity over every store Groundwork for the history copy and the ClickHouse adapter of ADR 0009 (#996), with no change in behaviour. The audit sink settings move from the accounts main package into pkg/auditstore/auditsink: Load reads AUDIT_SINK and its mode's settings through a getenv, and Open builds a swap value's store of record (tables ensured, read half on the Storage Read API) and archive. The service and a history-copy command can then read one configuration. The settings, their validation and their messages are unchanged, and so are the tests, which move with them. business.AuditHistoryReader reads back every copy a store of record holds of this deployment's events in a time window, duplicates included, with the details each copy carries. The BigQuery store implements it with one Storage Read API session per table, as every other read. The read parity test becomes a suite over every store of record: the relay fans each batch, its redeliveries and another deployment's copy out to each store under test, and the refused-read check counts reads that reach the store instead of the fake's sessions. Co-Authored-By: Claude Opus 5.5 --- module/services/accounts/code/audit_sink.go | 263 ++-------------- .../pkg/auditstore/auditsink/auditsink.go | 297 ++++++++++++++++++ .../auditstore/auditsink/auditsink_test.go} | 73 ++--- .../auditstore/bigquerystore/bigquerystore.go | 13 +- .../pkg/auditstore/bigquerystore/reader.go | 62 ++++ .../code/pkg/business/audit_history.go | 29 ++ .../pkg/business/audit_store_parity_test.go | 222 +++++++++---- module/services/accounts/code/work.go | 19 +- 8 files changed, 621 insertions(+), 357 deletions(-) create mode 100644 module/services/accounts/code/pkg/auditstore/auditsink/auditsink.go rename module/services/accounts/code/{work_audit_test.go => pkg/auditstore/auditsink/auditsink_test.go} (72%) create mode 100644 module/services/accounts/code/pkg/business/audit_history.go diff --git a/module/services/accounts/code/audit_sink.go b/module/services/accounts/code/audit_sink.go index 9142b7e99..142ef52fe 100644 --- a/module/services/accounts/code/audit_sink.go +++ b/module/services/accounts/code/audit_sink.go @@ -1,206 +1,30 @@ package main import ( - "accounts/pkg/auditstore/bigquerystore" - "accounts/pkg/auditstore/gcsarchive" + "accounts/pkg/auditstore/auditsink" "accounts/pkg/business" "accounts/pkg/infra" "context" "fmt" - "net/url" "os" - "regexp" - "strconv" - "strings" - "time" - - "cloud.google.com/go/bigquery" - bqstorage "cloud.google.com/go/bigquery/storage/apiv1" - "cloud.google.com/go/storage" -) - -// The audit sink settings. AUDIT_SINK selects the mode (business.AuditSinkMode); -// each mode's settings are required only under it, and a missing one fails -// startup rather than degrading the audit trail. -const ( - envAuditSink = "AUDIT_SINK" - - // AUDIT_SINK=both (ADR 0006). - envAuditExternalURL = "AUDIT_EXTERNAL_URL" - envAuditExternalToken = "AUDIT_EXTERNAL_TOKEN" - - // AUDIT_SINK=bigquery (ADR 0009). Credentials are Application Default - // Credentials only — on Kubernetes, the pod's workload identity — so there - // is deliberately no key-file or token setting. - envAuditBigQueryProject = "AUDIT_BIGQUERY_PROJECT" - envAuditBigQueryDataset = "AUDIT_BIGQUERY_DATASET" - envAuditArchiveURL = "AUDIT_ARCHIVE_URL" - envAuditDeploymentID = "AUDIT_DEPLOYMENT_ID" - envAuditContentRetentionDays = "AUDIT_CONTENT_RETENTION_DAYS" - // Optional relay tuning. - envAuditRelayBatchSize = "AUDIT_RELAY_BATCH_SIZE" - envAuditRelayMaxWait = "AUDIT_RELAY_MAX_WAIT" ) -// auditSinkConfig is the resolved audit sink: the mode and, for the modes that -// write outside Postgres, where. -type auditSinkConfig struct { - mode business.AuditSinkMode - // external is the tee's destination under AUDIT_SINK=both. - external business.ExternalAuditSink - // bigQuery is the store of record and archive under AUDIT_SINK=bigquery. - bigQuery *bigQueryAuditSwap +// configuredAuditSink reads AUDIT_SINK and the settings its mode requires +// (auditsink.Load). +func configuredAuditSink() (auditsink.Config, error) { + return auditsink.Load(os.Getenv) } -// configuredAuditSink reads AUDIT_SINK and the settings its mode requires. -// "postgres" (the default) keeps the durable emitter alone. "both" adds an -// asynchronous tee to an HTTP endpoint while audit_events stays the store of -// record. "bigquery" swaps the store of record to BigQuery, with Postgres -// keeping only the transactional queue. "external" is refused: no destination -// outside Postgres can join the transaction a change commits in. -func configuredAuditSink() (auditSinkConfig, error) { - mode, err := business.ParseAuditSinkMode(os.Getenv(envAuditSink)) +// newAuditSwap builds a swap value (ADR 0009): its store of record and archive +// (auditsink.Open), and the relay that drains the queue into them on the +// relay's own pool. The service reads the returned store; the returned close +// releases every client. +func newAuditSwap(ctx context.Context, types business.DeclaredAuditEventTypeReader, swap *auditsink.Swap) (*business.AuditRelay, business.AuditStore, func(), error) { + opened, err := auditsink.Open(ctx, swap) if err != nil { - return auditSinkConfig{}, err - } - config := auditSinkConfig{mode: mode} - switch mode { - case business.AuditSinkBoth: - config.external, err = configuredExternalAuditSink() - case business.AuditSinkBigQuery: - config.bigQuery, err = configuredBigQueryAuditSwap() - } - if err != nil { - return auditSinkConfig{}, err - } - return config, nil -} - -func configuredExternalAuditSink() (business.ExternalAuditSink, error) { - endpoint := strings.TrimSpace(os.Getenv(envAuditExternalURL)) - if endpoint == "" { - return nil, fmt.Errorf("AUDIT_SINK=both requires %s for the external audit destination", envAuditExternalURL) - } - return business.NewHTTPAuditSink(business.HTTPAuditSinkConfig{ - Endpoint: endpoint, - Token: strings.TrimSpace(os.Getenv(envAuditExternalToken)), - }) -} - -// bigQueryAuditSwap is the validated AUDIT_SINK=bigquery configuration. -type bigQueryAuditSwap struct { - project string - dataset string - archive auditArchiveLocation - deploymentID string - contentDetailRetention time.Duration - batchSize int - maxWait time.Duration -} - -// auditArchiveLocation is a parsed AUDIT_ARCHIVE_URL. The scheme picks the -// archive writer; gs (Google Cloud Storage) is the one built. -type auditArchiveLocation struct { - scheme string - bucket string -} - -// auditArchiveSchemeGCS is the AUDIT_ARCHIVE_URL scheme of a GCS bucket. -const auditArchiveSchemeGCS = "gs" - -// gcsBucketNamePattern is GCS's bucket naming rule, loosely: lowercase -// letters, digits, '-', '_' and '.', starting and ending alphanumeric. -var gcsBucketNamePattern = regexp.MustCompile(`^[a-z0-9][a-z0-9._-]{1,220}[a-z0-9]$`) - -// parseAuditArchiveURL reads AUDIT_ARCHIVE_URL: gs://, nothing more. -// Another scheme names an archive no writer exists for yet, which is refused -// at startup rather than left to fail at the first delivery. -func parseAuditArchiveURL(raw string) (auditArchiveLocation, error) { - parsed, err := url.Parse(raw) - if err != nil || parsed.Scheme == "" { - return auditArchiveLocation{}, fmt.Errorf("%s must be a URL like gs://; got %q", envAuditArchiveURL, raw) - } - if parsed.Scheme != auditArchiveSchemeGCS { - return auditArchiveLocation{}, fmt.Errorf("%s scheme %q has no archive writer; the supported scheme is %s:// (got %q)", - envAuditArchiveURL, parsed.Scheme, auditArchiveSchemeGCS, raw) - } - if parsed.User != nil || parsed.Port() != "" || (parsed.Path != "" && parsed.Path != "/") || parsed.RawQuery != "" || parsed.Fragment != "" { - return auditArchiveLocation{}, fmt.Errorf("%s names a bucket and nothing else, like gs://; got %q", envAuditArchiveURL, raw) - } - if !gcsBucketNamePattern.MatchString(parsed.Host) { - return auditArchiveLocation{}, fmt.Errorf("%s bucket %q is not a valid bucket name", envAuditArchiveURL, parsed.Host) - } - return auditArchiveLocation{scheme: parsed.Scheme, bucket: parsed.Host}, nil -} - -// auditDeploymentIDPattern keeps the deployment id usable as an archive path -// segment and a warehouse value without escaping. -var auditDeploymentIDPattern = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$`) - -// configuredBigQueryAuditSwap reads and validates every AUDIT_SINK=bigquery -// setting, reporting all the missing ones at once. -func configuredBigQueryAuditSwap() (*bigQueryAuditSwap, error) { - swap := &bigQueryAuditSwap{ - project: strings.TrimSpace(os.Getenv(envAuditBigQueryProject)), - dataset: strings.TrimSpace(os.Getenv(envAuditBigQueryDataset)), - deploymentID: strings.TrimSpace(os.Getenv(envAuditDeploymentID)), - batchSize: business.DefaultAuditRelayBatchSize, - maxWait: business.DefaultAuditRelayMaxWait, - } - archiveURL := strings.TrimSpace(os.Getenv(envAuditArchiveURL)) - retentionDays := strings.TrimSpace(os.Getenv(envAuditContentRetentionDays)) - var missing []string - for _, setting := range []struct{ name, value string }{ - {envAuditBigQueryProject, swap.project}, - {envAuditBigQueryDataset, swap.dataset}, - {envAuditArchiveURL, archiveURL}, - {envAuditDeploymentID, swap.deploymentID}, - {envAuditContentRetentionDays, retentionDays}, - } { - if setting.value == "" { - missing = append(missing, setting.name) - } - } - if len(missing) > 0 { - return nil, fmt.Errorf("AUDIT_SINK=bigquery requires %s", strings.Join(missing, ", ")) - } - - days, err := strconv.Atoi(retentionDays) - if err != nil || days < 1 { - return nil, fmt.Errorf("%s must be a whole number of days, at least 1; got %q", envAuditContentRetentionDays, retentionDays) - } - swap.contentDetailRetention = time.Duration(days) * 24 * time.Hour - if !auditDeploymentIDPattern.MatchString(swap.deploymentID) { - return nil, fmt.Errorf("%s must be letters, digits, '.', '_' or '-' (at most 128, starting with a letter or digit); got %q", - envAuditDeploymentID, swap.deploymentID) - } - if swap.archive, err = parseAuditArchiveURL(archiveURL); err != nil { - return nil, err - } - if raw := strings.TrimSpace(os.Getenv(envAuditRelayBatchSize)); raw != "" { - size, err := strconv.Atoi(raw) - if err != nil || size < 1 || size > business.MaxAuditRelayBatchSize { - return nil, fmt.Errorf("%s must be between 1 and %d; got %q", envAuditRelayBatchSize, business.MaxAuditRelayBatchSize, raw) - } - swap.batchSize = size - } - if raw := strings.TrimSpace(os.Getenv(envAuditRelayMaxWait)); raw != "" { - wait, err := time.ParseDuration(raw) - if err != nil || wait <= 0 || wait > time.Hour { - return nil, fmt.Errorf("%s must be a positive duration of at most an hour, like 5s; got %q", envAuditRelayMaxWait, raw) - } - swap.maxWait = wait + return nil, nil, nil, err } - return swap, nil -} - -// newBigQueryAuditSwap builds the bigquery swap value: the BigQuery store of -// record (its tables created if missing) with its read half on the Storage -// Read API, the GCS archive, and the relay that drains the queue on the -// relay's own pool. Every client uses Application Default Credentials. The -// service reads the returned store; the returned close releases them all. -func newBigQueryAuditSwap(ctx context.Context, types business.DeclaredAuditEventTypeReader, swap *bigQueryAuditSwap) (*business.AuditRelay, business.AuditStore, func(), error) { - var closers []func() + closers := []func(){opened.Close} closeAll := func() { for i := len(closers) - 1; i >= 0; i-- { closers[i]() @@ -211,55 +35,6 @@ func newBigQueryAuditSwap(ctx context.Context, types business.DeclaredAuditEvent return nil, nil, nil, err } - bigQueryClient, err := bigquery.NewClient(ctx, swap.project) - if err != nil { - return fail(fmt.Errorf("audit store: bigquery client: %w", err)) - } - closers = append(closers, func() { _ = bigQueryClient.Close() }) - // Reads run no query job: the service holds the append grant, and append - // plus job creation would let it run DML against the store of record. - readClient, err := bqstorage.NewBigQueryReadClient(ctx) - if err != nil { - return fail(fmt.Errorf("audit store: bigquery storage read client: %w", err)) - } - closers = append(closers, func() { _ = readClient.Close() }) - reader, err := bigquerystore.NewReader(bigquerystore.ReadConfig{ - Client: readClient, - Project: swap.project, - Dataset: swap.dataset, - DeploymentID: swap.deploymentID, - }) - if err != nil { - return fail(err) - } - warehouse, err := bigquerystore.New(bigQueryClient, bigquerystore.Config{ - Dataset: swap.dataset, - ContentDetailRetention: swap.contentDetailRetention, - Reader: reader, - }) - if err != nil { - return fail(err) - } - ensureCtx, cancel := context.WithTimeout(ctx, time.Minute) - err = warehouse.Ensure(ensureCtx) - cancel() - if err != nil { - return fail(err) - } - - if swap.archive.scheme != auditArchiveSchemeGCS { - return fail(fmt.Errorf("audit archive: no writer for scheme %q", swap.archive.scheme)) - } - storageClient, err := storage.NewClient(ctx) - if err != nil { - return fail(fmt.Errorf("audit archive: storage client: %w", err)) - } - closers = append(closers, func() { _ = storageClient.Close() }) - archive, err := gcsarchive.New(storageClient, swap.archive.bucket) - if err != nil { - return fail(err) - } - pool, err := infra.NewAuditRelayPool(ctx) if err != nil { return fail(fmt.Errorf("audit relay: database pool: %w", err)) @@ -272,15 +47,15 @@ func newBigQueryAuditSwap(ctx context.Context, types business.DeclaredAuditEvent relay, err := business.NewAuditRelay(business.AuditRelayConfig{ Queue: queue, - Store: warehouse, - Archive: archive, + Store: opened.Store, + Archive: opened.Archive, Types: types, - DeploymentID: swap.deploymentID, - BatchSize: swap.batchSize, - MaxWait: swap.maxWait, + DeploymentID: swap.DeploymentID, + BatchSize: swap.RelayBatchSize, + MaxWait: swap.RelayMaxWait, }) if err != nil { return fail(err) } - return relay, warehouse, closeAll, nil + return relay, opened.Store, closeAll, nil } diff --git a/module/services/accounts/code/pkg/auditstore/auditsink/auditsink.go b/module/services/accounts/code/pkg/auditstore/auditsink/auditsink.go new file mode 100644 index 000000000..9165dca8a --- /dev/null +++ b/module/services/accounts/code/pkg/auditstore/auditsink/auditsink.go @@ -0,0 +1,297 @@ +// Package auditsink reads the audit sink settings and opens what they name: the +// store of record and the locked archive of a swap value (ADR 0009). The +// accounts service and the one-time history copy both read the same +// settings through it, so a deployment configures the swap once. +package auditsink + +import ( + "context" + "errors" + "fmt" + "net/url" + "regexp" + "strconv" + "strings" + "time" + + "accounts/pkg/auditstore/bigquerystore" + "accounts/pkg/auditstore/gcsarchive" + "accounts/pkg/business" + + "cloud.google.com/go/bigquery" + bqstorage "cloud.google.com/go/bigquery/storage/apiv1" + "cloud.google.com/go/storage" +) + +// The audit sink settings. AUDIT_SINK selects the mode (business.AuditSinkMode); +// each mode's settings are required only under it, and a missing one fails +// startup rather than degrading the audit trail. +const ( + EnvSink = "AUDIT_SINK" + + // AUDIT_SINK=both (ADR 0006). + EnvExternalURL = "AUDIT_EXTERNAL_URL" + EnvExternalToken = "AUDIT_EXTERNAL_TOKEN" + + // AUDIT_SINK=bigquery (ADR 0009). Credentials are Application Default + // Credentials only — on Kubernetes, the pod's workload identity — so there + // is deliberately no key-file or token setting. + EnvBigQueryProject = "AUDIT_BIGQUERY_PROJECT" + EnvBigQueryDataset = "AUDIT_BIGQUERY_DATASET" + EnvArchiveURL = "AUDIT_ARCHIVE_URL" + EnvDeploymentID = "AUDIT_DEPLOYMENT_ID" + EnvContentRetentionDays = "AUDIT_CONTENT_RETENTION_DAYS" + // Optional relay tuning. + EnvRelayBatchSize = "AUDIT_RELAY_BATCH_SIZE" + EnvRelayMaxWait = "AUDIT_RELAY_MAX_WAIT" +) + +// Config is the resolved audit sink: the mode and, for the modes that write +// outside Postgres, where. +type Config struct { + Mode business.AuditSinkMode + // External is the tee's destination under AUDIT_SINK=both. + External business.ExternalAuditSink + // Swap is the store of record and archive under a swap value. + Swap *Swap +} + +// Swap is a validated swap-value configuration. +type Swap struct { + Mode business.AuditSinkMode + // BigQuery names the store of record under AUDIT_SINK=bigquery. + BigQuery *BigQuery + Archive ArchiveLocation + // DeploymentID stamps every record and confines every read. + DeploymentID string + // ContentRetention is how long a content-class event's details are kept. + ContentRetention time.Duration + RelayBatchSize int + RelayMaxWait time.Duration +} + +// BigQuery is where the BigQuery store of record lives. +type BigQuery struct { + Project string + Dataset string +} + +// ArchiveLocation is a parsed AUDIT_ARCHIVE_URL. The scheme picks the archive +// writer; gs (Google Cloud Storage) is the one built. +type ArchiveLocation struct { + Scheme string + Bucket string +} + +// ArchiveSchemeGCS is the AUDIT_ARCHIVE_URL scheme of a GCS bucket. +const ArchiveSchemeGCS = "gs" + +// Load reads AUDIT_SINK and the settings its mode requires through getenv +// (os.Getenv in production). "postgres" (the default) keeps the durable +// emitter alone. "both" adds an asynchronous tee to an HTTP endpoint while +// audit_events stays the store of record. "bigquery" swaps the store of record +// to BigQuery, with Postgres keeping only the transactional queue. "external" +// is refused: no destination outside Postgres can join the transaction a +// change commits in. +func Load(getenv func(string) string) (Config, error) { + mode, err := business.ParseAuditSinkMode(getenv(EnvSink)) + if err != nil { + return Config{}, err + } + config := Config{Mode: mode} + switch { + case mode == business.AuditSinkBoth: + config.External, err = loadExternal(getenv) + case mode.Swaps(): + config.Swap, err = loadSwap(mode, getenv) + } + if err != nil { + return Config{}, err + } + return config, nil +} + +func loadExternal(getenv func(string) string) (business.ExternalAuditSink, error) { + endpoint := strings.TrimSpace(getenv(EnvExternalURL)) + if endpoint == "" { + return nil, fmt.Errorf("AUDIT_SINK=both requires %s for the external audit destination", EnvExternalURL) + } + return business.NewHTTPAuditSink(business.HTTPAuditSinkConfig{ + Endpoint: endpoint, + Token: strings.TrimSpace(getenv(EnvExternalToken)), + }) +} + +// gcsBucketNamePattern is GCS's bucket naming rule, loosely: lowercase +// letters, digits, '-', '_' and '.', starting and ending alphanumeric. +var gcsBucketNamePattern = regexp.MustCompile(`^[a-z0-9][a-z0-9._-]{1,220}[a-z0-9]$`) + +// ParseArchiveURL reads AUDIT_ARCHIVE_URL: gs://, nothing more. +// Another scheme names an archive no writer exists for yet, which is refused +// at startup rather than left to fail at the first delivery. +func ParseArchiveURL(raw string) (ArchiveLocation, error) { + parsed, err := url.Parse(raw) + if err != nil || parsed.Scheme == "" { + return ArchiveLocation{}, fmt.Errorf("%s must be a URL like gs://; got %q", EnvArchiveURL, raw) + } + if parsed.Scheme != ArchiveSchemeGCS { + return ArchiveLocation{}, fmt.Errorf("%s scheme %q has no archive writer; the supported scheme is %s:// (got %q)", + EnvArchiveURL, parsed.Scheme, ArchiveSchemeGCS, raw) + } + if parsed.User != nil || parsed.Port() != "" || (parsed.Path != "" && parsed.Path != "/") || parsed.RawQuery != "" || parsed.Fragment != "" { + return ArchiveLocation{}, fmt.Errorf("%s names a bucket and nothing else, like gs://; got %q", EnvArchiveURL, raw) + } + if !gcsBucketNamePattern.MatchString(parsed.Host) { + return ArchiveLocation{}, fmt.Errorf("%s bucket %q is not a valid bucket name", EnvArchiveURL, parsed.Host) + } + return ArchiveLocation{Scheme: parsed.Scheme, Bucket: parsed.Host}, nil +} + +// deploymentIDPattern keeps the deployment id usable as an archive path segment +// and a warehouse value without escaping. +var deploymentIDPattern = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$`) + +// loadSwap reads and validates every setting of a swap value, reporting all +// the missing ones at once. +func loadSwap(mode business.AuditSinkMode, getenv func(string) string) (*Swap, error) { + swap := &Swap{ + Mode: mode, + BigQuery: &BigQuery{Project: strings.TrimSpace(getenv(EnvBigQueryProject)), Dataset: strings.TrimSpace(getenv(EnvBigQueryDataset))}, + DeploymentID: strings.TrimSpace(getenv(EnvDeploymentID)), + RelayBatchSize: business.DefaultAuditRelayBatchSize, + RelayMaxWait: business.DefaultAuditRelayMaxWait, + } + archiveURL := strings.TrimSpace(getenv(EnvArchiveURL)) + retentionDays := strings.TrimSpace(getenv(EnvContentRetentionDays)) + var missing []string + for _, setting := range []struct{ name, value string }{ + {EnvBigQueryProject, swap.BigQuery.Project}, + {EnvBigQueryDataset, swap.BigQuery.Dataset}, + {EnvArchiveURL, archiveURL}, + {EnvDeploymentID, swap.DeploymentID}, + {EnvContentRetentionDays, retentionDays}, + } { + if setting.value == "" { + missing = append(missing, setting.name) + } + } + if len(missing) > 0 { + return nil, fmt.Errorf("AUDIT_SINK=%s requires %s", mode, strings.Join(missing, ", ")) + } + + days, err := strconv.Atoi(retentionDays) + if err != nil || days < 1 { + return nil, fmt.Errorf("%s must be a whole number of days, at least 1; got %q", EnvContentRetentionDays, retentionDays) + } + swap.ContentRetention = time.Duration(days) * 24 * time.Hour + if !deploymentIDPattern.MatchString(swap.DeploymentID) { + return nil, fmt.Errorf("%s must be letters, digits, '.', '_' or '-' (at most 128, starting with a letter or digit); got %q", + EnvDeploymentID, swap.DeploymentID) + } + if swap.Archive, err = ParseArchiveURL(archiveURL); err != nil { + return nil, err + } + if raw := strings.TrimSpace(getenv(EnvRelayBatchSize)); raw != "" { + size, err := strconv.Atoi(raw) + if err != nil || size < 1 || size > business.MaxAuditRelayBatchSize { + return nil, fmt.Errorf("%s must be between 1 and %d; got %q", EnvRelayBatchSize, business.MaxAuditRelayBatchSize, raw) + } + swap.RelayBatchSize = size + } + if raw := strings.TrimSpace(getenv(EnvRelayMaxWait)); raw != "" { + wait, err := time.ParseDuration(raw) + if err != nil || wait <= 0 || wait > time.Hour { + return nil, fmt.Errorf("%s must be a positive duration of at most an hour, like 5s; got %q", EnvRelayMaxWait, raw) + } + swap.RelayMaxWait = wait + } + return swap, nil +} + +// Opened is a swap value's store of record and archive, built and checked. +type Opened struct { + // Store is the store of record: the relay appends to it and the service + // reads it. + Store business.AuditStore + // History reads back a window of what the store holds, which the history + // copy verifies against. + History business.AuditHistoryReader + // Archive is the locked archive. + Archive business.AuditArchive + closers []func() +} + +// Close releases every client Open built. +func (o *Opened) Close() { + for i := len(o.closers) - 1; i >= 0; i-- { + o.closers[i]() + } + o.closers = nil +} + +// Open builds the store of record of a swap value — its tables created if +// missing, its read half on the Storage Read API — and the archive. Every +// client uses Application Default Credentials. +func Open(ctx context.Context, swap *Swap) (*Opened, error) { + if swap == nil { + return nil, errors.New("audit sink: no swap value to open") + } + opened := &Opened{} + fail := func(err error) (*Opened, error) { + opened.Close() + return nil, err + } + if swap.Mode != business.AuditSinkBigQuery || swap.BigQuery == nil { + return fail(fmt.Errorf("audit sink: no store of record for AUDIT_SINK=%s", swap.Mode)) + } + + bigQueryClient, err := bigquery.NewClient(ctx, swap.BigQuery.Project) + if err != nil { + return fail(fmt.Errorf("audit store: bigquery client: %w", err)) + } + opened.closers = append(opened.closers, func() { _ = bigQueryClient.Close() }) + // Reads run no query job: the service holds the append grant, and append + // plus job creation would let it run DML against the store of record. + readClient, err := bqstorage.NewBigQueryReadClient(ctx) + if err != nil { + return fail(fmt.Errorf("audit store: bigquery storage read client: %w", err)) + } + opened.closers = append(opened.closers, func() { _ = readClient.Close() }) + reader, err := bigquerystore.NewReader(bigquerystore.ReadConfig{ + Client: readClient, + Project: swap.BigQuery.Project, + Dataset: swap.BigQuery.Dataset, + DeploymentID: swap.DeploymentID, + }) + if err != nil { + return fail(err) + } + warehouse, err := bigquerystore.New(bigQueryClient, bigquerystore.Config{ + Dataset: swap.BigQuery.Dataset, + ContentDetailRetention: swap.ContentRetention, + Reader: reader, + }) + if err != nil { + return fail(err) + } + ensureCtx, cancel := context.WithTimeout(ctx, time.Minute) + err = warehouse.Ensure(ensureCtx) + cancel() + if err != nil { + return fail(err) + } + opened.Store, opened.History = warehouse, warehouse + + if swap.Archive.Scheme != ArchiveSchemeGCS { + return fail(fmt.Errorf("audit archive: no writer for scheme %q", swap.Archive.Scheme)) + } + storageClient, err := storage.NewClient(ctx) + if err != nil { + return fail(fmt.Errorf("audit archive: storage client: %w", err)) + } + opened.closers = append(opened.closers, func() { _ = storageClient.Close() }) + if opened.Archive, err = gcsarchive.New(storageClient, swap.Archive.Bucket); err != nil { + return fail(err) + } + return opened, nil +} diff --git a/module/services/accounts/code/work_audit_test.go b/module/services/accounts/code/pkg/auditstore/auditsink/auditsink_test.go similarity index 72% rename from module/services/accounts/code/work_audit_test.go rename to module/services/accounts/code/pkg/auditstore/auditsink/auditsink_test.go index f5c54b36f..13b4a5088 100644 --- a/module/services/accounts/code/work_audit_test.go +++ b/module/services/accounts/code/pkg/auditstore/auditsink/auditsink_test.go @@ -1,6 +1,7 @@ -package main +package auditsink import ( + "os" "testing" "time" @@ -11,36 +12,36 @@ import ( func TestConfiguredAuditSinkDefaultsToPostgres(t *testing.T) { t.Setenv("AUDIT_SINK", "") - sink, err := configuredAuditSink() + sink, err := Load(os.Getenv) require.NoError(t, err) - require.Equal(t, business.AuditSinkPostgres, sink.mode) - require.Nil(t, sink.external) - require.Nil(t, sink.bigQuery) + require.Equal(t, business.AuditSinkPostgres, sink.Mode) + require.Nil(t, sink.External) + require.Nil(t, sink.Swap) t.Setenv("AUDIT_SINK", "postgres") - sink, err = configuredAuditSink() + sink, err = Load(os.Getenv) require.NoError(t, err) - require.Equal(t, business.AuditSinkPostgres, sink.mode) - require.False(t, sink.mode.Swaps()) + require.Equal(t, business.AuditSinkPostgres, sink.Mode) + require.False(t, sink.Mode.Swaps()) } func TestConfiguredAuditSinkBothRequiresExternalURL(t *testing.T) { t.Setenv("AUDIT_SINK", "both") t.Setenv("AUDIT_EXTERNAL_URL", "") - _, err := configuredAuditSink() + _, err := Load(os.Getenv) require.ErrorContains(t, err, "AUDIT_EXTERNAL_URL") t.Setenv("AUDIT_EXTERNAL_URL", "https://warehouse.example/audit") - sink, err := configuredAuditSink() + sink, err := Load(os.Getenv) require.NoError(t, err) - require.Equal(t, business.AuditSinkBoth, sink.mode) - require.False(t, sink.mode.Swaps()) - require.IsType(t, &business.HTTPAuditSink{}, sink.external) + require.Equal(t, business.AuditSinkBoth, sink.Mode) + require.False(t, sink.Mode.Swaps()) + require.IsType(t, &business.HTTPAuditSink{}, sink.External) } func TestConfiguredAuditSinkRejectsExternalOnly(t *testing.T) { t.Setenv("AUDIT_SINK", "external") - _, err := configuredAuditSink() + _, err := Load(os.Getenv) require.ErrorContains(t, err, "not permitted") require.ErrorContains(t, err, "bigquery", "the refusal names the swap value that does what external cannot") } @@ -48,7 +49,7 @@ func TestConfiguredAuditSinkRejectsExternalOnly(t *testing.T) { func TestConfiguredAuditSinkRejectsUnknownMode(t *testing.T) { for _, value := range []string{"kafka", "clickhouse"} { t.Setenv("AUDIT_SINK", value) - _, err := configuredAuditSink() + _, err := Load(os.Getenv) require.ErrorContains(t, err, "must be postgres, both or bigquery", value) } } @@ -68,32 +69,32 @@ func setBigQuerySwap(t *testing.T) { func TestConfiguredAuditSinkBigQuery(t *testing.T) { setBigQuerySwap(t) - sink, err := configuredAuditSink() + sink, err := Load(os.Getenv) require.NoError(t, err) - require.Equal(t, business.AuditSinkBigQuery, sink.mode) - require.True(t, sink.mode.Swaps()) - require.Nil(t, sink.external) - require.Equal(t, &bigQueryAuditSwap{ - project: "example-project", - dataset: "audit", - archive: auditArchiveLocation{scheme: "gs", bucket: "example-audit-archive"}, - deploymentID: "acme-prod-1", - contentDetailRetention: 90 * 24 * time.Hour, - batchSize: business.DefaultAuditRelayBatchSize, - maxWait: business.DefaultAuditRelayMaxWait, - }, sink.bigQuery) + require.Equal(t, business.AuditSinkBigQuery, sink.Mode) + require.True(t, sink.Mode.Swaps()) + require.Nil(t, sink.External) + require.Equal(t, &Swap{ + Mode: business.AuditSinkBigQuery, + BigQuery: &BigQuery{Project: "example-project", Dataset: "audit"}, + Archive: ArchiveLocation{Scheme: "gs", Bucket: "example-audit-archive"}, + DeploymentID: "acme-prod-1", + ContentRetention: 90 * 24 * time.Hour, + RelayBatchSize: business.DefaultAuditRelayBatchSize, + RelayMaxWait: business.DefaultAuditRelayMaxWait, + }, sink.Swap) t.Setenv("AUDIT_ARCHIVE_URL", "gs://example-audit-archive/") - sink, err = configuredAuditSink() + sink, err = Load(os.Getenv) require.NoError(t, err, "a trailing slash still names only the bucket") - require.Equal(t, "example-audit-archive", sink.bigQuery.archive.bucket) + require.Equal(t, "example-audit-archive", sink.Swap.Archive.Bucket) t.Setenv("AUDIT_RELAY_BATCH_SIZE", "200") t.Setenv("AUDIT_RELAY_MAX_WAIT", "750ms") - sink, err = configuredAuditSink() + sink, err = Load(os.Getenv) require.NoError(t, err) - require.Equal(t, 200, sink.bigQuery.batchSize) - require.Equal(t, 750*time.Millisecond, sink.bigQuery.maxWait) + require.Equal(t, 200, sink.Swap.RelayBatchSize) + require.Equal(t, 750*time.Millisecond, sink.Swap.RelayMaxWait) } func TestConfiguredAuditSinkBigQueryNamesEveryMissingSetting(t *testing.T) { @@ -104,13 +105,13 @@ func TestConfiguredAuditSinkBigQueryNamesEveryMissingSetting(t *testing.T) { } { t.Setenv(name, "") } - _, err := configuredAuditSink() + _, err := Load(os.Getenv) require.EqualError(t, err, "AUDIT_SINK=bigquery requires AUDIT_BIGQUERY_PROJECT, AUDIT_BIGQUERY_DATASET, "+ "AUDIT_ARCHIVE_URL, AUDIT_DEPLOYMENT_ID, AUDIT_CONTENT_RETENTION_DAYS") setBigQuerySwap(t) t.Setenv("AUDIT_ARCHIVE_URL", " ") - _, err = configuredAuditSink() + _, err = Load(os.Getenv) require.EqualError(t, err, "AUDIT_SINK=bigquery requires AUDIT_ARCHIVE_URL", "a blank value is a missing one") } @@ -138,7 +139,7 @@ func TestConfiguredAuditSinkBigQueryRejectsInvalidSettings(t *testing.T) { t.Run(tc.name+"="+tc.value, func(t *testing.T) { setBigQuerySwap(t) t.Setenv(tc.name, tc.value) - _, err := configuredAuditSink() + _, err := Load(os.Getenv) require.ErrorContains(t, err, tc.want) }) } diff --git a/module/services/accounts/code/pkg/auditstore/bigquerystore/bigquerystore.go b/module/services/accounts/code/pkg/auditstore/bigquerystore/bigquerystore.go index 61049e0c9..6c633487b 100644 --- a/module/services/accounts/code/pkg/auditstore/bigquerystore/bigquerystore.go +++ b/module/services/accounts/code/pkg/auditstore/bigquerystore/bigquerystore.go @@ -132,7 +132,18 @@ func (s *Store) LatestSourceSyncEvents(ctx context.Context, scope business.Audit return s.reader.LatestSourceSyncEvents(ctx, scope, sources) } -var _ business.AuditStore = (*Store)(nil) +// ReadStoredAuditEvents implements business.AuditHistoryReader. +func (s *Store) ReadStoredAuditEvents(ctx context.Context, from, to time.Time, visit func(business.StoredAuditEvent) error) error { + if s.reader == nil { + return errWriteOnly + } + return s.reader.ReadStoredAuditEvents(ctx, from, to, visit) +} + +var ( + _ business.AuditStore = (*Store)(nil) + _ business.AuditHistoryReader = (*Store)(nil) +) // EventsSchema is the events table: every event's envelope, the SHA-256 of its // canonical details, and the details themselves for security-class events. diff --git a/module/services/accounts/code/pkg/auditstore/bigquerystore/reader.go b/module/services/accounts/code/pkg/auditstore/bigquerystore/reader.go index 7f79d9280..4aaf0827b 100644 --- a/module/services/accounts/code/pkg/auditstore/bigquerystore/reader.go +++ b/module/services/accounts/code/pkg/auditstore/bigquerystore/reader.go @@ -515,6 +515,68 @@ func (r *Reader) LatestSourceSyncEvents(ctx context.Context, scope business.Audi return out, nil } +// ReadStoredAuditEvents implements business.AuditHistoryReader: one session +// over each table for the window, every copy of every event of this +// deployment, with the details each copy has. A content-class event's details +// come from the details table; when the copies there disagree, a copy whose +// hash does not match the event's is the one reported, so the disagreement is +// what verification sees. +func (r *Reader) ReadStoredAuditEvents(ctx context.Context, from, to time.Time, visit func(business.StoredAuditEvent) error) error { + w := window{lo: &from, hi: &to, hiExclusive: true} + detailsWhere := r.scope(business.PlatformAuditScope()) + w.restrict(detailsWhere) + copies := map[string][]storedDetail{} + err := r.scan(ctx, DetailsTable, detailFields, detailsWhere, func(row arrowRow) error { + var detail storedDetail + var eventID, deploymentID string + for name, into := range map[string]*string{ + "event_id": &eventID, "deployment_id": &deploymentID, "details_sha256": &detail.sha256, "details": &detail.details, + } { + value, err := row.str(name) + if err != nil { + return err + } + *into = value + } + if deploymentID == r.deploymentID { + copies[eventID] = append(copies[eventID], detail) + } + return nil + }) + if err != nil { + return err + } + where := r.scope(business.PlatformAuditScope()) + w.restrict(where) + return r.scan(ctx, EventsTable, eventFields, where, func(row arrowRow) error { + stored, err := decodeEventRow(row) + if err != nil { + return err + } + if stored.deploymentID != r.deploymentID { + return nil + } + event := business.StoredAuditEvent{ + DeploymentID: stored.deploymentID, + Entry: stored.entry, + Retention: business.AuditRetentionClass(stored.retentionClass), + DetailsSHA256: stored.detailsSHA256, + Details: stored.details, + HasDetails: stored.hasDetails, + } + if event.Retention != business.RetentionSecurity { + event.Details, event.HasDetails = "", false + for _, detail := range copies[stored.entry.ID] { + event.Details, event.HasDetails = detail.details, true + if detail.sha256 != stored.detailsSHA256 { + break + } + } + } + return visit(event) + }) +} + // scan creates one read session over table and hands every row it returns to // visit, serially. The session's streams are read in parallel; a stream broken // by a transient error is reopened at the row it reached. diff --git a/module/services/accounts/code/pkg/business/audit_history.go b/module/services/accounts/code/pkg/business/audit_history.go new file mode 100644 index 000000000..0a1a16fc2 --- /dev/null +++ b/module/services/accounts/code/pkg/business/audit_history.go @@ -0,0 +1,29 @@ +package business + +import ( + "context" + "time" +) + +// StoredAuditEvent is one copy of an event as a store of record holds it: the +// envelope, the retention class and details hash it was written with, and the +// full details the store keeps for it — in the events row for a security-class +// event, in the details table for a content-class one, and none for a +// content-class event past the content window. +type StoredAuditEvent struct { + DeploymentID string + Entry AuditEntry // the envelope; Payload is never set + Retention AuditRetentionClass + DetailsSHA256 string + Details string + HasDetails bool +} + +// AuditHistoryReader reads back what a store of record holds of this +// deployment's events that occurred in [from, to), across every organization +// and the platform, every copy of a duplicated event included — the read-back +// the history copy verifies against (ADR 0009, item 7). Like every read of a +// store of record it runs no DML; on BigQuery it is a Storage Read API session. +type AuditHistoryReader interface { + ReadStoredAuditEvents(ctx context.Context, from, to time.Time, visit func(StoredAuditEvent) error) error +} diff --git a/module/services/accounts/code/pkg/business/audit_store_parity_test.go b/module/services/accounts/code/pkg/business/audit_store_parity_test.go index fedac8e4f..5673a32a7 100644 --- a/module/services/accounts/code/pkg/business/audit_store_parity_test.go +++ b/module/services/accounts/code/pkg/business/audit_store_parity_test.go @@ -23,15 +23,16 @@ import ( "google.golang.org/grpc/status" ) -// Read parity of the audit store swap (ADR 0009): one fixture, written once -// through the emitter as a postgres deployment writes it (audit_events) and -// once as a bigquery deployment does (the queue, drained by the relay into the -// warehouse), must read back identically through the service — the activity -// list page by page, every aggregation, both export formats and the -// readable-source query. The warehouse is the fake Storage Read API, which -// evaluates the reader's row restrictions; it also holds what Postgres does -// not: events delivered twice, and another deployment's and another -// organization's events, none of which may change an answer. +// Read parity of the audit store swap (ADR 0009) — the conformance suite every +// store of record passes. One fixture, written once through the emitter as a +// postgres deployment writes it (audit_events) and once as a swap deployment +// does (the queue, drained by the relay into each store under test), must read +// back identically through the service: the activity list page by page, every +// aggregation, both export formats and the readable-source query. Each store +// also holds what Postgres does not: events delivered twice, and another +// deployment's and another organization's events, none of which may change an +// answer. A read without a scope is refused, and a refused resource read never +// reaches the store. const ( parityProject = "parity-project" @@ -39,38 +40,121 @@ const ( parityDeployment = "deployment-parity" ) -// parityWarehouse is the relay's store of record: each batch's rows appended -// to the fake, as the BigQuery writer appends them. -type parityWarehouse struct { - fake *bigqueryfake.Server - batches []business.AuditBatch +// parityStore is one store of record under test. +type parityStore struct { + // writer is what the relay appends to. + writer business.AuditStoreWriter + // store is what the service reads. + store business.AuditStore + // eventIDs lists every event id the store holds, once per copy. + eventIDs func(t *testing.T) []string +} + +// parityStores builds every store of record the suite runs against. A store +// that needs a server the environment does not provide returns nil, after +// logging how to provide one. +var parityStores = map[string]func(t *testing.T) *parityStore{ + "bigquery": newBigQueryParityStore, } -func (w *parityWarehouse) AppendAuditBatch(_ context.Context, batch business.AuditBatch) error { +// bigQueryAppender appends a batch's rows to the fake, as the BigQuery writer +// streams them. +type bigQueryAppender struct{ fake *bigqueryfake.Server } + +func (a bigQueryAppender) AppendAuditBatch(_ context.Context, batch business.AuditBatch) error { events, details := bigquerystore.BatchRows(batch) for _, row := range events { - if err := w.fake.Insert(bigqueryfake.TablePath(parityProject, parityDataset, bigquerystore.EventsTable), row.Values); err != nil { + if err := a.fake.Insert(bigqueryfake.TablePath(parityProject, parityDataset, bigquerystore.EventsTable), row.Values); err != nil { return err } } for _, row := range details { - if err := w.fake.Insert(bigqueryfake.TablePath(parityProject, parityDataset, bigquerystore.DetailsTable), row.Values); err != nil { + if err := a.fake.Insert(bigqueryfake.TablePath(parityProject, parityDataset, bigquerystore.DetailsTable), row.Values); err != nil { + return err + } + } + return nil +} + +// newBigQueryParityStore is the BigQuery store over the fake Storage Read API, +// which evaluates the reader's row restrictions and serves Arrow batches over +// several streams. +func newBigQueryParityStore(t *testing.T) *parityStore { + fake := bigqueryfake.New() + eventsPath := bigqueryfake.TablePath(parityProject, parityDataset, bigquerystore.EventsTable) + fake.CreateTable(eventsPath, bigquerystore.EventsSchema()) + fake.CreateTable(bigqueryfake.TablePath(parityProject, parityDataset, bigquerystore.DetailsTable), bigquerystore.DetailsSchema()) + client, err := bigquery.NewClient(testCtx, parityProject, option.WithoutAuthentication(), option.WithEndpoint("http://127.0.0.1:1")) + require.NoError(t, err) + t.Cleanup(func() { _ = client.Close() }) + reader, err := bigquerystore.NewReader(bigquerystore.ReadConfig{ + Client: fake, Project: parityProject, Dataset: parityDataset, DeploymentID: parityDeployment, + }) + require.NoError(t, err) + store, err := bigquerystore.New(client, bigquerystore.Config{Dataset: parityDataset, ContentDetailRetention: 30 * 24 * time.Hour, Reader: reader}) + require.NoError(t, err) + return &parityStore{ + writer: bigQueryAppender{fake: fake}, + store: store, + eventIDs: func(*testing.T) []string { + var ids []string + for _, row := range fake.Rows(eventsPath) { + ids = append(ids, row["event_id"].(string)) + } + return ids + }, + } +} + +// fanOut appends every batch to every store under test, and remembers it. +type fanOut struct { + writers []business.AuditStoreWriter + batches []business.AuditBatch +} + +func (f *fanOut) AppendAuditBatch(ctx context.Context, batch business.AuditBatch) error { + for _, writer := range f.writers { + if err := writer.AppendAuditBatch(ctx, batch); err != nil { return err } } - w.batches = append(w.batches, batch) + f.batches = append(f.batches, batch) return nil } +// countingStore counts the reads that reach a store. +type countingStore struct { + business.AuditStore + reads int +} + +func (c *countingStore) ListAuditEvents(ctx context.Context, read business.AuditRead) ([]business.AuditEntry, string, error) { + c.reads++ + return c.AuditStore.ListAuditEvents(ctx, read) +} + +func (c *countingStore) AggregateAuditEvents(ctx context.Context, read business.AuditRead, spec business.AuditAggregationSpec) ([]business.AuditAggregateBucket, error) { + c.reads++ + return c.AuditStore.AggregateAuditEvents(ctx, read, spec) +} + +func (c *countingStore) ExportAuditEvents(ctx context.Context, read business.AuditRead) ([]business.AuditEntry, error) { + c.reads++ + return c.AuditStore.ExportAuditEvents(ctx, read) +} + +func (c *countingStore) LatestSourceSyncEvents(ctx context.Context, scope business.AuditReadScope, sources []string) (map[string]business.AuditSourceSyncEvent, error) { + c.reads++ + return c.AuditStore.LatestSourceSyncEvents(ctx, scope, sources) +} + type discardArchive struct{} func (discardArchive) WriteAuditBatch(context.Context, business.AuditBatch) error { return nil } type parityFixture struct { - warehouse *parityWarehouse + stores map[string]*parityStore postgres *business.Service - swapped *business.Service - store *bigquerystore.Store org, otherOrg string actor, other string sourceOrg string @@ -201,17 +285,22 @@ func newParityFixture(t *testing.T) *parityFixture { } } - fake := bigqueryfake.New() - fake.CreateTable(bigqueryfake.TablePath(parityProject, parityDataset, bigquerystore.EventsTable), bigquerystore.EventsSchema()) - fake.CreateTable(bigqueryfake.TablePath(parityProject, parityDataset, bigquerystore.DetailsTable), bigquerystore.DetailsSchema()) - f.warehouse = &parityWarehouse{fake: fake} + f.stores = map[string]*parityStore{} + delivery := &fanOut{} + for name, open := range parityStores { + if store := open(t); store != nil { + f.stores[name] = store + delivery.writers = append(delivery.writers, store.writer) + } + } + require.NotEmpty(t, f.stores) pool, err := infra.NewAuditRelayPool(ctx) require.NoError(t, err) t.Cleanup(pool.Close) queue, err := infra.NewPostgresAuditQueue(pool) require.NoError(t, err) relay, err := business.NewAuditRelay(business.AuditRelayConfig{ - Queue: queue, Store: f.warehouse, Archive: discardArchive{}, Types: testStore, + Queue: queue, Store: delivery, Archive: discardArchive{}, Types: testStore, DeploymentID: parityDeployment, BatchSize: 4, MaxWait: time.Nanosecond, }) require.NoError(t, err) @@ -220,51 +309,49 @@ func newParityFixture(t *testing.T) *parityFixture { require.GreaterOrEqual(t, delivered, len(entries)) // A relay that restarted between its writes and its queue delete delivers a - // batch again; the warehouse then holds those events twice. - require.GreaterOrEqual(t, len(f.warehouse.batches), 3) - for _, again := range []business.AuditBatch{f.warehouse.batches[0], f.warehouse.batches[len(f.warehouse.batches)/2]} { + // batch again; the store then holds those events twice. + batches := delivery.batches + require.GreaterOrEqual(t, len(batches), 3) + for _, again := range []business.AuditBatch{batches[0], batches[len(batches)/2]} { again.ID = uuid.NewString() - require.NoError(t, f.warehouse.AppendAuditBatch(ctx, again)) + require.NoError(t, delivery.AppendAuditBatch(ctx, again)) } // Another deployment writing the same organization's events into the same // tables is outside every read of this one. - foreign := f.warehouse.batches[1] + foreign := batches[1] foreign.ID, foreign.DeploymentID = uuid.NewString(), "deployment-other" - require.NoError(t, f.warehouse.AppendAuditBatch(ctx, foreign)) - - client, err := bigquery.NewClient(ctx, parityProject, option.WithoutAuthentication(), option.WithEndpoint("http://127.0.0.1:1")) - require.NoError(t, err) - t.Cleanup(func() { _ = client.Close() }) - reader, err := bigquerystore.NewReader(bigquerystore.ReadConfig{ - Client: fake, Project: parityProject, Dataset: parityDataset, DeploymentID: parityDeployment, - }) - require.NoError(t, err) - f.store, err = bigquerystore.New(client, bigquerystore.Config{Dataset: parityDataset, ContentDetailRetention: 30 * 24 * time.Hour, Reader: reader}) - require.NoError(t, err) + require.NoError(t, delivery.AppendAuditBatch(ctx, foreign)) f.postgres, err = business.NewService(testStore) require.NoError(t, err) - f.swapped, err = business.NewService(testStore) - require.NoError(t, err) - f.swapped.SetAuditStore(f.store) return f } -func TestAuditStoreParityPostgresAndBigQuery(t *testing.T) { +func TestAuditStoreParity(t *testing.T) { f := newParityFixture(t) + for name, store := range f.stores { + t.Run(name, func(t *testing.T) { runAuditStoreParity(t, f, store) }) + } +} + +func runAuditStoreParity(t *testing.T, f *parityFixture, under *parityStore) { ctx := testCtx - eventsPath := bigqueryfake.TablePath(parityProject, parityDataset, bigquerystore.EventsTable) - stored := map[string]int{} - for _, row := range f.warehouse.fake.Rows(eventsPath) { - stored[row["event_id"].(string)]++ + counted := &countingStore{AuditStore: under.store} + swapped, err := business.NewService(testStore) + require.NoError(t, err) + swapped.SetAuditStore(counted) + + copies := map[string]int{} + for _, id := range under.eventIDs(t) { + copies[id]++ } duplicated := 0 - for _, n := range stored { + for _, n := range copies { if n > 1 { duplicated++ } } - require.Positive(t, duplicated, "the warehouse holds events more than once") + require.Positive(t, duplicated, "the store holds events more than once") week := time.Now().Add(-7 * 24 * time.Hour) recent := time.Now().Add(-3 * time.Hour) @@ -288,7 +375,7 @@ func TestAuditStoreParityPostgresAndBigQuery(t *testing.T) { t.Run("list", func(t *testing.T) { // The fixture is what the comparisons compare: not two empty answers. - all, _, _, err := f.swapped.QueryAuditLog(ctx, business.AuditQuery{OrgID: f.org, PageSize: 100}) + all, _, _, err := swapped.QueryAuditLog(ctx, business.AuditQuery{OrgID: f.org, PageSize: 100}) require.NoError(t, err) require.Len(t, all, 12) for name, q := range queries { @@ -299,7 +386,7 @@ func TestAuditStoreParityPostgresAndBigQuery(t *testing.T) { for { want, wantNext, wantTotal, err := f.postgres.QueryAuditLog(ctx, q) require.NoError(t, err, name) - got, gotNext, gotTotal, err := f.swapped.QueryAuditLog(ctx, q) + got, gotNext, gotTotal, err := swapped.QueryAuditLog(ctx, q) require.NoError(t, err, name) require.Equal(t, want, got, "%s, page size %d, page %d", name, size, pages) require.Equal(t, wantNext, gotNext, name) @@ -351,7 +438,7 @@ func TestAuditStoreParityPostgresAndBigQuery(t *testing.T) { if queryName == "organization" { require.NotEmpty(t, want, specName) } - got, err := f.swapped.AggregateAuditLog(ctx, q, spec) + got, err := swapped.AggregateAuditLog(ctx, q, spec) require.NoError(t, err, "%s / %s", queryName, specName) require.Equal(t, want, got, "%s / %s", queryName, specName) } @@ -371,7 +458,7 @@ func TestAuditStoreParityPostgresAndBigQuery(t *testing.T) { } { want, wantType, _, err := f.postgres.ExportAuditLog(ctx, export.org, format, export.actor, export.eventType, export.eventTypes) require.NoError(t, err) - got, gotType, _, err := f.swapped.ExportAuditLog(ctx, export.org, format, export.actor, export.eventType, export.eventTypes) + got, gotType, _, err := swapped.ExportAuditLog(ctx, export.org, format, export.actor, export.eventType, export.eventTypes) require.NoError(t, err) require.Equal(t, wantType, gotType) require.Greater(t, len(want), 200, "%s export of %+v holds events", format, export) @@ -386,7 +473,7 @@ func TestAuditStoreParityPostgresAndBigQuery(t *testing.T) { require.NoError(t, testStore.WithSourceReadSnapshot(verified, f.sourceOrg, func(snapshot context.Context) error { want, err := f.postgres.LatestSourceSyncRequests(snapshot, f.sourceOrg, asked) require.NoError(t, err) - got, err := f.swapped.LatestSourceSyncRequests(snapshot, f.sourceOrg, asked) + got, err := swapped.LatestSourceSyncRequests(snapshot, f.sourceOrg, asked) require.NoError(t, err) require.Equal(t, want, got) require.Len(t, got, len(f.sources), "a source never synced has no request") @@ -399,26 +486,27 @@ func TestAuditStoreParityPostgresAndBigQuery(t *testing.T) { }) t.Run("scope", func(t *testing.T) { - _, _, err := f.store.ListAuditEvents(ctx, business.AuditRead{Query: business.AuditQuery{OrgID: f.org}}) + store := under.store + _, _, err := store.ListAuditEvents(ctx, business.AuditRead{Query: business.AuditQuery{OrgID: f.org}}) require.ErrorIs(t, err, business.ErrAuditReadUnscoped) - _, err = f.store.AggregateAuditEvents(ctx, business.AuditRead{}, business.AuditAggregationSpec{}) + _, err = store.AggregateAuditEvents(ctx, business.AuditRead{}, business.AuditAggregationSpec{}) require.ErrorIs(t, err, business.ErrAuditReadUnscoped) - _, err = f.store.ExportAuditEvents(ctx, business.AuditRead{}) + _, err = store.ExportAuditEvents(ctx, business.AuditRead{}) require.ErrorIs(t, err, business.ErrAuditReadUnscoped) - _, err = f.store.LatestSourceSyncEvents(ctx, business.AuditReadScope{}, f.sources) + _, err = store.LatestSourceSyncEvents(ctx, business.AuditReadScope{}, f.sources) require.ErrorIs(t, err, business.ErrAuditReadUnscoped) - _, _, err = f.store.ListAuditEvents(ctx, business.AuditRead{ + _, _, err = store.ListAuditEvents(ctx, business.AuditRead{ Scope: business.OrganizationAuditScope(f.org), Query: business.AuditQuery{OrgID: f.otherOrg}, }) require.Error(t, err, "a query naming another organization than its scope is refused") - // A resource read is authorized in Postgres before the warehouse is read; - // a reader without the grant reads nothing from it. - before := len(f.warehouse.fake.Sessions()) - _, err = f.swapped.AggregateAuditLogForReader(ctx, f.actor, business.AuditQuery{ + // A resource read is authorized in Postgres before the store is read; a + // reader without the grant reads nothing from it. + before := counted.reads + _, err = swapped.AggregateAuditLogForReader(ctx, f.actor, business.AuditQuery{ OrgID: f.org, Resource: "datasource", ResourceID: uuid.NewString(), EventType: string(business.EventDatasourceSourceSynced), }, business.AuditAggregationSpec{}) require.Equal(t, codes.PermissionDenied, status.Code(err), fmt.Sprint(err)) - require.Len(t, f.warehouse.fake.Sessions(), before, "a refused read never reaches the warehouse") + require.Equal(t, before, counted.reads, "a refused read never reaches the store") }) } diff --git a/module/services/accounts/code/work.go b/module/services/accounts/code/work.go index cc27ecc48..9c5af7a4a 100644 --- a/module/services/accounts/code/work.go +++ b/module/services/accounts/code/work.go @@ -544,9 +544,9 @@ func doWork(ctx context.Context) (Clean, error) { } var auditEmitterOpts []business.DurableAuditEmitterOption switch { - case auditSink.mode == business.AuditSinkBoth: + case auditSink.Mode == business.AuditSinkBoth: auditEmitterOpts = append(auditEmitterOpts, business.WithExternalTee()) - case auditSink.mode.Swaps(): + case auditSink.Mode.Swaps(): auditEmitterOpts = append(auditEmitterOpts, business.WithQueuedRecords()) } // Every org-scoped audit record publishes its external domain event in the @@ -572,12 +572,13 @@ func doWork(ctx context.Context) (Clean, error) { var auditRelay *business.AuditRelay closeAuditRelay := func() {} - if auditSink.bigQuery != nil { - // Under the swap BigQuery is the store of record for writes and reads - // alike: the activity list, aggregates, exports and the readable-source - // query read it, scoped to the caller's organization and this deployment. + if auditSink.Swap != nil { + // Under a swap value the warehouse is the store of record for writes and + // reads alike: the activity list, aggregates, exports and the + // readable-source query read it, scoped to the caller's organization and + // this deployment. var auditStore business.AuditStore - auditRelay, auditStore, closeAuditRelay, err = newBigQueryAuditSwap(ctx, store, auditSink.bigQuery) + auditRelay, auditStore, closeAuditRelay, err = newAuditSwap(ctx, store, auditSink.Swap) if err != nil { return nil, err } @@ -585,14 +586,14 @@ func doWork(ctx context.Context) (Clean, error) { } var auditExportWorker *jobs.Worker - if auditSink.mode == business.AuditSinkBoth { + if auditSink.Mode == business.AuditSinkBoth { // The tee carries org-scoped events only. Control-plane / platform-admin // audit events (NULL org) are Postgres-only: the job platform reserves // global-scope enqueue for the privileged worker pool, which opens its own // transaction and so cannot commit atomically with the audit row. Surface // this so operators don't assume the warehouse holds platform events. w.Warn("AUDIT_SINK=both tees only org-scoped audit events to the external sink; control-plane/platform-admin (NULL-org) events remain Postgres-only") - auditExportHandler, err := business.NewAuditExportJobHandler(auditSink.external, store) + auditExportHandler, err := business.NewAuditExportJobHandler(auditSink.External, store) if err != nil { return nil, err } From 1226c8f949ec94e278bf6da3eb780913404ffa04 Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Sat, 3 Oct 2026 20:43:06 -0400 Subject: [PATCH 10/69] feat(audit): copy a switching deployment's audit history into the store of record, verify it, then drop it ADR 0009 (#996) step 4. A deployment switching from AUDIT_SINK=postgres to a swap value keeps the audit_events rows it already holds. The new audit-history-copy command copies them into the store of record and the locked archive. Each event is classified per event type and given the same envelope and hash as a live one, in batches written archive first, as the relay writes them. The command then verifies the copy by reading it back through the Storage Read API: every row present, every copy carrying the row's envelope, retention class and details hash, the stored details hashing to it, and per-organization counts. Only with -confirm-drop, after this run verified every copied partition, and after a recount shows nothing changed and no unverified partition ends before the cutoff, does it drop the copied monthly partitions with audit_events_drop_partitions_before, the function RunRetention uses. No row is deleted. The copy is resumable without state. Each day window is read back before anything is written, and only missing events are copied, so a run after an interrupted one finishes it. The archive may then hold a batch twice, as a relay retry leaves it. -verify-only re-checks without copying, -through YYYY-MM limits the copy and the drop's cutoff to the months up to it, and a content-class event's details are required only inside the content window. The command reads the service's own AUDIT_* settings and runs only under a swap value. The Postgres side (ListAuditPartitions, ReadAuditHistory, CountAuditHistory) takes partition bounds from the catalog and reads rows as the queue does. The control-plane call-site list gains the copy's three platform operations. Against the BigQuery emulator (ghcr.io/goccy/bigquery-emulator), the read parity suite also runs over real streaming inserts and the emulator's Storage Read API, which applies the reader's row restrictions with a GoogleSQL engine. It passes, and a test checks that the server, not the service, narrows what is streamed back. That emulator sends each Arrow batch as a whole stream with its own schema, and the reader now decodes such a batch on its own. Co-Authored-By: Claude Opus 5.5 --- module/DATABASE_AUTHORITY.md | 5 + module/FEATURES.md | 2 +- module/services/accounts/AGENTS.md | 8 + .../code/cmd/audit-history-copy/main.go | 183 +++++++ .../code/cmd/audit-history-copy/main_test.go | 94 ++++ .../auditstore/bigquerystore/emulator_test.go | 107 ++++ .../pkg/auditstore/bigquerystore/reader.go | 48 +- .../code/pkg/business/audit_history.go | 472 ++++++++++++++++++ .../code/pkg/business/audit_history_test.go | 433 ++++++++++++++++ .../pkg/business/audit_store_parity_test.go | 66 ++- .../code/pkg/infra/postgres_audit_history.go | 132 +++++ .../pkg/infra/postgres_audit_history_test.go | 208 ++++++++ .../testdata/control_plane_call_sites.txt | 3 + 13 files changed, 1748 insertions(+), 13 deletions(-) create mode 100644 module/services/accounts/code/cmd/audit-history-copy/main.go create mode 100644 module/services/accounts/code/cmd/audit-history-copy/main_test.go create mode 100644 module/services/accounts/code/pkg/business/audit_history_test.go create mode 100644 module/services/accounts/code/pkg/infra/postgres_audit_history.go create mode 100644 module/services/accounts/code/pkg/infra/postgres_audit_history_test.go diff --git a/module/DATABASE_AUTHORITY.md b/module/DATABASE_AUTHORITY.md index 23e22bb26..be5659834 100644 --- a/module/DATABASE_AUTHORITY.md +++ b/module/DATABASE_AUTHORITY.md @@ -506,6 +506,11 @@ All tenant/user rows remain constrained by their forced RLS policies. Retention deletes and token-based pre-auth reads/updates use the audited control-plane boundary; they are intentionally not granted to `app_tenant`. +So does the one-time audit history copy of a deployment switching to a swap +value (`cmd/audit-history-copy`, ADR 0009): it reads `audit_events` across +organizations and, once its copy is verified and confirmed, drops the copied +partitions through `audit_events_drop_partitions_before`, as retention does. It +deletes no row. The infrastructure suite compares PostgreSQL's complete public-table inventory to this executable matrix, so adding a table without classifying and granting it fails the gate. diff --git a/module/FEATURES.md b/module/FEATURES.md index d88a70603..52334aa08 100644 --- a/module/FEATURES.md +++ b/module/FEATURES.md @@ -398,7 +398,7 @@ see `JOBS.md` for the exact boundary and sequencing. | Export (JSON/CSV) | ✅ | `audit_export.go` | | Impersonation tracking | ✅ | Records both real actor + viewed-as user | | Replay / event sourcing| ❌ | Audit log is read-only history; not used to reconstruct state | -| Warehouse store of record | 🟡 | `AUDIT_SINK=bigquery` (ADR 0009): each record commits to a Postgres queue on the mutation's transaction and a relay delivers it to BigQuery and a locked GCS archive. The activity list, aggregates, exports and the readable-source query read BigQuery through the Storage Read API — no query jobs — scoped to the caller's organization and the deployment, with each event returned once; Postgres history is not yet copied over; ClickHouse is not built | +| Warehouse store of record | 🟡 | `AUDIT_SINK=bigquery` (ADR 0009): each record commits to a Postgres queue on the mutation's transaction and a relay delivers it to BigQuery and a locked GCS archive. The activity list, aggregates, exports and the readable-source query read BigQuery through the Storage Read API — no query jobs — scoped to the caller's organization and the deployment, with each event returned once. `audit-history-copy` copies a switching deployment's `audit_events` history into the store and the archive, verifies it by reading it back, and drops the copied partitions only with `-confirm-drop`; ClickHouse is not built | ### Admin panel diff --git a/module/services/accounts/AGENTS.md b/module/services/accounts/AGENTS.md index 7267ecd06..3f156413e 100644 --- a/module/services/accounts/AGENTS.md +++ b/module/services/accounts/AGENTS.md @@ -83,6 +83,14 @@ survives a restart and reaches every replica, and is only served when it is evaluated in the service (`pkg/auditstore/auditeval`) with the Postgres reads' semantics; `pkg/business/audit_store_parity_test.go` holds the two to one fixture. +- **History copy** (`cmd/audit-history-copy`, `pkg/business/audit_history.go`): + a deployment switching to a swap value copies its `audit_events` rows into the + store of record and the archive — classified and hashed as the relay does a + live event — and verifies them by reading them back. It reads back before it + writes, so a rerun finishes an interrupted run. Only with `-confirm-drop`, + after a verification pass in the same run and a recount showing nothing + changed, does it drop the copied partitions, through the function retention + uses; it never deletes a row. - A composed **module** with no frontend half declares its own audit event types through `ModuleCapabilitiesService.DeclareAuditEventTypes` (`pkg/business/module_audit_declarations.go`): the same validator diff --git a/module/services/accounts/code/cmd/audit-history-copy/main.go b/module/services/accounts/code/cmd/audit-history-copy/main.go new file mode 100644 index 000000000..d7ecf80cf --- /dev/null +++ b/module/services/accounts/code/cmd/audit-history-copy/main.go @@ -0,0 +1,183 @@ +// Command audit-history-copy is the one-time history copy of a deployment +// switching from AUDIT_SINK=postgres to a swap value (ADR 0009, item 7). It +// copies the audit_events rows the deployment already holds into the store of +// record and the locked archive — each event classified, serialized and hashed +// exactly as the relay does a live one — and verifies the copy by reading it +// back: every row present, every copy identical, per organization. Only with +// -confirm-drop, and only after this run verified every copied partition, it +// drops the copied monthly partitions, the way retention drops them; no row is +// ever deleted. +// +// Usage: +// +// audit-history-copy [-database-url "$DATABASE_URL"] [-through YYYY-MM] [-verify-only] [-confirm-drop] [-batch-size N] +// +// It reads the same AUDIT_* settings as the accounts service and runs only +// under a swap value: run it with the service's own environment, after the +// service has switched, so nothing writes audit_events any more. The +// connection principal must be a member of app_control_plane, the authority +// retention runs under. +// +// It is resumable: a run reads back what the store holds before writing, so a +// run after an interrupted one copies only what is missing. -verify-only +// copies nothing, for a re-check once the store has caught up. +// +// Exit status: 0 done; 1 failed; 2 usage; 3 verification failed, nothing +// dropped. +package main + +import ( + "context" + "errors" + "flag" + "fmt" + "io" + "os" + "sort" + "time" + + "accounts/pkg/auditstore/auditsink" + "accounts/pkg/business" + "accounts/pkg/infra" +) + +func main() { + os.Exit(run(os.Args[1:], os.Getenv, os.Stdout, os.Stderr)) +} + +// Writes to the process's stdout/stderr have no meaningful recovery. +func line(w io.Writer, a ...any) { _, _ = fmt.Fprintln(w, a...) } + +// options is a parsed command line. +type options struct { + databaseURL string + through *time.Time + batchSize int + copyOptions business.AuditHistoryCopyOptions +} + +// parse reads the command line and the sink settings, refusing everything a +// run cannot start from. +func parse(args []string, getenv func(string) string, stderr io.Writer) (options, *auditsink.Swap, int) { + fs := flag.NewFlagSet("audit-history-copy", flag.ContinueOnError) + fs.SetOutput(stderr) + databaseURL := fs.String("database-url", getenv("DATABASE_URL"), "Postgres connection URL (defaults to $DATABASE_URL)") + through := fs.String("through", "", "copy only the partitions up to and including this month, YYYY-MM (default every partition)") + verifyOnly := fs.Bool("verify-only", false, "copy nothing; verify what the store already holds") + confirmDrop := fs.Bool("confirm-drop", false, "after a verification pass, drop the copied partitions") + batchSize := fs.Int("batch-size", business.DefaultAuditRelayBatchSize, "events per copied batch") + if err := fs.Parse(args); err != nil { + return options{}, nil, 2 + } + if fs.NArg() > 0 { + line(stderr, "audit-history-copy: unexpected arguments:", fs.Args()) + return options{}, nil, 2 + } + opts := options{ + databaseURL: *databaseURL, + batchSize: *batchSize, + copyOptions: business.AuditHistoryCopyOptions{VerifyOnly: *verifyOnly, ConfirmDrop: *confirmDrop}, + } + if *through != "" { + monthStart, err := time.Parse("2006-01", *through) + if err != nil { + line(stderr, "audit-history-copy: -through must be a month, YYYY-MM; got", *through) + return options{}, nil, 2 + } + end := monthStart.AddDate(0, 1, 0) + opts.through = &end + } + if opts.databaseURL == "" { + line(stderr, "audit-history-copy: -database-url (or $DATABASE_URL) is required") + return options{}, nil, 2 + } + sink, err := auditsink.Load(getenv) + if err != nil { + line(stderr, "audit-history-copy:", err) + return options{}, nil, 1 + } + if sink.Swap == nil { + line(stderr, fmt.Sprintf("audit-history-copy: AUDIT_SINK is %s; the history copy runs only under a swap value, "+ + "after the service has switched, so nothing writes audit_events any more", sink.Mode)) + return options{}, nil, 1 + } + return opts, sink.Swap, 0 +} + +func run(args []string, getenv func(string) string, stdout, stderr io.Writer) int { + opts, swap, code := parse(args, getenv, stderr) + if swap == nil { + return code + } + ctx := context.Background() + store, err := infra.NewPostgresStoreFromURL(ctx, opts.databaseURL) + if err != nil { + line(stderr, "audit-history-copy:", err) + return 1 + } + defer store.Close() + opened, err := auditsink.Open(ctx, swap) + if err != nil { + line(stderr, "audit-history-copy:", err) + return 1 + } + defer opened.Close() + + copier, err := business.NewAuditHistoryCopy(business.AuditHistoryCopyConfig{ + Source: store, + Store: opened.Store, + Archive: opened.Archive, + ReadBack: opened.History, + Types: store, + DeploymentID: swap.DeploymentID, + ContentRetention: swap.ContentRetention, + BatchSize: opts.batchSize, + Through: opts.through, + Progress: func(message string) { line(stdout, message) }, + }) + if err != nil { + line(stderr, "audit-history-copy:", err) + return 2 + } + report, err := copier.Run(ctx, opts.copyOptions) + return summarize(stdout, stderr, report, err) +} + +// summarize prints what the run found and maps it to the exit status. Pure over +// its inputs so the mapping is testable without a database or a warehouse. +func summarize(stdout, stderr io.Writer, report business.AuditHistoryReport, err error) int { + for _, partition := range report.Partitions { + orgs := make([]string, 0, len(partition.Orgs)) + for org := range partition.Orgs { + orgs = append(orgs, org) + } + sort.Strings(orgs) + for _, org := range orgs { + count := partition.Orgs[org] + name := org + if name == "" { + name = "(platform)" + } + line(stdout, fmt.Sprintf(" %s %s: %d in Postgres, %d verified", partition.Partition.Name, name, count.Postgres, count.Verified)) + } + for _, problem := range partition.Problems { + line(stderr, fmt.Sprintf(" %s: %s", partition.Partition.Name, problem)) + } + if hidden := partition.Failures - len(partition.Problems); hidden > 0 { + line(stderr, fmt.Sprintf(" %s: and %d more", partition.Partition.Name, hidden)) + } + } + switch { + case errors.Is(err, business.ErrAuditHistoryUnverified): + line(stderr, "audit-history-copy:", err) + return 3 + case err != nil: + line(stderr, "audit-history-copy:", err) + return 1 + case report.Dropped > 0: + line(stdout, fmt.Sprintf("verified; dropped %d partitions ending at or before %s", report.Dropped, report.Cutoff.UTC().Format(time.RFC3339))) + default: + line(stdout, "verified") + } + return 0 +} diff --git a/module/services/accounts/code/cmd/audit-history-copy/main_test.go b/module/services/accounts/code/cmd/audit-history-copy/main_test.go new file mode 100644 index 000000000..ca08cf172 --- /dev/null +++ b/module/services/accounts/code/cmd/audit-history-copy/main_test.go @@ -0,0 +1,94 @@ +package main + +import ( + "bytes" + "errors" + "testing" + "time" + + "accounts/pkg/business" + + "github.com/stretchr/testify/require" +) + +func env(values map[string]string) func(string) string { + return func(name string) string { return values[name] } +} + +var swapEnv = map[string]string{ + "DATABASE_URL": "postgres://example.invalid/accounts", + "AUDIT_SINK": "bigquery", + "AUDIT_BIGQUERY_PROJECT": "example-project", + "AUDIT_BIGQUERY_DATASET": "audit", + "AUDIT_ARCHIVE_URL": "gs://example-audit-archive", + "AUDIT_DEPLOYMENT_ID": "deployment-1", + "AUDIT_CONTENT_RETENTION_DAYS": "90", +} + +func TestParseRunsOnlyUnderASwapValue(t *testing.T) { + for _, sink := range []string{"", "postgres", "both"} { + values := map[string]string{"DATABASE_URL": "postgres://example.invalid/accounts", "AUDIT_SINK": sink, "AUDIT_EXTERNAL_URL": "https://audit.example/ingest"} + var stderr bytes.Buffer + _, swap, code := parse(nil, env(values), &stderr) + require.Nil(t, swap, sink) + require.Equal(t, 1, code, sink) + require.Contains(t, stderr.String(), "runs only under a swap value", sink) + } + + var stderr bytes.Buffer + opts, swap, code := parse([]string{"-through", "2026-06", "-confirm-drop", "-batch-size", "200"}, env(swapEnv), &stderr) + require.Equal(t, 0, code, stderr.String()) + require.NotNil(t, swap) + require.Equal(t, "deployment-1", swap.DeploymentID) + require.Equal(t, time.Date(2026, 7, 1, 0, 0, 0, 0, time.UTC), *opts.through, "-through names the last month copied") + require.True(t, opts.copyOptions.ConfirmDrop) + require.False(t, opts.copyOptions.VerifyOnly) + require.Equal(t, 200, opts.batchSize) +} + +func TestParseRefusesWhatARunCannotStartFrom(t *testing.T) { + for name, tc := range map[string]struct { + args []string + values map[string]string + code int + want string + }{ + "a malformed month": {args: []string{"-through", "June"}, values: swapEnv, code: 2, want: "YYYY-MM"}, + "a stray argument": {args: []string{"now"}, values: swapEnv, code: 2, want: "unexpected arguments"}, + "no database": {values: map[string]string{"AUDIT_SINK": "bigquery"}, code: 2, want: "-database-url"}, + "a swap missing parts": {values: map[string]string{"DATABASE_URL": "postgres://example.invalid/a", "AUDIT_SINK": "bigquery"}, code: 1, want: "requires AUDIT_BIGQUERY_PROJECT"}, + } { + t.Run(name, func(t *testing.T) { + var stderr bytes.Buffer + _, swap, code := parse(tc.args, env(tc.values), &stderr) + require.Nil(t, swap) + require.Equal(t, tc.code, code) + require.Contains(t, stderr.String(), tc.want) + }) + } +} + +func TestSummarizeExitStatus(t *testing.T) { + partition := business.AuditHistoryPartitionReport{ + Partition: business.AuditHistoryPartition{Name: "audit_events_2026_08"}, + Orgs: map[string]business.AuditHistoryOrgCount{"": {Postgres: 2, Verified: 1}, "org-1": {Postgres: 3, Verified: 3}}, + Problems: []string{"event e-1: missing from the store"}, + Failures: 4, + } + var stdout, stderr bytes.Buffer + code := summarize(&stdout, &stderr, business.AuditHistoryReport{Partitions: []business.AuditHistoryPartitionReport{partition}}, business.ErrAuditHistoryUnverified) + require.Equal(t, 3, code) + require.Contains(t, stdout.String(), "audit_events_2026_08 (platform): 2 in Postgres, 1 verified") + require.Contains(t, stdout.String(), "audit_events_2026_08 org-1: 3 in Postgres, 3 verified") + require.Contains(t, stderr.String(), "missing from the store") + require.Contains(t, stderr.String(), "and 3 more") + + stdout.Reset() + require.Equal(t, 1, summarize(&stdout, &stderr, business.AuditHistoryReport{}, errors.New("warehouse unavailable"))) + require.Equal(t, 0, summarize(&stdout, &stderr, business.AuditHistoryReport{Verified: true}, nil)) + require.Contains(t, stdout.String(), "verified") + stdout.Reset() + cutoff := time.Date(2026, 9, 1, 0, 0, 0, 0, time.UTC) + require.Equal(t, 0, summarize(&stdout, &stderr, business.AuditHistoryReport{Verified: true, Dropped: 2, Cutoff: cutoff}, nil)) + require.Contains(t, stdout.String(), "dropped 2 partitions ending at or before 2026-09-01T00:00:00Z") +} diff --git a/module/services/accounts/code/pkg/auditstore/bigquerystore/emulator_test.go b/module/services/accounts/code/pkg/auditstore/bigquerystore/emulator_test.go index 126ec0bb6..a50c52fc0 100644 --- a/module/services/accounts/code/pkg/auditstore/bigquerystore/emulator_test.go +++ b/module/services/accounts/code/pkg/auditstore/bigquerystore/emulator_test.go @@ -10,10 +10,13 @@ import ( "accounts/pkg/business" "cloud.google.com/go/bigquery" + bqstorage "cloud.google.com/go/bigquery/storage/apiv1" "github.com/google/uuid" "github.com/stretchr/testify/require" "google.golang.org/api/iterator" "google.golang.org/api/option" + "google.golang.org/grpc" + "google.golang.org/grpc/credentials/insecure" ) // The adapter against a BigQuery emulator, when one is reachable: @@ -138,3 +141,107 @@ func TestEmulatorAppendRoutesRecordsByRetentionClass(t *testing.T) { } require.Equal(t, []string{content.Entry.ID}, details, "only the content event has a details row") } + +// emulatorReader is a reader on the emulator's Storage Read API, at +// AUDIT_BIGQUERY_EMULATOR_GRPC_HOST: +// +// docker run --rm -p 9050:9050 -p 9060:9060 ghcr.io/goccy/bigquery-emulator --project=test-project +// AUDIT_BIGQUERY_EMULATOR_HOST=localhost:9050 AUDIT_BIGQUERY_EMULATOR_GRPC_HOST=localhost:9060 go test ./pkg/auditstore/bigquerystore +// +// The emulator evaluates a read session's row restriction with a GoogleSQL +// engine, so this is where the restriction text the reader writes is parsed +// by something other than the fake that shares its author. +func emulatorReader(t *testing.T, dataset, deploymentID string) *Reader { + t.Helper() + host := strings.TrimSpace(os.Getenv("AUDIT_BIGQUERY_EMULATOR_GRPC_HOST")) + if host == "" { + t.Skip("AUDIT_BIGQUERY_EMULATOR_GRPC_HOST is not set; see emulatorReader for how to run against an emulator") + } + client, err := bqstorage.NewBigQueryReadClient(context.Background(), option.WithEndpoint(host), option.WithoutAuthentication(), + option.WithGRPCDialOption(grpc.WithTransportCredentials(insecure.NewCredentials()))) + require.NoError(t, err) + t.Cleanup(func() { _ = client.Close() }) + // The emulator serves one stream per session. + reader, err := NewReader(ReadConfig{Client: client, Project: "test-project", Dataset: dataset, DeploymentID: deploymentID, MaxStreams: 1}) + require.NoError(t, err) + return reader +} + +func TestEmulatorReadsThroughTheStorageReadAPI(t *testing.T) { + client := emulatorClient(t) + ctx := context.Background() + dataset := emulatorDataset(t, client) + reader := emulatorReader(t, dataset, "deployment-1") + store, err := New(client, Config{Dataset: dataset, ContentDetailRetention: 30 * 24 * time.Hour, Reader: reader}) + require.NoError(t, err) + require.NoError(t, store.Ensure(ctx)) + + org := "22222222-2222-2222-2222-222222222222" + security := record(t, business.RetentionSecurity, org) + content := record(t, business.RetentionContent, org) + content.Entry.EventType = business.EventDatasourceSourceSynced + content.Entry.CreatedAt = occurredAt.Add(time.Minute) + other := record(t, business.RetentionSecurity, "33333333-3333-3333-3333-333333333333") + batch := business.AuditBatch{ID: "batch-1", DeploymentID: "deployment-1", ComposedAt: occurredAt, + Records: []business.AuditRecord{security, content, other}} + require.NoError(t, store.AppendAuditBatch(ctx, batch)) + batch.ID = "batch-1-again" + require.NoError(t, store.AppendAuditBatch(ctx, batch), "a redelivery") + batch.ID, batch.DeploymentID = "batch-elsewhere", "deployment-2" + require.NoError(t, store.AppendAuditBatch(ctx, batch), "another deployment") + + from, to := occurredAt.Add(-time.Hour), occurredAt.Add(time.Hour) + read := business.AuditRead{Scope: business.OrganizationAuditScope(org), Query: business.AuditQuery{OrgID: org, From: &from, To: &to}} + entries, next, err := store.ListAuditEvents(ctx, read) + require.NoError(t, err) + require.Empty(t, next) + require.Len(t, entries, 2, "each event once, this organization and deployment only") + require.Equal(t, content.Entry.ID, entries[0].ID, "newest first") + require.Equal(t, map[string]any{"method": "password"}, entries[0].Payload, "content details are joined") + require.Equal(t, map[string]any{"method": "password"}, entries[1].Payload) + + read.Query.EventTypes = []string{string(business.EventAuthLogin), string(business.EventDatasourceSourceSynced)} + read.Query.ResourceID = "session-1" + buckets, err := store.AggregateAuditEvents(ctx, read, business.AuditAggregationSpec{GroupBy: []string{"event_type"}}) + require.NoError(t, err) + require.Len(t, buckets, 2) + for _, bucket := range buckets { + require.Equal(t, int64(1), bucket.Count, bucket.Key) + } + + exported, err := store.ExportAuditEvents(ctx, business.AuditRead{Scope: business.PlatformAuditScope(), Query: business.AuditQuery{ + PayloadContains: map[string]any{"method": "pass\"word\né"}, + }}) + require.NoError(t, err, "a literal with a quote, a newline and a non-ASCII rune parses") + require.Empty(t, exported) + exported, err = store.ExportAuditEvents(ctx, business.AuditRead{Scope: business.PlatformAuditScope()}) + require.NoError(t, err) + require.Len(t, exported, 3, "the platform read spans organizations, never deployments") + + copies := 0 + require.NoError(t, store.ReadStoredAuditEvents(ctx, from, to, func(event business.StoredAuditEvent) error { + copies++ + require.Equal(t, "deployment-1", event.DeploymentID) + require.True(t, event.HasDetails) + require.Equal(t, business.AuditDetailsSHA256(event.Details), event.DetailsSHA256) + return nil + })) + require.Equal(t, 6, copies, "the read-back sees every copy") + + // The server, not the service, applies the restriction: what it streams + // back is already narrowed, at microsecond precision, and a literal stays a + // literal. + served := func(build func(*restriction)) int { + where := &restriction{} + build(where) + n := 0 + require.NoError(t, reader.scan(ctx, EventsTable, eventFields, where, func(arrowRow) error { n++; return nil })) + return n + } + require.Equal(t, 9, served(func(*restriction) {})) + require.Equal(t, 6, served(func(w *restriction) { w.eq("deployment_id", "deployment-1") })) + require.Equal(t, 9, served(func(w *restriction) { w.timeBound("occurred_at", "<=", content.Entry.CreatedAt) })) + require.Equal(t, 6, served(func(w *restriction) { w.timeBound("occurred_at", "<", content.Entry.CreatedAt) })) + require.Equal(t, 3, served(func(w *restriction) { w.in("event_type", []string{string(business.EventDatasourceSourceSynced), "saas.none.none"}) })) + require.Zero(t, served(func(w *restriction) { w.eq("resource_id", `session-1" OR TRUE OR "`+"\n\u00e9\\") })) +} diff --git a/module/services/accounts/code/pkg/auditstore/bigquerystore/reader.go b/module/services/accounts/code/pkg/auditstore/bigquerystore/reader.go index 4aaf0827b..110bc8361 100644 --- a/module/services/accounts/code/pkg/auditstore/bigquerystore/reader.go +++ b/module/services/accounts/code/pkg/auditstore/bigquerystore/reader.go @@ -660,13 +660,16 @@ func (r *Reader) consume(rows storagepb.BigQueryRead_ReadRowsClient, schema []by return offset, err } batch := response.GetArrowRecordBatch() - if batch == nil || response.GetRowCount() == 0 { + if batch == nil || len(batch.GetSerializedRecordBatch()) == 0 { continue } - if err := decodeBatch(schema, batch.GetSerializedRecordBatch(), visit); err != nil { + // The offset to resume from counts the rows decoded, which is what the + // response's row count reports when it is set. + rows, err := decodeBatch(schema, batch.GetSerializedRecordBatch(), visit) + offset += int64(rows) + if err != nil { return offset, err } - offset += response.GetRowCount() } } @@ -687,14 +690,25 @@ func (e decodeError) Error() string { return e.err.Error() } func (e decodeError) Unwrap() error { return e.err } // decodeBatch decodes one serialized Arrow record batch against the session's -// serialized schema and hands each row to visit. A visit error is returned as -// it is; a decoding failure as a decodeError. -func decodeBatch(schema, batch []byte, visit func(arrowRow) error) error { - reader, err := ipc.NewReader(io.MultiReader(bytes.NewReader(schema), bytes.NewReader(batch)), ipc.WithAllocator(memory.DefaultAllocator)) +// serialized schema, hands each row to visit, and returns how many rows it +// visited. A visit error is returned as it is; a decoding failure as a +// decodeError. +// +// BigQuery sends the schema once, on the session, and each batch without it, +// so the two are read as one stream. A batch that carries its own schema is a +// whole stream already — the BigQuery emulator sends them so — and is read +// alone. +func decodeBatch(schema, batch []byte, visit func(arrowRow) error) (int, error) { + stream := io.MultiReader(bytes.NewReader(schema), bytes.NewReader(batch)) + if carriesSchema(batch) { + stream = bytes.NewReader(batch) + } + reader, err := ipc.NewReader(stream, ipc.WithAllocator(memory.DefaultAllocator)) if err != nil { - return decodeError{fmt.Errorf("decode arrow batch: %w", err)} + return 0, decodeError{fmt.Errorf("decode arrow batch: %w", err)} } defer reader.Release() + visited := 0 for reader.Next() { record := reader.Record() columns := make(map[string]arrow.Array, record.NumCols()) @@ -703,14 +717,26 @@ func decodeBatch(schema, batch []byte, visit func(arrowRow) error) error { } for i := 0; i < int(record.NumRows()); i++ { if err := visit(arrowRow{columns: columns, index: i}); err != nil { - return err + return visited, err } + visited++ } } if err := reader.Err(); err != nil { - return decodeError{fmt.Errorf("decode arrow batch: %w", err)} + return visited, decodeError{fmt.Errorf("decode arrow batch: %w", err)} } - return nil + return visited, nil +} + +// carriesSchema reports whether a serialized batch begins with a schema message. +func carriesSchema(batch []byte) bool { + messages := ipc.NewMessageReader(bytes.NewReader(batch)) + defer messages.Release() + message, err := messages.Message() + if err != nil { + return false + } + return message.Type() == ipc.MessageSchema } // arrowRow is one row of a decoded record batch. diff --git a/module/services/accounts/code/pkg/business/audit_history.go b/module/services/accounts/code/pkg/business/audit_history.go index 0a1a16fc2..3e1a9bcce 100644 --- a/module/services/accounts/code/pkg/business/audit_history.go +++ b/module/services/accounts/code/pkg/business/audit_history.go @@ -2,7 +2,13 @@ package business import ( "context" + "errors" + "fmt" + "sort" + "strings" "time" + + "github.com/google/uuid" ) // StoredAuditEvent is one copy of an event as a store of record holds it: the @@ -27,3 +33,469 @@ type StoredAuditEvent struct { type AuditHistoryReader interface { ReadStoredAuditEvents(ctx context.Context, from, to time.Time, visit func(StoredAuditEvent) error) error } + +// The one-time history copy of ADR 0009, item 7: switching a deployment from +// postgres to a swap value copies the audit_events rows it already holds into +// the store of record and the archive, verifies the copy by reading it back, +// and only then — and only when the operator confirms — drops the copied +// monthly partitions, the way retention drops them. No row is deleted, so the +// append-only triggers stay as they are. + +// AuditHistoryPartition is one monthly partition of audit_events: the events +// that occurred in [From, To). +type AuditHistoryPartition struct { + Name string + From, To time.Time +} + +// AuditHistoryCursor is a position in a partition's (created_at, id) order. +type AuditHistoryCursor struct { + CreatedAt time.Time + ID string +} + +// AuditHistorySource is audit_events as the history copy reads and retires it. +// Every method runs under WithControlPlane: the copy spans every organization. +type AuditHistorySource interface { + WithControlPlane(ctx context.Context, fn func(ctx context.Context) error) error + // ListAuditPartitions lists the monthly partitions, oldest first, with the + // bounds Postgres holds for them. + ListAuditPartitions(ctx context.Context) ([]AuditHistoryPartition, error) + // ReadAuditHistory returns at most limit events that occurred in + // [from, to), in (created_at, id) order and after the cursor when one is + // given — each exactly as the queue would carry it, its payload decoded + // with json.Number so its canonical details are the ones a live event of + // the same row would have. + ReadAuditHistory(ctx context.Context, from, to time.Time, after *AuditHistoryCursor, limit int) ([]AuditEntry, error) + // CountAuditHistory counts the events that occurred in [from, to) per + // organization ("" for the platform's). + CountAuditHistory(ctx context.Context, from, to time.Time) (map[string]int64, error) + // DropAuditPartitionsBefore drops every monthly partition whose whole range + // is before the cutoff — the mechanism RunRetention uses. + DropAuditPartitionsBefore(ctx context.Context, before time.Time) (int64, error) +} + +// AuditHistoryCopyConfig wires a history copy. +type AuditHistoryCopyConfig struct { + Source AuditHistorySource + Store AuditStoreWriter + Archive AuditArchive + ReadBack AuditHistoryReader + // Types serves the declared half of the registry, so each event is + // classified exactly as the relay classifies a live one. + Types DeclaredAuditEventTypeReader + // DeploymentID stamps every copied record, as the relay's. + DeploymentID string + // ContentRetention is the details window: a content-class event older than + // it has no details left to verify in the store. + ContentRetention time.Duration + // BatchSize is the most events one copied batch carries (default + // DefaultAuditRelayBatchSize). + BatchSize int + // Through, when set, limits the copy to the partitions that end at or + // before it, and is the drop's cutoff. Unset is every partition. + Through *time.Time + // Progress, when set, is told what the copy is doing. + Progress func(message string) + + // Now and NewBatchID are seams for tests; nil uses the clock and a UUID. + Now func() time.Time + NewBatchID func() string +} + +// AuditHistoryCopyOptions chooses what a run may do beyond verifying. +type AuditHistoryCopyOptions struct { + // VerifyOnly copies nothing: it verifies what the store already holds. + VerifyOnly bool + // ConfirmDrop drops the copied partitions once this run has verified every + // one of them. Without it nothing is ever dropped. + ConfirmDrop bool +} + +// AuditHistoryOrgCount is one organization's events in a partition: how many +// Postgres holds and how many of those the store holds, verified. +type AuditHistoryOrgCount struct { + Postgres int64 + Verified int64 +} + +// AuditHistoryPartitionReport is what a run found in one partition. +type AuditHistoryPartitionReport struct { + Partition AuditHistoryPartition + // Copied is how many events this run wrote; the rest were already there. + Copied int + // Orgs counts the partition's events per organization ("" is the + // platform's). + Orgs map[string]AuditHistoryOrgCount + // Problems are the verification failures found, the first + // maxAuditHistoryProblems of them. + Problems []string + // Failures is how many there were in all. + Failures int +} + +// AuditHistoryReport is what a run did. +type AuditHistoryReport struct { + Partitions []AuditHistoryPartitionReport + // Verified is true when every partition's events are in the store, each + // copy identical to its row. + Verified bool + // Cutoff is the drop's cutoff: every partition ending at or before it was + // copied and verified. + Cutoff time.Time + // Dropped is how many partitions were dropped; DropRefused says why none + // were when a drop was confirmed and did not happen. + Dropped int64 + DropRefused string +} + +// ErrAuditHistoryUnverified is returned when a run's verification failed; the +// report says where. +var ErrAuditHistoryUnverified = errors.New("audit history copy: verification failed; nothing was dropped") + +const maxAuditHistoryProblems = 20 + +// auditHistoryPageSize is the rows one read of audit_events returns. +const auditHistoryPageSize = 1000 + +// auditHistoryWindow is the span the copy reads, writes and verifies at a +// time: a day keeps the read-back of a busy deployment in memory. +const auditHistoryWindow = 24 * time.Hour + +// auditHistoryDetailsMargin is how far inside the content window a +// content-class event's details must still be in the store: the store expires +// details by whole day partitions, so the last day or two of the window may +// already be gone. +const auditHistoryDetailsMargin = 2 * 24 * time.Hour + +// AuditHistoryCopy is the one-time history copy. +type AuditHistoryCopy struct { + cfg AuditHistoryCopyConfig +} + +// NewAuditHistoryCopy validates cfg. +func NewAuditHistoryCopy(cfg AuditHistoryCopyConfig) (*AuditHistoryCopy, error) { + if cfg.Source == nil || cfg.Store == nil || cfg.Archive == nil || cfg.ReadBack == nil { + return nil, errors.New("audit history copy: source, store, archive and read-back are required") + } + if strings.TrimSpace(cfg.DeploymentID) == "" { + return nil, errors.New("audit history copy: deployment id is required") + } + if cfg.ContentRetention < 24*time.Hour { + return nil, errors.New("audit history copy: content retention must be at least one day") + } + if cfg.BatchSize == 0 { + cfg.BatchSize = DefaultAuditRelayBatchSize + } + if cfg.BatchSize < 1 || cfg.BatchSize > MaxAuditRelayBatchSize { + return nil, fmt.Errorf("audit history copy: batch size must be between 1 and %d", MaxAuditRelayBatchSize) + } + if cfg.Now == nil { + cfg.Now = time.Now + } + if cfg.NewBatchID == nil { + cfg.NewBatchID = func() string { return uuid.NewString() } + } + if cfg.Progress == nil { + cfg.Progress = func(string) {} + } + return &AuditHistoryCopy{cfg: cfg}, nil +} + +// Run copies, verifies and — when confirmed and verified — drops. It is +// resumable: a run reads back what the store already holds before it writes, +// copies only the events missing from it, and verifies every event, so a run +// after an interrupted one finishes what that one started. A verification +// failure blocks the drop and returns ErrAuditHistoryUnverified. +func (c *AuditHistoryCopy) Run(ctx context.Context, opts AuditHistoryCopyOptions) (AuditHistoryReport, error) { + var report AuditHistoryReport + partitions, err := c.partitions(ctx) + if err != nil { + return report, err + } + if len(partitions) == 0 { + report.Verified = true + c.cfg.Progress("no audit_events partitions to copy") + return report, nil + } + if c.cfg.Through != nil { + report.Cutoff = *c.cfg.Through + } else { + report.Cutoff = partitions[len(partitions)-1].To + } + resolver := NewAuditEventResolver(c.cfg.Types) + report.Verified = true + for _, partition := range partitions { + partitionReport, err := c.copyPartition(ctx, resolver, partition, opts) + report.Partitions = append(report.Partitions, partitionReport) + if err != nil { + report.Verified = false + return report, err + } + if partitionReport.Failures > 0 { + report.Verified = false + } + } + if !report.Verified { + return report, ErrAuditHistoryUnverified + } + if !opts.ConfirmDrop { + c.cfg.Progress("verified; nothing dropped without the drop confirmation") + return report, nil + } + if refused, err := c.drop(ctx, &report, partitions); err != nil || refused != "" { + report.DropRefused = refused + if err == nil { + err = fmt.Errorf("audit history copy: drop refused: %s", refused) + } + return report, err + } + return report, nil +} + +// partitions lists the partitions the run covers: every one, or those ending +// at or before Through. +func (c *AuditHistoryCopy) partitions(ctx context.Context) ([]AuditHistoryPartition, error) { + var all []AuditHistoryPartition + err := c.cfg.Source.WithControlPlane(ctx, func(ctx context.Context) error { + var err error + all, err = c.cfg.Source.ListAuditPartitions(ctx) + return err + }) + if err != nil { + return nil, fmt.Errorf("audit history copy: list partitions: %w", err) + } + var out []AuditHistoryPartition + for _, partition := range all { + if c.cfg.Through == nil || !partition.To.After(*c.cfg.Through) { + out = append(out, partition) + } + } + sort.Slice(out, func(i, j int) bool { return out[i].From.Before(out[j].From) }) + return out, nil +} + +// copyPartition copies and verifies one partition, a window at a time. +func (c *AuditHistoryCopy) copyPartition(ctx context.Context, resolver *AuditEventResolver, partition AuditHistoryPartition, opts AuditHistoryCopyOptions) (AuditHistoryPartitionReport, error) { + report := AuditHistoryPartitionReport{Partition: partition, Orgs: map[string]AuditHistoryOrgCount{}} + for from := partition.From; from.Before(partition.To); from = from.Add(auditHistoryWindow) { + to := from.Add(auditHistoryWindow) + if to.After(partition.To) { + to = partition.To + } + if err := c.copyWindow(ctx, resolver, from, to, opts, &report); err != nil { + return report, fmt.Errorf("audit history copy: %s [%s, %s): %w", partition.Name, + from.UTC().Format(time.RFC3339), to.UTC().Format(time.RFC3339), err) + } + } + var rows, verified int64 + for _, count := range report.Orgs { + rows += count.Postgres + verified += count.Verified + } + c.cfg.Progress(fmt.Sprintf("%s: %d events, %d copied by this run, %d verified, %d problems", + partition.Name, rows, report.Copied, verified, report.Failures)) + return report, nil +} + +// copyWindow reads a window's rows, reads back what the store holds of it, +// writes what is missing, and verifies every row. +func (c *AuditHistoryCopy) copyWindow(ctx context.Context, resolver *AuditEventResolver, from, to time.Time, opts AuditHistoryCopyOptions, report *AuditHistoryPartitionReport) error { + records, err := c.readWindow(ctx, resolver, from, to) + if err != nil { + return err + } + if len(records) == 0 { + return nil + } + stored, err := c.readBack(ctx, from, to) + if err != nil { + return err + } + if !opts.VerifyOnly { + var missing []AuditRecord + for _, record := range records { + if len(stored[record.Entry.ID]) == 0 { + missing = append(missing, record) + } + } + if len(missing) > 0 { + for start := 0; start < len(missing); start += c.cfg.BatchSize { + end := min(start+c.cfg.BatchSize, len(missing)) + if err := c.write(ctx, missing[start:end]); err != nil { + return err + } + report.Copied += end - start + } + if stored, err = c.readBack(ctx, from, to); err != nil { + return err + } + } + } + for _, record := range records { + org := record.Entry.OrgID + count := report.Orgs[org] + count.Postgres++ + if problem := c.verify(record, stored[record.Entry.ID]); problem != "" { + report.Failures++ + if len(report.Problems) < maxAuditHistoryProblems { + report.Problems = append(report.Problems, fmt.Sprintf("event %s: %s", record.Entry.ID, problem)) + } + } else { + count.Verified++ + } + report.Orgs[org] = count + } + return nil +} + +// readWindow reads a window's rows from audit_events, a page at a time, and +// classifies and hashes each exactly as the relay does a live event. +func (c *AuditHistoryCopy) readWindow(ctx context.Context, resolver *AuditEventResolver, from, to time.Time) ([]AuditRecord, error) { + var records []AuditRecord + var after *AuditHistoryCursor + for { + var page []AuditEntry + err := c.cfg.Source.WithControlPlane(ctx, func(ctx context.Context) error { + var err error + page, err = c.cfg.Source.ReadAuditHistory(ctx, from, to, after, auditHistoryPageSize) + return err + }) + if err != nil { + return nil, fmt.Errorf("read audit_events: %w", err) + } + for _, entry := range page { + resolved, err := resolver.Resolve(ctx, entry.EventType) + if err != nil { + return nil, err + } + record, err := NewAuditRecord(entry, resolved.RetentionClass()) + if err != nil { + return nil, err + } + records = append(records, record) + } + if len(page) < auditHistoryPageSize { + return records, nil + } + last := page[len(page)-1] + after = &AuditHistoryCursor{CreatedAt: last.CreatedAt, ID: last.ID} + } +} + +// readBack reads every copy the store holds of the window's events. +func (c *AuditHistoryCopy) readBack(ctx context.Context, from, to time.Time) (map[string][]StoredAuditEvent, error) { + stored := map[string][]StoredAuditEvent{} + err := c.cfg.ReadBack.ReadStoredAuditEvents(ctx, from, to, func(event StoredAuditEvent) error { + stored[event.Entry.ID] = append(stored[event.Entry.ID], event) + return nil + }) + if err != nil { + return nil, fmt.Errorf("read back the store: %w", err) + } + return stored, nil +} + +// write delivers one batch as the relay does: the archive first, then the +// store. A failure leaves the rest of the window for the next run. +func (c *AuditHistoryCopy) write(ctx context.Context, records []AuditRecord) error { + batch := AuditBatch{ + ID: c.cfg.NewBatchID(), + DeploymentID: c.cfg.DeploymentID, + ComposedAt: c.cfg.Now().UTC(), + Records: records, + } + if err := c.cfg.Archive.WriteAuditBatch(ctx, batch); err != nil { + return fmt.Errorf("archive batch %s: %w", batch.ID, err) + } + if err := c.cfg.Store.AppendAuditBatch(ctx, batch); err != nil { + return fmt.Errorf("append batch %s: %w", batch.ID, err) + } + return nil +} + +// verify reports what is wrong with the store's copies of record, or "". +// Every copy must carry the record's envelope, retention class and details +// hash, and whatever details it carries must hash to it; a security-class +// event's details must be there, and a content-class event's while it is +// inside the content window. +func (c *AuditHistoryCopy) verify(record AuditRecord, copies []StoredAuditEvent) string { + if len(copies) == 0 { + return "missing from the store" + } + want := record.Entry + for _, stored := range copies { + got := stored.Entry + switch { + case stored.DeploymentID != c.cfg.DeploymentID: + return fmt.Sprintf("stored under deployment %q", stored.DeploymentID) + case got.OrgID != want.OrgID || got.ActorID != want.ActorID || got.ActorType != want.ActorType || + got.EventType != want.EventType || got.SchemaVersion != want.SchemaVersion || + got.Resource != want.Resource || got.ResourceID != want.ResourceID || got.IPAddress != want.IPAddress || + got.ImpersonatedBy != want.ImpersonatedBy || got.IsImpersonated != want.IsImpersonated || + got.ClientID != want.ClientID || !got.CreatedAt.Equal(want.CreatedAt): + return "stored envelope differs from the row" + case stored.Retention != record.Retention: + return fmt.Sprintf("stored as %s, classified %s", stored.Retention, record.Retention) + case stored.DetailsSHA256 != record.DetailsSHA256: + return "stored details hash differs from the row's" + case stored.HasDetails && AuditDetailsSHA256(stored.Details) != record.DetailsSHA256: + return "stored details do not hash to the row's hash" + case !stored.HasDetails && record.Retention == RetentionSecurity: + return "security-class details are missing" + case !stored.HasDetails && want.CreatedAt.After(c.cfg.Now().Add(-c.cfg.ContentRetention+auditHistoryDetailsMargin)): + return "content details are missing inside the content window" + } + } + return "" +} + +// drop checks that nothing changed since verification and drops the copied +// partitions. It refuses — returning why — when a partition's rows changed, or +// when the cutoff would drop a partition this run did not verify. +func (c *AuditHistoryCopy) drop(ctx context.Context, report *AuditHistoryReport, verified []AuditHistoryPartition) (string, error) { + covered := map[string]bool{} + for _, partition := range verified { + covered[partition.Name] = true + } + var refused string + err := c.cfg.Source.WithControlPlane(ctx, func(ctx context.Context) error { + current, err := c.cfg.Source.ListAuditPartitions(ctx) + if err != nil { + return err + } + for _, partition := range current { + if !partition.To.After(report.Cutoff) && !covered[partition.Name] { + refused = fmt.Sprintf("partition %s ends before the cutoff and was not verified by this run", partition.Name) + return nil + } + } + for _, partitionReport := range report.Partitions { + counts, err := c.cfg.Source.CountAuditHistory(ctx, partitionReport.Partition.From, partitionReport.Partition.To) + if err != nil { + return err + } + for org, count := range partitionReport.Orgs { + if counts[org] != count.Postgres { + refused = fmt.Sprintf("partition %s changed since it was verified", partitionReport.Partition.Name) + return nil + } + } + for org, n := range counts { + if _, known := partitionReport.Orgs[org]; !known && n > 0 { + refused = fmt.Sprintf("partition %s changed since it was verified", partitionReport.Partition.Name) + return nil + } + } + } + report.Dropped, err = c.cfg.Source.DropAuditPartitionsBefore(ctx, report.Cutoff) + return err + }) + if err != nil { + return "", fmt.Errorf("audit history copy: drop: %w", err) + } + if refused == "" { + c.cfg.Progress(fmt.Sprintf("dropped %d partitions ending at or before %s", report.Dropped, report.Cutoff.UTC().Format(time.RFC3339))) + } + return refused, nil +} diff --git a/module/services/accounts/code/pkg/business/audit_history_test.go b/module/services/accounts/code/pkg/business/audit_history_test.go new file mode 100644 index 000000000..fcbb46894 --- /dev/null +++ b/module/services/accounts/code/pkg/business/audit_history_test.go @@ -0,0 +1,433 @@ +package business + +import ( + "context" + "errors" + "fmt" + "sort" + "testing" + "time" + + "github.com/stretchr/testify/require" +) + +// The history copy against in-memory stand-ins for audit_events, the store of +// record and the archive: what it copies, what it verifies, when it drops, and +// how a run after an interrupted one finishes the job. + +type historySource struct { + partitions []AuditHistoryPartition + rows []AuditEntry + drops []time.Time + // beforeCount runs before every count, so a test can change the table + // between verification and the drop. + beforeCount func(*historySource) +} + +func (s *historySource) WithControlPlane(ctx context.Context, fn func(context.Context) error) error { + return fn(ctx) +} + +func (s *historySource) ListAuditPartitions(context.Context) ([]AuditHistoryPartition, error) { + return append([]AuditHistoryPartition(nil), s.partitions...), nil +} + +func (s *historySource) ReadAuditHistory(_ context.Context, from, to time.Time, after *AuditHistoryCursor, limit int) ([]AuditEntry, error) { + sorted := append([]AuditEntry(nil), s.rows...) + sort.Slice(sorted, func(i, j int) bool { + if !sorted[i].CreatedAt.Equal(sorted[j].CreatedAt) { + return sorted[i].CreatedAt.Before(sorted[j].CreatedAt) + } + return sorted[i].ID < sorted[j].ID + }) + var out []AuditEntry + for _, row := range sorted { + if row.CreatedAt.Before(from) || !row.CreatedAt.Before(to) { + continue + } + if after != nil && (row.CreatedAt.Before(after.CreatedAt) || row.CreatedAt.Equal(after.CreatedAt) && row.ID <= after.ID) { + continue + } + out = append(out, row) + if len(out) == limit { + break + } + } + return out, nil +} + +func (s *historySource) CountAuditHistory(_ context.Context, from, to time.Time) (map[string]int64, error) { + if s.beforeCount != nil { + s.beforeCount(s) + } + counts := map[string]int64{} + for _, row := range s.rows { + if !row.CreatedAt.Before(from) && row.CreatedAt.Before(to) { + counts[row.OrgID]++ + } + } + return counts, nil +} + +func (s *historySource) DropAuditPartitionsBefore(_ context.Context, before time.Time) (int64, error) { + s.drops = append(s.drops, before) + var kept []AuditHistoryPartition + var dropped int64 + for _, partition := range s.partitions { + if partition.To.After(before) { + kept = append(kept, partition) + } else { + dropped++ + } + } + s.partitions = kept + return dropped, nil +} + +// historyStore is a store of record that keeps what it is given, as the +// BigQuery tables do: a content-class event's details beside its events row. +type historyStore struct { + stored []StoredAuditEvent + appends int + failAppend int // the 1-based append that fails; 0 never + tamper func(*StoredAuditEvent) +} + +func (s *historyStore) AppendAuditBatch(_ context.Context, batch AuditBatch) error { + s.appends++ + if s.appends == s.failAppend { + return errors.New("warehouse unavailable") + } + for _, record := range batch.Records { + entry := record.Entry + entry.Payload = nil + event := StoredAuditEvent{ + DeploymentID: batch.DeploymentID, Entry: entry, Retention: record.Retention, + DetailsSHA256: record.DetailsSHA256, Details: record.Details, HasDetails: true, + } + if s.tamper != nil { + s.tamper(&event) + } + s.stored = append(s.stored, event) + } + return nil +} + +func (s *historyStore) ReadStoredAuditEvents(_ context.Context, from, to time.Time, visit func(StoredAuditEvent) error) error { + for _, event := range s.stored { + if !event.Entry.CreatedAt.Before(from) && event.Entry.CreatedAt.Before(to) { + if err := visit(event); err != nil { + return err + } + } + } + return nil +} + +func (s *historyStore) copies() map[string]int { + out := map[string]int{} + for _, event := range s.stored { + out[event.Entry.ID]++ + } + return out +} + +type historyArchive struct{ batches []AuditBatch } + +func (a *historyArchive) WriteAuditBatch(_ context.Context, batch AuditBatch) error { + a.batches = append(a.batches, batch) + return nil +} + +var historyNow = time.Date(2026, 10, 3, 12, 0, 0, 0, time.UTC) + +func month(year int, m time.Month) AuditHistoryPartition { + from := time.Date(year, m, 1, 0, 0, 0, 0, time.UTC) + return AuditHistoryPartition{Name: fmt.Sprintf("audit_events_%04d_%02d", year, m), From: from, To: from.AddDate(0, 1, 0)} +} + +func historyRow(n int, org string, eventType EventType, at time.Time) AuditEntry { + return AuditEntry{ + ID: fmt.Sprintf("00000000-0000-4000-8000-%012d", n), OrgID: org, ActorID: "11111111-1111-4111-8111-111111111111", + ActorType: ActorTypeUser, EventType: eventType, SchemaVersion: 1, Resource: "datasource", + ResourceID: fmt.Sprintf("source-%d", n), CreatedAt: at, + Payload: map[string]any{"n": n, "repo": "acme/handbook"}, + } +} + +type historyFixture struct { + source *historySource + store *historyStore + archive *historyArchive +} + +func newHistoryFixture() *historyFixture { + orgA, orgB := "aaaaaaaa-0000-4000-8000-000000000001", "bbbbbbbb-0000-4000-8000-000000000002" + source := &historySource{partitions: []AuditHistoryPartition{month(2026, time.August), month(2026, time.September), month(2026, time.October)}} + n := 0 + for day := time.Date(2026, 8, 30, 6, 0, 0, 0, time.UTC); day.Before(time.Date(2026, 9, 3, 0, 0, 0, 0, time.UTC)); day = day.Add(10 * time.Hour) { + for _, org := range []string{orgA, orgB, ""} { + n++ + eventType := EventDatasourceSourceSynced // content + if n%3 == 0 { + eventType = EventAuthLogin // security + } + source.rows = append(source.rows, historyRow(n, org, eventType, day)) + } + } + return &historyFixture{source: source, store: &historyStore{}, archive: &historyArchive{}} +} + +func (f *historyFixture) copier(t *testing.T, cfg AuditHistoryCopyConfig) *AuditHistoryCopy { + t.Helper() + cfg.Source, cfg.Store, cfg.Archive, cfg.ReadBack = f.source, f.store, f.archive, f.store + cfg.DeploymentID = "deployment-1" + if cfg.ContentRetention == 0 { + cfg.ContentRetention = 400 * 24 * time.Hour + } + if cfg.BatchSize == 0 { + cfg.BatchSize = 4 + } + cfg.Now = func() time.Time { return historyNow } + copier, err := NewAuditHistoryCopy(cfg) + require.NoError(t, err) + return copier +} + +func TestAuditHistoryCopyCopiesVerifiesAndDropsOnlyWhenConfirmed(t *testing.T) { + f := newHistoryFixture() + report, err := f.copier(t, AuditHistoryCopyConfig{}).Run(context.Background(), AuditHistoryCopyOptions{}) + require.NoError(t, err) + require.True(t, report.Verified) + require.Empty(t, f.source.drops, "nothing is dropped without the confirmation") + require.Equal(t, time.Date(2026, 11, 1, 0, 0, 0, 0, time.UTC), report.Cutoff) + + copies := f.store.copies() + require.Len(t, copies, len(f.source.rows)) + for _, row := range f.source.rows { + require.Equal(t, 1, copies[row.ID], "copied once: %s", row.ID) + } + var orgs = map[string]int64{} + for _, partition := range report.Partitions { + for org, count := range partition.Orgs { + require.Equal(t, count.Postgres, count.Verified) + orgs[org] += count.Postgres + } + } + require.Equal(t, int64(len(f.source.rows)/3), orgs[""], "platform events are copied too") + + // Each copy carries the envelope, class and hash a live event of the row gets. + resolver := NewAuditEventResolver(nil) + for _, batch := range f.archive.batches { + require.Equal(t, "deployment-1", batch.DeploymentID) + require.LessOrEqual(t, len(batch.Records), 4) + for _, record := range batch.Records { + resolved, err := resolver.Resolve(context.Background(), record.Entry.EventType) + require.NoError(t, err) + want, err := NewAuditRecord(record.Entry, resolved.RetentionClass()) + require.NoError(t, err) + require.Equal(t, want.DetailsSHA256, record.DetailsSHA256) + require.Equal(t, want.Retention, record.Retention) + } + } + + // A second run finds everything there: it copies nothing, verifies, and with + // the confirmation drops every partition up to the cutoff. + appends := f.store.appends + report, err = f.copier(t, AuditHistoryCopyConfig{}).Run(context.Background(), AuditHistoryCopyOptions{ConfirmDrop: true}) + require.NoError(t, err) + require.Equal(t, appends, f.store.appends, "a resumed run copies nothing already there") + require.Equal(t, []time.Time{report.Cutoff}, f.source.drops) + require.Equal(t, int64(3), report.Dropped) + require.Empty(t, f.source.partitions) +} + +func TestAuditHistoryCopyResumesAfterAnInterruption(t *testing.T) { + f := newHistoryFixture() + f.store.failAppend = 3 + _, err := f.copier(t, AuditHistoryCopyConfig{}).Run(context.Background(), AuditHistoryCopyOptions{ConfirmDrop: true}) + require.ErrorContains(t, err, "warehouse unavailable") + require.Empty(t, f.source.drops, "an interrupted run drops nothing") + partial := len(f.store.copies()) + require.Positive(t, partial) + require.Less(t, partial, len(f.source.rows)) + + f.store.failAppend = 0 + report, err := f.copier(t, AuditHistoryCopyConfig{}).Run(context.Background(), AuditHistoryCopyOptions{ConfirmDrop: true}) + require.NoError(t, err) + require.True(t, report.Verified) + copied := 0 + for _, partition := range report.Partitions { + copied += partition.Copied + } + require.Equal(t, len(f.source.rows)-partial, copied, "the second run copies only what the first did not") + for id, n := range f.store.copies() { + require.Equal(t, 1, n, "event %s is in the store once", id) + } + archived := map[string]int{} + for _, batch := range f.archive.batches { + for _, record := range batch.Records { + archived[record.Entry.ID]++ + } + } + require.Len(t, archived, len(f.source.rows)) + twice := 0 + for _, n := range archived { + if n == 2 { + twice++ + } + } + require.Equal(t, 4, twice, "the batch archived before its append failed is archived again, as a relay retry would") + require.Equal(t, int64(3), report.Dropped) +} + +func TestAuditHistoryVerificationFailureBlocksTheDrop(t *testing.T) { + for name, tc := range map[string]struct { + prepare func(*historyFixture) + opts AuditHistoryCopyOptions + want string + }{ + "an event the store does not hold": { + opts: AuditHistoryCopyOptions{VerifyOnly: true, ConfirmDrop: true}, + want: "missing from the store", + }, + "a hash that differs": { + prepare: func(f *historyFixture) { + f.store.tamper = func(e *StoredAuditEvent) { + if e.Entry.ID == f.source.rows[4].ID { + e.DetailsSHA256 = AuditDetailsSHA256("{}") + } + } + }, + opts: AuditHistoryCopyOptions{ConfirmDrop: true}, + want: "details hash differs", + }, + "details that do not hash to it": { + prepare: func(f *historyFixture) { + f.store.tamper = func(e *StoredAuditEvent) { + if e.Entry.ID == f.source.rows[2].ID { + e.Details = `{"n":0}` + } + } + }, + opts: AuditHistoryCopyOptions{ConfirmDrop: true}, + want: "do not hash", + }, + "an envelope that differs": { + prepare: func(f *historyFixture) { + f.store.tamper = func(e *StoredAuditEvent) { + if e.Entry.ID == f.source.rows[1].ID { + e.Entry.OrgID = "cccccccc-0000-4000-8000-000000000003" + } + } + }, + opts: AuditHistoryCopyOptions{ConfirmDrop: true}, + want: "envelope differs", + }, + "content details missing inside the window": { + prepare: func(f *historyFixture) { + f.store.tamper = func(e *StoredAuditEvent) { + if e.Retention == RetentionContent { + e.Details, e.HasDetails = "", false + } + } + }, + opts: AuditHistoryCopyOptions{ConfirmDrop: true}, + want: "content details are missing", + }, + } { + t.Run(name, func(t *testing.T) { + f := newHistoryFixture() + if tc.prepare != nil { + tc.prepare(f) + } + report, err := f.copier(t, AuditHistoryCopyConfig{}).Run(context.Background(), tc.opts) + require.ErrorIs(t, err, ErrAuditHistoryUnverified) + require.False(t, report.Verified) + require.Empty(t, f.source.drops) + var problems []string + for _, partition := range report.Partitions { + problems = append(problems, partition.Problems...) + } + require.NotEmpty(t, problems) + require.Contains(t, problems[0], tc.want) + }) + } +} + +func TestAuditHistoryContentDetailsPastTheWindowAreNotRequired(t *testing.T) { + f := newHistoryFixture() + f.store.tamper = func(e *StoredAuditEvent) { + if e.Retention == RetentionContent { + e.Details, e.HasDetails = "", false // expired with their day partition + } + } + report, err := f.copier(t, AuditHistoryCopyConfig{ContentRetention: 7 * 24 * time.Hour}).Run(context.Background(), AuditHistoryCopyOptions{ConfirmDrop: true}) + require.NoError(t, err) + require.True(t, report.Verified) + require.Equal(t, int64(3), report.Dropped) +} + +func TestAuditHistoryDropIsRefusedWhenAPartitionChangedSinceVerification(t *testing.T) { + f := newHistoryFixture() + f.source.beforeCount = func(s *historySource) { + s.beforeCount = nil + s.rows = append(s.rows, historyRow(999, "", EventAuthLogin, time.Date(2026, 9, 2, 0, 0, 0, 0, time.UTC))) + } + report, err := f.copier(t, AuditHistoryCopyConfig{}).Run(context.Background(), AuditHistoryCopyOptions{ConfirmDrop: true}) + require.ErrorContains(t, err, "changed since it was verified") + require.True(t, report.Verified) + require.Contains(t, report.DropRefused, "audit_events_2026_09") + require.Empty(t, f.source.drops) +} + +func TestAuditHistoryThroughLimitsThePartitionsAndTheCutoff(t *testing.T) { + f := newHistoryFixture() + through := time.Date(2026, 9, 1, 0, 0, 0, 0, time.UTC) + report, err := f.copier(t, AuditHistoryCopyConfig{Through: &through}).Run(context.Background(), AuditHistoryCopyOptions{ConfirmDrop: true}) + require.NoError(t, err) + require.Len(t, report.Partitions, 1) + require.Equal(t, "audit_events_2026_08", report.Partitions[0].Partition.Name) + require.Equal(t, []time.Time{through}, f.source.drops) + for _, event := range f.store.stored { + require.True(t, event.Entry.CreatedAt.Before(through), "only August is copied") + } + + // A partition the run did not verify that the cutoff would drop refuses it. + f = newHistoryFixture() + f.source.partitions = append([]AuditHistoryPartition{{Name: "audit_events_2026_07", From: month(2026, time.July).From, To: month(2026, time.July).To}}, f.source.partitions...) + copier := f.copier(t, AuditHistoryCopyConfig{Through: &through}) + copier.cfg.Source = &listingAfterVerification{historySource: f.source, extra: AuditHistoryPartition{Name: "audit_events_2026_06", From: month(2026, time.June).From, To: month(2026, time.June).To}} + _, err = copier.Run(context.Background(), AuditHistoryCopyOptions{ConfirmDrop: true}) + require.ErrorContains(t, err, "audit_events_2026_06 ends before the cutoff and was not verified") + require.Empty(t, f.source.drops) +} + +// listingAfterVerification lists one more partition on every listing after the +// first, as a partition created while the copy ran would appear. +type listingAfterVerification struct { + *historySource + extra AuditHistoryPartition + listed int +} + +func (l *listingAfterVerification) ListAuditPartitions(ctx context.Context) ([]AuditHistoryPartition, error) { + l.listed++ + partitions, err := l.historySource.ListAuditPartitions(ctx) + if l.listed > 1 { + partitions = append(partitions, l.extra) + } + return partitions, err +} + +func TestAuditHistoryVerifyOnlyWritesNothing(t *testing.T) { + f := newHistoryFixture() + _, err := f.copier(t, AuditHistoryCopyConfig{}).Run(context.Background(), AuditHistoryCopyOptions{}) + require.NoError(t, err) + appends, archived := f.store.appends, len(f.archive.batches) + report, err := f.copier(t, AuditHistoryCopyConfig{}).Run(context.Background(), AuditHistoryCopyOptions{VerifyOnly: true}) + require.NoError(t, err) + require.True(t, report.Verified) + require.Equal(t, appends, f.store.appends) + require.Len(t, f.archive.batches, archived) +} diff --git a/module/services/accounts/code/pkg/business/audit_store_parity_test.go b/module/services/accounts/code/pkg/business/audit_store_parity_test.go index 5673a32a7..34bca2ed9 100644 --- a/module/services/accounts/code/pkg/business/audit_store_parity_test.go +++ b/module/services/accounts/code/pkg/business/audit_store_parity_test.go @@ -4,7 +4,10 @@ package business_test import ( "context" + "errors" "fmt" + "os" + "strings" "testing" "time" @@ -16,10 +19,14 @@ import ( "accounts/pkg/infra" "cloud.google.com/go/bigquery" + bqstorage "cloud.google.com/go/bigquery/storage/apiv1" "github.com/google/uuid" "github.com/stretchr/testify/require" + "google.golang.org/api/iterator" "google.golang.org/api/option" + "google.golang.org/grpc" "google.golang.org/grpc/codes" + "google.golang.org/grpc/credentials/insecure" "google.golang.org/grpc/status" ) @@ -54,7 +61,8 @@ type parityStore struct { // that needs a server the environment does not provide returns nil, after // logging how to provide one. var parityStores = map[string]func(t *testing.T) *parityStore{ - "bigquery": newBigQueryParityStore, + "bigquery": newBigQueryParityStore, + "bigquery-emulator": newBigQueryEmulatorParityStore, } // bigQueryAppender appends a batch's rows to the fake, as the BigQuery writer @@ -106,6 +114,62 @@ func newBigQueryParityStore(t *testing.T) *parityStore { } } +// newBigQueryEmulatorParityStore is the BigQuery store against the BigQuery +// emulator: rows streamed through tabledata.insertAll, and read back through +// its Storage Read API, which evaluates the row restrictions with a GoogleSQL +// engine. It runs when both of its addresses are set: +// +// docker run --rm -p 9050:9050 -p 9060:9060 ghcr.io/goccy/bigquery-emulator --project=test-project +// AUDIT_BIGQUERY_EMULATOR_HOST=localhost:9050 AUDIT_BIGQUERY_EMULATOR_GRPC_HOST=localhost:9060 +// +// The project is the one the bigquerystore emulator tests use, so one emulator +// serves both. +func newBigQueryEmulatorParityStore(t *testing.T) *parityStore { + const project = "test-project" + rest, grpcHost := os.Getenv("AUDIT_BIGQUERY_EMULATOR_HOST"), os.Getenv("AUDIT_BIGQUERY_EMULATOR_GRPC_HOST") + if rest == "" || grpcHost == "" { + t.Log("bigquery-emulator: AUDIT_BIGQUERY_EMULATOR_HOST and AUDIT_BIGQUERY_EMULATOR_GRPC_HOST are not set; not run") + return nil + } + client, err := bigquery.NewClient(testCtx, project, option.WithEndpoint("http://"+rest), option.WithoutAuthentication()) + require.NoError(t, err) + t.Cleanup(func() { _ = client.Close() }) + dataset := "parity_" + strings.ReplaceAll(uuid.NewString(), "-", "")[:12] + require.NoError(t, client.Dataset(dataset).Create(testCtx, &bigquery.DatasetMetadata{})) + t.Cleanup(func() { _ = client.Dataset(dataset).DeleteWithContents(context.Background()) }) + readClient, err := bqstorage.NewBigQueryReadClient(testCtx, option.WithEndpoint(grpcHost), option.WithoutAuthentication(), + option.WithGRPCDialOption(grpc.WithTransportCredentials(insecure.NewCredentials()))) + require.NoError(t, err) + t.Cleanup(func() { _ = readClient.Close() }) + // The emulator serves one stream per session. + reader, err := bigquerystore.NewReader(bigquerystore.ReadConfig{ + Client: readClient, Project: project, Dataset: dataset, DeploymentID: parityDeployment, MaxStreams: 1, + }) + require.NoError(t, err) + store, err := bigquerystore.New(client, bigquerystore.Config{Dataset: dataset, ContentDetailRetention: 30 * 24 * time.Hour, Reader: reader}) + require.NoError(t, err) + require.NoError(t, store.Ensure(testCtx)) + return &parityStore{ + writer: store, + store: store, + eventIDs: func(t *testing.T) []string { + var ids []string + rows := client.Dataset(dataset).Table(bigquerystore.EventsTable).Read(testCtx) + for { + var row struct { + EventID string `bigquery:"event_id"` + } + err := rows.Next(&row) + if errors.Is(err, iterator.Done) { + return ids + } + require.NoError(t, err) + ids = append(ids, row.EventID) + } + }, + } +} + // fanOut appends every batch to every store under test, and remembers it. type fanOut struct { writers []business.AuditStoreWriter diff --git a/module/services/accounts/code/pkg/infra/postgres_audit_history.go b/module/services/accounts/code/pkg/infra/postgres_audit_history.go new file mode 100644 index 000000000..bc61fba8d --- /dev/null +++ b/module/services/accounts/code/pkg/infra/postgres_audit_history.go @@ -0,0 +1,132 @@ +package infra + +import ( + "context" + "fmt" + "time" + + "accounts/pkg/business" +) + +// audit_events as the one-time history copy reads and retires it +// (business.AuditHistorySource). Every method runs on the control-plane +// transaction the caller opened: the copy spans every organization. + +// auditPartitionsSQL lists the monthly partitions of audit_events — the ones +// audit_events_ensure_partition creates and audit_events_drop_partitions_before +// drops — with the bounds Postgres holds for each, parsed back from its own +// rendering of them so no time zone is assumed. +const auditPartitionsSQL = ` + SELECT c.relname, + (regexp_match(pg_get_expr(c.relpartbound, c.oid), 'FROM \(''([^'']+)''\) TO \(''([^'']+)''\)'))[1]::timestamptz, + (regexp_match(pg_get_expr(c.relpartbound, c.oid), 'FROM \(''([^'']+)''\) TO \(''([^'']+)''\)'))[2]::timestamptz + FROM pg_inherits i + JOIN pg_class c ON c.oid = i.inhrelid + JOIN pg_class p ON p.oid = i.inhparent + WHERE p.oid = 'public.audit_events'::regclass + AND c.relname ~ '^audit_events_[0-9]{4}_[0-9]{2}$' + ORDER BY 2` + +// ListAuditPartitions implements business.AuditHistorySource. +func (s *PostgresStore) ListAuditPartitions(ctx context.Context) ([]business.AuditHistoryPartition, error) { + rows, err := s.getQueryExecutor(ctx).Query(ctx, auditPartitionsSQL) + if err != nil { + return nil, err + } + defer rows.Close() + var out []business.AuditHistoryPartition + for rows.Next() { + var partition business.AuditHistoryPartition + var from, to *time.Time + if err := rows.Scan(&partition.Name, &from, &to); err != nil { + return nil, err + } + if from == nil || to == nil { + return nil, fmt.Errorf("audit partition %s has no range bounds", partition.Name) + } + partition.From, partition.To = from.UTC(), to.UTC() + out = append(out, partition) + } + return out, rows.Err() +} + +// auditHistorySelectSQL reads rows as the queue does (auditQueueSelectSQL): +// every nullable column as text, so an entry carries exactly what a live +// event of the same row would. +const auditHistorySelectSQL = ` + SELECT id::text, event_type, schema_version, + COALESCE(actor_id::text, ''), actor_type, resource, + COALESCE(resource_id, ''), COALESCE(org_id::text, ''), payload, + COALESCE(ip_address, ''), created_at, + COALESCE(impersonated_by::text, ''), is_impersonated, + COALESCE(client_id, '') + FROM public.audit_events + WHERE created_at >= $1 AND created_at < $2` + +// ReadAuditHistory implements business.AuditHistorySource. +func (s *PostgresStore) ReadAuditHistory(ctx context.Context, from, to time.Time, after *business.AuditHistoryCursor, limit int) ([]business.AuditEntry, error) { + if limit < 1 { + return nil, fmt.Errorf("audit history: limit must be positive, got %d", limit) + } + query := auditHistorySelectSQL + args := []any{from, to} + if after != nil { + query += ` AND (created_at, id) > ($3, $4::uuid)` + args = append(args, after.CreatedAt, after.ID) + } + query += fmt.Sprintf(` ORDER BY created_at, id LIMIT $%d`, len(args)+1) + args = append(args, limit) + rows, err := s.getQueryExecutor(ctx).Query(ctx, query, args...) + if err != nil { + return nil, err + } + defer rows.Close() + var out []business.AuditEntry + for rows.Next() { + var entry business.AuditEntry + var eventType string + var payload []byte + if err := rows.Scan( + &entry.ID, &eventType, &entry.SchemaVersion, + &entry.ActorID, &entry.ActorType, &entry.Resource, + &entry.ResourceID, &entry.OrgID, &payload, + &entry.IPAddress, &entry.CreatedAt, + &entry.ImpersonatedBy, &entry.IsImpersonated, + &entry.ClientID, + ); err != nil { + return nil, err + } + entry.EventType = business.EventType(eventType) + entry.CreatedAt = entry.CreatedAt.UTC() + if entry.Payload, err = decodeQueuedPayload(payload); err != nil { + return nil, fmt.Errorf("audit history: payload of event %s: %w", entry.ID, err) + } + out = append(out, entry) + } + return out, rows.Err() +} + +// CountAuditHistory implements business.AuditHistorySource. +func (s *PostgresStore) CountAuditHistory(ctx context.Context, from, to time.Time) (map[string]int64, error) { + rows, err := s.getQueryExecutor(ctx).Query(ctx, ` + SELECT COALESCE(org_id::text, ''), count(*) + FROM public.audit_events + WHERE created_at >= $1 AND created_at < $2 + GROUP BY 1`, from, to) + if err != nil { + return nil, err + } + defer rows.Close() + counts := map[string]int64{} + for rows.Next() { + var org string + var n int64 + if err := rows.Scan(&org, &n); err != nil { + return nil, err + } + counts[org] = n + } + return counts, rows.Err() +} + +var _ business.AuditHistorySource = (*PostgresStore)(nil) diff --git a/module/services/accounts/code/pkg/infra/postgres_audit_history_test.go b/module/services/accounts/code/pkg/infra/postgres_audit_history_test.go new file mode 100644 index 000000000..0fe020074 --- /dev/null +++ b/module/services/accounts/code/pkg/infra/postgres_audit_history_test.go @@ -0,0 +1,208 @@ +//go:build !pure + +package infra_test + +import ( + "context" + "fmt" + "testing" + "time" + + "accounts/pkg/auditstore/bigqueryfake" + "accounts/pkg/auditstore/bigquerystore" + "accounts/pkg/business" + + "github.com/google/uuid" + "github.com/stretchr/testify/require" +) + +// The history copy against the real audit_events (ADR 0009, item 7): its +// partitions and their bounds, its rows read back as the queue reads them, and +// a whole run — copy into the warehouse (the fake Storage Read API), verify by +// reading back, then drop — over a month no install provisions, so the run's +// drop reaches nothing else. A run interrupted before its drop leaves that +// month behind with its rows (they cannot be deleted); the next run's copy +// takes them along and retires the month, so assertions name this run's rows. + +const historyMonth = "2002-03-01" + +func historyPartitionExists(t *testing.T) bool { + t.Helper() + var exists bool + require.NoError(t, testStore.WithControlPlane(testCtx, func(ctx context.Context) error { + return txFromCtx(t, ctx).QueryRow(ctx, `SELECT to_regclass('public.audit_events_2002_03') IS NOT NULL`).Scan(&exists) + })) + return exists +} + +// seedHistoryMonth provisions the month and writes events into it through the +// emitter, as a postgres deployment wrote them. +func seedHistoryMonth(t *testing.T) []business.AuditEntry { + t.Helper() + require.NoError(t, testStore.WithControlPlane(testCtx, func(ctx context.Context) error { + _, err := txFromCtx(t, ctx).Exec(ctx, `SELECT audit_events_ensure_partition($1::date)`, historyMonth) + return err + })) + emitter, err := business.NewDurableAuditEmitter(testStore, testStore) + require.NoError(t, err) + orgs := []string{uuid.NewString(), uuid.NewString(), ""} + base := time.Date(2002, 3, 30, 22, 0, 0, 0, time.UTC) + var entries []business.AuditEntry + for i := 0; i < 9; i++ { + org := orgs[i%3] + eventType := business.EventDatasourceSourceSynced + if i%2 == 0 { + eventType = business.EventAuthLogin + } + entry := business.AuditEntry{ + ID: uuid.NewString(), OrgID: org, ActorID: uuid.NewString(), ActorType: business.ActorTypeUser, + EventType: eventType, SchemaVersion: 1, Resource: "datasource", ResourceID: fmt.Sprintf("source-%d", i), + // Two events share an instant, so the read pages on (created_at, id). + CreatedAt: base.Add(time.Duration(i/2) * 90 * time.Minute), + Payload: map[string]any{"n": i, "ratio": 1.50, "label": ""}, + } + write := func(ctx context.Context) error { return emitter.EmitTx(ctx, entry) } + if org == "" { + require.NoError(t, testStore.WithControlPlane(testCtx, write)) + } else { + require.NoError(t, testStore.WithOrgTx(testCtx, org, write)) + } + entries = append(entries, entry) + } + return entries +} + +func TestAuditHistorySourceReadsPartitionsAndRowsAsTheQueueDoes(t *testing.T) { + entries := seedHistoryMonth(t) + from := time.Date(2002, 3, 1, 0, 0, 0, 0, time.UTC) + to := from.AddDate(0, 1, 0) + require.NoError(t, testStore.WithControlPlane(testCtx, func(ctx context.Context) error { + partitions, err := testStore.ListAuditPartitions(ctx) + require.NoError(t, err) + found := false + for i, partition := range partitions { + if i > 0 { + require.False(t, partition.From.Before(partitions[i-1].From), "oldest first") + } + if partition.Name == "audit_events_2002_03" { + found = true + require.Equal(t, from, partition.From) + require.Equal(t, to, partition.To) + } + } + require.True(t, found) + + // Paged two at a time on (created_at, id), every row once, in order. + var read []business.AuditEntry + var after *business.AuditHistoryCursor + for { + page, err := testStore.ReadAuditHistory(ctx, from, to, after, 2) + require.NoError(t, err) + read = append(read, page...) + if len(page) < 2 { + break + } + last := page[len(page)-1] + after = &business.AuditHistoryCursor{CreatedAt: last.CreatedAt, ID: last.ID} + } + byID := map[string]business.AuditEntry{} + for i, entry := range read { + if i > 0 { + previous := read[i-1] + require.True(t, previous.CreatedAt.Before(entry.CreatedAt) || previous.CreatedAt.Equal(entry.CreatedAt) && previous.ID < entry.ID) + } + byID[entry.ID] = entry + } + for _, want := range entries { + got, ok := byID[want.ID] + require.True(t, ok, "every row is read once") + require.Equal(t, want.OrgID, got.OrgID) + require.Equal(t, want.CreatedAt, got.CreatedAt) + record, err := business.NewAuditRecord(got, business.RetentionContent) + require.NoError(t, err) + require.Equal(t, `{"label":"","n":`+fmt.Sprint(want.Payload["n"])+`,"ratio":1.5}`, record.Details, + "the canonical details of the row, as the queue's are") + } + + counts, err := testStore.CountAuditHistory(ctx, from, to) + require.NoError(t, err) + require.Equal(t, int64(3), counts[entries[0].OrgID]) + require.Equal(t, int64(3), counts[entries[1].OrgID]) + require.GreaterOrEqual(t, counts[""], int64(3), "platform events count under no organization") + return nil + })) + t.Run("the copy", func(t *testing.T) { runHistoryCopy(t, entries) }) +} + +// historyWarehouse appends to the fake as the BigQuery writer streams. +type historyWarehouse struct{ fake *bigqueryfake.Server } + +func (w historyWarehouse) AppendAuditBatch(_ context.Context, batch business.AuditBatch) error { + events, details := bigquerystore.BatchRows(batch) + for _, row := range events { + if err := w.fake.Insert(bigqueryfake.TablePath("history-project", "audit", bigquerystore.EventsTable), row.Values); err != nil { + return err + } + } + for _, row := range details { + if err := w.fake.Insert(bigqueryfake.TablePath("history-project", "audit", bigquerystore.DetailsTable), row.Values); err != nil { + return err + } + } + return nil +} + +type historyArchive struct{ records int } + +func (a *historyArchive) WriteAuditBatch(_ context.Context, batch business.AuditBatch) error { + a.records += len(batch.Records) + return nil +} + +func runHistoryCopy(t *testing.T, entries []business.AuditEntry) { + fake := bigqueryfake.New() + fake.CreateTable(bigqueryfake.TablePath("history-project", "audit", bigquerystore.EventsTable), bigquerystore.EventsSchema()) + fake.CreateTable(bigqueryfake.TablePath("history-project", "audit", bigquerystore.DetailsTable), bigquerystore.DetailsSchema()) + reader, err := bigquerystore.NewReader(bigquerystore.ReadConfig{Client: fake, Project: "history-project", Dataset: "audit", DeploymentID: "deployment-history"}) + require.NoError(t, err) + archive := &historyArchive{} + through := time.Date(2002, 4, 1, 0, 0, 0, 0, time.UTC) + copier, err := business.NewAuditHistoryCopy(business.AuditHistoryCopyConfig{ + Source: testStore, Store: historyWarehouse{fake: fake}, Archive: archive, ReadBack: reader, Types: testStore, + DeploymentID: "deployment-history", ContentRetention: 365 * 24 * time.Hour, BatchSize: 2, Through: &through, + }) + require.NoError(t, err) + + report, err := copier.Run(testCtx, business.AuditHistoryCopyOptions{}) + require.NoError(t, err) + require.True(t, report.Verified) + require.True(t, historyPartitionExists(t), "nothing is dropped without the confirmation") + require.GreaterOrEqual(t, archive.records, len(entries)) + + // The warehouse answers what audit_events did, through the same reads. + for _, org := range []string{entries[0].OrgID, entries[1].OrgID} { + got, _, err := reader.ListAuditEvents(testCtx, business.AuditRead{ + Scope: business.OrganizationAuditScope(org), Query: business.AuditQuery{OrgID: org, PageSize: 100}, + }) + require.NoError(t, err) + require.Len(t, got, 3) + var want []business.AuditEntry + require.NoError(t, testStore.WithOrgTx(testCtx, org, func(ctx context.Context) error { + var err error + want, _, _, err = testStore.QueryAuditLog(ctx, business.AuditQuery{OrgID: org, PageSize: 100}) + return err + })) + require.Equal(t, want, got) + } + + report, err = copier.Run(testCtx, business.AuditHistoryCopyOptions{ConfirmDrop: true}) + require.NoError(t, err) + require.Positive(t, report.Dropped) + require.Equal(t, through, report.Cutoff) + require.False(t, historyPartitionExists(t), "the copied month is dropped, the way retention drops one") + var copied int + for _, partition := range report.Partitions { + copied += partition.Copied + } + require.Zero(t, copied, "the confirming run found everything copied") +} diff --git a/module/services/accounts/code/pkg/infra/testdata/control_plane_call_sites.txt b/module/services/accounts/code/pkg/infra/testdata/control_plane_call_sites.txt index 506b51bbc..f792607d2 100644 --- a/module/services/accounts/code/pkg/infra/testdata/control_plane_call_sites.txt +++ b/module/services/accounts/code/pkg/infra/testdata/control_plane_call_sites.txt @@ -27,6 +27,9 @@ pkg/auth/pg/session_store.go: (*SessionStore).FindByRefreshHash: WithControlPlan pkg/auth/pg/session_store.go: (*SessionStore).RevokeFamily: WithControlPlane pkg/auth/pg/session_store.go: (*SessionStore).RotateRefresh: WithControlPlane pkg/business/audit.go: (*DurableAuditEmitter).Emit: WithControlPlane +pkg/business/audit_history.go: (*AuditHistoryCopy).drop: WithControlPlane +pkg/business/audit_history.go: (*AuditHistoryCopy).partitions: WithControlPlane +pkg/business/audit_history.go: (*AuditHistoryCopy).readWindow: WithControlPlane pkg/business/audit_store_read.go: (postgresAuditReader).within: WithControlPlane pkg/business/connector_credentials.go: (*Service).SourceSigningSecret: WithControlPlane pkg/business/datasource_budget.go: (datasourceScheduler).Acquire: WithControlPlane From 9a2dc17e785fdfc4cf55842006968fae6e68b133 Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Sat, 3 Oct 2026 21:00:01 -0400 Subject: [PATCH 11/69] feat(audit): expose queue lag and alert when delivery stalls Read the transactional audit queue through the relay's worker pool and publish its depth and oldest-row age while telemetry is enabled. The age grows when archive or warehouse delivery fails and returns to zero after the queue clears. Add an alert for a sustained five-minute delay. Validation: the focused pure Go telemetry and alert tests pass, and the accounts service compiles. The PostgreSQL integration test could not complete because the local Codefly store process was killed during startup. --- module/services/accounts/code/audit_sink.go | 20 ++- .../code/pkg/business/audit_relay_metrics.go | 129 ++++++++++++++++++ .../pkg/business/audit_relay_metrics_test.go | 68 +++++++++ .../code/pkg/infra/postgres_audit_queue.go | 14 ++ .../pkg/infra/postgres_audit_queue_test.go | 27 ++++ .../accounts/code/pkg/metrics/slo_pack.json | 8 ++ module/services/accounts/code/work.go | 14 +- 7 files changed, 274 insertions(+), 6 deletions(-) create mode 100644 module/services/accounts/code/pkg/business/audit_relay_metrics.go create mode 100644 module/services/accounts/code/pkg/business/audit_relay_metrics_test.go diff --git a/module/services/accounts/code/audit_sink.go b/module/services/accounts/code/audit_sink.go index 142ef52fe..ea4c2977c 100644 --- a/module/services/accounts/code/audit_sink.go +++ b/module/services/accounts/code/audit_sink.go @@ -7,6 +7,9 @@ import ( "context" "fmt" "os" + "time" + + "go.opentelemetry.io/otel" ) // configuredAuditSink reads AUDIT_SINK and the settings its mode requires @@ -19,10 +22,10 @@ func configuredAuditSink() (auditsink.Config, error) { // (auditsink.Open), and the relay that drains the queue into them on the // relay's own pool. The service reads the returned store; the returned close // releases every client. -func newAuditSwap(ctx context.Context, types business.DeclaredAuditEventTypeReader, swap *auditsink.Swap) (*business.AuditRelay, business.AuditStore, func(), error) { +func newAuditSwap(ctx context.Context, types business.DeclaredAuditEventTypeReader, swap *auditsink.Swap, metricsEnabled bool) (*business.AuditRelay, *business.AuditRelayMonitor, business.AuditStore, func(), error) { opened, err := auditsink.Open(ctx, swap) if err != nil { - return nil, nil, nil, err + return nil, nil, nil, nil, err } closers := []func(){opened.Close} closeAll := func() { @@ -30,9 +33,9 @@ func newAuditSwap(ctx context.Context, types business.DeclaredAuditEventTypeRead closers[i]() } } - fail := func(err error) (*business.AuditRelay, business.AuditStore, func(), error) { + fail := func(err error) (*business.AuditRelay, *business.AuditRelayMonitor, business.AuditStore, func(), error) { closeAll() - return nil, nil, nil, err + return nil, nil, nil, nil, err } pool, err := infra.NewAuditRelayPool(ctx) @@ -57,5 +60,12 @@ func newAuditSwap(ctx context.Context, types business.DeclaredAuditEventTypeRead if err != nil { return fail(err) } - return relay, opened.Store, closeAll, nil + var monitor *business.AuditRelayMonitor + if metricsEnabled { + monitor, err = business.NewAuditRelayMonitor(queue, otel.Meter("github.com/codefly-dev/module-saas-starter/accounts"), 10*time.Second) + if err != nil { + return fail(err) + } + } + return relay, monitor, opened.Store, closeAll, nil } diff --git a/module/services/accounts/code/pkg/business/audit_relay_metrics.go b/module/services/accounts/code/pkg/business/audit_relay_metrics.go new file mode 100644 index 000000000..39f890a3e --- /dev/null +++ b/module/services/accounts/code/pkg/business/audit_relay_metrics.go @@ -0,0 +1,129 @@ +package business + +import ( + "context" + "errors" + "fmt" + "sync" + "time" + + "github.com/codefly-dev/core/wool" + "go.opentelemetry.io/otel/metric" +) + +// AuditQueueSnapshot is the payload-free state of the transactional queue. +type AuditQueueSnapshot struct { + Depth int64 + OldestEnqueuedAt *time.Time +} + +// AuditQueueMetricsSource reads the queue without taking the relay lease. +type AuditQueueMetricsSource interface { + Snapshot(context.Context) (AuditQueueSnapshot, error) +} + +// AuditRelayMonitor records queue depth and the age of the oldest row. A +// failed relay leaves its rows queued, so the age keeps rising until delivery +// succeeds; zero is recorded for an empty queue. +type AuditRelayMonitor struct { + source AuditQueueMetricsSource + depth metric.Int64Gauge + age metric.Float64Gauge + interval time.Duration + now func() time.Time + + mu sync.Mutex + cancel context.CancelFunc + done chan struct{} +} + +func NewAuditRelayMonitor(source AuditQueueMetricsSource, meter metric.Meter, interval time.Duration) (*AuditRelayMonitor, error) { + if source == nil || meter == nil { + return nil, errors.New("audit relay metrics: source and meter are required") + } + if interval <= 0 { + return nil, errors.New("audit relay metrics: interval must be positive") + } + depth, err := meter.Int64Gauge("saas.audit_queue.depth", metric.WithDescription("Queued audit events awaiting archive and warehouse delivery.")) + if err != nil { + return nil, err + } + age, err := meter.Float64Gauge("saas.audit_queue.oldest_age", metric.WithUnit("s"), + metric.WithDescription("Age of the oldest queued audit event.")) + if err != nil { + return nil, err + } + return &AuditRelayMonitor{source: source, depth: depth, age: age, interval: interval, now: time.Now}, nil +} + +func (m *AuditRelayMonitor) RunOnce(ctx context.Context) error { + snapshot, err := m.source.Snapshot(ctx) + if err != nil { + return fmt.Errorf("audit relay metrics: read queue: %w", err) + } + if snapshot.Depth < 0 || (snapshot.Depth == 0 && snapshot.OldestEnqueuedAt != nil) || + (snapshot.Depth > 0 && snapshot.OldestEnqueuedAt == nil) { + return errors.New("audit relay metrics: inconsistent queue snapshot") + } + age := 0.0 + if snapshot.OldestEnqueuedAt != nil { + age = m.now().Sub(*snapshot.OldestEnqueuedAt).Seconds() + if age < 0 { + age = 0 + } + } + m.depth.Record(ctx, snapshot.Depth) + m.age.Record(ctx, age) + return nil +} + +func (m *AuditRelayMonitor) Start(parent context.Context) { + m.mu.Lock() + if m.cancel != nil { + m.mu.Unlock() + return + } + ctx, cancel := context.WithCancel(parent) + m.cancel = cancel + m.done = make(chan struct{}) + done := m.done + m.mu.Unlock() + + go func() { + defer close(done) + run := func() { + attempt, cancel := context.WithTimeout(ctx, 5*time.Second) + defer cancel() + if err := m.RunOnce(attempt); err != nil && !errors.Is(err, context.Canceled) { + wool.Get(ctx).In("audit.relay.metrics").Warn("queue observation failed", wool.ErrField(err)) + } + } + run() + ticker := time.NewTicker(m.interval) + defer ticker.Stop() + for { + select { + case <-ctx.Done(): + return + case <-ticker.C: + run() + } + } + }() +} + +func (m *AuditRelayMonitor) Shutdown(ctx context.Context) error { + m.mu.Lock() + cancel, done := m.cancel, m.done + m.mu.Unlock() + if cancel == nil { + return nil + } + cancel() + select { + case <-done: + return nil + case <-ctx.Done(): + return ctx.Err() + } +} diff --git a/module/services/accounts/code/pkg/business/audit_relay_metrics_test.go b/module/services/accounts/code/pkg/business/audit_relay_metrics_test.go new file mode 100644 index 000000000..3062d59e1 --- /dev/null +++ b/module/services/accounts/code/pkg/business/audit_relay_metrics_test.go @@ -0,0 +1,68 @@ +package business + +import ( + "context" + "testing" + "time" + + "github.com/stretchr/testify/require" + sdkmetric "go.opentelemetry.io/otel/sdk/metric" + "go.opentelemetry.io/otel/sdk/metric/metricdata" +) + +type auditQueueSnapshotFake struct{ snapshot AuditQueueSnapshot } + +func (f *auditQueueSnapshotFake) Snapshot(context.Context) (AuditQueueSnapshot, error) { + return f.snapshot, nil +} + +func TestAuditRelayMonitorReportsLagUntilTheQueueClears(t *testing.T) { + now := time.Date(2026, 10, 3, 12, 0, 0, 0, time.UTC) + oldest := now.Add(-6 * time.Minute) + source := &auditQueueSnapshotFake{snapshot: AuditQueueSnapshot{Depth: 7, OldestEnqueuedAt: &oldest}} + reader := sdkmetric.NewManualReader() + provider := sdkmetric.NewMeterProvider(sdkmetric.WithReader(reader)) + t.Cleanup(func() { require.NoError(t, provider.Shutdown(t.Context())) }) + monitor, err := NewAuditRelayMonitor(source, provider.Meter("audit-relay-test"), time.Second) + require.NoError(t, err) + monitor.now = func() time.Time { return now } + + require.NoError(t, monitor.RunOnce(t.Context())) + require.Equal(t, int64(7), auditDepth(t, reader)) + require.Equal(t, 360.0, auditAge(t, reader)) + + source.snapshot = AuditQueueSnapshot{} + require.NoError(t, monitor.RunOnce(t.Context())) + require.Equal(t, int64(0), auditDepth(t, reader)) + require.Equal(t, 0.0, auditAge(t, reader)) +} + +func auditDepth(t *testing.T, reader *sdkmetric.ManualReader) int64 { + t.Helper() + var exported metricdata.ResourceMetrics + require.NoError(t, reader.Collect(t.Context(), &exported)) + for _, scope := range exported.ScopeMetrics { + for _, item := range scope.Metrics { + if item.Name == "saas.audit_queue.depth" { + return item.Data.(metricdata.Gauge[int64]).DataPoints[0].Value + } + } + } + t.Fatal("audit queue depth was not exported") + return 0 +} + +func auditAge(t *testing.T, reader *sdkmetric.ManualReader) float64 { + t.Helper() + var exported metricdata.ResourceMetrics + require.NoError(t, reader.Collect(t.Context(), &exported)) + for _, scope := range exported.ScopeMetrics { + for _, item := range scope.Metrics { + if item.Name == "saas.audit_queue.oldest_age" { + return item.Data.(metricdata.Gauge[float64]).DataPoints[0].Value + } + } + } + t.Fatal("audit queue age was not exported") + return 0 +} diff --git a/module/services/accounts/code/pkg/infra/postgres_audit_queue.go b/module/services/accounts/code/pkg/infra/postgres_audit_queue.go index 56df2ef26..c185c6028 100644 --- a/module/services/accounts/code/pkg/infra/postgres_audit_queue.go +++ b/module/services/accounts/code/pkg/infra/postgres_audit_queue.go @@ -6,6 +6,7 @@ import ( "encoding/json" "errors" "fmt" + "time" "accounts/pkg/business" @@ -56,6 +57,19 @@ func NewPostgresAuditQueue(pool *pgxpool.Pool) (*PostgresAuditQueue, error) { return &PostgresAuditQueue{pool: pool}, nil } +// Snapshot reads the queue's depth and oldest row for relay-lag telemetry. +// The worker pool has the queue's cross-organization SELECT grant; request +// traffic never receives this pool. +func (q *PostgresAuditQueue) Snapshot(ctx context.Context) (business.AuditQueueSnapshot, error) { + var depth int64 + var oldest *time.Time + err := q.pool.QueryRow(ctx, `SELECT count(*), min(enqueued_at) FROM public.audit_event_queue`).Scan(&depth, &oldest) + if err != nil { + return business.AuditQueueSnapshot{}, fmt.Errorf("audit queue: observe: %w", err) + } + return business.AuditQueueSnapshot{Depth: depth, OldestEnqueuedAt: oldest}, nil +} + // auditQueueSelectSQL reads the oldest queued events whose writing // transactions are below the snapshot's xmin — every transaction that old has // finished, so nothing still in flight can later commit a row ahead of these. diff --git a/module/services/accounts/code/pkg/infra/postgres_audit_queue_test.go b/module/services/accounts/code/pkg/infra/postgres_audit_queue_test.go index 8fc40532b..64d344a59 100644 --- a/module/services/accounts/code/pkg/infra/postgres_audit_queue_test.go +++ b/module/services/accounts/code/pkg/infra/postgres_audit_queue_test.go @@ -72,6 +72,33 @@ func drainAll(t *testing.T, queue *infra.PostgresAuditQueue) []business.QueuedAu return got } +func TestAuditQueueSnapshotTracksOldestRowAndClears(t *testing.T) { + pool := auditRelayPool(t) + emptyAuditQueue(t, pool) + orgID := seedOrg(t, seedUser(t)) + queue, err := infra.NewPostgresAuditQueue(pool) + require.NoError(t, err) + before, err := queue.Snapshot(testCtx) + require.NoError(t, err) + require.EqualValues(t, 0, before.Depth) + require.Nil(t, before.OldestEnqueuedAt) + + require.NoError(t, testStore.WithOrgTx(testCtx, orgID, func(ctx context.Context) error { + return testStore.EnqueueAuditEvent(ctx, queuedEntry(orgID, business.EventSessionRevoked)) + })) + after, err := queue.Snapshot(testCtx) + require.NoError(t, err) + require.EqualValues(t, 1, after.Depth) + require.NotNil(t, after.OldestEnqueuedAt) + require.WithinDuration(t, time.Now(), *after.OldestEnqueuedAt, time.Minute) + + drainAll(t, queue) + cleared, err := queue.Snapshot(testCtx) + require.NoError(t, err) + require.EqualValues(t, 0, cleared.Depth) + require.Nil(t, cleared.OldestEnqueuedAt) +} + func TestAuditQueueTenantWritesOnlyItsOwnOrganizationAndReadsNothing(t *testing.T) { pool := auditRelayPool(t) emptyAuditQueue(t, pool) diff --git a/module/services/accounts/code/pkg/metrics/slo_pack.json b/module/services/accounts/code/pkg/metrics/slo_pack.json index 87e17c39f..c41fbc4f4 100644 --- a/module/services/accounts/code/pkg/metrics/slo_pack.json +++ b/module/services/accounts/code/pkg/metrics/slo_pack.json @@ -100,6 +100,14 @@ "expression_language": "promql", "expression": "max_over_time(saas_jobs_oldest_ready_age_seconds[10m]) > 300" }, + { + "id": "audit_relay_lag", + "signal": "saas.audit_queue.oldest_age", + "condition": "the oldest queued audit event has waited over 5m for 5m", + "runbook": "audit-relay", + "expression_language": "promql", + "expression": "min_over_time(saas_audit_queue_oldest_age_seconds[5m]) > 300" + }, { "id": "integration_failures", "signal": "saas.jobs.completed", diff --git a/module/services/accounts/code/work.go b/module/services/accounts/code/work.go index 9c5af7a4a..b5e8a21b2 100644 --- a/module/services/accounts/code/work.go +++ b/module/services/accounts/code/work.go @@ -571,6 +571,7 @@ func doWork(ctx context.Context) (Clean, error) { } var auditRelay *business.AuditRelay + var auditRelayMonitor *business.AuditRelayMonitor closeAuditRelay := func() {} if auditSink.Swap != nil { // Under a swap value the warehouse is the store of record for writes and @@ -578,7 +579,7 @@ func doWork(ctx context.Context) (Clean, error) { // readable-source query read it, scoped to the caller's organization and // this deployment. var auditStore business.AuditStore - auditRelay, auditStore, closeAuditRelay, err = newAuditSwap(ctx, store, auditSink.Swap) + auditRelay, auditRelayMonitor, auditStore, closeAuditRelay, err = newAuditSwap(ctx, store, auditSink.Swap, otelMetricProvider != nil) if err != nil { return nil, err } @@ -1148,6 +1149,9 @@ func doWork(ctx context.Context) (Clean, error) { if auditRelay != nil { auditRelay.Start(ctx) } + if auditRelayMonitor != nil { + auditRelayMonitor.Start(ctx) + } if emailWorker != nil { emailWorker.Start(ctx) } @@ -1193,6 +1197,14 @@ func doWork(ctx context.Context) (Clean, error) { } cancel() } + if auditRelayMonitor != nil { + sw.Info("stopping audit relay metrics monitor") + shutdownCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + if err := auditRelayMonitor.Shutdown(shutdownCtx); err != nil { + sw.Warn("audit relay metrics monitor shutdown timed out", wool.ErrField(err)) + } + cancel() + } closeAuditRelay() if jobOperationsMonitor != nil { sw.Info("stopping durable job metrics monitor") From b84e1a58cdfdb2f464f4f6948a13101f7cae20bd Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Sat, 3 Oct 2026 21:10:16 -0400 Subject: [PATCH 12/69] docs(audit): add the relay lag response runbook The audit queue alert pack names an audit-relay runbook. Explain how to read its depth and age, restore a failed destination without losing queued events, and reconcile distinct event ids after recovery. Validation: document links, naming gate and diff whitespace checks pass. --- module/MEASUREMENT_RUNBOOKS.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/module/MEASUREMENT_RUNBOOKS.md b/module/MEASUREMENT_RUNBOOKS.md index 2b1022a84..60ad3e95a 100644 --- a/module/MEASUREMENT_RUNBOOKS.md +++ b/module/MEASUREMENT_RUNBOOKS.md @@ -31,6 +31,23 @@ provider responses into an incident channel. 5. Confirm queue age drains, no new terminal failures appear, and logical destination counts match distinct source identities. +## `audit-relay` + +1. Read `saas.audit_queue.depth` and `saas.audit_queue.oldest_age` together. + An absent series means telemetry is unavailable; a zero age with zero + depth means the queue has drained. Do not infer a healthy relay from a + missing series. +2. Check the Accounts audit relay's bounded error logs and the health of its + declared warehouse and locked archive. Confirm the deployed `AUDIT_SINK`, + deployment id, destination, and workload identity without printing secrets. +3. Restore the failing destination or identity and let the relay retry the + transactional queue. Do not delete queue rows, turn off audit emission, + or send around the relay. A retry may duplicate a warehouse row or archive + object; readers deduplicate by event id. +4. Confirm depth and age return to zero, then compare source event ids with + distinct warehouse and archive event ids for the affected interval. Check + that service reads still enforce organization and deployment scope. + ## `integrations` 1. Separate transport failure, timeout, rate limiting, authentication, From 6952a57e04b5c7fb81b4fb225a5a8dbcfd7e1fe6 Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Sat, 3 Oct 2026 21:08:55 -0400 Subject: [PATCH 13/69] feat(audit): add the ClickHouse store of record and an S3 archive writer MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ADR 0009 names two warehouse adapters. BigQuery shipped; this adds ClickHouse, for deployments outside Google Cloud and on-premises, and an S3 archive writer, so any warehouse can pair with a GCS or an S3 (or S3-compatible) locked archive. AUDIT_SINK=clickhouse is now a swap value. Its settings are AUDIT_CLICKHOUSE_DSN (required; credentials ride in it, from the deployment's secret, and it is never logged or echoed in an error), AUDIT_CLICKHOUSE_CLUSTER (optional; tables are then created ON CLUSTER as ReplicatedMergeTree), AUDIT_EVENTS_RETENTION_DAYS (required; the compliance window, the events table's TTL, at least the content window) and the shared archive, deployment, content-window and relay settings. Missing ones are reported together. The bigquery settings and their messages are unchanged. The store (pkg/auditstore/clickhousestore) keeps the BigQuery adapter's two tables and rows: Ensure requires the database, creates the tables when missing (partitioned by month, ordered by deployment, organization and time, a TTL per tier) and refuses a table whose columns or TTL differ. A relay batch is one insert per table, never a row at a time. Reads carry the deployment always and the organization unless the read is the platform read, with every value a server-side query parameter; an unscoped read, or a query naming another organization than its scope, is refused before any statement runs. ClickHouse does the scope, window, envelope filters, deduplication by event id, order, keyset paging, aggregation, readable-source query and the details join. The service (auditeval) evaluates only what ClickHouse cannot compute as Postgres does: number and nested payload containment, the ->> text of fractional or nested values, and payloads ClickHouse's parser refuses (an integer beyond 64 bits) — an aggregation counts such rows and falls back to the service when any are present. percentile_cont is interpolated in the service over values ClickHouse collects per group. Reads pin precise_float_parsing and allow_simdjson: without either, ClickHouse reads some decimal texts to a double other than float8in's. The archive object name and JSONL move to pkg/auditstore/archiveformat, shared by the GCS writer (byte-identical output) and the new S3 writer, which puts with If-None-Match: * and a SHA-256 checksum, treats a 412 on a retried put as its lost acknowledgement, and takes credentials, region and endpoint from the AWS default chain and environment only. Azure Blob still has no writer and is refused at startup. Evidence: the read-parity suite against Postgres passes for bigquery and, against a ClickHouse 25.8 container, for clickhouse, with added cases (paired and null payload filters, numeric and fractional payload dimensions, numeric metrics by type, category by month). The adapter's own server tests compare every read with auditeval over the records, prove which aggregations run in ClickHouse, and fail when either pinned setting is dropped. The S3 writer passes against a fake endpoint and a MinIO bucket with Object Lock; Open runs end to end against both containers. govulncheck is clean. Co-Authored-By: Claude Opus 5.5 --- module/FEATURES.md | 15 +- module/services/accounts/AGENTS.md | 7 +- module/services/accounts/code/go.mod | 28 +- module/services/accounts/code/go.sum | 58 +- .../auditstore/archiveformat/archiveformat.go | 109 ++++ .../archiveformat/archiveformat_test.go | 104 ++++ .../pkg/auditstore/auditeval/aggregate.go | 11 +- .../code/pkg/auditstore/auditeval/eval.go | 6 + .../pkg/auditstore/auditeval/eval_test.go | 10 +- .../code/pkg/auditstore/auditeval/page.go | 13 + .../pkg/auditstore/auditsink/auditsink.go | 257 +++++++-- .../auditstore/auditsink/auditsink_test.go | 123 +++- .../pkg/auditstore/auditsink/open_test.go | 85 +++ .../auditstore/clickhousestore/aggregate.go | 257 +++++++++ .../clickhousestore/clickhousestore.go | 527 ++++++++++++++++++ .../clickhousestore/clickhousestore_test.go | 348 ++++++++++++ .../pkg/auditstore/clickhousestore/query.go | 271 +++++++++ .../pkg/auditstore/clickhousestore/reader.go | 519 +++++++++++++++++ .../auditstore/clickhousestore/server_test.go | 429 ++++++++++++++ .../pkg/auditstore/gcsarchive/gcsarchive.go | 105 +--- .../auditstore/gcsarchive/gcsarchive_test.go | 62 +-- .../auditstore/s3archive/objectlock_test.go | 87 +++ .../pkg/auditstore/s3archive/s3archive.go | 134 +++++ .../auditstore/s3archive/s3archive_test.go | 170 ++++++ .../accounts/code/pkg/business/audit_store.go | 11 +- .../pkg/business/audit_store_parity_test.go | 77 +++ .../code/pkg/business/audit_store_test.go | 6 +- module/services/accounts/code/work.go | 6 +- 28 files changed, 3598 insertions(+), 237 deletions(-) create mode 100644 module/services/accounts/code/pkg/auditstore/archiveformat/archiveformat.go create mode 100644 module/services/accounts/code/pkg/auditstore/archiveformat/archiveformat_test.go create mode 100644 module/services/accounts/code/pkg/auditstore/auditsink/open_test.go create mode 100644 module/services/accounts/code/pkg/auditstore/clickhousestore/aggregate.go create mode 100644 module/services/accounts/code/pkg/auditstore/clickhousestore/clickhousestore.go create mode 100644 module/services/accounts/code/pkg/auditstore/clickhousestore/clickhousestore_test.go create mode 100644 module/services/accounts/code/pkg/auditstore/clickhousestore/query.go create mode 100644 module/services/accounts/code/pkg/auditstore/clickhousestore/reader.go create mode 100644 module/services/accounts/code/pkg/auditstore/clickhousestore/server_test.go create mode 100644 module/services/accounts/code/pkg/auditstore/s3archive/objectlock_test.go create mode 100644 module/services/accounts/code/pkg/auditstore/s3archive/s3archive.go create mode 100644 module/services/accounts/code/pkg/auditstore/s3archive/s3archive_test.go diff --git a/module/FEATURES.md b/module/FEATURES.md index 52334aa08..94b396880 100644 --- a/module/FEATURES.md +++ b/module/FEATURES.md @@ -398,7 +398,7 @@ see `JOBS.md` for the exact boundary and sequencing. | Export (JSON/CSV) | ✅ | `audit_export.go` | | Impersonation tracking | ✅ | Records both real actor + viewed-as user | | Replay / event sourcing| ❌ | Audit log is read-only history; not used to reconstruct state | -| Warehouse store of record | 🟡 | `AUDIT_SINK=bigquery` (ADR 0009): each record commits to a Postgres queue on the mutation's transaction and a relay delivers it to BigQuery and a locked GCS archive. The activity list, aggregates, exports and the readable-source query read BigQuery through the Storage Read API — no query jobs — scoped to the caller's organization and the deployment, with each event returned once. `audit-history-copy` copies a switching deployment's `audit_events` history into the store and the archive, verifies it by reading it back, and drops the copied partitions only with `-confirm-drop`; ClickHouse is not built | +| Warehouse store of record | 🟡 | `AUDIT_SINK=bigquery` or `AUDIT_SINK=clickhouse` (ADR 0009): each record commits to a Postgres queue in the mutation's transaction, then a relay delivers it to the warehouse and a locked GCS or S3 archive. Reads are scoped to the organization and deployment and deduplicated by event id; BigQuery uses the Storage Read API without query jobs, and ClickHouse uses SQL with service evaluation where needed. The shared parity suite compares both with Postgres. `audit-history-copy` verifies a switching deployment's copied history before a confirmed monthly-partition drop. No Azure Blob archive writer exists yet. | ### Admin panel @@ -654,12 +654,15 @@ Environment variables consumed by the api: | `POSTHOG_PROJECT_ID` | Positive PostHog project ID; required only in PostHog mode | | `POSTHOG_HOST` | Explicit HTTPS or local capture origin | | `POSTHOG_API_HOST` | Separate PostHog management/deletion origin | -| `AUDIT_SINK` | `postgres` (default), `both` (tee to `AUDIT_EXTERNAL_URL`, optional `AUDIT_EXTERNAL_TOKEN`), or `bigquery` (swap the audit store of record, ADR 0009); `external` is refused | +| `AUDIT_SINK` | `postgres` (default), `both` (tee to `AUDIT_EXTERNAL_URL`, optional `AUDIT_EXTERNAL_TOKEN`), or `bigquery` / `clickhouse` (swap the audit store of record, ADR 0009); `external` is refused | | `AUDIT_BIGQUERY_PROJECT`, `AUDIT_BIGQUERY_DATASET` | Required under `bigquery`: the store of record. The dataset must exist; the `audit_events` and `audit_event_details` tables are created if missing. Credentials are Application Default Credentials only. The service's identity holds no `bigquery.jobs.create`: it writes with streaming inserts and reads through the Storage Read API (`bigquery.tables.getData`, `bigquery.readsessions.create`/`getData`/`update`), because append plus job creation would permit DML | -| `AUDIT_ARCHIVE_URL` | Required under `bigquery`: the locked archive, as `gs://` (the scheme picks the writer; GCS is the one built). The bucket carries a retention lock the deployment sets and receives one object per relay batch | -| `AUDIT_DEPLOYMENT_ID` | Required under `bigquery`: stamped on every record and the archive path | -| `AUDIT_CONTENT_RETENTION_DAYS` | Required under `bigquery`: days a content-class event's full details are kept (the details table's partition expiration) | -| `AUDIT_RELAY_BATCH_SIZE`, `AUDIT_RELAY_MAX_WAIT` | Optional under `bigquery`: events per delivery (default 500, at most 5000) and how long a partial batch waits (default `5s`) | +| `AUDIT_CLICKHOUSE_DSN` | Required under `clickhouse`: the store of record, as `clickhouse://:@:9000/` (add `?secure=true` for TLS), from the deployment's secret — never logged. The database must exist; the `audit_events` and `audit_event_details` tables are created if missing (partitioned by month, a TTL per tier) and refused if their columns or TTL differ. The user needs `CREATE TABLE`, `INSERT` and `SELECT` on that database, and no `ALTER`, `DELETE`, `TRUNCATE` or `DROP`. Tested against ClickHouse 25.8 | +| `AUDIT_CLICKHOUSE_CLUSTER` | Optional under `clickhouse`: a cluster name or macro (`{cluster}`); the tables are then created `ON CLUSTER` as `ReplicatedMergeTree`, coordinated by ClickHouse Keeper under the servers' default replica path and name | +| `AUDIT_EVENTS_RETENTION_DAYS` | Required under `clickhouse`: the compliance window in days — the events table's TTL — at least `AUDIT_CONTENT_RETENTION_DAYS` | +| `AUDIT_ARCHIVE_URL` | Required under a swap value: the locked archive, as `gs://` or `s3://` (the scheme picks the writer; any warehouse pairs with either). The bucket carries a retention lock the deployment sets (GCS Bucket Lock, S3 Object Lock default retention) and receives one create-only object per relay batch. S3 takes credentials and region from the AWS default chain and `AWS_REGION`, and an S3-compatible store's endpoint from `AWS_ENDPOINT_URL_S3` (addressed path-style); the writer needs `s3:PutObject` only | +| `AUDIT_DEPLOYMENT_ID` | Required under a swap value: stamped on every record and the archive path | +| `AUDIT_CONTENT_RETENTION_DAYS` | Required under a swap value: days a content-class event's full details are kept (the details table's partition expiration on BigQuery, its TTL on ClickHouse) | +| `AUDIT_RELAY_BATCH_SIZE`, `AUDIT_RELAY_MAX_WAIT` | Optional under a swap value: events per delivery (default 500, at most 5000) and how long a partial batch waits (default `5s`) | | `ERROR_TRACKING_MODE` | Explicit `disabled` or `sentry`; rejects partial config | | `SENTRY_DSN` | Server Sentry DSN, required in Sentry mode | | `OBSERVABILITY_EXPORTER` | Required in-graph collector output: `debug` or `otlphttp`; the collector refuses to start when it is unset | diff --git a/module/services/accounts/AGENTS.md b/module/services/accounts/AGENTS.md index 3f156413e..bf6fc959b 100644 --- a/module/services/accounts/AGENTS.md +++ b/module/services/accounts/AGENTS.md @@ -81,8 +81,11 @@ survives a restart and reaches every replica, and is only served when it is service also holds the append grant, and append plus job creation would allow DML — so they use the Storage Read API with row restrictions and are evaluated in the service (`pkg/auditstore/auditeval`) with the Postgres - reads' semantics; `pkg/business/audit_store_parity_test.go` holds the two to - one fixture. + reads' semantics. The ClickHouse reads run in SQL with bound parameters and + fall back to `auditeval` only where ClickHouse cannot compute what Postgres + does (`pkg/auditstore/clickhousestore/reader.go` lists where); + `pkg/business/audit_store_parity_test.go` holds every store to one fixture + (ClickHouse joins when `AUDIT_CLICKHOUSE_TEST_DSN` names a server). - **History copy** (`cmd/audit-history-copy`, `pkg/business/audit_history.go`): a deployment switching to a swap value copies its `audit_events` rows into the store of record and the archive — classified and hashed as the relay does a diff --git a/module/services/accounts/code/go.mod b/module/services/accounts/code/go.mod index 472ab909d..aab8ed60f 100644 --- a/module/services/accounts/code/go.mod +++ b/module/services/accounts/code/go.mod @@ -9,7 +9,12 @@ require ( cloud.google.com/go/storage v1.69.0 connectrpc.com/connect v1.21.0 connectrpc.com/otelconnect v0.9.0 + github.com/ClickHouse/clickhouse-go/v2 v2.48.0 github.com/apache/arrow/go/v15 v15.0.2 + github.com/aws/aws-sdk-go-v2 v1.47.1 + github.com/aws/aws-sdk-go-v2/config v1.33.6 + github.com/aws/aws-sdk-go-v2/credentials v1.20.6 + github.com/aws/aws-sdk-go-v2/service/s3 v1.114.0 github.com/codefly-dev/core v0.5.11-0.20260926231217-d481a2d49683 github.com/codefly-dev/interface-cache/go/cache v0.2.2-0.20260926205419-327a6e1d9aba github.com/codefly-dev/sdk-go v0.2.1-0.20260926212413-b91ce91eb23e @@ -60,13 +65,29 @@ require ( github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.14.1 // indirect github.com/Azure/azure-sdk-for-go/sdk/internal v1.12.0 // indirect github.com/AzureAD/microsoft-authentication-library-for-go v1.9.0 // indirect + github.com/ClickHouse/ch-go v0.74.0 // indirect github.com/GoogleCloudPlatform/opentelemetry-operations-go/detectors/gcp v1.35.0 // indirect github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/metric v0.57.0 // indirect github.com/GoogleCloudPlatform/opentelemetry-operations-go/internal/resourcemapping v0.57.0 // indirect github.com/Masterminds/semver v1.5.0 // indirect github.com/Masterminds/semver/v3 v3.5.0 // indirect github.com/VividCortex/ewma v1.2.0 // indirect + github.com/andybalholm/brotli v1.2.2 // indirect github.com/antlr4-go/antlr/v4 v4.13.1 // indirect + github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.20 // indirect + github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.20.1 // indirect + github.com/aws/aws-sdk-go-v2/internal/configsources v1.5.4 // indirect + github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.8.4 // indirect + github.com/aws/aws-sdk-go-v2/internal/v4a v1.5.4 // indirect + github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.19 // indirect + github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.11.5 // indirect + github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.14.4 // indirect + github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.20.4 // indirect + github.com/aws/aws-sdk-go-v2/service/signin v1.10.1 // indirect + github.com/aws/aws-sdk-go-v2/service/sso v1.38.1 // indirect + github.com/aws/aws-sdk-go-v2/service/ssooidc v1.43.1 // indirect + github.com/aws/aws-sdk-go-v2/service/sts v1.51.1 // indirect + github.com/aws/smithy-go v1.28.2 // indirect github.com/beorn7/perks v1.0.1 // indirect github.com/blang/semver v3.5.1+incompatible // indirect github.com/cenkalti/backoff/v5 v5.0.3 // indirect @@ -79,6 +100,8 @@ require ( github.com/fatih/color v1.19.0 // indirect github.com/felixge/httpsnoop v1.1.0 // indirect github.com/fxamacker/cbor/v2 v2.9.3 // indirect + github.com/go-faster/city v1.0.1 // indirect + github.com/go-faster/errors v0.7.1 // indirect github.com/go-jose/go-jose/v4 v4.1.4 // indirect github.com/go-logr/logr v1.4.4 // indirect github.com/go-logr/stdr v1.2.2 // indirect @@ -123,8 +146,9 @@ require ( github.com/mitchellh/mapstructure v1.5.0 // indirect github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect github.com/oklog/ulid/v2 v2.1.2 // indirect + github.com/paulmach/orb v0.13.0 // indirect github.com/philhofer/fwd v1.2.0 // indirect - github.com/pierrec/lz4/v4 v4.1.18 // indirect + github.com/pierrec/lz4/v4 v4.1.27 // indirect github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c // indirect github.com/pkg/errors v0.9.1 // indirect github.com/planetscale/vtprotobuf v0.6.1-0.20240319094008-0393e58bdf10 // indirect @@ -134,8 +158,10 @@ require ( github.com/prometheus/otlptranslator v1.0.0 // indirect github.com/prometheus/procfs v0.22.0 // indirect github.com/santhosh-tekuri/jsonschema/v6 v6.0.3 // indirect + github.com/segmentio/asm v1.2.1 // indirect github.com/shirou/gopsutil/v3 v3.24.5 // indirect github.com/shoenig/go-m1cpu v0.2.2 // indirect + github.com/shopspring/decimal v1.4.0 // indirect github.com/spiffe/go-spiffe/v2 v2.7.0 // indirect github.com/tinylib/msgp v1.6.4 // indirect github.com/tklauser/go-sysconf v0.4.0 // indirect diff --git a/module/services/accounts/code/go.sum b/module/services/accounts/code/go.sum index 3b82ece3e..71052e3d7 100644 --- a/module/services/accounts/code/go.sum +++ b/module/services/accounts/code/go.sum @@ -48,6 +48,10 @@ github.com/AzureAD/microsoft-authentication-extensions-for-go/cache v0.1.1 h1:WJ github.com/AzureAD/microsoft-authentication-extensions-for-go/cache v0.1.1/go.mod h1:tCcJZ0uHAmvjsVYzEFivsRTN00oz5BEsRgQHu5JZ9WE= github.com/AzureAD/microsoft-authentication-library-for-go v1.9.0 h1:MDT4FxAPve5FnYn6vOL1r7RCRDG+l9cI7a5LlCuHsqA= github.com/AzureAD/microsoft-authentication-library-for-go v1.9.0/go.mod h1:Y33QHnf0FfdVewFFISOGe20mkZbxX4H839o955/PoeI= +github.com/ClickHouse/ch-go v0.74.0 h1:uYs2m4wIt0ZHSM1E72rg0maCfzhR2V3xWb/vZEgpeWE= +github.com/ClickHouse/ch-go v0.74.0/go.mod h1:sZ/r+8ttZMjyrP9PuFbgoVbth1ywIu2LIQNA2vgko6M= +github.com/ClickHouse/clickhouse-go/v2 v2.48.0 h1:auzd4VkapQYhQF8F2Gog7s3x78Bi1JZmByxGbrw3C+4= +github.com/ClickHouse/clickhouse-go/v2 v2.48.0/go.mod h1:lBjUCPRG6RpRQdMbkXq+JV8rY0/O5lw+Z7jShgReFjM= github.com/GoogleCloudPlatform/opentelemetry-operations-go/detectors/gcp v1.35.0 h1:bN1gA3of5bXtbnLsRPrwfmbbe7A5UWFlcTHseujLnpc= github.com/GoogleCloudPlatform/opentelemetry-operations-go/detectors/gcp v1.35.0/go.mod h1:Yj5vHEz/aAepZGliRJsA6uvHAVAQyEwajq9ORCHPxzM= github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/metric v0.57.0 h1:jLdiS1vO+XJFyDSWRHBx56r4s/NNtcl5J6KyCcWUX/w= @@ -66,10 +70,48 @@ github.com/ProtonMail/go-crypto v1.4.1 h1:9RfcZHqEQUvP8RzecWEUafnZVtEvrBVL9BiF67 github.com/ProtonMail/go-crypto v1.4.1/go.mod h1:e1OaTyu5SYVrO9gKOEhTc+5UcXtTUa+P3uLudwcgPqo= github.com/VividCortex/ewma v1.2.0 h1:f58SaIzcDXrSy3kWaHNvuJgJ3Nmz59Zji6XoJR/q1ow= github.com/VividCortex/ewma v1.2.0/go.mod h1:nz4BbCtbLyFDeC9SUHbtcT5644juEuWfUAUnGx7j5l4= +github.com/andybalholm/brotli v1.2.2 h1:HzTuoo2ErYQqf5qvcJInB8uvqSVxRttzkFexPWtnceM= +github.com/andybalholm/brotli v1.2.2/go.mod h1:rzTDkvFWvIrjDXZHkuS16NPggd91W3kUSvPlQ1pLaKY= github.com/antlr4-go/antlr/v4 v4.13.1 h1:SqQKkuVZ+zWkMMNkjy5FZe5mr5WURWnlpmOuzYWrPrQ= github.com/antlr4-go/antlr/v4 v4.13.1/go.mod h1:GKmUxMtwp6ZgGwZSva4eWPC5mS6vUAmOABFgjdkM7Nw= github.com/apache/arrow/go/v15 v15.0.2 h1:60IliRbiyTWCWjERBCkO1W4Qun9svcYoZrSLcyOsMLE= github.com/apache/arrow/go/v15 v15.0.2/go.mod h1:DGXsR3ajT524njufqf95822i+KTh+yea1jass9YXgjA= +github.com/aws/aws-sdk-go-v2 v1.47.1 h1:uOIZnp4PK3ZhKI0dNrJrhTEsLxbpXHTAJlwoS1pvAtw= +github.com/aws/aws-sdk-go-v2 v1.47.1/go.mod h1:bttEH6JqnUL8LepvDVfdrds/fZ5bCIxzpe3abyUrhDU= +github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.20 h1:GPRlPwz40I2B2VrBEASOA3Bi77NyeqejNLkifosX0rs= +github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.20/go.mod h1:g7PNzKcsOKWb4fkSRBA7BZVAS6Y8IcxzN+nRohhQ1Q8= +github.com/aws/aws-sdk-go-v2/config v1.33.6 h1:MBjkSTLczek/UgiK+EYPIoRTqE7gP8vtW3OFbFo7Nug= +github.com/aws/aws-sdk-go-v2/config v1.33.6/go.mod h1:grRAFzdAZJrwcbasJRg2MPvIrVjtlfXllHssN6+E1JE= +github.com/aws/aws-sdk-go-v2/credentials v1.20.6 h1:NpAFXCU7NzXNkdGK3zQTtsRJ+3v9tZQV0xcdRw8uBdw= +github.com/aws/aws-sdk-go-v2/credentials v1.20.6/go.mod h1:mcZCoiPnyMvP8VMNbygNX5lLqSlkYJIMPODylQMurOk= +github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.20.1 h1:8gALAAmacnIXh+z6VkdDanv4/IkG5APdg4DZLDTmLog= +github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.20.1/go.mod h1:Z7IJhJU+poOdJjUR2wpyY21ossQ1XS/R3Lk9Msq5kM4= +github.com/aws/aws-sdk-go-v2/internal/configsources v1.5.4 h1:CLq4+8UHCI+ZZYl/EuJxXovaIVN2xeeT8JV+dsApQ5E= +github.com/aws/aws-sdk-go-v2/internal/configsources v1.5.4/go.mod h1:Wv4q5sAM04xAMkoOedxLx2inVf6K5FdxYp+A61L+q/0= +github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.8.4 h1:dD4MR81I7YkpEBRk6UP9rocC2QnT3qVuXwzlYTtfGEs= +github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.8.4/go.mod h1:EcXV1kAFd5XwSkDHlj94gnF3q5CkJyYiIJfH8N0VmrE= +github.com/aws/aws-sdk-go-v2/internal/v4a v1.5.4 h1:7Wo47d/xn/7KttCSBd8EGYeZ7ULRFRkUHr6vkZPBzVQ= +github.com/aws/aws-sdk-go-v2/internal/v4a v1.5.4/go.mod h1:tDB2IVC1xC3vX8o+6uRlzhTxP3g1b77CZXFX/oD2FnQ= +github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.19 h1:bAdDl/HkGCcGPoe25ToSHEw23VIxt6CT5fLcg111BKg= +github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.19/go.mod h1:KaUzbLxv4CeSxh6ZCl9B4m7CuFenS8kUEaDs+f/DQr4= +github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.11.5 h1:/TYsZXdA8UTa+WCtCYSAJIr1vwl0+eho6TUgJGwFFO8= +github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.11.5/go.mod h1:qPqp1Uwd/BqdhPufv6oem9j5J7HNsgc2V22dUiDPn+s= +github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.14.4 h1:29SvnfGhXjTl8ONxFwbj2rs6lbhiFXD2CgFQmbT/bXY= +github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.14.4/go.mod h1:wm04I5DMuNVvZHFe/dHnUxincvNbbK7AiNBbYsQivek= +github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.20.4 h1:pPiWfgeNxqluKEph7hvU88kuGKBPOWzO+Dk9t2zqqNs= +github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.20.4/go.mod h1:YlwGoIUDG/3kBQbdNOVs/xKZ9J01G8e/6D1mRBj9uTk= +github.com/aws/aws-sdk-go-v2/service/s3 v1.114.0 h1:VMAdYqr4Jn/8ATs9BHC5riwrs0d6m1Z2ohFriSwZwm0= +github.com/aws/aws-sdk-go-v2/service/s3 v1.114.0/go.mod h1:9APRWGLFITKD+xzWSIyT9V7QV4bNlEuIieWlzXgGFlI= +github.com/aws/aws-sdk-go-v2/service/signin v1.10.1 h1:DzCCWLzcIRQ77F3DEUljud7bEjTgFOIKXP52NmVRyhU= +github.com/aws/aws-sdk-go-v2/service/signin v1.10.1/go.mod h1:xpo/geVldu8payT375WekctUzopG/hBU7miiqItMUlw= +github.com/aws/aws-sdk-go-v2/service/sso v1.38.1 h1:Umtl/0YZhng4xndfW3lKJrYYP7NLEjI6bGXVomwLcs0= +github.com/aws/aws-sdk-go-v2/service/sso v1.38.1/go.mod h1:rRD/dnm7q0HYE/I5TMaPgkWyyUGLcwuxHLABsLnQ3e0= +github.com/aws/aws-sdk-go-v2/service/ssooidc v1.43.1 h1:orIWdNiLgzrhu/11RcPPKO/SBzUUymbUQuZbSPImghg= +github.com/aws/aws-sdk-go-v2/service/ssooidc v1.43.1/go.mod h1:skwM/xsbR/1ReUTesv9BhpJp1VjajR7DWQnuVLwiXsQ= +github.com/aws/aws-sdk-go-v2/service/sts v1.51.1 h1:0HOqZXRvMytH6bFHVIc0oJX07sZjfhz0zXtjs6gdE8s= +github.com/aws/aws-sdk-go-v2/service/sts v1.51.1/go.mod h1:26zA0GhDrLo+yiLI2yXWxqB1PdsShfLikoI7GOEgugM= +github.com/aws/smithy-go v1.28.2 h1:myhcykQcatTul2B/zITjDk203G7t0awUAs1hVry5Bvg= +github.com/aws/smithy-go v1.28.2/go.mod h1:YE2RhdIuDbA5E5bTdciG9KrW3+TiEONeUWCqxX9i1Fc= github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM= github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw= github.com/blang/semver v3.5.1+incompatible h1:cQNTCjp13qL8KC3Nbxr/y2Bqb63oX6wdnnjpJbkM4JQ= @@ -147,6 +189,10 @@ github.com/fxamacker/cbor/v2 v2.9.3 h1:oQBnFATpNdY8gJHTndDDv5Xl4QqNaz51G5LLEPhng github.com/fxamacker/cbor/v2 v2.9.3/go.mod h1:vM4b+DJCtHn+zz7h3FFp/hDAI9WNWCsZj23V5ytsSxQ= github.com/go-errors/errors v1.4.2 h1:J6MZopCL4uSllY1OfXM374weqZFFItUbrImctkmUxIA= github.com/go-errors/errors v1.4.2/go.mod h1:sIVyrIiJhuEF+Pj9Ebtd6P/rEYROXFi3BopGUQ5a5Og= +github.com/go-faster/city v1.0.1 h1:4WAxSZ3V2Ws4QRDrscLEDcibJY8uf41H6AhXDrNDcGw= +github.com/go-faster/city v1.0.1/go.mod h1:jKcUJId49qdW3L1qKHH/3wPeUstCVpVSXTM6vO3VcTw= +github.com/go-faster/errors v0.7.1 h1:MkJTnDoEdi9pDabt1dpWf7AA8/BaSYZqibYyhZ20AYg= +github.com/go-faster/errors v0.7.1/go.mod h1:5ySTjWFiphBs07IKuiL69nxdfd5+fzh1u7FPGZP2quo= github.com/go-git/gcfg v1.5.1-0.20230307220236-3a3c6141e376 h1:+zs/tPmkDkHx3U66DAb0lQFJrpS6731Oaa12ikc+DiI= github.com/go-git/gcfg v1.5.1-0.20230307220236-3a3c6141e376/go.mod h1:an3vInlBmSxCcxctByoQdvwPiA7DTK7jaaFDBTtu0ic= github.com/go-git/go-billy/v5 v5.9.0 h1:jItGXszUDRtR/AlferWPTMN4j38BQ88XnXKbilmmBPA= @@ -305,11 +351,13 @@ github.com/opencontainers/go-digest v1.0.0 h1:apOUWs51W5PlhuyGyz9FCeeBIOUDA/6nW8 github.com/opencontainers/go-digest v1.0.0/go.mod h1:0JzlMkj0TRzQZfJkVvzbP0HBR3IKzErnv2BNG4W4MAM= github.com/opencontainers/image-spec v1.1.1 h1:y0fUlFfIZhPF1W537XOLg0/fcx6zcHCJwooC2xJA040= github.com/opencontainers/image-spec v1.1.1/go.mod h1:qpqAh3Dmcf36wStyyWU+kCeDgrGnAve2nCC8+7h8Q0M= +github.com/paulmach/orb v0.13.0 h1:r7n7mQGGF+cj/CbcivEj9J3HGK+XR+yXnvzRdq9saIw= +github.com/paulmach/orb v0.13.0/go.mod h1:6scRWINywA2Jf05dcjOfLfxrUIMECvTSG2MVbRLxu/k= github.com/pborman/getopt v0.0.0-20170112200414-7148bc3a4c30/go.mod h1:85jBQOZwpVEaDAr341tbn15RS4fCAsIst0qp7i8ex1o= github.com/philhofer/fwd v1.2.0 h1:e6DnBTl7vGY+Gz322/ASL4Gyp1FspeMvx1RNDoToZuM= github.com/philhofer/fwd v1.2.0/go.mod h1:RqIHx9QI14HlwKwm98g9Re5prTQ6LdeRQn+gXJFxsJM= -github.com/pierrec/lz4/v4 v4.1.18 h1:xaKrnTkyoqfh1YItXl56+6KJNVYWlEEPuAQW9xsplYQ= -github.com/pierrec/lz4/v4 v4.1.18/go.mod h1:gZWDp/Ze/IJXGXf23ltt2EXimqmTUXEy0GFuRQyBid4= +github.com/pierrec/lz4/v4 v4.1.27 h1:+PhzhWDrjRj89TH2sw43nE3+4+W8lSxIuQadEHZyjUk= +github.com/pierrec/lz4/v4 v4.1.27/go.mod h1:EoQMVJgeeEOMsCqCzqFm2O0cJvljX2nGZjcRIPL34O4= github.com/pjbgf/sha1cd v0.6.0 h1:3WJ8Wz8gvDz29quX1OcEmkAlUg9diU4GxJHqs0/XiwU= github.com/pjbgf/sha1cd v0.6.0/go.mod h1:lhpGlyHLpQZoxMv8HcgXvZEhcGs0PG/vsZnEJ7H0iCM= github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c h1:+mdjkGKdHQG3305AYmdv1U2eRNDiU2ErMBj1gwrq8eQ= @@ -341,6 +389,8 @@ github.com/rs/cors v1.11.1 h1:eU3gRzXLRK57F5rKMGMZURNdIG4EoAmX8k94r9wXWHA= github.com/rs/cors v1.11.1/go.mod h1:XyqrcTp5zjWr1wsJ8PIRZssZ8b/WMcMf71DJnit4EMU= github.com/santhosh-tekuri/jsonschema/v6 v6.0.3 h1:1EYB5IzjZawrrnELUi78f9fPu57HuXjmddZPjrls/28= github.com/santhosh-tekuri/jsonschema/v6 v6.0.3/go.mod h1:JXeL+ps8p7/KNMjDQk3TCwPpBy0wYklyWTfbkIzdIFU= +github.com/segmentio/asm v1.2.1 h1:DTNbBqs57ioxAD4PrArqftgypG4/qNpXoJx8TVXxPR0= +github.com/segmentio/asm v1.2.1/go.mod h1:BqMnlJP91P8d+4ibuonYZw9mfnzI9HfxselHZr5aAcs= github.com/sergi/go-diff v1.4.0 h1:n/SP9D5ad1fORl+llWyN+D6qoUETXNZARKjyY2/KVCw= github.com/sergi/go-diff v1.4.0/go.mod h1:A0bzQcvG0E7Rwjx0REVgAGH58e96+X0MeOfepqsbeW4= github.com/shirou/gopsutil/v3 v3.24.5 h1:i0t8kL+kQTvpAYToeuiVk3TgDeKOFioZO3Ztz/iZ9pI= @@ -349,6 +399,8 @@ github.com/shoenig/go-m1cpu v0.2.2 h1:4nc55oVv7nygGnfI9bhLCLzUEs4794y0Bkqx4q2zy7 github.com/shoenig/go-m1cpu v0.2.2/go.mod h1:KkDOw6m3ZJQAPHbrzkZki4hnx+pDRR1Lo+ldA56wD5w= github.com/shoenig/test v1.7.0 h1:eWcHtTXa6QLnBvm0jgEabMRN/uJ4DMV3M8xUGgRkZmk= github.com/shoenig/test v1.7.0/go.mod h1:UxJ6u/x2v/TNs/LoLxBNJRV9DiwBBKYxXSyczsBHFoI= +github.com/shopspring/decimal v1.4.0 h1:bxl37RwXBklmTi0C79JfXCEBD1cqqHt0bbgBAGFp81k= +github.com/shopspring/decimal v1.4.0/go.mod h1:gawqmDU56v4yIKSwfBSFip1HdCCXN8/+DMd9qYNcwME= github.com/skeema/knownhosts v1.3.2 h1:EDL9mgf4NzwMXCTfaxSD/o/a5fxDw/xL9nkU28JjdBg= github.com/skeema/knownhosts v1.3.2/go.mod h1:bEg3iQAuw+jyiw+484wwFJoKSLwcfd7fqRy+N0QTiow= github.com/spiffe/go-spiffe/v2 v2.7.0 h1:uXe1MflJoHw58wAUvxVlcM7WpKtijWG7I1UidcGh6g4= @@ -372,6 +424,8 @@ github.com/xanzy/ssh-agent v0.3.3 h1:+/15pJfg/RsTxqYcX6fHqOXZwwMP+2VyYWJeWM2qQFM github.com/xanzy/ssh-agent v0.3.3/go.mod h1:6dzNDKs0J9rVPHPhaGCukekBHKqfl+L3KghI1Bc68Uw= github.com/xlab/treeprint v1.2.0 h1:HzHnuAF1plUN2zGlAFHbSQP2qJ0ZAD3XF5XD7OesXRQ= github.com/xlab/treeprint v1.2.0/go.mod h1:gj5Gd3gPdKtR1ikdDK6fnFLdmIS0X30kTTuNd/WEJu0= +github.com/xyproto/randomstring v1.0.5 h1:YtlWPoRdgMu3NZtP45drfy1GKoojuR7hmRcnhZqKjWU= +github.com/xyproto/randomstring v1.0.5/go.mod h1:rgmS5DeNXLivK7YprL0pY+lTuhNQW3iGxZ18UQApw/E= github.com/yoheimuta/go-protoparser/v4 v4.14.2 h1:/P/LlX1CF9NaTWEltGcIZVvNlPbhABuAnBtAWpb3+74= github.com/yoheimuta/go-protoparser/v4 v4.14.2/go.mod h1:AHNNnSWnb0UoL4QgHPiOAg2BniQceFscPI5X/BZNHl8= github.com/yusufpapurcu/wmi v1.2.4 h1:zFUKzehAFReQwLys1b/iSMl+JQGSCSjtVqQn9bBrPo0= diff --git a/module/services/accounts/code/pkg/auditstore/archiveformat/archiveformat.go b/module/services/accounts/code/pkg/auditstore/archiveformat/archiveformat.go new file mode 100644 index 000000000..9351b531b --- /dev/null +++ b/module/services/accounts/code/pkg/auditstore/archiveformat/archiveformat.go @@ -0,0 +1,109 @@ +// Package archiveformat is the one shape of the audit store swap's locked +// archive (ADR 0009), whichever object store holds it: one JSON Lines object +// per relay batch, named by the batch, full for security-class events and +// content-free for content-class ones. Every archive writer (gcsarchive, +// s3archive) names and encodes its objects here, so an archive reader reads +// every bucket the same way, and moving a deployment between object stores +// changes where the objects are, never what they say. +package archiveformat + +import ( + "bytes" + "encoding/json" + "fmt" + "time" + + "accounts/pkg/business" +) + +// ContentType is every archive object's media type. +const ContentType = "application/x-ndjson" + +// ObjectName names a batch's object: +// +// ///
/-.jsonl +// +// The date path is when the relay composed the batch, so a day's objects list +// together and in delivery order. The batch id is unique to one delivery +// attempt: a retried batch is a new object, never a rewrite of an old one. +func ObjectName(batch business.AuditBatch) (string, error) { + if !safeSegment(batch.DeploymentID) || !safeSegment(batch.ID) { + return "", fmt.Errorf("audit archive: deployment id %q and batch id %q must be plain path segments", batch.DeploymentID, batch.ID) + } + at := batch.ComposedAt.UTC() + return fmt.Sprintf("%s/%s/%s-%s.jsonl", + batch.DeploymentID, at.Format("2006/01/02"), at.Format("20060102T150405.000000000Z"), batch.ID), nil +} + +func safeSegment(segment string) bool { + if segment == "" || segment == "." || segment == ".." { + return false + } + for _, r := range segment { + if !(r >= 'a' && r <= 'z' || r >= 'A' && r <= 'Z' || r >= '0' && r <= '9' || r == '-' || r == '_' || r == '.') { + return false + } + } + return true +} + +// Line is one archive line: an event's envelope, the SHA-256 of its canonical +// details, and — for a security-class event only — the details themselves. A +// content-class line is content-free. Details is the canonical string as the +// store keeps it, so a reader verifies a line by hashing that string. +type Line struct { + EventID string `json:"event_id"` + DeploymentID string `json:"deployment_id"` + BatchID string `json:"batch_id"` + OrgID string `json:"org_id,omitempty"` + ActorID string `json:"actor_id,omitempty"` + ActorType string `json:"actor_type"` + EventType string `json:"event_type"` + SchemaVersion int `json:"schema_version"` + Resource string `json:"resource"` + ResourceID string `json:"resource_id,omitempty"` + OccurredAt string `json:"occurred_at"` + IPAddress string `json:"ip_address,omitempty"` + ImpersonatedBy string `json:"impersonated_by,omitempty"` + IsImpersonated bool `json:"is_impersonated"` + ClientID string `json:"client_id,omitempty"` + RetentionClass string `json:"retention_class"` + DetailsSHA256 string `json:"details_sha256"` + Details string `json:"details,omitempty"` +} + +// Encode renders a batch as JSON Lines, one line per record in batch order. +func Encode(batch business.AuditBatch) ([]byte, error) { + var buf bytes.Buffer + encoder := json.NewEncoder(&buf) + encoder.SetEscapeHTML(false) + for _, record := range batch.Records { + entry := record.Entry + line := Line{ + EventID: entry.ID, + DeploymentID: batch.DeploymentID, + BatchID: batch.ID, + OrgID: entry.OrgID, + ActorID: entry.ActorID, + ActorType: entry.ActorType, + EventType: string(entry.EventType), + SchemaVersion: entry.SchemaVersion, + Resource: entry.Resource, + ResourceID: entry.ResourceID, + OccurredAt: entry.CreatedAt.UTC().Format(time.RFC3339Nano), + IPAddress: entry.IPAddress, + ImpersonatedBy: entry.ImpersonatedBy, + IsImpersonated: entry.IsImpersonated, + ClientID: entry.ClientID, + RetentionClass: string(record.Retention), + DetailsSHA256: record.DetailsSHA256, + } + if record.Retention == business.RetentionSecurity { + line.Details = record.Details + } + if err := encoder.Encode(line); err != nil { + return nil, fmt.Errorf("audit archive: encode event %s: %w", entry.ID, err) + } + } + return buf.Bytes(), nil +} diff --git a/module/services/accounts/code/pkg/auditstore/archiveformat/archiveformat_test.go b/module/services/accounts/code/pkg/auditstore/archiveformat/archiveformat_test.go new file mode 100644 index 000000000..57034c521 --- /dev/null +++ b/module/services/accounts/code/pkg/auditstore/archiveformat/archiveformat_test.go @@ -0,0 +1,104 @@ +package archiveformat + +import ( + "bufio" + "bytes" + "encoding/json" + "testing" + "time" + + "accounts/pkg/business" + + "github.com/stretchr/testify/require" +) + +var composedAt = time.Date(2026, 10, 2, 14, 30, 5, 987654321, time.UTC) + +func batchOf(t *testing.T, id string, classes ...business.AuditRetentionClass) business.AuditBatch { + t.Helper() + batch := business.AuditBatch{ID: id, DeploymentID: "deployment-1", ComposedAt: composedAt} + for _, class := range classes { + record, err := business.NewAuditRecord(business.AuditEntry{ + ID: business.NewIDString(), + OrgID: "22222222-2222-2222-2222-222222222222", + ActorID: "11111111-1111-1111-1111-111111111111", + ActorType: business.ActorTypeUser, + EventType: business.EventAuthLogin, + SchemaVersion: 1, + Resource: "session", + Payload: map[string]any{"method": "password", "note": "&"}, + CreatedAt: composedAt.Add(-time.Second), + }, class) + require.NoError(t, err) + batch.Records = append(batch.Records, record) + } + return batch +} + +func TestObjectNameIsPerBatchUnderTheDeploymentAndDay(t *testing.T) { + name, err := ObjectName(batchOf(t, "4b6d0d1e-0f3c-4c1a-9a55-6f0c2b1d7e10")) + require.NoError(t, err) + require.Equal(t, "deployment-1/2026/10/02/20261002T143005.987654321Z-4b6d0d1e-0f3c-4c1a-9a55-6f0c2b1d7e10.jsonl", name) + + other, err := ObjectName(batchOf(t, "9e2f4a6b-1c3d-4e5f-8a7b-0c1d2e3f4a5b")) + require.NoError(t, err) + require.NotEqual(t, name, other, "a redelivered batch carries a new id, so it is a new object") + + for _, bad := range []business.AuditBatch{ + {ID: "batch", DeploymentID: "../escape", ComposedAt: composedAt}, + {ID: "a/b", DeploymentID: "deployment-1", ComposedAt: composedAt}, + {ID: "", DeploymentID: "deployment-1", ComposedAt: composedAt}, + } { + _, err := ObjectName(bad) + require.Error(t, err, "%+v", bad) + } +} + +func TestEncodeWritesSecurityInFullAndContentWithoutItsDetails(t *testing.T) { + batch := batchOf(t, "batch-1", business.RetentionSecurity, business.RetentionContent) + body, err := Encode(batch) + require.NoError(t, err) + + var lines []Line + scanner := bufio.NewScanner(bytes.NewReader(body)) + for scanner.Scan() { + var line Line + require.NoError(t, json.Unmarshal(scanner.Bytes(), &line)) + lines = append(lines, line) + } + require.Len(t, lines, 2, "one line per record, in batch order") + + security, content := lines[0], lines[1] + require.Equal(t, batch.Records[0].Entry.ID, security.EventID) + require.Equal(t, "security", security.RetentionClass) + require.Equal(t, `{"method":"password","note":"&"}`, security.Details) + require.Equal(t, business.AuditDetailsSHA256(security.Details), security.DetailsSHA256, + "a reader verifies a line by hashing its details string") + + require.Equal(t, batch.Records[1].Entry.ID, content.EventID) + require.Equal(t, "content", content.RetentionClass) + require.Empty(t, content.Details, "a content-class line is content-free") + require.Equal(t, batch.Records[1].DetailsSHA256, content.DetailsSHA256, "its hash proves what the details table holds") + require.NotContains(t, string(body[bytes.IndexByte(body, '\n')+1:]), "password") + + for _, line := range lines { + require.Equal(t, "deployment-1", line.DeploymentID) + require.Equal(t, "batch-1", line.BatchID) + require.Equal(t, "22222222-2222-2222-2222-222222222222", line.OrgID) + require.Equal(t, "2026-10-02T14:30:04.987654321Z", line.OccurredAt) + } +} + +// The bytes of an archive line are a format every archive reader depends on, +// whichever object store holds them: pinned here byte for byte. +func TestEncodeIsByteStable(t *testing.T) { + batch := batchOf(t, "batch-1", business.RetentionSecurity) + batch.Records[0].Entry.ID = "33333333-3333-3333-3333-333333333333" + body, err := Encode(batch) + require.NoError(t, err) + require.Equal(t, `{"event_id":"33333333-3333-3333-3333-333333333333","deployment_id":"deployment-1","batch_id":"batch-1",`+ + `"org_id":"22222222-2222-2222-2222-222222222222","actor_id":"11111111-1111-1111-1111-111111111111","actor_type":"user",`+ + `"event_type":"saas.auth.login","schema_version":1,"resource":"session","occurred_at":"2026-10-02T14:30:04.987654321Z",`+ + `"is_impersonated":false,"retention_class":"security","details_sha256":"`+batch.Records[0].DetailsSHA256+`",`+ + `"details":"{\"method\":\"password\",\"note\":\"&\"}"}`+"\n", string(body)) +} diff --git a/module/services/accounts/code/pkg/auditstore/auditeval/aggregate.go b/module/services/accounts/code/pkg/auditstore/auditeval/aggregate.go index c3d4da7dc..e651a6248 100644 --- a/module/services/accounts/code/pkg/auditstore/auditeval/aggregate.go +++ b/module/services/accounts/code/pkg/auditstore/auditeval/aggregate.go @@ -278,7 +278,7 @@ func (a *Aggregator) Buckets() []business.AuditAggregateBucket { case "max": bucket.Metrics[alias] = acc.max case "percentile": - bucket.Metrics[alias] = percentileCont(acc.values, metric.Percentile) + bucket.Metrics[alias] = PercentileCont(acc.values, metric.Percentile) } } for _, derived := range a.spec.Derived { @@ -293,15 +293,18 @@ func (a *Aggregator) Buckets() []business.AuditAggregateBucket { return out } -// percentileCont is Postgres's percentile_cont: the value at fraction p of the -// sorted inputs, interpolated linearly between the two nearest. +// PercentileCont is Postgres's percentile_cont: the value at fraction p of the +// sorted inputs, interpolated linearly between the two nearest. values need +// not be sorted and is not modified; it must not be empty. A store that +// aggregates in its own engine collects a group's values there and computes +// the percentile here, so the interpolation rounds as Postgres's does. // // The interpolation is written as Postgres's float8_lerp writes it, lo + pct // * (hi - lo), and left for the compiler to treat as C compilers treat it: Go // and GCC both fuse that multiply-add into one rounding on arm64 and neither // does on baseline amd64, so the result matches the Postgres of the same // architecture to the last bit. Forcing either rounding would match only one. -func percentileCont(values []float64, p float64) float64 { +func PercentileCont(values []float64, p float64) float64 { sorted := append([]float64(nil), values...) sort.Float64s(sorted) position := p * float64(len(sorted)-1) diff --git a/module/services/accounts/code/pkg/auditstore/auditeval/eval.go b/module/services/accounts/code/pkg/auditstore/auditeval/eval.go index ef765572d..04a33e1c8 100644 --- a/module/services/accounts/code/pkg/auditstore/auditeval/eval.go +++ b/module/services/accounts/code/pkg/auditstore/auditeval/eval.go @@ -219,6 +219,12 @@ func (m *Matcher) EventTypes() ([]string, bool) { // must join every candidate's details before Match can decide. func (m *Matcher) NeedsPayload() bool { return m.payload != nil } +// PayloadFilter is the payload filter as Match evaluates it — the query's +// PayloadContains read back the way Postgres binds it as jsonb, numbers as +// json.Number — or nil when the read has none. A store that pushes part of it +// down reads it here, so what it pushes is exactly what Match decides. +func (m *Matcher) PayloadFilter() any { return m.payload } + // InScope reports whether an event belongs to the read's scope: its // organization's, or for the platform scope any event at all. func (m *Matcher) InScope(entry business.AuditEntry) bool { diff --git a/module/services/accounts/code/pkg/auditstore/auditeval/eval_test.go b/module/services/accounts/code/pkg/auditstore/auditeval/eval_test.go index ca6e7e6c9..fc2eb067d 100644 --- a/module/services/accounts/code/pkg/auditstore/auditeval/eval_test.go +++ b/module/services/accounts/code/pkg/auditstore/auditeval/eval_test.go @@ -99,11 +99,11 @@ func TestTimeBucket(t *testing.T) { } func TestPercentileCont(t *testing.T) { - require.Equal(t, 5.0, percentileCont([]float64{5}, 0.9)) - require.Equal(t, 2.5, percentileCont([]float64{4, 1, 3, 2}, 0.5)) - require.Equal(t, 4.0, percentileCont([]float64{4, 1, 3, 2}, 1)) - require.Equal(t, 5.0, percentileCont([]float64{20, 0, 10}, 0.25)) - require.Equal(t, 0.5+1.75/512, percentileCont([]float64{100, 0.5, 2.25}, 1.0/1024)) + require.Equal(t, 5.0, PercentileCont([]float64{5}, 0.9)) + require.Equal(t, 2.5, PercentileCont([]float64{4, 1, 3, 2}, 0.5)) + require.Equal(t, 4.0, PercentileCont([]float64{4, 1, 3, 2}, 1)) + require.Equal(t, 5.0, PercentileCont([]float64{20, 0, 10}, 0.25)) + require.Equal(t, 0.5+1.75/512, PercentileCont([]float64{100, 0.5, 2.25}, 1.0/1024)) } func entryAt(id int, at time.Time) business.AuditEntry { diff --git a/module/services/accounts/code/pkg/auditstore/auditeval/page.go b/module/services/accounts/code/pkg/auditstore/auditeval/page.go index 3d6e9ae35..4f6465e1b 100644 --- a/module/services/accounts/code/pkg/auditstore/auditeval/page.go +++ b/module/services/accounts/code/pkg/auditstore/auditeval/page.go @@ -52,6 +52,19 @@ func NewPage(q business.AuditQuery) (*Page, error) { return p, nil } +// Size is the page size: the query's, or DefaultPageSize when it names none. +func (p *Page) Size() int { return p.size } + +// After is the keyset position the page token names — the last event of the +// previous page — and whether there is one. A store that pages in its own +// engine starts strictly after it. +func (p *Page) After() (time.Time, string, bool) { + if p.after == nil { + return time.Time{}, "", false + } + return p.after.at, p.after.id, true +} + // Before is the occurrence the page token names, if any: no event at or after // it beyond the token's own tie-break can be on this page. func (p *Page) Before() *time.Time { diff --git a/module/services/accounts/code/pkg/auditstore/auditsink/auditsink.go b/module/services/accounts/code/pkg/auditstore/auditsink/auditsink.go index 9165dca8a..de3563cc0 100644 --- a/module/services/accounts/code/pkg/auditstore/auditsink/auditsink.go +++ b/module/services/accounts/code/pkg/auditstore/auditsink/auditsink.go @@ -15,12 +15,15 @@ import ( "time" "accounts/pkg/auditstore/bigquerystore" + "accounts/pkg/auditstore/clickhousestore" "accounts/pkg/auditstore/gcsarchive" + "accounts/pkg/auditstore/s3archive" "accounts/pkg/business" "cloud.google.com/go/bigquery" bqstorage "cloud.google.com/go/bigquery/storage/apiv1" "cloud.google.com/go/storage" + "github.com/ClickHouse/clickhouse-go/v2" ) // The audit sink settings. AUDIT_SINK selects the mode (business.AuditSinkMode); @@ -36,8 +39,20 @@ const ( // AUDIT_SINK=bigquery (ADR 0009). Credentials are Application Default // Credentials only — on Kubernetes, the pod's workload identity — so there // is deliberately no key-file or token setting. - EnvBigQueryProject = "AUDIT_BIGQUERY_PROJECT" - EnvBigQueryDataset = "AUDIT_BIGQUERY_DATASET" + EnvBigQueryProject = "AUDIT_BIGQUERY_PROJECT" + EnvBigQueryDataset = "AUDIT_BIGQUERY_DATASET" + + // AUDIT_SINK=clickhouse (ADR 0009). The DSN names the server, the database + // and the credentials, and comes from the deployment's secret: it has no + // default and is never logged or echoed in an error. The cluster, when set, + // makes the tables replicated across it. The events retention is the + // compliance window, the events table's TTL; BigQuery's events table does + // not expire, so AUDIT_SINK=bigquery does not read it. + EnvClickHouseDSN = "AUDIT_CLICKHOUSE_DSN" + EnvClickHouseCluster = "AUDIT_CLICKHOUSE_CLUSTER" + EnvEventsRetentionDays = "AUDIT_EVENTS_RETENTION_DAYS" + + // Every swap value. EnvArchiveURL = "AUDIT_ARCHIVE_URL" EnvDeploymentID = "AUDIT_DEPLOYMENT_ID" EnvContentRetentionDays = "AUDIT_CONTENT_RETENTION_DAYS" @@ -61,7 +76,9 @@ type Swap struct { Mode business.AuditSinkMode // BigQuery names the store of record under AUDIT_SINK=bigquery. BigQuery *BigQuery - Archive ArchiveLocation + // ClickHouse names the store of record under AUDIT_SINK=clickhouse. + ClickHouse *ClickHouse + Archive ArchiveLocation // DeploymentID stamps every record and confines every read. DeploymentID string // ContentRetention is how long a content-class event's details are kept. @@ -76,23 +93,48 @@ type BigQuery struct { Dataset string } +// ClickHouse is where the ClickHouse store of record lives. +type ClickHouse struct { + // DSN carries the credentials: it is passed to the driver and nowhere else. + DSN string + // Database is the database the DSN names. + Database string + // Cluster, when set, creates the tables ON CLUSTER, replicated. + Cluster string + // EventsRetention is the events table's TTL: the compliance window. + EventsRetention time.Duration +} + +// String names the store without its DSN, so a ClickHouse printed by mistake +// never shows the credentials. +func (c ClickHouse) String() string { + return fmt.Sprintf("clickhouse database %q (cluster %q)", c.Database, c.Cluster) +} + +// GoString is String: %#v of a Swap prints its fields this way. +func (c ClickHouse) GoString() string { return c.String() } + // ArchiveLocation is a parsed AUDIT_ARCHIVE_URL. The scheme picks the archive -// writer; gs (Google Cloud Storage) is the one built. +// writer: gs (Google Cloud Storage) or s3 (Amazon S3, or an S3-compatible +// store). Any warehouse pairs with any archive. type ArchiveLocation struct { Scheme string Bucket string } -// ArchiveSchemeGCS is the AUDIT_ARCHIVE_URL scheme of a GCS bucket. -const ArchiveSchemeGCS = "gs" +// The AUDIT_ARCHIVE_URL schemes with a writer. +const ( + ArchiveSchemeGCS = "gs" + ArchiveSchemeS3 = "s3" +) // Load reads AUDIT_SINK and the settings its mode requires through getenv // (os.Getenv in production). "postgres" (the default) keeps the durable // emitter alone. "both" adds an asynchronous tee to an HTTP endpoint while -// audit_events stays the store of record. "bigquery" swaps the store of record -// to BigQuery, with Postgres keeping only the transactional queue. "external" -// is refused: no destination outside Postgres can join the transaction a -// change commits in. +// audit_events stays the store of record. "bigquery" and "clickhouse" swap the +// store of record to that warehouse, with Postgres keeping only the +// transactional queue. "external" is refused: no destination outside Postgres +// can join the transaction a change commits in. func Load(getenv func(string) string) (Config, error) { mode, err := business.ParseAuditSinkMode(getenv(EnvSink)) if err != nil { @@ -122,26 +164,32 @@ func loadExternal(getenv func(string) string) (business.ExternalAuditSink, error }) } -// gcsBucketNamePattern is GCS's bucket naming rule, loosely: lowercase -// letters, digits, '-', '_' and '.', starting and ending alphanumeric. -var gcsBucketNamePattern = regexp.MustCompile(`^[a-z0-9][a-z0-9._-]{1,220}[a-z0-9]$`) +// Bucket naming rules, loosely. GCS: lowercase letters, digits, '-', '_' and +// '.', starting and ending alphanumeric. S3: lowercase letters, digits, '-' +// and '.', 3 to 63 of them, starting and ending alphanumeric. +var bucketNamePatterns = map[string]*regexp.Regexp{ + ArchiveSchemeGCS: regexp.MustCompile(`^[a-z0-9][a-z0-9._-]{1,220}[a-z0-9]$`), + ArchiveSchemeS3: regexp.MustCompile(`^[a-z0-9][a-z0-9.-]{1,61}[a-z0-9]$`), +} -// ParseArchiveURL reads AUDIT_ARCHIVE_URL: gs://, nothing more. -// Another scheme names an archive no writer exists for yet, which is refused -// at startup rather than left to fail at the first delivery. +// ParseArchiveURL reads AUDIT_ARCHIVE_URL: gs:// or s3://, +// nothing more. Another scheme (an Azure container, say) names an archive no +// writer exists for yet, which is refused at startup rather than left to fail +// at the first delivery. func ParseArchiveURL(raw string) (ArchiveLocation, error) { parsed, err := url.Parse(raw) if err != nil || parsed.Scheme == "" { - return ArchiveLocation{}, fmt.Errorf("%s must be a URL like gs://; got %q", EnvArchiveURL, raw) + return ArchiveLocation{}, fmt.Errorf("%s must be a URL like gs:// or s3://; got %q", EnvArchiveURL, raw) } - if parsed.Scheme != ArchiveSchemeGCS { - return ArchiveLocation{}, fmt.Errorf("%s scheme %q has no archive writer; the supported scheme is %s:// (got %q)", - EnvArchiveURL, parsed.Scheme, ArchiveSchemeGCS, raw) + pattern, supported := bucketNamePatterns[parsed.Scheme] + if !supported { + return ArchiveLocation{}, fmt.Errorf("%s scheme %q has no archive writer; the supported schemes are %s:// and %s:// (got %q)", + EnvArchiveURL, parsed.Scheme, ArchiveSchemeGCS, ArchiveSchemeS3, raw) } if parsed.User != nil || parsed.Port() != "" || (parsed.Path != "" && parsed.Path != "/") || parsed.RawQuery != "" || parsed.Fragment != "" { - return ArchiveLocation{}, fmt.Errorf("%s names a bucket and nothing else, like gs://; got %q", EnvArchiveURL, raw) + return ArchiveLocation{}, fmt.Errorf("%s names a bucket and nothing else, like %s://; got %q", EnvArchiveURL, parsed.Scheme, raw) } - if !gcsBucketNamePattern.MatchString(parsed.Host) { + if !pattern.MatchString(parsed.Host) { return ArchiveLocation{}, fmt.Errorf("%s bucket %q is not a valid bucket name", EnvArchiveURL, parsed.Host) } return ArchiveLocation{Scheme: parsed.Scheme, Bucket: parsed.Host}, nil @@ -151,26 +199,45 @@ func ParseArchiveURL(raw string) (ArchiveLocation, error) { // and a warehouse value without escaping. var deploymentIDPattern = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$`) +// setting is one required setting of a swap value and what it was set to. +type setting struct{ name, value string } + // loadSwap reads and validates every setting of a swap value, reporting all // the missing ones at once. func loadSwap(mode business.AuditSinkMode, getenv func(string) string) (*Swap, error) { swap := &Swap{ Mode: mode, - BigQuery: &BigQuery{Project: strings.TrimSpace(getenv(EnvBigQueryProject)), Dataset: strings.TrimSpace(getenv(EnvBigQueryDataset))}, DeploymentID: strings.TrimSpace(getenv(EnvDeploymentID)), RelayBatchSize: business.DefaultAuditRelayBatchSize, RelayMaxWait: business.DefaultAuditRelayMaxWait, } archiveURL := strings.TrimSpace(getenv(EnvArchiveURL)) retentionDays := strings.TrimSpace(getenv(EnvContentRetentionDays)) - var missing []string - for _, setting := range []struct{ name, value string }{ - {EnvBigQueryProject, swap.BigQuery.Project}, - {EnvBigQueryDataset, swap.BigQuery.Dataset}, + shared := []setting{ {EnvArchiveURL, archiveURL}, {EnvDeploymentID, swap.DeploymentID}, {EnvContentRetentionDays, retentionDays}, - } { + } + var required []setting + var eventsRetentionDays string + switch mode { + case business.AuditSinkBigQuery: + swap.BigQuery = &BigQuery{Project: strings.TrimSpace(getenv(EnvBigQueryProject)), Dataset: strings.TrimSpace(getenv(EnvBigQueryDataset))} + required = append([]setting{ + {EnvBigQueryProject, swap.BigQuery.Project}, + {EnvBigQueryDataset, swap.BigQuery.Dataset}, + }, shared...) + case business.AuditSinkClickHouse: + swap.ClickHouse = &ClickHouse{DSN: strings.TrimSpace(getenv(EnvClickHouseDSN)), Cluster: strings.TrimSpace(getenv(EnvClickHouseCluster))} + eventsRetentionDays = strings.TrimSpace(getenv(EnvEventsRetentionDays)) + required = append(append([]setting{ + {EnvClickHouseDSN, swap.ClickHouse.DSN}, + }, shared...), setting{EnvEventsRetentionDays, eventsRetentionDays}) + default: + return nil, fmt.Errorf("audit sink: %s is not a swap value", mode) + } + var missing []string + for _, setting := range required { if setting.value == "" { missing = append(missing, setting.name) } @@ -184,6 +251,11 @@ func loadSwap(mode business.AuditSinkMode, getenv func(string) string) (*Swap, e return nil, fmt.Errorf("%s must be a whole number of days, at least 1; got %q", EnvContentRetentionDays, retentionDays) } swap.ContentRetention = time.Duration(days) * 24 * time.Hour + if swap.ClickHouse != nil { + if err := loadClickHouse(swap, eventsRetentionDays); err != nil { + return nil, err + } + } if !deploymentIDPattern.MatchString(swap.DeploymentID) { return nil, fmt.Errorf("%s must be letters, digits, '.', '_' or '-' (at most 128, starting with a letter or digit); got %q", EnvDeploymentID, swap.DeploymentID) @@ -208,6 +280,38 @@ func loadSwap(mode business.AuditSinkMode, getenv func(string) string) (*Swap, e return swap, nil } +// loadClickHouse validates the ClickHouse settings: a DSN the driver parses +// and that names its database, a cluster name ON CLUSTER can take, and an +// events window at least as long as the content window — content details +// outliving their event would be details of an event the store no longer has. +func loadClickHouse(swap *Swap, eventsRetentionDays string) error { + settings := swap.ClickHouse + days, err := strconv.Atoi(eventsRetentionDays) + if err != nil || days < 1 { + return fmt.Errorf("%s must be a whole number of days, at least 1; got %q", EnvEventsRetentionDays, eventsRetentionDays) + } + settings.EventsRetention = time.Duration(days) * 24 * time.Hour + if settings.EventsRetention < swap.ContentRetention { + return fmt.Errorf("%s (%d) must be at least %s (%d): content details are kept no longer than their events", + EnvEventsRetentionDays, days, EnvContentRetentionDays, int(swap.ContentRetention/(24*time.Hour))) + } + options, err := clickhouse.ParseDSN(settings.DSN) + if err != nil { + // The parse error can quote the DSN, credentials included, so none of + // it is reported. + return fmt.Errorf("%s is not a valid ClickHouse DSN; it looks like clickhouse://:@:9000/", EnvClickHouseDSN) + } + if options.Auth.Database == "" { + return fmt.Errorf("%s must name the database, like clickhouse://:9000/", EnvClickHouseDSN) + } + settings.Database = options.Auth.Database + if settings.Cluster != "" && !clickhousestore.ValidCluster(settings.Cluster) { + return fmt.Errorf("%s must be a cluster name (letters, digits, '_', '.', '-') or a macro like {cluster}; got %q", + EnvClickHouseCluster, settings.Cluster) + } + return nil +} + // Opened is a swap value's store of record and archive, built and checked. type Opened struct { // Store is the store of record: the relay appends to it and the service @@ -230,8 +334,9 @@ func (o *Opened) Close() { } // Open builds the store of record of a swap value — its tables created if -// missing, its read half on the Storage Read API — and the archive. Every -// client uses Application Default Credentials. +// missing — and the archive its URL's scheme names. BigQuery and GCS clients +// use Application Default Credentials; ClickHouse connects with the DSN; the +// S3 client takes the AWS SDK's default chain. func Open(ctx context.Context, swap *Swap) (*Opened, error) { if swap == nil { return nil, errors.New("audit sink: no swap value to open") @@ -241,20 +346,40 @@ func Open(ctx context.Context, swap *Swap) (*Opened, error) { opened.Close() return nil, err } - if swap.Mode != business.AuditSinkBigQuery || swap.BigQuery == nil { - return fail(fmt.Errorf("audit sink: no store of record for AUDIT_SINK=%s", swap.Mode)) + var err error + switch { + case swap.Mode == business.AuditSinkBigQuery && swap.BigQuery != nil: + err = opened.openBigQuery(ctx, swap) + case swap.Mode == business.AuditSinkClickHouse && swap.ClickHouse != nil: + err = opened.openClickHouse(ctx, swap) + default: + err = fmt.Errorf("audit sink: no store of record for AUDIT_SINK=%s", swap.Mode) + } + if err != nil { + return fail(err) } + if err := opened.openArchive(ctx, swap.Archive); err != nil { + return fail(err) + } + return opened, nil +} +// ensureTimeout bounds creating and checking the store's tables at startup. +const ensureTimeout = time.Minute + +// openBigQuery builds the BigQuery store: streaming inserts for the relay, and +// reads on the Storage Read API. +func (opened *Opened) openBigQuery(ctx context.Context, swap *Swap) error { bigQueryClient, err := bigquery.NewClient(ctx, swap.BigQuery.Project) if err != nil { - return fail(fmt.Errorf("audit store: bigquery client: %w", err)) + return fmt.Errorf("audit store: bigquery client: %w", err) } opened.closers = append(opened.closers, func() { _ = bigQueryClient.Close() }) // Reads run no query job: the service holds the append grant, and append // plus job creation would let it run DML against the store of record. readClient, err := bqstorage.NewBigQueryReadClient(ctx) if err != nil { - return fail(fmt.Errorf("audit store: bigquery storage read client: %w", err)) + return fmt.Errorf("audit store: bigquery storage read client: %w", err) } opened.closers = append(opened.closers, func() { _ = readClient.Close() }) reader, err := bigquerystore.NewReader(bigquerystore.ReadConfig{ @@ -264,7 +389,7 @@ func Open(ctx context.Context, swap *Swap) (*Opened, error) { DeploymentID: swap.DeploymentID, }) if err != nil { - return fail(err) + return err } warehouse, err := bigquerystore.New(bigQueryClient, bigquerystore.Config{ Dataset: swap.BigQuery.Dataset, @@ -272,26 +397,66 @@ func Open(ctx context.Context, swap *Swap) (*Opened, error) { Reader: reader, }) if err != nil { - return fail(err) + return err } - ensureCtx, cancel := context.WithTimeout(ctx, time.Minute) + ensureCtx, cancel := context.WithTimeout(ctx, ensureTimeout) err = warehouse.Ensure(ensureCtx) cancel() if err != nil { - return fail(err) + return err } opened.Store, opened.History = warehouse, warehouse + return nil +} - if swap.Archive.Scheme != ArchiveSchemeGCS { - return fail(fmt.Errorf("audit archive: no writer for scheme %q", swap.Archive.Scheme)) +// openClickHouse connects with the DSN and builds the ClickHouse store, which +// writes and reads over that one connection pool. +func (opened *Opened) openClickHouse(ctx context.Context, swap *Swap) error { + options, err := clickhouse.ParseDSN(swap.ClickHouse.DSN) + if err != nil { + // Load has parsed it already; never echo it. + return fmt.Errorf("audit store: %s is not a valid ClickHouse DSN", EnvClickHouseDSN) } - storageClient, err := storage.NewClient(ctx) + conn, err := clickhouse.Open(options) if err != nil { - return fail(fmt.Errorf("audit archive: storage client: %w", err)) + return fmt.Errorf("audit store: clickhouse connection: %w", err) } - opened.closers = append(opened.closers, func() { _ = storageClient.Close() }) - if opened.Archive, err = gcsarchive.New(storageClient, swap.Archive.Bucket); err != nil { - return fail(err) + opened.closers = append(opened.closers, func() { _ = conn.Close() }) + warehouse, err := clickhousestore.New(conn, clickhousestore.Config{ + Database: swap.ClickHouse.Database, + Cluster: swap.ClickHouse.Cluster, + DeploymentID: swap.DeploymentID, + EventsRetention: swap.ClickHouse.EventsRetention, + ContentDetailRetention: swap.ContentRetention, + }) + if err != nil { + return err } - return opened, nil + ensureCtx, cancel := context.WithTimeout(ctx, ensureTimeout) + err = warehouse.Ensure(ensureCtx) + cancel() + if err != nil { + return err + } + opened.Store, opened.History = warehouse, warehouse + return nil +} + +// openArchive builds the writer the archive URL's scheme names. +func (opened *Opened) openArchive(ctx context.Context, location ArchiveLocation) error { + var err error + switch location.Scheme { + case ArchiveSchemeGCS: + storageClient, clientErr := storage.NewClient(ctx) + if clientErr != nil { + return fmt.Errorf("audit archive: storage client: %w", clientErr) + } + opened.closers = append(opened.closers, func() { _ = storageClient.Close() }) + opened.Archive, err = gcsarchive.New(storageClient, location.Bucket) + case ArchiveSchemeS3: + opened.Archive, err = s3archive.NewFromEnvironment(ctx, location.Bucket) + default: + err = fmt.Errorf("audit archive: no writer for scheme %q", location.Scheme) + } + return err } diff --git a/module/services/accounts/code/pkg/auditstore/auditsink/auditsink_test.go b/module/services/accounts/code/pkg/auditstore/auditsink/auditsink_test.go index 13b4a5088..32acec00d 100644 --- a/module/services/accounts/code/pkg/auditstore/auditsink/auditsink_test.go +++ b/module/services/accounts/code/pkg/auditstore/auditsink/auditsink_test.go @@ -1,6 +1,7 @@ package auditsink import ( + "fmt" "os" "testing" "time" @@ -43,14 +44,14 @@ func TestConfiguredAuditSinkRejectsExternalOnly(t *testing.T) { t.Setenv("AUDIT_SINK", "external") _, err := Load(os.Getenv) require.ErrorContains(t, err, "not permitted") - require.ErrorContains(t, err, "bigquery", "the refusal names the swap value that does what external cannot") + require.ErrorContains(t, err, "bigquery or clickhouse", "the refusal names the swap values that do what external cannot") } func TestConfiguredAuditSinkRejectsUnknownMode(t *testing.T) { - for _, value := range []string{"kafka", "clickhouse"} { + for _, value := range []string{"kafka", "redshift"} { t.Setenv("AUDIT_SINK", value) _, err := Load(os.Getenv) - require.ErrorContains(t, err, "must be postgres, both or bigquery", value) + require.ErrorContains(t, err, "must be postgres, both, bigquery or clickhouse", value) } } @@ -124,9 +125,11 @@ func TestConfiguredAuditSinkBigQueryRejectsInvalidSettings(t *testing.T) { {"AUDIT_CONTENT_RETENTION_DAYS", "-1", "AUDIT_CONTENT_RETENTION_DAYS"}, {"AUDIT_DEPLOYMENT_ID", "acme/prod", "AUDIT_DEPLOYMENT_ID"}, {"AUDIT_DEPLOYMENT_ID", "..", "AUDIT_DEPLOYMENT_ID"}, - {"AUDIT_ARCHIVE_URL", "example-audit-archive", "like gs://"}, - {"AUDIT_ARCHIVE_URL", "s3://example-audit-archive", `scheme "s3" has no archive writer`}, + {"AUDIT_ARCHIVE_URL", "example-audit-archive", "like gs:// or s3://"}, + {"AUDIT_ARCHIVE_URL", "azblob://example-audit-archive", `scheme "azblob" has no archive writer; the supported schemes are gs:// and s3://`}, {"AUDIT_ARCHIVE_URL", "https://storage.example/archive", `scheme "https" has no archive writer`}, + {"AUDIT_ARCHIVE_URL", "s3://example-audit-archive/audit", "names a bucket and nothing else, like s3://"}, + {"AUDIT_ARCHIVE_URL", "s3://example_audit_archive", "not a valid bucket name"}, {"AUDIT_ARCHIVE_URL", "gs://example-audit-archive/audit", "names a bucket and nothing else"}, {"AUDIT_ARCHIVE_URL", "gs://user@example-audit-archive", "names a bucket and nothing else"}, {"AUDIT_ARCHIVE_URL", "gs://Example_Archive", "not a valid bucket name"}, @@ -144,3 +147,113 @@ func TestConfiguredAuditSinkBigQueryRejectsInvalidSettings(t *testing.T) { }) } } + +func TestConfiguredAuditSinkTakesAnS3Archive(t *testing.T) { + setBigQuerySwap(t) + t.Setenv("AUDIT_ARCHIVE_URL", "s3://example-audit-archive") + sink, err := Load(os.Getenv) + require.NoError(t, err, "any warehouse pairs with any archive") + require.Equal(t, ArchiveLocation{Scheme: "s3", Bucket: "example-audit-archive"}, sink.Swap.Archive) +} + +// setClickHouseSwap sets every required clickhouse setting to a valid value. +func setClickHouseSwap(t *testing.T) { + t.Helper() + t.Setenv("AUDIT_SINK", "clickhouse") + t.Setenv("AUDIT_CLICKHOUSE_DSN", "clickhouse://audit:s3cr3t-value@clickhouse.example:9440/audit?secure=true") + t.Setenv("AUDIT_CLICKHOUSE_CLUSTER", "") + t.Setenv("AUDIT_EVENTS_RETENTION_DAYS", "2555") + t.Setenv("AUDIT_ARCHIVE_URL", "s3://example-audit-archive") + t.Setenv("AUDIT_DEPLOYMENT_ID", "acme-prod-1") + t.Setenv("AUDIT_CONTENT_RETENTION_DAYS", "90") + t.Setenv("AUDIT_RELAY_BATCH_SIZE", "") + t.Setenv("AUDIT_RELAY_MAX_WAIT", "") +} + +func TestConfiguredAuditSinkClickHouse(t *testing.T) { + setClickHouseSwap(t) + sink, err := Load(os.Getenv) + require.NoError(t, err) + require.Equal(t, business.AuditSinkClickHouse, sink.Mode) + require.True(t, sink.Mode.Swaps()) + require.Equal(t, &Swap{ + Mode: business.AuditSinkClickHouse, + ClickHouse: &ClickHouse{ + DSN: "clickhouse://audit:s3cr3t-value@clickhouse.example:9440/audit?secure=true", + Database: "audit", + EventsRetention: 2555 * 24 * time.Hour, + }, + Archive: ArchiveLocation{Scheme: "s3", Bucket: "example-audit-archive"}, + DeploymentID: "acme-prod-1", + ContentRetention: 90 * 24 * time.Hour, + RelayBatchSize: business.DefaultAuditRelayBatchSize, + RelayMaxWait: business.DefaultAuditRelayMaxWait, + }, sink.Swap) + require.Nil(t, sink.Swap.BigQuery) + + t.Setenv("AUDIT_CLICKHOUSE_CLUSTER", "{cluster}") + t.Setenv("AUDIT_ARCHIVE_URL", "gs://example-audit-archive") + sink, err = Load(os.Getenv) + require.NoError(t, err) + require.Equal(t, "{cluster}", sink.Swap.ClickHouse.Cluster) + require.Equal(t, "gs", sink.Swap.Archive.Scheme, "any warehouse pairs with any archive") + + t.Setenv("AUDIT_EVENTS_RETENTION_DAYS", "90") + _, err = Load(os.Getenv) + require.NoError(t, err, "the events window may equal the content window") +} + +func TestConfiguredAuditSinkClickHouseNamesEveryMissingSetting(t *testing.T) { + t.Setenv("AUDIT_SINK", "clickhouse") + for _, name := range []string{ + "AUDIT_CLICKHOUSE_DSN", "AUDIT_ARCHIVE_URL", "AUDIT_DEPLOYMENT_ID", + "AUDIT_CONTENT_RETENTION_DAYS", "AUDIT_EVENTS_RETENTION_DAYS", + "AUDIT_BIGQUERY_PROJECT", "AUDIT_BIGQUERY_DATASET", + } { + t.Setenv(name, "") + } + _, err := Load(os.Getenv) + require.EqualError(t, err, "AUDIT_SINK=clickhouse requires AUDIT_CLICKHOUSE_DSN, AUDIT_ARCHIVE_URL, "+ + "AUDIT_DEPLOYMENT_ID, AUDIT_CONTENT_RETENTION_DAYS, AUDIT_EVENTS_RETENTION_DAYS") + + setClickHouseSwap(t) + t.Setenv("AUDIT_CLICKHOUSE_DSN", " \n") + _, err = Load(os.Getenv) + require.EqualError(t, err, "AUDIT_SINK=clickhouse requires AUDIT_CLICKHOUSE_DSN", "a blank value is a missing one") +} + +func TestConfiguredAuditSinkClickHouseRejectsInvalidSettings(t *testing.T) { + for _, tc := range []struct { + name, value, want string + }{ + {"AUDIT_EVENTS_RETENTION_DAYS", "0", "AUDIT_EVENTS_RETENTION_DAYS must be a whole number of days"}, + {"AUDIT_EVENTS_RETENTION_DAYS", "7y", "AUDIT_EVENTS_RETENTION_DAYS must be a whole number of days"}, + {"AUDIT_EVENTS_RETENTION_DAYS", "30", "AUDIT_EVENTS_RETENTION_DAYS (30) must be at least AUDIT_CONTENT_RETENTION_DAYS (90)"}, + {"AUDIT_CONTENT_RETENTION_DAYS", "0", "AUDIT_CONTENT_RETENTION_DAYS"}, + {"AUDIT_CLICKHOUSE_DSN", "clickhouse://audit:s3cr3t-value@clickhouse.example:9440", "must name the database"}, + {"AUDIT_CLICKHOUSE_DSN", "clickhouse://audit:s3cr3t-value@/audit", "not a valid ClickHouse DSN"}, + {"AUDIT_CLICKHOUSE_DSN", "clickhouse://audit:s3cr3t-value@clickhouse.example:9440/audit?dial_timeout=s3cr3t-value", "not a valid ClickHouse DSN"}, + {"AUDIT_CLICKHOUSE_DSN", "clickhouse://audit:s3cr3t-value\x7f@clickhouse.example/audit", "not a valid ClickHouse DSN"}, + {"AUDIT_CLICKHOUSE_CLUSTER", "prod' ON CLUSTER 'x", "AUDIT_CLICKHOUSE_CLUSTER must be a cluster name"}, + {"AUDIT_DEPLOYMENT_ID", "acme/prod", "AUDIT_DEPLOYMENT_ID"}, + {"AUDIT_ARCHIVE_URL", "azblob://example-audit-archive", "has no archive writer"}, + } { + t.Run(tc.name+"="+tc.value, func(t *testing.T) { + setClickHouseSwap(t) + t.Setenv(tc.name, tc.value) + _, err := Load(os.Getenv) + require.ErrorContains(t, err, tc.want) + require.NotContains(t, err.Error(), "s3cr3t-value", "a refusal never echoes the DSN's credentials") + }) + } +} + +func TestClickHouseSettingsNeverPrintTheDSN(t *testing.T) { + setClickHouseSwap(t) + sink, err := Load(os.Getenv) + require.NoError(t, err) + for _, format := range []string{"%v", "%+v", "%#v", "%s"} { + require.NotContains(t, fmt.Sprintf(format, *sink.Swap.ClickHouse), "s3cr3t-value", format) + require.NotContains(t, fmt.Sprintf(format, sink.Swap.ClickHouse), "s3cr3t-value", format) + } +} diff --git a/module/services/accounts/code/pkg/auditstore/auditsink/open_test.go b/module/services/accounts/code/pkg/auditstore/auditsink/open_test.go new file mode 100644 index 000000000..0147f697f --- /dev/null +++ b/module/services/accounts/code/pkg/auditstore/auditsink/open_test.go @@ -0,0 +1,85 @@ +package auditsink + +import ( + "context" + "net/url" + "os" + "strings" + "testing" + "time" + + "accounts/pkg/auditstore/s3archive" + "accounts/pkg/business" + + "github.com/ClickHouse/clickhouse-go/v2" + "github.com/aws/aws-sdk-go-v2/aws" + "github.com/aws/aws-sdk-go-v2/config" + "github.com/aws/aws-sdk-go-v2/service/s3" + "github.com/google/uuid" + "github.com/stretchr/testify/require" +) + +// Open end to end — the settings as the service reads them, a ClickHouse store +// of record and an S3 archive — when both servers are reachable (see +// clickhousestore and s3archive for how to run them): +// +// AUDIT_CLICKHOUSE_TEST_DSN=clickhouse://:@127.0.0.1:9000/default \ +// AUDIT_S3_TEST_ENDPOINT=http://127.0.0.1:9002 AWS_ACCESS_KEY_ID= AWS_SECRET_ACCESS_KEY= \ +// AWS_REGION=us-east-1 go test ./pkg/auditstore/auditsink +// +// It is skipped otherwise. The test creates the database and the bucket; Open +// creates neither. +func TestOpenClickHouseWithAnS3Archive(t *testing.T) { + dsn := strings.TrimSpace(os.Getenv("AUDIT_CLICKHOUSE_TEST_DSN")) + endpoint := strings.TrimSpace(os.Getenv("AUDIT_S3_TEST_ENDPOINT")) + if dsn == "" || endpoint == "" { + t.Skip("AUDIT_CLICKHOUSE_TEST_DSN and AUDIT_S3_TEST_ENDPOINT are not both set") + } + ctx := context.Background() + options, err := clickhouse.ParseDSN(dsn) + require.NoError(t, err) + admin, err := clickhouse.Open(options) + require.NoError(t, err) + t.Cleanup(func() { _ = admin.Close() }) + database := "audit_open_" + strings.ReplaceAll(uuid.NewString(), "-", "")[:12] + require.NoError(t, admin.Exec(ctx, "CREATE DATABASE "+database)) + t.Cleanup(func() { _ = admin.Exec(context.Background(), "DROP DATABASE IF EXISTS "+database) }) + + t.Setenv("AWS_ENDPOINT_URL_S3", endpoint) + awsConfig, err := config.LoadDefaultConfig(ctx) + require.NoError(t, err) + bucket := "audit-open-" + strings.ReplaceAll(uuid.NewString(), "-", "")[:12] + _, err = s3.NewFromConfig(awsConfig, s3archive.PathStyleForEndpointOverride).CreateBucket(ctx, &s3.CreateBucketInput{Bucket: aws.String(bucket)}) + require.NoError(t, err) + + scoped, err := url.Parse(dsn) + require.NoError(t, err) + scoped.Path = "/" + database + setClickHouseSwap(t) + t.Setenv("AUDIT_CLICKHOUSE_DSN", scoped.String()) + t.Setenv("AUDIT_ARCHIVE_URL", "s3://"+bucket) + sink, err := Load(os.Getenv) + require.NoError(t, err) + opened, err := Open(ctx, sink.Swap) + require.NoError(t, err) + t.Cleanup(opened.Close) + + org := uuid.NewString() + record, err := business.NewAuditRecord(business.AuditEntry{ + ID: uuid.NewString(), OrgID: org, ActorType: business.ActorTypeSystem, EventType: business.EventAuthLogin, + SchemaVersion: 1, Resource: "session", Payload: map[string]any{"method": "password"}, + CreatedAt: time.Now().UTC().Truncate(time.Microsecond), + }, business.RetentionSecurity) + require.NoError(t, err) + batch := business.AuditBatch{ID: uuid.NewString(), DeploymentID: sink.Swap.DeploymentID, ComposedAt: time.Now(), Records: []business.AuditRecord{record}} + require.NoError(t, opened.Store.AppendAuditBatch(ctx, batch)) + require.NoError(t, opened.Archive.WriteAuditBatch(ctx, batch)) + + entries, _, err := opened.Store.ListAuditEvents(ctx, business.AuditRead{ + Scope: business.OrganizationAuditScope(org), Query: business.AuditQuery{OrgID: org}, + }) + require.NoError(t, err) + require.Len(t, entries, 1) + require.Equal(t, record.Entry.ID, entries[0].ID) + require.Equal(t, map[string]any{"method": "password"}, entries[0].Payload) +} diff --git a/module/services/accounts/code/pkg/auditstore/clickhousestore/aggregate.go b/module/services/accounts/code/pkg/auditstore/clickhousestore/aggregate.go new file mode 100644 index 000000000..517f6b5f5 --- /dev/null +++ b/module/services/accounts/code/pkg/auditstore/clickhousestore/aggregate.go @@ -0,0 +1,257 @@ +package clickhousestore + +import ( + "fmt" + "sort" + "strings" + + "accounts/pkg/business" +) + +// The aggregation, in ClickHouse. AggregateAuditLog's dimensions and metrics +// run as one GROUP BY over the deduplicated events, ordered as Postgres orders +// them, wherever ClickHouse computes exactly what Postgres does: +// +// - event_type, actor and category (the event-type index bound as two +// parallel arrays) are column values; +// - a time bucket is the UTC day, ISO week or month, printed as Postgres's +// to_char prints it; +// - count and count_distinct over a column are exact; +// - a payload key's numeric reading (a JSON number, or a JSON string holding +// a plain decimal) parses to the same double — ClickHouse's JSON parser +// and its precise_float_parsing both round correctly, as float8in does — +// so min and max are the same, and sum and avg add the same values. The +// order of the additions is ClickHouse's there and the scan's in +// Postgres, which no plan fixes: sums of exactly representable values +// agree to the bit, as two Postgres plans would; +// - percentile_cont's inputs are collected per group and interpolated in +// the service (auditeval.PercentileCont), whose rounding matches +// Postgres's; +// - a payload key's ->> text is exact for a string, a boolean, an integer +// or no value at all. +// +// It is not exact for the ->> text of a fractional number (jsonb keeps its +// scale and digits, ClickHouse reprints the double) or of an object or array +// (jsonb's key order and spacing), nor for any payload ClickHouse's parser +// refuses — valid JSON such as an integer beyond 64 bits, which every JSON +// function then reads as empty. The statement counts the rows where a payload +// dimension or count_distinct meets such a value, and the unparsed payloads +// of any aggregation that reads payloads; when any row counts, the +// aggregation is evaluated in the service over the deduplicated rows instead +// (auditeval), so the answer is Postgres's either way. + +// aggregation is a rendered aggregation statement and how to read its rows: +// per group, the dimensions d0..dn, the count, each metric, each metric's +// sample count, and the inexact-row count. +type aggregation struct { + statement string + args []any + dims int + metrics []business.AuditMetric + aliases []string +} + +// numericReading is the regular expression Postgres's numeric reading applies +// to a JSON string (auditNumericExpr); ClickHouse's match is RE2, which reads +// it identically. +const numericReading = `'^-?[0-9]+(\\.[0-9]+)?$'` + +// payloadExprs renders the SQL forms of payload keys over the payload column, +// binding each key once. +type payloadExprs struct { + p *sqlParams + keys map[string]string + inexact []string + inexactK map[string]bool +} + +func (x *payloadExprs) key(name string) string { + if bound, ok := x.keys[name]; ok { + return bound + } + bound := x.p.text(name) + x.keys[name] = bound + return bound +} + +// text is payload ->> key, NULL when the key is missing or null. A text of a +// kind ClickHouse cannot print as jsonb does is NULL here too, and counted as +// inexact. +func (x *payloadExprs) text(name string) string { + k := x.key(name) + if !x.inexactK[name] { + x.inexactK[name] = true + x.inexact = append(x.inexact, "JSONType(payload, "+k+") IN ('Double', 'Array', 'Object')") + } + return "multiIf(JSONType(payload, " + k + ") = 'String', JSONExtractString(payload, " + k + "), " + + "JSONType(payload, " + k + ") IN ('Int64', 'UInt64', 'Bool'), JSONExtractRaw(payload, " + k + "), NULL)" +} + +// number is the numeric reading of a payload key, NULL when it has none. +func (x *payloadExprs) number(name string) string { + k := x.key(name) + return "multiIf(JSONType(payload, " + k + ") IN ('Int64', 'UInt64', 'Double'), JSONExtractFloat(payload, " + k + "), " + + "JSONType(payload, " + k + ") = 'String' AND match(JSONExtractString(payload, " + k + "), " + numericReading + "), " + + "toFloat64OrNull(JSONExtractString(payload, " + k + ")), NULL)" +} + +// usesPayload reports whether an aggregation reads payloads. +func usesPayload(pl plan, spec business.AuditAggregationSpec) bool { + if pl.m.NeedsPayload() { + return true + } + for _, dimension := range spec.GroupBy { + if strings.HasPrefix(dimension, "payload:") { + return true + } + } + for _, metric := range spec.Metrics { + if strings.HasPrefix(metric.Field, "payload:") { + return true + } + } + return false +} + +// buildAggregation renders the aggregation of a read whose payload filter, if +// any, is pushed down. ok is false when the read can match nothing. +func (s *Store) buildAggregation(pl plan, spec business.AuditAggregationSpec, types business.AuditEventTypeIndex) (aggregation, bool, error) { + p := &sqlParams{} + payload := usesPayload(pl, spec) + var source string + if payload { + joined, ok := s.joinedEvents(p, pl, false) + if !ok { + return aggregation{}, false, nil + } + source = joined + if pl.payload != nil { + source = "SELECT * FROM (" + source + ") WHERE " + payloadCandidates(p, "payload", pl.payload) + } + } else { + events, ok := s.eventsWhere(p, pl) + if !ok { + return aggregation{}, false, nil + } + source = "SELECT event_type, actor_id, occurred_at, resource, resource_id FROM " + EventsTable + + " WHERE " + events.String() + " LIMIT 1 BY event_id" + } + + x := &payloadExprs{p: p, keys: map[string]string{}, inexactK: map[string]bool{}} + var names, categories string + category := func() (string, string) { + if names == "" { + // Sorted, so the statement is the same for the same index. + typeNames := make([]string, 0, len(types)) + for name := range types { + typeNames = append(typeNames, name) + } + sort.Strings(typeNames) + typeCategories := make([]string, len(typeNames)) + for i, name := range typeNames { + typeCategories[i] = types[name].Category + } + names, categories = p.texts(typeNames), p.texts(typeCategories) + } + return names, categories + } + + dims := spec.GroupBy + if len(dims) == 0 { + dims = []string{"event_type"} + } + var selects, groups, order []string + for i, dim := range dims { + var expr string + if key, ok := strings.CutPrefix(dim, "payload:"); ok && key != "" { + expr = "ifNull(" + x.text(key) + ", '')" + } else { + switch dim { + case "event_type": + expr = "toString(event_type)" + case "actor": + expr = "actor_id" + case "category": + n, c := category() + expr = "transform(toString(event_type), " + n + ", " + c + ", 'unknown')" + case "time": + expr = timeBucket(spec.Bucket) + default: + return aggregation{}, false, fmt.Errorf("audit: unhandled group dimension %q", dim) + } + } + alias := fmt.Sprintf("d%d", i) + selects = append(selects, expr+" AS "+alias) + groups = append(groups, alias) + order = append(order, alias+" ASC") + } + selects = append(selects, "count() AS cnt") + + aliases := make([]string, len(spec.Metrics)) + samples := make([]string, len(spec.Metrics)) + for i, metric := range spec.Metrics { + aliases[i] = metric.ResolvedAlias() + var value, sample string + switch metric.Op { + case "", "count": + value, sample = "toFloat64(count())", "count()" + case "count_distinct": + if key, ok := strings.CutPrefix(metric.Field, "payload:"); ok && key != "" { + text := x.text(key) + value, sample = "toFloat64(uniqExact("+text+"))", "count("+text+")" + break + } + switch metric.Field { + case "actor_id", "resource_id": + // Empty is no value, as NULL is in Postgres. + value = "toFloat64(uniqExactIf(" + metric.Field + ", " + metric.Field + " != ''))" + sample = "countIf(" + metric.Field + " != '')" + case "event_type", "resource": + value, sample = "toFloat64(uniqExact("+metric.Field+"))", "count()" + case "category": + n, c := category() + known := "if(has(" + n + ", toString(event_type)), transform(toString(event_type), " + n + ", " + c + ", ''), NULL)" + value, sample = "toFloat64(uniqExact("+known+"))", "count("+known+")" + default: + return aggregation{}, false, fmt.Errorf("audit: unhandled count_distinct field %q", metric.Field) + } + case "sum", "avg", "min", "max": + number := x.number(strings.TrimPrefix(metric.Field, "payload:")) + value, sample = "ifNull("+metric.Op+"("+number+"), 0)", "count("+number+")" + case "percentile": + number := x.number(strings.TrimPrefix(metric.Field, "payload:")) + value, sample = "groupArray("+number+")", "count("+number+")" + default: + return aggregation{}, false, fmt.Errorf("audit: unhandled metric op %q", metric.Op) + } + selects = append(selects, fmt.Sprintf("%s AS m%d", value, i)) + samples[i] = fmt.Sprintf("%s AS s%d", sample, i) + } + selects = append(selects, samples...) + inexact := "toUInt64(0)" + if payload { + inexact = "countIf(" + strings.Join(append([]string{unparsed("payload")}, x.inexact...), " OR ") + ")" + } + selects = append(selects, inexact+" AS inexact") + + orderBy := "cnt DESC, " + strings.Join(order, ", ") + if len(dims) == 1 && dims[0] == "time" { + orderBy = "d0 ASC" + } + statement := "SELECT " + strings.Join(selects, ", ") + " FROM (" + source + ") GROUP BY " + strings.Join(groups, ", ") + " ORDER BY " + orderBy + return aggregation{statement: statement, args: p.args, dims: len(dims), metrics: spec.Metrics, aliases: aliases}, true, nil +} + +// timeBucket is the time dimension: the UTC start of the event's day (the +// default), ISO week or month, as to_char(date_trunc(grain, created_at), +// 'YYYY-MM-DD"T"HH24:MI:SSOF') prints it under the UTC session time zone. +func timeBucket(bucket string) string { + start := "toDate(occurred_at, 'UTC')" + switch bucket { + case "week": + start = "toMonday(occurred_at, 'UTC')" + case "month": + start = "toStartOfMonth(occurred_at, 'UTC')" + } + return "concat(toString(" + start + "), 'T00:00:00+00')" +} diff --git a/module/services/accounts/code/pkg/auditstore/clickhousestore/clickhousestore.go b/module/services/accounts/code/pkg/auditstore/clickhousestore/clickhousestore.go new file mode 100644 index 000000000..8b7923ded --- /dev/null +++ b/module/services/accounts/code/pkg/auditstore/clickhousestore/clickhousestore.go @@ -0,0 +1,527 @@ +// Package clickhousestore is the ClickHouse adapter of the audit store swap +// (ADR 0009): both halves of business.AuditStore, and the history read-back +// (business.AuditHistoryReader). +// +// It keeps the same two tables as the BigQuery adapter, with the same rows: an +// events table holding every event's envelope, the SHA-256 of its canonical +// details, and the details themselves for security-class events; and a details +// table holding the details of content-class events. Every row carries the +// deployment id; a platform event has no organization. Where BigQuery keeps a +// NULL, ClickHouse keeps an empty string — no column here holds a value whose +// empty form means something else. +// +// The tables live in the database the DSN names, which the deployment creates; +// the store creates the tables when they are missing (MergeTree, or +// ReplicatedMergeTree on a cluster), partitioned by month, and expires each +// tier with a table TTL at its window. The service's ClickHouse user needs +// CREATE TABLE, INSERT and SELECT on that database and nothing more: no ALTER, +// DELETE, UPDATE, TRUNCATE or DROP, so it cannot rewrite the store of record. +// Its credentials ride in the DSN, which comes from the deployment's secret and +// is never logged. +// +// Inserts are batched: one insert per table per relay batch, never a row at a +// time (a single-row insert creates a data part per insert and ends in +// too-many-parts errors). Reads run in ClickHouse wherever ClickHouse computes +// exactly what the Postgres reads compute, and in the service (auditeval) +// where it does not; reader.go says which. +package clickhousestore + +import ( + "context" + "errors" + "fmt" + "regexp" + "sort" + "strconv" + "strings" + "time" + + "accounts/pkg/business" + + "github.com/ClickHouse/clickhouse-go/v2" + "github.com/ClickHouse/clickhouse-go/v2/lib/driver" +) + +// Table names. Both live in the DSN's database. +const ( + EventsTable = "audit_events" + DetailsTable = "audit_event_details" +) + +// Conn is what the store uses of a clickhouse-go native connection; +// clickhouse.Open's driver.Conn satisfies it. +type Conn interface { + Query(ctx context.Context, query string, args ...any) (driver.Rows, error) + Exec(ctx context.Context, query string, args ...any) error + PrepareBatch(ctx context.Context, query string, opts ...driver.PrepareBatchOption) (driver.Batch, error) +} + +// Config names where the store writes and what it reads. +type Config struct { + // Database is the database the connection uses, as its DSN names it. It + // must already exist: the store creates tables in it, never the database. + Database string + // Cluster, when set, creates both tables ON CLUSTER as ReplicatedMergeTree, + // coordinated by ClickHouse Keeper under the servers' default replica path + // and name. Empty creates them as MergeTree on the one server. + Cluster string + // DeploymentID confines every read to this deployment's events. + DeploymentID string + // EventsRetention is the events table's TTL: the compliance window, in + // whole days. + EventsRetention time.Duration + // ContentDetailRetention is the details table's TTL: how long a + // content-class event's full details are kept, in whole days. + ContentDetailRetention time.Duration +} + +// clusterPattern is a cluster name as remote_servers declares one, or a macro +// that names it ({cluster}). It is the one name a statement carries as text, +// since ON CLUSTER takes no parameter, so it is held to these characters. +var clusterPattern = regexp.MustCompile(`^(?:[A-Za-z0-9_][A-Za-z0-9_.-]{0,127}|\{[A-Za-z0-9_]{1,64}\})$`) + +// ValidCluster reports whether name can be a configured cluster. +func ValidCluster(name string) bool { return clusterPattern.MatchString(name) } + +// Store appends relay batches to the events and details tables and reads them. +type Store struct { + conn Conn + database string + cluster string + deploymentID string + eventsDays int + detailDays int + readSettings clickhouse.Settings + // inService, when set, is told each time an aggregation is evaluated in the + // service rather than in ClickHouse; tests use it to prove which ran. + inService func() +} + +// New builds a store over conn. +func New(conn Conn, cfg Config) (*Store, error) { + if conn == nil { + return nil, errors.New("clickhouse audit store: connection is required") + } + if strings.TrimSpace(cfg.Database) == "" { + return nil, errors.New("clickhouse audit store: database is required") + } + if strings.TrimSpace(cfg.DeploymentID) == "" { + return nil, errors.New("clickhouse audit store: deployment id is required") + } + if cfg.Cluster != "" && !ValidCluster(cfg.Cluster) { + return nil, fmt.Errorf("clickhouse audit store: cluster %q is not a valid cluster name", cfg.Cluster) + } + eventsDays, err := wholeDays("events retention", cfg.EventsRetention) + if err != nil { + return nil, err + } + detailDays, err := wholeDays("content detail retention", cfg.ContentDetailRetention) + if err != nil { + return nil, err + } + return &Store{ + conn: conn, + database: cfg.Database, + cluster: cfg.Cluster, + deploymentID: cfg.DeploymentID, + eventsDays: eventsDays, + detailDays: detailDays, + // Pinned per read, whatever the user's profile says, because each one + // changes an answer. + readSettings: clickhouse.Settings{ + // The numeric reading of a payload string parses to the double + // float8in returns only with ClickHouse's precise parser. + "precise_float_parsing": 1, + // The JSON parser that rounds a JSON number as float8in does; the + // other one does not. + "allow_simdjson": 1, + // An unmatched details row reads as '' — no details — not NULL. + "join_use_nulls": 0, + }, + }, nil +} + +func wholeDays(name string, window time.Duration) (int, error) { + const day = 24 * time.Hour + if window < day || window%day != 0 { + return 0, fmt.Errorf("clickhouse audit store: %s must be a whole number of days, at least one; got %s", name, window) + } + return int(window / day), nil +} + +var ( + _ business.AuditStore = (*Store)(nil) + _ business.AuditHistoryReader = (*Store)(nil) +) + +// Column is one column of a table the store creates. +type Column struct { + Name string + Type string + Comment string +} + +// EventsColumns is the events table: every event's envelope, the SHA-256 of +// its canonical details, and the details themselves for security-class events. +func EventsColumns() []Column { + return []Column{ + {Name: "event_id", Type: "String"}, + {Name: "deployment_id", Type: "LowCardinality(String)"}, + {Name: "org_id", Type: "String", Comment: "Empty for a platform event."}, + {Name: "actor_id", Type: "String", Comment: "Empty when the event names no actor."}, + {Name: "actor_type", Type: "LowCardinality(String)"}, + {Name: "event_type", Type: "LowCardinality(String)"}, + {Name: "schema_version", Type: "Int64"}, + {Name: "resource", Type: "LowCardinality(String)", Comment: "The target's resource type."}, + {Name: "resource_id", Type: "String", Comment: "The target's id; empty when none."}, + {Name: "occurred_at", Type: "DateTime64(6, 'UTC')"}, + {Name: "ip_address", Type: "String"}, + {Name: "impersonated_by", Type: "String"}, + {Name: "is_impersonated", Type: "Bool"}, + {Name: "client_id", Type: "String"}, + {Name: "retention_class", Type: "LowCardinality(String)"}, + {Name: "details_sha256", Type: "String", Comment: "Lowercase hex SHA-256 of the canonical details string."}, + {Name: "details", Type: "String", Comment: "Canonical details JSON of a security-class event; empty for a content-class one."}, + } +} + +// DetailsColumns is the details table: the full details of content-class +// events, kept for the content window. +func DetailsColumns() []Column { + return []Column{ + {Name: "event_id", Type: "String"}, + {Name: "deployment_id", Type: "LowCardinality(String)"}, + {Name: "org_id", Type: "String", Comment: "Empty for a platform event."}, + {Name: "event_type", Type: "LowCardinality(String)"}, + {Name: "occurred_at", Type: "DateTime64(6, 'UTC')"}, + {Name: "details_sha256", Type: "String"}, + {Name: "details", Type: "String", Comment: "Canonical details JSON."}, + } +} + +// tableSpec is what Ensure creates and what it requires of a table it finds. +type tableSpec struct { + name string + comment string + columns []Column + days int +} + +func (s *Store) tableSpecs() []tableSpec { + return []tableSpec{ + { + name: EventsTable, + comment: "Audit store of record: one row per audit event (ADR 0009).", + columns: EventsColumns(), + days: s.eventsDays, + }, + { + name: DetailsTable, + comment: "Full details of content-class audit events, kept for the content window (ADR 0009).", + columns: DetailsColumns(), + days: s.detailDays, + }, + } +} + +// CreateStatements are the statements Ensure runs for a missing table, in +// table order. +func (s *Store) CreateStatements() []string { + specs := s.tableSpecs() + out := make([]string, len(specs)) + for i, spec := range specs { + out[i] = s.createStatement(spec) + } + return out +} + +// createStatement is one table's CREATE TABLE: partitioned by month, sorted +// by deployment, organization and time — every read names the deployment and, +// but for the platform read, the organization, and the activity list walks +// time newest first — and expiring rows at the table's window. Every word of +// it is the store's own: column comments and the window are constants and +// configuration the store validated, and the cluster matched clusterPattern. +func (s *Store) createStatement(spec tableSpec) string { + var b strings.Builder + b.WriteString("CREATE TABLE IF NOT EXISTS " + spec.name) + engine := "MergeTree" + if s.cluster != "" { + b.WriteString(" ON CLUSTER '" + s.cluster + "'") + engine = "ReplicatedMergeTree" + } + b.WriteString("\n(\n") + for i, column := range spec.columns { + b.WriteString(" " + column.Name + " " + column.Type) + if column.Comment != "" { + b.WriteString(" COMMENT " + quoteLiteral(column.Comment)) + } + if i < len(spec.columns)-1 { + b.WriteString(",") + } + b.WriteString("\n") + } + b.WriteString(")\nENGINE = " + engine + "\n") + b.WriteString("PARTITION BY toYYYYMM(occurred_at)\n") + b.WriteString("ORDER BY (deployment_id, org_id, occurred_at, event_id)\n") + b.WriteString("TTL occurred_at + INTERVAL " + strconv.Itoa(spec.days) + " DAY\n") + b.WriteString("COMMENT " + quoteLiteral(spec.comment)) + return b.String() +} + +// quoteLiteral quotes one of the store's own constant strings. +func quoteLiteral(text string) string { + return "'" + strings.NewReplacer(`\`, `\\`, `'`, `\'`).Replace(text) + "'" +} + +// unknownDatabase is ClickHouse's UNKNOWN_DATABASE error code. +const unknownDatabase = 81 + +// Ensure verifies the database exists, creates each table that is missing, +// and refuses a table it finds that does not match: the service's user holds +// no ALTER, so a mismatch is the deployment's to fix, and a table expiring at +// a window other than the configured one would keep events or content longer +// or shorter than the deployment declared. +func (s *Store) Ensure(ctx context.Context) error { + var current string + var found uint64 + err := s.queryRow(ctx, "SELECT currentDatabase(), count() FROM system.databases WHERE name = currentDatabase()", nil, ¤t, &found) + if err != nil { + return s.ensureError("read the current database", err) + } + if found == 0 { + return s.ensureError("", nil) + } + if current != s.database { + return fmt.Errorf("clickhouse audit store: the connection uses database %q, the configured one is %q", current, s.database) + } + for _, spec := range s.tableSpecs() { + existing, err := s.describe(ctx, spec.name) + if err != nil { + return s.ensureError("read table "+spec.name, err) + } + if existing == nil { + if err := s.conn.Exec(ctx, s.createStatement(spec)); err != nil { + return s.ensureError("create table "+spec.name, err) + } + // Another replica may have created it first; read back what exists. + if existing, err = s.describe(ctx, spec.name); err != nil { + return s.ensureError("read table "+spec.name, err) + } + if existing == nil { + return fmt.Errorf("clickhouse audit store: table %s.%s was not created", s.database, spec.name) + } + } + if err := conforms(spec, existing); err != nil { + return fmt.Errorf("clickhouse audit store: table %s.%s: %w", s.database, spec.name, err) + } + } + return nil +} + +// ensureError describes a failure of Ensure: the database missing — whether +// the connection could not open it or found it gone — or what failed. +func (s *Store) ensureError(doing string, err error) error { + var exception *clickhouse.Exception + if err == nil || (errors.As(err, &exception) && exception.Code == unknownDatabase) { + return fmt.Errorf("clickhouse audit store: database %q does not exist; the deployment creates it", s.database) + } + return fmt.Errorf("clickhouse audit store: %s: %w", doing, err) +} + +// tableShape is what system.tables and system.columns say of a table. +type tableShape struct { + engineFull string + columns map[string]string +} + +func (s *Store) describe(ctx context.Context, table string) (*tableShape, error) { + p := &sqlParams{} + rows, err := s.conn.Query(ctx, "SELECT engine_full FROM system.tables WHERE database = currentDatabase() AND name = "+p.text(table), p.args...) + if err != nil { + return nil, err + } + var shape *tableShape + for rows.Next() { + shape = &tableShape{columns: map[string]string{}} + if err := rows.Scan(&shape.engineFull); err != nil { + _ = rows.Close() + return nil, err + } + } + if err := closeRows(rows); err != nil || shape == nil { + return nil, err + } + p = &sqlParams{} + rows, err = s.conn.Query(ctx, "SELECT name, type FROM system.columns WHERE database = currentDatabase() AND table = "+p.text(table), p.args...) + if err != nil { + return nil, err + } + for rows.Next() { + var name, typ string + if err := rows.Scan(&name, &typ); err != nil { + _ = rows.Close() + return nil, err + } + shape.columns[name] = typ + } + if err := closeRows(rows); err != nil { + return nil, err + } + return shape, nil +} + +// ttlClause is the TTL clause of an engine_full, up to its SETTINGS. +var ttlClause = regexp.MustCompile(` TTL (.+?)(?: SETTINGS |$)`) + +// ttlDays is the one TTL the store writes, as ClickHouse prints it back +// (occurred_at + toIntervalDay(N)) or as the statement spelled it. +var ttlDays = regexp.MustCompile(`^occurred_at \+ (?:toIntervalDay\((\d+)\)|INTERVAL (\d+) DAY)$`) + +// conforms reports how an existing table differs from what the store writes +// and reads: a column missing or of another type, or rows that expire at +// another window or by another rule. +func conforms(spec tableSpec, existing *tableShape) error { + for _, want := range spec.columns { + got, ok := existing.columns[want.Name] + if !ok { + return fmt.Errorf("has no column %s", want.Name) + } + if got != want.Type { + return fmt.Errorf("column %s is %s, want %s", want.Name, got, want.Type) + } + } + want := fmt.Sprintf("occurred_at + INTERVAL %d DAY", spec.days) + clause := ttlClause.FindStringSubmatch(existing.engineFull) + if clause == nil { + return fmt.Errorf("has no TTL; the configured window is %s", want) + } + days := ttlDays.FindStringSubmatch(strings.TrimSpace(clause[1])) + if days == nil { + return fmt.Errorf("expires rows by %q, the configured window is %s", clause[1], want) + } + got := days[1] + days[2] + if got != strconv.Itoa(spec.days) { + return fmt.Errorf("rows expire after %s days, the configured window is %d days", got, spec.days) + } + return nil +} + +// insertStatement is the INSERT a batch of rows of a table is sent with. +func insertStatement(table string, columns []Column) string { + names := make([]string, len(columns)) + for i, column := range columns { + names[i] = column.Name + } + return "INSERT INTO " + table + " (" + strings.Join(names, ", ") + ")" +} + +// BatchRows is the rows a batch appends, in each table's column order: one +// events-table row per record, and a details-table row for each content-class +// record, whose details go there instead of its events row — the rows +// bigquerystore.BatchRows writes, with an empty string where BigQuery keeps a +// NULL. +func BatchRows(batch business.AuditBatch) (events, details [][]any) { + for _, record := range batch.Records { + events = append(events, EventRow(batch.DeploymentID, record)) + if record.Retention != business.RetentionSecurity { + details = append(details, DetailRow(batch.DeploymentID, record)) + } + } + return events, details +} + +// EventRow is a record's events-table row, in EventsColumns order. +func EventRow(deploymentID string, record business.AuditRecord) []any { + entry := record.Entry + details := "" + if record.Retention == business.RetentionSecurity { + details = record.Details + } + return []any{ + entry.ID, deploymentID, entry.OrgID, entry.ActorID, entry.ActorType, string(entry.EventType), + int64(entry.SchemaVersion), entry.Resource, entry.ResourceID, entry.CreatedAt.UTC(), entry.IPAddress, + entry.ImpersonatedBy, entry.IsImpersonated, entry.ClientID, string(record.Retention), record.DetailsSHA256, details, + } +} + +// DetailRow is a content-class record's details-table row, in DetailsColumns +// order. +func DetailRow(deploymentID string, record business.AuditRecord) []any { + entry := record.Entry + return []any{entry.ID, deploymentID, entry.OrgID, string(entry.EventType), entry.CreatedAt.UTC(), record.DetailsSHA256, record.Details} +} + +// AppendAuditBatch implements business.AuditStoreWriter: the batch's rows +// (BatchRows) as one insert into the events table and, when the batch holds +// content-class events, one into the details table. Each insert is one block, +// written atomically; a failure between the two leaves the events without +// their details until the relay delivers the batch again, and every read +// returns each event once by event id. +func (s *Store) AppendAuditBatch(ctx context.Context, batch business.AuditBatch) error { + events, details := BatchRows(batch) + if err := s.insert(ctx, EventsTable, EventsColumns(), events); err != nil { + return err + } + return s.insert(ctx, DetailsTable, DetailsColumns(), details) +} + +func (s *Store) insert(ctx context.Context, table string, columns []Column, rows [][]any) error { + if len(rows) == 0 { + return nil + } + batch, err := s.conn.PrepareBatch(ctx, insertStatement(table, columns)) + if err != nil { + return fmt.Errorf("clickhouse audit store: append to %s: %w", table, err) + } + defer func() { + if !batch.IsSent() { + _ = batch.Abort() + } + }() + for _, row := range rows { + if err := batch.Append(row...); err != nil { + return fmt.Errorf("clickhouse audit store: append to %s: event %v: %w", table, row[0], err) + } + } + if err := batch.Send(); err != nil { + return fmt.Errorf("clickhouse audit store: append to %s: %w", table, err) + } + return nil +} + +// queryRow runs a statement expected to return at most one row and scans it. +func (s *Store) queryRow(ctx context.Context, statement string, args []any, dest ...any) error { + rows, err := s.conn.Query(ctx, statement, args...) + if err != nil { + return err + } + found := false + for rows.Next() { + if found { + continue + } + found = true + if err := rows.Scan(dest...); err != nil { + _ = rows.Close() + return err + } + } + if err := closeRows(rows); err != nil { + return err + } + if !found { + return errors.New("no row") + } + return nil +} + +// closeRows closes rows and reports the first error the iteration met. +func closeRows(rows driver.Rows) error { + err := rows.Err() + if closeErr := rows.Close(); err == nil { + err = closeErr + } + return err +} + +func sortStrings(values []string) { sort.Strings(values) } diff --git a/module/services/accounts/code/pkg/auditstore/clickhousestore/clickhousestore_test.go b/module/services/accounts/code/pkg/auditstore/clickhousestore/clickhousestore_test.go new file mode 100644 index 000000000..29c77b442 --- /dev/null +++ b/module/services/accounts/code/pkg/auditstore/clickhousestore/clickhousestore_test.go @@ -0,0 +1,348 @@ +package clickhousestore + +import ( + "context" + "encoding/json" + "errors" + "strings" + "testing" + "time" + + "accounts/pkg/auditstore/auditeval" + "accounts/pkg/business" + + "github.com/ClickHouse/clickhouse-go/v2/lib/driver" + "github.com/stretchr/testify/require" +) + +const day = 24 * time.Hour + +func validConfig() Config { + return Config{Database: "audit", DeploymentID: "deployment-1", EventsRetention: 2555 * day, ContentDetailRetention: 30 * day} +} + +// panicConn fails a test that reaches the server: every method of the +// embedded nil interface panics. +type panicConn struct{ Conn } + +func TestCreateStatementsOnOneServer(t *testing.T) { + store, err := New(panicConn{}, validConfig()) + require.NoError(t, err) + statements := store.CreateStatements() + require.Len(t, statements, 2) + require.Equal(t, `CREATE TABLE IF NOT EXISTS audit_events +( + event_id String, + deployment_id LowCardinality(String), + org_id String COMMENT 'Empty for a platform event.', + actor_id String COMMENT 'Empty when the event names no actor.', + actor_type LowCardinality(String), + event_type LowCardinality(String), + schema_version Int64, + resource LowCardinality(String) COMMENT 'The target\'s resource type.', + resource_id String COMMENT 'The target\'s id; empty when none.', + occurred_at DateTime64(6, 'UTC'), + ip_address String, + impersonated_by String, + is_impersonated Bool, + client_id String, + retention_class LowCardinality(String), + details_sha256 String COMMENT 'Lowercase hex SHA-256 of the canonical details string.', + details String COMMENT 'Canonical details JSON of a security-class event; empty for a content-class one.' +) +ENGINE = MergeTree +PARTITION BY toYYYYMM(occurred_at) +ORDER BY (deployment_id, org_id, occurred_at, event_id) +TTL occurred_at + INTERVAL 2555 DAY +COMMENT 'Audit store of record: one row per audit event (ADR 0009).'`, statements[0]) + require.Equal(t, `CREATE TABLE IF NOT EXISTS audit_event_details +( + event_id String, + deployment_id LowCardinality(String), + org_id String COMMENT 'Empty for a platform event.', + event_type LowCardinality(String), + occurred_at DateTime64(6, 'UTC'), + details_sha256 String, + details String COMMENT 'Canonical details JSON.' +) +ENGINE = MergeTree +PARTITION BY toYYYYMM(occurred_at) +ORDER BY (deployment_id, org_id, occurred_at, event_id) +TTL occurred_at + INTERVAL 30 DAY +COMMENT 'Full details of content-class audit events, kept for the content window (ADR 0009).'`, statements[1]) +} + +func TestCreateStatementsOnACluster(t *testing.T) { + for _, cluster := range []string{"audit_cluster", "{cluster}"} { + cfg := validConfig() + cfg.Cluster = cluster + store, err := New(panicConn{}, cfg) + require.NoError(t, err) + for _, statement := range store.CreateStatements() { + require.Contains(t, statement, " ON CLUSTER '"+cluster+"'\n(") + require.Contains(t, statement, "\nENGINE = ReplicatedMergeTree\n", + "replicated under the servers' default replica path and name") + } + } +} + +func TestNewRefusesAnIncompleteConfiguration(t *testing.T) { + _, err := New(nil, validConfig()) + require.ErrorContains(t, err, "connection is required") + for name, tc := range map[string]struct { + mutate func(*Config) + want string + }{ + "database": {func(c *Config) { c.Database = " " }, "database is required"}, + "deployment": {func(c *Config) { c.DeploymentID = "" }, "deployment id is required"}, + "events window": {func(c *Config) { c.EventsRetention = 0 }, "events retention must be a whole number of days"}, + "partial day": {func(c *Config) { c.EventsRetention = 30*day + time.Hour }, "events retention must be a whole number of days"}, + "content window": {func(c *Config) { c.ContentDetailRetention = time.Hour }, "content detail retention must be a whole number of days"}, + "quoted cluster": {func(c *Config) { c.Cluster = "a' ON CLUSTER 'b" }, "not a valid cluster name"}, + "macro in a name": {func(c *Config) { c.Cluster = "x{cluster}" }, "not a valid cluster name"}, + "cluster too long": {func(c *Config) { c.Cluster = strings.Repeat("c", 129) }, "not a valid cluster name"}, + } { + cfg := validConfig() + tc.mutate(&cfg) + _, err := New(panicConn{}, cfg) + require.ErrorContains(t, err, tc.want, name) + } +} + +func TestConformsStatesTheMismatch(t *testing.T) { + store, err := New(panicConn{}, validConfig()) + require.NoError(t, err) + details := store.tableSpecs()[1] + shape := func() *tableShape { + columns := map[string]string{} + for _, column := range details.columns { + columns[column.Name] = column.Type + } + return &tableShape{ + engineFull: "MergeTree PARTITION BY toYYYYMM(occurred_at) ORDER BY (deployment_id, org_id, occurred_at, event_id) " + + "TTL occurred_at + toIntervalDay(30) SETTINGS index_granularity = 8192", + columns: columns, + } + } + require.NoError(t, conforms(details, shape())) + replicated := shape() + replicated.engineFull = "ReplicatedMergeTree('/clickhouse/tables/{uuid}/{shard}', '{replica}') PARTITION BY toYYYYMM(occurred_at) " + + "ORDER BY (deployment_id, org_id, occurred_at, event_id) TTL occurred_at + toIntervalDay(30)" + require.NoError(t, conforms(details, replicated), "a TTL clause at the end, with no SETTINGS after it") + + missing := shape() + delete(missing.columns, "details_sha256") + require.EqualError(t, conforms(details, missing), "has no column details_sha256") + retyped := shape() + retyped.columns["occurred_at"] = "DateTime64(3, 'UTC')" + require.EqualError(t, conforms(details, retyped), "column occurred_at is DateTime64(3, 'UTC'), want DateTime64(6, 'UTC')") + longer := shape() + longer.engineFull = strings.Replace(longer.engineFull, "toIntervalDay(30)", "toIntervalDay(400)", 1) + require.EqualError(t, conforms(details, longer), "rows expire after 400 days, the configured window is 30 days") + none := shape() + none.engineFull = "MergeTree ORDER BY (deployment_id, org_id, occurred_at, event_id) SETTINGS index_granularity = 8192" + require.EqualError(t, conforms(details, none), "has no TTL; the configured window is occurred_at + INTERVAL 30 DAY") + other := shape() + other.engineFull = strings.Replace(other.engineFull, "TTL occurred_at + toIntervalDay(30)", + "TTL occurred_at + toIntervalDay(30) DELETE WHERE org_id = ''", 1) + require.ErrorContains(t, conforms(details, other), "expires rows by") +} + +func TestReadsRefuseAnUnscopedReadBeforeReachingTheServer(t *testing.T) { + store, err := New(panicConn{}, validConfig()) + require.NoError(t, err) + ctx := context.Background() + org := "22222222-2222-2222-2222-222222222222" + + _, _, err = store.ListAuditEvents(ctx, business.AuditRead{Query: business.AuditQuery{OrgID: org}}) + require.ErrorIs(t, err, business.ErrAuditReadUnscoped) + _, err = store.AggregateAuditEvents(ctx, business.AuditRead{}, business.AuditAggregationSpec{}) + require.ErrorIs(t, err, business.ErrAuditReadUnscoped) + _, err = store.ExportAuditEvents(ctx, business.AuditRead{}) + require.ErrorIs(t, err, business.ErrAuditReadUnscoped) + _, err = store.LatestSourceSyncEvents(ctx, business.AuditReadScope{}, []string{"source-1"}) + require.ErrorIs(t, err, business.ErrAuditReadUnscoped) + + _, _, err = store.ListAuditEvents(ctx, business.AuditRead{ + Scope: business.OrganizationAuditScope(org), Query: business.AuditQuery{OrgID: "33333333-3333-3333-3333-333333333333"}, + }) + require.ErrorContains(t, err, "outside the organization", "a query naming another organization than its scope") + _, _, err = store.ListAuditEvents(ctx, business.AuditRead{Scope: business.PlatformAuditScope(), Query: business.AuditQuery{OrgID: org}}) + require.ErrorContains(t, err, "outside the platform scope") + _, err = store.LatestSourceSyncEvents(ctx, business.PlatformAuditScope(), []string{"source-1"}) + require.ErrorContains(t, err, "reads one organization") + _, err = store.AggregateAuditEvents(ctx, business.AuditRead{Scope: business.OrganizationAuditScope(org), Query: business.AuditQuery{OrgID: org}}, + business.AuditAggregationSpec{GroupBy: []string{"category"}}) + require.ErrorContains(t, err, "needs the event-type index", "a category without the index is refused, as auditeval refuses it") + + // A read that can match no event type runs no statement. + none := business.AuditQuery{OrgID: org, EventType: "saas.auth.login", EventTypes: []string{"saas.org.updated"}} + entries, next, err := store.ListAuditEvents(ctx, business.AuditRead{Scope: business.OrganizationAuditScope(org), Query: none}) + require.NoError(t, err) + require.Nil(t, entries) + require.Empty(t, next) +} + +// recordingConn records the batches the store prepares. +type recordingConn struct { + Conn + batches []*recordingBatch + fail error +} + +func (c *recordingConn) PrepareBatch(_ context.Context, query string, _ ...driver.PrepareBatchOption) (driver.Batch, error) { + batch := &recordingBatch{query: query, fail: c.fail} + c.batches = append(c.batches, batch) + return batch, nil +} + +type recordingBatch struct { + driver.Batch + query string + rows [][]any + sent int + aborted bool + fail error +} + +func (b *recordingBatch) Append(values ...any) error { + if b.fail != nil { + return b.fail + } + b.rows = append(b.rows, values) + return nil +} + +func (b *recordingBatch) Send() error { b.sent++; return nil } +func (b *recordingBatch) Abort() error { b.aborted = true; return nil } +func (b *recordingBatch) IsSent() bool { return b.sent > 0 || b.aborted } + +var occurredAt = time.Date(2026, 10, 2, 14, 30, 5, 123456000, time.UTC) + +func record(t *testing.T, class business.AuditRetentionClass, org string) business.AuditRecord { + t.Helper() + r, err := business.NewAuditRecord(business.AuditEntry{ + ID: business.NewIDString(), + OrgID: org, + ActorID: "11111111-1111-1111-1111-111111111111", + ActorType: business.ActorTypeUser, + EventType: business.EventAuthLogin, + SchemaVersion: 1, + Resource: "session", + ResourceID: "session-1", + Payload: map[string]any{"method": "password"}, + CreatedAt: occurredAt, + }, class) + require.NoError(t, err) + return r +} + +func TestAppendAuditBatchIsOneInsertPerTable(t *testing.T) { + conn := &recordingConn{} + store, err := New(conn, validConfig()) + require.NoError(t, err) + security := record(t, business.RetentionSecurity, "22222222-2222-2222-2222-222222222222") + content := record(t, business.RetentionContent, "") + other := record(t, business.RetentionContent, "22222222-2222-2222-2222-222222222222") + batch := business.AuditBatch{ID: "batch-1", DeploymentID: "deployment-1", Records: []business.AuditRecord{security, content, other}} + require.NoError(t, store.AppendAuditBatch(context.Background(), batch)) + + require.Len(t, conn.batches, 2, "one insert per table for the whole relay batch, never one per row") + events, details := conn.batches[0], conn.batches[1] + require.Equal(t, "INSERT INTO audit_events (event_id, deployment_id, org_id, actor_id, actor_type, event_type, schema_version, "+ + "resource, resource_id, occurred_at, ip_address, impersonated_by, is_impersonated, client_id, retention_class, details_sha256, details)", events.query) + require.Equal(t, "INSERT INTO audit_event_details (event_id, deployment_id, org_id, event_type, occurred_at, details_sha256, details)", details.query) + require.Equal(t, 1, events.sent) + require.Equal(t, 1, details.sent) + require.Len(t, events.rows, 3, "every record has an events row") + require.Len(t, details.rows, 2, "only content-class records have a details row") + + require.Equal(t, []any{ + security.Entry.ID, "deployment-1", "22222222-2222-2222-2222-222222222222", "11111111-1111-1111-1111-111111111111", "user", + "saas.auth.login", int64(1), "session", "session-1", occurredAt, "", "", false, "", "security", security.DetailsSHA256, + `{"method":"password"}`, + }, events.rows[0], "a security-class event keeps its details in the events row") + require.Equal(t, "", events.rows[1][len(events.rows[1])-1], "a content-class events row is content-free") + require.Equal(t, "", events.rows[1][2], "a platform event has no organization") + require.Equal(t, []any{content.Entry.ID, "deployment-1", "", "saas.auth.login", occurredAt, content.DetailsSHA256, `{"method":"password"}`}, details.rows[0]) + + conn.batches = nil + require.NoError(t, store.AppendAuditBatch(context.Background(), business.AuditBatch{ID: "batch-2", DeploymentID: "deployment-1", + Records: []business.AuditRecord{security}})) + require.Len(t, conn.batches, 1, "a batch without content-class events inserts no details") + + conn.batches, conn.fail = nil, errors.New("block too large") + err = store.AppendAuditBatch(context.Background(), batch) + require.ErrorContains(t, err, "block too large") + require.True(t, conn.batches[0].aborted, "a batch that failed to compose is aborted, never sent partially") + require.Zero(t, conn.batches[0].sent) +} + +// No value a read carries is ever part of a statement's text: every one is a +// server-side parameter. +func TestStatementsCarryNoValues(t *testing.T) { + store, err := New(panicConn{}, validConfig()) + require.NoError(t, err) + org := "22222222-2222-2222-2222-222222222222" + from := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC) + values := []string{"value-actor", "value-resource", "value-id", "value-client", "value-key", "value-text", "value-type", "value-category"} + read := business.AuditRead{ + Scope: business.OrganizationAuditScope(org), + Query: business.AuditQuery{ + OrgID: org, ActorID: "44444444-4444-4444-4444-444444444444", Resource: "value-resource", ResourceID: "value-id", + ClientID: "value-client", EventTypes: []string{"value-type", "saas.auth.login"}, From: &from, + PayloadContains: map[string]any{"value-key": "value-text", "flag": true}, + }, + Types: business.AuditEventTypeIndex{"value-type": {Category: "value-category"}}, + } + m, err := auditeval.NewMatcher(read) + require.NoError(t, err) + pl := newPlan(m, read.Query) + require.NotNil(t, pl.payload, "a filter of strings and booleans is pushed down") + + p := &sqlParams{} + joined, ok := store.joinedEvents(p, pl, true) + require.True(t, ok) + agg, ok, err := store.buildAggregation(pl, business.AuditAggregationSpec{ + GroupBy: []string{"category", "payload:value-key"}, + Metrics: []business.AuditMetric{{Op: "percentile", Field: "payload:value-key", Percentile: 0.5}, {Op: "count_distinct", Field: "category"}}, + }, read.Types) + require.NoError(t, err) + require.True(t, ok) + for _, statement := range []string{joined, agg.statement} { + for _, value := range append(values, org, "deployment-1", "44444444", "2026") { + require.NotContains(t, statement, value) + } + require.Contains(t, statement, "{p0:String}") + } +} + +func TestParamEscaperKeepsEveryByte(t *testing.T) { + require.Equal(t, `a\\b\tc\nd\re\0f'g}{h`, paramEscaper.Replace("a\\b\tc\nd\re\x00f'g}{h")) +} + +func TestPushablePayload(t *testing.T) { + for filter, want := range map[string]bool{ + `{"boundary":"b-1"}`: true, + `{"flag":true,"note":null}`: true, + `{"count":3}`: false, + `{"nested":{"a":"q"}}`: false, + `{"tags":["x"]}`: false, + `{"":"empty key"}`: false, + `{"boundary":"b-1","count":1}`: false, + } { + m, err := auditeval.NewMatcher(business.AuditRead{ + Scope: business.PlatformAuditScope(), + Query: business.AuditQuery{PayloadContains: decode(t, filter)}, + }) + require.NoError(t, err) + require.Equal(t, want, newPlan(m, business.AuditQuery{}).payload != nil, filter) + } +} + +func decode(t *testing.T, text string) map[string]any { + t.Helper() + var out map[string]any + require.NoError(t, json.Unmarshal([]byte(text), &out)) + return out +} diff --git a/module/services/accounts/code/pkg/auditstore/clickhousestore/query.go b/module/services/accounts/code/pkg/auditstore/clickhousestore/query.go new file mode 100644 index 000000000..2b76a47b3 --- /dev/null +++ b/module/services/accounts/code/pkg/auditstore/clickhousestore/query.go @@ -0,0 +1,271 @@ +package clickhousestore + +import ( + "strconv" + "strings" + "time" + + "accounts/pkg/auditstore/auditeval" + "accounts/pkg/business" + + "github.com/ClickHouse/clickhouse-go/v2" +) + +// The statements the read half builds. Every value a read carries — the +// deployment, the organization, ids, event types, payload keys and values, +// time bounds, limits — is a server-side query parameter ({name:Type}), sent +// beside the statement and parsed by the server as a value of that type, so no +// value is ever part of the SQL text. Only the statement's shape varies with +// the read: which predicates, dimensions and metrics it has. + +// sqlParams collects one statement's query parameters. +type sqlParams struct { + args []any +} + +// paramEscaper escapes a String parameter for the server, which reads a +// parameter value in its escaped text format: a backslash, tab, newline, +// carriage return or NUL in the value would otherwise be read as an escape +// sequence or the end of the value. clickhouse-go sends a top-level String +// parameter as given, so the store escapes it; it quotes and escapes the +// elements of an Array(String) parameter itself. +var paramEscaper = strings.NewReplacer(`\`, `\\`, "\t", `\t`, "\n", `\n`, "\r", `\r`, "\x00", `\0`) + +func (p *sqlParams) name() string { + return "p" + strconv.Itoa(len(p.args)) +} + +// text binds a String. +func (p *sqlParams) text(value string) string { + name := p.name() + p.args = append(p.args, clickhouse.Named(name, paramEscaper.Replace(value))) + return "{" + name + ":String}" +} + +// texts binds an Array(String). +func (p *sqlParams) texts(values []string) string { + name := p.name() + if values == nil { + values = []string{} + } + p.args = append(p.args, clickhouse.Named(name, values)) + return "{" + name + ":Array(String)}" +} + +// instant binds a time at the microsecond precision occurred_at keeps; a finer +// part is truncated, as Postgres truncates a bound parameter. +func (p *sqlParams) instant(at time.Time) string { + name := p.name() + p.args = append(p.args, clickhouse.DateNamed(name, at, clickhouse.MicroSeconds)) + return "{" + name + ":DateTime64(6, 'UTC')}" +} + +// count binds a non-negative integer. +func (p *sqlParams) count(n int) string { + name := p.name() + p.args = append(p.args, clickhouse.Named(name, strconv.Itoa(n))) + return "{" + name + ":UInt64}" +} + +// where is a conjunction under construction. +type where []string + +func (w *where) and(condition string) { *w = append(*w, condition) } + +func (w where) String() string { + if len(w) == 0 { + return "1" + } + return strings.Join(w, " AND ") +} + +// scope confines a statement over either table to this deployment always, and +// to the read's organization unless the read is the explicit platform read. +func (s *Store) scope(p *sqlParams, scope business.AuditReadScope) where { + w := where{"deployment_id = " + p.text(s.deploymentID)} + if !scope.Platform() { + w.and("org_id = " + p.text(scope.OrgID())) + } + return w +} + +// window adds the read's time bounds, both inclusive. +func (w *where) window(p *sqlParams, from, to *time.Time) { + if from != nil { + w.and("occurred_at >= " + p.instant(*from)) + } + if to != nil { + w.and("occurred_at <= " + p.instant(*to)) + } +} + +// eventTypes adds the event types the read can match. It reports false when +// the read can match none, so the statement need not run. +func (w *where) eventTypes(p *sqlParams, m *auditeval.Matcher) bool { + types, restricted := m.EventTypes() + switch { + case !restricted: + case len(types) == 0: + return false + case len(types) == 1: + w.and("event_type = " + p.text(types[0])) + default: + w.and("has(" + p.texts(types) + ", event_type)") + } + return true +} + +// plan is what a read pushes into SQL. +type plan struct { + m *auditeval.Matcher + q business.AuditQuery + // payload is the payload filter's SQL form over a details column, when + // every pair of it has one (pushablePayload); nil otherwise. + payload map[string]any + // none is set when the read can match no event type at all. + none bool +} + +func newPlan(m *auditeval.Matcher, q business.AuditQuery) plan { + pl := plan{m: m, q: q} + if filter, ok := m.PayloadFilter().(map[string]any); ok && pushablePayload(filter) { + pl.payload = filter + } + if types, restricted := m.EventTypes(); restricted && len(types) == 0 { + pl.none = true + } + return pl +} + +// pushablePayload reports whether a payload filter has an exact SQL form: +// every key non-empty and every value a string, a boolean or null at the top +// level. jsonb containment of such a filter is "the payload has each key with +// exactly that value", which ClickHouse's JSON functions decide the same way. +// A number (jsonb compares numerically, at any precision) or a nested object or +// array (containment recurses) is evaluated in the service instead. +func pushablePayload(filter map[string]any) bool { + for key, value := range filter { + if key == "" { + return false + } + switch value.(type) { + case string, bool, nil: + default: + return false + } + } + return len(filter) > 0 +} + +// unparsed is true for details ClickHouse cannot parse as JSON — a document +// valid to Postgres that the server's parser refuses, such as an integer +// beyond 64 bits. Every JSON function reads such a document as empty, so a +// statement that pushes payload predicates down treats it as unknown: a +// candidate the service evaluates, never a silent miss. +func unparsed(column string) string { + return "(" + column + " != '' AND NOT isValidJSON(" + column + "))" +} + +// payloadMatches is the pushed payload filter over a details column: true when +// the details hold every pair exactly. +func payloadMatches(p *sqlParams, column string, filter map[string]any) string { + keys := make([]string, 0, len(filter)) + for key := range filter { + keys = append(keys, key) + } + sortStrings(keys) + var terms []string + for _, key := range keys { + k := p.text(key) + switch value := filter[key].(type) { + case string: + terms = append(terms, "(JSONType("+column+", "+k+") = 'String' AND JSONExtractString("+column+", "+k+") = "+p.text(value)+")") + case bool: + want := "0" + if value { + want = "1" + } + terms = append(terms, "(JSONType("+column+", "+k+") = 'Bool' AND JSONExtractBool("+column+", "+k+") = "+want+")") + case nil: + terms = append(terms, "(JSONType("+column+", "+k+") = 'Null' AND JSONHas("+column+", "+k+"))") + } + } + return "(" + strings.Join(terms, " AND ") + ")" +} + +// payloadCandidates is a details column that matches the pushed filter, or +// that ClickHouse cannot parse and so must be decided by the service. +func payloadCandidates(p *sqlParams, column string, filter map[string]any) string { + return "(" + unparsed(column) + " OR " + payloadMatches(p, column, filter) + ")" +} + +// eventsWhere is the condition on the events table for a read: scope, window, +// event types, the envelope filters and — when pushed — the payload filter: +// a security-class event's own details, or a content-class event whose details +// row (by id and hash) satisfies it. ok is false when the read can match +// nothing. +func (s *Store) eventsWhere(p *sqlParams, pl plan) (where, bool) { + if pl.none { + return nil, false + } + w := s.scope(p, pl.m.Scope()) + w.window(p, pl.m.From(), pl.m.To()) + w.eventTypes(p, pl.m) + if actor := pl.m.ActorID(); actor != "" { + w.and("actor_id = " + p.text(actor)) + } + if pl.q.Resource != "" { + w.and("resource = " + p.text(pl.q.Resource)) + } + if pl.q.ResourceID != "" { + w.and("resource_id = " + p.text(pl.q.ResourceID)) + } + if pl.q.ClientID != "" { + w.and("client_id = " + p.text(pl.q.ClientID)) + } + if pl.payload != nil { + details := s.detailsWhere(p, pl) + details.and(payloadCandidates(p, "details", pl.payload)) + w.and("((retention_class = " + p.text(string(business.RetentionSecurity)) + " AND " + payloadCandidates(p, "details", pl.payload) + ")" + + " OR (retention_class != " + p.text(string(business.RetentionSecurity)) + " AND (event_id, details_sha256) IN (" + + "SELECT event_id, details_sha256 FROM " + DetailsTable + " WHERE " + details.String() + ")))") + } + return w, true +} + +// detailsWhere is the condition on the details table that bounds the details +// a read can join: scope, window and event types. +func (s *Store) detailsWhere(p *sqlParams, pl plan) where { + w := s.scope(p, pl.m.Scope()) + w.window(p, pl.m.From(), pl.m.To()) + w.eventTypes(p, pl.m) + return w +} + +// eventColumns lists the events-table columns every row-returning read +// selects, in the order scanStored reads them. +const eventColumns = "deployment_id, event_id, org_id, actor_id, actor_type, event_type, schema_version, resource, resource_id, " + + "occurred_at, ip_address, impersonated_by, is_impersonated, client_id, retention_class, details_sha256, details" + +// joinedEvents is the statement shape that returns each matching event once, +// with its details: a security-class event's from its own row, a content-class +// event's from the details table, joined by id and hash. A content-class event +// past the content window has none. +func (s *Store) joinedEvents(p *sqlParams, pl plan, ordered bool) (string, bool) { + events, ok := s.eventsWhere(p, pl) + if !ok { + return "", false + } + details := s.detailsWhere(p, pl) + security := p.text(string(business.RetentionSecurity)) + statement := "SELECT e.deployment_id, e.event_id, e.org_id, e.actor_id, e.actor_type, e.event_type, e.schema_version, e.resource, " + + "e.resource_id, e.occurred_at, e.ip_address, e.impersonated_by, e.is_impersonated, e.client_id, e.retention_class, e.details_sha256, " + + "if(e.retention_class = " + security + ", e.details, d.details) AS payload " + + "FROM (SELECT " + eventColumns + " FROM " + EventsTable + " WHERE " + events.String() + " LIMIT 1 BY event_id) AS e " + + "LEFT JOIN (SELECT event_id, details_sha256, any(details) AS details FROM " + DetailsTable + " WHERE " + details.String() + + " GROUP BY event_id, details_sha256) AS d ON e.event_id = d.event_id AND e.details_sha256 = d.details_sha256" + if ordered { + statement += " ORDER BY e.occurred_at DESC, e.event_id DESC" + } + return statement, true +} diff --git a/module/services/accounts/code/pkg/auditstore/clickhousestore/reader.go b/module/services/accounts/code/pkg/auditstore/clickhousestore/reader.go new file mode 100644 index 000000000..d05619786 --- /dev/null +++ b/module/services/accounts/code/pkg/auditstore/clickhousestore/reader.go @@ -0,0 +1,519 @@ +package clickhousestore + +import ( + "context" + "errors" + "fmt" + "time" + + "accounts/pkg/auditstore/auditeval" + "accounts/pkg/business" + + "github.com/ClickHouse/clickhouse-go/v2" + "github.com/ClickHouse/clickhouse-go/v2/lib/driver" +) + +// The read half of the ClickHouse store (ADR 0009). +// +// Postgres row-level security does not reach ClickHouse, so every statement +// carries the read's scope itself: this deployment always, and the caller's +// organization unless the read is the explicit platform read. A read without a +// scope, or whose query names another organization than its scope, is refused +// before any statement runs (business.AuditRead.Validate, through +// auditeval.NewMatcher). +// +// ClickHouse does the work it reproduces exactly: the scope, the time window, +// the envelope filters (actor, event type and the types a category or +// namespace names, resource, client), deduplication by event id (an event +// delivered twice is stored twice), the activity list's order and keyset +// paging, the aggregation (aggregate.go), the readable-source query and the +// join of content details by id and hash. A payload filter whose pairs are all +// top-level strings, booleans or nulls is pushed down too. The service +// evaluates what ClickHouse cannot reproduce: containment of a number or a +// nested value (jsonb compares numbers at any precision and recurses into +// containers), the ->> text of a fractional number or a nested value, and +// details ClickHouse's parser refuses. Every row a read returns is still +// matched by the service (auditeval.Matcher), which is the authority on every +// predicate: a statement may return a candidate the service then drops, never +// drop one the service would keep. + +// errUnscopedRow is a row outside the read's scope or deployment, which no +// statement here can return; it fails the read rather than being dropped. +var errUnscopedRow = errors.New("clickhouse audit store: a statement returned an event outside the read's scope") + +// readContext carries the settings every read runs with. +func (s *Store) readContext(ctx context.Context) context.Context { + return clickhouse.Context(ctx, clickhouse.WithSettings(s.readSettings)) +} + +// storedRow is one events-table row as a read selects it (eventColumns). +type storedRow struct { + deploymentID string + entry business.AuditEntry + retentionClass string + detailsSHA256 string + details string +} + +// scanStored reads a row selected as eventColumns, with the details column +// last: the events row's own, or the joined payload. +func scanStored(rows driver.Rows, extra ...any) (storedRow, error) { + var row storedRow + var eventType string + var schemaVersion int64 + var occurredAt time.Time + entry := &row.entry + dest := []any{ + &row.deploymentID, &entry.ID, &entry.OrgID, &entry.ActorID, &entry.ActorType, &eventType, &schemaVersion, + &entry.Resource, &entry.ResourceID, &occurredAt, &entry.IPAddress, &entry.ImpersonatedBy, &entry.IsImpersonated, + &entry.ClientID, &row.retentionClass, &row.detailsSHA256, &row.details, + } + if err := rows.Scan(append(dest, extra...)...); err != nil { + return storedRow{}, err + } + entry.EventType = business.EventType(eventType) + entry.SchemaVersion = int(schemaVersion) + // The process's local zone at microsecond precision: what a Postgres read + // hands back. + entry.CreatedAt = time.UnixMicro(occurredAt.UnixMicro()) + return row, nil +} + +func (r storedRow) event() *auditeval.Event { + return &auditeval.Event{Entry: r.entry, Details: r.details, HasDetails: r.details != "", DetailsSHA256: r.detailsSHA256} +} + +// inScope reports whether a row is one the read may see. +func (s *Store) inScope(m *auditeval.Matcher, row storedRow) bool { + return row.deploymentID == s.deploymentID && m.InScope(row.entry) +} + +// scan runs a row-returning statement and hands each row to visit, which +// returns false to stop early. +func (s *Store) scan(ctx context.Context, statement string, args []any, visit func(driver.Rows) (bool, error)) error { + // Closing rows reads them to the end; cancelling the statement's context is + // what stops the server, so a read that has its answer cancels first. + ctx, cancel := context.WithCancel(ctx) + defer cancel() + rows, err := s.conn.Query(s.readContext(ctx), statement, args...) + if err != nil { + return fmt.Errorf("clickhouse audit store: read: %w", err) + } + for rows.Next() { + more, err := visit(rows) + if err != nil || !more { + cancel() + _ = rows.Close() + return err + } + } + if err := closeRows(rows); err != nil { + return fmt.Errorf("clickhouse audit store: read: %w", err) + } + return nil +} + +// ListAuditEvents implements business.AuditReader: the page in keyset order, +// read in chunks newest first, each chunk deduplicated, ordered and limited in +// ClickHouse. The page is complete once it holds a whole page and the proof of +// another, or the history runs out. +func (s *Store) ListAuditEvents(ctx context.Context, read business.AuditRead) ([]business.AuditEntry, string, error) { + m, err := auditeval.NewMatcher(read) + if err != nil { + return nil, "", err + } + page, err := auditeval.NewPage(read.Query) + if err != nil { + return nil, "", err + } + pl := newPlan(m, read.Query) + if pl.none { + return nil, "", nil + } + // A payload filter the statement carries returns matches alone, so a chunk + // of one more than a page fills it; one the service evaluates may reject + // candidates, so it reads further at a time. + chunk := page.Size() + 1 + if m.NeedsPayload() && pl.payload == nil && chunk < unpushedChunk { + chunk = unpushedChunk + } + at, id, after := page.After() + for { + events, err := s.listChunk(ctx, pl, at, id, after, chunk) + if err != nil { + return nil, "", err + } + if m.NeedsPayload() { + if err := s.joinDetails(ctx, m.Scope(), events); err != nil { + return nil, "", err + } + } + for _, event := range events { + ok, err := m.Match(event) + if err != nil { + return nil, "", err + } + if ok { + page.Offer(event) + } + } + if page.Full() || len(events) < chunk { + break + } + last := events[len(events)-1].Entry + at, id, after = last.CreatedAt, last.ID, true + } + events, next := page.Result() + if !m.NeedsPayload() { + if err := s.joinDetails(ctx, m.Scope(), events); err != nil { + return nil, "", err + } + } + return results(events), next, nil +} + +// unpushedChunk is how many candidates a list reads at a time when the service +// evaluates its payload filter. +const unpushedChunk = 1000 + +// listChunk reads up to limit events strictly after the keyset position +// (when after), newest first, each once. +func (s *Store) listChunk(ctx context.Context, pl plan, at time.Time, id string, after bool, limit int) ([]*auditeval.Event, error) { + p := &sqlParams{} + w, ok := s.eventsWhere(p, pl) + if !ok { + return nil, nil + } + if after { + bound := p.instant(at) + w.and("occurred_at <= " + bound) + w.and("(occurred_at, event_id) < (" + bound + ", " + p.text(id) + ")") + } + statement := "SELECT " + eventColumns + " FROM " + EventsTable + " WHERE " + w.String() + + " ORDER BY occurred_at DESC, event_id DESC LIMIT 1 BY event_id LIMIT " + p.count(limit) + var events []*auditeval.Event + err := s.scan(ctx, statement, p.args, func(rows driver.Rows) (bool, error) { + row, err := scanStored(rows) + if err != nil { + return false, fmt.Errorf("clickhouse audit store: read %s: %w", EventsTable, err) + } + if !s.inScope(pl.m, row) { + return false, errUnscopedRow + } + events = append(events, row.event()) + return true, nil + }) + return events, err +} + +// joinDetails reads the details of the given events that have none yet — the +// content-class events — in one statement bounded by their occurrences and +// ids. An event whose details are past the content window keeps none. +func (s *Store) joinDetails(ctx context.Context, scope business.AuditReadScope, events []*auditeval.Event) error { + var ids []string + var lo, hi time.Time + for _, event := range events { + if event.HasDetails { + continue + } + at := event.Entry.CreatedAt + if len(ids) == 0 || at.Before(lo) { + lo = at + } + if len(ids) == 0 || at.After(hi) { + hi = at + } + ids = append(ids, event.Entry.ID) + } + if len(ids) == 0 { + return nil + } + p := &sqlParams{} + w := s.scope(p, scope) + w.window(p, &lo, &hi) + w.and("has(" + p.texts(ids) + ", event_id)") + statement := "SELECT event_id, details_sha256, any(details) FROM " + DetailsTable + " WHERE " + w.String() + + " GROUP BY event_id, details_sha256" + type key struct{ id, sha string } + details := map[key]string{} + err := s.scan(ctx, statement, p.args, func(rows driver.Rows) (bool, error) { + var id, sha, text string + if err := rows.Scan(&id, &sha, &text); err != nil { + return false, fmt.Errorf("clickhouse audit store: read %s: %w", DetailsTable, err) + } + details[key{id, sha}] = text + return true, nil + }) + if err != nil { + return err + } + for _, event := range events { + if event.HasDetails { + continue + } + if text, ok := details[key{event.Entry.ID, event.DetailsSHA256}]; ok && text != "" { + event.Details, event.HasDetails = text, true + } + } + return nil +} + +func results(events []*auditeval.Event) []business.AuditEntry { + if len(events) == 0 { + return nil + } + out := make([]business.AuditEntry, len(events)) + for i, event := range events { + out[i] = event.Result() + } + return out +} + +// joined streams every matching event once, with its details, to visit — in +// list order when ordered — after the service has matched it. +func (s *Store) joined(ctx context.Context, m *auditeval.Matcher, pl plan, ordered bool, visit func(*auditeval.Event) error) error { + p := &sqlParams{} + statement, ok := s.joinedEvents(p, pl, ordered) + if !ok { + return nil + } + return s.scan(ctx, statement, p.args, func(rows driver.Rows) (bool, error) { + row, err := scanStored(rows) + if err != nil { + return false, fmt.Errorf("clickhouse audit store: read %s: %w", EventsTable, err) + } + if !s.inScope(m, row) { + return false, errUnscopedRow + } + event := row.event() + ok, err := m.Match(event) + if err != nil || !ok { + return err == nil, err + } + return true, visit(event) + }) +} + +// ExportAuditEvents implements business.AuditReader: every matching event, +// once each, newest first, with its payload. +func (s *Store) ExportAuditEvents(ctx context.Context, read business.AuditRead) ([]business.AuditEntry, error) { + m, err := auditeval.NewMatcher(read) + if err != nil { + return nil, err + } + var events []*auditeval.Event + err = s.joined(ctx, m, newPlan(m, read.Query), true, func(event *auditeval.Event) error { + events = append(events, event) + return nil + }) + if err != nil { + return nil, err + } + return results(events), nil +} + +// AggregateAuditEvents implements business.AuditReader: one GROUP BY in +// ClickHouse when it is exact for this read (aggregate.go), and otherwise the +// deduplicated, matched events aggregated in the service. +func (s *Store) AggregateAuditEvents(ctx context.Context, read business.AuditRead, spec business.AuditAggregationSpec) ([]business.AuditAggregateBucket, error) { + m, err := auditeval.NewMatcher(read) + if err != nil { + return nil, err + } + // The service's aggregator is built either way: it refuses what Postgres + // refuses, and it is the fallback. + aggregator, err := auditeval.NewAggregator(spec, read.Types) + if err != nil { + return nil, err + } + pl := newPlan(m, read.Query) + if pl.none { + return nil, nil + } + if !m.NeedsPayload() || pl.payload != nil { + buckets, exact, err := s.aggregateInClickHouse(ctx, pl, spec, read.Types) + if err != nil || exact { + return buckets, err + } + } + if s.inService != nil { + s.inService() + } + err = s.joined(ctx, m, pl, false, aggregator.Add) + if err != nil { + return nil, err + } + return aggregator.Buckets(), nil +} + +// aggregateInClickHouse runs the aggregation statement. exact is false when a +// row it grouped is one ClickHouse cannot read as Postgres does, and the +// buckets are then discarded. +func (s *Store) aggregateInClickHouse(ctx context.Context, pl plan, spec business.AuditAggregationSpec, types business.AuditEventTypeIndex) ([]business.AuditAggregateBucket, bool, error) { + agg, ok, err := s.buildAggregation(pl, spec, types) + if err != nil || !ok { + return nil, err == nil, err + } + var out []business.AuditAggregateBucket + var inexact uint64 + err = s.scan(ctx, agg.statement, agg.args, func(rows driver.Rows) (bool, error) { + keys := make([]string, agg.dims) + var count, rowInexact uint64 + values := make([]float64, len(agg.metrics)) + arrays := make([][]float64, len(agg.metrics)) + samples := make([]uint64, len(agg.metrics)) + dest := make([]any, 0, agg.dims+2+2*len(agg.metrics)) + for i := range keys { + dest = append(dest, &keys[i]) + } + dest = append(dest, &count) + for i, metric := range agg.metrics { + if metric.Op == "percentile" { + dest = append(dest, &arrays[i]) + } else { + dest = append(dest, &values[i]) + } + } + for i := range samples { + dest = append(dest, &samples[i]) + } + dest = append(dest, &rowInexact) + if err := rows.Scan(dest...); err != nil { + return false, fmt.Errorf("clickhouse audit store: read aggregation: %w", err) + } + inexact += rowInexact + if inexact > 0 { + return false, nil + } + bucket := business.AuditAggregateBucket{ + Keys: keys, + Key: keys[0], + Count: int64(count), + Samples: make(map[string]int64, len(agg.aliases)), + Metrics: make(map[string]float64, len(agg.aliases)+len(spec.Derived)), + } + for i, metric := range agg.metrics { + alias := agg.aliases[i] + bucket.Samples[alias] = int64(samples[i]) + if samples[i] == 0 { + continue + } + if metric.Op == "percentile" { + bucket.Metrics[alias] = auditeval.PercentileCont(arrays[i], metric.Percentile) + } else { + bucket.Metrics[alias] = values[i] + } + } + for _, derived := range spec.Derived { + numerator, numeratorOK := bucket.Metrics[derived.Numerator] + denominator, denominatorOK := bucket.Metrics[derived.Denominator] + if numeratorOK && denominatorOK && denominator != 0 { + bucket.Metrics[derived.Alias] = numerator / denominator + } + } + out = append(out, bucket) + return true, nil + }) + if err != nil { + return nil, false, err + } + if inexact > 0 { + return nil, false, nil + } + return out, true, nil +} + +// LatestSourceSyncEvents implements business.AuditSourceSyncReader: per +// source, its newest sync request in the scope's organization — by +// occurrence, and by event id between two at one instant — in one GROUP BY. +func (s *Store) LatestSourceSyncEvents(ctx context.Context, scope business.AuditReadScope, sources []string) (map[string]business.AuditSourceSyncEvent, error) { + if err := scope.Validate(); err != nil { + return nil, err + } + if scope.Platform() { + return nil, errors.New("clickhouse audit store: the readable-source query reads one organization") + } + out := map[string]business.AuditSourceSyncEvent{} + seen := map[string]bool{} + var ids []string + for _, source := range sources { + if !seen[source] { + seen[source] = true + ids = append(ids, source) + } + } + if len(ids) == 0 { + return out, nil + } + p := &sqlParams{} + w := s.scope(p, scope) + w.and("event_type = " + p.text(string(business.EventDatasourceSourceSynced))) + w.and("resource = " + p.text("datasource")) + w.and("has(" + p.texts(ids) + ", resource_id)") + statement := "SELECT resource_id, max(occurred_at), argMax(actor_id, (occurred_at, event_id)) FROM " + EventsTable + + " WHERE " + w.String() + " GROUP BY resource_id" + err := s.scan(ctx, statement, p.args, func(rows driver.Rows) (bool, error) { + var source, actor string + var at time.Time + if err := rows.Scan(&source, &at, &actor); err != nil { + return false, fmt.Errorf("clickhouse audit store: read %s: %w", EventsTable, err) + } + if !seen[source] { + return false, errUnscopedRow + } + out[source] = business.AuditSourceSyncEvent{RequestedAt: time.UnixMicro(at.UnixMicro()), ActorID: actor} + return true, nil + }) + if err != nil { + return nil, err + } + return out, nil +} + +// ReadStoredAuditEvents implements business.AuditHistoryReader: every copy of +// every event of this deployment in [from, to), each with the details its +// store holds. A content-class event's details come from the details table; +// when the copies there disagree, a copy whose hash does not match the +// event's is the one reported, so the disagreement is what verification sees. +func (s *Store) ReadStoredAuditEvents(ctx context.Context, from, to time.Time, visit func(business.StoredAuditEvent) error) error { + p := &sqlParams{} + window := func() string { + return "deployment_id = " + p.text(s.deploymentID) + " AND occurred_at >= " + p.instant(from) + " AND occurred_at < " + p.instant(to) + } + events := window() + details := window() + statement := "SELECT e.deployment_id, e.event_id, e.org_id, e.actor_id, e.actor_type, e.event_type, e.schema_version, e.resource, " + + "e.resource_id, e.occurred_at, e.ip_address, e.impersonated_by, e.is_impersonated, e.client_id, e.retention_class, " + + "e.details_sha256, e.details, arrayMap(c -> c.1, d.copies), arrayMap(c -> c.2, d.copies) " + + "FROM (SELECT " + eventColumns + " FROM " + EventsTable + " WHERE " + events + ") AS e " + + "LEFT JOIN (SELECT event_id, groupArray((details_sha256, details)) AS copies FROM " + DetailsTable + + " WHERE " + details + " GROUP BY event_id) AS d ON e.event_id = d.event_id" + return s.scan(ctx, statement, p.args, func(rows driver.Rows) (bool, error) { + var hashes, texts []string + row, err := scanStored(rows, &hashes, &texts) + if err != nil { + return false, fmt.Errorf("clickhouse audit store: read %s: %w", EventsTable, err) + } + if row.deploymentID != s.deploymentID { + return false, errUnscopedRow + } + event := business.StoredAuditEvent{ + DeploymentID: row.deploymentID, + Entry: row.entry, + Retention: business.AuditRetentionClass(row.retentionClass), + DetailsSHA256: row.detailsSHA256, + Details: row.details, + HasDetails: row.details != "", + } + if event.Retention != business.RetentionSecurity { + event.Details, event.HasDetails = "", false + for i, text := range texts { + event.Details, event.HasDetails = text, true + if hashes[i] != row.detailsSHA256 { + break + } + } + } + return true, visit(event) + }) +} diff --git a/module/services/accounts/code/pkg/auditstore/clickhousestore/server_test.go b/module/services/accounts/code/pkg/auditstore/clickhousestore/server_test.go new file mode 100644 index 000000000..83bada483 --- /dev/null +++ b/module/services/accounts/code/pkg/auditstore/clickhousestore/server_test.go @@ -0,0 +1,429 @@ +package clickhousestore + +import ( + "context" + "encoding/json" + "os" + "strings" + "testing" + "time" + + "accounts/pkg/auditstore/auditeval" + "accounts/pkg/business" + + "github.com/ClickHouse/clickhouse-go/v2" + "github.com/ClickHouse/clickhouse-go/v2/lib/driver" + "github.com/google/uuid" + "github.com/stretchr/testify/require" +) + +// The adapter against a ClickHouse server, when one is reachable: +// +// docker run --rm -p 9000:9000 -e CLICKHOUSE_USER= -e CLICKHOUSE_PASSWORD= \ +// clickhouse/clickhouse-server:25.8 +// AUDIT_CLICKHOUSE_TEST_DSN=clickhouse://:@127.0.0.1:9000/default \ +// go test ./pkg/auditstore/clickhousestore +// +// It is skipped otherwise. Each test creates its own database — the store +// itself never creates one; the deployment does — and drops it afterwards. +// The same variable enrolls ClickHouse in the read-parity suite against +// Postgres (pkg/business/audit_store_parity_test.go). + +const testDeployment = "deployment-1" + +// serverDatabase creates a database for one test and connects to it. +func serverDatabase(t *testing.T) (driver.Conn, string) { + t.Helper() + dsn := strings.TrimSpace(os.Getenv("AUDIT_CLICKHOUSE_TEST_DSN")) + if dsn == "" { + t.Skip("AUDIT_CLICKHOUSE_TEST_DSN is not set; see the comment above for how to run against a ClickHouse server") + } + options, err := clickhouse.ParseDSN(dsn) + require.NoError(t, err) + admin, err := clickhouse.Open(options) + require.NoError(t, err) + t.Cleanup(func() { _ = admin.Close() }) + database := "audit_" + strings.ReplaceAll(uuid.NewString(), "-", "")[:12] + require.NoError(t, admin.Exec(context.Background(), "CREATE DATABASE "+database)) + t.Cleanup(func() { _ = admin.Exec(context.Background(), "DROP DATABASE IF EXISTS "+database) }) + + scoped := *options + scoped.Auth.Database = database + conn, err := clickhouse.Open(&scoped) + require.NoError(t, err) + t.Cleanup(func() { _ = conn.Close() }) + return conn, database +} + +func serverStore(t *testing.T, conn driver.Conn, database string, mutate ...func(*Config)) *Store { + t.Helper() + cfg := Config{ + Database: database, + DeploymentID: testDeployment, + EventsRetention: 2555 * 24 * time.Hour, + ContentDetailRetention: 30 * 24 * time.Hour, + } + for _, m := range mutate { + m(&cfg) + } + store, err := New(conn, cfg) + require.NoError(t, err) + return store +} + +func TestServerEnsureRefusesAMissingDatabase(t *testing.T) { + conn, database := serverDatabase(t) + require.NoError(t, conn.Exec(context.Background(), "DROP DATABASE "+database)) + store := serverStore(t, conn, database) + require.ErrorContains(t, store.Ensure(context.Background()), "does not exist; the deployment creates it") +} + +func TestServerEnsureCreatesBothTablesOnceAndRefusesAMismatch(t *testing.T) { + conn, database := serverDatabase(t) + ctx := context.Background() + store := serverStore(t, conn, database) + require.NoError(t, store.Ensure(ctx)) + require.NoError(t, store.Ensure(ctx), "a second start finds both tables and accepts them") + + for table, days := range map[string]string{EventsTable: "2555", DetailsTable: "30"} { + var engine string + require.NoError(t, store.queryRow(ctx, "SELECT engine_full FROM system.tables WHERE database = currentDatabase() AND name = '"+table+"'", nil, &engine)) + require.Contains(t, engine, "MergeTree PARTITION BY toYYYYMM(occurred_at) ORDER BY (deployment_id, org_id, occurred_at, event_id)", table) + require.Contains(t, engine, "TTL occurred_at + toIntervalDay("+days+")", table) + } + + other := serverStore(t, conn, database, func(cfg *Config) { cfg.ContentDetailRetention = 400 * 24 * time.Hour }) + require.ErrorContains(t, other.Ensure(ctx), "rows expire after 30 days, the configured window is 400 days", + "a details table expiring at another window is refused, never silently kept") + other = serverStore(t, conn, database, func(cfg *Config) { cfg.EventsRetention = 3650 * 24 * time.Hour }) + require.ErrorContains(t, other.Ensure(ctx), "table "+database+".audit_events: rows expire after 2555 days") + + require.NoError(t, conn.Exec(ctx, "ALTER TABLE "+DetailsTable+" MODIFY COLUMN details Nullable(String)")) + require.ErrorContains(t, store.Ensure(ctx), "column details is Nullable(String), want String") + require.NoError(t, conn.Exec(ctx, "ALTER TABLE "+DetailsTable+" MODIFY COLUMN details String")) + require.NoError(t, conn.Exec(ctx, "ALTER TABLE "+DetailsTable+" REMOVE TTL")) + require.ErrorContains(t, store.Ensure(ctx), "has no TTL") + require.NoError(t, conn.Exec(ctx, "ALTER TABLE "+DetailsTable+" MODIFY TTL toDateTime(occurred_at) + toIntervalDay(30)")) + require.ErrorContains(t, store.Ensure(ctx), "expires rows by") + require.NoError(t, conn.Exec(ctx, "ALTER TABLE "+DetailsTable+" MODIFY TTL occurred_at + INTERVAL 30 DAY")) + require.NoError(t, store.Ensure(ctx), "the table is as the store writes it again") + require.NoError(t, conn.Exec(ctx, "ALTER TABLE "+DetailsTable+" DROP COLUMN details_sha256")) + require.ErrorContains(t, store.Ensure(ctx), "has no column details_sha256") +} + +// fixture is a set of records, the batches they were appended in, and the +// reference reads over them. +type fixture struct { + org, otherOrg string + actor string + records []business.AuditRecord +} + +func newRecord(t *testing.T, org, actor string, eventType business.EventType, resourceID string, at time.Time, class business.AuditRetentionClass, payload map[string]any) business.AuditRecord { + t.Helper() + record, err := business.NewAuditRecord(business.AuditEntry{ + ID: uuid.NewString(), + OrgID: org, + ActorID: actor, + ActorType: business.ActorTypeUser, + EventType: eventType, + SchemaVersion: 1, + Resource: "datasource", + ResourceID: resourceID, + Payload: payload, + CreatedAt: at, + }, class) + require.NoError(t, err) + return record +} + +func appendBatch(t *testing.T, store *Store, deployment string, records ...business.AuditRecord) { + t.Helper() + require.NoError(t, store.AppendAuditBatch(context.Background(), business.AuditBatch{ + ID: uuid.NewString(), DeploymentID: deployment, ComposedAt: time.Now(), Records: records, + })) +} + +// reference is the read answered by auditeval over the records themselves, +// once each: the semantics every store reproduces. +func (f *fixture) referenceList(t *testing.T, read business.AuditRead) ([]business.AuditEntry, string) { + t.Helper() + m, err := auditeval.NewMatcher(read) + require.NoError(t, err) + page, err := auditeval.NewPage(read.Query) + require.NoError(t, err) + for _, event := range f.events() { + ok, err := m.Match(event) + require.NoError(t, err) + if ok { + page.Offer(event) + } + } + events, next := page.Result() + return results(events), next +} + +func (f *fixture) referenceAggregate(t *testing.T, read business.AuditRead, spec business.AuditAggregationSpec) []business.AuditAggregateBucket { + t.Helper() + m, err := auditeval.NewMatcher(read) + require.NoError(t, err) + aggregator, err := auditeval.NewAggregator(spec, read.Types) + require.NoError(t, err) + for _, event := range f.events() { + ok, err := m.Match(event) + require.NoError(t, err) + if ok { + require.NoError(t, aggregator.Add(event)) + } + } + return aggregator.Buckets() +} + +func (f *fixture) events() []*auditeval.Event { + out := make([]*auditeval.Event, len(f.records)) + for i, record := range f.records { + entry := record.Entry + entry.Payload = nil + entry.CreatedAt = time.UnixMicro(entry.CreatedAt.UnixMicro()) + out[i] = &auditeval.Event{Entry: entry, Details: record.Details, HasDetails: true, DetailsSHA256: record.DetailsSHA256} + } + return out +} + +func TestServerReadsMatchTheReferenceSemantics(t *testing.T) { + conn, database := serverDatabase(t) + ctx := context.Background() + store := serverStore(t, conn, database) + require.NoError(t, store.Ensure(ctx)) + + f := &fixture{org: uuid.NewString(), otherOrg: uuid.NewString(), actor: uuid.NewString()} + base := time.Now().UTC().Truncate(time.Microsecond).Add(-time.Minute) + tricky := "doc\\1\t'x'}{p0:String}\n" + security, content := business.RetentionSecurity, business.RetentionContent + f.records = []business.AuditRecord{ + newRecord(t, f.org, f.actor, business.EventAuthLogin, "", base, security, map[string]any{"method": "password", "flag": true}), + newRecord(t, f.org, f.actor, business.EventDocumentRead, tricky, base.Add(-time.Hour), content, map[string]any{"boundary": "b-1", "duration_ms": 12.5, "count": 3}), + newRecord(t, f.org, "", business.EventDocumentRead, "doc-2", base.Add(-2*time.Hour), content, map[string]any{"boundary": "b\\2", "duration_ms": "7.25", "note": nil}), + // An integer beyond 64 bits: valid JSON, which ClickHouse's parser refuses. + newRecord(t, f.org, f.actor, business.EventDocumentRead, "doc-3", base.Add(-3*time.Hour), content, map[string]any{"boundary": "b-1", "big": jsonNumber("123456789012345678901234567890"), "duration_ms": 4}), + newRecord(t, f.org, f.actor, business.EventDocumentRead, "doc-4", base.Add(-30*time.Hour), content, map[string]any{"boundary": "b-3", "nested": map[string]any{"zz": 1, "a": []any{"q"}}, "duration_ms": jsonNumber("1.50")}), + newRecord(t, f.org, f.actor, business.EventOrgUpdated, "", base.Add(-50*time.Hour), security, map[string]any{"name": "First"}), + newRecord(t, f.otherOrg, f.actor, business.EventDocumentRead, "doc-1", base.Add(-time.Hour), content, map[string]any{"boundary": "b-1"}), + newRecord(t, "", f.actor, business.EventAuthLogin, "", base.Add(-10*time.Minute), security, map[string]any{"method": "passkey"}), + } + appendBatch(t, store, testDeployment, f.records[:4]...) + appendBatch(t, store, testDeployment, f.records[4:]...) + appendBatch(t, store, testDeployment, f.records[1:3]...) // delivered again + appendBatch(t, store, "deployment-other", f.records[0], f.records[5]) // another deployment's copy + foreign := newRecord(t, f.org, f.actor, business.EventAuthLogin, "", base, security, map[string]any{"method": "password"}) + appendBatch(t, store, "deployment-other", foreign) + + var parts uint64 + require.NoError(t, store.queryRow(ctx, "SELECT count() FROM system.parts WHERE database = currentDatabase() AND table = 'audit_events' AND active", nil, &parts)) + require.LessOrEqual(t, parts, uint64(5), "every append is one insert, one part, per table") + + types := business.AuditEventTypeIndex{ + string(business.EventAuthLogin): {Category: "auth", Namespace: "saas"}, + string(business.EventDocumentRead): {Category: "document", Namespace: "saas"}, + string(business.EventOrgUpdated): {Category: "organization", Namespace: "saas"}, + } + week := base.Add(-48 * time.Hour) + queries := map[string]business.AuditQuery{ + "organization": {OrgID: f.org}, + "actor": {OrgID: f.org, ActorID: f.actor}, + "types": {OrgID: f.org, EventTypes: []string{string(business.EventDocumentRead), string(business.EventOrgUpdated)}}, + "category": {OrgID: f.org, Category: "document"}, + "tricky resource": {OrgID: f.org, ResourceID: tricky}, + "payload string": {OrgID: f.org, PayloadContains: map[string]any{"boundary": "b-1"}}, + "payload escaped": {OrgID: f.org, PayloadContains: map[string]any{"boundary": "b\\2"}}, + "payload bool": {OrgID: f.org, PayloadContains: map[string]any{"flag": true}}, + "payload on a type": {OrgID: f.org, EventType: string(business.EventAuthLogin), PayloadContains: map[string]any{"method": "password"}}, + "payload null": {OrgID: f.org, PayloadContains: map[string]any{"note": nil}}, + "payload number": {OrgID: f.org, PayloadContains: map[string]any{"count": 3}}, + "payload nested": {OrgID: f.org, PayloadContains: map[string]any{"nested": map[string]any{"a": []any{"q"}}}}, + "payload big": {OrgID: f.org, PayloadContains: map[string]any{"big": jsonNumber("123456789012345678901234567890")}}, + "window": {OrgID: f.org, From: &week, To: &base}, + "platform": {}, + "platform for one": {ActorID: f.actor}, + } + specs := map[string]business.AuditAggregationSpec{ + "default": {}, + "category": {GroupBy: []string{"category", "actor"}, Metrics: []business.AuditMetric{{Op: "count_distinct", Field: "category"}, {Op: "count_distinct", Field: "actor_id"}}}, + "week": {GroupBy: []string{"time"}, Bucket: "week", Metrics: []business.AuditMetric{{Op: "count_distinct", Field: "resource_id"}}}, + "text": {GroupBy: []string{"payload:boundary", "payload:method"}, Metrics: []business.AuditMetric{{Op: "count_distinct", Field: "payload:count"}}}, + "nested": {GroupBy: []string{"payload:nested"}}, + "fraction": {GroupBy: []string{"payload:duration_ms"}}, + "numbers": {GroupBy: []string{"event_type"}, Metrics: []business.AuditMetric{ + {Op: "count", Alias: "n"}, + {Op: "sum", Field: "payload:duration_ms"}, + {Op: "avg", Field: "payload:duration_ms"}, + {Op: "min", Field: "payload:duration_ms"}, + {Op: "max", Field: "payload:count"}, + {Op: "percentile", Field: "payload:duration_ms", Percentile: 0.3, Alias: "p30"}, + }, Derived: []business.AuditDerivedMetric{{Alias: "ratio", Numerator: "sum_duration_ms", Denominator: "n"}}}, + } + inService := 0 + store.inService = func() { inService++ } + // Which aggregations ClickHouse runs, and which the service does: the + // answers are compared below either way. + for _, tc := range []struct { + query, spec string + want int + }{ + {"organization", "default", 0}, + {"tricky resource", "numbers", 0}, + {"payload on a type", "week", 0}, + {"payload string", "week", 1}, // a candidate ClickHouse cannot parse + {"tricky resource", "fraction", 1}, + {"tricky resource", "nested", 0}, + {"organization", "numbers", 1}, + {"payload number", "default", 1}, + } { + before := inService + q := queries[tc.query] + _, err := store.AggregateAuditEvents(ctx, business.AuditRead{Scope: business.OrganizationAuditScope(q.OrgID), Query: q, Types: types}, specs[tc.spec]) + require.NoError(t, err) + require.Equal(t, tc.want, inService-before, "%s / %s", tc.query, tc.spec) + } + + for name, q := range queries { + scope := business.OrganizationAuditScope(q.OrgID) + if q.OrgID == "" { + scope = business.PlatformAuditScope() + } + for _, size := range []int32{0, 1, 2} { + q := q + q.PageSize = size + for pages := 0; ; pages++ { + read := business.AuditRead{Scope: scope, Query: q, Types: types} + want, wantNext := f.referenceList(t, read) + got, gotNext, err := store.ListAuditEvents(ctx, read) + require.NoError(t, err, name) + require.Equal(t, want, got, "%s, page size %d, page %d", name, size, pages) + require.Equal(t, wantNext, gotNext, name) + if wantNext == "" { + break + } + q.PageToken = wantNext + } + } + read := business.AuditRead{Scope: scope, Query: q, Types: types} + all := read + all.Query.PageSize = 1000 + want, _ := f.referenceList(t, all) + got, err := store.ExportAuditEvents(ctx, read) + require.NoError(t, err, name) + require.Equal(t, want, got, "export %s", name) + + for specName, spec := range specs { + want := f.referenceAggregate(t, read, spec) + got, err := store.AggregateAuditEvents(ctx, read, spec) + require.NoError(t, err, "%s / %s", name, specName) + require.Equal(t, want, got, "%s / %s", name, specName) + } + } + + t.Run("scope", func(t *testing.T) { + _, _, err := store.ListAuditEvents(ctx, business.AuditRead{Query: business.AuditQuery{OrgID: f.org}}) + require.ErrorIs(t, err, business.ErrAuditReadUnscoped) + _, _, err = store.ListAuditEvents(ctx, business.AuditRead{Scope: business.OrganizationAuditScope(f.org), Query: business.AuditQuery{OrgID: f.otherOrg}}) + require.ErrorContains(t, err, "outside the") + _, err = store.LatestSourceSyncEvents(ctx, business.PlatformAuditScope(), []string{"doc-1"}) + require.ErrorContains(t, err, "one organization") + }) + + t.Run("history", func(t *testing.T) { + copies := map[string]int{} + err := store.ReadStoredAuditEvents(ctx, base.Add(-100*time.Hour), base.Add(time.Microsecond), func(event business.StoredAuditEvent) error { + require.Equal(t, testDeployment, event.DeploymentID) + copies[event.Entry.ID]++ + for _, record := range f.records { + if record.Entry.ID == event.Entry.ID { + require.Equal(t, record.Retention, event.Retention) + require.Equal(t, record.DetailsSHA256, event.DetailsSHA256) + require.True(t, event.HasDetails) + require.Equal(t, record.Details, event.Details) + } + } + return nil + }) + require.NoError(t, err) + require.Len(t, copies, len(f.records), "every event of this deployment, and none of another's") + require.Equal(t, 2, copies[f.records[1].Entry.ID], "every copy") + require.Equal(t, 1, copies[f.records[0].Entry.ID]) + }) +} + +// jsonNumber is a payload number with exactly this text, as the queue reads it. +func jsonNumber(text string) any { return json.Number(text) } + +func TestServerLatestSourceSyncEvents(t *testing.T) { + conn, database := serverDatabase(t) + ctx := context.Background() + store := serverStore(t, conn, database) + require.NoError(t, store.Ensure(ctx)) + + org, actor, other := uuid.NewString(), uuid.NewString(), uuid.NewString() + base := time.Now().UTC().Truncate(time.Microsecond) + synced := func(source, by string, at time.Time) business.AuditRecord { + return newRecord(t, org, by, business.EventDatasourceSourceSynced, source, at, business.RetentionContent, map[string]any{"job_id": uuid.NewString()}) + } + newest := synced("source-1", actor, base.Add(-time.Hour)) + appendBatch(t, store, testDeployment, + synced("source-1", other, base.Add(-3*time.Hour)), newest, + synced("source-2", "", base.Add(-2*time.Hour)), + newRecord(t, uuid.NewString(), other, business.EventDatasourceSourceSynced, "source-3", base, business.RetentionContent, nil), + ) + appendBatch(t, store, testDeployment, newest) + appendBatch(t, store, "deployment-other", synced("source-1", other, base)) + + got, err := store.LatestSourceSyncEvents(ctx, business.OrganizationAuditScope(org), []string{"source-1", "source-2", "source-3", "source-1"}) + require.NoError(t, err) + require.Equal(t, map[string]business.AuditSourceSyncEvent{ + "source-1": {RequestedAt: time.UnixMicro(base.Add(-time.Hour).UnixMicro()), ActorID: actor}, + "source-2": {RequestedAt: time.UnixMicro(base.Add(-2 * time.Hour).UnixMicro())}, + }, got, "another organization's and another deployment's requests are outside the read") +} + +// Decimal texts whose nearest double a fast parser misses: ClickHouse reads +// each — as a JSON number and as a JSON string — to the double Postgres's +// float8in reads, so the metrics over them run in ClickHouse and agree. +func TestServerNumericReadingIsExact(t *testing.T) { + conn, database := serverDatabase(t) + ctx := context.Background() + store := serverStore(t, conn, database) + require.NoError(t, store.Ensure(ctx)) + + f := &fixture{org: uuid.NewString()} + base := time.Now().UTC().Truncate(time.Microsecond) + for i, text := range []string{ + "96.55639781", "0.000000000000005720547203731686", "0.000000000000011375683516399455", + "376.8549944162621642512", "844.8835565338717500304", "801.3627100269752912500", "9007199254740993", "-12345678901234567", + } { + at := base.Add(-time.Duration(i) * time.Minute) + f.records = append(f.records, + newRecord(t, f.org, "", business.EventDocumentRead, "doc", at, business.RetentionContent, map[string]any{"label": "number " + text, "value": jsonNumber(text)}), + newRecord(t, f.org, "", business.EventDocumentRead, "doc", at, business.RetentionContent, map[string]any{"label": "string " + text, "value": text}), + ) + } + appendBatch(t, store, testDeployment, f.records...) + + inService := 0 + store.inService = func() { inService++ } + read := business.AuditRead{Scope: business.OrganizationAuditScope(f.org), Query: business.AuditQuery{OrgID: f.org}} + for _, metric := range []string{"sum", "avg", "min", "max", "percentile"} { + // One group per value: each metric is that value's reading itself. + spec := business.AuditAggregationSpec{GroupBy: []string{"payload:label"}, Metrics: []business.AuditMetric{ + {Op: metric, Field: "payload:value", Percentile: 0.37}, + }} + got, err := store.AggregateAuditEvents(ctx, read, spec) + require.NoError(t, err) + require.Equal(t, f.referenceAggregate(t, read, spec), got, metric) + // And across them all, in one group. + spec.GroupBy = []string{"event_type"} + got, err = store.AggregateAuditEvents(ctx, read, spec) + require.NoError(t, err) + require.Equal(t, f.referenceAggregate(t, read, spec), got, metric) + } + require.Zero(t, inService, "every one ran in ClickHouse") +} diff --git a/module/services/accounts/code/pkg/auditstore/gcsarchive/gcsarchive.go b/module/services/accounts/code/pkg/auditstore/gcsarchive/gcsarchive.go index 8d3222bc4..de1a54201 100644 --- a/module/services/accounts/code/pkg/auditstore/gcsarchive/gcsarchive.go +++ b/module/services/accounts/code/pkg/auditstore/gcsarchive/gcsarchive.go @@ -1,5 +1,6 @@ // Package gcsarchive writes the audit store swap's locked archive (ADR 0009) to -// a Google Cloud Storage bucket: one JSON Lines object per relay batch. +// a Google Cloud Storage bucket: one JSON Lines object per relay batch, named +// and encoded as every archive is (archiveformat). // // The writer's authority is storage.objects.create alone. It never overwrites — // every write carries a does-not-exist precondition — and never deletes; the @@ -10,25 +11,20 @@ package gcsarchive import ( - "bytes" "context" - "encoding/json" "errors" "fmt" "hash/crc32" "net/http" "strings" - "time" + "accounts/pkg/auditstore/archiveformat" "accounts/pkg/business" "cloud.google.com/go/storage" "google.golang.org/api/googleapi" ) -// ContentType is every archive object's media type. -const ContentType = "application/x-ndjson" - // objectCreator creates one object only if none of that name exists. type objectCreator interface { CreateObject(ctx context.Context, name string, body []byte) error @@ -56,11 +52,11 @@ func (a *Archive) WriteAuditBatch(ctx context.Context, batch business.AuditBatch if len(batch.Records) == 0 { return errors.New("gcs audit archive: refusing an empty batch") } - name, err := ObjectName(batch) + name, err := archiveformat.ObjectName(batch) if err != nil { return err } - body, err := Encode(batch) + body, err := archiveformat.Encode(batch) if err != nil { return err } @@ -70,95 +66,6 @@ func (a *Archive) WriteAuditBatch(ctx context.Context, batch business.AuditBatch return nil } -// ObjectName names a batch's object: -// -// ///
/-.jsonl -// -// The date path is when the relay composed the batch, so a day's objects list -// together and in delivery order. The batch id is unique to one delivery -// attempt: a retried batch is a new object, never a rewrite of an old one. -func ObjectName(batch business.AuditBatch) (string, error) { - if !safeSegment(batch.DeploymentID) || !safeSegment(batch.ID) { - return "", fmt.Errorf("gcs audit archive: deployment id %q and batch id %q must be plain path segments", batch.DeploymentID, batch.ID) - } - at := batch.ComposedAt.UTC() - return fmt.Sprintf("%s/%s/%s-%s.jsonl", - batch.DeploymentID, at.Format("2006/01/02"), at.Format("20060102T150405.000000000Z"), batch.ID), nil -} - -func safeSegment(segment string) bool { - if segment == "" || segment == "." || segment == ".." { - return false - } - for _, r := range segment { - if !(r >= 'a' && r <= 'z' || r >= 'A' && r <= 'Z' || r >= '0' && r <= '9' || r == '-' || r == '_' || r == '.') { - return false - } - } - return true -} - -// Line is one archive line: an event's envelope, the SHA-256 of its canonical -// details, and — for a security-class event only — the details themselves. A -// content-class line is content-free. Details is the canonical string as the -// store keeps it, so a reader verifies a line by hashing that string. -type Line struct { - EventID string `json:"event_id"` - DeploymentID string `json:"deployment_id"` - BatchID string `json:"batch_id"` - OrgID string `json:"org_id,omitempty"` - ActorID string `json:"actor_id,omitempty"` - ActorType string `json:"actor_type"` - EventType string `json:"event_type"` - SchemaVersion int `json:"schema_version"` - Resource string `json:"resource"` - ResourceID string `json:"resource_id,omitempty"` - OccurredAt string `json:"occurred_at"` - IPAddress string `json:"ip_address,omitempty"` - ImpersonatedBy string `json:"impersonated_by,omitempty"` - IsImpersonated bool `json:"is_impersonated"` - ClientID string `json:"client_id,omitempty"` - RetentionClass string `json:"retention_class"` - DetailsSHA256 string `json:"details_sha256"` - Details string `json:"details,omitempty"` -} - -// Encode renders a batch as JSON Lines, one line per record in batch order. -func Encode(batch business.AuditBatch) ([]byte, error) { - var buf bytes.Buffer - encoder := json.NewEncoder(&buf) - encoder.SetEscapeHTML(false) - for _, record := range batch.Records { - entry := record.Entry - line := Line{ - EventID: entry.ID, - DeploymentID: batch.DeploymentID, - BatchID: batch.ID, - OrgID: entry.OrgID, - ActorID: entry.ActorID, - ActorType: entry.ActorType, - EventType: string(entry.EventType), - SchemaVersion: entry.SchemaVersion, - Resource: entry.Resource, - ResourceID: entry.ResourceID, - OccurredAt: entry.CreatedAt.UTC().Format(time.RFC3339Nano), - IPAddress: entry.IPAddress, - ImpersonatedBy: entry.ImpersonatedBy, - IsImpersonated: entry.IsImpersonated, - ClientID: entry.ClientID, - RetentionClass: string(record.Retention), - DetailsSHA256: record.DetailsSHA256, - } - if record.Retention == business.RetentionSecurity { - line.Details = record.Details - } - if err := encoder.Encode(line); err != nil { - return nil, fmt.Errorf("gcs audit archive: encode event %s: %w", entry.ID, err) - } - } - return buf.Bytes(), nil -} - // gcsBucket creates objects in one bucket. type gcsBucket struct { handle *storage.BucketHandle @@ -173,7 +80,7 @@ var castagnoli = crc32.MakeTable(crc32.Castagnoli) // so the precondition failure is the acknowledgement it lost. func (b gcsBucket) CreateObject(ctx context.Context, name string, body []byte) error { writer := b.handle.Object(name).If(storage.Conditions{DoesNotExist: true}).NewWriter(ctx) - writer.ContentType = ContentType + writer.ContentType = archiveformat.ContentType writer.ChunkSize = len(body) writer.CRC32C = crc32.Checksum(body, castagnoli) writer.SendCRC32C = true diff --git a/module/services/accounts/code/pkg/auditstore/gcsarchive/gcsarchive_test.go b/module/services/accounts/code/pkg/auditstore/gcsarchive/gcsarchive_test.go index a6f9546c7..23b97738c 100644 --- a/module/services/accounts/code/pkg/auditstore/gcsarchive/gcsarchive_test.go +++ b/module/services/accounts/code/pkg/auditstore/gcsarchive/gcsarchive_test.go @@ -1,14 +1,12 @@ package gcsarchive import ( - "bufio" - "bytes" "context" - "encoding/json" "errors" "testing" "time" + "accounts/pkg/auditstore/archiveformat" "accounts/pkg/business" "github.com/stretchr/testify/require" @@ -37,60 +35,6 @@ func batchOf(t *testing.T, id string, classes ...business.AuditRetentionClass) b return batch } -func TestObjectNameIsPerBatchUnderTheDeploymentAndDay(t *testing.T) { - name, err := ObjectName(batchOf(t, "4b6d0d1e-0f3c-4c1a-9a55-6f0c2b1d7e10")) - require.NoError(t, err) - require.Equal(t, "deployment-1/2026/10/02/20261002T143005.987654321Z-4b6d0d1e-0f3c-4c1a-9a55-6f0c2b1d7e10.jsonl", name) - - other, err := ObjectName(batchOf(t, "9e2f4a6b-1c3d-4e5f-8a7b-0c1d2e3f4a5b")) - require.NoError(t, err) - require.NotEqual(t, name, other, "a redelivered batch carries a new id, so it is a new object") - - for _, bad := range []business.AuditBatch{ - {ID: "batch", DeploymentID: "../escape", ComposedAt: composedAt}, - {ID: "a/b", DeploymentID: "deployment-1", ComposedAt: composedAt}, - {ID: "", DeploymentID: "deployment-1", ComposedAt: composedAt}, - } { - _, err := ObjectName(bad) - require.Error(t, err, "%+v", bad) - } -} - -func TestEncodeWritesSecurityInFullAndContentWithoutItsDetails(t *testing.T) { - batch := batchOf(t, "batch-1", business.RetentionSecurity, business.RetentionContent) - body, err := Encode(batch) - require.NoError(t, err) - - var lines []Line - scanner := bufio.NewScanner(bytes.NewReader(body)) - for scanner.Scan() { - var line Line - require.NoError(t, json.Unmarshal(scanner.Bytes(), &line)) - lines = append(lines, line) - } - require.Len(t, lines, 2, "one line per record, in batch order") - - security, content := lines[0], lines[1] - require.Equal(t, batch.Records[0].Entry.ID, security.EventID) - require.Equal(t, "security", security.RetentionClass) - require.Equal(t, `{"method":"password","note":"&"}`, security.Details) - require.Equal(t, business.AuditDetailsSHA256(security.Details), security.DetailsSHA256, - "a reader verifies a line by hashing its details string") - - require.Equal(t, batch.Records[1].Entry.ID, content.EventID) - require.Equal(t, "content", content.RetentionClass) - require.Empty(t, content.Details, "a content-class line is content-free") - require.Equal(t, batch.Records[1].DetailsSHA256, content.DetailsSHA256, "its hash proves what the details table holds") - require.NotContains(t, string(body[bytes.IndexByte(body, '\n')+1:]), "password") - - for _, line := range lines { - require.Equal(t, "deployment-1", line.DeploymentID) - require.Equal(t, "batch-1", line.BatchID) - require.Equal(t, "22222222-2222-2222-2222-222222222222", line.OrgID) - require.Equal(t, "2026-10-02T14:30:04.987654321Z", line.OccurredAt) - } -} - // fakeBucket records created objects and refuses to overwrite one. type fakeBucket struct { objects map[string][]byte @@ -115,9 +59,9 @@ func TestWriteAuditBatchWritesOneObjectPerBatch(t *testing.T) { first := batchOf(t, "batch-1", business.RetentionSecurity, business.RetentionContent, business.RetentionContent) require.NoError(t, archive.WriteAuditBatch(context.Background(), first)) require.Len(t, bucket.objects, 1, "a whole batch is one object") - name, err := ObjectName(first) + name, err := archiveformat.ObjectName(first) require.NoError(t, err) - want, err := Encode(first) + want, err := archiveformat.Encode(first) require.NoError(t, err) require.Equal(t, want, bucket.objects[name]) diff --git a/module/services/accounts/code/pkg/auditstore/s3archive/objectlock_test.go b/module/services/accounts/code/pkg/auditstore/s3archive/objectlock_test.go new file mode 100644 index 000000000..38789c562 --- /dev/null +++ b/module/services/accounts/code/pkg/auditstore/s3archive/objectlock_test.go @@ -0,0 +1,87 @@ +package s3archive + +import ( + "context" + "io" + "os" + "strings" + "testing" + + "accounts/pkg/auditstore/archiveformat" + "accounts/pkg/business" + + "github.com/aws/aws-sdk-go-v2/aws" + "github.com/aws/aws-sdk-go-v2/config" + "github.com/aws/aws-sdk-go-v2/service/s3" + "github.com/aws/aws-sdk-go-v2/service/s3/types" + "github.com/google/uuid" + "github.com/stretchr/testify/require" +) + +// The writer against an S3-compatible store with Object Lock, when one is +// reachable: +// +// docker run --rm -p 9000:9000 -e MINIO_ROOT_USER= -e MINIO_ROOT_PASSWORD= \ +// quay.io/minio/minio server /data +// AUDIT_S3_TEST_ENDPOINT=http://127.0.0.1:9000 AWS_ACCESS_KEY_ID= AWS_SECRET_ACCESS_KEY= \ +// AWS_REGION=us-east-1 go test ./pkg/auditstore/s3archive +// +// It is skipped otherwise, and no gate runs it. The test creates a bucket with +// Object Lock and a default retention — the writer itself never creates or +// configures a bucket; the deployment does — and the writer is built the way +// the service builds it, from the environment. +func TestObjectLockBucket(t *testing.T) { + endpoint := strings.TrimSpace(os.Getenv("AUDIT_S3_TEST_ENDPOINT")) + if endpoint == "" { + t.Skip("AUDIT_S3_TEST_ENDPOINT is not set; see the comment above for how to run against an S3-compatible store") + } + t.Setenv("AWS_ENDPOINT_URL_S3", endpoint) + ctx := context.Background() + cfg, err := config.LoadDefaultConfig(ctx) + require.NoError(t, err) + admin := s3.NewFromConfig(cfg, PathStyleForEndpointOverride) + + bucket := "audit-archive-" + strings.ReplaceAll(uuid.NewString(), "-", "")[:12] + _, err = admin.CreateBucket(ctx, &s3.CreateBucketInput{Bucket: aws.String(bucket), ObjectLockEnabledForBucket: aws.Bool(true)}) + require.NoError(t, err) + _, err = admin.PutObjectLockConfiguration(ctx, &s3.PutObjectLockConfigurationInput{ + Bucket: aws.String(bucket), + ObjectLockConfiguration: &types.ObjectLockConfiguration{ + ObjectLockEnabled: types.ObjectLockEnabledEnabled, + Rule: &types.ObjectLockRule{DefaultRetention: &types.DefaultRetention{ + Mode: types.ObjectLockRetentionModeGovernance, Days: aws.Int32(1), + }}, + }, + }) + require.NoError(t, err) + + archive, err := NewFromEnvironment(ctx, bucket) + require.NoError(t, err) + first := batchOf(t, uuid.NewString(), business.RetentionSecurity, business.RetentionContent) + require.NoError(t, archive.WriteAuditBatch(ctx, first), "an Object Lock bucket accepts the put: it carries a checksum") + require.NoError(t, archive.WriteAuditBatch(ctx, first), "a second put of the same name is refused by the store and taken as acknowledged") + second := batchOf(t, uuid.NewString(), business.RetentionContent) + require.NoError(t, archive.WriteAuditBatch(ctx, second)) + + listed, err := admin.ListObjectVersions(ctx, &s3.ListObjectVersionsInput{Bucket: aws.String(bucket)}) + require.NoError(t, err) + require.Len(t, listed.Versions, 2, "one object per batch, and the repeated put wrote no new version") + + for _, batch := range []business.AuditBatch{first, second} { + name, err := archiveformat.ObjectName(batch) + require.NoError(t, err) + object, err := admin.GetObject(ctx, &s3.GetObjectInput{Bucket: aws.String(bucket), Key: aws.String(name)}) + require.NoError(t, err) + body, err := io.ReadAll(object.Body) + require.NoError(t, object.Body.Close()) + require.NoError(t, err) + want, err := archiveformat.Encode(batch) + require.NoError(t, err) + require.Equal(t, string(want), string(body)) + require.Equal(t, archiveformat.ContentType, aws.ToString(object.ContentType)) + + retention, err := admin.GetObjectRetention(ctx, &s3.GetObjectRetentionInput{Bucket: aws.String(bucket), Key: aws.String(name)}) + require.NoError(t, err) + require.Equal(t, types.ObjectLockRetentionModeGovernance, retention.Retention.Mode, "the bucket's default retention locks every object") + } +} diff --git a/module/services/accounts/code/pkg/auditstore/s3archive/s3archive.go b/module/services/accounts/code/pkg/auditstore/s3archive/s3archive.go new file mode 100644 index 000000000..9f21d182b --- /dev/null +++ b/module/services/accounts/code/pkg/auditstore/s3archive/s3archive.go @@ -0,0 +1,134 @@ +// Package s3archive writes the audit store swap's locked archive (ADR 0009) to +// an Amazon S3 bucket, or a bucket of an S3-compatible object store: one JSON +// Lines object per relay batch, named and encoded as every archive is +// (archiveformat), so it reads exactly as a GCS archive does. +// +// The writer's authority is s3:PutObject alone. It never overwrites — every +// write carries If-None-Match: * — and never deletes; the bucket's Object Lock +// default retention, which the deployment configures, is what keeps anyone +// else from doing either before the window ends. Every put carries the +// object's SHA-256, which S3 verifies before it stores the object and which an +// Object Lock bucket requires. +// +// Credentials, region and endpoint come from the AWS SDK's default chain and +// the environment only — on Kubernetes the pod's workload identity, and +// AWS_REGION, plus AWS_ENDPOINT_URL_S3 for an S3-compatible store — so there is +// deliberately no key setting of the kit's own. +package s3archive + +import ( + "bytes" + "context" + "crypto/sha256" + "encoding/base64" + "errors" + "fmt" + "net/http" + "strings" + + "accounts/pkg/auditstore/archiveformat" + "accounts/pkg/business" + + "github.com/aws/aws-sdk-go-v2/aws" + "github.com/aws/aws-sdk-go-v2/config" + "github.com/aws/aws-sdk-go-v2/service/s3" + "github.com/aws/aws-sdk-go-v2/service/s3/types" +) + +// objectPutter is the one S3 call the writer makes; *s3.Client satisfies it. +type objectPutter interface { + PutObject(ctx context.Context, params *s3.PutObjectInput, optFns ...func(*s3.Options)) (*s3.PutObjectOutput, error) +} + +// Archive writes relay batches as archive objects into one bucket. +type Archive struct { + client objectPutter + bucket string +} + +// New writes into bucket through client. +func New(client *s3.Client, bucket string) (*Archive, error) { + if client == nil { + return nil, errors.New("s3 audit archive: client is required") + } + return newArchive(client, bucket) +} + +func newArchive(client objectPutter, bucket string) (*Archive, error) { + if strings.TrimSpace(bucket) == "" { + return nil, errors.New("s3 audit archive: bucket is required") + } + return &Archive{client: client, bucket: bucket}, nil +} + +// NewFromEnvironment writes into bucket through a client the AWS SDK's +// default chain configures: credentials from the environment, the shared +// files or the platform's identity; the region from AWS_REGION; and the +// endpoint from AWS_ENDPOINT_URL_S3 (or AWS_ENDPOINT_URL) when the bucket is in +// an S3-compatible store rather than S3 itself. A missing region is refused +// here, at startup, rather than at the first delivery. +func NewFromEnvironment(ctx context.Context, bucket string) (*Archive, error) { + cfg, err := config.LoadDefaultConfig(ctx) + if err != nil { + return nil, fmt.Errorf("s3 audit archive: aws configuration: %w", err) + } + if strings.TrimSpace(cfg.Region) == "" { + return nil, errors.New("s3 audit archive: no AWS region is configured; set AWS_REGION") + } + return New(s3.NewFromConfig(cfg, PathStyleForEndpointOverride), bucket) +} + +// PathStyleForEndpointOverride addresses buckets by path when the endpoint +// is overridden (AWS_ENDPOINT_URL_S3): S3-compatible stores serve bucket paths +// at their one endpoint, while virtual-hosted addressing would need a DNS +// name per bucket. S3 itself, reached through its own endpoints, keeps the +// SDK's default addressing. +func PathStyleForEndpointOverride(o *s3.Options) { + if o.BaseEndpoint != nil { + o.UsePathStyle = true + } +} + +// WriteAuditBatch implements business.AuditArchive. +func (a *Archive) WriteAuditBatch(ctx context.Context, batch business.AuditBatch) error { + if len(batch.Records) == 0 { + return errors.New("s3 audit archive: refusing an empty batch") + } + name, err := archiveformat.ObjectName(batch) + if err != nil { + return err + } + body, err := archiveformat.Encode(batch) + if err != nil { + return err + } + if err := a.putIfAbsent(ctx, name, body); err != nil { + return fmt.Errorf("s3 audit archive: write %s: %w", name, err) + } + return nil +} + +// putIfAbsent uploads body in one request that S3 refuses with 412 if an +// object of that name exists, carrying the body's SHA-256 so a corrupted +// upload is refused. The SDK retries the request; if a retry finds the object +// already there, that object is this upload's own — the name carries a batch +// id no other attempt uses — so the precondition failure is the +// acknowledgement it lost, exactly as the GCS writer treats it. +func (a *Archive) putIfAbsent(ctx context.Context, name string, body []byte) error { + sum := sha256.Sum256(body) + _, err := a.client.PutObject(ctx, &s3.PutObjectInput{ + Bucket: aws.String(a.bucket), + Key: aws.String(name), + Body: bytes.NewReader(body), + ContentLength: aws.Int64(int64(len(body))), + ContentType: aws.String(archiveformat.ContentType), + IfNoneMatch: aws.String("*"), + ChecksumAlgorithm: types.ChecksumAlgorithmSha256, + ChecksumSHA256: aws.String(base64.StdEncoding.EncodeToString(sum[:])), + }) + var response interface{ HTTPStatusCode() int } + if errors.As(err, &response) && response.HTTPStatusCode() == http.StatusPreconditionFailed { + return nil + } + return err +} diff --git a/module/services/accounts/code/pkg/auditstore/s3archive/s3archive_test.go b/module/services/accounts/code/pkg/auditstore/s3archive/s3archive_test.go new file mode 100644 index 000000000..7fa0b9233 --- /dev/null +++ b/module/services/accounts/code/pkg/auditstore/s3archive/s3archive_test.go @@ -0,0 +1,170 @@ +package s3archive + +import ( + "context" + "crypto/sha256" + "encoding/base64" + "io" + "net/http" + "net/http/httptest" + "strings" + "sync" + "testing" + "time" + + "accounts/pkg/auditstore/archiveformat" + "accounts/pkg/business" + + "github.com/aws/aws-sdk-go-v2/aws" + "github.com/aws/aws-sdk-go-v2/credentials" + "github.com/aws/aws-sdk-go-v2/service/s3" + "github.com/stretchr/testify/require" +) + +var composedAt = time.Date(2026, 10, 2, 14, 30, 5, 987654321, time.UTC) + +func batchOf(t *testing.T, id string, classes ...business.AuditRetentionClass) business.AuditBatch { + t.Helper() + batch := business.AuditBatch{ID: id, DeploymentID: "deployment-1", ComposedAt: composedAt} + for _, class := range classes { + record, err := business.NewAuditRecord(business.AuditEntry{ + ID: business.NewIDString(), + OrgID: "22222222-2222-2222-2222-222222222222", + ActorID: "11111111-1111-1111-1111-111111111111", + ActorType: business.ActorTypeUser, + EventType: business.EventAuthLogin, + SchemaVersion: 1, + Resource: "session", + Payload: map[string]any{"method": "password"}, + CreatedAt: composedAt.Add(-time.Second), + }, class) + require.NoError(t, err) + batch.Records = append(batch.Records, record) + } + return batch +} + +// fakeS3 is a bucket behind an S3 endpoint that honours If-None-Match: * and +// verifies the SHA-256 a put carries. failNext answers the next put of a name +// with that status after storing it — an acknowledgement lost in transit. +type fakeS3 struct { + mu sync.Mutex + objects map[string][]byte + headers []http.Header + failNext map[string]int + refuse int +} + +func (f *fakeS3) ServeHTTP(w http.ResponseWriter, r *http.Request) { + f.mu.Lock() + defer f.mu.Unlock() + if r.Method != http.MethodPut { + w.WriteHeader(http.StatusMethodNotAllowed) + return + } + f.headers = append(f.headers, r.Header.Clone()) + if f.refuse != 0 { + w.WriteHeader(f.refuse) + _, _ = io.WriteString(w, `AccessDenieddenied`) + return + } + key := strings.TrimPrefix(r.URL.Path, "/audit-archive/") + body, _ := io.ReadAll(r.Body) + sum := sha256.Sum256(body) + if r.Header.Get("X-Amz-Checksum-Sha256") != base64.StdEncoding.EncodeToString(sum[:]) { + w.WriteHeader(http.StatusBadRequest) + _, _ = io.WriteString(w, `BadDigestchecksum`) + return + } + if _, exists := f.objects[key]; exists && r.Header.Get("If-None-Match") == "*" { + w.WriteHeader(http.StatusPreconditionFailed) + _, _ = io.WriteString(w, `PreconditionFailedexists`) + return + } + f.objects[key] = body + if status := f.failNext[key]; status != 0 { + delete(f.failNext, key) + w.WriteHeader(status) + _, _ = io.WriteString(w, `InternalErrorlost`) + return + } + w.WriteHeader(http.StatusOK) +} + +func fakeArchive(t *testing.T) (*Archive, *fakeS3) { + t.Helper() + fake := &fakeS3{objects: map[string][]byte{}, failNext: map[string]int{}} + server := httptest.NewServer(fake) + t.Cleanup(server.Close) + client := s3.New(s3.Options{ + Region: "us-east-1", + BaseEndpoint: aws.String(server.URL), + Credentials: credentials.NewStaticCredentialsProvider("test", "test", ""), + }, PathStyleForEndpointOverride) + archive, err := New(client, "audit-archive") + require.NoError(t, err) + return archive, fake +} + +func TestWriteAuditBatchPutsOneObjectThatNeverOverwrites(t *testing.T) { + archive, fake := fakeArchive(t) + ctx := context.Background() + batch := batchOf(t, "batch-1", business.RetentionSecurity, business.RetentionContent) + require.NoError(t, archive.WriteAuditBatch(ctx, batch)) + + name, err := archiveformat.ObjectName(batch) + require.NoError(t, err) + want, err := archiveformat.Encode(batch) + require.NoError(t, err) + require.Equal(t, map[string][]byte{name: want}, fake.objects, "one object, named and encoded as every archive is") + + sent := fake.headers[0] + require.Equal(t, "*", sent.Get("If-None-Match"), "every put is create-only") + require.Equal(t, archiveformat.ContentType, sent.Get("Content-Type")) + require.NotEmpty(t, sent.Get("X-Amz-Checksum-Sha256"), "the put carries the object's SHA-256") + require.Equal(t, "SHA256", sent.Get("X-Amz-Sdk-Checksum-Algorithm")) + + require.NoError(t, archive.WriteAuditBatch(ctx, batch), + "a put that finds its own object (412) is the acknowledgement it lost") + require.Equal(t, want, fake.objects[name], "and the object is not rewritten") +} + +func TestWriteAuditBatchTreatsARetriedPutsPreconditionFailureAsItsAcknowledgement(t *testing.T) { + archive, fake := fakeArchive(t) + batch := batchOf(t, "batch-2", business.RetentionContent) + name, err := archiveformat.ObjectName(batch) + require.NoError(t, err) + fake.failNext[name] = http.StatusInternalServerError + + require.NoError(t, archive.WriteAuditBatch(context.Background(), batch)) + require.Len(t, fake.headers, 2, "the SDK retried the put whose answer was lost") + require.Len(t, fake.objects, 1) +} + +func TestWriteAuditBatchRefusals(t *testing.T) { + archive, fake := fakeArchive(t) + ctx := context.Background() + require.ErrorContains(t, archive.WriteAuditBatch(ctx, batchOf(t, "batch-3")), "empty batch") + + bad := batchOf(t, "../escape", business.RetentionContent) + require.ErrorContains(t, archive.WriteAuditBatch(ctx, bad), "plain path segments") + + fake.refuse = http.StatusForbidden + err := archive.WriteAuditBatch(ctx, batchOf(t, "batch-4", business.RetentionContent)) + require.ErrorContains(t, err, "AccessDenied", "any other refusal fails the delivery, which the relay retries") + require.Empty(t, fake.objects) + + _, err = New(nil, "bucket") + require.Error(t, err) + _, err = newArchive(&s3.Client{}, " ") + require.Error(t, err) +} + +func TestNewFromEnvironmentNeedsARegion(t *testing.T) { + t.Setenv("AWS_REGION", "") + t.Setenv("AWS_DEFAULT_REGION", "") + t.Setenv("AWS_CONFIG_FILE", t.TempDir()+"/absent") + t.Setenv("AWS_PROFILE", "") + _, err := NewFromEnvironment(context.Background(), "audit-archive") + require.ErrorContains(t, err, "AWS_REGION") +} diff --git a/module/services/accounts/code/pkg/business/audit_store.go b/module/services/accounts/code/pkg/business/audit_store.go index 6a1aba522..57f4f5685 100644 --- a/module/services/accounts/code/pkg/business/audit_store.go +++ b/module/services/accounts/code/pkg/business/audit_store.go @@ -31,6 +31,9 @@ const ( // AuditSinkBigQuery is a swap value: BigQuery is the store of record and // Postgres keeps only the transactional queue. AuditSinkBigQuery AuditSinkMode = "bigquery" + // AuditSinkClickHouse is a swap value: ClickHouse is the store of record and + // Postgres keeps only the transactional queue. + AuditSinkClickHouse AuditSinkMode = "clickhouse" ) // ParseAuditSinkMode reads an AUDIT_SINK value. Empty is the default. @@ -42,19 +45,21 @@ func ParseAuditSinkMode(raw string) (AuditSinkMode, error) { return AuditSinkBoth, nil case string(AuditSinkBigQuery): return AuditSinkBigQuery, nil + case string(AuditSinkClickHouse): + return AuditSinkClickHouse, nil case "external": return "", errors.New("AUDIT_SINK=external is not permitted: no external destination can join the transaction " + "a change commits in, so it cannot receive the change's record atomically; use AUDIT_SINK=both to copy " + - "records to an HTTP endpoint, or a swap value (bigquery), which commits each record to a Postgres queue first") + "records to an HTTP endpoint, or a swap value (bigquery or clickhouse), which commits each record to a Postgres queue first") default: - return "", fmt.Errorf("AUDIT_SINK must be %s, %s or %s", AuditSinkPostgres, AuditSinkBoth, AuditSinkBigQuery) + return "", fmt.Errorf("AUDIT_SINK must be %s, %s, %s or %s", AuditSinkPostgres, AuditSinkBoth, AuditSinkBigQuery, AuditSinkClickHouse) } } // Swaps reports whether the mode makes a warehouse the audit store of record, // so the emitter writes the queue instead of audit_events. func (m AuditSinkMode) Swaps() bool { - return m == AuditSinkBigQuery + return m == AuditSinkBigQuery || m == AuditSinkClickHouse } // AuditRecorder writes the record of one event on the caller's transaction and diff --git a/module/services/accounts/code/pkg/business/audit_store_parity_test.go b/module/services/accounts/code/pkg/business/audit_store_parity_test.go index 34bca2ed9..789d57993 100644 --- a/module/services/accounts/code/pkg/business/audit_store_parity_test.go +++ b/module/services/accounts/code/pkg/business/audit_store_parity_test.go @@ -13,6 +13,7 @@ import ( "accounts/pkg/auditstore/bigqueryfake" "accounts/pkg/auditstore/bigquerystore" + "accounts/pkg/auditstore/clickhousestore" "accounts/pkg/auth" "accounts/pkg/business" gen "accounts/pkg/gen/saas/accounts/v1" @@ -20,6 +21,7 @@ import ( "cloud.google.com/go/bigquery" bqstorage "cloud.google.com/go/bigquery/storage/apiv1" + "github.com/ClickHouse/clickhouse-go/v2" "github.com/google/uuid" "github.com/stretchr/testify/require" "google.golang.org/api/iterator" @@ -63,6 +65,7 @@ type parityStore struct { var parityStores = map[string]func(t *testing.T) *parityStore{ "bigquery": newBigQueryParityStore, "bigquery-emulator": newBigQueryEmulatorParityStore, + "clickhouse": newClickHouseParityStore, } // bigQueryAppender appends a batch's rows to the fake, as the BigQuery writer @@ -170,6 +173,58 @@ func newBigQueryEmulatorParityStore(t *testing.T) *parityStore { } } +// newClickHouseParityStore is the ClickHouse store on a server named by +// AUDIT_CLICKHOUSE_TEST_DSN, in a database created for this run (the store +// never creates one; the deployment does), written through the store's own +// batched inserts. +func newClickHouseParityStore(t *testing.T) *parityStore { + dsn := strings.TrimSpace(os.Getenv("AUDIT_CLICKHOUSE_TEST_DSN")) + if dsn == "" { + t.Log("clickhouse: skipped; set AUDIT_CLICKHOUSE_TEST_DSN=clickhouse://:@:9000/default " + + "to run the suite against a ClickHouse server, e.g. docker run -p 9000:9000 clickhouse/clickhouse-server") + return nil + } + options, err := clickhouse.ParseDSN(dsn) + require.NoError(t, err) + admin, err := clickhouse.Open(options) + require.NoError(t, err) + t.Cleanup(func() { _ = admin.Close() }) + database := "audit_parity_" + strings.ReplaceAll(uuid.NewString(), "-", "")[:12] + require.NoError(t, admin.Exec(testCtx, "CREATE DATABASE "+database)) + t.Cleanup(func() { _ = admin.Exec(context.Background(), "DROP DATABASE IF EXISTS "+database) }) + + scoped := *options + scoped.Auth.Database = database + conn, err := clickhouse.Open(&scoped) + require.NoError(t, err) + t.Cleanup(func() { _ = conn.Close() }) + store, err := clickhousestore.New(conn, clickhousestore.Config{ + Database: database, + DeploymentID: parityDeployment, + EventsRetention: 2555 * 24 * time.Hour, + ContentDetailRetention: 30 * 24 * time.Hour, + }) + require.NoError(t, err) + require.NoError(t, store.Ensure(testCtx)) + return &parityStore{ + writer: store, + store: store, + eventIDs: func(t *testing.T) []string { + rows, err := conn.Query(testCtx, "SELECT event_id FROM "+clickhousestore.EventsTable) + require.NoError(t, err) + defer func() { _ = rows.Close() }() + var ids []string + for rows.Next() { + var id string + require.NoError(t, rows.Scan(&id)) + ids = append(ids, id) + } + require.NoError(t, rows.Err()) + return ids + }, + } +} + // fanOut appends every batch to every store under test, and remembers it. type fanOut struct { writers []business.AuditStoreWriter @@ -432,6 +487,8 @@ func runAuditStoreParity(t *testing.T, f *parityFixture, under *parityStore) { "payload": {OrgID: f.org, PayloadContains: map[string]any{"boundary": "b-1"}}, "payload number": {OrgID: f.org, PayloadContains: map[string]any{"result_count": 3.0}}, "payload nested": {OrgID: f.org, PayloadContains: map[string]any{"nested": map[string]any{"a": "q"}, "tags": []any{"x"}}}, + "payload pair": {OrgID: f.org, PayloadContains: map[string]any{"boundary": "b-2", "outcome": "denied"}}, + "payload absent": {OrgID: f.org, PayloadContains: map[string]any{"outcome": nil}}, "window": {OrgID: f.org, From: &week, To: &recent}, "platform for actor": {ActorID: f.actor}, "other organization": {OrgID: f.otherOrg}, @@ -480,6 +537,26 @@ func runAuditStoreParity(t *testing.T, f *parityFixture, under *parityStore) { {Op: "count_distinct", Field: "resource"}, }}, "nested payload dimension": {GroupBy: []string{"payload:tags", "payload:nested", "payload:label"}}, + "numeric payload dimension": {GroupBy: []string{"payload:result_count"}, Metrics: []business.AuditMetric{ + {Op: "count_distinct", Field: "payload:result_count"}, + }}, + "fractional payload dimension": {GroupBy: []string{"payload:duration_ms"}}, + "metrics by type": {GroupBy: []string{"event_type"}, Metrics: []business.AuditMetric{ + {Op: "count", Alias: "events"}, + {Op: "sum", Field: "payload:duration_ms"}, + {Op: "avg", Field: "payload:result_count"}, + {Op: "min", Field: "payload:duration_ms"}, + {Op: "max", Field: "payload:result_count"}, + {Op: "percentile", Field: "payload:duration_ms", Percentile: 0.25, Alias: "p25"}, + {Op: "count_distinct", Field: "payload:outcome"}, + {Op: "count_distinct", Field: "category"}, + {Op: "count_distinct", Field: "actor_id"}, + }, Derived: []business.AuditDerivedMetric{ + {Alias: "duration_per_event", Numerator: "sum_duration_ms", Denominator: "events"}, + }}, + "category by month": {GroupBy: []string{"category", "time"}, Bucket: "month", Metrics: []business.AuditMetric{ + {Op: "count_distinct", Field: "resource_id"}, + }}, "metrics": {GroupBy: []string{"payload:boundary"}, Metrics: []business.AuditMetric{ {Op: "count", Alias: "reads"}, {Op: "sum", Field: "payload:duration_ms"}, diff --git a/module/services/accounts/code/pkg/business/audit_store_test.go b/module/services/accounts/code/pkg/business/audit_store_test.go index 5defac587..8e802aa75 100644 --- a/module/services/accounts/code/pkg/business/audit_store_test.go +++ b/module/services/accounts/code/pkg/business/audit_store_test.go @@ -17,19 +17,21 @@ func TestParseAuditSinkMode(t *testing.T) { " Postgres ": AuditSinkPostgres, "both": AuditSinkBoth, "BigQuery": AuditSinkBigQuery, + "clickhouse": AuditSinkClickHouse, } { got, err := ParseAuditSinkMode(raw) require.NoError(t, err, raw) require.Equal(t, want, got, raw) } require.True(t, AuditSinkBigQuery.Swaps()) + require.True(t, AuditSinkClickHouse.Swaps()) require.False(t, AuditSinkPostgres.Swaps()) require.False(t, AuditSinkBoth.Swaps(), "the tee keeps audit_events as the store of record") _, err := ParseAuditSinkMode("external") require.ErrorContains(t, err, "not permitted") - _, err = ParseAuditSinkMode("clickhouse") - require.ErrorContains(t, err, "must be postgres, both or bigquery", "a swap value with no adapter yet is refused, not ignored") + _, err = ParseAuditSinkMode("kafka") + require.ErrorContains(t, err, "must be postgres, both, bigquery or clickhouse", "an unknown value is refused, not ignored") } func TestCanonicalAuditDetails(t *testing.T) { diff --git a/module/services/accounts/code/work.go b/module/services/accounts/code/work.go index b5e8a21b2..863a3034b 100644 --- a/module/services/accounts/code/work.go +++ b/module/services/accounts/code/work.go @@ -535,9 +535,9 @@ func doWork(ctx context.Context) (Clean, error) { // AUDIT_SINK=both additionally tees each org-scoped audit row to an external // warehouse, fed asynchronously from the durable outbox — Postgres stays the // atomic source of truth (an external destination cannot join its tx). - // AUDIT_SINK=bigquery swaps the store of record (ADR 0009): the record is a - // queue row on that same transaction, and the audit relay delivers it to - // BigQuery and the archive afterwards. + // AUDIT_SINK=bigquery or clickhouse swaps the store of record (ADR 0009): + // the record is a queue row on that same transaction, and the audit relay + // delivers it to the warehouse and the archive afterwards. auditSink, err := configuredAuditSink() if err != nil { return nil, err From 25fc80d422c4ef89da2c2fd366adf62e9fd65b79 Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Sat, 3 Oct 2026 21:24:19 -0400 Subject: [PATCH 14/69] docs(adr): record ClickHouse audit settings --- module/docs/adr/0009-audit-store-swap.md | 27 ++++++++++++++---------- 1 file changed, 16 insertions(+), 11 deletions(-) diff --git a/module/docs/adr/0009-audit-store-swap.md b/module/docs/adr/0009-audit-store-swap.md index 614fa89b3..d86918dcb 100644 --- a/module/docs/adr/0009-audit-store-swap.md +++ b/module/docs/adr/0009-audit-store-swap.md @@ -58,13 +58,16 @@ Postgres transaction. A deployment that selects a swap value sets, beside `AUDIT_SINK`: the warehouse (`AUDIT_BIGQUERY_PROJECT` and `AUDIT_BIGQUERY_DATASET` for - `bigquery`), `AUDIT_ARCHIVE_URL` — the locked archive as a URL whose scheme - picks the archive writer, `gs://` for GCS, any scheme without a - writer refused at startup — `AUDIT_CONTENT_RETENTION_DAYS` (the details - window of item 5) and `AUDIT_DEPLOYMENT_ID`, stamped on every record. A - missing setting fails startup. Credentials are the platform's ambient - identity (on Kubernetes, the pod's workload identity), never a key in - configuration. + `bigquery`; `AUDIT_CLICKHOUSE_DSN` and `AUDIT_EVENTS_RETENTION_DAYS` for + `clickhouse`), `AUDIT_ARCHIVE_URL` — the locked archive as a URL whose scheme + picks the archive writer (`gs://` for GCS or `s3://` for S3; + any scheme without a writer is refused at startup) — + `AUDIT_CONTENT_RETENTION_DAYS` (the details window of item 5) and + `AUDIT_DEPLOYMENT_ID`, stamped on every record. ClickHouse may additionally + set `AUDIT_CLICKHOUSE_CLUSTER` for replicated tables. The events retention + must be at least the content-detail window. A missing required setting + fails startup. Credentials are the platform's ambient identity (on + Kubernetes, the pod's workload identity), never a key in configuration. 2. **Postgres keeps only the transactional queue.** `EmitTx` still writes inside the caller's transaction (and `Emit` inside its own, for the observational @@ -171,8 +174,9 @@ Postgres transaction. A **locked object-storage archive** holds the compliance copy, one object per relay batch: full for security-class events, content-free (envelope plus hash) for content-class events. Locked means write-once under a retention - lock the operator cannot shorten: GCS Bucket Lock, S3 Object Lock, Azure Blob - immutability policies, or an object-lock-capable store on-premises. + lock the operator cannot shorten. The first two writers target GCS Bucket + Lock and S3 Object Lock. Azure Blob immutability policies still need a + writer; an on-premises S3-compatible lock must be qualified before use. Archive objects are named per batch. A retried batch writes a new object, so the archive may hold an event more than once. Every archive reader — @@ -186,8 +190,9 @@ Postgres transaction. - Inserts are batched — the relay's batches, `async_insert`, or both. A single-row insert creates a data part per insert and ends in too-many-parts errors. - - Tables are replicated (`Replicated*MergeTree`), coordinated by ClickHouse - Keeper. + - A single-node deployment uses MergeTree. When + `AUDIT_CLICKHOUSE_CLUSTER` is set, tables use `ReplicatedMergeTree` with + ClickHouse Keeper; the cluster must exist before the adapter starts. - Tables are partitioned by month, and table TTL expires each tier at its window. - Growing past one shard is a planned operation, not an automatic one — or From f4ad4f99c93253f5f60b5bab72c824dfd8669e10 Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Mon, 5 Oct 2026 11:59:58 -0400 Subject: [PATCH 15/69] feat(audit): ship bounded qualification and content-bound history receipts --- module/AUDIT_OPERATIONS.md | 79 ++++ module/services/accounts/AGENTS.md | 8 + .../services/accounts/code/audit_commands.go | 21 + .../code/cmd/audit-history-copy/main.go | 180 +------ .../accounts/code/cmd/audit-qualify/main.go | 9 + .../accounts/code/pkg/auditops/common.go | 68 +++ .../accounts/code/pkg/auditops/history.go | 269 +++++++++++ .../auditops/history_test.go} | 36 +- .../code/pkg/auditops/qualification.go | 443 ++++++++++++++++++ .../code/pkg/auditops/qualification_test.go | 228 +++++++++ .../code/pkg/business/audit_history.go | 63 ++- .../code/pkg/business/audit_history_test.go | 65 ++- .../pkg/infra/audit_qualification_test.go | 61 +++ .../pkg/infra/postgres_audit_history_test.go | 2 +- .../code/pkg/infra/postgres_webhooks_test.go | 5 +- .../testdata/control_plane_call_sites.txt | 2 + ...down.sql => 18_audit_event_queue.down.sql} | 0 ...eue.up.sql => 18_audit_event_queue.up.sql} | 0 ...audit_event_type_retention_class.down.sql} | 0 ...9_audit_event_type_retention_class.up.sql} | 0 20 files changed, 1345 insertions(+), 194 deletions(-) create mode 100644 module/AUDIT_OPERATIONS.md create mode 100644 module/services/accounts/code/audit_commands.go create mode 100644 module/services/accounts/code/cmd/audit-qualify/main.go create mode 100644 module/services/accounts/code/pkg/auditops/common.go create mode 100644 module/services/accounts/code/pkg/auditops/history.go rename module/services/accounts/code/{cmd/audit-history-copy/main_test.go => pkg/auditops/history_test.go} (70%) create mode 100644 module/services/accounts/code/pkg/auditops/qualification.go create mode 100644 module/services/accounts/code/pkg/auditops/qualification_test.go create mode 100644 module/services/accounts/code/pkg/infra/audit_qualification_test.go rename module/services/store/migrations/{17_audit_event_queue.down.sql => 18_audit_event_queue.down.sql} (100%) rename module/services/store/migrations/{17_audit_event_queue.up.sql => 18_audit_event_queue.up.sql} (100%) rename module/services/store/migrations/{18_audit_event_type_retention_class.down.sql => 19_audit_event_type_retention_class.down.sql} (100%) rename module/services/store/migrations/{18_audit_event_type_retention_class.up.sql => 19_audit_event_type_retention_class.up.sql} (100%) diff --git a/module/AUDIT_OPERATIONS.md b/module/AUDIT_OPERATIONS.md new file mode 100644 index 000000000..684a01694 --- /dev/null +++ b/module/AUDIT_OPERATIONS.md @@ -0,0 +1,79 @@ +# Audit warehouse operator commands + +The accounts image contains its audit commands in the same executable as the +service. Use `./app audit-history-copy` and `./app audit-qualify` from `/app`. +The standalone Go commands under accounts' `code/cmd` use the same implementation. +Generated service entry points and build recipes are not modified. + +Commands use the runtime's injected Codefly database capabilities and `AUDIT_*` +configuration. They run only under `AUDIT_SINK=bigquery` or `clickhouse` and never +accept database URLs in arguments. `DATABASE_URL` is an explicit integration-test +environment override. Operator output contains no credentials, payloads or raw +provider errors. A destination failure returns a safe `error_code` and a nonzero +exit status. + +## Capabilities and destination preflight + +`./app audit-history-copy -capabilities-json` (also supported by `audit-qualify`) +returns `codefly/audit-tools/v1`, the history and qualification report schemas, +`expected_partitions_sha256: true`, and a nonsecret `configuration` map. It does +not connect or mutate. An operator must verify that map against the intended +warehouse, archive, deployment and content-retention policy before running a +command. The map deliberately excludes a ClickHouse DSN and all database secrets. + +## History copy, verification and removal + +Run `./app audit-history-copy -json -through YYYY-MM` to copy through the selected +month into the warehouse and archive. Copy is resumable; existing copies are +verified before missing rows are appended. `-verify-only` writes nothing. + +A `codefly/audit-history/v1` receipt includes `deployment_id`, `sink`, +`configuration`, `through`, `observed_at`, `verified`, `partitions_sha256`, +`partitions`, `dropped`, `cutoff`, and `error_code` (empty on success). Each +partition names `name`, `from`, `to`, per-organization `orgs` counts, +`events_sha256`, and `failures`. Payloads and provider error messages are excluded. + +`partitions_sha256` binds the deployment, cutoff, partition names and ranges, +per-organization counts, and a canonical source stream of every event envelope, +retention class and details hash. Run timing and the number copied by a particular +attempt are excluded, so successful copy and verification runs agree. Equal counts +with changed event content produce a different authorization digest. + +Removing partitions requires an explicit month, a verification-only pass, and +`-confirm-drop -expected-partitions-sha256 `. +The copier verifies again and compares the approved digest before any DDL. It also +rechecks the source partition set and counts in the control-plane drop transaction. +An unverified copy, missing confirmation or changed digest drops nothing. The +operator must separately establish complete cutover acceptance, authenticate the +receipt and pin the running image, destination and pod identity before removal. + +## Bounded live qualification + +A prepare phase is an explicit audit mutation. Run `./app audit-qualify -json +-prepare -run-id -organization -organization +`. It uses `RecordTx` under the queued mode to commit six +fixtures together: security and content events for both organizations and the +platform. No domain events or webhook jobs are published. If an organization is +absent, or Postgres receives rows in `audit_events` in the qualification window, +the fixture transaction aborts. Use a fresh run ID for every prepare attempt. + +Verify with `-verify` and the same run ID and organizations, plus `-window-from` +and `-window-to` copied from the prepare receipt. `-timeout` defaults to two minutes +and cannot exceed five. Verify emits nothing and waits for the service's normal +relay; it does not create a second worker or interfere with a running relay. + +The `codefly/audit-qualification/v1` receipt includes the destination/configuration, +run ID, observation time, five-minute event window, six event IDs/scopes/classes +and content hashes, and a `checks` map with `passed`, `failed` or `pending` values. +Successful probes verify every stored copy's envelope, class and details hash; +scoped list, aggregation, export and readable-source results; organization-scope +refusals; zero Postgres audit rows in the window; and an actually observed empty +cell queue. Warehouse scans refuse more than 20,000 events in the bounded window. + +`deployment_isolation`, archive object/lock enforcement, gateway authorization, +controlled relay interruption/recovery and alert delivery remain **pending** until +separate live evidence proves them. Seeing fixtures in the warehouse does not +prove those checks. A zero exit status says the command's probes passed; it does +not mean the full cutover is accepted or that history may be removed. Archive +observers should scan through `observed_at` as well as the event window, because +relay retries can compose archive objects after their events occurred. diff --git a/module/services/accounts/AGENTS.md b/module/services/accounts/AGENTS.md index 95cbd9555..99d09e066 100644 --- a/module/services/accounts/AGENTS.md +++ b/module/services/accounts/AGENTS.md @@ -398,6 +398,14 @@ viewer may read out. which `Subscribe` refuses to a module principal, so a consumer chooses its own refresh — per read, or cached against a TTL it accepts. +## Audit warehouse operator interface + +The deployed accounts executable owns `audit-history-copy` and `audit-qualify` +subcommands through authored init dispatch. Their sanitized receipts, production +database capabilities and content-bound removal guard are documented in +[../../AUDIT_OPERATIONS.md](../../AUDIT_OPERATIONS.md). A qualification receipt +retains pending checks for every live behavior it did not observe. + ## Mesh reachability is the composition's to grant A composed module reaches the internal tier on the named `authority` endpoint diff --git a/module/services/accounts/code/audit_commands.go b/module/services/accounts/code/audit_commands.go new file mode 100644 index 000000000..0c54094a6 --- /dev/null +++ b/module/services/accounts/code/audit_commands.go @@ -0,0 +1,21 @@ +package main + +import ( + "accounts/pkg/auditops" + "os" +) + +// Operator commands run before the generated service entry point. Keeping +// dispatch in authored code makes them part of the same immutable image and +// identity as the service without modifying agent-generated main.go/builders. +func init() { + if len(os.Args) < 2 { + return + } + switch os.Args[1] { + case "audit-history-copy": + os.Exit(auditops.RunHistory(os.Args[2:], os.Getenv, os.Stdout, os.Stderr)) + case "audit-qualify": + os.Exit(auditops.RunQualification(os.Args[2:], os.Getenv, os.Stdout, os.Stderr)) + } +} diff --git a/module/services/accounts/code/cmd/audit-history-copy/main.go b/module/services/accounts/code/cmd/audit-history-copy/main.go index d7ecf80cf..a609b858d 100644 --- a/module/services/accounts/code/cmd/audit-history-copy/main.go +++ b/module/services/accounts/code/cmd/audit-history-copy/main.go @@ -1,183 +1,9 @@ -// Command audit-history-copy is the one-time history copy of a deployment -// switching from AUDIT_SINK=postgres to a swap value (ADR 0009, item 7). It -// copies the audit_events rows the deployment already holds into the store of -// record and the locked archive — each event classified, serialized and hashed -// exactly as the relay does a live one — and verifies the copy by reading it -// back: every row present, every copy identical, per organization. Only with -// -confirm-drop, and only after this run verified every copied partition, it -// drops the copied monthly partitions, the way retention drops them; no row is -// ever deleted. -// -// Usage: -// -// audit-history-copy [-database-url "$DATABASE_URL"] [-through YYYY-MM] [-verify-only] [-confirm-drop] [-batch-size N] -// -// It reads the same AUDIT_* settings as the accounts service and runs only -// under a swap value: run it with the service's own environment, after the -// service has switched, so nothing writes audit_events any more. The -// connection principal must be a member of app_control_plane, the authority -// retention runs under. -// -// It is resumable: a run reads back what the store holds before writing, so a -// run after an interrupted one copies only what is missing. -verify-only -// copies nothing, for a re-check once the store has caught up. -// -// Exit status: 0 done; 1 failed; 2 usage; 3 verification failed, nothing -// dropped. +// Command audit-history-copy uses the same operator interface shipped in accounts. package main import ( - "context" - "errors" - "flag" - "fmt" - "io" + "accounts/pkg/auditops" "os" - "sort" - "time" - - "accounts/pkg/auditstore/auditsink" - "accounts/pkg/business" - "accounts/pkg/infra" ) -func main() { - os.Exit(run(os.Args[1:], os.Getenv, os.Stdout, os.Stderr)) -} - -// Writes to the process's stdout/stderr have no meaningful recovery. -func line(w io.Writer, a ...any) { _, _ = fmt.Fprintln(w, a...) } - -// options is a parsed command line. -type options struct { - databaseURL string - through *time.Time - batchSize int - copyOptions business.AuditHistoryCopyOptions -} - -// parse reads the command line and the sink settings, refusing everything a -// run cannot start from. -func parse(args []string, getenv func(string) string, stderr io.Writer) (options, *auditsink.Swap, int) { - fs := flag.NewFlagSet("audit-history-copy", flag.ContinueOnError) - fs.SetOutput(stderr) - databaseURL := fs.String("database-url", getenv("DATABASE_URL"), "Postgres connection URL (defaults to $DATABASE_URL)") - through := fs.String("through", "", "copy only the partitions up to and including this month, YYYY-MM (default every partition)") - verifyOnly := fs.Bool("verify-only", false, "copy nothing; verify what the store already holds") - confirmDrop := fs.Bool("confirm-drop", false, "after a verification pass, drop the copied partitions") - batchSize := fs.Int("batch-size", business.DefaultAuditRelayBatchSize, "events per copied batch") - if err := fs.Parse(args); err != nil { - return options{}, nil, 2 - } - if fs.NArg() > 0 { - line(stderr, "audit-history-copy: unexpected arguments:", fs.Args()) - return options{}, nil, 2 - } - opts := options{ - databaseURL: *databaseURL, - batchSize: *batchSize, - copyOptions: business.AuditHistoryCopyOptions{VerifyOnly: *verifyOnly, ConfirmDrop: *confirmDrop}, - } - if *through != "" { - monthStart, err := time.Parse("2006-01", *through) - if err != nil { - line(stderr, "audit-history-copy: -through must be a month, YYYY-MM; got", *through) - return options{}, nil, 2 - } - end := monthStart.AddDate(0, 1, 0) - opts.through = &end - } - if opts.databaseURL == "" { - line(stderr, "audit-history-copy: -database-url (or $DATABASE_URL) is required") - return options{}, nil, 2 - } - sink, err := auditsink.Load(getenv) - if err != nil { - line(stderr, "audit-history-copy:", err) - return options{}, nil, 1 - } - if sink.Swap == nil { - line(stderr, fmt.Sprintf("audit-history-copy: AUDIT_SINK is %s; the history copy runs only under a swap value, "+ - "after the service has switched, so nothing writes audit_events any more", sink.Mode)) - return options{}, nil, 1 - } - return opts, sink.Swap, 0 -} - -func run(args []string, getenv func(string) string, stdout, stderr io.Writer) int { - opts, swap, code := parse(args, getenv, stderr) - if swap == nil { - return code - } - ctx := context.Background() - store, err := infra.NewPostgresStoreFromURL(ctx, opts.databaseURL) - if err != nil { - line(stderr, "audit-history-copy:", err) - return 1 - } - defer store.Close() - opened, err := auditsink.Open(ctx, swap) - if err != nil { - line(stderr, "audit-history-copy:", err) - return 1 - } - defer opened.Close() - - copier, err := business.NewAuditHistoryCopy(business.AuditHistoryCopyConfig{ - Source: store, - Store: opened.Store, - Archive: opened.Archive, - ReadBack: opened.History, - Types: store, - DeploymentID: swap.DeploymentID, - ContentRetention: swap.ContentRetention, - BatchSize: opts.batchSize, - Through: opts.through, - Progress: func(message string) { line(stdout, message) }, - }) - if err != nil { - line(stderr, "audit-history-copy:", err) - return 2 - } - report, err := copier.Run(ctx, opts.copyOptions) - return summarize(stdout, stderr, report, err) -} - -// summarize prints what the run found and maps it to the exit status. Pure over -// its inputs so the mapping is testable without a database or a warehouse. -func summarize(stdout, stderr io.Writer, report business.AuditHistoryReport, err error) int { - for _, partition := range report.Partitions { - orgs := make([]string, 0, len(partition.Orgs)) - for org := range partition.Orgs { - orgs = append(orgs, org) - } - sort.Strings(orgs) - for _, org := range orgs { - count := partition.Orgs[org] - name := org - if name == "" { - name = "(platform)" - } - line(stdout, fmt.Sprintf(" %s %s: %d in Postgres, %d verified", partition.Partition.Name, name, count.Postgres, count.Verified)) - } - for _, problem := range partition.Problems { - line(stderr, fmt.Sprintf(" %s: %s", partition.Partition.Name, problem)) - } - if hidden := partition.Failures - len(partition.Problems); hidden > 0 { - line(stderr, fmt.Sprintf(" %s: and %d more", partition.Partition.Name, hidden)) - } - } - switch { - case errors.Is(err, business.ErrAuditHistoryUnverified): - line(stderr, "audit-history-copy:", err) - return 3 - case err != nil: - line(stderr, "audit-history-copy:", err) - return 1 - case report.Dropped > 0: - line(stdout, fmt.Sprintf("verified; dropped %d partitions ending at or before %s", report.Dropped, report.Cutoff.UTC().Format(time.RFC3339))) - default: - line(stdout, "verified") - } - return 0 -} +func main() { os.Exit(auditops.RunHistory(os.Args[1:], os.Getenv, os.Stdout, os.Stderr)) } diff --git a/module/services/accounts/code/cmd/audit-qualify/main.go b/module/services/accounts/code/cmd/audit-qualify/main.go new file mode 100644 index 000000000..de0cc4daa --- /dev/null +++ b/module/services/accounts/code/cmd/audit-qualify/main.go @@ -0,0 +1,9 @@ +// Command audit-qualify is also shipped as `accounts audit-qualify`. +package main + +import ( + "accounts/pkg/auditops" + "os" +) + +func main() { os.Exit(auditops.RunQualification(os.Args[1:], os.Getenv, os.Stdout, os.Stderr)) } diff --git a/module/services/accounts/code/pkg/auditops/common.go b/module/services/accounts/code/pkg/auditops/common.go new file mode 100644 index 000000000..bd988f508 --- /dev/null +++ b/module/services/accounts/code/pkg/auditops/common.go @@ -0,0 +1,68 @@ +// Package auditops owns the audit operator commands shipped in the accounts +// executable. Commands use the deployed service's injected identity and +// configuration; they never accept database credentials in command arguments. +package auditops + +import ( + "accounts/pkg/auditstore/auditsink" + "accounts/pkg/infra" + "context" + "encoding/json" + "io" + "strconv" + "time" + + "github.com/codefly-dev/core/wool" + codefly "github.com/codefly-dev/sdk-go" +) + +// These commands report safe error codes. Provider logs may include URLs or +// credentials, so their logger is deliberately disabled during operator work. +type quietLog struct{} + +func (quietLog) Process(*wool.Log) {} +func commandContext(ctx context.Context) (context.Context, error) { + provider, err := codefly.Init(ctx) + if err != nil { + return nil, err + } + return provider.WithLogger(quietLog{}).Inject(ctx), nil +} +func openStore(ctx context.Context, testURL string) (*infra.PostgresStore, error) { + if testURL != "" { + return infra.NewPostgresStoreFromURL(ctx, testURL) + } + return infra.NewPostgresStore(ctx) +} +func safeConfiguration(swap *auditsink.Swap) map[string]string { + out := map[string]string{ + "AUDIT_SINK": string(swap.Mode), "AUDIT_DEPLOYMENT_ID": swap.DeploymentID, + "AUDIT_ARCHIVE_URL": swap.Archive.Scheme + "://" + swap.Archive.Bucket, + "AUDIT_CONTENT_RETENTION_DAYS": strconv.Itoa(int(swap.ContentRetention / (24 * time.Hour))), + } + if swap.BigQuery != nil { + out["AUDIT_BIGQUERY_PROJECT"] = swap.BigQuery.Project + out["AUDIT_BIGQUERY_DATASET"] = swap.BigQuery.Dataset + } + if swap.ClickHouse != nil { + out["AUDIT_EVENTS_RETENTION_DAYS"] = strconv.Itoa(int(swap.ClickHouse.EventsRetention / (24 * time.Hour))) + } + return out +} +func writeCapabilities(out io.Writer, getenv func(string) string) int { + cfg, err := auditsink.Load(getenv) + if err != nil || cfg.Swap == nil { + return 1 + } + body := struct { + Schema string `json:"schema"` + HistoryReport string `json:"history_report"` + QualificationReport string `json:"qualification_report"` + ExpectedPartitionsSHA256 bool `json:"expected_partitions_sha256"` + Configuration map[string]string `json:"configuration"` + }{"codefly/audit-tools/v1", "codefly/audit-history/v1", "codefly/audit-qualification/v1", true, safeConfiguration(cfg.Swap)} + if json.NewEncoder(out).Encode(body) != nil { + return 1 + } + return 0 +} diff --git a/module/services/accounts/code/pkg/auditops/history.go b/module/services/accounts/code/pkg/auditops/history.go new file mode 100644 index 000000000..4431c3755 --- /dev/null +++ b/module/services/accounts/code/pkg/auditops/history.go @@ -0,0 +1,269 @@ +// Command audit-history-copy is the one-time history copy of a deployment +// switching from AUDIT_SINK=postgres to a swap value (ADR 0009, item 7). It +// copies the audit_events rows the deployment already holds into the store of +// record and the locked archive — each event classified, serialized and hashed +// exactly as the relay does a live one — and verifies the copy by reading it +// back: every row present, every copy identical, per organization. Only with +// -confirm-drop, and only after this run verified every copied partition, it +// drops the copied monthly partitions, the way retention drops them; no row is +// ever deleted. +// +// Usage: +// +// audit-history-copy [-through YYYY-MM] [-verify-only] [-confirm-drop -expected-partitions-sha256 HEX] [-json] +// +// It reads the same AUDIT_* settings as the accounts service and runs only +// under a swap value: run it with the service's own environment, after the +// service has switched, so nothing writes audit_events any more. The +// connection principal must be a member of app_control_plane, the authority +// retention runs under. +// +// It is resumable: a run reads back what the store holds before writing, so a +// run after an interrupted one copies only what is missing. -verify-only +// copies nothing, for a re-check once the store has caught up. +// +// Exit status: 0 done; 1 failed; 2 usage; 3 verification failed, nothing +// dropped. +package auditops + +import ( + "context" + "encoding/json" + "errors" + "flag" + "fmt" + "io" + "regexp" + "sort" + "time" + + "accounts/pkg/auditstore/auditsink" + "accounts/pkg/business" +) + +// Writes to the process's stdout/stderr have no meaningful recovery. +func line(w io.Writer, a ...any) { _, _ = fmt.Fprintln(w, a...) } + +// options is a parsed command line. +type options struct { + databaseURL string + through *time.Time + batchSize int + copyOptions business.AuditHistoryCopyOptions + json bool + capabilities bool + throughMonth string +} + +// parse reads the command line and the sink settings, refusing everything a +// run cannot start from. +func parse(args []string, getenv func(string) string, stderr io.Writer) (options, *auditsink.Swap, int) { + fs := flag.NewFlagSet("audit-history-copy", flag.ContinueOnError) + fs.SetOutput(stderr) + databaseURL := getenv("DATABASE_URL") + machine := fs.Bool("json", false, "emit a sanitized machine receipt") + capabilities := fs.Bool("capabilities-json", false, "report supported command schemas and destination without connecting") + expected := fs.String("expected-partitions-sha256", "", "digest from prior verification, required for drop") + through := fs.String("through", "", "copy only the partitions up to and including this month, YYYY-MM (default every partition)") + verifyOnly := fs.Bool("verify-only", false, "copy nothing; verify what the store already holds") + confirmDrop := fs.Bool("confirm-drop", false, "after a verification pass, drop the copied partitions") + batchSize := fs.Int("batch-size", business.DefaultAuditRelayBatchSize, "events per copied batch") + if err := fs.Parse(args); err != nil { + return options{}, nil, 2 + } + if fs.NArg() > 0 { + line(stderr, "audit-history-copy: unexpected arguments:", fs.Args()) + return options{}, nil, 2 + } + opts := options{ + databaseURL: databaseURL, + json: *machine, capabilities: *capabilities, throughMonth: *through, + batchSize: *batchSize, + copyOptions: business.AuditHistoryCopyOptions{VerifyOnly: *verifyOnly, ConfirmDrop: *confirmDrop, ExpectedPartitionsSHA256: *expected}, + } + if *through != "" { + monthStart, err := time.Parse("2006-01", *through) + if err != nil { + line(stderr, "audit-history-copy: -through must be a month, YYYY-MM; got", *through) + return options{}, nil, 2 + } + end := monthStart.AddDate(0, 1, 0) + opts.through = &end + } + if *confirmDrop && (*through == "" || !*verifyOnly || !regexp.MustCompile(`^[a-f0-9]{64}$`).MatchString(*expected)) { + line(stderr, "audit-history-copy: drop requires -through YYYY-MM, -verify-only and -expected-partitions-sha256") + return options{}, nil, 2 + } + sink, err := auditsink.Load(getenv) + if err != nil { + line(stderr, "audit-history-copy: invalid audit destination configuration") + return options{}, nil, 1 + } + if sink.Swap == nil { + line(stderr, fmt.Sprintf("audit-history-copy: AUDIT_SINK is %s; the history copy runs only under a swap value, "+ + "after the service has switched, so nothing writes audit_events any more", sink.Mode)) + return options{}, nil, 1 + } + return opts, sink.Swap, 0 +} + +func RunHistory(args []string, getenv func(string) string, stdout, stderr io.Writer) int { + opts, swap, code := parse(args, getenv, stderr) + if swap == nil { + return code + } + if opts.capabilities { + return writeCapabilities(stdout, getenv) + } + ctx, err := commandContext(context.Background()) + if err != nil { + return historyFailure(stdout, stderr, opts, swap, "runtime_configuration_failed") + } + store, err := openStore(ctx, opts.databaseURL) + if err != nil { + return historyFailure(stdout, stderr, opts, swap, "operation_failed") + } + defer store.Close() + opened, err := auditsink.Open(ctx, swap) + if err != nil { + return historyFailure(stdout, stderr, opts, swap, "operation_failed") + } + defer opened.Close() + + copier, err := business.NewAuditHistoryCopy(business.AuditHistoryCopyConfig{ + Source: store, + Store: opened.Store, + Archive: opened.Archive, + ReadBack: opened.History, + Types: store, + DeploymentID: swap.DeploymentID, + ContentRetention: swap.ContentRetention, + BatchSize: opts.batchSize, + Through: opts.through, + Progress: func(message string) { + if !opts.json { + line(stdout, message) + } + }, + }) + if err != nil { + return historyFailure(stdout, stderr, opts, swap, "invalid_options") + } + report, err := copier.Run(ctx, opts.copyOptions) + if opts.json { + return writeHistory(stdout, opts, swap, report, err) + } + return summarize(stdout, stderr, report, err) +} + +// summarize prints what the run found and maps it to the exit status. Pure over +// its inputs so the mapping is testable without a database or a warehouse. +func summarize(stdout, stderr io.Writer, report business.AuditHistoryReport, err error) int { + for _, partition := range report.Partitions { + orgs := make([]string, 0, len(partition.Orgs)) + for org := range partition.Orgs { + orgs = append(orgs, org) + } + sort.Strings(orgs) + for _, org := range orgs { + count := partition.Orgs[org] + name := org + if name == "" { + name = "(platform)" + } + line(stdout, fmt.Sprintf(" %s %s: %d in Postgres, %d verified", partition.Partition.Name, name, count.Postgres, count.Verified)) + } + for _, problem := range partition.Problems { + line(stderr, fmt.Sprintf(" %s: %s", partition.Partition.Name, problem)) + } + if hidden := partition.Failures - len(partition.Problems); hidden > 0 { + line(stderr, fmt.Sprintf(" %s: and %d more", partition.Partition.Name, hidden)) + } + } + switch { + case errors.Is(err, business.ErrAuditHistoryUnverified): + line(stderr, "audit-history-copy:", err) + return 3 + case err != nil: + line(stderr, "audit-history-copy: operation_failed") + return 1 + case report.Dropped > 0: + line(stdout, fmt.Sprintf("verified; dropped %d partitions ending at or before %s", report.Dropped, report.Cutoff.UTC().Format(time.RFC3339))) + default: + line(stdout, "verified") + } + return 0 +} + +// HistoryReceipt is the public machine evidence. It excludes payloads, secrets +// and provider errors; the digest still binds every source event and hash. +type HistoryReceipt struct { + Schema string `json:"schema"` + DeploymentID string `json:"deployment_id"` + Sink string `json:"sink"` + Configuration map[string]string `json:"configuration"` + Through string `json:"through"` + ObservedAt time.Time `json:"observed_at"` + Verified bool `json:"verified"` + PartitionsSHA256 string `json:"partitions_sha256"` + Partitions []HistoryPartition `json:"partitions"` + Dropped []string `json:"dropped"` + Cutoff time.Time `json:"cutoff"` + ErrorCode string `json:"error_code"` +} +type HistoryPartition struct { + Name string `json:"name"` + From time.Time `json:"from"` + To time.Time `json:"to"` + Orgs []HistoryOrg `json:"orgs"` + Failures int `json:"failures"` + EventsSHA256 string `json:"events_sha256"` +} +type HistoryOrg struct { + OrgID string `json:"org_id"` + Postgres int64 `json:"postgres"` + Verified int64 `json:"verified"` +} + +func writeHistory(out io.Writer, opts options, swap *auditsink.Swap, report business.AuditHistoryReport, err error) int { + receipt := HistoryReceipt{Schema: "codefly/audit-history/v1", DeploymentID: swap.DeploymentID, Sink: string(swap.Mode), Configuration: safeConfiguration(swap), Through: opts.throughMonth, ObservedAt: time.Now().UTC(), Verified: report.Verified, PartitionsSHA256: report.PartitionsSHA256, Partitions: []HistoryPartition{}, Dropped: report.DroppedNames, Cutoff: report.Cutoff} + if receipt.Dropped == nil { + receipt.Dropped = []string{} + } + code := 0 + if err != nil { + code = 1 + receipt.ErrorCode = "operation_failed" + } + if errors.Is(err, business.ErrAuditHistoryUnverified) { + code = 3 + receipt.ErrorCode = "verification_failed" + } + if report.DropRefused != "" { + receipt.ErrorCode = "drop_refused" + } + for _, p := range report.Partitions { + part := HistoryPartition{Name: p.Partition.Name, From: p.Partition.From, To: p.Partition.To, Failures: p.Failures, EventsSHA256: p.EventsSHA256, Orgs: []HistoryOrg{}} + ids := make([]string, 0, len(p.Orgs)) + for id := range p.Orgs { + ids = append(ids, id) + } + sort.Strings(ids) + for _, id := range ids { + c := p.Orgs[id] + part.Orgs = append(part.Orgs, HistoryOrg{id, c.Postgres, c.Verified}) + } + receipt.Partitions = append(receipt.Partitions, part) + } + if json.NewEncoder(out).Encode(receipt) != nil { + return 1 + } + return code +} +func historyFailure(out, stderr io.Writer, opts options, swap *auditsink.Swap, code string) int { + if opts.json { + return writeHistory(out, opts, swap, business.AuditHistoryReport{}, errors.New(code)) + } + line(stderr, "audit-history-copy:", code) + return 1 +} diff --git a/module/services/accounts/code/cmd/audit-history-copy/main_test.go b/module/services/accounts/code/pkg/auditops/history_test.go similarity index 70% rename from module/services/accounts/code/cmd/audit-history-copy/main_test.go rename to module/services/accounts/code/pkg/auditops/history_test.go index ca08cf172..da0e31b28 100644 --- a/module/services/accounts/code/cmd/audit-history-copy/main_test.go +++ b/module/services/accounts/code/pkg/auditops/history_test.go @@ -1,4 +1,4 @@ -package main +package auditops import ( "bytes" @@ -36,13 +36,13 @@ func TestParseRunsOnlyUnderASwapValue(t *testing.T) { } var stderr bytes.Buffer - opts, swap, code := parse([]string{"-through", "2026-06", "-confirm-drop", "-batch-size", "200"}, env(swapEnv), &stderr) + opts, swap, code := parse([]string{"-through", "2026-06", "-confirm-drop", "-verify-only", "-expected-partitions-sha256", "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", "-batch-size", "200"}, env(swapEnv), &stderr) require.Equal(t, 0, code, stderr.String()) require.NotNil(t, swap) require.Equal(t, "deployment-1", swap.DeploymentID) require.Equal(t, time.Date(2026, 7, 1, 0, 0, 0, 0, time.UTC), *opts.through, "-through names the last month copied") require.True(t, opts.copyOptions.ConfirmDrop) - require.False(t, opts.copyOptions.VerifyOnly) + require.True(t, opts.copyOptions.VerifyOnly) require.Equal(t, 200, opts.batchSize) } @@ -55,8 +55,8 @@ func TestParseRefusesWhatARunCannotStartFrom(t *testing.T) { }{ "a malformed month": {args: []string{"-through", "June"}, values: swapEnv, code: 2, want: "YYYY-MM"}, "a stray argument": {args: []string{"now"}, values: swapEnv, code: 2, want: "unexpected arguments"}, - "no database": {values: map[string]string{"AUDIT_SINK": "bigquery"}, code: 2, want: "-database-url"}, - "a swap missing parts": {values: map[string]string{"DATABASE_URL": "postgres://example.invalid/a", "AUDIT_SINK": "bigquery"}, code: 1, want: "requires AUDIT_BIGQUERY_PROJECT"}, + "unbounded drop": {args: []string{"-confirm-drop"}, values: swapEnv, code: 2, want: "drop requires"}, + "a swap missing parts": {values: map[string]string{"DATABASE_URL": "postgres://example.invalid/a", "AUDIT_SINK": "bigquery"}, code: 1, want: "invalid audit destination configuration"}, } { t.Run(name, func(t *testing.T) { var stderr bytes.Buffer @@ -92,3 +92,29 @@ func TestSummarizeExitStatus(t *testing.T) { require.Equal(t, 0, summarize(&stdout, &stderr, business.AuditHistoryReport{Verified: true, Dropped: 2, Cutoff: cutoff}, nil)) require.Contains(t, stdout.String(), "dropped 2 partitions ending at or before 2026-09-01T00:00:00Z") } + +func TestCapabilitiesAndHistoryMachineOutputExcludeSecretsAndProviderErrors(t *testing.T) { + var out, stderr bytes.Buffer + require.Equal(t, 0, RunHistory([]string{"-capabilities-json"}, env(swapEnv), &out, &stderr)) + require.Contains(t, out.String(), "codefly/audit-tools/v1") + require.Contains(t, out.String(), "expected_partitions_sha256") + require.NotContains(t, out.String(), "DATABASE_URL") + require.NotContains(t, out.String(), "postgres://") + out.Reset() + _, swap, code := parse(nil, env(swapEnv), &stderr) + require.Zero(t, code) + require.Equal(t, 1, writeHistory(&out, options{json: true}, swap, business.AuditHistoryReport{}, errors.New("provider URL containing password=private"))) + require.NotContains(t, out.String(), "private") + require.Contains(t, out.String(), "operation_failed") + // Production takes its scoped Codefly capability, so absence of the explicit + // integration-test URL is valid at parse time. + values := map[string]string{} + for k, v := range swapEnv { + values[k] = v + } + delete(values, "DATABASE_URL") + opts, swap, code := parse(nil, env(values), &stderr) + require.Zero(t, code) + require.NotNil(t, swap) + require.Empty(t, opts.databaseURL) +} diff --git a/module/services/accounts/code/pkg/auditops/qualification.go b/module/services/accounts/code/pkg/auditops/qualification.go new file mode 100644 index 000000000..8fa921094 --- /dev/null +++ b/module/services/accounts/code/pkg/auditops/qualification.go @@ -0,0 +1,443 @@ +package auditops + +import ( + "accounts/pkg/auditstore/auditsink" + "accounts/pkg/business" + "accounts/pkg/infra" + "context" + "encoding/json" + "errors" + "flag" + "github.com/google/uuid" + "github.com/jackc/pgx/v5/pgxpool" + "io" + "reflect" + "time" +) + +// QualificationReceipt contains only fixture identifiers, hashes and checks +// actually observed. Gateway, archive, alert and controlled outage acceptance +// need separate observers and remain pending here. +type QualificationReceipt struct { + Schema string `json:"schema"` + DeploymentID string `json:"deployment_id"` + Sink string `json:"sink"` + Configuration map[string]string `json:"configuration"` + RunID string `json:"run_id"` + ObservedAt time.Time `json:"observed_at"` + WindowFrom time.Time `json:"window_from"` + WindowTo time.Time `json:"window_to"` + Events []QualificationEvent `json:"events"` + Checks map[string]string `json:"checks"` + ErrorCode string `json:"error_code"` +} +type QualificationEvent struct { + ID string `json:"id"` + OrgID string `json:"org_id"` + Retention business.AuditRetentionClass `json:"retention"` + DetailsSHA256 string `json:"details_sha256"` +} +type organizations []string + +func (o *organizations) String() string { return "" } +func (o *organizations) Set(value string) error { + id, err := uuid.Parse(value) + if err != nil || id == uuid.Nil { + return errors.New("organization must be a nonzero UUID") + } + *o = append(*o, id.String()) + return nil +} + +type qualificationOptions struct { + orgs organizations + runID uuid.UUID + prepare bool + from, to time.Time + timeout time.Duration +} + +func parseQualification(args []string, stderr io.Writer, now time.Time) (qualificationOptions, bool, error) { + var o qualificationOptions + fs := flag.NewFlagSet("audit-qualify", flag.ContinueOnError) + fs.SetOutput(stderr) + fs.Var(&o.orgs, "organization", "existing organization UUID; repeat twice") + run := fs.String("run-id", "", "unique qualification UUID") + prep := fs.Bool("prepare", false, "write six bounded fixtures through RecordTx") + verify := fs.Bool("verify", false, "observe fixtures without writing") + from := fs.String("window-from", "", "prepare receipt window_from") + to := fs.String("window-to", "", "prepare receipt window_to") + deadline := fs.Duration("timeout", 2*time.Minute, "polling deadline, at most five minutes") + _ = fs.Bool("json", false, "emit machine evidence") + cap := fs.Bool("capabilities-json", false, "report capabilities without connecting") + if err := fs.Parse(args); err != nil { + return o, false, errors.New("invalid flags") + } + if fs.NArg() != 0 { + return o, false, errors.New("unexpected arguments") + } + if *cap { + return o, true, nil + } + var err error + o.runID, err = uuid.Parse(*run) + if err != nil || o.runID == uuid.Nil || len(o.orgs) != 2 || o.orgs[0] == o.orgs[1] || *prep == *verify { + return o, false, errors.New("unique run UUID, two distinct organization UUIDs and exactly one phase required") + } + if *deadline <= 0 || *deadline > 5*time.Minute { + return o, false, errors.New("timeout must be positive and at most five minutes") + } + o.prepare, o.timeout = *prep, *deadline + if *prep { + if *from != "" || *to != "" { + return o, false, errors.New("prepare chooses its own window") + } + o.from = now.UTC().Truncate(time.Microsecond) + o.to = o.from.Add(5 * time.Minute) + } else { + o.from, err = time.Parse(time.RFC3339Nano, *from) + if err != nil { + return o, false, errors.New("receipt window-from required") + } + o.to, err = time.Parse(time.RFC3339Nano, *to) + if err != nil { + return o, false, errors.New("receipt window-to required") + } + o.from, o.to = o.from.UTC(), o.to.UTC() + if !o.to.After(o.from) || o.to.Sub(o.from) > 5*time.Minute || o.from.Before(now.Add(-24*time.Hour)) || o.from.After(now.Add(time.Minute)) { + return o, false, errors.New("recent receipt window of at most five minutes required") + } + } + return o, false, nil +} +func qualificationRecords(o qualificationOptions) ([]business.AuditRecord, error) { + var records []business.AuditRecord + for _, org := range []string{o.orgs[0], o.orgs[1], ""} { + for _, class := range []business.AuditRetentionClass{business.RetentionSecurity, business.RetentionContent} { + id := uuid.NewSHA1(o.runID, []byte(org+"/"+string(class))).String() + typ := business.EventAuthLogin + payload := map[string]any{"method": "audit-qualification", "probe_run_id": o.runID.String()} + if class == business.RetentionContent { + typ = business.EventDatasourceSourceSynced + payload = map[string]any{"job_id": o.runID.String(), "repo": "example/audit-qualification", "probe_run_id": o.runID.String()} + } + e := business.AuditEntry{ID: id, OrgID: org, ActorType: business.ActorTypeSystem, EventType: typ, SchemaVersion: 1, Resource: "audit-qualification", ResourceID: id, CreatedAt: o.from, Payload: payload, IdempotencyKey: "audit-qualification/" + o.runID.String() + "/" + string(class)} + if class == business.RetentionContent { + e.Resource = "datasource" + } + record, err := business.NewAuditRecord(e, class) + if err != nil { + return nil, err + } + records = append(records, record) + } + } + return records, nil +} +func newQualificationReceipt(o qualificationOptions, s *auditsink.Swap, records []business.AuditRecord) QualificationReceipt { + r := QualificationReceipt{Schema: "codefly/audit-qualification/v1", DeploymentID: s.DeploymentID, Sink: string(s.Mode), Configuration: safeConfiguration(s), RunID: o.runID.String(), ObservedAt: time.Now().UTC(), WindowFrom: o.from, WindowTo: o.to, Events: []QualificationEvent{}, Checks: map[string]string{}} + for _, key := range []string{"transaction_queue", "queue_drained", "warehouse_readback", "scoped_list", "aggregation", "export", "readable_sources", "organization_isolation", "deployment_isolation", "postgres_no_new_rows", "archive_object_readback", "archive_lock_enforcement", "gateway_authorization", "relay_interruption_recovery", "alert_delivery"} { + r.Checks[key] = "pending" + } + for _, v := range records { + r.Events = append(r.Events, QualificationEvent{v.Entry.ID, v.Entry.OrgID, v.Retention, v.DetailsSHA256}) + } + return r +} +func RunQualification(args []string, getenv func(string) string, stdout, stderr io.Writer) int { + o, cap, err := parseQualification(args, stderr, time.Now()) + if err != nil { + line(stderr, "audit-qualify: invalid qualification options") + return 2 + } + if cap { + return writeCapabilities(stdout, getenv) + } + cfg, err := auditsink.Load(getenv) + if err != nil || cfg.Swap == nil { + line(stderr, "audit-qualify: configured warehouse required") + return 1 + } + records, err := qualificationRecords(o) + if err != nil { + return 1 + } + r := newQualificationReceipt(o, cfg.Swap, records) + fail := func(code string) int { + r.ObservedAt = time.Now().UTC() + r.ErrorCode = code + _ = json.NewEncoder(stdout).Encode(r) + return 1 + } + ctx, err := commandContext(context.Background()) + if err != nil { + return fail("runtime_configuration_failed") + } + ctx, cancel := context.WithTimeout(ctx, o.timeout) + defer cancel() + db, err := openStore(ctx, getenv("DATABASE_URL")) + if err != nil { + return fail("database_unavailable") + } + defer db.Close() + if o.prepare { + emitter, err := business.NewDurableAuditEmitter(db, db, business.WithQueuedRecords()) + if err != nil { + return fail("fixture_configuration_failed") + } + if err = prepareQualification(ctx, o, db, emitter, records, &r); err != nil { + return fail("fixture_transaction_failed") + } + } else { + opened, err := auditsink.Open(ctx, cfg.Swap) + if err != nil { + return fail("warehouse_unavailable") + } + defer opened.Close() + if err = verifyQualification(ctx, o, db, opened.Store, opened.History, records, &r); err != nil { + return fail("qualification_failed") + } + var depthProbe func(context.Context) (business.AuditQueueSnapshot, error) + var poolErr error + var pool *pgxpool.Pool + if getenv("DATABASE_URL") != "" { + pool, poolErr = infra.NewAuditRelayPoolFromURL(ctx, getenv("DATABASE_URL")) + } else { + pool, poolErr = infra.NewAuditRelayPool(ctx) + } + if poolErr != nil { + return fail("queue_unavailable") + } + defer pool.Close() + queue, queueErr := infra.NewPostgresAuditQueue(pool) + if queueErr != nil { + return fail("queue_unavailable") + } + depthProbe = queue.Snapshot + if waitForQueueDrain(ctx, depthProbe) != nil { + r.Checks["queue_drained"] = "failed" + return fail("queue_not_drained") + } + r.Checks["queue_drained"] = "passed" + } + r.ObservedAt = time.Now().UTC() + if json.NewEncoder(stdout).Encode(r) != nil { + return 1 + } + return 0 +} + +type qualificationDB interface { + WithControlPlane(context.Context, func(context.Context) error) error + OrganizationIDExists(context.Context, string) (bool, error) + CountAuditHistory(context.Context, time.Time, time.Time) (map[string]int64, error) +} + +func postgresProbe(ctx context.Context, o qualificationOptions, db qualificationDB) error { + counts, err := db.CountAuditHistory(ctx, o.from, o.to) + if err != nil { + return err + } + for _, n := range counts { + if n != 0 { + return errors.New("postgres received new audit rows") + } + } + return nil +} +func prepareQualification(ctx context.Context, o qualificationOptions, db qualificationDB, recorder business.AuditRecorder, records []business.AuditRecord, r *QualificationReceipt) error { + err := db.WithControlPlane(ctx, func(ctx context.Context) error { + for _, org := range o.orgs { + exists, err := db.OrganizationIDExists(ctx, org) + if err != nil || !exists { + return errors.New("existing organization required") + } + } + for _, v := range records { + if err := recorder.RecordTx(ctx, v.Entry); err != nil { + return err + } + } + return postgresProbe(ctx, o, db) + }) + if err != nil { + r.Checks["transaction_queue"] = "failed" + return err + } + r.Checks["transaction_queue"], r.Checks["postgres_no_new_rows"] = "passed", "passed" + return nil +} + +var errFixturesPending = errors.New("fixtures not delivered yet") + +func warehouseProbe(ctx context.Context, o qualificationOptions, history business.AuditHistoryReader, records []business.AuditRecord, deployment string) error { + expected := map[string]business.AuditRecord{} + for _, v := range records { + expected[v.Entry.ID] = v + } + seen := map[string]bool{} + scanned := 0 + err := history.ReadStoredAuditEvents(ctx, o.from, o.to, func(got business.StoredAuditEvent) error { + scanned++ + if scanned > 20000 { + return errors.New("event read bound exceeded") + } + want, ok := expected[got.Entry.ID] + if !ok { + return nil + } + if got.DeploymentID != deployment || got.Retention != want.Retention || got.DetailsSHA256 != want.DetailsSHA256 || !got.HasDetails || business.AuditDetailsSHA256(got.Details) != want.DetailsSHA256 || !sameEnvelope(got.Entry, want.Entry) { + return errors.New("fixture envelope or details hash differs") + } + seen[got.Entry.ID] = true + return nil + }) + if err != nil { + return err + } + if len(seen) != len(expected) { + return errFixturesPending + } + return nil +} +func verifyQualification(ctx context.Context, o qualificationOptions, db qualificationDB, store business.AuditStore, history business.AuditHistoryReader, records []business.AuditRecord, r *QualificationReceipt) error { + for { + err := warehouseProbe(ctx, o, history, records, r.DeploymentID) + if err == nil { + r.Checks["warehouse_readback"], r.Checks["transaction_queue"] = "passed", "passed" + break + } + if !errors.Is(err, errFixturesPending) { + r.Checks["warehouse_readback"] = "failed" + return err + } + select { + case <-ctx.Done(): + r.Checks["warehouse_readback"] = "failed" + return ctx.Err() + case <-time.After(2 * time.Second): + } + } + if err := db.WithControlPlane(ctx, func(ctx context.Context) error { return postgresProbe(ctx, o, db) }); err != nil { + r.Checks["postgres_no_new_rows"] = "failed" + return err + } + r.Checks["postgres_no_new_rows"] = "passed" + for _, key := range []string{"scoped_list", "aggregation", "export", "readable_sources", "organization_isolation"} { + r.Checks[key] = "passed" + } + for _, org := range []string{o.orgs[0], o.orgs[1], ""} { + scope := business.PlatformAuditScope() + if org != "" { + scope = business.OrganizationAuditScope(org) + } + read := business.AuditRead{Scope: scope, Query: business.AuditQuery{OrgID: org, From: &o.from, To: &o.to, PayloadContains: map[string]any{"probe_run_id": o.runID.String()}, PageSize: 20}} + want := map[string]business.AuditRecord{} + for _, v := range records { + if org == "" || v.Entry.OrgID == org { + want[v.Entry.ID] = v + } + } + entries, token, err := store.ListAuditEvents(ctx, read) + if err != nil || token != "" || !matchEntries(entries, want) { + r.Checks["scoped_list"], r.Checks["organization_isolation"] = "failed", "failed" + continue + } + exported, err := store.ExportAuditEvents(ctx, read) + if err != nil || !matchEntries(exported, want) { + r.Checks["export"] = "failed" + } + buckets, err := store.AggregateAuditEvents(ctx, read, business.AuditAggregationSpec{GroupBy: []string{"event_type"}, Metrics: []business.AuditMetric{{Op: "count"}}}) + expectedCounts := map[string]float64{} + for _, v := range want { + expectedCounts[string(v.Entry.EventType)]++ + } + observedCounts := map[string]float64{} + valid := err == nil + for _, b := range buckets { + if len(b.Keys) != 1 || observedCounts[b.Keys[0]] != 0 || b.Metrics["count"] != expectedCounts[b.Keys[0]] { + valid = false + continue + } + observedCounts[b.Keys[0]] = b.Metrics["count"] + } + if !valid || !reflect.DeepEqual(observedCounts, expectedCounts) { + r.Checks["aggregation"] = "failed" + } + if org != "" { + var source string + for _, v := range records { + if v.Entry.OrgID == org && v.Retention == business.RetentionContent { + source = v.Entry.ResourceID + } + } + allSources := []string{} + for _, v := range records { + if v.Retention == business.RetentionContent { + allSources = append(allSources, v.Entry.ResourceID) + } + } + sources, err := store.LatestSourceSyncEvents(ctx, scope, allSources) + if err != nil || len(sources) != 1 || !sources[source].RequestedAt.Equal(o.from) || sources[source].ActorID != "" { + r.Checks["readable_sources"] = "failed" + } + } + } + _, _, err := store.ListAuditEvents(ctx, business.AuditRead{}) + if err == nil { + r.Checks["organization_isolation"] = "failed" + } + _, _, err = store.ListAuditEvents(ctx, business.AuditRead{Scope: business.OrganizationAuditScope(o.orgs[0]), Query: business.AuditQuery{OrgID: o.orgs[1]}}) + if err == nil { + r.Checks["organization_isolation"] = "failed" + } + for _, v := range r.Checks { + if v == "failed" { + return errors.New("read qualification failed") + } + } + return nil +} +func sameEnvelope(got, want business.AuditEntry) bool { + got.Payload, want.Payload = nil, nil + got.IdempotencyKey, want.IdempotencyKey = "", "" + got.CreatedAt, want.CreatedAt = got.CreatedAt.UTC(), want.CreatedAt.UTC() + return reflect.DeepEqual(got, want) +} +func matchEntries(entries []business.AuditEntry, want map[string]business.AuditRecord) bool { + if len(entries) != len(want) { + return false + } + seen := map[string]bool{} + for _, entry := range entries { + record, ok := want[entry.ID] + if !ok || seen[entry.ID] || !sameEnvelope(entry, record.Entry) { + return false + } + details, err := business.CanonicalAuditDetails(entry.Payload) + if err != nil || business.AuditDetailsSHA256(details) != record.DetailsSHA256 { + return false + } + seen[entry.ID] = true + } + return true +} + +// waitForQueueDrain observes the whole cell queue on the worker authority; +// delivery of just the fixtures does not establish that the queue is empty. +func waitForQueueDrain(ctx context.Context, observe func(context.Context) (business.AuditQueueSnapshot, error)) error { + for { + snapshot, err := observe(ctx) + if err != nil { + return err + } + if snapshot.Depth == 0 { + return nil + } + select { + case <-ctx.Done(): + return ctx.Err() + case <-time.After(time.Second): + } + } +} diff --git a/module/services/accounts/code/pkg/auditops/qualification_test.go b/module/services/accounts/code/pkg/auditops/qualification_test.go new file mode 100644 index 000000000..89d35f9ea --- /dev/null +++ b/module/services/accounts/code/pkg/auditops/qualification_test.go @@ -0,0 +1,228 @@ +package auditops + +import ( + "accounts/pkg/auditstore/bigqueryfake" + "accounts/pkg/auditstore/bigquerystore" + "accounts/pkg/business" + "bytes" + "context" + "encoding/json" + "github.com/google/uuid" + "github.com/stretchr/testify/require" + "testing" + "time" +) + +var probeNow = time.Date(2026, 10, 5, 12, 0, 0, 0, time.UTC) + +func probeOptions() qualificationOptions { + return qualificationOptions{orgs: organizations{"aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa", "bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb"}, runID: uuid.MustParse("cccccccc-cccc-4ccc-8ccc-cccccccccccc"), from: probeNow, to: probeNow.Add(5 * time.Minute), timeout: time.Minute} +} + +type probeDB struct { + exists bool + count int64 + writes []business.AuditEntry +} + +func (d *probeDB) WithControlPlane(ctx context.Context, fn func(context.Context) error) error { + n := len(d.writes) + err := fn(ctx) + if err != nil { + d.writes = d.writes[:n] + } + return err +} +func (d *probeDB) OrganizationIDExists(context.Context, string) (bool, error) { return d.exists, nil } +func (d *probeDB) CountAuditHistory(context.Context, time.Time, time.Time) (map[string]int64, error) { + return map[string]int64{"": d.count}, nil +} +func (d *probeDB) RecordTx(_ context.Context, e business.AuditEntry) error { + d.writes = append(d.writes, e) + return nil +} + +type probeWarehouse struct { + business.AuditStore + records []business.AuditRecord + corrupt bool + leak bool +} + +func (s *probeWarehouse) ReadStoredAuditEvents(_ context.Context, _, _ time.Time, visit func(business.StoredAuditEvent) error) error { + for _, r := range s.records { + event := business.StoredAuditEvent{DeploymentID: "deployment-1", Entry: r.Entry, Retention: r.Retention, DetailsSHA256: r.DetailsSHA256, Details: r.Details, HasDetails: true} + if s.corrupt { + event.Details = "{}" + } + if err := visit(event); err != nil { + return err + } + } + return nil +} +func (s *probeWarehouse) ListAuditEvents(_ context.Context, read business.AuditRead) ([]business.AuditEntry, string, error) { + if err := read.Validate(); err != nil { + return nil, "", err + } + var entries []business.AuditEntry + for _, r := range s.records { + if read.Scope.Platform() || r.Entry.OrgID == read.Scope.OrgID() || s.leak { + entries = append(entries, r.Entry) + } + } + return entries, "", nil +} +func (s *probeWarehouse) ExportAuditEvents(ctx context.Context, read business.AuditRead) ([]business.AuditEntry, error) { + v, _, err := s.ListAuditEvents(ctx, read) + return v, err +} +func (s *probeWarehouse) AggregateAuditEvents(ctx context.Context, read business.AuditRead, _ business.AuditAggregationSpec) ([]business.AuditAggregateBucket, error) { + v, _, err := s.ListAuditEvents(ctx, read) + counts := map[string]float64{} + for _, e := range v { + counts[string(e.EventType)]++ + } + var buckets []business.AuditAggregateBucket + for typ, n := range counts { + buckets = append(buckets, business.AuditAggregateBucket{Key: typ, Keys: []string{typ}, Metrics: map[string]float64{"count": n}}) + } + return buckets, err +} +func (s *probeWarehouse) LatestSourceSyncEvents(_ context.Context, scope business.AuditReadScope, ids []string) (map[string]business.AuditSourceSyncEvent, error) { + out := map[string]business.AuditSourceSyncEvent{} + for _, r := range s.records { + for _, id := range ids { + if id == r.Entry.ID && r.Entry.OrgID == scope.OrgID() && r.Retention == business.RetentionContent { + out[id] = business.AuditSourceSyncEvent{RequestedAt: r.Entry.CreatedAt} + } + } + } + return out, nil +} +func probeReceipt(t *testing.T, o qualificationOptions, records []business.AuditRecord) QualificationReceipt { + t.Helper() + var stderr bytes.Buffer + _, swap, code := parse(nil, env(swapEnv), &stderr) + require.Zero(t, code) + return newQualificationReceipt(o, swap, records) +} +func TestQualificationCommitsOnlyIntoExistingOrganizationsAndRefusesNewPostgresRows(t *testing.T) { + o := probeOptions() + records, err := qualificationRecords(o) + require.NoError(t, err) + r := probeReceipt(t, o, records) + db := &probeDB{exists: false} + require.Error(t, prepareQualification(context.Background(), o, db, db, records, &r)) + require.Empty(t, db.writes) + db.exists = true + db.count = 1 + require.Error(t, prepareQualification(context.Background(), o, db, db, records, &r)) + require.Empty(t, db.writes, "failed acceptance aborts fixture transaction") + db.count = 0 + require.NoError(t, prepareQualification(context.Background(), o, db, db, records, &r)) + require.Len(t, db.writes, 6) + require.Equal(t, "passed", r.Checks["transaction_queue"]) + require.Equal(t, "pending", r.Checks["warehouse_readback"]) +} +func TestQualificationChecksHashesAndOrganizationIsolationWithoutClaimingOtherAcceptance(t *testing.T) { + o := probeOptions() + records, err := qualificationRecords(o) + require.NoError(t, err) + db := &probeDB{exists: true} + store := &probeWarehouse{records: records} + r := probeReceipt(t, o, records) + require.NoError(t, verifyQualification(context.Background(), o, db, store, store, records, &r)) + for _, key := range []string{"warehouse_readback", "scoped_list", "aggregation", "export", "readable_sources", "organization_isolation"} { + require.Equal(t, "passed", r.Checks[key]) + } + for _, key := range []string{"deployment_isolation", "archive_lock_enforcement", "relay_interruption_recovery", "gateway_authorization", "alert_delivery"} { + require.Equal(t, "pending", r.Checks[key]) + } + store.corrupt = true + r = probeReceipt(t, o, records) + require.Error(t, verifyQualification(context.Background(), o, db, store, store, records, &r)) + require.Equal(t, "failed", r.Checks["warehouse_readback"]) + store.corrupt = false + store.leak = true + r = probeReceipt(t, o, records) + require.Error(t, verifyQualification(context.Background(), o, db, store, store, records, &r)) + require.Equal(t, "failed", r.Checks["organization_isolation"]) + store.leak = false + store.records = store.records[:5] + require.ErrorIs(t, warehouseProbe(context.Background(), o, store, records, "deployment-1"), errFixturesPending) + require.Error(t, warehouseProbe(context.Background(), o, &probeWarehouse{records: records}, records, "other-deployment")) +} +func TestQualificationBoundsAndDeterministicReceiptFixtures(t *testing.T) { + o := probeOptions() + records, err := qualificationRecords(o) + require.NoError(t, err) + again, err := qualificationRecords(o) + require.NoError(t, err) + require.Equal(t, records, again) + other := o + other.runID = uuid.New() + newRun, err := qualificationRecords(other) + require.NoError(t, err) + require.NotEqual(t, records[0].Entry.ID, newRun[0].Entry.ID) + args := []string{"-run-id", o.runID.String(), "-organization", o.orgs[0], "-organization", o.orgs[1], "-verify", "-window-from", o.from.Format(time.RFC3339Nano), "-window-to", o.to.Format(time.RFC3339Nano)} + var stderr bytes.Buffer + _, _, err = parseQualification(args, &stderr, probeNow) + require.NoError(t, err) + _, _, err = parseQualification(append(args, "-timeout", "6m"), &stderr, probeNow) + require.Error(t, err) + _, _, err = parseQualification(append(args, "-prepare"), &stderr, probeNow) + require.Error(t, err) + var out bytes.Buffer + r := probeReceipt(t, o, records) + r.ErrorCode = "qualification_failed" + require.NoError(t, jsonEncode(&out, r)) + require.NotContains(t, out.String(), "method") + require.NotContains(t, out.String(), "repo") + require.NotContains(t, out.String(), "postgres://") +} +func jsonEncode(out *bytes.Buffer, v any) error { return json.NewEncoder(out).Encode(v) } + +func TestQueueDrainedRequiresAnObservedEmptyQueue(t *testing.T) { + calls := 0 + require.NoError(t, waitForQueueDrain(context.Background(), func(context.Context) (business.AuditQueueSnapshot, error) { + calls++ + return business.AuditQueueSnapshot{Depth: 0}, nil + })) + require.Equal(t, 1, calls) + ctx, cancel := context.WithCancel(context.Background()) + cancel() + require.Error(t, waitForQueueDrain(ctx, func(context.Context) (business.AuditQueueSnapshot, error) { + return business.AuditQueueSnapshot{Depth: 2}, nil + })) +} + +// Exercise the qualifier through the actual warehouse reader and Storage Read +// fake. Its source selector requires datasource envelopes, unlike an arbitrary +// stand-in that would accept every event with a matching resource identifier. +type readerProbeStore struct{ *bigquerystore.Reader } + +func (readerProbeStore) AppendAuditBatch(context.Context, business.AuditBatch) error { return nil } +func TestQualificationAgainstWarehouseReaderAndArrowStreams(t *testing.T) { + o := probeOptions() + records, err := qualificationRecords(o) + require.NoError(t, err) + fake := bigqueryfake.New() + fake.CreateTable(bigqueryfake.TablePath("example-project", "audit", bigquerystore.EventsTable), bigquerystore.EventsSchema()) + fake.CreateTable(bigqueryfake.TablePath("example-project", "audit", bigquerystore.DetailsTable), bigquerystore.DetailsSchema()) + for _, deployment := range []string{"deployment-1", "different-deployment"} { + events, details := bigquerystore.BatchRows(business.AuditBatch{ID: uuid.NewString(), DeploymentID: deployment, ComposedAt: o.from, Records: records}) + for _, row := range events { + require.NoError(t, fake.Insert(bigqueryfake.TablePath("example-project", "audit", bigquerystore.EventsTable), row.Values)) + } + for _, row := range details { + require.NoError(t, fake.Insert(bigqueryfake.TablePath("example-project", "audit", bigquerystore.DetailsTable), row.Values)) + } + } + reader, err := bigquerystore.NewReader(bigquerystore.ReadConfig{Client: fake, Project: "example-project", Dataset: "audit", DeploymentID: "deployment-1", Now: func() time.Time { return o.to }}) + require.NoError(t, err) + receipt := probeReceipt(t, o, records) + require.NoError(t, verifyQualification(context.Background(), o, &probeDB{exists: true}, readerProbeStore{reader}, reader, records, &receipt)) + require.Equal(t, "passed", receipt.Checks["readable_sources"]) + require.Equal(t, "pending", receipt.Checks["deployment_isolation"], "fake coverage cannot count as live peer-cell evidence") +} diff --git a/module/services/accounts/code/pkg/business/audit_history.go b/module/services/accounts/code/pkg/business/audit_history.go index 3e1a9bcce..fe729ea9b 100644 --- a/module/services/accounts/code/pkg/business/audit_history.go +++ b/module/services/accounts/code/pkg/business/audit_history.go @@ -2,8 +2,12 @@ package business import ( "context" + "crypto/sha256" + "encoding/hex" + "encoding/json" "errors" "fmt" + "hash" "sort" "strings" "time" @@ -110,6 +114,9 @@ type AuditHistoryCopyOptions struct { // ConfirmDrop drops the copied partitions once this run has verified every // one of them. Without it nothing is ever dropped. ConfirmDrop bool + // ExpectedPartitionsSHA256 is the digest of a previous verification receipt. + // A confirmed drop re-verifies and refuses a changed plan before any DDL. + ExpectedPartitionsSHA256 string } // AuditHistoryOrgCount is one organization's events in a partition: how many @@ -122,6 +129,8 @@ type AuditHistoryOrgCount struct { // AuditHistoryPartitionReport is what a run found in one partition. type AuditHistoryPartitionReport struct { Partition AuditHistoryPartition + // EventsSHA256 binds the source event envelopes, classes and details hashes. + EventsSHA256 string // Copied is how many events this run wrote; the rest were already there. Copied int // Orgs counts the partition's events per organization ("" is the @@ -136,7 +145,9 @@ type AuditHistoryPartitionReport struct { // AuditHistoryReport is what a run did. type AuditHistoryReport struct { - Partitions []AuditHistoryPartitionReport + Partitions []AuditHistoryPartitionReport + PartitionsSHA256 string + DroppedNames []string // Verified is true when every partition's events are in the store, each // copy identical to its row. Verified bool @@ -213,7 +224,11 @@ func (c *AuditHistoryCopy) Run(ctx context.Context, opts AuditHistoryCopyOptions if err != nil { return report, err } + if c.cfg.Through != nil { + report.Cutoff = *c.cfg.Through + } if len(partitions) == 0 { + report.PartitionsSHA256 = c.planDigest(report) report.Verified = true c.cfg.Progress("no audit_events partitions to copy") return report, nil @@ -236,6 +251,7 @@ func (c *AuditHistoryCopy) Run(ctx context.Context, opts AuditHistoryCopyOptions report.Verified = false } } + report.PartitionsSHA256 = c.planDigest(report) if !report.Verified { return report, ErrAuditHistoryUnverified } @@ -243,6 +259,10 @@ func (c *AuditHistoryCopy) Run(ctx context.Context, opts AuditHistoryCopyOptions c.cfg.Progress("verified; nothing dropped without the drop confirmation") return report, nil } + if c.cfg.Through == nil || opts.ExpectedPartitionsSHA256 == "" || opts.ExpectedPartitionsSHA256 != report.PartitionsSHA256 { + report.DropRefused = "explicit cutoff and matching verification digest required" + return report, errors.New("audit history copy: drop refused: explicit cutoff and matching verification digest required") + } if refused, err := c.drop(ctx, &report, partitions); err != nil || refused != "" { report.DropRefused = refused if err == nil { @@ -278,16 +298,18 @@ func (c *AuditHistoryCopy) partitions(ctx context.Context) ([]AuditHistoryPartit // copyPartition copies and verifies one partition, a window at a time. func (c *AuditHistoryCopy) copyPartition(ctx context.Context, resolver *AuditEventResolver, partition AuditHistoryPartition, opts AuditHistoryCopyOptions) (AuditHistoryPartitionReport, error) { report := AuditHistoryPartitionReport{Partition: partition, Orgs: map[string]AuditHistoryOrgCount{}} + digest := sha256.New() for from := partition.From; from.Before(partition.To); from = from.Add(auditHistoryWindow) { to := from.Add(auditHistoryWindow) if to.After(partition.To) { to = partition.To } - if err := c.copyWindow(ctx, resolver, from, to, opts, &report); err != nil { + if err := c.copyWindow(ctx, resolver, from, to, opts, &report, digest); err != nil { return report, fmt.Errorf("audit history copy: %s [%s, %s): %w", partition.Name, from.UTC().Format(time.RFC3339), to.UTC().Format(time.RFC3339), err) } } + report.EventsSHA256 = hex.EncodeToString(digest.Sum(nil)) var rows, verified int64 for _, count := range report.Orgs { rows += count.Postgres @@ -300,7 +322,7 @@ func (c *AuditHistoryCopy) copyPartition(ctx context.Context, resolver *AuditEve // copyWindow reads a window's rows, reads back what the store holds of it, // writes what is missing, and verifies every row. -func (c *AuditHistoryCopy) copyWindow(ctx context.Context, resolver *AuditEventResolver, from, to time.Time, opts AuditHistoryCopyOptions, report *AuditHistoryPartitionReport) error { +func (c *AuditHistoryCopy) copyWindow(ctx context.Context, resolver *AuditEventResolver, from, to time.Time, opts AuditHistoryCopyOptions, report *AuditHistoryPartitionReport, digest hash.Hash) error { records, err := c.readWindow(ctx, resolver, from, to) if err != nil { return err @@ -333,6 +355,16 @@ func (c *AuditHistoryCopy) copyWindow(ctx context.Context, resolver *AuditEventR } } for _, record := range records { + envelope := record.Entry + envelope.Payload, envelope.IdempotencyKey = nil, "" + envelope.CreatedAt = envelope.CreatedAt.UTC() + // Encoding an explicit envelope plus the content hash binds source bytes + // without retaining payloads or a deployment-sized stream in memory. + _ = json.NewEncoder(digest).Encode(struct { + Entry AuditEntry + Retention AuditRetentionClass + DetailsSHA256 string + }{envelope, record.Retention, record.DetailsSHA256}) org := record.Entry.OrgID count := report.Orgs[org] count.Postgres++ @@ -495,7 +527,32 @@ func (c *AuditHistoryCopy) drop(ctx context.Context, report *AuditHistoryReport, return "", fmt.Errorf("audit history copy: drop: %w", err) } if refused == "" { + for _, p := range report.Partitions { + report.DroppedNames = append(report.DroppedNames, p.Partition.Name) + } c.cfg.Progress(fmt.Sprintf("dropped %d partitions ending at or before %s", report.Dropped, report.Cutoff.UTC().Format(time.RFC3339))) } return refused, nil } + +// planDigest excludes run timing and copied counts: copy, verify and drop runs +// of unchanged source events produce the same authorization digest. +func (c *AuditHistoryCopy) planDigest(report AuditHistoryReport) string { + type partition struct { + Name string + From, To time.Time + Orgs map[string]AuditHistoryOrgCount + EventsSHA256 string + } + parts := make([]partition, 0, len(report.Partitions)) + for _, p := range report.Partitions { + parts = append(parts, partition{p.Partition.Name, p.Partition.From.UTC(), p.Partition.To.UTC(), p.Orgs, p.EventsSHA256}) + } + body, _ := json.Marshal(struct { + Deployment string + Cutoff time.Time + Partitions []partition + }{c.cfg.DeploymentID, report.Cutoff.UTC(), parts}) + sum := sha256.Sum256(body) + return hex.EncodeToString(sum[:]) +} diff --git a/module/services/accounts/code/pkg/business/audit_history_test.go b/module/services/accounts/code/pkg/business/audit_history_test.go index fcbb46894..2f7f92f4b 100644 --- a/module/services/accounts/code/pkg/business/audit_history_test.go +++ b/module/services/accounts/code/pkg/business/audit_history_test.go @@ -181,6 +181,10 @@ func newHistoryFixture() *historyFixture { func (f *historyFixture) copier(t *testing.T, cfg AuditHistoryCopyConfig) *AuditHistoryCopy { t.Helper() cfg.Source, cfg.Store, cfg.Archive, cfg.ReadBack = f.source, f.store, f.archive, f.store + if cfg.Through == nil && len(f.source.partitions) > 0 { + through := f.source.partitions[len(f.source.partitions)-1].To + cfg.Through = &through + } cfg.DeploymentID = "deployment-1" if cfg.ContentRetention == 0 { cfg.ContentRetention = 400 * 24 * time.Hour @@ -234,7 +238,7 @@ func TestAuditHistoryCopyCopiesVerifiesAndDropsOnlyWhenConfirmed(t *testing.T) { // A second run finds everything there: it copies nothing, verifies, and with // the confirmation drops every partition up to the cutoff. appends := f.store.appends - report, err = f.copier(t, AuditHistoryCopyConfig{}).Run(context.Background(), AuditHistoryCopyOptions{ConfirmDrop: true}) + report, err = f.copier(t, AuditHistoryCopyConfig{}).Run(context.Background(), f.dropOptions(t, AuditHistoryCopyConfig{})) require.NoError(t, err) require.Equal(t, appends, f.store.appends, "a resumed run copies nothing already there") require.Equal(t, []time.Time{report.Cutoff}, f.source.drops) @@ -245,7 +249,7 @@ func TestAuditHistoryCopyCopiesVerifiesAndDropsOnlyWhenConfirmed(t *testing.T) { func TestAuditHistoryCopyResumesAfterAnInterruption(t *testing.T) { f := newHistoryFixture() f.store.failAppend = 3 - _, err := f.copier(t, AuditHistoryCopyConfig{}).Run(context.Background(), AuditHistoryCopyOptions{ConfirmDrop: true}) + _, err := f.copier(t, AuditHistoryCopyConfig{}).Run(context.Background(), f.dropOptions(t, AuditHistoryCopyConfig{})) require.ErrorContains(t, err, "warehouse unavailable") require.Empty(t, f.source.drops, "an interrupted run drops nothing") partial := len(f.store.copies()) @@ -253,7 +257,7 @@ func TestAuditHistoryCopyResumesAfterAnInterruption(t *testing.T) { require.Less(t, partial, len(f.source.rows)) f.store.failAppend = 0 - report, err := f.copier(t, AuditHistoryCopyConfig{}).Run(context.Background(), AuditHistoryCopyOptions{ConfirmDrop: true}) + report, err := f.copier(t, AuditHistoryCopyConfig{}).Run(context.Background(), f.dropOptions(t, AuditHistoryCopyConfig{})) require.NoError(t, err) require.True(t, report.Verified) copied := 0 @@ -362,7 +366,7 @@ func TestAuditHistoryContentDetailsPastTheWindowAreNotRequired(t *testing.T) { e.Details, e.HasDetails = "", false // expired with their day partition } } - report, err := f.copier(t, AuditHistoryCopyConfig{ContentRetention: 7 * 24 * time.Hour}).Run(context.Background(), AuditHistoryCopyOptions{ConfirmDrop: true}) + report, err := f.copier(t, AuditHistoryCopyConfig{ContentRetention: 7 * 24 * time.Hour}).Run(context.Background(), f.dropOptions(t, AuditHistoryCopyConfig{ContentRetention: 7 * 24 * time.Hour})) require.NoError(t, err) require.True(t, report.Verified) require.Equal(t, int64(3), report.Dropped) @@ -374,7 +378,7 @@ func TestAuditHistoryDropIsRefusedWhenAPartitionChangedSinceVerification(t *test s.beforeCount = nil s.rows = append(s.rows, historyRow(999, "", EventAuthLogin, time.Date(2026, 9, 2, 0, 0, 0, 0, time.UTC))) } - report, err := f.copier(t, AuditHistoryCopyConfig{}).Run(context.Background(), AuditHistoryCopyOptions{ConfirmDrop: true}) + report, err := f.copier(t, AuditHistoryCopyConfig{}).Run(context.Background(), f.dropOptions(t, AuditHistoryCopyConfig{})) require.ErrorContains(t, err, "changed since it was verified") require.True(t, report.Verified) require.Contains(t, report.DropRefused, "audit_events_2026_09") @@ -384,7 +388,7 @@ func TestAuditHistoryDropIsRefusedWhenAPartitionChangedSinceVerification(t *test func TestAuditHistoryThroughLimitsThePartitionsAndTheCutoff(t *testing.T) { f := newHistoryFixture() through := time.Date(2026, 9, 1, 0, 0, 0, 0, time.UTC) - report, err := f.copier(t, AuditHistoryCopyConfig{Through: &through}).Run(context.Background(), AuditHistoryCopyOptions{ConfirmDrop: true}) + report, err := f.copier(t, AuditHistoryCopyConfig{Through: &through}).Run(context.Background(), f.dropOptions(t, AuditHistoryCopyConfig{Through: &through})) require.NoError(t, err) require.Len(t, report.Partitions, 1) require.Equal(t, "audit_events_2026_08", report.Partitions[0].Partition.Name) @@ -397,8 +401,9 @@ func TestAuditHistoryThroughLimitsThePartitionsAndTheCutoff(t *testing.T) { f = newHistoryFixture() f.source.partitions = append([]AuditHistoryPartition{{Name: "audit_events_2026_07", From: month(2026, time.July).From, To: month(2026, time.July).To}}, f.source.partitions...) copier := f.copier(t, AuditHistoryCopyConfig{Through: &through}) + digest := f.dropOptions(t, AuditHistoryCopyConfig{Through: &through}) copier.cfg.Source = &listingAfterVerification{historySource: f.source, extra: AuditHistoryPartition{Name: "audit_events_2026_06", From: month(2026, time.June).From, To: month(2026, time.June).To}} - _, err = copier.Run(context.Background(), AuditHistoryCopyOptions{ConfirmDrop: true}) + _, err = copier.Run(context.Background(), digest) require.ErrorContains(t, err, "audit_events_2026_06 ends before the cutoff and was not verified") require.Empty(t, f.source.drops) } @@ -431,3 +436,49 @@ func TestAuditHistoryVerifyOnlyWritesNothing(t *testing.T) { require.Equal(t, appends, f.store.appends) require.Len(t, f.archive.batches, archived) } + +// Construct the explicit prior verification receipt against a separate copy of +// the source, so interruption/tamper tests retain their original effects. +func (f *historyFixture) dropOptions(t *testing.T, cfg AuditHistoryCopyConfig) AuditHistoryCopyOptions { + t.Helper() + clone := &historyFixture{source: &historySource{partitions: append([]AuditHistoryPartition(nil), f.source.partitions...), rows: append([]AuditEntry(nil), f.source.rows...)}, store: &historyStore{}, archive: &historyArchive{}} + report, err := clone.copier(t, cfg).Run(context.Background(), AuditHistoryCopyOptions{}) + require.NoError(t, err) + return AuditHistoryCopyOptions{ConfirmDrop: true, ExpectedPartitionsSHA256: report.PartitionsSHA256} +} + +func TestAuditHistoryDropDigestBindsContentAndNeedsAPriorVerification(t *testing.T) { + f := newHistoryFixture() + copier := f.copier(t, AuditHistoryCopyConfig{}) + plan, err := copier.Run(context.Background(), AuditHistoryCopyOptions{}) + require.NoError(t, err) + verify, err := copier.Run(context.Background(), AuditHistoryCopyOptions{VerifyOnly: true}) + require.NoError(t, err) + require.Equal(t, plan.PartitionsSHA256, verify.PartitionsSHA256, "copy counts and run timing do not alter a verification authorization") + _, err = copier.Run(context.Background(), AuditHistoryCopyOptions{ConfirmDrop: true, VerifyOnly: true}) + require.ErrorContains(t, err, "matching verification digest") + require.Empty(t, f.source.drops) + // Equal counts are insufficient: model a source and destination rewritten + // together after the receipt, preserving IDs and counts but changing content. + f.source.rows[0].Payload = map[string]any{"n": 999, "repo": "example/changed"} + resolver := NewAuditEventResolver(nil) + resolved, err := resolver.Resolve(context.Background(), f.source.rows[0].EventType) + require.NoError(t, err) + record, err := NewAuditRecord(f.source.rows[0], resolved.RetentionClass()) + require.NoError(t, err) + for i := range f.store.stored { + if f.store.stored[i].Entry.ID == record.Entry.ID { + f.store.stored[i].Details = record.Details + f.store.stored[i].DetailsSHA256 = record.DetailsSHA256 + } + } + current, err := copier.Run(context.Background(), AuditHistoryCopyOptions{ConfirmDrop: true, VerifyOnly: true, ExpectedPartitionsSHA256: plan.PartitionsSHA256}) + require.ErrorContains(t, err, "matching verification digest") + require.True(t, current.Verified, "current copy is valid but the operator approved different content") + require.NotEqual(t, plan.PartitionsSHA256, current.PartitionsSHA256) + require.Empty(t, f.source.drops) + copier.cfg.Through = nil + _, err = copier.Run(context.Background(), AuditHistoryCopyOptions{ConfirmDrop: true, VerifyOnly: true, ExpectedPartitionsSHA256: current.PartitionsSHA256}) + require.ErrorContains(t, err, "explicit cutoff") + require.Empty(t, f.source.drops) +} diff --git a/module/services/accounts/code/pkg/infra/audit_qualification_test.go b/module/services/accounts/code/pkg/infra/audit_qualification_test.go new file mode 100644 index 000000000..390e03bc8 --- /dev/null +++ b/module/services/accounts/code/pkg/infra/audit_qualification_test.go @@ -0,0 +1,61 @@ +//go:build !pure + +package infra_test + +import ( + "accounts/pkg/auditops" + "accounts/pkg/business" + "bytes" + "context" + "encoding/json" + "testing" + + "github.com/google/uuid" + "github.com/stretchr/testify/require" +) + +func TestAuditQualificationPrepareUsesRuntimeAuthorityAndTransactionalQueue(t *testing.T) { + orgA, orgB := seedOrg(t, seedUser(t)), seedOrg(t, seedUser(t)) + values := map[string]string{"AUDIT_SINK": "bigquery", "AUDIT_BIGQUERY_PROJECT": "example-project", "AUDIT_BIGQUERY_DATASET": "audit", "AUDIT_ARCHIVE_URL": "gs://example-audit-archive", "AUDIT_DEPLOYMENT_ID": "deployment-1", "AUDIT_CONTENT_RETENTION_DAYS": "90"} + getenv := func(key string) string { return values[key] } + var out, stderr bytes.Buffer + args := []string{"-json", "-prepare", "-run-id", uuid.NewString(), "-organization", orgA, "-organization", orgB} + require.Zero(t, auditops.RunQualification(args, getenv, &out, &stderr), stderr.String()+out.String()) + var receipt auditops.QualificationReceipt + require.NoError(t, json.Unmarshal(out.Bytes(), &receipt)) + require.Len(t, receipt.Events, 6) + require.Equal(t, "passed", receipt.Checks["transaction_queue"]) + require.Equal(t, "pending", receipt.Checks["queue_drained"]) + require.Empty(t, receipt.ErrorCode) + ids := []string{} + for _, e := range receipt.Events { + ids = append(ids, e.ID) + } + pool := auditRelayPool(t) + var n int + require.NoError(t, pool.QueryRow(testCtx, `SELECT count(*) FROM audit_event_queue WHERE id=ANY($1::uuid[])`, ids).Scan(&n)) + require.Equal(t, 6, n) + require.NoError(t, testStore.WithControlPlane(testCtx, func(ctx context.Context) error { + counts, err := testStore.CountAuditHistory(ctx, receipt.WindowFrom, receipt.WindowTo) + require.NoError(t, err) + for _, count := range counts { + require.Zero(t, count) + } + return nil + })) + // The receipt's platform event and both organizations were queued together; + // no tee/job fan-out was invoked by the record-only probe. + require.Equal(t, business.RetentionSecurity, receipt.Events[0].Retention) + out.Reset() + stderr.Reset() + missingArgs := []string{"-json", "-prepare", "-run-id", uuid.NewString(), "-organization", orgA, "-organization", uuid.NewString()} + require.Equal(t, 1, auditops.RunQualification(missingArgs, getenv, &out, &stderr)) + require.NoError(t, json.Unmarshal(out.Bytes(), &receipt)) + require.Equal(t, "fixture_transaction_failed", receipt.ErrorCode) + ids = nil + for _, e := range receipt.Events { + ids = append(ids, e.ID) + } + require.NoError(t, pool.QueryRow(testCtx, `SELECT count(*) FROM audit_event_queue WHERE id=ANY($1::uuid[])`, ids).Scan(&n)) + require.Zero(t, n, "a missing organization writes no fixtures") +} diff --git a/module/services/accounts/code/pkg/infra/postgres_audit_history_test.go b/module/services/accounts/code/pkg/infra/postgres_audit_history_test.go index 0fe020074..4cb2d3841 100644 --- a/module/services/accounts/code/pkg/infra/postgres_audit_history_test.go +++ b/module/services/accounts/code/pkg/infra/postgres_audit_history_test.go @@ -195,7 +195,7 @@ func runHistoryCopy(t *testing.T, entries []business.AuditEntry) { require.Equal(t, want, got) } - report, err = copier.Run(testCtx, business.AuditHistoryCopyOptions{ConfirmDrop: true}) + report, err = copier.Run(testCtx, business.AuditHistoryCopyOptions{ConfirmDrop: true, VerifyOnly: true, ExpectedPartitionsSHA256:report.PartitionsSHA256}) require.NoError(t, err) require.Positive(t, report.Dropped) require.Equal(t, through, report.Cutoff) diff --git a/module/services/accounts/code/pkg/infra/postgres_webhooks_test.go b/module/services/accounts/code/pkg/infra/postgres_webhooks_test.go index 89e35cf10..8e58ad7d4 100644 --- a/module/services/accounts/code/pkg/infra/postgres_webhooks_test.go +++ b/module/services/accounts/code/pkg/infra/postgres_webhooks_test.go @@ -17,6 +17,7 @@ import ( "time" codefly "github.com/codefly-dev/sdk-go" + "github.com/google/uuid" "github.com/jackc/pgx/v5/pgxpool" "github.com/stretchr/testify/require" @@ -103,7 +104,9 @@ func runPostgresInfraTests(m *testing.M) int { sdk.WithDebug(), sdk.WithSharedControlChannel(), sdk.WithExcludedDependencies("cache", "vault", "telemetry"), - sdk.WithNamingScope("test-infra"), + // Each run owns a fresh database. A fixed scope can reopen an old + // unmerged ledger after migrations are renumbered against current main. + sdk.WithNamingScope("test-infra-"+uuid.NewString()[:12]), sdk.WithTimeout(90*time.Second), sdk.WithSilence("store"), ) diff --git a/module/services/accounts/code/pkg/infra/testdata/control_plane_call_sites.txt b/module/services/accounts/code/pkg/infra/testdata/control_plane_call_sites.txt index 27ff940f5..e50781ab3 100644 --- a/module/services/accounts/code/pkg/infra/testdata/control_plane_call_sites.txt +++ b/module/services/accounts/code/pkg/infra/testdata/control_plane_call_sites.txt @@ -21,6 +21,8 @@ pkg/adapters/rpcs.go: (*UserServer).DeleteUser: System() pkg/adapters/rpcs.go: (*UserServer).DeleteUser: System() pkg/adapters/rpcs.go: (*UserServer).GetUser: System() pkg/adapters/rpcs.go: userDataIdentity: System() +pkg/auditops/qualification.go: prepareQualification: WithControlPlane +pkg/auditops/qualification.go: verifyQualification: WithControlPlane pkg/auth/pg/client_session_store.go: (*SessionStore).AuthorizeClientSession: WithControlPlane pkg/auth/pg/session_store.go: (*SessionStore).ExchangeOrganization: WithControlPlane pkg/auth/pg/session_store.go: (*SessionStore).FindByRefreshHash: WithControlPlane diff --git a/module/services/store/migrations/17_audit_event_queue.down.sql b/module/services/store/migrations/18_audit_event_queue.down.sql similarity index 100% rename from module/services/store/migrations/17_audit_event_queue.down.sql rename to module/services/store/migrations/18_audit_event_queue.down.sql diff --git a/module/services/store/migrations/17_audit_event_queue.up.sql b/module/services/store/migrations/18_audit_event_queue.up.sql similarity index 100% rename from module/services/store/migrations/17_audit_event_queue.up.sql rename to module/services/store/migrations/18_audit_event_queue.up.sql diff --git a/module/services/store/migrations/18_audit_event_type_retention_class.down.sql b/module/services/store/migrations/19_audit_event_type_retention_class.down.sql similarity index 100% rename from module/services/store/migrations/18_audit_event_type_retention_class.down.sql rename to module/services/store/migrations/19_audit_event_type_retention_class.down.sql diff --git a/module/services/store/migrations/18_audit_event_type_retention_class.up.sql b/module/services/store/migrations/19_audit_event_type_retention_class.up.sql similarity index 100% rename from module/services/store/migrations/18_audit_event_type_retention_class.up.sql rename to module/services/store/migrations/19_audit_event_type_retention_class.up.sql From 59bdacf79b5d0eed680b7352ab64093195a16565 Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Mon, 5 Oct 2026 12:09:22 -0400 Subject: [PATCH 16/69] style(audit): apply required Go import formatting --- module/services/accounts/code/pkg/auditops/qualification.go | 5 +++-- .../accounts/code/pkg/auditops/qualification_test.go | 5 +++-- .../code/pkg/auditstore/bigquerystore/emulator_test.go | 4 +++- .../accounts/code/pkg/infra/postgres_audit_history_test.go | 2 +- 4 files changed, 10 insertions(+), 6 deletions(-) diff --git a/module/services/accounts/code/pkg/auditops/qualification.go b/module/services/accounts/code/pkg/auditops/qualification.go index 8fa921094..11abee841 100644 --- a/module/services/accounts/code/pkg/auditops/qualification.go +++ b/module/services/accounts/code/pkg/auditops/qualification.go @@ -8,11 +8,12 @@ import ( "encoding/json" "errors" "flag" - "github.com/google/uuid" - "github.com/jackc/pgx/v5/pgxpool" "io" "reflect" "time" + + "github.com/google/uuid" + "github.com/jackc/pgx/v5/pgxpool" ) // QualificationReceipt contains only fixture identifiers, hashes and checks diff --git a/module/services/accounts/code/pkg/auditops/qualification_test.go b/module/services/accounts/code/pkg/auditops/qualification_test.go index 89d35f9ea..93a1fdadb 100644 --- a/module/services/accounts/code/pkg/auditops/qualification_test.go +++ b/module/services/accounts/code/pkg/auditops/qualification_test.go @@ -7,10 +7,11 @@ import ( "bytes" "context" "encoding/json" - "github.com/google/uuid" - "github.com/stretchr/testify/require" "testing" "time" + + "github.com/google/uuid" + "github.com/stretchr/testify/require" ) var probeNow = time.Date(2026, 10, 5, 12, 0, 0, 0, time.UTC) diff --git a/module/services/accounts/code/pkg/auditstore/bigquerystore/emulator_test.go b/module/services/accounts/code/pkg/auditstore/bigquerystore/emulator_test.go index a50c52fc0..8365dfbc9 100644 --- a/module/services/accounts/code/pkg/auditstore/bigquerystore/emulator_test.go +++ b/module/services/accounts/code/pkg/auditstore/bigquerystore/emulator_test.go @@ -242,6 +242,8 @@ func TestEmulatorReadsThroughTheStorageReadAPI(t *testing.T) { require.Equal(t, 6, served(func(w *restriction) { w.eq("deployment_id", "deployment-1") })) require.Equal(t, 9, served(func(w *restriction) { w.timeBound("occurred_at", "<=", content.Entry.CreatedAt) })) require.Equal(t, 6, served(func(w *restriction) { w.timeBound("occurred_at", "<", content.Entry.CreatedAt) })) - require.Equal(t, 3, served(func(w *restriction) { w.in("event_type", []string{string(business.EventDatasourceSourceSynced), "saas.none.none"}) })) + require.Equal(t, 3, served(func(w *restriction) { + w.in("event_type", []string{string(business.EventDatasourceSourceSynced), "saas.none.none"}) + })) require.Zero(t, served(func(w *restriction) { w.eq("resource_id", `session-1" OR TRUE OR "`+"\n\u00e9\\") })) } diff --git a/module/services/accounts/code/pkg/infra/postgres_audit_history_test.go b/module/services/accounts/code/pkg/infra/postgres_audit_history_test.go index 4cb2d3841..8a31110db 100644 --- a/module/services/accounts/code/pkg/infra/postgres_audit_history_test.go +++ b/module/services/accounts/code/pkg/infra/postgres_audit_history_test.go @@ -195,7 +195,7 @@ func runHistoryCopy(t *testing.T, entries []business.AuditEntry) { require.Equal(t, want, got) } - report, err = copier.Run(testCtx, business.AuditHistoryCopyOptions{ConfirmDrop: true, VerifyOnly: true, ExpectedPartitionsSHA256:report.PartitionsSHA256}) + report, err = copier.Run(testCtx, business.AuditHistoryCopyOptions{ConfirmDrop: true, VerifyOnly: true, ExpectedPartitionsSHA256: report.PartitionsSHA256}) require.NoError(t, err) require.Positive(t, report.Dropped) require.Equal(t, through, report.Cutoff) From ac94469b8b214abed019f05edf2171c0d3dca347 Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Mon, 5 Oct 2026 16:33:28 -0400 Subject: [PATCH 17/69] docs(adr): correct ADR 0009 settings, retention and relay ordering Settings: AUDIT_CLICKHOUSE_DSN carries the user and password, so it is a secret delivered as one rather than plain configuration; an s3:// archive needs AWS_REGION (and AWS_ENDPOINT_URL_S3 for a compatible store); the relay tuning settings AUDIT_RELAY_BATCH_SIZE and AUDIT_RELAY_MAX_WAIT are listed; the events/content retention ordering is stated as ClickHouse-only. Retention: on BigQuery the events table has no kit-set expiry; on ClickHouse AUDIT_EVENTS_RETENTION_DAYS sets a TTL. The locked archive is the compliance record, security events in full and content events as envelope plus SHA-256. The security class lists what the registry declares and names it the source of truth. Design: the archive is written before the warehouse; history removal drops an explicit list of verified partitions in one locking, re-counting database function; a permanently rejected row moves to a quarantine table; the queue is monitored under every sink; BigQuery reads hold bounded memory; the role-catalog importer resolves the sink as the service does. Cross-references: the tee is option C of ADR 0006's Options considered, not B; the sink loader is audit_sink.go and auditsink.Load, not work.go. Co-Authored-By: Claude Sonnet 5.5 --- .../0006-audit-sink-and-retention-tiers.md | 2 +- module/docs/adr/0009-audit-store-swap.md | 170 ++++++++++++------ 2 files changed, 121 insertions(+), 51 deletions(-) diff --git a/module/docs/adr/0006-audit-sink-and-retention-tiers.md b/module/docs/adr/0006-audit-sink-and-retention-tiers.md index b081f354a..59f998675 100644 --- a/module/docs/adr/0006-audit-sink-and-retention-tiers.md +++ b/module/docs/adr/0006-audit-sink-and-retention-tiers.md @@ -1,6 +1,6 @@ # ADR 0006: Audit sink and retention tiers — a swappable compliance backend, Postgres as the atomic hot source of truth -- Status: Proposed; superseded in part by [ADR 0009](./0009-audit-store-swap.md) — the sink/tee decision ("a tee, not a swap": Decision item 2, option B) +- Status: Proposed; superseded in part by [ADR 0009](./0009-audit-store-swap.md) — the sink/tee decision ("a tee, not a swap": Decision item 2; option C of Options considered) - Date: 2026-09-02 - Task: prepares the audit performance/size and compliance workstreams (see #430). Refines — for the **sink and retention dimension only** — the diff --git a/module/docs/adr/0009-audit-store-swap.md b/module/docs/adr/0009-audit-store-swap.md index d86918dcb..4b7b4dea8 100644 --- a/module/docs/adr/0009-audit-store-swap.md +++ b/module/docs/adr/0009-audit-store-swap.md @@ -3,8 +3,9 @@ - Status: Proposed - Date: 2026-10-02 - Supersedes, in part: [ADR 0006](./0006-audit-sink-and-retention-tiers.md) — its - sink decision, "a tee, not a swap" (Decision item 2, option B), and for a - deployment that selects a swap value, its retention tiers (Decision item 3). + sink decision, "a tee, not a swap" (Decision item 2; option C of its Options + considered), and for a deployment that selects a swap value, its retention + tiers (Decision item 3). ADR 0006's admission policy and sizing stand. - Refines: [ADR 0003](./0003-typed-audit-event-registry.md) — "analytics stays in Postgres" now holds for the `postgres` default only. The registry and the @@ -56,18 +57,35 @@ Postgres transaction. default for every deployment that does not opt in. `both` — ADR 0006's tee — keeps its current behaviour, for compatibility. `external` stays refused. - A deployment that selects a swap value sets, beside `AUDIT_SINK`: the - warehouse (`AUDIT_BIGQUERY_PROJECT` and `AUDIT_BIGQUERY_DATASET` for - `bigquery`; `AUDIT_CLICKHOUSE_DSN` and `AUDIT_EVENTS_RETENTION_DAYS` for - `clickhouse`), `AUDIT_ARCHIVE_URL` — the locked archive as a URL whose scheme - picks the archive writer (`gs://` for GCS or `s3://` for S3; - any scheme without a writer is refused at startup) — - `AUDIT_CONTENT_RETENTION_DAYS` (the details window of item 5) and - `AUDIT_DEPLOYMENT_ID`, stamped on every record. ClickHouse may additionally - set `AUDIT_CLICKHOUSE_CLUSTER` for replicated tables. The events retention - must be at least the content-detail window. A missing required setting - fails startup. Credentials are the platform's ambient identity (on - Kubernetes, the pod's workload identity), never a key in configuration. + A deployment that selects a swap value sets, beside `AUDIT_SINK`: + - the warehouse: `AUDIT_BIGQUERY_PROJECT` and `AUDIT_BIGQUERY_DATASET` for + `bigquery`; `AUDIT_CLICKHOUSE_DSN` and `AUDIT_EVENTS_RETENTION_DAYS` for + `clickhouse`, which may additionally set `AUDIT_CLICKHOUSE_CLUSTER` for + replicated tables; + - `AUDIT_ARCHIVE_URL` — the locked archive as a URL whose scheme picks the + archive writer (`gs://` for GCS or `s3://` for S3; any + scheme without a writer is refused at startup); + - `AUDIT_CONTENT_RETENTION_DAYS` — the details window of item 5; + - `AUDIT_DEPLOYMENT_ID`, stamped on every record; + - optionally, the relay's tuning: `AUDIT_RELAY_BATCH_SIZE` (the most events + one delivery carries; default 500) and `AUDIT_RELAY_MAX_WAIT` (how long a + partial batch waits for more events before it is delivered anyway; + default 5s, at most 60s). + + A missing required setting fails startup. `AUDIT_EVENTS_RETENTION_DAYS` is a + ClickHouse setting only — BigQuery's events table has no expiry the kit sets + (item 5) — and ClickHouse requires it to be at least the content-detail + window, since details must not outlive their event. + + Credentials never go in configuration. BigQuery and GCS use the platform's + ambient identity (on Kubernetes, the pod's workload identity). An `s3://` + archive uses the AWS SDK's default credential chain and also needs + `AWS_REGION` — startup refuses without it — and `AWS_ENDPOINT_URL_S3` when + the bucket is in an S3-compatible store rather than S3 itself. The one + credential that has no ambient identity is `AUDIT_CLICKHOUSE_DSN`, which + carries the ClickHouse user and password: it is a secret, delivered to the + service the way the deployment delivers its other secrets, never plain + configuration, and never logged or echoed in an error. 2. **Postgres keeps only the transactional queue.** `EmitTx` still writes inside the caller's transaction (and `Emit` inside its own, for the observational @@ -92,11 +110,28 @@ Postgres transaction. every read returns each event once by event id (item 4); - **ordered per organization** (platform-level events form one more ordering key); - - each batch is **appended to the warehouse and written as one object to the - locked archive** (item 5) — the archive copy is written within the batch, - not filled in later; + - each batch is **written as one object to the locked archive first, then + appended to the warehouse** (item 5) — the archive copy is written within + the batch, not filled in later, and a failure between the two writes + repeats the archive write, not the warehouse's. The relay remembers which + writes the batch at the head of the queue has had acknowledged, so an + outage of one side does not write that batch to the other side again; only + a restart forgets, which is the redelivery above; - the batch's queue rows are **deleted only after both writes are - acknowledged**. + acknowledged**; + - a row the warehouse **permanently rejects** — one that fails alone while + the other rows of the same attempt succeed — does not stall the queue. It + moves, in one transaction, to an `audit_event_quarantine` table beside the + queue and is counted in `saas.audit_queue.quarantined`. The kit never + deletes a quarantined row; replaying one is future work. + + The queue is monitored: `saas.audit_queue.depth` and + `saas.audit_queue.oldest_age` report how much is waiting and for how long. + When the queue cannot be read they go absent rather than staying at their + last value, and each failed read counts in + `saas.audit_queue.snapshot_errors`. The monitoring runs under every sink, not + only a swap value: under a non-warehouse sink a non-empty queue — rows left + by an earlier swap, which nothing drains — is logged at startup. The idempotency reservation (`audit_event_idempotency`) stays in Postgres beside the queue row, as it is today. @@ -135,7 +170,8 @@ Postgres transaction. read), the time window, so partitions outside it are pruned, and the event types when the read names them. The service then sorts, pages, deduplicates, joins content details and aggregates the rows it streams back, - with the semantics of the Postgres reads. + with the semantics of the Postgres reads, in bounded memory whatever the size + of the history in the window. The reason is the grants. The service that reads is the service that appends, so it holds `bigquery.tables.updateData`. With @@ -147,9 +183,15 @@ Postgres transaction. 5. **Retention tiers.** Every registered event type has a **retention class** in the typed registry (ADR 0003): - - **security** — authentication, failed authentication, permission and role - changes, admin actions, data exports, configuration changes; - - **content** — everything else. + - **security** — the record of who could get in and what they were allowed to + do: authentication (including failures), credentials issued and revoked, + permission, role, membership, sharing and delegation changes, + administrative and operator actions, data leaving the platform (exports, + replays), and configuration changes. The registry's declarations + (`pkg/business/audit_registry.go`) are the source of truth for which types + are in this class; + - **content** — everything else: what happened to a tenant's own content and + operations. A code-owned type declares its class beside its durability, and cannot be registered without one. A type declared by a solution (its manifest) or a @@ -165,18 +207,26 @@ Postgres transaction. The warehouse keeps two tables: - an **events** table holding the envelope — actor, organization, event type, target id, timestamp — plus a hash of the details, and the full - details as well for security-class types; kept for the compliance window. - An event's outcome stays where it is today: in the event type's name - (`saas.approval.denied`) or in the payload's `outcome` field; + details as well for security-class types. An event's outcome stays where it + is today: in the event type's name (`saas.approval.denied`) or in the + payload's `outcome` field. How long the table keeps a row depends on the + warehouse. On ClickHouse, `AUDIT_EVENTS_RETENTION_DAYS` sets the table's + TTL. On BigQuery the kit sets no expiry on the events table; a deployment + that wants one sets a table or partition expiration on its dataset; - a **details** table holding the full details of content-class types, kept - for a shorter, configurable window. - - A **locked object-storage archive** holds the compliance copy, one object - per relay batch: full for security-class events, content-free (envelope plus - hash) for content-class events. Locked means write-once under a retention - lock the operator cannot shorten. The first two writers target GCS Bucket - Lock and S3 Object Lock. Azure Blob immutability policies still need a - writer; an on-premises S3-compatible lock must be qualified before use. + for the shorter content window, `AUDIT_CONTENT_RETENTION_DAYS` — a + partition expiration on BigQuery, a table TTL on ClickHouse. + + The **compliance record is the locked object-storage archive**, one object + per relay batch: the full event for security-class types, and for + content-class types the envelope plus the SHA-256 of the details, never the + details themselves. Locked means write-once under a retention lock the + operator cannot shorten, set to the deployment's compliance window (ADR + 0006's sizing assumes seven years). The warehouse's events table serves + reads; it is not what a compliance regime relies on. The first two writers + target GCS Bucket Lock and S3 Object Lock. Azure Blob immutability policies + still need a writer; an on-premises S3-compatible lock must be qualified + before use. Archive objects are named per batch. A retried batch writes a new object, so the archive may hold an event more than once. Every archive reader — @@ -184,7 +234,8 @@ Postgres transaction. deduplicates by event id, and the per-event hash makes duplicates provably identical. - Windows are deployment configuration, not code. + The content window is deployment configuration, not code; so is the lock + period of the archive bucket, and the events TTL on ClickHouse. 6. **ClickHouse operating requirements.** - Inserts are batched — the relay's batches, `async_insert`, or both. A @@ -193,18 +244,24 @@ Postgres transaction. - A single-node deployment uses MergeTree. When `AUDIT_CLICKHOUSE_CLUSTER` is set, tables use `ReplicatedMergeTree` with ClickHouse Keeper; the cluster must exist before the adapter starts. - - Tables are partitioned by month, and table TTL expires each tier at its - window. + - Tables are partitioned by month, and a table TTL expires each table at its + window: `AUDIT_EVENTS_RETENTION_DAYS` for events, + `AUDIT_CONTENT_RETENTION_DAYS` for details. - Growing past one shard is a planned operation, not an automatic one — or the deployment uses a managed ClickHouse with shared storage. 7. **History migration.** Switching a deployment from `postgres` to a swap value includes a one-time copy of the existing `audit_events` rows into the warehouse and the archive, verified per organization by row counts and by - per-row hashes. Once verified, the copied monthly partitions are detached - and dropped — the mechanism `RunRetention` already uses — so no row is - deleted and the append-only triggers stay as they are. From then on - `audit_events` receives no new rows, and Postgres holds only the queue. + per-row hashes. Once verified, the copied monthly partitions are removed + with `DROP TABLE`, as `RunRetention` drops expired ones, so no row is + deleted and the append-only triggers stay as they are. The removal is one + database function (migration 20) that takes an explicit list of the + verified partitions, locks them, re-counts each against the count that was + verified, and refuses on any mismatch — a partition that gained a row since + verification is never dropped. `--through`, the cutoff of the copy, accepts + completed months only. From then on `audit_events` receives no new rows, and + Postgres holds only the queue. 8. **Conformance.** Every adapter passes one shared test suite: read deduplication (after a batch is appended twice, every read returns each @@ -251,10 +308,15 @@ The sink shape itself: **What changes in code** (paths under `module/services/accounts/code/`): -- `work.go` — `configuredAuditSink` admits `bigquery` and `clickhouse`, builds - the adapter and the archive writer, and starts the relay. The `external` - refusal stays; its message, which today says Postgres is the source of truth - under every value, is reworded. +- `audit_sink.go` and `pkg/auditstore/auditsink` — `auditsink.Load` admits + `bigquery` and `clickhouse`, reads and validates their settings, and + `auditsink.Open` builds the adapter and the archive writer; `audit_sink.go` + starts the relay on them. The `external` refusal stays; its message, which + today says Postgres is the source of truth under every value, is reworded. + The one-time history copy and the role-catalog importer resolve the sink + through the same `auditsink.Load` as the service, so a deployment configures + the swap once and neither tool writes to a different store than the service + reads; the importer never falls back silently to `audit_events`. - `pkg/business/audit.go` — under a swap value, the emitter's write inserts into the queue table in place of `audit_events`, for org-scoped and platform-level entries alike. `EmitTx`'s transaction contract, `VerifyAuditWiring` and the @@ -279,13 +341,18 @@ The sink shape itself: class, required the way `Durability` is. A declared type's class comes from its declaration — the manifest event, or the module declaration message — and is stored on its `audit_event_types` row, `content` when it states none. -- `pkg/business/retention.go` — partition drop keeps running for `audit_events`, - and is also how the history migration (item 7) removes copied partitions. - Under a swap value, tier expiry belongs to the warehouse (TTL or table - expiration) and to the archive's lock. +- `pkg/business/retention.go` — partition drop keeps running for `audit_events`; + the history migration (item 7) removes copied partitions with `DROP TABLE` + as well, through its own verified-partition function. Under a swap value, + expiry belongs to the warehouse where it sets one — the details table on + both, the events table on ClickHouse only — and to the archive's lock. - `store/migrations` — a new queue table, under the same tenant row-level security as `audit_events` (`audit_events_tenant`, in `1_baseline.up.sql`), - with delete granted to the relay. `audit_events` itself is untouched: its + with delete granted to the relay; its down migration refuses to drop a + non-empty queue, since the rows in it are audit records no warehouse holds + yet. A second migration (21) adds the `audit_event_quarantine` table of item + 2, and migration 20 the partition-removal function of item 7. `audit_events` + itself is untouched: its `audit_events_no_delete` and `audit_events_no_update` triggers and its `SELECT`/`INSERT`-only grants stay as they are. Stored history in the warehouse is guarded by the service-side scope of item 4, not by these @@ -308,6 +375,9 @@ The sink shape itself: - Under a swap value, reads trail writes: an event is listed once the relay has delivered it, not when its mutation commits. - A deployment that opts in runs two more systems — the warehouse and the - archive bucket — and a relay whose lag needs watching. + archive bucket — and a relay whose lag and quarantine table need watching. +- On BigQuery the events table is never expired by the kit and grows until the + deployment sets an expiry on its dataset; the locked archive, not that table, + is the compliance record. - Each adapter must stay at parity with the Postgres implementation; the conformance suite is what holds it there. From 62b4ba46b112d5847212c1f896ae57b42db3ace5 Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Mon, 5 Oct 2026 17:06:37 -0400 Subject: [PATCH 18/69] fix(audit): set aside the row a warehouse refuses instead of stalling the queue behind it One queued row the warehouse rejects for its own content stopped delivery for every tenant: the relay gave up at the first failed batch, and the runbook forbids deleting queue rows, so everything behind that row waited for as long as the row existed. When a batch write fails the relay now splits the batch and retries the halves, down to single rows. A single row is moved to the new audit_event_quarantine table (migration 21: the whole queued row, the error and the time) only on this chain of evidence: the store refused it alone, a witness write of a row it had already acknowledged was then accepted, and the store refused the row alone a second time. During an outage every write fails, so nothing is set aside, and a search in which no write has been accepted gives up after 16 writes instead of probing every subrange. One statement moves the row (DELETE feeding INSERT), so it is in the queue or the quarantine, never neither, and the verdict is kept in memory until the row leaves the queue so a lost commit does not turn proof back into doubt. The kit never deletes a quarantined row; replay is a follow-up and the docs say so. Archive invariant: an archive treats a precondition failure on an object name as "my own retried batch, same rows". Splitting must therefore never write different rows under a name already used. The archive object is written once per batch, whole, before any splitting, and only the warehouse write is split: both stores key a record by event id and never use the batch name. A retry of the same rows keeps the batch id; any other set of rows takes a new one; rows already archived are not archived again when a batch is retried after partial progress, so a set-aside row is always in the archive already. Also in this change: - every batch of a pass gets its own deadline; the 2-minute timeout used to wrap the whole drain loop. The delete/move that records an acknowledged delivery runs on a context of its own, so a delivery that used its budget still records what it delivered. - the retry backoff starts over after a pass that made progress, which partial delivery now makes common. - the comments claiming the queue never hands a batch over ahead of an earlier event are corrected: the gate is transaction visibility, not sequence order, which is harmless because both stores key by event id and every read orders by event time. Co-Authored-By: Claude Sonnet 5.5 --- .../accounts/code/pkg/business/audit_relay.go | 504 +++++++++++++++--- .../code/pkg/business/audit_relay_metrics.go | 3 + .../business/audit_relay_quarantine_test.go | 287 ++++++++++ .../code/pkg/business/audit_relay_test.go | 105 +++- .../infra/postgres_audit_quarantine_test.go | 195 +++++++ .../code/pkg/infra/postgres_audit_queue.go | 161 ++++-- .../pkg/infra/postgres_audit_queue_test.go | 39 +- .../21_audit_event_quarantine.down.sql | 18 + .../21_audit_event_quarantine.up.sql | 68 +++ 9 files changed, 1212 insertions(+), 168 deletions(-) create mode 100644 module/services/accounts/code/pkg/business/audit_relay_quarantine_test.go create mode 100644 module/services/accounts/code/pkg/infra/postgres_audit_quarantine_test.go create mode 100644 module/services/store/migrations/21_audit_event_quarantine.down.sql create mode 100644 module/services/store/migrations/21_audit_event_quarantine.up.sql diff --git a/module/services/accounts/code/pkg/business/audit_relay.go b/module/services/accounts/code/pkg/business/audit_relay.go index 0408cc719..38fa1864c 100644 --- a/module/services/accounts/code/pkg/business/audit_relay.go +++ b/module/services/accounts/code/pkg/business/audit_relay.go @@ -22,19 +22,56 @@ type QueuedAuditEvent struct { EnqueuedAt time.Time } +// QuarantinedAuditEvent is a queued row the relay set aside, with why. +type QuarantinedAuditEvent struct { + Seq int64 + // Reason is the error the store gave when it refused the row. + Reason string +} + +// AuditDeliveryOutcome is what the relay decided about the rows one Drain +// handed it. The queue applies it in the transaction that read them. +type AuditDeliveryOutcome struct { + // Delivered are the rows the archive and the store both acknowledged. The + // queue deletes them. + Delivered []int64 + // Quarantined are the rows the relay set aside because the store refused + // them while it was accepting others. The queue moves them, whole, into the + // quarantine; it never deletes them. + Quarantined []QuarantinedAuditEvent + // Pending is why any other row stays queued, nil when none does. + Pending error +} + +// AuditDrainResult is how many rows one Drain removed from the queue. +type AuditDrainResult struct { + Delivered int + Quarantined int +} + +// Removed is every row the drain took out of the queue. +func (r AuditDrainResult) Removed() int { return r.Delivered + r.Quarantined } + // AuditQueue is the relay's side of the transactional queue. type AuditQueue interface { - // Drain hands deliver the oldest queued events whose transactions have - // committed — at most limit, in queue order — while holding a lease no other - // relay holds at the same time. When deliver reports true it deletes exactly - // those rows and returns how many; when deliver reports false or an error it - // deletes nothing. A relay that finds the lease held elsewhere returns 0 and - // no error without calling deliver. + // Drain hands deliver the oldest queued events whose writing transactions + // have finished — at most limit, in queue order — while holding a lease no + // other relay holds at the same time. It applies the outcome deliver + // returns in the same transaction: the delivered rows are deleted, the + // quarantined rows are moved to the quarantine, every other row stays. When + // deliver returns an error it changes nothing. When the outcome leaves rows + // queued, Drain returns what it removed together with the outcome's Pending + // error. A relay that finds the lease held elsewhere returns zero and no + // error without calling deliver. // - // The queue never reports an event before every event queued ahead of it has - // committed or rolled back, so a batch cannot overtake an earlier event of - // the same organization that was still in flight. - Drain(ctx context.Context, limit int, deliver func(ctx context.Context, events []QueuedAuditEvent) (bool, error)) (int, error) + // The queue holds back a row while a transaction that began before the row's + // own is still running, so a late commit is not left behind a batch that was + // already handed over. That gate is transaction visibility, not sequence + // order: a row whose transaction took its transaction id earlier but its + // sequence number later can be handed over ahead of a lower sequence number + // still to commit. Nothing downstream depends on delivery order: both stores + // key a record by event id and every read orders by event time. + Drain(ctx context.Context, limit int, deliver func(ctx context.Context, events []QueuedAuditEvent) (AuditDeliveryOutcome, error)) (AuditDrainResult, error) } // Relay defaults. A batch of 500 is BigQuery's recommended streaming request @@ -43,10 +80,28 @@ const ( DefaultAuditRelayBatchSize = 500 MaxAuditRelayBatchSize = 5000 DefaultAuditRelayMaxWait = 5 * time.Second - // auditRelayAttemptTimeout bounds one delivery, which holds a queue - // transaction open across the archive and warehouse writes. - auditRelayAttemptTimeout = 2 * time.Minute - auditRelayMaxBackoff = time.Minute + // MaxAuditRelayMaxWait is the longest a partial batch may be configured to + // wait. The relay-lag alert fires at five minutes, so a wait that long would + // page on a healthy relay; a minute leaves room under it. + MaxAuditRelayMaxWait = time.Minute + // auditRelayBatchTimeout bounds one batch's delivery, which holds a queue + // transaction open across the archive and warehouse writes. Every batch of + // a pass has its own. + auditRelayBatchTimeout = 2 * time.Minute + auditRelayMaxBackoff = time.Minute + // AuditRelayBookkeepingTimeout bounds deleting acknowledged rows once their + // writes are done. It runs on a context of its own so a delivery that used + // its whole batch budget still gets to record what it delivered. + AuditRelayBookkeepingTimeout = 30 * time.Second + // auditRelayMaxSplitWrites bounds the store writes one batch spends + // isolating the rows a refused batch write holds. + auditRelayMaxSplitWrites = 64 + // auditRelayProbeWrites is how many store writes in a row may fail, with none + // accepted, before the relay concludes that the store is unreachable rather + // than refusing rows, and stops searching. + auditRelayProbeWrites = 16 + // auditRelayMaxReasonLength bounds the error text kept with a quarantined row. + auditRelayMaxReasonLength = 1000 ) // AuditRelayConfig wires a relay. @@ -79,10 +134,21 @@ type AuditRelayConfig struct { // returns each event once by event id. // // The archive is written first so that a failure between the two writes -// repeats the archive write, not the store's. +// repeats the archive write, not the store's. The archive object is written +// whole and once: an archive treats a precondition failure on an object name as +// "this batch, already written", so a name must never be written again with +// different rows. Only the store write is split when the store refuses a batch; +// it keys each record by event id and has no use for a batch's name. +// +// A row the store refuses must not hold the rest of the queue behind it. When a +// batch write fails the relay splits the batch and retries the halves, down to +// single rows. A single row is set aside — moved whole to the quarantine, +// never deleted — only when the store refused it alone, accepted another row +// afterwards, and refused it again on a second try: during an outage every +// write fails and nothing is set aside. // -// While it runs, the relay remembers which writes the batch at the head of the -// queue has already had acknowledged, so retrying it during an outage of one +// While it runs, the relay remembers what each queued row has had acknowledged +// and what it decided about it, so retrying a batch during an outage of one // side does not write it to the other side again: a warehouse outage does not // fill the locked archive with copies of one batch, and a lost delete is // retried as a delete. Only a restart forgets, which is the at-least-once case @@ -95,12 +161,17 @@ type AuditRelay struct { deploymentID string batchSize int maxWait time.Duration + batchTimeout time.Duration now func() time.Time newBatchID func() string - // drainMu serializes drains, which share head. + // drainMu serializes drains, which share rows and archiving. drainMu sync.Mutex - head batchProgress + // rows is what the relay knows of each queued row it has handled, by + // sequence number. A row leaves it when the queue no longer returns it. + rows map[int64]*rowState + // archiving is the archive write that may be in flight or unacknowledged. + archiving archiveAttempt mu sync.Mutex started bool @@ -130,8 +201,10 @@ func NewAuditRelay(cfg AuditRelayConfig) (*AuditRelay, error) { deploymentID: cfg.DeploymentID, batchSize: cfg.BatchSize, maxWait: cfg.MaxWait, + batchTimeout: auditRelayBatchTimeout, now: cfg.Now, newBatchID: cfg.NewBatchID, + rows: map[int64]*rowState{}, } if relay.now == nil { relay.now = time.Now @@ -144,81 +217,140 @@ func NewAuditRelay(cfg AuditRelayConfig) (*AuditRelay, error) { // DrainOnce delivers every batch that is due — each full batch, then a partial // one whose oldest event has waited MaxWait — and returns how many events it -// delivered. It stops at the first failure, which leaves the failed batch and -// everything behind it queued, in order, for the next attempt. +// delivered. Each batch has a deadline of its own. It stops at the first batch +// that leaves rows queued, which stay queued, in order, for the next attempt; +// the events it delivered before that, and in that batch, are counted. func (r *AuditRelay) DrainOnce(ctx context.Context) (int, error) { + pass, err := r.drain(ctx, nil) + return pass.Delivered, err +} + +// drain is DrainOnce with a way to end the pass between batches: stopped is +// asked before each batch, and a pass it ends is not a failure. +func (r *AuditRelay) drain(ctx context.Context, stopped func() bool) (AuditDrainResult, error) { r.drainMu.Lock() defer r.drainMu.Unlock() - delivered := 0 + var pass AuditDrainResult for { - n, err := r.queue.Drain(ctx, r.batchSize, r.deliver) - delivered += n - if err != nil { - return delivered, err + if stopped != nil && stopped() { + return pass, nil } - if n > 0 { - // Its rows are gone: nothing about the head batch is worth remembering. - r.head = batchProgress{} + batch, cancel := context.WithTimeout(ctx, r.batchTimeout) + result, err := r.queue.Drain(batch, r.batchSize, r.deliver) + cancel() + pass.Delivered += result.Delivered + pass.Quarantined += result.Quarantined + if err != nil { + return pass, err } // Fewer than a full batch means the queue held no more that was due: // nothing, a partial batch still waiting, a partial batch now delivered, // or a lease held by another relay. - if n < r.batchSize { - return delivered, nil + if result.Removed() < r.batchSize { + return pass, nil } } } -// batchProgress is what the relay knows about the batch at the head of the -// queue: which queued rows it holds, and which of the two writes acknowledged it. -type batchProgress struct { - rows string - batch AuditBatch - archived bool +// rowState is what the relay remembers about one queued row. +type rowState struct { + record AuditRecord + composed bool + // archive is the archive object that holds the row, once one acknowledged. + archive *archiveRef appended bool + // refused is why the row was set aside; empty until it is. + refused string +} + +// archiveRef names the archive object a row was acknowledged in. +type archiveRef struct { + id string + composedAt time.Time +} + +// archiveAttempt is the batch the relay last tried to archive and had not +// acknowledged, so a retry of exactly the same rows keeps its name and any +// other set of rows takes a new one. +type archiveAttempt struct { + key string + batch AuditBatch +} + +// queuedRow pairs a row the queue handed over with what the relay remembers. +type queuedRow struct { + event QueuedAuditEvent + state *rowState } // rowsKey identifies a run of queued rows by their sequence numbers. -func rowsKey(events []QueuedAuditEvent) string { +func rowsKey(rows []*queuedRow) string { var key strings.Builder - for _, event := range events { - key.WriteString(strconv.FormatInt(event.Seq, 36)) + for _, row := range rows { + key.WriteString(strconv.FormatInt(row.event.Seq, 36)) key.WriteByte(',') } return key.String() } +// rowsOf returns the relay's record of each event, forgetting every row the +// queue did not hand over: a row not returned is one already removed. +func (r *AuditRelay) rowsOf(events []QueuedAuditEvent) []*queuedRow { + current := make(map[int64]*rowState, len(events)) + rows := make([]*queuedRow, len(events)) + for i, event := range events { + state := r.rows[event.Seq] + if state == nil { + state = &rowState{} + } + current[event.Seq] = state + rows[i] = &queuedRow{event: event, state: state} + } + r.rows = current + return rows +} + // deliver is the relay's half of one Drain: decide whether the events are due, -// compose them into a batch, and write it to the archive and then the store — -// each only if it has not already acknowledged these exact rows. -func (r *AuditRelay) deliver(ctx context.Context, events []QueuedAuditEvent) (bool, error) { +// compose them, write them to the archive and then the store — each only for +// the rows that have not already had it acknowledged — and report which rows +// were delivered, which were set aside, and why any other stays queued. +func (r *AuditRelay) deliver(ctx context.Context, events []QueuedAuditEvent) (AuditDeliveryOutcome, error) { + rows := r.rowsOf(events) if len(events) == 0 { - return false, nil + return AuditDeliveryOutcome{}, nil } if len(events) < r.batchSize && r.now().Sub(oldestEnqueued(events)) < r.maxWait { - return false, nil + return AuditDeliveryOutcome{}, nil } - if key := rowsKey(events); r.head.rows != key { - batch, err := r.compose(ctx, events) - if err != nil { - return false, err - } - r.head = batchProgress{rows: key, batch: batch} + if err := r.compose(ctx, rows); err != nil { + return settle(rows, err), nil + } + if err := r.archiveRows(ctx, rows); err != nil { + return settle(rows, err), nil } - batch := r.head.batch - if !r.head.archived { - if err := r.archive.WriteAuditBatch(ctx, batch); err != nil { - return false, fmt.Errorf("audit relay: archive batch %s: %w", batch.ID, err) + return settle(rows, r.appendRows(ctx, rows)), nil +} + +// settle reports what is decided about rows: every row acknowledged by both +// sides is delivered, every row judged undeliverable is set aside, and the +// rest stay queued for the reason in pending. +func settle(rows []*queuedRow, pending error) AuditDeliveryOutcome { + var outcome AuditDeliveryOutcome + for _, row := range rows { + switch { + case row.state.appended: + outcome.Delivered = append(outcome.Delivered, row.event.Seq) + case row.state.refused != "": + outcome.Quarantined = append(outcome.Quarantined, QuarantinedAuditEvent{Seq: row.event.Seq, Reason: row.state.refused}) } - r.head.archived = true } - if !r.head.appended { - if err := r.store.AppendAuditBatch(ctx, batch); err != nil { - return false, fmt.Errorf("audit relay: append batch %s: %w", batch.ID, err) + if len(outcome.Delivered)+len(outcome.Quarantined) < len(rows) { + outcome.Pending = pending + if outcome.Pending == nil { + outcome.Pending = errors.New("audit relay: rows remain queued") } - r.head.appended = true } - return true, nil + return outcome } func oldestEnqueued(events []QueuedAuditEvent) time.Time { @@ -231,28 +363,209 @@ func oldestEnqueued(events []QueuedAuditEvent) time.Time { return oldest } -// compose classifies and hashes each event. A registry that cannot be read is -// an error, retried with the batch — never mistaken for an unregistered type. -func (r *AuditRelay) compose(ctx context.Context, events []QueuedAuditEvent) (AuditBatch, error) { +// compose classifies and hashes each row it has not composed. A registry that +// cannot be read is an error, retried with the batch — never mistaken for an +// unregistered type. A row whose own details cannot be serialized is set aside, +// but only when another row composed: with none composing, nothing shows that +// the rows, not the relay, are at fault. +func (r *AuditRelay) compose(ctx context.Context, rows []*queuedRow) error { resolver := NewAuditEventResolver(r.types) - batch := AuditBatch{ - ID: r.newBatchID(), - DeploymentID: r.deploymentID, - ComposedAt: r.now().UTC(), - Records: make([]AuditRecord, 0, len(events)), - } - for _, event := range events { - resolved, err := resolver.Resolve(ctx, event.Entry.EventType) + composed := false + var unreadable []*queuedRow + var first error + for _, row := range rows { + state := row.state + if state.refused != "" { + continue + } + if state.composed { + composed = true + continue + } + resolved, err := resolver.Resolve(ctx, row.event.Entry.EventType) if err != nil { - return AuditBatch{}, err + return err } - record, err := NewAuditRecord(event.Entry, resolved.RetentionClass()) + record, err := NewAuditRecord(row.event.Entry, resolved.RetentionClass()) if err != nil { - return AuditBatch{}, err + if first == nil { + first = err + } + unreadable = append(unreadable, row) + continue } - batch.Records = append(batch.Records, record) + state.record, state.composed = record, true + composed = true + } + if len(unreadable) == 0 { + return nil } - return batch, nil + if !composed { + return first + } + for _, row := range unreadable { + r.setAside(ctx, row, first) + } + return nil +} + +// archiveRows writes the composed rows the archive has not acknowledged as one +// object, whole. The same rows retried keep the batch's name; any other set +// takes a new one. +func (r *AuditRelay) archiveRows(ctx context.Context, rows []*queuedRow) error { + var todo []*queuedRow + for _, row := range rows { + if row.state.composed && row.state.archive == nil && row.state.refused == "" { + todo = append(todo, row) + } + } + if len(todo) == 0 { + return nil + } + if key := rowsKey(todo); r.archiving.key != key { + batch := AuditBatch{ + ID: r.newBatchID(), + DeploymentID: r.deploymentID, + ComposedAt: r.now().UTC(), + Records: make([]AuditRecord, len(todo)), + } + for i, row := range todo { + batch.Records[i] = row.state.record + } + r.archiving = archiveAttempt{key: key, batch: batch} + } + batch := r.archiving.batch + if err := r.archive.WriteAuditBatch(ctx, batch); err != nil { + return fmt.Errorf("audit relay: archive batch %s: %w", batch.ID, err) + } + ref := &archiveRef{id: batch.ID, composedAt: batch.ComposedAt} + for _, row := range todo { + row.state.archive = ref + } + r.archiving = archiveAttempt{} + return nil +} + +// appendRows writes the archived rows the store has not acknowledged. The +// whole run goes first; when the store refuses it, the run is split and the +// halves retried, so a row the store cannot take does not hold the rest back. +func (r *AuditRelay) appendRows(ctx context.Context, rows []*queuedRow) error { + var todo []*queuedRow + for _, row := range rows { + if row.state.archive != nil && !row.state.appended && row.state.refused == "" { + todo = append(todo, row) + } + } + if len(todo) == 0 { + return nil + } + + writes := 0 + write := func(run []*queuedRow) error { + writes++ + first := run[0].state.archive + batch := AuditBatch{ID: first.id, DeploymentID: r.deploymentID, ComposedAt: first.composedAt, Records: make([]AuditRecord, len(run))} + for i, row := range run { + batch.Records[i] = row.state.record + } + if err := r.store.AppendAuditBatch(ctx, batch); err != nil { + return err + } + for _, row := range run { + row.state.appended = true + } + return nil + } + + err := write(todo) + if err == nil { + return nil + } + failure := fmt.Errorf("audit relay: append %d rows of batch %s: %w", len(todo), todo[0].state.archive.id, err) + if len(todo) == 1 { + return failure + } + + var alone []*queuedRow + accepted := 0 + // Search breadth-first, so the first accepted run turns up within a write or + // two when only one row is bad, and an outage ends after a bounded number of + // writes instead of after one per row. + failed := [][]*queuedRow{todo} +search: + for len(failed) > 0 { + var next [][]*queuedRow + for _, run := range failed { + middle := len(run) / 2 + for _, half := range [][]*queuedRow{run[:middle], run[middle:]} { + if ctx.Err() != nil || writes >= auditRelayMaxSplitWrites || (accepted == 0 && writes >= auditRelayProbeWrites) { + break search + } + switch err := write(half); { + case err == nil: + accepted++ + case len(half) == 1: + alone = append(alone, half[0]) + default: + next = append(next, half) + } + } + } + failed = next + } + + // A row is set aside only on this chain of evidence: the store refused it + // alone; afterwards the store took a row it had already acknowledged, again + // (so it was reachable after the refusal, and a duplicate is something every + // read already discards); and it refuses the row alone once more. Without a + // row to write as that witness nothing is set aside. + if witness := lastAppended(rows); len(alone) > 0 && witness != nil && ctx.Err() == nil { + if write([]*queuedRow{witness}) == nil { + for _, refused := range alone { + if ctx.Err() != nil { + break + } + if err := write([]*queuedRow{refused}); err != nil && ctx.Err() == nil { + r.setAside(ctx, refused, err) + } + } + } + } + + unresolved := 0 + for _, row := range todo { + if !row.state.appended && row.state.refused == "" { + unresolved++ + } + } + if unresolved == 0 { + return nil + } + return failure +} + +// lastAppended is the last row the store has acknowledged, nil when none has. +func lastAppended(rows []*queuedRow) *queuedRow { + for i := len(rows) - 1; i >= 0; i-- { + if rows[i].state.appended { + return rows[i] + } + } + return nil +} + +// setAside records the decision that row cannot be delivered. The decision is +// kept until the row leaves the queue, so a lost commit does not turn it back +// into doubt. +func (r *AuditRelay) setAside(ctx context.Context, row *queuedRow, cause error) { + reason := cause.Error() + if len(reason) > auditRelayMaxReasonLength { + reason = reason[:auditRelayMaxReasonLength] + } + row.state.refused = reason + wool.Get(ctx).In("audit.relay").Error("audit event set aside: the store refused it while accepting others; it stays in the quarantine and is not delivered", + wool.Field("event_id", row.event.Entry.ID), wool.Field("event_type", string(row.event.Entry.EventType)), + wool.Field("queue_seq", strconv.FormatInt(row.event.Seq, 10)), wool.ErrField(cause)) } // pollInterval is how often the loop looks for due events: a fifth of MaxWait, @@ -269,8 +582,9 @@ func (r *AuditRelay) pollInterval() time.Duration { } // Start runs the relay loop until Shutdown. A failed delivery is logged and -// retried with a backoff that doubles up to a minute; the failed batch stays -// queued, so nothing behind it is delivered out of order. Start is idempotent. +// retried with a backoff that doubles up to a minute and starts over whenever a +// pass made progress; the rows it could not deliver stay queued. Start is +// idempotent. func (r *AuditRelay) Start(parent context.Context) { r.mu.Lock() if r.started { @@ -291,18 +605,19 @@ func (r *AuditRelay) Start(parent context.Context) { for { // A delivery in flight is not cancelled by Shutdown: cancelling it // between the store write and the queue delete is exactly what turns - // one delivery into two. It is bounded by its own timeout instead. - attempt, cancelAttempt := context.WithTimeout(context.WithoutCancel(ctx), auditRelayAttemptTimeout) - _, err := r.DrainOnce(attempt) - cancelAttempt() + // one delivery into two. Each batch is bounded by its own timeout + // instead, and Shutdown ends the pass between batches. + pass, err := r.drain(context.WithoutCancel(ctx), func() bool { return ctx.Err() != nil }) + backoff = relayBackoffAfter(backoff, pass.Removed(), err) wait := r.pollInterval() if err != nil { - backoff = nextAuditRelayBackoff(backoff) wait = backoff - w.Warn("audit relay delivery failed; the batch stays queued and is retried", + w.Warn("audit relay delivery failed; the rows stay queued and are retried", wool.Field("retry_in", backoff.String()), wool.ErrField(err)) - } else { - backoff = 0 + } + if pass.Quarantined > 0 { + w.Error("audit relay set rows aside in the quarantine", + wool.Field("rows", strconv.Itoa(pass.Quarantined))) } select { case <-ctx.Done(): @@ -313,6 +628,21 @@ func (r *AuditRelay) Start(parent context.Context) { }() } +// relayBackoffAfter is the wait before the next pass: none after a pass that +// succeeded, one doubling further after a failure that made no progress, and +// the first step again after a failure that did — a relay moving rows is not a +// relay that is stuck, and should not be slowed as if it were. +func relayBackoffAfter(previous time.Duration, progress int, err error) time.Duration { + switch { + case err == nil: + return 0 + case progress > 0: + return nextAuditRelayBackoff(0) + default: + return nextAuditRelayBackoff(previous) + } +} + func nextAuditRelayBackoff(previous time.Duration) time.Duration { if previous <= 0 { return time.Second diff --git a/module/services/accounts/code/pkg/business/audit_relay_metrics.go b/module/services/accounts/code/pkg/business/audit_relay_metrics.go index 39f890a3e..cf058d945 100644 --- a/module/services/accounts/code/pkg/business/audit_relay_metrics.go +++ b/module/services/accounts/code/pkg/business/audit_relay_metrics.go @@ -15,6 +15,9 @@ import ( type AuditQueueSnapshot struct { Depth int64 OldestEnqueuedAt *time.Time + // Quarantined is how many rows the relay has set aside, and which nothing + // deletes. + Quarantined int64 } // AuditQueueMetricsSource reads the queue without taking the relay lease. diff --git a/module/services/accounts/code/pkg/business/audit_relay_quarantine_test.go b/module/services/accounts/code/pkg/business/audit_relay_quarantine_test.go new file mode 100644 index 000000000..5dcb09bd7 --- /dev/null +++ b/module/services/accounts/code/pkg/business/audit_relay_quarantine_test.go @@ -0,0 +1,287 @@ +package business + +import ( + "context" + "errors" + "fmt" + "slices" + "sync" + "testing" + "time" + + "github.com/stretchr/testify/require" +) + +// A queued row the store refuses for good must not hold every other tenant's +// events behind it (ADR 0009). The relay isolates it by splitting a refused +// batch, and sets it aside only on proof that the store itself was reachable. + +// rejects makes a store write fail while the batch holds any of ids, the way a +// warehouse refuses a row it cannot accept. +func rejects(ids ...string) func(AuditBatch) error { + return func(batch AuditBatch) error { + for _, record := range batch.Records { + if slices.Contains(ids, record.Entry.ID) { + return fmt.Errorf("row %s rejected: invalid value", record.Entry.ID) + } + } + return nil + } +} + +func queuedIDs(rows []QueuedAuditEvent) []string { + ids := make([]string, len(rows)) + for i, row := range rows { + ids[i] = row.Entry.ID + } + return ids +} + +func TestAuditRelaySetsAsideARowTheStoreRefusesAndDeliversTheRest(t *testing.T) { + f := newRelayFixture(t, 10, time.Second, nil) + ids := f.enqueue(time.Minute, EventSessionRevoked, repeatOrg(teeOrgID, 10)...) + poison := ids[3] + f.warehouse.fail = rejects(poison) + + delivered, err := f.relay.DrainOnce(context.Background()) + require.NoError(t, err, "a refused row that is set aside is not a failed delivery") + require.Equal(t, 9, delivered) + require.Empty(t, f.queue.remaining(), "the rest of the batch is delivered, not held behind the refused row") + require.Len(t, f.warehouse.byID, 9) + require.NotContains(t, f.warehouse.byID, poison) + + set := f.queue.quarantined() + require.Equal(t, []string{poison}, queuedIDs(set)) + + require.Len(t, f.archive.order, 1, "the archive holds the batch once, whole") + require.ElementsMatch(t, ids, f.archive.eventIDs(), "including the row the store refused") +} + +func TestAuditRelayNeverSetsAsideAnythingDuringAnOutage(t *testing.T) { + f := newRelayFixture(t, 16, time.Second, nil) + f.enqueue(time.Minute, EventSessionRevoked, repeatOrg(teeOrgID, 16)...) + writes := 0 + f.warehouse.fail = func(AuditBatch) error { + writes++ + return errors.New("warehouse unavailable") + } + + delivered, err := f.relay.DrainOnce(context.Background()) + require.ErrorContains(t, err, "warehouse unavailable") + require.Zero(t, delivered) + require.Len(t, f.queue.remaining(), 16, "everything stays queued") + require.Empty(t, f.queue.quarantined(), "a failure that no other row's success contradicts proves nothing about the row") + require.LessOrEqual(t, writes, 16, "and the search for a refused row gives up instead of writing every subrange") +} + +func TestAuditRelayDoesNotSetAsideRowsThatFailAfterTheStoreWentDown(t *testing.T) { + f := newRelayFixture(t, 8, time.Second, nil) + ids := f.enqueue(time.Minute, EventSessionRevoked, repeatOrg(teeOrgID, 8)...) + // The whole batch fails, the first half is accepted, then the store is gone. + calls := 0 + f.warehouse.fail = func(AuditBatch) error { + calls++ + if calls == 2 { + return nil + } + return errors.New("warehouse unavailable") + } + + delivered, err := f.relay.DrainOnce(context.Background()) + require.Error(t, err) + require.Equal(t, 4, delivered, "what the store accepted is delivered") + require.Empty(t, f.queue.quarantined(), "rows that failed after the last accepted write were not refused for their content") + require.Equal(t, ids[4:], queuedIDs(f.queue.remaining())) +} + +func TestAuditRelayRetriesATransientRefusalBeforeSettingAnythingAside(t *testing.T) { + f := newRelayFixture(t, 4, time.Second, nil) + f.enqueue(time.Minute, EventSessionRevoked, repeatOrg(teeOrgID, 4)...) + // The whole batch, then the first half and the first row alone fail; every + // other write is accepted, the witness and the first row's second try + // included. + script := []bool{false, false, true, false, true} + calls := 0 + f.warehouse.fail = func(AuditBatch) error { + calls++ + if calls <= len(script) && !script[calls-1] { + return errors.New("blip") + } + return nil + } + + delivered, err := f.relay.DrainOnce(context.Background()) + require.NoError(t, err) + require.Equal(t, 4, delivered) + require.Empty(t, f.queue.remaining()) + require.Empty(t, f.queue.quarantined(), "a row the store accepts on a second try was never refused") + require.Len(t, f.warehouse.byID, 4) +} + +func TestAuditRelayRefusalOfEveryRowIsIndistinguishableFromAnOutage(t *testing.T) { + f := newRelayFixture(t, 4, time.Second, nil) + ids := f.enqueue(time.Minute, EventSessionRevoked, repeatOrg(teeOrgID, 4)...) + f.warehouse.fail = rejects(ids...) + + _, err := f.relay.DrainOnce(context.Background()) + require.Error(t, err) + require.Len(t, f.queue.remaining(), 4) + require.Empty(t, f.queue.quarantined(), "with no row accepted there is no evidence the store was reachable") +} + +// An archive treats a precondition failure on an object name as "this batch, +// already written". A name must therefore never be written again with +// different rows. +func TestAuditRelayNeverWritesDifferentRowsUnderAnArchiveName(t *testing.T) { + f := newRelayFixture(t, 8, time.Second, nil) + archive := &namedArchive{} + f.relay.archive = archive + ids := f.enqueue(time.Minute, EventSessionRevoked, repeatOrg(teeOrgID, 8)...) + f.warehouse.fail = rejects(ids[5]) + + _, err := f.relay.DrainOnce(context.Background()) + require.NoError(t, err) + require.Len(t, f.queue.quarantined(), 1) + require.Equal(t, 1, archive.writes(), "splitting the store write does not re-archive the halves") + + // An archive write whose acknowledgement was lost, then more rows queued: + // the grown batch is a different set of rows and takes a name of its own. + g := newRelayFixture(t, 4, time.Second, nil) + lossy := &namedArchive{loseFirstAck: true} + g.relay.archive = lossy + g.enqueue(time.Minute, EventSessionRevoked, repeatOrg(teeOrgID, 2)...) + _, err = g.relay.DrainOnce(context.Background()) + require.Error(t, err) + g.enqueue(time.Minute, EventSessionRevoked, repeatOrg(teeOrgID, 2)...) + _, err = g.relay.DrainOnce(context.Background()) + require.NoError(t, err) + require.Empty(t, lossy.conflicts(), "a name is never reused for different rows") + require.Empty(t, g.queue.remaining()) +} + +// namedArchive behaves as the object-store writers do: writing a name that +// exists succeeds without writing, because it is taken to be the same batch. +type namedArchive struct { + mu sync.Mutex + objects map[string][]string + loseFirstAck bool + written int + reused []string +} + +func (a *namedArchive) WriteAuditBatch(_ context.Context, batch AuditBatch) error { + a.mu.Lock() + defer a.mu.Unlock() + if a.objects == nil { + a.objects = map[string][]string{} + } + ids := make([]string, len(batch.Records)) + for i, record := range batch.Records { + ids[i] = record.Entry.ID + } + if existing, ok := a.objects[batch.ID]; ok { + if !slices.Equal(existing, ids) { + a.reused = append(a.reused, batch.ID) + } + return nil + } + a.objects[batch.ID] = ids + a.written++ + if a.loseFirstAck { + a.loseFirstAck = false + return errors.New("acknowledgement lost") + } + return nil +} + +func (a *namedArchive) writes() int { + a.mu.Lock() + defer a.mu.Unlock() + return a.written +} + +func (a *namedArchive) conflicts() []string { + a.mu.Lock() + defer a.mu.Unlock() + return slices.Clone(a.reused) +} + +// The set-aside decision is kept until the rows leave the queue, so a lost +// commit does not turn proof into doubt. +func TestAuditRelayKeepsItsVerdictWhenTheCommitIsLost(t *testing.T) { + f := newRelayFixture(t, 6, time.Second, nil) + ids := f.enqueue(time.Minute, EventSessionRevoked, repeatOrg(teeOrgID, 6)...) + writes := 0 + refuse := rejects(ids[2]) + f.warehouse.fail = func(batch AuditBatch) error { + writes++ + return refuse(batch) + } + f.queue.crashes = 1 + + _, err := f.relay.DrainOnce(context.Background()) + require.Error(t, err) + require.Len(t, f.queue.remaining(), 6, "the lost commit leaves every row queued") + written := writes + + delivered, err := f.relay.DrainOnce(context.Background()) + require.NoError(t, err) + require.Equal(t, 5, delivered) + require.Equal(t, written, writes, "nothing is written twice and nothing is re-judged") + require.Equal(t, []string{ids[2]}, queuedIDs(f.queue.quarantined())) + require.Empty(t, f.queue.remaining()) +} + +// unencodable is a payload value no store can serialize. +func unencodable() map[string]any { return map[string]any{"bad": make(chan int)} } + +func TestAuditRelaySetsAsideARowItCannotComposeOnlyWhenOthersCan(t *testing.T) { + f := newRelayFixture(t, 3, time.Second, nil) + ids := f.enqueue(time.Minute, EventSessionRevoked, repeatOrg(teeOrgID, 3)...) + f.queue.rows[1].Entry.Payload = unencodable() + + delivered, err := f.relay.DrainOnce(context.Background()) + require.NoError(t, err) + require.Equal(t, 2, delivered) + require.Equal(t, []string{ids[1]}, queuedIDs(f.queue.quarantined())) + require.Empty(t, f.queue.remaining()) + + g := newRelayFixture(t, 2, time.Second, nil) + g.enqueue(time.Minute, EventSessionRevoked, repeatOrg(teeOrgID, 2)...) + for i := range g.queue.rows { + g.queue.rows[i].Entry.Payload = unencodable() + } + _, err = g.relay.DrainOnce(context.Background()) + require.Error(t, err) + require.Len(t, g.queue.remaining(), 2, "when no row composes, nothing shows the rows rather than the relay at fault") + require.Empty(t, g.queue.quarantined()) +} + +// Each batch is bounded on its own: a delivery that is working must not be cut +// off because earlier batches in the same pass used up a shared budget. +func TestAuditRelayGivesEveryBatchItsOwnDeadline(t *testing.T) { + f := newRelayFixture(t, 2, time.Second, nil) + f.enqueue(time.Minute, EventSessionRevoked, repeatOrg(teeOrgID, 6)...) + f.queue.onDrain = func(ctx context.Context) { + time.Sleep(15 * time.Millisecond) + } + + _, err := f.relay.DrainOnce(context.Background()) + require.NoError(t, err) + + deadlines := f.queue.deadlines() + require.Len(t, deadlines, 4, "three full batches and the empty look that ends the pass") + for i, deadline := range deadlines { + require.NotZero(t, deadline, "batch %d has a deadline of its own", i) + } + for i := 1; i < len(deadlines); i++ { + require.True(t, deadlines[i].After(deadlines[i-1]), "batch %d's deadline starts when it does", i) + } +} + +func TestAuditRelayBackoffResetsOnProgress(t *testing.T) { + failure := errors.New("warehouse unavailable") + require.Equal(t, time.Duration(0), relayBackoffAfter(8*time.Second, 3, nil), "success clears it") + require.Equal(t, 16*time.Second, relayBackoffAfter(8*time.Second, 0, failure), "no progress keeps doubling") + require.Equal(t, time.Second, relayBackoffAfter(32*time.Second, 5, failure), "a failure after progress starts over") +} diff --git a/module/services/accounts/code/pkg/business/audit_relay_test.go b/module/services/accounts/code/pkg/business/audit_relay_test.go index f0b33e38b..ecb4602bb 100644 --- a/module/services/accounts/code/pkg/business/audit_relay_test.go +++ b/module/services/accounts/code/pkg/business/audit_relay_test.go @@ -18,42 +18,84 @@ import ( // every organization's events kept in queue order. These run it against // in-memory fakes of the three sides. -// memQueue is a fake AuditQueue with the Postgres queue's delete-on-success -// semantics. +// memQueue is a fake AuditQueue with the Postgres queue's semantics: what the +// relay delivers is deleted, what it sets aside moves to the quarantine, and +// the rest stays. type memQueue struct { mu sync.Mutex rows []QueuedAuditEvent - leased bool // another relay holds the lease - // crashes makes the next n successful deliveries fail before their delete - // commits, the way a crash or a lost connection between the writes and the - // delete would. + held []QueuedAuditEvent // the quarantine + leased bool // another relay holds the lease + // crashes makes the next n removals fail before their commit, the way a + // crash or a lost connection between the writes and the delete would. crashes int drains int + // onDrain runs at the start of every Drain, and ctxDeadlines records the + // deadline each Drain's context carried (zero for none). + onDrain func(context.Context) + ctxDeadlines []time.Time } -func (q *memQueue) Drain(ctx context.Context, limit int, deliver func(context.Context, []QueuedAuditEvent) (bool, error)) (int, error) { +func (q *memQueue) Drain(ctx context.Context, limit int, deliver func(context.Context, []QueuedAuditEvent) (AuditDeliveryOutcome, error)) (AuditDrainResult, error) { + if q.onDrain != nil { + q.onDrain(ctx) + } q.mu.Lock() q.drains++ + deadline, _ := ctx.Deadline() + q.ctxDeadlines = append(q.ctxDeadlines, deadline) if q.leased { q.mu.Unlock() - return 0, nil + return AuditDrainResult{}, nil } events := slices.Clone(q.rows[:min(limit, len(q.rows))]) q.mu.Unlock() - delivered, err := deliver(ctx, events) - if err != nil || !delivered { - return 0, err + outcome, err := deliver(ctx, events) + if err != nil { + return AuditDrainResult{}, err + } + result := AuditDrainResult{Delivered: len(outcome.Delivered), Quarantined: len(outcome.Quarantined)} + if result.Removed() == 0 { + return AuditDrainResult{}, outcome.Pending } q.mu.Lock() defer q.mu.Unlock() if q.crashes > 0 { q.crashes-- - return 0, errors.New("connection lost before the delete committed") + return AuditDrainResult{}, errors.New("connection lost before the commit") + } + gone := map[int64]bool{} + for _, seq := range outcome.Delivered { + gone[seq] = true + } + for _, set := range outcome.Quarantined { + gone[set.Seq] = true + } + var kept []QueuedAuditEvent + for _, row := range q.rows { + switch { + case !gone[row.Seq]: + kept = append(kept, row) + case slices.ContainsFunc(outcome.Quarantined, func(set QuarantinedAuditEvent) bool { return set.Seq == row.Seq }): + q.held = append(q.held, row) + } } - q.rows = q.rows[len(events):] - return len(events), nil + q.rows = kept + return result, outcome.Pending +} + +func (q *memQueue) quarantined() []QueuedAuditEvent { + q.mu.Lock() + defer q.mu.Unlock() + return slices.Clone(q.held) +} + +func (q *memQueue) deadlines() []time.Time { + q.mu.Lock() + defer q.mu.Unlock() + return slices.Clone(q.ctxDeadlines) } func (q *memQueue) remaining() []QueuedAuditEvent { @@ -355,10 +397,9 @@ func TestAuditRelayRetriesOnlyTheUnacknowledgedWrite(t *testing.T) { t.Run("warehouse outage", func(t *testing.T) { f := newRelayFixture(t, 3, time.Second, nil) f.enqueue(time.Minute, EventSessionRevoked, repeatOrg(teeOrgID, 3)...) - outage := 4 + outage := true f.warehouse.fail = func(AuditBatch) error { - if outage > 0 { - outage-- + if outage { return errors.New("warehouse unavailable") } return nil @@ -367,6 +408,7 @@ func TestAuditRelayRetriesOnlyTheUnacknowledgedWrite(t *testing.T) { _, err := f.relay.DrainOnce(context.Background()) require.Error(t, err) } + outage = false delivered, err := f.relay.DrainOnce(context.Background()) require.NoError(t, err) require.Equal(t, 3, delivered) @@ -387,30 +429,40 @@ func TestAuditRelayRetriesOnlyTheUnacknowledgedWrite(t *testing.T) { require.Len(t, f.archive.order, 1) require.Len(t, f.warehouse.appended, 3, "neither side is written twice") }) - t.Run("a grown batch is a new batch", func(t *testing.T) { + t.Run("a grown batch archives only its new rows, as an object of their own", func(t *testing.T) { f := newRelayFixture(t, 4, time.Second, nil) - f.enqueue(time.Minute, EventSessionRevoked, repeatOrg(teeOrgID, 2)...) + first := f.enqueue(time.Minute, EventSessionRevoked, repeatOrg(teeOrgID, 2)...) f.warehouse.fail = func(AuditBatch) error { return errors.New("warehouse unavailable") } _, err := f.relay.DrainOnce(context.Background()) require.Error(t, err) - f.enqueue(time.Minute, EventSessionRevoked, repeatOrg(teeOrgID, 2)...) + second := f.enqueue(time.Minute, EventSessionRevoked, repeatOrg(teeOrgID, 2)...) f.warehouse.fail = nil delivered, err := f.relay.DrainOnce(context.Background()) require.NoError(t, err) require.Equal(t, 4, delivered) require.Len(t, f.archive.order, 2, "different rows are archived as their own object") - require.Len(t, f.archive.objects[f.archive.order[1]].Records, 4) + require.ElementsMatch(t, second, recordIDs(f.archive.objects[f.archive.order[1]]), "and rows already archived are not archived again") + require.ElementsMatch(t, first, recordIDs(f.archive.objects[f.archive.order[0]])) + require.Len(t, f.warehouse.byID, 4) }) } -func TestAuditRelayStopsAtTheFirstFailedBatch(t *testing.T) { +func recordIDs(batch AuditBatch) []string { + ids := make([]string, len(batch.Records)) + for i, record := range batch.Records { + ids[i] = record.Entry.ID + } + return ids +} + +func TestAuditRelayStopsAtTheFirstBatchThatLeavesRowsQueued(t *testing.T) { f := newRelayFixture(t, 2, time.Second, nil) ids := f.enqueue(time.Minute, EventSessionRevoked, repeatOrg(teeOrgID, 6)...) appends := 0 f.warehouse.fail = func(AuditBatch) error { appends++ - if appends == 2 { + if appends >= 2 { return errors.New("warehouse unavailable") } return nil @@ -419,11 +471,8 @@ func TestAuditRelayStopsAtTheFirstFailedBatch(t *testing.T) { delivered, err := f.relay.DrainOnce(context.Background()) require.Error(t, err) require.Equal(t, 2, delivered) - var remaining []string - for _, row := range f.queue.remaining() { - remaining = append(remaining, row.Entry.ID) - } - require.Equal(t, ids[2:], remaining, "the failed batch and everything behind it stay queued, in order") + require.Equal(t, ids[2:], queuedIDs(f.queue.remaining()), "the failed batch and everything behind it stay queued, in order") + require.Empty(t, f.queue.quarantined()) } func TestAuditRelayLeaseHeldElsewhereDeliversNothing(t *testing.T) { diff --git a/module/services/accounts/code/pkg/infra/postgres_audit_quarantine_test.go b/module/services/accounts/code/pkg/infra/postgres_audit_quarantine_test.go new file mode 100644 index 000000000..dd2e1ced8 --- /dev/null +++ b/module/services/accounts/code/pkg/infra/postgres_audit_quarantine_test.go @@ -0,0 +1,195 @@ +//go:build !pure + +package infra_test + +import ( + "context" + "errors" + "testing" + "time" + + "github.com/stretchr/testify/require" + + "accounts/pkg/business" + "accounts/pkg/infra" + "accounts/pkg/infra/storetx" +) + +// The audit relay's quarantine (migration 21): who may touch it, and that moving +// a row into it is one transaction. + +func enqueueN(t *testing.T, orgID string, n int) []business.AuditEntry { + t.Helper() + entries := make([]business.AuditEntry, n) + for i := range entries { + entries[i] = queuedEntry(orgID, business.EventSessionRevoked) + entry := entries[i] + require.NoError(t, testStore.WithOrgTx(testCtx, orgID, func(ctx context.Context) error { + return testStore.EnqueueAuditEvent(ctx, entry) + })) + } + return entries +} + +func TestAuditQueueMovesASetAsideRowToTheQuarantineWholeAndKeepsTheRest(t *testing.T) { + pool := auditRelayPool(t) + emptyAuditQueue(t, pool) + orgID := seedOrg(t, seedUser(t)) + queue, err := infra.NewPostgresAuditQueue(pool) + require.NoError(t, err) + + entries := enqueueN(t, orgID, 5) + before, err := queue.Snapshot(testCtx) + require.NoError(t, err) + + var seqs []int64 + removed, err := queue.Drain(testCtx, 10, func(_ context.Context, events []business.QueuedAuditEvent) (business.AuditDeliveryOutcome, error) { + require.Len(t, events, 5) + for _, event := range events { + seqs = append(seqs, event.Seq) + } + return business.AuditDeliveryOutcome{ + Delivered: []int64{seqs[0], seqs[1], seqs[3]}, + Quarantined: []business.QuarantinedAuditEvent{{Seq: seqs[2], Reason: "row rejected: invalid value"}}, + Pending: errors.New("the fifth row could not be written"), + }, nil + }) + require.ErrorContains(t, err, "fifth row", "what stays queued is reported with what was removed") + require.Equal(t, business.AuditDrainResult{Delivered: 3, Quarantined: 1}, removed) + + remaining := drainAllWithoutDeleting(t, queue) + require.Len(t, remaining, 1, "the row the relay could not write stays queued") + require.Equal(t, entries[4].ID, remaining[0].Entry.ID) + + var ( + quarantinedSeq int64 + eventType, resource, why string + resourceID, actorID, org *string + payload string + createdAt, quarantinedAt time.Time + enqueuedAt time.Time + ) + require.NoError(t, pool.QueryRow(testCtx, ` + SELECT seq, event_type, resource, resource_id, actor_id::text, org_id::text, payload::text, created_at, enqueued_at, quarantined_at, error + FROM audit_event_quarantine WHERE id = $1`, entries[2].ID). + Scan(&quarantinedSeq, &eventType, &resource, &resourceID, &actorID, &org, &payload, &createdAt, &enqueuedAt, &quarantinedAt, &why)) + require.Equal(t, seqs[2], quarantinedSeq, "the queue's sequence number is kept") + require.Equal(t, string(entries[2].EventType), eventType) + require.Equal(t, entries[2].Resource, resource) + require.Equal(t, entries[2].ResourceID, *resourceID) + require.Equal(t, entries[2].ActorID, *actorID) + require.Equal(t, orgID, *org) + require.True(t, entries[2].CreatedAt.Equal(createdAt), "the event's own time is kept") + require.JSONEq(t, `{"ratio": 1.5, "label": ""}`, payload, "and its payload, whole") + require.Equal(t, "row rejected: invalid value", why) + require.WithinDuration(t, time.Now(), quarantinedAt, time.Minute) + require.False(t, enqueuedAt.After(quarantinedAt)) + + after, err := queue.Snapshot(testCtx) + require.NoError(t, err) + require.Equal(t, before.Quarantined+1, after.Quarantined, "the snapshot counts the quarantine") + require.EqualValues(t, 1, after.Depth) + + var inQueue int + require.NoError(t, pool.QueryRow(testCtx, `SELECT count(*) FROM audit_event_queue WHERE id = $1`, entries[2].ID).Scan(&inQueue)) + require.Zero(t, inQueue, "a row is in the queue or the quarantine, never both") +} + +// drainAllWithoutDeleting reads what a drain would be handed and leaves it. +func drainAllWithoutDeleting(t *testing.T, queue *infra.PostgresAuditQueue) []business.QueuedAuditEvent { + t.Helper() + var got []business.QueuedAuditEvent + _, err := queue.Drain(testCtx, 1000, func(_ context.Context, events []business.QueuedAuditEvent) (business.AuditDeliveryOutcome, error) { + got = events + return business.AuditDeliveryOutcome{}, nil + }) + require.NoError(t, err) + return got +} + +func TestAuditQueueQuarantineMoveIsOneTransaction(t *testing.T) { + pool := auditRelayPool(t) + emptyAuditQueue(t, pool) + orgID := seedOrg(t, seedUser(t)) + queue, err := infra.NewPostgresAuditQueue(pool) + require.NoError(t, err) + entries := enqueueN(t, orgID, 2) + + // A quarantine row already holds the sequence number: the move cannot insert. + var firstSeq int64 + require.NoError(t, pool.QueryRow(testCtx, `SELECT min(seq) FROM audit_event_queue`).Scan(&firstSeq)) + _, err = pool.Exec(testCtx, ` + INSERT INTO audit_event_quarantine (seq, xact_id, id, event_type, schema_version, actor_type, resource, created_at, enqueued_at, error) + VALUES ($1, pg_current_xact_id(), $2, 'saas.session.revoked', 1, 'user', 'session', now(), now(), 'earlier')`, + firstSeq, business.NewIDString()) + require.NoError(t, err) + + _, err = queue.Drain(testCtx, 10, func(_ context.Context, events []business.QueuedAuditEvent) (business.AuditDeliveryOutcome, error) { + return business.AuditDeliveryOutcome{ + Delivered: []int64{events[1].Seq}, + Quarantined: []business.QuarantinedAuditEvent{{Seq: events[0].Seq, Reason: "refused"}}, + }, nil + }) + require.Error(t, err) + + remaining := drainAllWithoutDeleting(t, queue) + require.Len(t, remaining, 2, "a move that fails takes the delivered row's delete back with it") + require.Equal(t, entries[0].ID, remaining[0].Entry.ID) + require.Equal(t, entries[1].ID, remaining[1].Entry.ID) +} + +func TestAuditQueueRefusesAnOutcomeThatNamesARowItDidNotRead(t *testing.T) { + pool := auditRelayPool(t) + emptyAuditQueue(t, pool) + orgID := seedOrg(t, seedUser(t)) + queue, err := infra.NewPostgresAuditQueue(pool) + require.NoError(t, err) + enqueueN(t, orgID, 2) + + for name, outcome := range map[string]func([]business.QueuedAuditEvent) business.AuditDeliveryOutcome{ + "a row that is not there": func([]business.QueuedAuditEvent) business.AuditDeliveryOutcome { + return business.AuditDeliveryOutcome{Delivered: []int64{1 << 50}} + }, + "a row delivered and set aside": func(events []business.QueuedAuditEvent) business.AuditDeliveryOutcome { + return business.AuditDeliveryOutcome{ + Delivered: []int64{events[0].Seq}, + Quarantined: []business.QuarantinedAuditEvent{{Seq: events[0].Seq, Reason: "twice"}}, + } + }, + } { + _, err := queue.Drain(testCtx, 10, func(_ context.Context, events []business.QueuedAuditEvent) (business.AuditDeliveryOutcome, error) { + return outcome(events), nil + }) + require.Error(t, err, name) + } + require.Len(t, drainAllWithoutDeleting(t, queue), 2) +} + +func TestAuditQuarantineIsReadableAndWritableOnlyByTheRelaysRole(t *testing.T) { + pool := auditRelayPool(t) + orgID := seedOrg(t, seedUser(t)) + + _, err := pool.Exec(testCtx, `DELETE FROM audit_event_quarantine`) + require.Error(t, err, "the relay cannot remove a quarantined row") + _, err = pool.Exec(testCtx, `UPDATE audit_event_quarantine SET error = 'edited'`) + require.Error(t, err, "nor change one") + + err = testStore.WithOrgTx(testCtx, orgID, func(ctx context.Context) error { + var n int + return storetx.Tx(ctx).QueryRow(ctx, `SELECT count(*) FROM audit_event_quarantine`).Scan(&n) + }) + require.Error(t, err, "a tenant cannot read the quarantine") + + err = testStore.WithControlPlane(testCtx, func(ctx context.Context) error { + var n int + return storetx.Tx(ctx).QueryRow(ctx, `SELECT count(*) FROM audit_event_quarantine`).Scan(&n) + }) + require.Error(t, err, "nor can the control plane") + err = testStore.WithControlPlane(testCtx, func(ctx context.Context) error { + _, err := storetx.Tx(ctx).Exec(ctx, ` + INSERT INTO audit_event_quarantine (seq, xact_id, id, event_type, schema_version, actor_type, resource, created_at, enqueued_at, error) + VALUES (1, pg_current_xact_id(), $1, 'saas.session.revoked', 1, 'user', 'session', now(), now(), 'x')`, business.NewIDString()) + return err + }) + require.Error(t, err, "or write to it") +} diff --git a/module/services/accounts/code/pkg/infra/postgres_audit_queue.go b/module/services/accounts/code/pkg/infra/postgres_audit_queue.go index c185c6028..d603196b5 100644 --- a/module/services/accounts/code/pkg/infra/postgres_audit_queue.go +++ b/module/services/accounts/code/pkg/infra/postgres_audit_queue.go @@ -6,7 +6,6 @@ import ( "encoding/json" "errors" "fmt" - "time" "accounts/pkg/business" @@ -57,22 +56,36 @@ func NewPostgresAuditQueue(pool *pgxpool.Pool) (*PostgresAuditQueue, error) { return &PostgresAuditQueue{pool: pool}, nil } -// Snapshot reads the queue's depth and oldest row for relay-lag telemetry. -// The worker pool has the queue's cross-organization SELECT grant; request -// traffic never receives this pool. +// Snapshot reads the queue's depth, its oldest row and the size of the +// quarantine for relay telemetry, in one statement so the three agree. The +// worker pool has the queue's and the quarantine's cross-organization SELECT +// grant; request traffic never receives this pool. func (q *PostgresAuditQueue) Snapshot(ctx context.Context) (business.AuditQueueSnapshot, error) { - var depth int64 - var oldest *time.Time - err := q.pool.QueryRow(ctx, `SELECT count(*), min(enqueued_at) FROM public.audit_event_queue`).Scan(&depth, &oldest) + var snapshot business.AuditQueueSnapshot + err := q.pool.QueryRow(ctx, ` + SELECT (SELECT count(*) FROM public.audit_event_queue), + (SELECT min(enqueued_at) FROM public.audit_event_queue), + (SELECT count(*) FROM public.audit_event_quarantine)`, + ).Scan(&snapshot.Depth, &snapshot.OldestEnqueuedAt, &snapshot.Quarantined) if err != nil { return business.AuditQueueSnapshot{}, fmt.Errorf("audit queue: observe: %w", err) } - return business.AuditQueueSnapshot{Depth: depth, OldestEnqueuedAt: oldest}, nil + return snapshot, nil } // auditQueueSelectSQL reads the oldest queued events whose writing -// transactions are below the snapshot's xmin — every transaction that old has -// finished, so nothing still in flight can later commit a row ahead of these. +// transactions took their ids before the oldest transaction still running +// (the snapshot's xmin): every transaction that old has finished, so its rows +// are complete. The gate is transaction visibility, not sequence order. A row +// can carry a lower sequence number than one already handed over when its +// transaction took its id later; that is harmless, because both stores key a +// record by event id and every read orders by event time. +// +// The same gate holds the whole queue behind a write transaction left open (a +// stuck session, a long migration), and behind rows whose transaction ids are +// ahead of the cluster's after a logical restore into a new cluster. Both +// stall delivery without losing anything; MEASUREMENT_RUNBOOKS.md (`audit-relay`) +// says how to tell them apart. const auditQueueSelectSQL = ` SELECT seq, id::text, event_type, schema_version, COALESCE(actor_id::text, ''), actor_type, resource, @@ -86,34 +99,37 @@ const auditQueueSelectSQL = ` LIMIT $1` // Drain implements business.AuditQueue. The whole delivery runs in one -// transaction holding the relay lease: read the batch, hand it to deliver, -// delete exactly its rows, commit. A delivery that fails, a crash, or a lost -// connection rolls the transaction back and leaves every row queued. +// transaction holding the relay lease: read the batch, hand it to deliver, then +// apply what deliver decided — delete the delivered rows and move the +// quarantined ones into audit_event_quarantine, in the same transaction — and +// commit. A delivery that fails, a crash, or a lost connection rolls the +// transaction back and leaves every row queued. func (q *PostgresAuditQueue) Drain( ctx context.Context, limit int, - deliver func(ctx context.Context, events []business.QueuedAuditEvent) (bool, error), -) (int, error) { + deliver func(ctx context.Context, events []business.QueuedAuditEvent) (business.AuditDeliveryOutcome, error), +) (business.AuditDrainResult, error) { + var none business.AuditDrainResult if limit < 1 { - return 0, fmt.Errorf("audit queue: limit must be positive, got %d", limit) + return none, fmt.Errorf("audit queue: limit must be positive, got %d", limit) } tx, err := q.pool.Begin(ctx) if err != nil { - return 0, err + return none, err } defer tx.Rollback(context.WithoutCancel(ctx)) //nolint:errcheck // rollback after commit is a no-op var leased bool if err := tx.QueryRow(ctx, `SELECT pg_try_advisory_xact_lock(hashtextextended($1, 0))`, auditRelayLeaseKey).Scan(&leased); err != nil { - return 0, fmt.Errorf("audit queue: take relay lease: %w", err) + return none, fmt.Errorf("audit queue: take relay lease: %w", err) } if !leased { - return 0, nil + return none, nil } rows, err := tx.Query(ctx, auditQueueSelectSQL, limit) if err != nil { - return 0, fmt.Errorf("audit queue: read: %w", err) + return none, fmt.Errorf("audit queue: read: %w", err) } var events []business.QueuedAuditEvent for rows.Next() { @@ -132,43 +148,112 @@ func (q *PostgresAuditQueue) Drain( &entry.ClientID, &event.EnqueuedAt, ); err != nil { rows.Close() - return 0, fmt.Errorf("audit queue: scan: %w", err) + return none, fmt.Errorf("audit queue: scan: %w", err) } entry.EventType = business.EventType(eventType) entry.CreatedAt = entry.CreatedAt.UTC() event.EnqueuedAt = event.EnqueuedAt.UTC() if entry.Payload, err = decodeQueuedPayload(payload); err != nil { rows.Close() - return 0, fmt.Errorf("audit queue: payload of event %s: %w", entry.ID, err) + return none, fmt.Errorf("audit queue: payload of event %s: %w", entry.ID, err) } events = append(events, event) } rows.Close() if err := rows.Err(); err != nil { - return 0, fmt.Errorf("audit queue: read: %w", err) + return none, fmt.Errorf("audit queue: read: %w", err) } - delivered, err := deliver(ctx, events) - if err != nil || !delivered { - return 0, err + outcome, err := deliver(ctx, events) + if err != nil { + return none, err + } + if len(outcome.Delivered)+len(outcome.Quarantined) == 0 { + return none, outcome.Pending + } + if err := checkOutcomeNamesOnlyTheseRows(outcome, events); err != nil { + return none, err } - seqs := make([]int64, len(events)) - for i, event := range events { - seqs[i] = event.Seq + // The writes are acknowledged, so recording them is worth finishing even when + // the delivery used up its own deadline. + bookkeeping, cancel := context.WithTimeout(context.WithoutCancel(ctx), business.AuditRelayBookkeepingTimeout) + defer cancel() + + if len(outcome.Delivered) > 0 { + tag, err := tx.Exec(bookkeeping, `DELETE FROM public.audit_event_queue WHERE seq = ANY($1)`, outcome.Delivered) + if err != nil { + return none, fmt.Errorf("audit queue: delete delivered rows: %w", err) + } + if tag.RowsAffected() != int64(len(outcome.Delivered)) { + // Only the lease holder deletes, so this is a broken invariant, not a race. + return none, fmt.Errorf("audit queue: deleted %d of %d delivered rows", tag.RowsAffected(), len(outcome.Delivered)) + } } - tag, err := tx.Exec(ctx, `DELETE FROM public.audit_event_queue WHERE seq = ANY($1)`, seqs) - if err != nil { - return 0, fmt.Errorf("audit queue: delete delivered rows: %w", err) + if len(outcome.Quarantined) > 0 { + seqs := make([]int64, len(outcome.Quarantined)) + reasons := make([]string, len(outcome.Quarantined)) + for i, set := range outcome.Quarantined { + seqs[i], reasons[i] = set.Seq, set.Reason + } + tag, err := tx.Exec(bookkeeping, auditQueueQuarantineSQL, seqs, reasons) + if err != nil { + return none, fmt.Errorf("audit queue: quarantine rows: %w", err) + } + if tag.RowsAffected() != int64(len(seqs)) { + return none, fmt.Errorf("audit queue: quarantined %d of %d rows", tag.RowsAffected(), len(seqs)) + } } - if tag.RowsAffected() != int64(len(seqs)) { - // Only the lease holder deletes, so this is a broken invariant, not a race. - return 0, fmt.Errorf("audit queue: deleted %d of %d delivered rows", tag.RowsAffected(), len(seqs)) + if err := tx.Commit(bookkeeping); err != nil { + return none, fmt.Errorf("audit queue: commit delivered batch: %w", err) } - if err := tx.Commit(ctx); err != nil { - return 0, fmt.Errorf("audit queue: commit delivered batch: %w", err) + return business.AuditDrainResult{Delivered: len(outcome.Delivered), Quarantined: len(outcome.Quarantined)}, outcome.Pending +} + +// auditQueueQuarantineSQL moves rows out of the queue into the quarantine in a +// single statement, so a row is in exactly one of the two whatever happens: +// the DELETE's rows feed the INSERT, and a failure of either undoes both. +const auditQueueQuarantineSQL = ` + WITH moved AS ( + DELETE FROM public.audit_event_queue WHERE seq = ANY($1::bigint[]) RETURNING * + ) + INSERT INTO public.audit_event_quarantine ( + seq, xact_id, id, event_type, schema_version, actor_id, actor_type, + resource, resource_id, org_id, payload, ip_address, created_at, + impersonated_by, is_impersonated, client_id, enqueued_at, error + ) + SELECT m.seq, m.xact_id, m.id, m.event_type, m.schema_version, m.actor_id, m.actor_type, + m.resource, m.resource_id, m.org_id, m.payload, m.ip_address, m.created_at, + m.impersonated_by, m.is_impersonated, m.client_id, m.enqueued_at, why.error + FROM moved m + JOIN unnest($1::bigint[], $2::text[]) AS why(seq, error) ON why.seq = m.seq` + +// checkOutcomeNamesOnlyTheseRows refuses an outcome that names a row this drain +// did not read, or names one twice: applying it would delete or move a row no +// one delivered. +func checkOutcomeNamesOnlyTheseRows(outcome business.AuditDeliveryOutcome, events []business.QueuedAuditEvent) error { + read := make(map[int64]bool, len(events)) + for _, event := range events { + read[event.Seq] = true + } + named := func(seq int64) error { + if !read[seq] { + return fmt.Errorf("audit queue: the outcome names row %d, which this drain did not read", seq) + } + delete(read, seq) + return nil + } + for _, seq := range outcome.Delivered { + if err := named(seq); err != nil { + return err + } + } + for _, set := range outcome.Quarantined { + if err := named(set.Seq); err != nil { + return err + } } - return len(events), nil + return nil } // decodeQueuedPayload decodes a queued payload keeping each number's text diff --git a/module/services/accounts/code/pkg/infra/postgres_audit_queue_test.go b/module/services/accounts/code/pkg/infra/postgres_audit_queue_test.go index 64d344a59..51502dc9a 100644 --- a/module/services/accounts/code/pkg/infra/postgres_audit_queue_test.go +++ b/module/services/accounts/code/pkg/infra/postgres_audit_queue_test.go @@ -61,12 +61,21 @@ func queuedEntry(orgID string, eventType business.EventType) business.AuditEntry } } +// deliveredAll is the outcome of a relay that delivered every row it was handed. +func deliveredAll(events []business.QueuedAuditEvent) business.AuditDeliveryOutcome { + var outcome business.AuditDeliveryOutcome + for _, event := range events { + outcome.Delivered = append(outcome.Delivered, event.Seq) + } + return outcome +} + func drainAll(t *testing.T, queue *infra.PostgresAuditQueue) []business.QueuedAuditEvent { t.Helper() var got []business.QueuedAuditEvent - _, err := queue.Drain(testCtx, 1000, func(_ context.Context, events []business.QueuedAuditEvent) (bool, error) { + _, err := queue.Drain(testCtx, 1000, func(_ context.Context, events []business.QueuedAuditEvent) (business.AuditDeliveryOutcome, error) { got = events - return true, nil + return deliveredAll(events), nil }) require.NoError(t, err) return got @@ -192,23 +201,23 @@ func TestAuditQueueDeletesOnlyWhatWasDelivered(t *testing.T) { queue, err := infra.NewPostgresAuditQueue(pool) require.NoError(t, err) - _, err = queue.Drain(testCtx, 3, func(context.Context, []business.QueuedAuditEvent) (bool, error) { - return false, errors.New("warehouse unavailable") + _, err = queue.Drain(testCtx, 3, func(context.Context, []business.QueuedAuditEvent) (business.AuditDeliveryOutcome, error) { + return business.AuditDeliveryOutcome{}, errors.New("warehouse unavailable") }) require.ErrorContains(t, err, "warehouse unavailable") - n, err := queue.Drain(testCtx, 3, func(context.Context, []business.QueuedAuditEvent) (bool, error) { - return false, nil + removed, err := queue.Drain(testCtx, 3, func(context.Context, []business.QueuedAuditEvent) (business.AuditDeliveryOutcome, error) { + return business.AuditDeliveryOutcome{}, nil }) require.NoError(t, err) - require.Zero(t, n, "a batch the relay is not ready to deliver is left queued") + require.Zero(t, removed.Removed(), "a batch the relay is not ready to deliver is left queued") var first []business.QueuedAuditEvent - n, err = queue.Drain(testCtx, 3, func(_ context.Context, events []business.QueuedAuditEvent) (bool, error) { + removed, err = queue.Drain(testCtx, 3, func(_ context.Context, events []business.QueuedAuditEvent) (business.AuditDeliveryOutcome, error) { first = events - return true, nil + return deliveredAll(events), nil }) require.NoError(t, err) - require.Equal(t, 3, n) + require.Equal(t, 3, removed.Delivered) rest := drainAll(t, queue) require.Len(t, rest, 2, "exactly the delivered rows were deleted") @@ -274,15 +283,15 @@ func TestAuditQueueLeaseAdmitsOneRelayAtATime(t *testing.T) { second, err := infra.NewPostgresAuditQueue(auditRelayPool(t)) require.NoError(t, err) - _, err = first.Drain(testCtx, 10, func(ctx context.Context, events []business.QueuedAuditEvent) (bool, error) { + _, err = first.Drain(testCtx, 10, func(ctx context.Context, events []business.QueuedAuditEvent) (business.AuditDeliveryOutcome, error) { require.Len(t, events, 1) - n, err := second.Drain(ctx, 10, func(context.Context, []business.QueuedAuditEvent) (bool, error) { + removed, err := second.Drain(ctx, 10, func(context.Context, []business.QueuedAuditEvent) (business.AuditDeliveryOutcome, error) { t.Fatal("a second relay must not be handed rows while the first holds the lease") - return false, nil + return business.AuditDeliveryOutcome{}, nil }) require.NoError(t, err) - require.Zero(t, n) - return true, nil + require.Zero(t, removed.Removed()) + return deliveredAll(events), nil }) require.NoError(t, err) } diff --git a/module/services/store/migrations/21_audit_event_quarantine.down.sql b/module/services/store/migrations/21_audit_event_quarantine.down.sql new file mode 100644 index 000000000..8a2db7c38 --- /dev/null +++ b/module/services/store/migrations/21_audit_event_quarantine.down.sql @@ -0,0 +1,18 @@ +-- Nothing deletes a quarantined row, and rolling this back would. With rows in +-- the quarantine the rollback refuses: replay or resolve them first. +-- +-- The check reads as the relay's role, the only one the quarantine's row level +-- security admits. +SET LOCAL ROLE app_job_worker; +DO $$ +DECLARE + quarantined bigint; +BEGIN + SELECT count(*) INTO quarantined FROM public.audit_event_quarantine; + IF quarantined > 0 THEN + RAISE EXCEPTION 'audit_event_quarantine still holds % audit events the relay set aside; refusing to drop the only copy of them. Replay or resolve them first.', quarantined; + END IF; +END +$$; +RESET ROLE; +DROP TABLE IF EXISTS public.audit_event_quarantine; diff --git a/module/services/store/migrations/21_audit_event_quarantine.up.sql b/module/services/store/migrations/21_audit_event_quarantine.up.sql new file mode 100644 index 000000000..9cc59fe6c --- /dev/null +++ b/module/services/store/migrations/21_audit_event_quarantine.up.sql @@ -0,0 +1,68 @@ +-- The audit relay's quarantine (ADR 0009). The relay delivers the transactional +-- queue to the warehouse and the archive in batches, and splits a batch the +-- warehouse refuses to find the row it cannot take. A row the warehouse refuses +-- on its own, while it accepts others, would otherwise sit at the head of the +-- queue forever and hold every organization's events behind it; the relay moves +-- it here instead, whole, together with why. +-- +-- Nothing deletes a quarantined row: not the relay, not retention. The row was +-- committed with a change to the platform and is evidence of it. An operator +-- replays or resolves quarantined rows; replay is not built yet, and until it +-- is the table is how the rows are kept and counted (saas.audit_queue.quarantined). +-- +-- It mirrors audit_event_queue column for column, so a row can be put back into +-- the queue as it was. It has no foreign key to the event-type registry, unlike +-- the queue: a quarantined row must outlive the registry entry it was written +-- under, which a retired type's removal would otherwise be blocked by. +CREATE TABLE public.audit_event_quarantine ( + -- The queue's sequence number, kept: it is what the relay's log and the + -- move name the row by. + seq bigint NOT NULL, + xact_id xid8 NOT NULL, + id uuid NOT NULL, + event_type text NOT NULL, + schema_version integer NOT NULL, + actor_id uuid, + actor_type text NOT NULL, + resource text NOT NULL, + resource_id text, + org_id uuid, + payload jsonb DEFAULT '{}'::jsonb NOT NULL, + ip_address text, + created_at timestamp with time zone NOT NULL, + impersonated_by uuid, + is_impersonated boolean DEFAULT false NOT NULL, + client_id text, + enqueued_at timestamp with time zone NOT NULL, + -- When the relay set the row aside, and what the store said when it refused. + quarantined_at timestamp with time zone DEFAULT now() NOT NULL, + error text NOT NULL, + CONSTRAINT audit_event_quarantine_actor_type_check CHECK ((actor_type = ANY (ARRAY['user'::text, 'api_key'::text, 'system'::text, 'agent'::text]))), + CONSTRAINT audit_event_quarantine_event_type_format CHECK ((event_type ~ '^[a-z][a-z0-9_]*(\.[a-z][a-z0-9_]*){2,}$'::text)), + CONSTRAINT audit_event_quarantine_impersonation_identity_complete CHECK (((is_impersonated AND (impersonated_by IS NOT NULL)) OR ((NOT is_impersonated) AND (impersonated_by IS NULL)))), + CONSTRAINT audit_event_quarantine_error_bounded CHECK ((char_length(error) <= 2000)) +); + +ALTER TABLE ONLY public.audit_event_quarantine + ADD CONSTRAINT audit_event_quarantine_pkey PRIMARY KEY (seq); + +ALTER TABLE public.audit_event_quarantine ENABLE ROW LEVEL SECURITY; +ALTER TABLE ONLY public.audit_event_quarantine FORCE ROW LEVEL SECURITY; + +-- Only the relay's role sees or writes the quarantine. Request traffic holds no +-- grant on it. The tenant policy is the one every tenant-scoped relation carries +-- (membership_integrity_findings is the precedent for a relation no request can +-- reach): it keeps the table inside the tenant boundary, so a grant added later +-- by mistake would still show an organization nothing but its own rows, and only +-- for reading. A platform event, with no organization, matches no tenant. +CREATE POLICY audit_event_quarantine_tenant ON public.audit_event_quarantine + FOR SELECT + USING (((org_id)::text = current_setting('app.current_org_id'::text, true))); + +CREATE POLICY app_job_worker_explicit_rows ON public.audit_event_quarantine + TO app_job_worker + USING ((CURRENT_USER = 'app_job_worker'::name)) + WITH CHECK ((CURRENT_USER = 'app_job_worker'::name)); + +-- The relay moves rows in and counts them. It cannot change or remove one. +GRANT SELECT,INSERT ON TABLE public.audit_event_quarantine TO app_job_worker; From 2b5f1b5aa383348be796a7ab15acdb4fe800b89d Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Mon, 5 Oct 2026 17:07:26 -0400 Subject: [PATCH 19/69] fix(audit): watch the audit queue under every sink and refuse a rollback that would drop it Only a warehouse sink built the relay, and with it the only queue monitor. Switching AUDIT_SINK back to postgres left queued events that nothing drained and nothing reported, and migration 18's down dropped the table with a comment as its only guard. Startup now opens the queue under postgres and both as well (never failing startup: refusing to start would take login down for a side table) and logs an error with the number of queued events, saying they reach the warehouse only when a warehouse sink is restored. The depth, age and quarantine series keep reporting for as long as the service runs. Migration 18's down raises when the queue holds a row; migration 21's down does the same for the quarantine. Both read as the relay's role, because the tables force row level security and the migrating session's own view of them cannot be trusted to show every row. The relay's metrics could also lie by omission. A failed read of the queue recorded nothing, so the synchronous gauges kept their last value, often 0, and the lag alert stayed green while the relay's database was gone. depth, oldest_age and the new quarantined gauge are now observable gauges whose callback reports nothing when the latest read failed or is older than twice the interval, and saas.audit_queue.snapshot_errors counts the failed reads. The four names are pinned by a test that also holds the alert pack to the series the monitor exports; the pack gains a telemetry gap alert and a quarantine alert. The runbook now covers the quarantine, rolling AUDIT_SINK back, and the two ways the transaction-id gate holds a queue without an error: a write transaction left open, and rows restored from another cluster whose transaction ids are ahead of the new one's. Co-Authored-By: Claude Sonnet 5.5 --- module/MEASUREMENT_RUNBOOKS.md | 71 ++++- .../accounts/code/audit_queue_observe_test.go | 88 +++++++ module/services/accounts/code/audit_sink.go | 63 ++++- .../code/pkg/business/audit_relay_metrics.go | 133 +++++++--- .../pkg/business/audit_relay_metrics_test.go | 245 +++++++++++++++--- .../pkg/business/audit_stranded_queue_test.go | 36 +++ .../pkg/infra/postgres_audit_rollback_test.go | 116 +++++++++ .../accounts/code/pkg/metrics/slo_pack.json | 16 ++ module/services/accounts/code/work.go | 5 + .../migrations/18_audit_event_queue.down.sql | 24 +- 10 files changed, 719 insertions(+), 78 deletions(-) create mode 100644 module/services/accounts/code/audit_queue_observe_test.go create mode 100644 module/services/accounts/code/pkg/business/audit_stranded_queue_test.go create mode 100644 module/services/accounts/code/pkg/infra/postgres_audit_rollback_test.go diff --git a/module/MEASUREMENT_RUNBOOKS.md b/module/MEASUREMENT_RUNBOOKS.md index 60ad3e95a..bd48fdef2 100644 --- a/module/MEASUREMENT_RUNBOOKS.md +++ b/module/MEASUREMENT_RUNBOOKS.md @@ -33,21 +33,82 @@ provider responses into an incident channel. ## `audit-relay` +Under `AUDIT_SINK=bigquery` or `clickhouse` each audit event is a row in the +Postgres `audit_event_queue`, written in the transaction of the change it +records. The accounts relay delivers it to the locked archive and then the +warehouse, and deletes the row only after both acknowledged. Delivery is +at-least-once: a retry may duplicate a warehouse row or archive object, and +readers deduplicate by event id. + 1. Read `saas.audit_queue.depth` and `saas.audit_queue.oldest_age` together. - An absent series means telemetry is unavailable; a zero age with zero - depth means the queue has drained. Do not infer a healthy relay from a - missing series. + A zero age with zero depth means the queue has drained. **An absent series + means the queue could not be read** (the latest read failed, or none was + made in the last two intervals), not that it is healthy: do not infer a + healthy relay from a missing series. `saas.audit_queue.snapshot_errors` + counts the failed reads, and the monitor runs under every `AUDIT_SINK`, so + the series are present whenever the service is. 2. Check the Accounts audit relay's bounded error logs and the health of its declared warehouse and locked archive. Confirm the deployed `AUDIT_SINK`, deployment id, destination, and workload identity without printing secrets. 3. Restore the failing destination or identity and let the relay retry the transactional queue. Do not delete queue rows, turn off audit emission, - or send around the relay. A retry may duplicate a warehouse row or archive - object; readers deduplicate by event id. + or send around the relay. Queue rows are never destroyed by the kit. 4. Confirm depth and age return to zero, then compare source event ids with distinct warehouse and archive event ids for the affected interval. Check that service reads still enforce organization and deployment scope. +### A row the warehouse refuses (`saas.audit_queue.quarantined`) + +One row the warehouse rejects for its own content must not hold every other +organization's events behind it. When a batch write fails, the relay splits the +batch and retries the halves, down to single rows. A single row is set aside only +when the warehouse refused it alone, accepted other rows afterwards, and refused +it again on a second try; during an outage every write fails and nothing is set +aside. The archive is written once, whole, before any of this, so a set-aside row +is already in the archive. + +A set-aside row moves, whole and with the error, into `audit_event_quarantine` +in the same transaction that deletes the delivered rows. The relay logs each at +error level (event id, type and queue sequence number, never the payload). +`saas.audit_queue.quarantined` counts the rows there and the +`audit_relay_quarantine` alert fires while it is above zero. The kit never +deletes a quarantined row. Replaying or resolving quarantined rows is not built +yet: until it is, read the table as the database owner (the relay's role may read +it), fix the cause, and keep the rows. + +### Switching `AUDIT_SINK` back to `postgres` (or `both`) + +Nothing drains the queue under `postgres` or `both`, and rows left in it are not +lost: they reach the warehouse only when a warehouse sink is restored. The +service does not refuse to start (that would take login down). At startup it logs +an error with the number of queued events, and the depth and age series keep +reporting them. To roll back for good, first restore the warehouse sink and let +the relay drain the queue to zero. + +Migration 18's down refuses (`audit_event_queue still holds N queued audit +events`) while any row is queued, and migration 21's down refuses while any row +is quarantined: rolling the schema back would drop the only copy of those events. +Do not work around the refusal by deleting rows. + +### The queue is full but nothing is delivered and nothing is logged + +The relay hands out only rows whose writing transaction is older than every +transaction still running. Two conditions hold the whole queue behind that gate +without an error: depth and age rise, `snapshot_errors` stays at zero, and the +relay logs nothing. + +- A write transaction left open: a stuck session, a long migration, a forgotten + `BEGIN`. Look for the oldest `xact_start` in `pg_stat_activity` and end that + transaction; the queue drains by itself behind it. +- A logical restore into a new cluster: restored rows carry transaction ids from + the old cluster, which can be ahead of the new cluster's, so the gate never + opens for them. Compare `min(xact_id)` and `max(xact_id)` of the queue with + `pg_snapshot_xmin(pg_current_snapshot())`; rows whose `xact_id` is at or past + `pg_snapshot_xmax(pg_current_snapshot())` came from another cluster. They are + committed and complete, and re-stamping their `xact_id` below the new + cluster's horizon (an owner-level update; the relay's role cannot) releases + them. Nothing is lost while they wait. + ## `integrations` 1. Separate transport failure, timeout, rate limiting, authentication, diff --git a/module/services/accounts/code/audit_queue_observe_test.go b/module/services/accounts/code/audit_queue_observe_test.go new file mode 100644 index 000000000..9792cdabd --- /dev/null +++ b/module/services/accounts/code/audit_queue_observe_test.go @@ -0,0 +1,88 @@ +package main + +import ( + "context" + "errors" + "sync" + "testing" + "time" + + "accounts/pkg/business" + + "github.com/codefly-dev/core/wool" + "github.com/stretchr/testify/require" +) + +// Events left in the audit queue by a warehouse sink that was switched back are +// delivered only when a warehouse sink returns. Startup says so, with the count, +// and does not refuse to start: that would take login down for a side table. + +type queueSource struct { + snapshot business.AuditQueueSnapshot + err error +} + +func (q queueSource) Snapshot(context.Context) (business.AuditQueueSnapshot, error) { + return q.snapshot, q.err +} + +type logCapture struct { + mu sync.Mutex + logs []*wool.Log +} + +func (c *logCapture) Process(log *wool.Log) { + c.mu.Lock() + defer c.mu.Unlock() + c.logs = append(c.logs, log) +} + +func (c *logCapture) at(level wool.Loglevel) []string { + c.mu.Lock() + defer c.mu.Unlock() + var out []string + for _, log := range c.logs { + if log.Level == level { + out = append(out, log.String()) + } + } + return out +} + +func captureLogs(t *testing.T) *logCapture { + t.Helper() + previous := wool.GlobalLogLevel() + wool.SetGlobalLogLevel(wool.TRACE) + capture := &logCapture{} + wool.SetFallbackLogger(capture) + t.Cleanup(func() { + wool.SetFallbackLogger(nil) + wool.SetGlobalLogLevel(previous) + }) + return capture +} + +func TestStartupReportsQueuedAuditEventsNoSinkWillDeliver(t *testing.T) { + oldest := time.Now().Add(-time.Hour) + queued := queueSource{snapshot: business.AuditQueueSnapshot{Depth: 42, OldestEnqueuedAt: &oldest}} + + logs := captureLogs(t) + warnStrandedAuditQueue(context.Background(), business.AuditSinkPostgres, queued) + errs := logs.at(wool.ERROR) + require.Len(t, errs, 1) + require.Contains(t, errs[0], "42", "the count of stranded events") + require.Contains(t, errs[0], "warehouse AUDIT_SINK", "and what brings them back") + + logs = captureLogs(t) + warnStrandedAuditQueue(context.Background(), business.AuditSinkBigQuery, queued) + require.Empty(t, logs.at(wool.ERROR), "under a warehouse sink the relay drains the queue") + + logs = captureLogs(t) + warnStrandedAuditQueue(context.Background(), business.AuditSinkPostgres, queueSource{}) + require.Empty(t, logs.at(wool.ERROR), "an empty queue is not reported") + + logs = captureLogs(t) + warnStrandedAuditQueue(context.Background(), business.AuditSinkBoth, queueSource{err: errors.New("database unavailable")}) + require.Empty(t, logs.at(wool.ERROR), "an unreadable queue is a warning, not a claim about stranded events") + require.Len(t, logs.at(wool.WARN), 1) +} diff --git a/module/services/accounts/code/audit_sink.go b/module/services/accounts/code/audit_sink.go index ea4c2977c..ac917df35 100644 --- a/module/services/accounts/code/audit_sink.go +++ b/module/services/accounts/code/audit_sink.go @@ -9,9 +9,13 @@ import ( "os" "time" + "github.com/codefly-dev/core/wool" "go.opentelemetry.io/otel" ) +// auditQueueMonitorInterval is how often the queue's depth and age are read. +const auditQueueMonitorInterval = 10 * time.Second + // configuredAuditSink reads AUDIT_SINK and the settings its mode requires // (auditsink.Load). func configuredAuditSink() (auditsink.Config, error) { @@ -62,10 +66,67 @@ func newAuditSwap(ctx context.Context, types business.DeclaredAuditEventTypeRead } var monitor *business.AuditRelayMonitor if metricsEnabled { - monitor, err = business.NewAuditRelayMonitor(queue, otel.Meter("github.com/codefly-dev/module-saas-starter/accounts"), 10*time.Second) + monitor, err = newAuditQueueMonitor(queue) if err != nil { return fail(err) } } return relay, monitor, opened.Store, closeAll, nil } + +func newAuditQueueMonitor(queue *infra.PostgresAuditQueue) (*business.AuditRelayMonitor, error) { + return business.NewAuditRelayMonitor(queue, otel.Meter("github.com/codefly-dev/module-saas-starter/accounts"), auditQueueMonitorInterval) +} + +// observeAuditQueue watches the audit queue under a sink that does not drain it +// (postgres, both), so the queue is never invisible. If a deployment switched +// back from a warehouse sink with events still queued, they are neither lost nor +// delivered — they wait for a warehouse sink to be restored — and the only thing +// that tells an operator so is this: an error at startup with the count, and the +// depth, age and quarantine series for as long as the service runs. +// +// It never fails startup. The queue is the audit trail's business, not login's: +// a service that refuses to start because a side table is unreadable takes +// every user down for a problem that costs them nothing. Any trouble here is a +// warning, and the missing series is itself what the telemetry alert reads. +func observeAuditQueue(ctx context.Context, mode business.AuditSinkMode, metricsEnabled bool) (*business.AuditRelayMonitor, func()) { + w := wool.Get(ctx).In("observeAuditQueue") + pool, err := infra.NewAuditRelayPool(ctx) + if err != nil { + w.Warn("audit queue is not observed: its database pool could not be opened", wool.ErrField(err)) + return nil, func() {} + } + queue, err := infra.NewPostgresAuditQueue(pool) + if err != nil { + pool.Close() + w.Warn("audit queue is not observed", wool.ErrField(err)) + return nil, func() {} + } + warnStrandedAuditQueue(ctx, mode, queue) + if !metricsEnabled { + pool.Close() + return nil, func() {} + } + monitor, err := newAuditQueueMonitor(queue) + if err != nil { + pool.Close() + w.Warn("audit queue metrics are not recorded", wool.ErrField(err)) + return nil, func() {} + } + return monitor, pool.Close +} + +// warnStrandedAuditQueue logs an error when events are queued under a sink that +// will not deliver them. +func warnStrandedAuditQueue(ctx context.Context, mode business.AuditSinkMode, queue business.AuditQueueMetricsSource) { + w := wool.Get(ctx).In("warnStrandedAuditQueue") + stranded, err := business.StrandedAuditQueue(ctx, mode, queue) + if err != nil { + w.Warn("audit queue could not be read at startup", wool.ErrField(err)) + return + } + if stranded > 0 { + w.Error("audit events are queued and AUDIT_SINK does not deliver them: they reach the warehouse only when a warehouse AUDIT_SINK (bigquery or clickhouse) is restored; until then they stay queued and new audit events are written to audit_events", + wool.Field("sink", string(mode)), wool.Field("queued_events", fmt.Sprint(stranded))) + } +} diff --git a/module/services/accounts/code/pkg/business/audit_relay_metrics.go b/module/services/accounts/code/pkg/business/audit_relay_metrics.go index cf058d945..4957f8556 100644 --- a/module/services/accounts/code/pkg/business/audit_relay_metrics.go +++ b/module/services/accounts/code/pkg/business/audit_relay_metrics.go @@ -16,7 +16,7 @@ type AuditQueueSnapshot struct { Depth int64 OldestEnqueuedAt *time.Time // Quarantined is how many rows the relay has set aside, and which nothing - // deletes. + // deletes: it only grows until an operator deals with them. Quarantined int64 } @@ -25,19 +25,35 @@ type AuditQueueMetricsSource interface { Snapshot(context.Context) (AuditQueueSnapshot, error) } -// AuditRelayMonitor records queue depth and the age of the oldest row. A -// failed relay leaves its rows queued, so the age keeps rising until delivery -// succeeds; zero is recorded for an empty queue. +// AuditRelayMonitor reports queue depth, the age of the oldest row and the size +// of the quarantine. A failed relay leaves its rows queued, so the age keeps +// rising until delivery succeeds; zero is reported for an empty queue. +// +// Each series is reported only while it is known. When the latest read of the +// queue failed, or the latest successful one is more than twice the interval +// old, depth, age and quarantine are absent rather than frozen at their last +// value: a gauge that kept "0" through a dead database or a dead monitor would +// keep the lag alert green while the relay was down. The failures are counted. type AuditRelayMonitor struct { - source AuditQueueMetricsSource - depth metric.Int64Gauge - age metric.Float64Gauge - interval time.Duration - now func() time.Time + source AuditQueueMetricsSource + interval time.Duration + now func() time.Time + snapshotErrors metric.Int64Counter + depth metric.Int64ObservableGauge + age metric.Float64ObservableGauge + quarantined metric.Int64ObservableGauge + registration metric.Registration - mu sync.Mutex - cancel context.CancelFunc - done chan struct{} + mu sync.Mutex + observed *observedQueue // nil until the first read, and after a failed one + cancel context.CancelFunc + done chan struct{} +} + +// observedQueue is the latest successful read of the queue and when it was made. +type observedQueue struct { + snapshot AuditQueueSnapshot + at time.Time } func NewAuditRelayMonitor(source AuditQueueMetricsSource, meter metric.Meter, interval time.Duration) (*AuditRelayMonitor, error) { @@ -47,37 +63,88 @@ func NewAuditRelayMonitor(source AuditQueueMetricsSource, meter metric.Meter, in if interval <= 0 { return nil, errors.New("audit relay metrics: interval must be positive") } - depth, err := meter.Int64Gauge("saas.audit_queue.depth", metric.WithDescription("Queued audit events awaiting archive and warehouse delivery.")) - if err != nil { + m := &AuditRelayMonitor{source: source, interval: interval, now: time.Now} + var err error + if m.depth, err = meter.Int64ObservableGauge("saas.audit_queue.depth", + metric.WithDescription("Queued audit events awaiting archive and warehouse delivery. Absent while the queue cannot be read.")); err != nil { return nil, err } - age, err := meter.Float64Gauge("saas.audit_queue.oldest_age", metric.WithUnit("s"), - metric.WithDescription("Age of the oldest queued audit event.")) - if err != nil { + if m.age, err = meter.Float64ObservableGauge("saas.audit_queue.oldest_age", metric.WithUnit("s"), + metric.WithDescription("Age of the oldest queued audit event. Absent while the queue cannot be read.")); err != nil { + return nil, err + } + if m.quarantined, err = meter.Int64ObservableGauge("saas.audit_queue.quarantined", + metric.WithDescription("Audit events the relay set aside because the store refused them. Nothing deletes them. Absent while the queue cannot be read.")); err != nil { + return nil, err + } + if m.snapshotErrors, err = meter.Int64Counter("saas.audit_queue.snapshot_errors", + metric.WithDescription("Reads of the audit queue's depth and age that failed.")); err != nil { + return nil, err + } + if m.registration, err = meter.RegisterCallback(m.observe, m.depth, m.age, m.quarantined); err != nil { return nil, err } - return &AuditRelayMonitor{source: source, depth: depth, age: age, interval: interval, now: time.Now}, nil + return m, nil +} + +// observe reports the latest reading while it is current. +func (m *AuditRelayMonitor) observe(_ context.Context, observer metric.Observer) error { + m.mu.Lock() + observed := m.observed + m.mu.Unlock() + if observed == nil { + return nil + } + now := m.now() + if now.Sub(observed.at) > 2*m.interval { + return nil + } + age := 0.0 + if observed.snapshot.OldestEnqueuedAt != nil { + age = max(0, now.Sub(*observed.snapshot.OldestEnqueuedAt).Seconds()) + } + observer.ObserveInt64(m.depth, observed.snapshot.Depth) + observer.ObserveFloat64(m.age, age) + observer.ObserveInt64(m.quarantined, observed.snapshot.Quarantined) + return nil } func (m *AuditRelayMonitor) RunOnce(ctx context.Context) error { snapshot, err := m.source.Snapshot(ctx) if err != nil { - return fmt.Errorf("audit relay metrics: read queue: %w", err) - } - if snapshot.Depth < 0 || (snapshot.Depth == 0 && snapshot.OldestEnqueuedAt != nil) || + err = fmt.Errorf("audit relay metrics: read queue: %w", err) + } else if snapshot.Depth < 0 || snapshot.Quarantined < 0 || (snapshot.Depth == 0 && snapshot.OldestEnqueuedAt != nil) || (snapshot.Depth > 0 && snapshot.OldestEnqueuedAt == nil) { - return errors.New("audit relay metrics: inconsistent queue snapshot") + err = errors.New("audit relay metrics: inconsistent queue snapshot") } - age := 0.0 - if snapshot.OldestEnqueuedAt != nil { - age = m.now().Sub(*snapshot.OldestEnqueuedAt).Seconds() - if age < 0 { - age = 0 - } + m.mu.Lock() + if err != nil { + m.observed = nil + } else { + m.observed = &observedQueue{snapshot: snapshot, at: m.now()} } - m.depth.Record(ctx, snapshot.Depth) - m.age.Record(ctx, age) - return nil + m.mu.Unlock() + if err != nil { + m.snapshotErrors.Add(ctx, 1) + } + return err +} + +// StrandedAuditQueue reports how many events sit in the audit queue of a +// deployment whose sink does not drain it. Under postgres or both nothing +// writes the queue and nothing reads it, so a row there was queued while a +// warehouse sink was configured and is delivered only when a warehouse sink is +// restored: the sink was switched back with events still waiting. Under a +// warehouse sink the relay drains the queue and nothing is stranded. +func StrandedAuditQueue(ctx context.Context, mode AuditSinkMode, source AuditQueueMetricsSource) (int64, error) { + if mode.Swaps() { + return 0, nil + } + snapshot, err := source.Snapshot(ctx) + if err != nil { + return 0, fmt.Errorf("audit relay metrics: read queue: %w", err) + } + return snapshot.Depth, nil } func (m *AuditRelayMonitor) Start(parent context.Context) { @@ -120,13 +187,13 @@ func (m *AuditRelayMonitor) Shutdown(ctx context.Context) error { cancel, done := m.cancel, m.done m.mu.Unlock() if cancel == nil { - return nil + return m.registration.Unregister() } cancel() select { case <-done: - return nil case <-ctx.Done(): return ctx.Err() } + return m.registration.Unregister() } diff --git a/module/services/accounts/code/pkg/business/audit_relay_metrics_test.go b/module/services/accounts/code/pkg/business/audit_relay_metrics_test.go index 3062d59e1..1ae86022e 100644 --- a/module/services/accounts/code/pkg/business/audit_relay_metrics_test.go +++ b/module/services/accounts/code/pkg/business/audit_relay_metrics_test.go @@ -2,6 +2,10 @@ package business import ( "context" + "errors" + "os" + "regexp" + "strings" "testing" "time" @@ -10,59 +14,228 @@ import ( "go.opentelemetry.io/otel/sdk/metric/metricdata" ) -type auditQueueSnapshotFake struct{ snapshot AuditQueueSnapshot } +type auditQueueSnapshotFake struct { + snapshot AuditQueueSnapshot + err error +} func (f *auditQueueSnapshotFake) Snapshot(context.Context) (AuditQueueSnapshot, error) { - return f.snapshot, nil + return f.snapshot, f.err } -func TestAuditRelayMonitorReportsLagUntilTheQueueClears(t *testing.T) { - now := time.Date(2026, 10, 3, 12, 0, 0, 0, time.UTC) - oldest := now.Add(-6 * time.Minute) - source := &auditQueueSnapshotFake{snapshot: AuditQueueSnapshot{Depth: 7, OldestEnqueuedAt: &oldest}} - reader := sdkmetric.NewManualReader() - provider := sdkmetric.NewMeterProvider(sdkmetric.WithReader(reader)) +// monitorFixture runs a monitor over a fake queue, a manual clock and a manual +// metric reader. +type monitorFixture struct { + source *auditQueueSnapshotFake + reader *sdkmetric.ManualReader + monitor *AuditRelayMonitor + now time.Time +} + +func newMonitorFixture(t *testing.T, interval time.Duration) *monitorFixture { + t.Helper() + f := &monitorFixture{ + source: &auditQueueSnapshotFake{}, + reader: sdkmetric.NewManualReader(), + now: time.Date(2026, 10, 3, 12, 0, 0, 0, time.UTC), + } + provider := sdkmetric.NewMeterProvider(sdkmetric.WithReader(f.reader)) t.Cleanup(func() { require.NoError(t, provider.Shutdown(t.Context())) }) - monitor, err := NewAuditRelayMonitor(source, provider.Meter("audit-relay-test"), time.Second) + monitor, err := NewAuditRelayMonitor(f.source, provider.Meter("audit-relay-test"), interval) require.NoError(t, err) - monitor.now = func() time.Time { return now } - - require.NoError(t, monitor.RunOnce(t.Context())) - require.Equal(t, int64(7), auditDepth(t, reader)) - require.Equal(t, 360.0, auditAge(t, reader)) - - source.snapshot = AuditQueueSnapshot{} - require.NoError(t, monitor.RunOnce(t.Context())) - require.Equal(t, int64(0), auditDepth(t, reader)) - require.Equal(t, 0.0, auditAge(t, reader)) + monitor.now = func() time.Time { return f.now } + t.Cleanup(func() { require.NoError(t, monitor.Shutdown(t.Context())) }) + f.monitor = monitor + return f } -func auditDepth(t *testing.T, reader *sdkmetric.ManualReader) int64 { +func (f *monitorFixture) collect(t *testing.T) map[string]metricdata.Metrics { t.Helper() var exported metricdata.ResourceMetrics - require.NoError(t, reader.Collect(t.Context(), &exported)) + require.NoError(t, f.reader.Collect(t.Context(), &exported)) + byName := map[string]metricdata.Metrics{} for _, scope := range exported.ScopeMetrics { for _, item := range scope.Metrics { - if item.Name == "saas.audit_queue.depth" { - return item.Data.(metricdata.Gauge[int64]).DataPoints[0].Value - } + byName[item.Name] = item } } - t.Fatal("audit queue depth was not exported") - return 0 + return byName } -func auditAge(t *testing.T, reader *sdkmetric.ManualReader) float64 { +// depth, age and quarantined read a gauge's single point, and report whether +// the series has one at all. +func (f *monitorFixture) depth(t *testing.T) (int64, bool) { t.Helper() - var exported metricdata.ResourceMetrics - require.NoError(t, reader.Collect(t.Context(), &exported)) - for _, scope := range exported.ScopeMetrics { - for _, item := range scope.Metrics { - if item.Name == "saas.audit_queue.oldest_age" { - return item.Data.(metricdata.Gauge[float64]).DataPoints[0].Value - } + return int64Point(t, f.collect(t), "saas.audit_queue.depth") +} + +func (f *monitorFixture) quarantined(t *testing.T) (int64, bool) { + t.Helper() + return int64Point(t, f.collect(t), "saas.audit_queue.quarantined") +} + +func (f *monitorFixture) age(t *testing.T) (float64, bool) { + t.Helper() + metric, ok := f.collect(t)["saas.audit_queue.oldest_age"] + if !ok { + return 0, false // an observable series nothing observed is not exported at all + } + points := metric.Data.(metricdata.Gauge[float64]).DataPoints + if len(points) == 0 { + return 0, false + } + require.Len(t, points, 1) + return points[0].Value, true +} + +func (f *monitorFixture) snapshotErrors(t *testing.T) int64 { + t.Helper() + metric, ok := f.collect(t)["saas.audit_queue.snapshot_errors"] + require.True(t, ok, "failed reads of the queue are counted") + var total int64 + for _, point := range metric.Data.(metricdata.Sum[int64]).DataPoints { + total += point.Value + } + return total +} + +func int64Point(t *testing.T, byName map[string]metricdata.Metrics, name string) (int64, bool) { + t.Helper() + metric, ok := byName[name] + if !ok { + return 0, false // an observable series nothing observed is not exported at all + } + points := metric.Data.(metricdata.Gauge[int64]).DataPoints + if len(points) == 0 { + return 0, false + } + require.Len(t, points, 1) + return points[0].Value, true +} + +func TestAuditRelayMonitorReportsLagUntilTheQueueClears(t *testing.T) { + f := newMonitorFixture(t, time.Second) + oldest := f.now.Add(-6 * time.Minute) + f.source.snapshot = AuditQueueSnapshot{Depth: 7, OldestEnqueuedAt: &oldest, Quarantined: 2} + + require.NoError(t, f.monitor.RunOnce(t.Context())) + depth, ok := f.depth(t) + require.True(t, ok) + require.Equal(t, int64(7), depth) + age, ok := f.age(t) + require.True(t, ok) + require.Equal(t, 360.0, age) + quarantined, ok := f.quarantined(t) + require.True(t, ok) + require.Equal(t, int64(2), quarantined, "rows set aside are reported until they are dealt with") + + f.source.snapshot = AuditQueueSnapshot{} + require.NoError(t, f.monitor.RunOnce(t.Context())) + depth, _ = f.depth(t) + require.Equal(t, int64(0), depth) + age, _ = f.age(t) + require.Equal(t, 0.0, age) + quarantined, _ = f.quarantined(t) + require.Equal(t, int64(0), quarantined) +} + +// A gauge that keeps its last value when the queue cannot be read reports a +// healthy relay while the relay is down: the series must go absent instead, +// and the failure be counted. +func TestAuditRelayMonitorDropsItsSeriesWhenTheLatestReadFailed(t *testing.T) { + f := newMonitorFixture(t, time.Second) + oldest := f.now.Add(-time.Minute) + f.source.snapshot = AuditQueueSnapshot{Depth: 7, OldestEnqueuedAt: &oldest} + require.NoError(t, f.monitor.RunOnce(t.Context())) + _, ok := f.depth(t) + require.True(t, ok) + + f.source.err = errors.New("database unavailable") + require.ErrorContains(t, f.monitor.RunOnce(t.Context()), "database unavailable") + _, ok = f.depth(t) + require.False(t, ok, "no depth is reported for a queue that could not be read") + _, ok = f.age(t) + require.False(t, ok, "no age either: a stale zero would read as a drained queue") + _, ok = f.quarantined(t) + require.False(t, ok) + require.Equal(t, int64(1), f.snapshotErrors(t)) + + f.source.err = nil + require.NoError(t, f.monitor.RunOnce(t.Context())) + depth, ok := f.depth(t) + require.True(t, ok, "the series returns with the next successful read") + require.Equal(t, int64(7), depth) + require.Equal(t, int64(1), f.snapshotErrors(t), "and the count of failures does not reset") +} + +func TestAuditRelayMonitorCountsAnInconsistentSnapshotAsAFailure(t *testing.T) { + f := newMonitorFixture(t, time.Second) + f.source.snapshot = AuditQueueSnapshot{Depth: 3} // rows but no oldest row + require.ErrorContains(t, f.monitor.RunOnce(t.Context()), "inconsistent") + _, ok := f.depth(t) + require.False(t, ok) + require.Equal(t, int64(1), f.snapshotErrors(t)) +} + +// A monitor that stopped reading — hung, or its loop gone — must not leave its +// last reading standing: a reading is reported for twice the interval. +func TestAuditRelayMonitorDropsItsSeriesWhenTheReadIsStale(t *testing.T) { + f := newMonitorFixture(t, 10*time.Second) + oldest := f.now.Add(-time.Minute) + f.source.snapshot = AuditQueueSnapshot{Depth: 4, OldestEnqueuedAt: &oldest} + require.NoError(t, f.monitor.RunOnce(t.Context())) + + f.now = f.now.Add(20 * time.Second) + depth, ok := f.depth(t) + require.True(t, ok, "a reading two intervals old is still current") + require.Equal(t, int64(4), depth) + age, _ := f.age(t) + require.Equal(t, 80.0, age, "and the age keeps rising between readings") + + f.now = f.now.Add(time.Second) + _, ok = f.depth(t) + require.False(t, ok, "past two intervals it is stale and reported as absent") + _, ok = f.age(t) + require.False(t, ok) +} + +// The alert pack names the monitor's series in Prometheus's spelling. The names +// are what an alert in another repository is written against, so a series the +// pack mentions must be one the monitor exports, and every series the monitor +// exports must be watched by some alert. +func TestAuditQueueAlertsNameTheSeriesTheMonitorExports(t *testing.T) { + f := newMonitorFixture(t, time.Second) + f.source.err = errors.New("unreadable") + require.Error(t, f.monitor.RunOnce(t.Context())) // creates the counter's series + f.source.err = nil + require.NoError(t, f.monitor.RunOnce(t.Context())) + + exported := map[string]string{} + for name, item := range f.collect(t) { + prom := strings.ReplaceAll(name, ".", "_") + if item.Unit == "s" { + prom += "_seconds" + } + if _, counter := item.Data.(metricdata.Sum[int64]); counter { + prom += "_total" } + exported[prom] = name + } + require.Equal(t, map[string]string{ + "saas_audit_queue_depth": "saas.audit_queue.depth", + "saas_audit_queue_oldest_age_seconds": "saas.audit_queue.oldest_age", + "saas_audit_queue_quarantined": "saas.audit_queue.quarantined", + "saas_audit_queue_snapshot_errors_total": "saas.audit_queue.snapshot_errors", + }, exported, "these four names are a contract with the alerts written against them") + + pack, err := os.ReadFile("../metrics/slo_pack.json") + require.NoError(t, err) + mentioned := map[string]bool{} + for _, name := range regexp.MustCompile(`saas_audit_queue_[a-z_]+`).FindAllString(string(pack), -1) { + mentioned[name] = true + require.Contains(t, exported, name, "the alert pack reads a series the monitor does not export") + } + for name := range exported { + require.True(t, mentioned[name], "%s is exported and watched by no alert", name) } - t.Fatal("audit queue age was not exported") - return 0 } diff --git a/module/services/accounts/code/pkg/business/audit_stranded_queue_test.go b/module/services/accounts/code/pkg/business/audit_stranded_queue_test.go new file mode 100644 index 000000000..ec756a40f --- /dev/null +++ b/module/services/accounts/code/pkg/business/audit_stranded_queue_test.go @@ -0,0 +1,36 @@ +package business + +import ( + "context" + "errors" + "testing" + "time" + + "github.com/stretchr/testify/require" +) + +// Switching AUDIT_SINK back to postgres leaves queued events that no relay +// drains. Startup must say so — with the count — instead of refusing to start +// (which would take login down) or saying nothing. +func TestStrandedAuditQueueCountsRowsNoSinkWillDrain(t *testing.T) { + oldest := time.Now().Add(-time.Hour) + queued := &auditQueueSnapshotFake{snapshot: AuditQueueSnapshot{Depth: 42, OldestEnqueuedAt: &oldest}} + + for _, mode := range []AuditSinkMode{AuditSinkPostgres, AuditSinkBoth} { + stranded, err := StrandedAuditQueue(context.Background(), mode, queued) + require.NoError(t, err) + require.Equal(t, int64(42), stranded, "under %s nothing drains the queue", mode) + } + for _, mode := range []AuditSinkMode{AuditSinkBigQuery, AuditSinkClickHouse} { + stranded, err := StrandedAuditQueue(context.Background(), mode, queued) + require.NoError(t, err) + require.Zero(t, stranded, "under %s the relay drains it", mode) + } + + stranded, err := StrandedAuditQueue(context.Background(), AuditSinkPostgres, &auditQueueSnapshotFake{}) + require.NoError(t, err) + require.Zero(t, stranded, "an empty queue strands nothing") + + _, err = StrandedAuditQueue(context.Background(), AuditSinkPostgres, &auditQueueSnapshotFake{err: errors.New("database unavailable")}) + require.ErrorContains(t, err, "database unavailable", "an unreadable queue is reported, not taken for an empty one") +} diff --git a/module/services/accounts/code/pkg/infra/postgres_audit_rollback_test.go b/module/services/accounts/code/pkg/infra/postgres_audit_rollback_test.go new file mode 100644 index 000000000..29733b6e8 --- /dev/null +++ b/module/services/accounts/code/pkg/infra/postgres_audit_rollback_test.go @@ -0,0 +1,116 @@ +//go:build !pure + +package infra_test + +import ( + "context" + "os" + "strings" + "testing" + + "github.com/jackc/pgx/v5" + "github.com/jackc/pgx/v5/pgxpool" + "github.com/stretchr/testify/require" + + "accounts/pkg/business" +) + +// The queue's and the quarantine's down migrations (18 and 21) drop the only +// copy of whatever rows they hold, so they refuse while any row is there. + +// migrationOwner runs fn on a connection with the authority migrations run +// under, inside a transaction it rolls back, so a down migration can be run +// against the real schema and leave it as it was. +func migrationOwner(t *testing.T, fn func(ctx context.Context, tx pgx.Tx)) { + t.Helper() + run := func(ctx context.Context, conn *pgx.Conn) { + tx, err := conn.Begin(ctx) + require.NoError(t, err) + defer func() { _ = tx.Rollback(ctx) }() + fn(ctx, tx) + } + if raw := os.Getenv("ACCOUNTS_INSTALLER_TEST_DATABASE_URL"); raw != "" { + conn, err := pgx.Connect(testCtx, raw) + require.NoError(t, err) + defer conn.Close(testCtx) + run(testCtx, conn) + return + } + asMigrationOwner(t, func(ctx context.Context, conn *pgxpool.Conn) { run(ctx, conn.Conn()) }) +} + +func readMigration(t *testing.T, name string) string { + t.Helper() + body, err := os.ReadFile("../../../../store/migrations/" + name) + require.NoError(t, err) + return string(body) +} + +// asRole runs sql as role inside the transaction and returns to the session's own. +func asRole(t *testing.T, ctx context.Context, tx pgx.Tx, role, sql string, args ...any) { + t.Helper() + _, err := tx.Exec(ctx, "SET LOCAL ROLE "+role) + require.NoError(t, err) + _, err = tx.Exec(ctx, sql, args...) + require.NoError(t, err) + _, err = tx.Exec(ctx, "RESET ROLE") + require.NoError(t, err) +} + +// Rolling the queue's migration back drops the only copy of every event the +// relay has not delivered, so it must refuse while any row is queued. +func TestRollingBackTheQueueRefusesWhileEventsAreQueued(t *testing.T) { + down := readMigration(t, "18_audit_event_queue.down.sql") + orgID := seedOrg(t, seedUser(t)) + + migrationOwner(t, func(ctx context.Context, tx pgx.Tx) { + asRole(t, ctx, tx, "app_job_worker", `DELETE FROM public.audit_event_queue`) + + _, err := tx.Exec(ctx, "SAVEPOINT empty") + require.NoError(t, err) + _, err = tx.Exec(ctx, down) + require.NoError(t, err, "an empty queue rolls back") + _, err = tx.Exec(ctx, "ROLLBACK TO SAVEPOINT empty") + require.NoError(t, err) + + asRole(t, ctx, tx, "app_control_plane", ` + INSERT INTO public.audit_event_queue (id, event_type, schema_version, actor_type, resource, org_id, created_at) + VALUES ($1, 'saas.session.revoked', 1, 'system', 'session', $2, now())`, business.NewIDString(), orgID) + _, err = tx.Exec(ctx, "SAVEPOINT queued") + require.NoError(t, err) + _, err = tx.Exec(ctx, down) + require.Error(t, err, "a queue with a row in it does not") + require.True(t, strings.Contains(err.Error(), "still holds 1 queued audit events"), err.Error()) + _, err = tx.Exec(ctx, "ROLLBACK TO SAVEPOINT queued") + require.NoError(t, err) + }) +} + +func TestRollingBackTheQuarantineRefusesWhileRowsAreSetAside(t *testing.T) { + down := readMigration(t, "21_audit_event_quarantine.down.sql") + + migrationOwner(t, func(ctx context.Context, tx pgx.Tx) { + countQuarantine := func() int { + _, err := tx.Exec(ctx, "SET LOCAL ROLE app_job_worker") + require.NoError(t, err) + var n int + require.NoError(t, tx.QueryRow(ctx, `SELECT count(*) FROM public.audit_event_quarantine`).Scan(&n)) + _, err = tx.Exec(ctx, "RESET ROLE") + require.NoError(t, err) + return n + } + held := countQuarantine() + + asRole(t, ctx, tx, "app_job_worker", ` + INSERT INTO public.audit_event_quarantine (seq, xact_id, id, event_type, schema_version, actor_type, resource, created_at, enqueued_at, error) + VALUES (9000000000, pg_current_xact_id(), $1, 'saas.session.revoked', 1, 'user', 'session', now(), now(), 'refused')`, business.NewIDString()) + _, err := tx.Exec(ctx, "SAVEPOINT held") + require.NoError(t, err) + _, err = tx.Exec(ctx, down) + require.Error(t, err, "the quarantine is never dropped with rows in it") + require.Contains(t, err.Error(), "audit_event_quarantine still holds") + _, err = tx.Exec(ctx, "ROLLBACK TO SAVEPOINT held") + require.NoError(t, err) + require.Equal(t, held+1, countQuarantine()) + }) +} diff --git a/module/services/accounts/code/pkg/metrics/slo_pack.json b/module/services/accounts/code/pkg/metrics/slo_pack.json index c41fbc4f4..991df2f31 100644 --- a/module/services/accounts/code/pkg/metrics/slo_pack.json +++ b/module/services/accounts/code/pkg/metrics/slo_pack.json @@ -108,6 +108,22 @@ "expression_language": "promql", "expression": "min_over_time(saas_audit_queue_oldest_age_seconds[5m]) > 300" }, + { + "id": "audit_relay_telemetry_gap", + "signal": "saas.audit_queue.snapshot_errors", + "condition": "the audit queue could not be read in the last 5m, or its depth series is absent: the lag alert cannot fire for a queue nobody can see", + "runbook": "audit-relay", + "expression_language": "promql", + "expression": "increase(saas_audit_queue_snapshot_errors_total[5m]) > 0 or absent(saas_audit_queue_depth)" + }, + { + "id": "audit_relay_quarantine", + "signal": "saas.audit_queue.quarantined", + "condition": "the relay holds audit events the warehouse refused; nothing deletes them", + "runbook": "audit-relay", + "expression_language": "promql", + "expression": "max_over_time(saas_audit_queue_quarantined[10m]) > 0" + }, { "id": "integration_failures", "signal": "saas.jobs.completed", diff --git a/module/services/accounts/code/work.go b/module/services/accounts/code/work.go index 99c7a94ba..05efc09b9 100644 --- a/module/services/accounts/code/work.go +++ b/module/services/accounts/code/work.go @@ -593,6 +593,11 @@ func doWork(ctx context.Context) (Clean, error) { return nil, err } service.SetAuditStore(auditStore) + } else { + // Under postgres and both nothing drains the audit queue, but it is still + // watched: events left in it by a warehouse sink that was switched back + // are reported, with their count, instead of waiting unseen. + auditRelayMonitor, closeAuditRelay = observeAuditQueue(ctx, auditSink.Mode, otelMetricProvider != nil) } var auditExportWorker *jobs.Worker diff --git a/module/services/store/migrations/18_audit_event_queue.down.sql b/module/services/store/migrations/18_audit_event_queue.down.sql index 59b5f73ec..822a9c4e9 100644 --- a/module/services/store/migrations/18_audit_event_queue.down.sql +++ b/module/services/store/migrations/18_audit_event_queue.down.sql @@ -1,4 +1,22 @@ --- Removing the queue removes every event still in it. Under the default --- (postgres) and the tee (both) nothing writes it; under a swap value, roll back --- only once the relay has drained it. +-- The queue holds the only copy of an event the relay has not yet delivered, so +-- removing it removes those events. Under the default (postgres) and the tee +-- (both) nothing writes it, and rolling back an empty queue is safe; with rows +-- in it the rollback refuses. Restore a warehouse sink and let the relay drain +-- the queue to depth 0 (MEASUREMENT_RUNBOOKS.md, audit-relay), then roll back. +-- +-- The check reads as the relay's role: the queue forces row level security, and +-- that role's policy admits every row, so the count is the real one rather than +-- whatever the migrating session's own policies happen to show it. +SET LOCAL ROLE app_job_worker; +DO $$ +DECLARE + queued bigint; +BEGIN + SELECT count(*) INTO queued FROM public.audit_event_queue; + IF queued > 0 THEN + RAISE EXCEPTION 'audit_event_queue still holds % queued audit events; refusing to drop the only copy of them. Restore a warehouse AUDIT_SINK and let the audit relay drain the queue to zero first.', queued; + END IF; +END +$$; +RESET ROLE; DROP TABLE IF EXISTS public.audit_event_queue; From 266ed09f9bf3ebc0c21330963bc53c117eec8dec Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Mon, 5 Oct 2026 17:07:41 -0400 Subject: [PATCH 20/69] fix(audit): make the role catalog import refuse an unset AUDIT_SINK and bound the relay settings The import records audit events through the same emitter as the service, so where they go depends on AUDIT_SINK. It read the variable itself and defaulted an empty one to postgres, so a deploy step that did not receive the setting wrote its role events to audit_events on a deployment whose store of record is a warehouse, where nothing reads them, and said nothing. The deployment passes the service's value to the step (service_environment: [AUDIT_SINK]); the import now reads it through the same auditsink package as the service (Mode, which Load uses, and RequireMode) and, unlike the service, has no default: an unset or blank value stops it with a message saying how to set it, before it touches the store. -audit-sink still overrides, and an explicit empty flag is no sink either. I did not make the import call Load: Load demands every setting of a warehouse (the ClickHouse DSN among them), and the deploy step should not inherit credentials that only the relay needs. A deployment that left AUDIT_SINK unset on the service must now set AUDIT_SINK=postgres explicitly; the step's comment says so. Also: - AUDIT_RELAY_MAX_WAIT was accepted up to an hour while the relay-lag alert fires at five minutes, so a legal setting could page on a healthy relay. It is capped at a minute. - retention days were converted with time.Duration(days)*24h, which overflows above about 106,751 days and goes negative. Both retention settings are bounded at a century before they are multiplied. Co-Authored-By: Claude Sonnet 5.5 --- module/deployment/jobs.codefly.yaml | 8 ++- module/services/accounts/AGENTS.md | 21 +++++++ .../code/cmd/role-catalog-import/main.go | 35 ++++++++--- .../code/cmd/role-catalog-import/main_test.go | 62 ++++++++++++++++++ .../pkg/auditstore/auditsink/auditsink.go | 63 +++++++++++++++---- .../auditstore/auditsink/auditsink_test.go | 53 ++++++++++++++++ 6 files changed, 219 insertions(+), 23 deletions(-) diff --git a/module/deployment/jobs.codefly.yaml b/module/deployment/jobs.codefly.yaml index 505a0506b..b0e6a3da0 100644 --- a/module/deployment/jobs.codefly.yaml +++ b/module/deployment/jobs.codefly.yaml @@ -15,7 +15,13 @@ deploy_jobs: # service's AUDIT_SINK: under a swap value (ADR 0009) its events go to the # queue the accounts relay delivers, not to audit_events. AUDIT_SINK is the one # audit setting the importer reads; the rest name destinations only the relay - # writes to. + # writes to. The importer reads it through the same auditsink package as the + # service but, unlike the service, has no default: if the Job does not receive + # the variable it exits before touching the store, rather than guess postgres + # and write its events where a warehouse deployment never reads them. A + # deployment that leaves AUDIT_SINK unset on the service (the default, + # postgres) must set AUDIT_SINK=postgres explicitly so that the step has a + # value to inherit. - name: role-catalog-import service: accounts command: role-catalog-import diff --git a/module/services/accounts/AGENTS.md b/module/services/accounts/AGENTS.md index 99d09e066..0bdbd2685 100644 --- a/module/services/accounts/AGENTS.md +++ b/module/services/accounts/AGENTS.md @@ -86,6 +86,27 @@ survives a restart and reaches every replica, and is only served when it is does (`pkg/auditstore/clickhousestore/reader.go` lists where); `pkg/business/audit_store_parity_test.go` holds every store to one fixture (ClickHouse joins when `AUDIT_CLICKHOUSE_TEST_DSN` names a server). +- **Audit relay and queue** (ADR 0009, `pkg/business/audit_relay.go`, + `pkg/infra/postgres_audit_queue.go`, migrations 18 and 21). Under a swap value + `EmitTx` writes a short-lived `audit_event_queue` row on the caller's + transaction; the relay drains it to the archive and then the warehouse and + deletes the row only after both acknowledged. It is at-least-once, so every + read dedupes by event id. Four rules a change must keep. (1) The archive object + is written once per batch, whole, under a name no other set of rows ever uses + — the object-store writers take a precondition failure on a name to mean "this + batch, already written" — so only the *warehouse* write is split when a batch + fails. (2) A row is set aside into `audit_event_quarantine` only after the + warehouse refused it alone, accepted another row afterwards, and refused it + again; an outage sets nothing aside. The move from queue to quarantine is one + statement, and nothing in the kit ever deletes a queue or quarantine row (the + down migrations refuse while either holds rows). (3) The queue is observed + under every `AUDIT_SINK` (`saas.audit_queue.{depth,oldest_age,quarantined, + snapshot_errors}`): a failed read makes the gauges absent, never stale, and + rows left in the queue under `postgres`/`both` are logged at startup with their + count — startup never refuses, because that would take login down. (4) A + process that records audit events outside the service (the role catalog + import) resolves `AUDIT_SINK` through `auditsink.RequireMode` and refuses an + unset value. Replaying quarantined rows is not built yet. - **History copy** (`cmd/audit-history-copy`, `pkg/business/audit_history.go`): a deployment switching to a swap value copies its `audit_events` rows into the store of record and the archive — classified and hashed as the relay does a diff --git a/module/services/accounts/code/cmd/role-catalog-import/main.go b/module/services/accounts/code/cmd/role-catalog-import/main.go index 4604e4080..74a3d9b13 100644 --- a/module/services/accounts/code/cmd/role-catalog-import/main.go +++ b/module/services/accounts/code/cmd/role-catalog-import/main.go @@ -5,15 +5,19 @@ // // Usage: // -// role-catalog-import -catalog roles.json -database-url "$DATABASE_URL" [-audit-sink "$AUDIT_SINK"] [-dry-run] [-force] +// role-catalog-import -catalog roles.json -database-url "$DATABASE_URL" [-audit-sink postgres|both|bigquery|clickhouse] [-dry-run] [-force] // // The connection principal must be a member of app_control_plane (the same // authority migrations run under); built-in roles cannot be written otherwise. // See AUTHZ.md ("Built-in role catalog import") for the format and workflow. // -// -audit-sink must be the deployment's AUDIT_SINK: under a swap value -// (ADR 0009) the import's audit events go to the transactional queue, which -// the accounts relay delivers, rather than to audit_events. +// The import must run under the deployment's AUDIT_SINK, read through the same +// auditsink package the accounts service reads it with (-audit-sink, else +// $AUDIT_SINK): under a swap value (ADR 0009) the import's audit events go to +// the transactional queue, which the accounts relay delivers, rather than to +// audit_events. Unlike the service, the import has no default: a deploy step that +// did not receive the setting stops, because guessing postgres would write its +// events where a warehouse deployment never reads them. package main import ( @@ -23,6 +27,7 @@ import ( "io" "os" + "accounts/pkg/auditstore/auditsink" "accounts/pkg/business" "accounts/pkg/infra" "accounts/pkg/rolecatalog" @@ -48,15 +53,10 @@ func run(args []string, stdout, stderr io.Writer) int { databaseURL := fs.String("database-url", os.Getenv("DATABASE_URL"), "Postgres connection URL (defaults to $DATABASE_URL)") dryRun := fs.Bool("dry-run", false, "print the plan without applying it") force := fs.Bool("force", false, "apply removals even when they would delete assignments or wipe the whole catalog") - auditSink := fs.String("audit-sink", os.Getenv("AUDIT_SINK"), "the deployment's AUDIT_SINK (defaults to $AUDIT_SINK; empty is postgres)") + auditSink := fs.String("audit-sink", "", "the deployment's AUDIT_SINK: postgres, both, bigquery or clickhouse (defaults to $AUDIT_SINK, which must then be set)") if err := fs.Parse(args); err != nil { return 2 } - sinkMode, err := business.ParseAuditSinkMode(*auditSink) - if err != nil { - line(stderr, "role-catalog-import:", err) - return 1 - } if *catalogPath == "" { line(stderr, "role-catalog-import: -catalog is required") @@ -67,6 +67,21 @@ func run(args []string, stdout, stderr io.Writer) int { return 1 } + // The flag, when given, stands in for the environment variable the service + // reads; either way the value is read and checked by auditsink, and a value + // that was never set is refused rather than defaulted. + lookup := os.LookupEnv + fs.Visit(func(f *flag.Flag) { + if f.Name == "audit-sink" { + lookup = func(string) (string, bool) { return *auditSink, true } + } + }) + sinkMode, err := auditsink.RequireMode(lookup) + if err != nil { + line(stderr, "role-catalog-import:", err) + return 1 + } + document, err := os.ReadFile(*catalogPath) if err != nil { line(stderr, "role-catalog-import:", err) diff --git a/module/services/accounts/code/cmd/role-catalog-import/main_test.go b/module/services/accounts/code/cmd/role-catalog-import/main_test.go index efff43258..627e2028e 100644 --- a/module/services/accounts/code/cmd/role-catalog-import/main_test.go +++ b/module/services/accounts/code/cmd/role-catalog-import/main_test.go @@ -2,6 +2,8 @@ package main import ( "bytes" + "os" + "path/filepath" "strings" "testing" @@ -78,3 +80,63 @@ func TestRunRequiresFlags(t *testing.T) { require.Equal(t, 2, run([]string{"-nonexistent-flag"}, &bytes.Buffer{}, &stderr)) require.True(t, strings.Contains(stderr.String(), "flag provided but not defined") || stderr.Len() > 0) } + +// unsetEnv removes name for the length of the test. +func unsetEnv(t *testing.T, name string) { + t.Helper() + t.Setenv(name, "") + require.NoError(t, os.Unsetenv(name)) +} + +func writeCatalog(t *testing.T) string { + t.Helper() + path := filepath.Join(t.TempDir(), "roles.json") + require.NoError(t, os.WriteFile(path, []byte(`{"version":1,"roles":[]}`), 0o600)) + return path +} + +// The import records audit events, and where they go depends on the +// deployment's AUDIT_SINK. A deploy step that lost the variable must stop, not +// guess postgres: on a warehouse deployment its events would land in +// audit_events, which nothing reads there. +func TestRunRefusesAnAuditSinkThatIsNotSet(t *testing.T) { + t.Setenv("DATABASE_URL", "") + catalog := writeCatalog(t) + // A database nothing listens on: reaching it means the sink check was passed. + database := "postgres://nobody@127.0.0.1:1/none?connect_timeout=1" + + unsetEnv(t, "AUDIT_SINK") + var stdout, stderr bytes.Buffer + require.Equal(t, 1, run([]string{"-catalog", catalog, "-database-url", database}, &stdout, &stderr)) + require.Contains(t, stderr.String(), "AUDIT_SINK is not set") + + t.Setenv("AUDIT_SINK", " ") + stderr.Reset() + require.Equal(t, 1, run([]string{"-catalog", catalog, "-database-url", database}, &stdout, &stderr)) + require.Contains(t, stderr.String(), "AUDIT_SINK is not set", "a blank value is as good as none") + + t.Setenv("AUDIT_SINK", "kafka") + stderr.Reset() + require.Equal(t, 1, run([]string{"-catalog", catalog, "-database-url", database}, &stdout, &stderr)) + require.Contains(t, stderr.String(), "AUDIT_SINK must be", "a set value is read by the service's own rules") +} + +func TestRunTakesTheAuditSinkFromTheEnvironmentOrTheFlag(t *testing.T) { + t.Setenv("DATABASE_URL", "") + catalog := writeCatalog(t) + database := "postgres://nobody@127.0.0.1:1/none?connect_timeout=1" + + t.Setenv("AUDIT_SINK", "bigquery") + var stdout, stderr bytes.Buffer + require.Equal(t, 1, run([]string{"-catalog", catalog, "-database-url", database}, &stdout, &stderr)) + require.NotContains(t, stderr.String(), "AUDIT_SINK", "a sink in the environment gets past the check, to the database") + + unsetEnv(t, "AUDIT_SINK") + stderr.Reset() + require.Equal(t, 1, run([]string{"-catalog", catalog, "-database-url", database, "-audit-sink", "postgres"}, &stdout, &stderr)) + require.NotContains(t, stderr.String(), "AUDIT_SINK", "and so does the flag") + + stderr.Reset() + require.Equal(t, 1, run([]string{"-catalog", catalog, "-database-url", database, "-audit-sink", ""}, &stdout, &stderr)) + require.Contains(t, stderr.String(), "AUDIT_SINK is not set", "a flag given empty is no sink either") +} diff --git a/module/services/accounts/code/pkg/auditstore/auditsink/auditsink.go b/module/services/accounts/code/pkg/auditstore/auditsink/auditsink.go index de3563cc0..e9eb88917 100644 --- a/module/services/accounts/code/pkg/auditstore/auditsink/auditsink.go +++ b/module/services/accounts/code/pkg/auditstore/auditsink/auditsink.go @@ -136,7 +136,7 @@ const ( // transactional queue. "external" is refused: no destination outside Postgres // can join the transaction a change commits in. func Load(getenv func(string) string) (Config, error) { - mode, err := business.ParseAuditSinkMode(getenv(EnvSink)) + mode, err := Mode(getenv) if err != nil { return Config{}, err } @@ -153,6 +153,31 @@ func Load(getenv func(string) string) (Config, error) { return config, nil } +// Mode reads AUDIT_SINK alone: the one place the setting is read, which Load and +// every process that must agree with the service go through. An unset value is +// the default, postgres, as it is for the service. +func Mode(getenv func(string) string) (business.AuditSinkMode, error) { + return business.ParseAuditSinkMode(getenv(EnvSink)) +} + +// RequireMode is Mode for a process that writes audit events apart from the +// service — the role catalog import, run as a deploy step — where an unset +// AUDIT_SINK is not the default but a lost setting. The service's value reaches +// such a process only because the deployment passes it on; when that does not +// happen, defaulting to postgres would write its events to audit_events on a +// deployment whose store of record is a warehouse, where nothing reads them. So +// it refuses, and says how to set it. lookup reports whether the variable is +// set at all (os.LookupEnv); a blank value is as good as none. +func RequireMode(lookup func(string) (string, bool)) (business.AuditSinkMode, error) { + raw, set := lookup(EnvSink) + if !set || strings.TrimSpace(raw) == "" { + return "", fmt.Errorf("%s is not set. This process records audit events, and where they go depends on it: "+ + "under bigquery or clickhouse they must reach the queue the audit relay delivers, not audit_events. "+ + "Pass the deployment's value (postgres, both, bigquery or clickhouse); it will not guess", EnvSink) + } + return Mode(func(string) string { return raw }) +} + func loadExternal(getenv func(string) string) (business.ExternalAuditSink, error) { endpoint := strings.TrimSpace(getenv(EnvExternalURL)) if endpoint == "" { @@ -246,11 +271,10 @@ func loadSwap(mode business.AuditSinkMode, getenv func(string) string) (*Swap, e return nil, fmt.Errorf("AUDIT_SINK=%s requires %s", mode, strings.Join(missing, ", ")) } - days, err := strconv.Atoi(retentionDays) - if err != nil || days < 1 { - return nil, fmt.Errorf("%s must be a whole number of days, at least 1; got %q", EnvContentRetentionDays, retentionDays) + var err error + if swap.ContentRetention, err = retentionDuration(EnvContentRetentionDays, retentionDays); err != nil { + return nil, err } - swap.ContentRetention = time.Duration(days) * 24 * time.Hour if swap.ClickHouse != nil { if err := loadClickHouse(swap, eventsRetentionDays); err != nil { return nil, err @@ -272,28 +296,43 @@ func loadSwap(mode business.AuditSinkMode, getenv func(string) string) (*Swap, e } if raw := strings.TrimSpace(getenv(EnvRelayMaxWait)); raw != "" { wait, err := time.ParseDuration(raw) - if err != nil || wait <= 0 || wait > time.Hour { - return nil, fmt.Errorf("%s must be a positive duration of at most an hour, like 5s; got %q", EnvRelayMaxWait, raw) + if err != nil || wait <= 0 || wait > business.MaxAuditRelayMaxWait { + return nil, fmt.Errorf("%s must be a positive duration of at most %s, like 5s; got %q", EnvRelayMaxWait, business.MaxAuditRelayMaxWait, raw) } swap.RelayMaxWait = wait } return swap, nil } +// maxRetentionDays bounds a retention setting at a century. A duration counts +// nanoseconds in an int64, which overflows to a negative value past about +// 106,751 days, and a retention that long is a mistake in the setting. +const maxRetentionDays = 36500 + +// retentionDuration reads a retention setting in whole days, bounding the count +// before it is multiplied into a duration. +func retentionDuration(name, raw string) (time.Duration, error) { + days, err := strconv.Atoi(raw) + if err != nil || days < 1 || days > maxRetentionDays { + return 0, fmt.Errorf("%s must be a whole number of days, at least 1 and at most %d; got %q", name, maxRetentionDays, raw) + } + return time.Duration(days) * 24 * time.Hour, nil +} + // loadClickHouse validates the ClickHouse settings: a DSN the driver parses // and that names its database, a cluster name ON CLUSTER can take, and an // events window at least as long as the content window — content details // outliving their event would be details of an event the store no longer has. func loadClickHouse(swap *Swap, eventsRetentionDays string) error { settings := swap.ClickHouse - days, err := strconv.Atoi(eventsRetentionDays) - if err != nil || days < 1 { - return fmt.Errorf("%s must be a whole number of days, at least 1; got %q", EnvEventsRetentionDays, eventsRetentionDays) + retention, err := retentionDuration(EnvEventsRetentionDays, eventsRetentionDays) + if err != nil { + return err } - settings.EventsRetention = time.Duration(days) * 24 * time.Hour + settings.EventsRetention = retention if settings.EventsRetention < swap.ContentRetention { return fmt.Errorf("%s (%d) must be at least %s (%d): content details are kept no longer than their events", - EnvEventsRetentionDays, days, EnvContentRetentionDays, int(swap.ContentRetention/(24*time.Hour))) + EnvEventsRetentionDays, int(retention/(24*time.Hour)), EnvContentRetentionDays, int(swap.ContentRetention/(24*time.Hour))) } options, err := clickhouse.ParseDSN(settings.DSN) if err != nil { diff --git a/module/services/accounts/code/pkg/auditstore/auditsink/auditsink_test.go b/module/services/accounts/code/pkg/auditstore/auditsink/auditsink_test.go index 32acec00d..212f4b6ec 100644 --- a/module/services/accounts/code/pkg/auditstore/auditsink/auditsink_test.go +++ b/module/services/accounts/code/pkg/auditstore/auditsink/auditsink_test.go @@ -96,6 +96,15 @@ func TestConfiguredAuditSinkBigQuery(t *testing.T) { require.NoError(t, err) require.Equal(t, 200, sink.Swap.RelayBatchSize) require.Equal(t, 750*time.Millisecond, sink.Swap.RelayMaxWait) + + t.Setenv("AUDIT_RELAY_MAX_WAIT", "60s") + sink, err = Load(os.Getenv) + require.NoError(t, err, "a minute is the longest a partial batch may wait") + require.Equal(t, time.Minute, sink.Swap.RelayMaxWait) + t.Setenv("AUDIT_CONTENT_RETENTION_DAYS", "36500") + sink, err = Load(os.Getenv) + require.NoError(t, err, "a century is the longest retention") + require.Equal(t, 36500*24*time.Hour, sink.Swap.ContentRetention) } func TestConfiguredAuditSinkBigQueryNamesEveryMissingSetting(t *testing.T) { @@ -138,6 +147,12 @@ func TestConfiguredAuditSinkBigQueryRejectsInvalidSettings(t *testing.T) { {"AUDIT_RELAY_MAX_WAIT", "5", "AUDIT_RELAY_MAX_WAIT"}, {"AUDIT_RELAY_MAX_WAIT", "-1s", "AUDIT_RELAY_MAX_WAIT"}, {"AUDIT_RELAY_MAX_WAIT", "2h", "AUDIT_RELAY_MAX_WAIT"}, + // The lag alert fires at five minutes, so a longer wait would page on a healthy relay. + {"AUDIT_RELAY_MAX_WAIT", "5m", "AUDIT_RELAY_MAX_WAIT"}, + {"AUDIT_RELAY_MAX_WAIT", "61s", "AUDIT_RELAY_MAX_WAIT"}, + // Beyond about 106,751 days a duration in nanoseconds overflows and goes negative. + {"AUDIT_CONTENT_RETENTION_DAYS", "106752", "AUDIT_CONTENT_RETENTION_DAYS"}, + {"AUDIT_CONTENT_RETENTION_DAYS", "36501", "AUDIT_CONTENT_RETENTION_DAYS"}, } { t.Run(tc.name+"="+tc.value, func(t *testing.T) { setBigQuerySwap(t) @@ -229,6 +244,7 @@ func TestConfiguredAuditSinkClickHouseRejectsInvalidSettings(t *testing.T) { {"AUDIT_EVENTS_RETENTION_DAYS", "0", "AUDIT_EVENTS_RETENTION_DAYS must be a whole number of days"}, {"AUDIT_EVENTS_RETENTION_DAYS", "7y", "AUDIT_EVENTS_RETENTION_DAYS must be a whole number of days"}, {"AUDIT_EVENTS_RETENTION_DAYS", "30", "AUDIT_EVENTS_RETENTION_DAYS (30) must be at least AUDIT_CONTENT_RETENTION_DAYS (90)"}, + {"AUDIT_EVENTS_RETENTION_DAYS", "106752", "AUDIT_EVENTS_RETENTION_DAYS must be a whole number of days"}, {"AUDIT_CONTENT_RETENTION_DAYS", "0", "AUDIT_CONTENT_RETENTION_DAYS"}, {"AUDIT_CLICKHOUSE_DSN", "clickhouse://audit:s3cr3t-value@clickhouse.example:9440", "must name the database"}, {"AUDIT_CLICKHOUSE_DSN", "clickhouse://audit:s3cr3t-value@/audit", "not a valid ClickHouse DSN"}, @@ -257,3 +273,40 @@ func TestClickHouseSettingsNeverPrintTheDSN(t *testing.T) { require.NotContains(t, fmt.Sprintf(format, sink.Swap.ClickHouse), "s3cr3t-value", format) } } + +// A process that records audit events apart from the service learns the sink +// only if the deployment passes it on. When that fails, "unset" must not read as +// the default: it would write to audit_events where the warehouse is the store. +func TestRequireModeRefusesASinkThatWasNeverSet(t *testing.T) { + env := func(values map[string]string) func(string) (string, bool) { + return func(name string) (string, bool) { v, ok := values[name]; return v, ok } + } + for name, values := range map[string]map[string]string{ + "unset": {}, + "blank": {"AUDIT_SINK": " "}, + "empty": {"AUDIT_SINK": ""}, + } { + _, err := RequireMode(env(values)) + require.ErrorContains(t, err, "AUDIT_SINK is not set", name) + } + + for value, want := range map[string]business.AuditSinkMode{ + "postgres": business.AuditSinkPostgres, + "both": business.AuditSinkBoth, + "bigquery": business.AuditSinkBigQuery, + " ClickHouse": business.AuditSinkClickHouse, + } { + mode, err := RequireMode(env(map[string]string{"AUDIT_SINK": value})) + require.NoError(t, err, value) + require.Equal(t, want, mode, value) + } + + _, err := RequireMode(env(map[string]string{"AUDIT_SINK": "external"})) + require.ErrorContains(t, err, "not permitted", "a set value is read by the service's own rules") +} + +func TestModeIsTheServicesDefaultWhenUnset(t *testing.T) { + mode, err := Mode(func(string) string { return "" }) + require.NoError(t, err) + require.Equal(t, business.AuditSinkPostgres, mode) +} From 302b2c506f13f4080723373608d711b529fa06f4 Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Mon, 5 Oct 2026 17:08:11 -0400 Subject: [PATCH 21/69] fix(audit): declare the payload keys the role catalog import records and refuse unknown declared enums The role catalog import records catalog_sha256 and catalog_source on every event, permissions_added, permissions_removed and assignments_removed on some, and a name on the deleted event, none of which saas.role.created, updated and deleted declared. Every import therefore logged "audit payload failed registry validation" for each event it recorded: advisory only, but a warning that fires on every run is a warning nobody reads. The three types now declare those fields. The counts stay decimal strings, as the importer has always written them, so rows already stored and new ones read the same. A test runs the import through a recorder that holds each payload to its registered schema. ModuleCapabilities.DeclareAuditEventTypes mapped a wire enum value this build does not know to the empty string, which the registry reads as the default. For retention that files evidence its author meant to keep for the compliance window under the shorter content window, with no refusal anywhere; the manifest path refuses a misspelt value, and the wire now does the same. Visibility had the same hole and gets the same fix: UNSPECIFIED stays the default, any other unmapped value is refused by the registry's own check. Co-Authored-By: Claude Sonnet 5.5 --- .../module_audit_declarations_test.go | 37 ++++++++++++++++++ .../adapters/module_capabilities_server.go | 31 +++++++++++++-- .../code/pkg/business/audit_registry.go | 27 +++++++++++-- .../pkg/infra/postgres_role_catalog_test.go | 38 +++++++++++++++++++ 4 files changed, 127 insertions(+), 6 deletions(-) diff --git a/module/services/accounts/code/pkg/adapters/module_audit_declarations_test.go b/module/services/accounts/code/pkg/adapters/module_audit_declarations_test.go index e13dfe5ae..bb5703510 100644 --- a/module/services/accounts/code/pkg/adapters/module_audit_declarations_test.go +++ b/module/services/accounts/code/pkg/adapters/module_audit_declarations_test.go @@ -1,6 +1,7 @@ package adapters import ( + "strings" "testing" "accounts/pkg/business" @@ -83,3 +84,39 @@ func TestModuleAuditRetentions_CoverEveryWireClass(t *testing.T) { } } } + +// A wire enum is open: a newer client can send a value this build has never +// heard of. Reading it as UNSPECIFIED would file a type its author meant to +// keep for the compliance window under the shorter content window — evidence +// that expires early with no refusal anywhere. A manifest that misspells its +// retention is refused, and so is the wire. +func TestModuleAuditDeclarations_RefuseAnEnumValueThisBuildDoesNotKnow(t *testing.T) { + unknownRetention := gen.ModuleAuditEventRetention(99) + _, err := business.ValidateAuditEventTypeDeclarations("acme", moduleAuditDeclarations([]*gen.ModuleAuditEventTypeDeclaration{ + {Type: "acme.access.granted", Retention: unknownRetention}, + })) + if err == nil { + t.Fatal("an unknown retention class was filed under the default") + } + if !strings.Contains(err.Error(), "retention") { + t.Errorf("the refusal does not name the retention: %v", err) + } + + unknownVisibility := gen.ModuleAuditEventVisibility(99) + _, err = business.ValidateAuditEventTypeDeclarations("acme", moduleAuditDeclarations([]*gen.ModuleAuditEventTypeDeclaration{ + {Type: "acme.access.granted", Visibility: unknownVisibility}, + })) + if err == nil { + t.Fatal("an unknown visibility was filed under the default") + } + if !strings.Contains(err.Error(), "visibility") { + t.Errorf("the refusal does not name the visibility: %v", err) + } + + // UNSPECIFIED stays the default it is documented to be. + if _, err := business.ValidateAuditEventTypeDeclarations("acme", moduleAuditDeclarations([]*gen.ModuleAuditEventTypeDeclaration{ + {Type: "acme.access.granted"}, + })); err != nil { + t.Errorf("an undeclared retention and visibility are refused: %v", err) + } +} diff --git a/module/services/accounts/code/pkg/adapters/module_capabilities_server.go b/module/services/accounts/code/pkg/adapters/module_capabilities_server.go index 4d77ad3c6..c626209c4 100644 --- a/module/services/accounts/code/pkg/adapters/module_capabilities_server.go +++ b/module/services/accounts/code/pkg/adapters/module_capabilities_server.go @@ -403,7 +403,8 @@ var moduleAuditFieldKinds = map[gen.ModuleAuditFieldKind]business.FieldKind{ // registry's. UNSPECIFIED has no entry and yields the empty string, which the // registry resolves to its default (tenant) — the same reading a manifest // declaration that omits `visibility` gets, so a module and a solution that say -// nothing say the same thing. +// nothing say the same thing. A value this build does not know is not +// UNSPECIFIED: see moduleAuditVisibility. var moduleAuditVisibilities = map[gen.ModuleAuditEventVisibility]string{ gen.ModuleAuditEventVisibility_MODULE_AUDIT_EVENT_VISIBILITY_TENANT: business.AuditVisibilityTenant, gen.ModuleAuditEventVisibility_MODULE_AUDIT_EVENT_VISIBILITY_EXTERNAL: business.AuditVisibilityExternal, @@ -418,6 +419,30 @@ var moduleAuditRetentions = map[gen.ModuleAuditEventRetention]string{ gen.ModuleAuditEventRetention_MODULE_AUDIT_EVENT_RETENTION_SECURITY: string(business.RetentionSecurity), } +// moduleAuditVisibility is the registry's visibility for a wire value. A value +// with no entry that is not UNSPECIFIED — an enum is open, and a newer client can +// send one this build has never heard of — is passed on as its number, which the +// registry's own check refuses by name, exactly as it refuses a misspelt manifest +// value. Mapping it to the default would silently file the type under a +// visibility its author did not choose. +func moduleAuditVisibility(v gen.ModuleAuditEventVisibility) string { + if class, ok := moduleAuditVisibilities[v]; ok || v == gen.ModuleAuditEventVisibility_MODULE_AUDIT_EVENT_VISIBILITY_UNSPECIFIED { + return class + } + return v.String() +} + +// moduleAuditRetention is the registry's retention class for a wire value, with +// the same rule: UNSPECIFIED is the default, a value this build does not know is +// refused. Mapping it to the default would file evidence its author meant to keep +// for the compliance window under the shorter content window, and say nothing. +func moduleAuditRetention(r gen.ModuleAuditEventRetention) string { + if class, ok := moduleAuditRetentions[r]; ok || r == gen.ModuleAuditEventRetention_MODULE_AUDIT_EVENT_RETENTION_UNSPECIFIED { + return class + } + return r.String() +} + // moduleAuditDeclarations converts the wire declarations to the registry's // declaration shape, the one a solution's manifest is decoded into as well. func moduleAuditDeclarations(types []*gen.ModuleAuditEventTypeDeclaration) []business.AuditEventTypeDeclaration { @@ -426,8 +451,8 @@ func moduleAuditDeclarations(types []*gen.ModuleAuditEventTypeDeclaration) []bus declaration := business.AuditEventTypeDeclaration{ Type: t.GetType(), Description: t.GetDescription(), - Visibility: moduleAuditVisibilities[t.GetVisibility()], - Retention: moduleAuditRetentions[t.GetRetention()], + Visibility: moduleAuditVisibility(t.GetVisibility()), + Retention: moduleAuditRetention(t.GetRetention()), } for _, f := range t.GetFields() { field := business.AuditFieldDeclaration{ diff --git a/module/services/accounts/code/pkg/business/audit_registry.go b/module/services/accounts/code/pkg/business/audit_registry.go index 2fa443863..bec14533f 100644 --- a/module/services/accounts/code/pkg/business/audit_registry.go +++ b/module/services/accounts/code/pkg/business/audit_registry.go @@ -288,6 +288,24 @@ func contentRetained(d AuditEventDefinition) AuditEventDefinition { } func str(name string) PayloadField { return PayloadField{Name: name, Kind: FieldString} } + +// catalogCount is a count the built-in role catalog import records, carried as +// the decimal string the importer writes, so a row written before this field was +// declared reads the same as one written after. +func catalogCount(name string) PayloadField { + return PayloadField{Name: name, Kind: FieldString, MaxLen: 10} +} + +// withCatalogProvenance adds the provenance the role catalog import stamps onto +// every event it records — the catalog's fingerprint and, when the operator +// named one, where it came from — to a role event's own fields. A role changed +// by a person carries neither. +func withCatalogProvenance(fields ...PayloadField) []PayloadField { + return append(fields, + PayloadField{Name: "catalog_sha256", Kind: FieldString, MaxLen: 64}, + PayloadField{Name: "catalog_source", Kind: FieldString, MaxLen: 1024}, + ) +} func strs(name string) PayloadField { return PayloadField{Name: name, Kind: FieldStringArray} } @@ -619,9 +637,12 @@ var auditEventCatalog = []AuditEventDefinition{ securityRetained(mutation(EventSolutionRegistrationUpdated, CategoryAccess, "A solution registered or replaced one half of its runtime registration.", str("solution_id"), str("publisher"), str("half"), PayloadField{Name: "revision", Kind: FieldInt}, strs("audit_namespaces_taken_over"))), securityRetained(mutation(EventSolutionRegistrationDeleted, CategoryAccess, "A solution registration was removed and tombstoned.", str("solution_id"), str("publisher"), PayloadField{Name: "revision", Kind: FieldInt})), securityRetained(mutation(EventAPIKeyRevoked, CategoryAccess, "An API key was revoked.", uid("key_id"))), - securityRetained(mutation(EventRoleCreated, CategoryAccess, "A role was created.", str("name"))), - securityRetained(mutation(EventRoleUpdated, CategoryAccess, "A role's description and permission set were replaced.", str("name"), strs("permissions"))), - securityRetained(mutation(EventRoleDeleted, CategoryAccess, "A role was deleted.")), + securityRetained(mutation(EventRoleCreated, CategoryAccess, "A role was created.", + withCatalogProvenance(str("name"), catalogCount("permissions_added"))...)), + securityRetained(mutation(EventRoleUpdated, CategoryAccess, "A role's description and permission set were replaced.", + withCatalogProvenance(str("name"), strs("permissions"), catalogCount("permissions_added"), catalogCount("permissions_removed"))...)), + securityRetained(mutation(EventRoleDeleted, CategoryAccess, "A role was deleted.", + withCatalogProvenance(str("name"), catalogCount("assignments_removed"))...)), securityRetained(mutation(EventRoleAssigned, CategoryAccess, "A role was assigned to a principal.", uid("role_id"), uid("subject_id"))), securityRetained(mutation(EventRoleRevoked, CategoryAccess, "A role assignment was revoked.", uid("role_id"))), securityRetained(mutation(EventSessionRevoked, CategoryAccess, "A session was revoked.")), diff --git a/module/services/accounts/code/pkg/infra/postgres_role_catalog_test.go b/module/services/accounts/code/pkg/infra/postgres_role_catalog_test.go index 795e2432c..09469cf55 100644 --- a/module/services/accounts/code/pkg/infra/postgres_role_catalog_test.go +++ b/module/services/accounts/code/pkg/infra/postgres_role_catalog_test.go @@ -346,6 +346,44 @@ func TestImportRoleCatalogStampsProvenanceOnAudit(t *testing.T) { require.Equal(t, "catalog-test:provenance", metadata["name"]) } +// validatingRecorder holds every payload the import records to the audit +// registry's schema for its type, as the emitter would, and keeps the +// complaints: the emitter only logs them, so drift between what the import +// writes and what the registry declares stays silent until someone reads a log. +type validatingRecorder struct { + inner business.AuditRecorder + problems []string +} + +func (v *validatingRecorder) RecordTx(ctx context.Context, entry business.AuditEntry) error { + if err := business.ValidatePayload(entry.EventType, entry.Payload); err != nil { + v.problems = append(v.problems, err.Error()) + } + return v.inner.RecordTx(ctx, entry) +} + +func TestImportRoleCatalogRecordsOnlyPayloadsTheRegistryDeclares(t *testing.T) { + resetCatalogRoles(t) + recorder := &validatingRecorder{inner: catalogAudit(t)} + options := infra.ImportOptions{Audit: recorder, Source: "roles.json"} + + created := parseCatalog(t, `{"version":1,"roles":[ + {"name":"catalog-test:payloads","description":"d","scope":"m","permissions":[{"resource":"x","action":"read"},{"resource":"y","action":"read"}]}]}`) + _, err := testStore.ImportRoleCatalog(testCtx, created, options) + require.NoError(t, err) + + updated := parseCatalog(t, `{"version":1,"roles":[ + {"name":"catalog-test:payloads","description":"d","scope":"m","permissions":[{"resource":"x","action":"read"},{"resource":"z","action":"read"}]}]}`) + _, err = testStore.ImportRoleCatalog(testCtx, updated, options) + require.NoError(t, err) + + options.Force = true + _, err = testStore.ImportRoleCatalog(testCtx, parseCatalog(t, `{"version":1,"roles":[]}`), options) + require.NoError(t, err) + + require.Empty(t, recorder.problems, "created, updated and deleted events each carry only fields their registered type declares") +} + func TestBuiltinRoleNamesAreUnique(t *testing.T) { err := testStore.WithControlPlane(testCtx, func(ctx context.Context) error { tx := storetx.Tx(ctx) From 897ffa316320d3760dd43b5e3292a65803b1238d Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Mon, 5 Oct 2026 17:00:31 -0400 Subject: [PATCH 22/69] fix(audit): retire only the partitions a history copy verified, atomically The history copy chose the partitions it verified by the bounds PostgreSQL holds, then dropped through audit_events_drop_partitions_before, which chooses by the month in a partition's NAME read in the session's time zone. Partitions created from a New York session end at 04:00 UTC, so a UTC session asked to retire everything before 1 September also dropped the August partition: a month whose first September hours nobody had verified (reproduced on PostgreSQL 16). The receipt was built from the plan, and the dropped count was never compared with it. The count recheck and the DROP were also two steps in READ COMMITTED with no lock: a row committed into a verified partition between them was destroyed with the table (reproduced with two sessions, and by a failing test here). Migration 20 adds audit_events_drop_verified_partitions(cutoff, names, expected_rows). In one transaction it locks audit_events and each named partition, confirms from pg_catalog that each is a monthly partition whose upper bound is at or before the cutoff, recounts each against the verified count, and drops exactly those, returning their names. Any mismatch raises SQLSTATE AH001 and rolls back. It is SECURITY DEFINER with pg_temp last, and EXECUTE goes to app_control_plane alone, the role that drops today. Partitions force row-level security, which holds a non-superuser owner to the policies and makes it count no rows, so the function lifts FORCE on each partition inside its own transaction before counting; the ACCESS EXCLUSIVE lock hides the change from other sessions and a refusal rolls it back. The copy now asks for the verified partitions by name with their row counts, refuses unless the names the database returns are exactly the verified set (rolling the transaction back before it commits), and builds the receipt from what was dropped. It stops calling the name-parsing function, which stays for retention. A confirmed drop also needs a cutoff that has passed. The copy now also counts, by table, the events older than the cutoff that no verified partition covers (default partition, odd names, a partition straddling the cutoff, a table that is not partitioned) and fails verification on any. A run with no partition to verify, and nothing older elsewhere, reports nothing to verify instead of verified. Tests: the time-zone case, the backdated row between recount and drop, a writer in flight (the drop waits, recounts, refuses; a count-then-drop variant of the function fails it), a later writer held off until commit, and every refusal leaving FORCE in place. They pass against a ledger installed by a NOSUPERUSER NOBYPASSRLS migration principal as well as a superuser. Co-Authored-By: Claude Sonnet 5.5 --- module/DATABASE_AUTHORITY.md | 59 ++- .../code/pkg/business/audit_history.go | 293 +++++++++++--- .../code/pkg/business/audit_history_test.go | 295 +++++++++++++- .../code/pkg/infra/postgres_audit_history.go | 80 +++- .../pkg/infra/postgres_audit_history_test.go | 375 ++++++++++++++++++ .../testdata/control_plane_call_sites.txt | 1 + .../20_audit_verified_partition_drop.down.sql | 3 + .../20_audit_verified_partition_drop.up.sql | 144 +++++++ 8 files changed, 1182 insertions(+), 68 deletions(-) create mode 100644 module/services/store/migrations/20_audit_verified_partition_drop.down.sql create mode 100644 module/services/store/migrations/20_audit_verified_partition_drop.up.sql diff --git a/module/DATABASE_AUTHORITY.md b/module/DATABASE_AUTHORITY.md index be5659834..62aebad21 100644 --- a/module/DATABASE_AUTHORITY.md +++ b/module/DATABASE_AUTHORITY.md @@ -508,9 +508,10 @@ Retention deletes and token-based pre-auth reads/updates use the audited control-plane boundary; they are intentionally not granted to `app_tenant`. So does the one-time audit history copy of a deployment switching to a swap value (`cmd/audit-history-copy`, ADR 0009): it reads `audit_events` across -organizations and, once its copy is verified and confirmed, drops the copied -partitions through `audit_events_drop_partitions_before`, as retention does. It -deletes no row. +organizations and, once its copy is verified and confirmed, drops exactly the +partitions it verified through `audit_events_drop_verified_partitions` (see +[Dropping verified partitions](#dropping-verified-partitions)), where retention +uses `audit_events_drop_partitions_before`. It deletes no row. The infrastructure suite compares PostgreSQL's complete public-table inventory to this executable matrix, so adding a table without classifying and granting it fails the gate. @@ -588,6 +589,58 @@ attached or detached, now needs a `BYPASSRLS` role to read every row, which the partition offload planned in [ADR 0006](./docs/adr/0006-audit-sink-and-retention-tiers.md) has to provision. +## Dropping verified partitions + +The one-time history copy retires the monthly partitions it copied into a +warehouse, and retires exactly those. `audit_events_drop_partitions_before` cannot +promise that: it picks partitions by the month in their *name*, read in the +calling session's time zone, while the copy verified partitions by the bounds +PostgreSQL holds. A partition created from a session in another zone has bounds on +that zone's midnights, so a UTC session asked to retire everything before 1 +September also dropped `audit_events_2026_08` created from New York, whose last +four hours are September's events, unverified. It also drops without looking: a +row committed into a partition after the copy's last recount, before the `DROP`, +was destroyed with the table. + +Migration `20_audit_verified_partition_drop` adds +`audit_events_drop_verified_partitions(cutoff, names, expected_rows)`, which takes +the explicit list of verified partition names with the row count each held when +verified, and in one transaction: + +1. locks `audit_events` and then each named partition `ACCESS EXCLUSIVE`, in name + order, with a 10-second lock wait: a writer in flight is waited for, a later + writer waits behind the drop, and a writer that does not finish makes the drop + refuse rather than hang. Inserters take the parent before a partition, so the + order cannot deadlock with them; +2. confirms from `pg_catalog`, after the lock, that each name is a monthly + partition of `audit_events` with a timestamp range whose upper bound is at or + before the cutoff; +3. recounts each partition and compares it with the verified count; +4. only then drops exactly the named partitions and returns their names. + +Any mismatch raises SQLSTATE `AH001` and the transaction rolls back with nothing +dropped; the command refuses unless the returned names equal the verified set +before it commits. The function never chooses a partition itself. + +It is `SECURITY DEFINER`, owned by the migration principal that owns the +partitions, with `pg_temp` last in its `search_path`, and `EXECUTE` granted to +`app_control_plane` alone, the role that holds the same authority through +`audit_events_drop_partitions_before` today; nothing is widened. The recount reads a +partition by name as its owner, and partitions *force* row-level security, which +holds an owner without `BYPASSRLS` (the migration principal) to the policies: +bound to no organization it would count no rows. The function therefore lifts +`FORCE` on each partition inside its own transaction before counting. The +partition is held `ACCESS EXCLUSIVE`, so no other session sees the change, and the +transaction either drops the partition or rolls the change back with everything +else; `TestVerifiedPartitionDropRefusesWhatItWasNotAskedToVerify` checks a refused +call leaves every partition forced. The history copy does not call +`audit_events_drop_partitions_before`; retention still does. + +`TestAuditHistoryDropsExactlyTheVerifiedPartitionsWhateverZoneCreatedThem` pins the +time-zone case, `TestAuditHistoryDropDoesNotDestroyARowCommittedAfterTheRecount` and +`TestVerifiedPartitionDropWaitsForAnInFlightInsertAndThenRefusesToDestroyIt` the +concurrent-insert case. + ## SECURITY DEFINER functions list pg_temp last A `SECURITY DEFINER` function runs with its owner's authority. Unless its diff --git a/module/services/accounts/code/pkg/business/audit_history.go b/module/services/accounts/code/pkg/business/audit_history.go index fe729ea9b..968cee4c3 100644 --- a/module/services/accounts/code/pkg/business/audit_history.go +++ b/module/services/accounts/code/pkg/business/audit_history.go @@ -42,8 +42,11 @@ type AuditHistoryReader interface { // postgres to a swap value copies the audit_events rows it already holds into // the store of record and the archive, verifies the copy by reading it back, // and only then — and only when the operator confirms — drops the copied -// monthly partitions, the way retention drops them. No row is deleted, so the -// append-only triggers stay as they are. +// monthly partitions. The partitions it drops are the ones it verified, named +// one by one: the database re-checks each against what was verified inside the +// drop's own transaction, and the copy builds its receipt from the names the +// database says it dropped. No row is deleted, so the append-only triggers stay +// as they are. // AuditHistoryPartition is one monthly partition of audit_events: the events // that occurred in [From, To). @@ -58,6 +61,26 @@ type AuditHistoryCursor struct { ID string } +// AuditHistoryDropTarget is one partition the copy verified and asks to have +// dropped, with the number of rows it held when verified. +type AuditHistoryDropTarget struct { + Name string + Rows int64 +} + +// AuditHistoryTableRows is how many events one table of audit_events holds. +type AuditHistoryTableRows struct { + Table string + Rows int64 +} + +// AuditHistoryDropRefusal is what a source returns when the database itself +// refused the drop because what the copy verified no longer holds. Nothing was +// dropped. +type AuditHistoryDropRefusal struct{ Reason string } + +func (e *AuditHistoryDropRefusal) Error() string { return "drop refused: " + e.Reason } + // AuditHistorySource is audit_events as the history copy reads and retires it. // Every method runs under WithControlPlane: the copy spans every organization. type AuditHistorySource interface { @@ -74,9 +97,18 @@ type AuditHistorySource interface { // CountAuditHistory counts the events that occurred in [from, to) per // organization ("" for the platform's). CountAuditHistory(ctx context.Context, from, to time.Time) (map[string]int64, error) - // DropAuditPartitionsBefore drops every monthly partition whose whole range - // is before the cutoff — the mechanism RunRetention uses. - DropAuditPartitionsBefore(ctx context.Context, before time.Time) (int64, error) + // CountAuditHistoryByTable counts the events that occurred in [from, to) + // per table that holds them: a monthly partition, the default partition, a + // partition under another name, or audit_events itself when it is not + // partitioned. A zero from is no lower bound. + CountAuditHistoryByTable(ctx context.Context, from, to time.Time) ([]AuditHistoryTableRows, error) + // DropVerifiedAuditPartitions drops exactly the named partitions, in one + // transaction with a last lock-and-recount of each against the number of + // rows it was verified with, and returns the names it dropped. The database + // refuses — with an *AuditHistoryDropRefusal, dropping nothing — when a + // partition is missing, ends after the cutoff, or holds a different number + // of rows. It never chooses a partition itself. + DropVerifiedAuditPartitions(ctx context.Context, cutoff time.Time, partitions []AuditHistoryDropTarget) ([]string, error) } // AuditHistoryCopyConfig wires a history copy. @@ -96,8 +128,9 @@ type AuditHistoryCopyConfig struct { // BatchSize is the most events one copied batch carries (default // DefaultAuditRelayBatchSize). BatchSize int - // Through, when set, limits the copy to the partitions that end at or - // before it, and is the drop's cutoff. Unset is every partition. + // Through, when set, limits the copy to the partitions whose bounds end at + // or before it, and is the drop's cutoff; a confirmed drop needs it, and + // refuses one that has not yet passed. Unset is every partition. Through *time.Time // Progress, when set, is told what the copy is doing. Progress func(message string) @@ -147,13 +180,22 @@ type AuditHistoryPartitionReport struct { type AuditHistoryReport struct { Partitions []AuditHistoryPartitionReport PartitionsSHA256 string - DroppedNames []string + // DroppedNames are the partitions the database reports it dropped; the + // copy refuses a drop whose names are not exactly the verified ones. + DroppedNames []string // Verified is true when every partition's events are in the store, each - // copy identical to its row. + // copy identical to its row, and nothing older than the cutoff is held + // outside the verified partitions. A run with no partition to verify is + // not verified. Verified bool // Cutoff is the drop's cutoff: every partition ending at or before it was // copied and verified. Cutoff time.Time + // Uncovered lists the tables that hold events older than the cutoff which + // no verified partition covers — the default partition, a partition under + // another name, one whose bounds straddle the cutoff, or audit_events + // itself when it is not partitioned. Any makes the run unverified. + Uncovered []AuditHistoryTableRows // Dropped is how many partitions were dropped; DropRefused says why none // were when a drop was confirmed and did not happen. Dropped int64 @@ -164,6 +206,16 @@ type AuditHistoryReport struct { // report says where. var ErrAuditHistoryUnverified = errors.New("audit history copy: verification failed; nothing was dropped") +// ErrAuditHistoryNothingToVerify is returned when no monthly partition ends at +// or before the cutoff and nothing older than it is held elsewhere: there is +// nothing for the run to have verified, so it claims nothing. +var ErrAuditHistoryNothingToVerify = errors.New("audit history copy: nothing to verify; no audit_events partition ends at or before the cutoff") + +// ErrAuditHistoryDropRefused is returned, wrapped with the reason, when a +// confirmed drop did not happen because what was verified no longer holds or +// the approval does not match; nothing was dropped. +var ErrAuditHistoryDropRefused = errors.New("audit history copy: drop refused") + const maxAuditHistoryProblems = 20 // auditHistoryPageSize is the rows one read of audit_events returns. @@ -214,29 +266,26 @@ func NewAuditHistoryCopy(cfg AuditHistoryCopyConfig) (*AuditHistoryCopy, error) } // Run copies, verifies and — when confirmed and verified — drops. It is -// resumable: a run reads back what the store already holds before it writes, -// copies only the events missing from it, and verifies every event, so a run -// after an interrupted one finishes what that one started. A verification -// failure blocks the drop and returns ErrAuditHistoryUnverified. +// resumable: a run reads each window's rows, reads back what the store already +// holds, appends only the events missing from it, reads the window back again +// and verifies every event, so a run after an interrupted one finishes what +// that one started. A verification failure blocks the drop and returns +// ErrAuditHistoryUnverified; a run with nothing to verify returns +// ErrAuditHistoryNothingToVerify; a drop that does not happen returns +// ErrAuditHistoryDropRefused with the reason. func (c *AuditHistoryCopy) Run(ctx context.Context, opts AuditHistoryCopyOptions) (AuditHistoryReport, error) { var report AuditHistoryReport partitions, err := c.partitions(ctx) if err != nil { return report, err } - if c.cfg.Through != nil { + switch { + case c.cfg.Through != nil: report.Cutoff = *c.cfg.Through - } - if len(partitions) == 0 { - report.PartitionsSHA256 = c.planDigest(report) - report.Verified = true - c.cfg.Progress("no audit_events partitions to copy") - return report, nil - } - if c.cfg.Through != nil { - report.Cutoff = *c.cfg.Through - } else { + case len(partitions) > 0: report.Cutoff = partitions[len(partitions)-1].To + default: + report.Cutoff = c.cfg.Now() } resolver := NewAuditEventResolver(c.cfg.Types) report.Verified = true @@ -251,7 +300,22 @@ func (c *AuditHistoryCopy) Run(ctx context.Context, opts AuditHistoryCopyOptions report.Verified = false } } + // What the verified partitions do not cover is not verified, whatever else + // holds: the partitions were chosen by their bounds, and an event older + // than the cutoff may sit in a table those bounds do not name. + if report.Uncovered, err = c.uncovered(ctx, report.Cutoff, partitions); err != nil { + report.Verified = false + return report, err + } + if len(report.Uncovered) > 0 { + report.Verified = false + } report.PartitionsSHA256 = c.planDigest(report) + if len(partitions) == 0 && len(report.Uncovered) == 0 { + report.Verified = false + c.cfg.Progress("no audit_events partition ends at or before the cutoff; nothing to verify") + return report, ErrAuditHistoryNothingToVerify + } if !report.Verified { return report, ErrAuditHistoryUnverified } @@ -259,20 +323,86 @@ func (c *AuditHistoryCopy) Run(ctx context.Context, opts AuditHistoryCopyOptions c.cfg.Progress("verified; nothing dropped without the drop confirmation") return report, nil } - if c.cfg.Through == nil || opts.ExpectedPartitionsSHA256 == "" || opts.ExpectedPartitionsSHA256 != report.PartitionsSHA256 { - report.DropRefused = "explicit cutoff and matching verification digest required" - return report, errors.New("audit history copy: drop refused: explicit cutoff and matching verification digest required") + switch { + case c.cfg.Through == nil || opts.ExpectedPartitionsSHA256 == "" || opts.ExpectedPartitionsSHA256 != report.PartitionsSHA256: + return report, c.refuse(&report, "explicit cutoff and matching verification digest required") + case report.Cutoff.After(c.cfg.Now()): + return report, c.refuse(&report, "the cutoff has not passed: -through must be a completed month") } - if refused, err := c.drop(ctx, &report, partitions); err != nil || refused != "" { - report.DropRefused = refused - if err == nil { - err = fmt.Errorf("audit history copy: drop refused: %s", refused) - } + refused, err := c.drop(ctx, &report, partitions) + if err != nil { return report, err } + if refused != "" { + return report, c.refuse(&report, refused) + } return report, nil } +// refuse records why a confirmed drop did not happen and returns the error that +// says so. +func (c *AuditHistoryCopy) refuse(report *AuditHistoryReport, reason string) error { + report.DropRefused = reason + return fmt.Errorf("%w: %s", ErrAuditHistoryDropRefused, reason) +} + +// uncovered lists the tables that hold events older than the cutoff which none +// of the verified partitions covers: it counts, by table, the events in every +// stretch of time before the cutoff that the verified partitions' bounds leave +// out. The default partition and a non-partitioned audit_events are in every +// such stretch. +func (c *AuditHistoryCopy) uncovered(ctx context.Context, cutoff time.Time, verified []AuditHistoryPartition) ([]AuditHistoryTableRows, error) { + type gap struct{ from, to time.Time } // a zero from is unbounded + var gaps []gap + var covered time.Time // everything before this was covered or is a gap already + for i, partition := range verified { + if i == 0 || partition.From.After(covered) { + if i == 0 { + gaps = append(gaps, gap{to: partition.From}) + } else { + gaps = append(gaps, gap{covered, partition.From}) + } + } + if partition.To.After(covered) { + covered = partition.To + } + } + if len(verified) == 0 { + gaps = append(gaps, gap{to: cutoff}) + } else { + gaps = append(gaps, gap{covered, cutoff}) + } + totals := map[string]int64{} + for _, g := range gaps { + if g.to.After(cutoff) { + g.to = cutoff + } + if !g.from.IsZero() && !g.to.After(g.from) { + continue + } + var found []AuditHistoryTableRows + err := c.cfg.Source.WithControlPlane(ctx, func(ctx context.Context) error { + var err error + found, err = c.cfg.Source.CountAuditHistoryByTable(ctx, g.from, g.to) + return err + }) + if err != nil { + return nil, fmt.Errorf("audit history copy: count events outside the verified partitions: %w", err) + } + for _, table := range found { + totals[table.Table] += table.Rows + } + } + var out []AuditHistoryTableRows + for table, rows := range totals { + if rows > 0 { + out = append(out, AuditHistoryTableRows{Table: table, Rows: rows}) + } + } + sort.Slice(out, func(i, j int) bool { return out[i].Table < out[j].Table }) + return out, nil +} + // partitions lists the partitions the run covers: every one, or those ending // at or before Through. func (c *AuditHistoryCopy) partitions(ctx context.Context) ([]AuditHistoryPartition, error) { @@ -482,24 +612,50 @@ func (c *AuditHistoryCopy) verify(record AuditRecord, copies []StoredAuditEvent) return "" } -// drop checks that nothing changed since verification and drops the copied -// partitions. It refuses — returning why — when a partition's rows changed, or -// when the cutoff would drop a partition this run did not verify. +// drop checks that nothing changed since verification and drops the verified +// partitions — those, named one by one, and no others. It refuses, returning +// why, when a verified partition is gone or has other bounds, when the cutoff +// would reach a partition this run did not verify, when a partition's rows +// changed, or when the database's own recheck, made under lock in the drop's +// transaction, disagrees. The database returns the names it dropped; a set that +// is not exactly the verified one rolls the transaction back before it commits. func (c *AuditHistoryCopy) drop(ctx context.Context, report *AuditHistoryReport, verified []AuditHistoryPartition) (string, error) { - covered := map[string]bool{} + byName := map[string]AuditHistoryPartition{} for _, partition := range verified { - covered[partition.Name] = true + byName[partition.Name] = partition + } + targets := make([]AuditHistoryDropTarget, 0, len(report.Partitions)) + for _, partitionReport := range report.Partitions { + var rows int64 + for _, count := range partitionReport.Orgs { + rows += count.Postgres + } + targets = append(targets, AuditHistoryDropTarget{Name: partitionReport.Partition.Name, Rows: rows}) } + sort.Slice(targets, func(i, j int) bool { return targets[i].Name < targets[j].Name }) + var refused string + var dropped []string err := c.cfg.Source.WithControlPlane(ctx, func(ctx context.Context) error { + refused, dropped = "", nil current, err := c.cfg.Source.ListAuditPartitions(ctx) if err != nil { return err } + now := map[string]AuditHistoryPartition{} for _, partition := range current { - if !partition.To.After(report.Cutoff) && !covered[partition.Name] { - refused = fmt.Sprintf("partition %s ends before the cutoff and was not verified by this run", partition.Name) - return nil + now[partition.Name] = partition + if !partition.To.After(report.Cutoff) { + if _, ok := byName[partition.Name]; !ok { + refused = fmt.Sprintf("partition %s ends before the cutoff and was not verified by this run", partition.Name) + return errAuditHistoryRollBack + } + } + } + for _, partition := range verified { + if got, ok := now[partition.Name]; !ok || !got.From.Equal(partition.From) || !got.To.Equal(partition.To) { + refused = fmt.Sprintf("partition %s changed since it was verified", partition.Name) + return errAuditHistoryRollBack } } for _, partitionReport := range report.Partitions { @@ -510,29 +666,68 @@ func (c *AuditHistoryCopy) drop(ctx context.Context, report *AuditHistoryReport, for org, count := range partitionReport.Orgs { if counts[org] != count.Postgres { refused = fmt.Sprintf("partition %s changed since it was verified", partitionReport.Partition.Name) - return nil + return errAuditHistoryRollBack } } for org, n := range counts { if _, known := partitionReport.Orgs[org]; !known && n > 0 { refused = fmt.Sprintf("partition %s changed since it was verified", partitionReport.Partition.Name) - return nil + return errAuditHistoryRollBack } } } - report.Dropped, err = c.cfg.Source.DropAuditPartitionsBefore(ctx, report.Cutoff) - return err + dropped, err = c.cfg.Source.DropVerifiedAuditPartitions(ctx, report.Cutoff, targets) + var refusal *AuditHistoryDropRefusal + if errors.As(err, &refusal) { + refused = refusal.Reason + return errAuditHistoryRollBack + } + if err != nil { + return err + } + sort.Strings(dropped) + if !sameNames(dropped, targets) { + refused = fmt.Sprintf("the database dropped %v, not the verified %v; rolled back", dropped, targetNames(targets)) + return errAuditHistoryRollBack + } + return nil }) + if refused != "" { + return refused, nil + } if err != nil { return "", fmt.Errorf("audit history copy: drop: %w", err) } - if refused == "" { - for _, p := range report.Partitions { - report.DroppedNames = append(report.DroppedNames, p.Partition.Name) + report.DroppedNames = dropped + report.Dropped = int64(len(dropped)) + c.cfg.Progress(fmt.Sprintf("dropped %d partitions ending at or before %s", report.Dropped, report.Cutoff.UTC().Format(time.RFC3339))) + return "", nil +} + +// errAuditHistoryRollBack ends the drop's transaction without committing it; the +// reason it was refused is held beside it. +var errAuditHistoryRollBack = errors.New("audit history copy: drop transaction rolled back") + +func targetNames(targets []AuditHistoryDropTarget) []string { + names := make([]string, 0, len(targets)) + for _, target := range targets { + names = append(names, target.Name) + } + return names +} + +// sameNames reports whether got, sorted, is exactly the targets' names. +func sameNames(got []string, targets []AuditHistoryDropTarget) bool { + want := targetNames(targets) // sorted: targets are + if len(got) != len(want) { + return false + } + for i := range got { + if got[i] != want[i] { + return false } - c.cfg.Progress(fmt.Sprintf("dropped %d partitions ending at or before %s", report.Dropped, report.Cutoff.UTC().Format(time.RFC3339))) } - return refused, nil + return true } // planDigest excludes run timing and copied counts: copy, verify and drop runs diff --git a/module/services/accounts/code/pkg/business/audit_history_test.go b/module/services/accounts/code/pkg/business/audit_history_test.go index 2f7f92f4b..3b9f89358 100644 --- a/module/services/accounts/code/pkg/business/audit_history_test.go +++ b/module/services/accounts/code/pkg/business/audit_history_test.go @@ -17,15 +17,35 @@ import ( type historySource struct { partitions []AuditHistoryPartition - rows []AuditEntry - drops []time.Time + // hidden are tables the monthly listing does not return — the default + // partition, a partition under another name — that still hold rows. + hidden []AuditHistoryPartition + rows []AuditEntry + // tableOf, when set, says which table a row sits in; the default is the + // partition whose bounds hold it. + tableOf func(AuditEntry) string + // drops are the partitions each drop asked for, dropCutoffs its cutoffs. + drops [][]AuditHistoryDropTarget + dropCutoffs []time.Time + // dropNames, when set, is the set the database claims to have dropped (and + // did drop), in place of the one asked for. + dropNames func([]AuditHistoryDropTarget) []string + // refusal, when set, is the database refusing the drop. + refusal string // beforeCount runs before every count, so a test can change the table // between verification and the drop. beforeCount func(*historySource) } +// WithControlPlane runs fn as one transaction: what fn drops stays dropped only +// if fn returns nil. func (s *historySource) WithControlPlane(ctx context.Context, fn func(context.Context) error) error { - return fn(ctx) + before := append([]AuditHistoryPartition(nil), s.partitions...) + if err := fn(ctx); err != nil { + s.partitions = before + return err + } + return nil } func (s *historySource) ListAuditPartitions(context.Context) ([]AuditHistoryPartition, error) { @@ -69,19 +89,55 @@ func (s *historySource) CountAuditHistory(_ context.Context, from, to time.Time) return counts, nil } -func (s *historySource) DropAuditPartitionsBefore(_ context.Context, before time.Time) (int64, error) { - s.drops = append(s.drops, before) +func (s *historySource) table(row AuditEntry) string { + if s.tableOf != nil { + return s.tableOf(row) + } + for _, partition := range append(append([]AuditHistoryPartition(nil), s.partitions...), s.hidden...) { + if !row.CreatedAt.Before(partition.From) && row.CreatedAt.Before(partition.To) { + return partition.Name + } + } + return "audit_events" +} + +func (s *historySource) CountAuditHistoryByTable(_ context.Context, from, to time.Time) ([]AuditHistoryTableRows, error) { + totals := map[string]int64{} + for _, row := range s.rows { + if row.CreatedAt.Before(to) && (from.IsZero() || !row.CreatedAt.Before(from)) { + totals[s.table(row)]++ + } + } + var out []AuditHistoryTableRows + for table, rows := range totals { + out = append(out, AuditHistoryTableRows{Table: table, Rows: rows}) + } + sort.Slice(out, func(i, j int) bool { return out[i].Table < out[j].Table }) + return out, nil +} + +func (s *historySource) DropVerifiedAuditPartitions(_ context.Context, cutoff time.Time, targets []AuditHistoryDropTarget) ([]string, error) { + s.drops = append(s.drops, append([]AuditHistoryDropTarget(nil), targets...)) + s.dropCutoffs = append(s.dropCutoffs, cutoff) + if s.refusal != "" { + return nil, &AuditHistoryDropRefusal{Reason: s.refusal} + } + names := targetNames(targets) + if s.dropNames != nil { + names = s.dropNames(targets) + } + gone := map[string]bool{} + for _, name := range names { + gone[name] = true + } var kept []AuditHistoryPartition - var dropped int64 for _, partition := range s.partitions { - if partition.To.After(before) { + if !gone[partition.Name] { kept = append(kept, partition) - } else { - dropped++ } } s.partitions = kept - return dropped, nil + return names, nil } // historyStore is a store of record that keeps what it is given, as the @@ -139,7 +195,9 @@ func (a *historyArchive) WriteAuditBatch(_ context.Context, batch AuditBatch) er return nil } -var historyNow = time.Date(2026, 10, 3, 12, 0, 0, 0, time.UTC) +// historyNow is after the last of the fixture's months: a drop needs a cutoff +// that has passed. +var historyNow = time.Date(2026, 11, 2, 12, 0, 0, 0, time.UTC) func month(year int, m time.Month) AuditHistoryPartition { from := time.Date(year, m, 1, 0, 0, 0, 0, time.UTC) @@ -241,8 +299,13 @@ func TestAuditHistoryCopyCopiesVerifiesAndDropsOnlyWhenConfirmed(t *testing.T) { report, err = f.copier(t, AuditHistoryCopyConfig{}).Run(context.Background(), f.dropOptions(t, AuditHistoryCopyConfig{})) require.NoError(t, err) require.Equal(t, appends, f.store.appends, "a resumed run copies nothing already there") - require.Equal(t, []time.Time{report.Cutoff}, f.source.drops) + require.Equal(t, []time.Time{report.Cutoff}, f.source.dropCutoffs) + require.Equal(t, []AuditHistoryDropTarget{ + {"audit_events_2026_08", 15}, {"audit_events_2026_09", 12}, {"audit_events_2026_10", 0}, + }, f.source.drops[0], "the database is asked for each verified partition by name, with the rows it was verified with") require.Equal(t, int64(3), report.Dropped) + require.Equal(t, []string{"audit_events_2026_08", "audit_events_2026_09", "audit_events_2026_10"}, report.DroppedNames, + "the receipt names what the database says it dropped") require.Empty(t, f.source.partitions) } @@ -392,7 +455,7 @@ func TestAuditHistoryThroughLimitsThePartitionsAndTheCutoff(t *testing.T) { require.NoError(t, err) require.Len(t, report.Partitions, 1) require.Equal(t, "audit_events_2026_08", report.Partitions[0].Partition.Name) - require.Equal(t, []time.Time{through}, f.source.drops) + require.Equal(t, []time.Time{through}, f.source.dropCutoffs) for _, event := range f.store.stored { require.True(t, event.Entry.CreatedAt.Before(through), "only August is copied") } @@ -482,3 +545,209 @@ func TestAuditHistoryDropDigestBindsContentAndNeedsAPriorVerification(t *testing require.ErrorContains(t, err, "explicit cutoff") require.Empty(t, f.source.drops) } + +// The drop is by name, the names are the verified ones, and the receipt carries +// what the database reports it dropped. A database that drops anything else — +// the way a drop by partition name in another time zone would — is rolled back +// before it commits, whatever it claims. +func TestAuditHistoryDropRefusesASetThatIsNotTheVerifiedOne(t *testing.T) { + for name, tc := range map[string]struct { + dropNames func([]AuditHistoryDropTarget) []string + want string + }{ + "one more than was verified": { + dropNames: func(targets []AuditHistoryDropTarget) []string { + return append(targetNames(targets), "audit_events_2026_11") + }, + want: "not the verified", + }, + "one fewer": { + dropNames: func(targets []AuditHistoryDropTarget) []string { return targetNames(targets)[:2] }, + want: "not the verified", + }, + "another in place of one": { + dropNames: func(targets []AuditHistoryDropTarget) []string { + return []string{"audit_events_2026_08", "audit_events_2026_09", "audit_events_2026_12"} + }, + want: "not the verified", + }, + } { + t.Run(name, func(t *testing.T) { + f := newHistoryFixture() + f.source.dropNames = tc.dropNames + options := f.dropOptions(t, AuditHistoryCopyConfig{}) + report, err := f.copier(t, AuditHistoryCopyConfig{}).Run(context.Background(), options) + require.ErrorIs(t, err, ErrAuditHistoryDropRefused) + require.Contains(t, report.DropRefused, tc.want) + require.Empty(t, report.DroppedNames) + require.Zero(t, report.Dropped) + require.Len(t, f.source.partitions, 3, "the transaction was rolled back") + }) + } +} + +func TestAuditHistoryDropReportsTheDatabasesRefusal(t *testing.T) { + f := newHistoryFixture() + f.source.refusal = "partition audit_events_2026_09 holds 13 rows, 12 were verified" + report, err := f.copier(t, AuditHistoryCopyConfig{}).Run(context.Background(), f.dropOptions(t, AuditHistoryCopyConfig{})) + require.ErrorIs(t, err, ErrAuditHistoryDropRefused) + require.ErrorContains(t, err, "holds 13 rows, 12 were verified") + require.Equal(t, f.source.refusal, report.DropRefused) + require.True(t, report.Verified, "the copy was verified; the drop was refused") + require.Empty(t, report.DroppedNames) + require.Len(t, f.source.partitions, 3) +} + +func TestAuditHistoryDropNeedsACutoffThatHasPassed(t *testing.T) { + f := newHistoryFixture() + copier := f.copier(t, AuditHistoryCopyConfig{}) + copier.cfg.Now = func() time.Time { return time.Date(2026, 10, 15, 0, 0, 0, 0, time.UTC) } + plan, err := copier.Run(context.Background(), AuditHistoryCopyOptions{}) + require.NoError(t, err, "copying and verifying through a month that has not ended is allowed") + report, err := copier.Run(context.Background(), AuditHistoryCopyOptions{ConfirmDrop: true, VerifyOnly: true, ExpectedPartitionsSHA256: plan.PartitionsSHA256}) + require.ErrorIs(t, err, ErrAuditHistoryDropRefused) + require.Contains(t, report.DropRefused, "completed month") + require.Empty(t, f.source.drops) +} + +// A run that finds no partition to verify has not verified anything, and says +// so as an outcome of its own rather than as success. +func TestAuditHistoryWithNoPartitionToVerifyClaimsNothing(t *testing.T) { + f := newHistoryFixture() + f.source.partitions, f.source.rows = nil, nil + through := time.Date(2026, 9, 1, 0, 0, 0, 0, time.UTC) + for _, opts := range []AuditHistoryCopyOptions{{}, {VerifyOnly: true, ConfirmDrop: true, ExpectedPartitionsSHA256: "ignored"}} { + report, err := f.copier(t, AuditHistoryCopyConfig{Through: &through}).Run(context.Background(), opts) + require.ErrorIs(t, err, ErrAuditHistoryNothingToVerify) + require.False(t, report.Verified) + require.Empty(t, report.Uncovered) + require.Empty(t, f.source.drops) + } +} + +// Events older than the cutoff that no verified partition holds fail the run: +// the partitions were chosen by their bounds, and the table can hold events +// elsewhere — in a default partition, a partition under another name, one that +// straddles the cutoff, or when it is not partitioned at all. +func TestAuditHistoryEventsOutsideTheVerifiedPartitionsFailVerification(t *testing.T) { + newYork := func(m time.Month) AuditHistoryPartition { + p := month(2026, m) + p.From, p.To = p.From.Add(4*time.Hour), p.To.Add(4*time.Hour) + return p + } + through := time.Date(2026, 9, 1, 0, 0, 0, 0, time.UTC) + for name, tc := range map[string]struct { + prepare func(*historyFixture) + table string + rows int64 + }{ + "a default partition": { + prepare: func(f *historyFixture) { + // An event from before the first partition, which only a default + // partition could hold. + f.source.rows = append(f.source.rows, historyRow(900, "", EventAuthLogin, time.Date(2025, 5, 1, 0, 0, 0, 0, time.UTC))) + f.source.tableOf = func(e AuditEntry) string { + if e.CreatedAt.Before(month(2026, time.August).From) { + return "audit_events_default" + } + return "audit_events_2026_08" + } + }, + table: "audit_events_default", rows: 1, + }, + "a partition under another name": { + prepare: func(f *historyFixture) { + // The listing returns monthly names only; this one holds August 30th. + odd := AuditHistoryPartition{Name: "audit_events_aug_late", From: time.Date(2026, 8, 30, 0, 0, 0, 0, time.UTC), To: time.Date(2026, 8, 31, 0, 0, 0, 0, time.UTC)} + f.source.hidden = []AuditHistoryPartition{odd} + f.source.partitions = []AuditHistoryPartition{{Name: "audit_events_2026_07", From: month(2026, time.July).From, To: odd.From}, {Name: "audit_events_2026_08", From: odd.To, To: month(2026, time.August).To}} + }, + table: "audit_events_aug_late", rows: 6, + }, + "a partition that straddles the cutoff": { + prepare: func(f *historyFixture) { + f.source.partitions = []AuditHistoryPartition{newYork(time.July), newYork(time.August)} + }, + table: "audit_events_2026_08", rows: 15, + }, + "a table that is not partitioned": { + prepare: func(f *historyFixture) { f.source.partitions = nil }, + table: "audit_events", rows: 15, + }, + } { + t.Run(name, func(t *testing.T) { + f := newHistoryFixture() + tc.prepare(f) + report, err := f.copier(t, AuditHistoryCopyConfig{Through: &through}).Run(context.Background(), AuditHistoryCopyOptions{}) + require.ErrorIs(t, err, ErrAuditHistoryUnverified) + require.False(t, report.Verified) + require.Contains(t, report.Uncovered, AuditHistoryTableRows{Table: tc.table, Rows: tc.rows}) + + // The digest of the unverified run authorizes nothing. + _, err = f.copier(t, AuditHistoryCopyConfig{Through: &through}).Run(context.Background(), + AuditHistoryCopyOptions{VerifyOnly: true, ConfirmDrop: true, ExpectedPartitionsSHA256: report.PartitionsSHA256}) + require.ErrorIs(t, err, ErrAuditHistoryUnverified) + require.Empty(t, f.source.drops) + }) + } +} + +// Contiguous verified partitions leave no gap to look in but the two ends, and +// nothing outside them passes unnoticed on either. +func TestAuditHistoryLooksForUncoveredEventsOnlyWhereThePartitionsAreNot(t *testing.T) { + f := newHistoryFixture() + queried := &countingTables{historySource: f.source} + through := time.Date(2026, 11, 1, 0, 0, 0, 0, time.UTC) + copier := f.copier(t, AuditHistoryCopyConfig{Through: &through}) + copier.cfg.Source = queried + _, err := copier.Run(context.Background(), AuditHistoryCopyOptions{}) + require.NoError(t, err) + require.Len(t, queried.intervals, 1, "before the first partition; the cutoff is the last partition's end") + require.True(t, queried.intervals[0][0].IsZero()) + require.Equal(t, month(2026, time.August).From, queried.intervals[0][1]) +} + +type countingTables struct { + *historySource + intervals [][2]time.Time +} + +func (c *countingTables) CountAuditHistoryByTable(ctx context.Context, from, to time.Time) ([]AuditHistoryTableRows, error) { + c.intervals = append(c.intervals, [2]time.Time{from, to}) + return c.historySource.CountAuditHistoryByTable(ctx, from, to) +} + +// The copy matches events by id within a read window while audit_events is keyed +// by (id, created_at). Two rows with one id in the same window cannot both match +// the copies found, so verification fails closed; in different windows each copy +// matches its own row. module/AUDIT_OPERATIONS.md says exactly this. +func TestAuditHistoryDuplicateIDsFailVerificationWithinAWindow(t *testing.T) { + for name, tc := range map[string]struct { + offset time.Duration + verified bool + }{ + "in the same UTC-day window": {offset: time.Hour, verified: false}, + "in a different window (a day)": {offset: 24 * time.Hour, verified: true}, + } { + t.Run(name, func(t *testing.T) { + f := newHistoryFixture() + twin := f.source.rows[0] + twin.CreatedAt = twin.CreatedAt.Add(tc.offset) + f.source.rows = append(f.source.rows, twin) + report, err := f.copier(t, AuditHistoryCopyConfig{}).Run(context.Background(), AuditHistoryCopyOptions{}) + if tc.verified { + require.NoError(t, err) + require.True(t, report.Verified) + require.Equal(t, 2, f.store.copies()[twin.ID], "one stored row per source row") + return + } + require.ErrorIs(t, err, ErrAuditHistoryUnverified) + require.False(t, report.Verified) + var problems []string + for _, partition := range report.Partitions { + problems = append(problems, partition.Problems...) + } + require.Contains(t, problems[0], "stored envelope differs from the row") + }) + } +} diff --git a/module/services/accounts/code/pkg/infra/postgres_audit_history.go b/module/services/accounts/code/pkg/infra/postgres_audit_history.go index bc61fba8d..fb9c10f4a 100644 --- a/module/services/accounts/code/pkg/infra/postgres_audit_history.go +++ b/module/services/accounts/code/pkg/infra/postgres_audit_history.go @@ -2,10 +2,14 @@ package infra import ( "context" + "errors" "fmt" + "strings" "time" "accounts/pkg/business" + + "github.com/jackc/pgx/v5/pgconn" ) // audit_events as the one-time history copy reads and retires it @@ -13,9 +17,10 @@ import ( // transaction the caller opened: the copy spans every organization. // auditPartitionsSQL lists the monthly partitions of audit_events — the ones -// audit_events_ensure_partition creates and audit_events_drop_partitions_before -// drops — with the bounds Postgres holds for each, parsed back from its own -// rendering of them so no time zone is assumed. +// audit_events_ensure_partition creates — with the bounds Postgres holds for +// each, parsed back from its own rendering of them so no time zone is assumed. +// The copy verifies the partitions this lists and drops those same ones by name, +// through audit_events_drop_verified_partitions, which reads the same bounds. const auditPartitionsSQL = ` SELECT c.relname, (regexp_match(pg_get_expr(c.relpartbound, c.oid), 'FROM \(''([^'']+)''\) TO \(''([^'']+)''\)'))[1]::timestamptz, @@ -129,4 +134,73 @@ func (s *PostgresStore) CountAuditHistory(ctx context.Context, from, to time.Tim return counts, rows.Err() } +// CountAuditHistoryByTable implements business.AuditHistorySource. It reads +// through the parent, so partition pruning keeps it to the partitions that +// overlap [from, to) and the default partition, and tableoid names the table a +// row sits in. +func (s *PostgresStore) CountAuditHistoryByTable(ctx context.Context, from, to time.Time) ([]business.AuditHistoryTableRows, error) { + query := ` + SELECT tableoid::regclass::text, count(*) + FROM public.audit_events + WHERE created_at < $1` + args := []any{to} + if !from.IsZero() { + query += ` AND created_at >= $2` + args = append(args, from) + } + query += ` GROUP BY tableoid ORDER BY 1` + rows, err := s.getQueryExecutor(ctx).Query(ctx, query, args...) + if err != nil { + return nil, err + } + defer rows.Close() + var out []business.AuditHistoryTableRows + for rows.Next() { + var table business.AuditHistoryTableRows + if err := rows.Scan(&table.Table, &table.Rows); err != nil { + return nil, err + } + out = append(out, table) + } + return out, rows.Err() +} + +// The SQLSTATE audit_events_drop_verified_partitions raises when it refuses, +// and the one PostgreSQL raises when a lock wait ends (lock_timeout). +const ( + auditDropRefusedState = "AH001" + lockNotAvailableState = "55P03" + auditDropRefusedLead = "audit history drop refused: " +) + +// DropVerifiedAuditPartitions implements business.AuditHistorySource. The +// database function locks audit_events and each named partition, confirms from +// the catalog that each is a partition whose upper bound is at or before the +// cutoff, recounts each against the number it was verified with, and only then +// drops exactly those partitions, all in the caller's transaction. +func (s *PostgresStore) DropVerifiedAuditPartitions(ctx context.Context, cutoff time.Time, partitions []business.AuditHistoryDropTarget) ([]string, error) { + names := make([]string, 0, len(partitions)) + rows := make([]int64, 0, len(partitions)) + for _, partition := range partitions { + names = append(names, partition.Name) + rows = append(rows, partition.Rows) + } + var dropped []string + err := s.getQueryExecutor(ctx).QueryRow(ctx, + `SELECT public.audit_events_drop_verified_partitions($1, $2::text[], $3::bigint[])`, cutoff, names, rows).Scan(&dropped) + var pgErr *pgconn.PgError + if errors.As(err, &pgErr) { + switch pgErr.Code { + case auditDropRefusedState: + return nil, &business.AuditHistoryDropRefusal{Reason: strings.TrimPrefix(pgErr.Message, auditDropRefusedLead)} + case lockNotAvailableState: + return nil, &business.AuditHistoryDropRefusal{Reason: "audit_events is locked by another transaction that did not finish in time; nothing was dropped"} + } + } + if err != nil { + return nil, err + } + return dropped, nil +} + var _ business.AuditHistorySource = (*PostgresStore)(nil) diff --git a/module/services/accounts/code/pkg/infra/postgres_audit_history_test.go b/module/services/accounts/code/pkg/infra/postgres_audit_history_test.go index 8a31110db..189651a75 100644 --- a/module/services/accounts/code/pkg/infra/postgres_audit_history_test.go +++ b/module/services/accounts/code/pkg/infra/postgres_audit_history_test.go @@ -5,12 +5,14 @@ package infra_test import ( "context" "fmt" + "strings" "testing" "time" "accounts/pkg/auditstore/bigqueryfake" "accounts/pkg/auditstore/bigquerystore" "accounts/pkg/business" + "accounts/pkg/infra" "github.com/google/uuid" "github.com/stretchr/testify/require" @@ -206,3 +208,376 @@ func runHistoryCopy(t *testing.T, entries []business.AuditEntry) { } require.Zero(t, copied, "the confirming run found everything copied") } + +// historyRig is the copy's other two sides: the fake Storage Read API the +// store of record reads back through, and an archive that counts. +type historyRig struct { + fake *bigqueryfake.Server + reader *bigquerystore.Reader + archive *historyArchive +} + +func newHistoryRig(t *testing.T) *historyRig { + t.Helper() + fake := bigqueryfake.New() + fake.CreateTable(bigqueryfake.TablePath("history-project", "audit", bigquerystore.EventsTable), bigquerystore.EventsSchema()) + fake.CreateTable(bigqueryfake.TablePath("history-project", "audit", bigquerystore.DetailsTable), bigquerystore.DetailsSchema()) + reader, err := bigquerystore.NewReader(bigquerystore.ReadConfig{Client: fake, Project: "history-project", Dataset: "audit", DeploymentID: "deployment-history"}) + require.NoError(t, err) + return &historyRig{fake: fake, reader: reader, archive: &historyArchive{}} +} + +func (r *historyRig) copier(t *testing.T, source business.AuditHistorySource, through time.Time) *business.AuditHistoryCopy { + t.Helper() + copier, err := business.NewAuditHistoryCopy(business.AuditHistoryCopyConfig{ + Source: source, Store: historyWarehouse{fake: r.fake}, Archive: r.archive, ReadBack: r.reader, Types: testStore, + DeploymentID: "deployment-history", ContentRetention: 365 * 24 * time.Hour, BatchSize: 2, Through: &through, + }) + require.NoError(t, err) + return copier +} + +// ensurePartitionUnderZone creates the month's partition from a session in the +// given time zone: PostgreSQL renders the month's first instant in the session's +// zone, so a zone other than UTC gives a partition whose bounds are not UTC +// midnights. +func ensurePartitionUnderZone(t *testing.T, zone, day string) { + t.Helper() + require.NoError(t, testStore.WithControlPlane(testCtx, func(ctx context.Context) error { + tx := txFromCtx(t, ctx) + if _, err := tx.Exec(ctx, fmt.Sprintf(`SET LOCAL TimeZone = '%s'`, zone)); err != nil { + return err + } + _, err := tx.Exec(ctx, `SELECT audit_events_ensure_partition($1::date)`, day) + return err + })) +} + +// insertHistoryRow commits a platform event at an explicit instant, the way a +// writer backdating a row would. +func insertHistoryRow(t *testing.T, at time.Time) { + t.Helper() + require.NoError(t, testStore.WithControlPlane(testCtx, func(ctx context.Context) error { + _, err := txFromCtx(t, ctx).Exec(ctx, + `INSERT INTO audit_events (id, event_type, resource, created_at) VALUES ($1, $2, 'example', $3)`, + uuid.NewString(), string(business.EventAuthLogin), at) + return err + })) +} + +func auditPartitionExists(t *testing.T, name string) bool { + t.Helper() + var exists bool + require.NoError(t, testStore.WithControlPlane(testCtx, func(ctx context.Context) error { + return txFromCtx(t, ctx).QueryRow(ctx, `SELECT to_regclass('public.' || $1) IS NOT NULL`, name).Scan(&exists) + })) + return exists +} + +// auditPartitionRows counts the rows a partition holds, through the parent. +func auditPartitionRows(t *testing.T, name string) int64 { + t.Helper() + var rows int64 + require.NoError(t, testStore.WithControlPlane(testCtx, func(ctx context.Context) error { + return txFromCtx(t, ctx).QueryRow(ctx, + `SELECT count(*) FROM audit_events WHERE tableoid = to_regclass('public.' || $1)`, name).Scan(&rows) + })) + return rows +} + +// retireTestMonths drops what a test left behind, by the retention function, so +// no month outlives the test that provisioned it. +func retireTestMonths(t *testing.T, before string) { + t.Helper() + t.Cleanup(func() { + require.NoError(t, testStore.WithControlPlane(testCtx, func(ctx context.Context) error { + _, err := txFromCtx(t, ctx).Exec(ctx, `SELECT audit_events_drop_partitions_before($1::timestamptz)`, before) + return err + })) + }) +} + +func droppedIn(report business.AuditHistoryReport, prefix string) []string { + var out []string + for _, name := range report.DroppedNames { + if strings.HasPrefix(name, prefix) { + out = append(out, name) + } + } + return out +} + +// Partitions created from a session whose zone is not UTC have bounds on that +// zone's midnights, so the month named audit_events_2003_08 ends at 04:00 UTC on +// the first of September. The copy verifies the partitions whose bounds end by +// the cutoff, and must drop exactly those: dropping by name would also take +// August, whose first four September hours are events no one verified. +func TestAuditHistoryDropsExactlyTheVerifiedPartitionsWhateverZoneCreatedThem(t *testing.T) { + retireTestMonths(t, "2004-01-01") + ensurePartitionUnderZone(t, "America/New_York", "2003-07-15") + ensurePartitionUnderZone(t, "America/New_York", "2003-08-15") + insertHistoryRow(t, time.Date(2003, 7, 10, 12, 0, 0, 0, time.UTC)) + insertHistoryRow(t, time.Date(2003, 7, 20, 12, 0, 0, 0, time.UTC)) + // September's first hours, in the partition named for August. + insertHistoryRow(t, time.Date(2003, 9, 1, 2, 0, 0, 0, time.UTC)) + through := time.Date(2003, 9, 1, 0, 0, 0, 0, time.UTC) + + rig := newHistoryRig(t) + copier := rig.copier(t, testStore, through) + report, err := copier.Run(testCtx, business.AuditHistoryCopyOptions{}) + require.NoError(t, err) + require.True(t, report.Verified) + require.Empty(t, report.Uncovered, "the August partition holds nothing older than the cutoff") + report, err = copier.Run(testCtx, business.AuditHistoryCopyOptions{ConfirmDrop: true, VerifyOnly: true, ExpectedPartitionsSHA256: report.PartitionsSHA256}) + require.NoError(t, err) + + require.Equal(t, []string{"audit_events_2003_07"}, droppedIn(report, "audit_events_2003_")) + require.Equal(t, int64(len(report.DroppedNames)), report.Dropped) + require.False(t, auditPartitionExists(t, "audit_events_2003_07")) + require.True(t, auditPartitionExists(t, "audit_events_2003_08"), "a partition the run did not verify is not dropped") + require.Equal(t, int64(1), auditPartitionRows(t, "audit_events_2003_08")) +} + +// A partition that straddles the cutoff holds events before it that no verified +// partition covers. The copy cannot have verified them, so it does not pass, +// and nothing is dropped. +func TestAuditHistoryDoesNotPassAPartitionThatStraddlesTheCutoff(t *testing.T) { + retireTestMonths(t, "2006-01-01") + ensurePartitionUnderZone(t, "America/New_York", "2005-10-15") + ensurePartitionUnderZone(t, "America/New_York", "2005-11-15") + insertHistoryRow(t, time.Date(2005, 10, 10, 12, 0, 0, 0, time.UTC)) + insertHistoryRow(t, time.Date(2005, 11, 20, 12, 0, 0, 0, time.UTC)) + through := time.Date(2005, 12, 1, 0, 0, 0, 0, time.UTC) + + rig := newHistoryRig(t) + copier := rig.copier(t, testStore, through) + report, err := copier.Run(testCtx, business.AuditHistoryCopyOptions{}) + require.ErrorIs(t, err, business.ErrAuditHistoryUnverified) + require.False(t, report.Verified) + require.Len(t, report.Uncovered, 1) + require.Contains(t, report.Uncovered[0].Table, "audit_events_2005_11") + require.Equal(t, int64(1), report.Uncovered[0].Rows) + + // Even with the digest the run printed, the confirmation drops nothing. + report, err = copier.Run(testCtx, business.AuditHistoryCopyOptions{ConfirmDrop: true, VerifyOnly: true, ExpectedPartitionsSHA256: report.PartitionsSHA256}) + require.ErrorIs(t, err, business.ErrAuditHistoryUnverified) + require.Empty(t, report.DroppedNames) + require.True(t, auditPartitionExists(t, "audit_events_2005_10")) + require.True(t, auditPartitionExists(t, "audit_events_2005_11"), "its November events were never verified") + require.Equal(t, int64(1), auditPartitionRows(t, "audit_events_2005_11")) +} + +// backdatingSource commits a backdated row right after the recount the drop +// makes of a verified partition, the window a count-then-drop leaves open. +type backdatingSource struct { + *infra.PostgresStore + afterCount func() +} + +func (s *backdatingSource) CountAuditHistory(ctx context.Context, from, to time.Time) (map[string]int64, error) { + counts, err := s.PostgresStore.CountAuditHistory(ctx, from, to) + if hook := s.afterCount; hook != nil { + s.afterCount = nil + hook() + } + return counts, err +} + +func TestAuditHistoryDropDoesNotDestroyARowCommittedAfterTheRecount(t *testing.T) { + retireTestMonths(t, "2004-06-01") + ensurePartitionUnderZone(t, "UTC", "2004-03-15") + for day := 3; day <= 5; day++ { + insertHistoryRow(t, time.Date(2004, 3, day, 12, 0, 0, 0, time.UTC)) + } + through := time.Date(2004, 4, 1, 0, 0, 0, 0, time.UTC) + + rig := newHistoryRig(t) + report, err := rig.copier(t, testStore, through).Run(testCtx, business.AuditHistoryCopyOptions{}) + require.NoError(t, err) + + racing := &backdatingSource{PostgresStore: testStore, afterCount: func() { + insertHistoryRow(t, time.Date(2004, 3, 6, 12, 0, 0, 0, time.UTC)) + }} + refused, err := rig.copier(t, racing, through).Run(testCtx, business.AuditHistoryCopyOptions{ + ConfirmDrop: true, VerifyOnly: true, ExpectedPartitionsSHA256: report.PartitionsSHA256, + }) + + require.ErrorIs(t, err, business.ErrAuditHistoryDropRefused, "the drop was refused, not reported as done") + require.Contains(t, refused.DropRefused, "audit_events_2004_03 holds 4 rows, 3 were verified") + require.Empty(t, refused.DroppedNames) + require.True(t, auditPartitionExists(t, "audit_events_2004_03"), "the backdated row's partition was dropped with it") + require.Equal(t, int64(4), auditPartitionRows(t, "audit_events_2004_03"), "a row committed after the recount is not destroyed") +} + +// dropVerified calls the database function as the history copy does, in its own +// control-plane transaction. +func dropVerified(cutoff time.Time, targets ...business.AuditHistoryDropTarget) ([]string, error) { + var dropped []string + err := testStore.WithControlPlane(testCtx, func(ctx context.Context) error { + var err error + dropped, err = testStore.DropVerifiedAuditPartitions(ctx, cutoff, targets) + return err + }) + return dropped, err +} + +// A writer whose insert is in flight when the drop starts is waited for: the +// drop locks the partition before it counts, so the count includes the row the +// writer then commits, disagrees with the number verified, and drops nothing. +// Counting first and locking after would see the old count, wait out the +// writer at the DROP, and take the writer's row with the table. +func TestVerifiedPartitionDropWaitsForAnInFlightInsertAndThenRefusesToDestroyIt(t *testing.T) { + retireTestMonths(t, "2008-01-01") + ensurePartitionUnderZone(t, "UTC", "2007-05-15") + insertHistoryRow(t, time.Date(2007, 5, 3, 12, 0, 0, 0, time.UTC)) + insertHistoryRow(t, time.Date(2007, 5, 4, 12, 0, 0, 0, time.UTC)) + cutoff := time.Date(2007, 6, 1, 0, 0, 0, 0, time.UTC) + + inserted, release := make(chan struct{}), make(chan struct{}) + writer := make(chan error, 1) + go func() { + writer <- testStore.WithControlPlane(testCtx, func(ctx context.Context) error { + if _, err := txFromCtx(t, ctx).Exec(ctx, + `INSERT INTO audit_events (id, event_type, resource, created_at) VALUES ($1, $2, 'example', $3)`, + uuid.NewString(), string(business.EventAuthLogin), time.Date(2007, 5, 6, 12, 0, 0, 0, time.UTC)); err != nil { + return err + } + close(inserted) + <-release + return nil + }) + }() + select { + case <-inserted: + case err := <-writer: + t.Fatalf("the in-flight insert failed: %v", err) + } + + type outcome struct { + dropped []string + err error + } + dropper := make(chan outcome, 1) + go func() { + dropped, err := dropVerified(cutoff, business.AuditHistoryDropTarget{Name: "audit_events_2007_05", Rows: 2}) + dropper <- outcome{dropped, err} + }() + select { + case got := <-dropper: + t.Fatalf("the drop did not wait for the insert in flight: %v, %v", got.dropped, got.err) + case <-time.After(750 * time.Millisecond): + } + close(release) + require.NoError(t, <-writer) + + got := <-dropper + var refusal *business.AuditHistoryDropRefusal + require.ErrorAs(t, got.err, &refusal) + require.Contains(t, refusal.Reason, "audit_events_2007_05 holds 3 rows, 2 were verified") + require.Empty(t, got.dropped) + require.Equal(t, int64(3), auditPartitionRows(t, "audit_events_2007_05"), "the writer's row survives") +} + +// Once the drop holds the partition nothing routes into it: a writer that +// arrives meanwhile waits, and when the drop commits it is told there is no +// partition for its row instead of having the row accepted and destroyed. +func TestVerifiedPartitionDropHoldsOffLaterInsertsUntilItCommits(t *testing.T) { + retireTestMonths(t, "2010-01-01") + ensurePartitionUnderZone(t, "UTC", "2009-05-15") + insertHistoryRow(t, time.Date(2009, 5, 3, 12, 0, 0, 0, time.UTC)) + cutoff := time.Date(2009, 6, 1, 0, 0, 0, 0, time.UTC) + + dropped, release := make(chan struct{}), make(chan struct{}) + dropper := make(chan error, 1) + go func() { + dropper <- testStore.WithControlPlane(testCtx, func(ctx context.Context) error { + names, err := testStore.DropVerifiedAuditPartitions(ctx, cutoff, []business.AuditHistoryDropTarget{{Name: "audit_events_2009_05", Rows: 1}}) + if err != nil { + return err + } + if len(names) != 1 || names[0] != "audit_events_2009_05" { + return fmt.Errorf("dropped %v", names) + } + close(dropped) + <-release + return nil + }) + }() + select { + case <-dropped: + case err := <-dropper: + t.Fatalf("the drop failed: %v", err) + } + + writer := make(chan error, 1) + go func() { + writer <- testStore.WithControlPlane(testCtx, func(ctx context.Context) error { + _, err := txFromCtx(t, ctx).Exec(ctx, + `INSERT INTO audit_events (id, event_type, resource, created_at) VALUES ($1, $2, 'example', $3)`, + uuid.NewString(), string(business.EventAuthLogin), time.Date(2009, 5, 6, 12, 0, 0, 0, time.UTC)) + return err + }) + }() + select { + case err := <-writer: + t.Fatalf("a writer got past the drop's lock before it committed: %v", err) + case <-time.After(750 * time.Millisecond): + } + close(release) + require.NoError(t, <-dropper) + require.ErrorContains(t, <-writer, "no partition") + require.False(t, auditPartitionExists(t, "audit_events_2009_05")) +} + +// The function names what it will drop and checks each name against the +// catalog; it never works out a set for itself, and a refusal changes nothing. +func TestVerifiedPartitionDropRefusesWhatItWasNotAskedToVerify(t *testing.T) { + retireTestMonths(t, "2012-01-01") + ensurePartitionUnderZone(t, "America/New_York", "2011-07-15") + ensurePartitionUnderZone(t, "America/New_York", "2011-08-15") + insertHistoryRow(t, time.Date(2011, 7, 10, 12, 0, 0, 0, time.UTC)) + cutoff := time.Date(2011, 9, 1, 0, 0, 0, 0, time.UTC) + + for name, tc := range map[string]struct { + targets []business.AuditHistoryDropTarget + want string + }{ + // Named explicitly, the August partition ends at 04:00 UTC on 1 September: + // the cutoff, which the name alone would have admitted, does not. + "a partition that ends after the cutoff": { + targets: []business.AuditHistoryDropTarget{{Name: "audit_events_2011_07", Rows: 1}, {Name: "audit_events_2011_08", Rows: 0}}, + want: "audit_events_2011_08 ends at 2011-09-01 04:00:00+00, after the cutoff", + }, + "a partition that is not there": {targets: []business.AuditHistoryDropTarget{{Name: "audit_events_2011_01", Rows: 0}}, want: "does not exist"}, + "a name that is not a month": {targets: []business.AuditHistoryDropTarget{{Name: "audit_events", Rows: 0}}, want: "not a monthly audit_events partition name"}, + "a count that is not the verified": {targets: []business.AuditHistoryDropTarget{{Name: "audit_events_2011_07", Rows: 2}}, want: "holds 1 rows, 2 were verified"}, + "nothing at all": {targets: nil, want: "required"}, + "a name twice": { + targets: []business.AuditHistoryDropTarget{{Name: "audit_events_2011_07", Rows: 1}, {Name: "audit_events_2011_07", Rows: 1}}, + want: "distinct", + }, + } { + t.Run(name, func(t *testing.T) { + dropped, err := dropVerified(cutoff, tc.targets...) + var refusal *business.AuditHistoryDropRefusal + require.ErrorAs(t, err, &refusal) + require.Contains(t, refusal.Reason, tc.want) + require.Empty(t, dropped) + require.True(t, auditPartitionExists(t, "audit_events_2011_07")) + require.True(t, auditPartitionExists(t, "audit_events_2011_08")) + require.Equal(t, int64(1), auditPartitionRows(t, "audit_events_2011_07")) + // The recount lifts FORCE on a partition for the length of its own + // transaction; a refusal rolls that back with everything else. + require.NoError(t, testStore.WithControlPlane(testCtx, func(ctx context.Context) error { + requirePartitionsSecured(t, ctx, txFromCtx(t, ctx)) + return nil + })) + }) + } + + dropped, err := dropVerified(cutoff, business.AuditHistoryDropTarget{Name: "audit_events_2011_07", Rows: 1}) + require.NoError(t, err) + require.Equal(t, []string{"audit_events_2011_07"}, dropped) + require.False(t, auditPartitionExists(t, "audit_events_2011_07")) + require.True(t, auditPartitionExists(t, "audit_events_2011_08")) +} diff --git a/module/services/accounts/code/pkg/infra/testdata/control_plane_call_sites.txt b/module/services/accounts/code/pkg/infra/testdata/control_plane_call_sites.txt index e50781ab3..ccfc9829c 100644 --- a/module/services/accounts/code/pkg/infra/testdata/control_plane_call_sites.txt +++ b/module/services/accounts/code/pkg/infra/testdata/control_plane_call_sites.txt @@ -32,6 +32,7 @@ pkg/business/audit.go: (*DurableAuditEmitter).Emit: WithControlPlane pkg/business/audit_history.go: (*AuditHistoryCopy).drop: WithControlPlane pkg/business/audit_history.go: (*AuditHistoryCopy).partitions: WithControlPlane pkg/business/audit_history.go: (*AuditHistoryCopy).readWindow: WithControlPlane +pkg/business/audit_history.go: (*AuditHistoryCopy).uncovered: WithControlPlane pkg/business/audit_store_read.go: (postgresAuditReader).within: WithControlPlane pkg/business/connector_credentials.go: (*Service).SourceSigningSecret: WithControlPlane pkg/business/datasource_budget.go: (datasourceScheduler).Acquire: WithControlPlane diff --git a/module/services/store/migrations/20_audit_verified_partition_drop.down.sql b/module/services/store/migrations/20_audit_verified_partition_drop.down.sql new file mode 100644 index 000000000..65429d805 --- /dev/null +++ b/module/services/store/migrations/20_audit_verified_partition_drop.down.sql @@ -0,0 +1,3 @@ +-- Reverts migration 20. The history copy would then have to retire partitions +-- through audit_events_drop_partitions_before again. +DROP FUNCTION public.audit_events_drop_verified_partitions(timestamp with time zone, text[], bigint[]); diff --git a/module/services/store/migrations/20_audit_verified_partition_drop.up.sql b/module/services/store/migrations/20_audit_verified_partition_drop.up.sql new file mode 100644 index 000000000..df01bfc66 --- /dev/null +++ b/module/services/store/migrations/20_audit_verified_partition_drop.up.sql @@ -0,0 +1,144 @@ +-- Migration 20: drop the audit_events partitions a history copy verified, and +-- exactly those, atomically with a last check of what they hold. +-- +-- The one-time history copy of a deployment switching to a warehouse store of +-- record (ADR 0009, item 7) copies audit_events into the warehouse, verifies the +-- copy, and then retires the copied monthly partitions. It used to do that +-- through audit_events_drop_partitions_before, which has two properties a +-- verified retirement cannot rely on: +-- +-- * It chooses partitions by the month in their NAME, read in the calling +-- session's time zone. The copy chooses the partitions it verified by the +-- bounds PostgreSQL holds for them. A partition created from a session in +-- another zone has bounds on that zone's midnights, so the two choices +-- differ: a drop session in UTC asked to retire everything before 1 September +-- also dropped audit_events_2026_08, whose last four hours (when created +-- from New York) are September's events, which no one had verified. +-- * It drops what it finds. A row committed into a partition between the +-- copy's last recount and the DROP was destroyed unseen: nothing locked the +-- partition, and a DROP waits for an inserting transaction and then takes its +-- row with the table. +-- +-- audit_events_drop_verified_partitions takes the EXPLICIT list of partitions +-- the copy verified, with the number of rows each held when verified, and in one +-- transaction: +-- +-- 1. locks audit_events and then each named partition ACCESS EXCLUSIVE, in a +-- fixed order, so no insert can route into a partition it is about to +-- count and drop, and an insert already in flight is waited for. The lock +-- wait is bounded; a writer that does not finish is a refusal, not a hang; +-- 2. confirms from pg_catalog, after the lock, that each name is a monthly +-- partition of audit_events with a timestamp range whose UPPER bound is at +-- or before the cutoff; +-- 3. recounts each partition and compares the count with the one verified; +-- 4. only then drops exactly the named partitions, and returns their names. +-- +-- Any mismatch raises SQLSTATE AH001 and the transaction rolls back with nothing +-- dropped. The caller compares the returned names with the verified set before +-- it commits. +-- +-- The recount reads the partition by name as its owner. Partitions FORCE +-- row-level security, which holds an owner without BYPASSRLS (the migration +-- principal) to the policies: bound to no organization it would count none of the +-- rows. The function therefore lifts FORCE on each partition for the duration of +-- its own transaction before counting. The partition is held ACCESS EXCLUSIVE, so +-- no other session sees the change, and the transaction either drops the +-- partition or rolls the change back with everything else. +-- +-- Owned by the migration principal, which owns the partitions. SECURITY DEFINER +-- with pg_temp last in its search_path, like every definer function in the +-- store. app_control_plane, the role retention and the history copy run as, is +-- the only runtime role that may execute it; it already holds the authority to +-- drop partitions through audit_events_drop_partitions_before, which stays in +-- place for retention and which the history copy no longer calls. +CREATE FUNCTION public.audit_events_drop_verified_partitions( + cutoff timestamp with time zone, + names text[], + expected_rows bigint[] +) RETURNS text[] + LANGUAGE plpgsql SECURITY DEFINER + SET search_path TO 'pg_catalog', 'public', 'pg_temp' + SET lock_timeout TO '10s' + AS $$ +DECLARE + parent CONSTANT regclass := 'public.audit_events'::regclass; + part_name text; + wanted bigint; + part regclass; + bound text; + bounds text[]; + upper_bound timestamp with time zone; + actual bigint; + dropped text[] := ARRAY[]::text[]; +BEGIN + IF cutoff IS NULL OR names IS NULL OR expected_rows IS NULL + OR array_ndims(names) IS DISTINCT FROM 1 OR array_ndims(expected_rows) IS DISTINCT FROM 1 + OR cardinality(names) = 0 OR cardinality(names) <> cardinality(expected_rows) THEN + RAISE EXCEPTION 'audit history drop refused: a cutoff and one expected row count per partition name are required' + USING ERRCODE = 'AH001'; + END IF; + IF array_position(names, NULL) IS NOT NULL OR array_position(expected_rows, NULL) IS NOT NULL + OR (SELECT count(DISTINCT n) FROM unnest(names) AS n) <> cardinality(names) + OR (SELECT min(e) FROM unnest(expected_rows) AS e) < 0 THEN + RAISE EXCEPTION 'audit history drop refused: partition names must be non-null and distinct, and every expected row count non-negative' + USING ERRCODE = 'AH001'; + END IF; + + -- Parent first, then the partitions by name: inserters take the parent + -- before a partition, so this order cannot deadlock with them. + LOCK TABLE public.audit_events IN ACCESS EXCLUSIVE MODE; + + FOR part_name, wanted IN + SELECT n, e FROM unnest(names, expected_rows) AS t(n, e) ORDER BY n + LOOP + IF part_name !~ '^audit_events_[0-9]{4}_[0-9]{2}$' THEN + RAISE EXCEPTION 'audit history drop refused: % is not a monthly audit_events partition name', part_name + USING ERRCODE = 'AH001'; + END IF; + part := pg_catalog.to_regclass(format('public.%I', part_name)); + IF part IS NULL THEN + RAISE EXCEPTION 'audit history drop refused: partition % does not exist', part_name + USING ERRCODE = 'AH001'; + END IF; + EXECUTE format('LOCK TABLE public.%I IN ACCESS EXCLUSIVE MODE', part_name); + + -- Read the bounds after the lock: PostgreSQL renders them in this + -- session's zone, with the offset, and the cast below reads them back in + -- the same one, so the instant is the partition's own. + SELECT pg_catalog.pg_get_expr(c.relpartbound, c.oid) INTO bound + FROM pg_catalog.pg_inherits i + JOIN pg_catalog.pg_class c ON c.oid = i.inhrelid + WHERE i.inhparent = parent AND c.oid = part AND c.relkind = 'r'; + IF NOT FOUND THEN + RAISE EXCEPTION 'audit history drop refused: % is not attached to audit_events', part_name + USING ERRCODE = 'AH001'; + END IF; + bounds := regexp_match(bound, '^FOR VALUES FROM \(''([^'']+)''\) TO \(''([^'']+)''\)$'); + IF bounds IS NULL THEN + RAISE EXCEPTION 'audit history drop refused: partition % has no timestamp range bounds', part_name + USING ERRCODE = 'AH001'; + END IF; + upper_bound := bounds[2]::timestamp with time zone; + IF upper_bound > cutoff THEN + RAISE EXCEPTION 'audit history drop refused: partition % ends at %, after the cutoff %', part_name, upper_bound, cutoff + USING ERRCODE = 'AH001'; + END IF; + + EXECUTE format('ALTER TABLE public.%I NO FORCE ROW LEVEL SECURITY', part_name); + EXECUTE format('SELECT count(*) FROM public.%I', part_name) INTO actual; + IF actual <> wanted THEN + RAISE EXCEPTION 'audit history drop refused: partition % holds % rows, % were verified', part_name, actual, wanted + USING ERRCODE = 'AH001'; + END IF; + END LOOP; + + FOR part_name IN SELECT n FROM unnest(names) AS n ORDER BY n LOOP + EXECUTE format('DROP TABLE public.%I', part_name); + dropped := dropped || part_name; + END LOOP; + RETURN dropped; +END; +$$; + +REVOKE ALL ON FUNCTION public.audit_events_drop_verified_partitions(timestamp with time zone, text[], bigint[]) FROM PUBLIC; +GRANT EXECUTE ON FUNCTION public.audit_events_drop_verified_partitions(timestamp with time zone, text[], bigint[]) TO app_control_plane; From f6ee3e21a5ba52df2f196b128be2f7207be2053d Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Mon, 5 Oct 2026 17:01:12 -0400 Subject: [PATCH 23/69] fix(audit): give audit-history-copy honest exit codes, a completed-month cutoff, and receipts that name the destination -through must be a completed month in UTC: the current and later months are refused as a usage error, so a run cannot verify and drop a month that is still being written. -batch-size outside 1 to the relay maximum now exits 2 like the other usage errors, as the command header says, rather than 1 from inside the copier. A refused drop used to exit 1 with only operation_failed on screen. It exits 4 now, and the non-JSON output prints why (a partition's recount disagreed, the digest did not match, the cutoff has not passed). A run with no partition to verify exits 5 and says nothing was verified, where it used to report verified with exit 0. Events older than the cutoff that no verified partition covers fail verification (exit 3) and are listed by table. The JSON receipt carries the same outcomes as error_code values (verification_failed, drop_refused, nothing_to_verify) plus drop_refused and uncovered. Receipts identified a ClickHouse destination by nothing: the safe configuration left out its database and cluster, so two stores differing only there read the same. Both are now included, the DSN, host and password never. Co-Authored-By: Claude Sonnet 5.5 --- .../accounts/code/pkg/auditops/common.go | 7 + .../accounts/code/pkg/auditops/history.go | 137 +++++++++++----- .../code/pkg/auditops/history_test.go | 150 +++++++++++++++++- .../code/pkg/auditops/qualification_test.go | 2 +- 4 files changed, 249 insertions(+), 47 deletions(-) diff --git a/module/services/accounts/code/pkg/auditops/common.go b/module/services/accounts/code/pkg/auditops/common.go index bd988f508..8dc4c7622 100644 --- a/module/services/accounts/code/pkg/auditops/common.go +++ b/module/services/accounts/code/pkg/auditops/common.go @@ -34,6 +34,11 @@ func openStore(ctx context.Context, testURL string) (*infra.PostgresStore, error } return infra.NewPostgresStore(ctx) } + +// safeConfiguration is what a receipt may say about where the commands pointed: +// enough to identify the warehouse, archive and deployment, and never a +// credential. A ClickHouse store is identified by its database and cluster (the +// DSN names a host and carries the password, and neither is reported). func safeConfiguration(swap *auditsink.Swap) map[string]string { out := map[string]string{ "AUDIT_SINK": string(swap.Mode), "AUDIT_DEPLOYMENT_ID": swap.DeploymentID, @@ -45,6 +50,8 @@ func safeConfiguration(swap *auditsink.Swap) map[string]string { out["AUDIT_BIGQUERY_DATASET"] = swap.BigQuery.Dataset } if swap.ClickHouse != nil { + out["AUDIT_CLICKHOUSE_DATABASE"] = swap.ClickHouse.Database + out["AUDIT_CLICKHOUSE_CLUSTER"] = swap.ClickHouse.Cluster out["AUDIT_EVENTS_RETENTION_DAYS"] = strconv.Itoa(int(swap.ClickHouse.EventsRetention / (24 * time.Hour))) } return out diff --git a/module/services/accounts/code/pkg/auditops/history.go b/module/services/accounts/code/pkg/auditops/history.go index 4431c3755..ef347a79a 100644 --- a/module/services/accounts/code/pkg/auditops/history.go +++ b/module/services/accounts/code/pkg/auditops/history.go @@ -3,27 +3,41 @@ // copies the audit_events rows the deployment already holds into the store of // record and the locked archive — each event classified, serialized and hashed // exactly as the relay does a live one — and verifies the copy by reading it -// back: every row present, every copy identical, per organization. Only with +// back: every row present, every copy identical, per organization, and no event +// older than the cutoff held outside the verified partitions. Only with // -confirm-drop, and only after this run verified every copied partition, it -// drops the copied monthly partitions, the way retention drops them; no row is -// ever deleted. +// drops those monthly partitions, named one by one: the database locks them, +// recounts each against what was verified and drops exactly them, in one +// transaction, and the receipt names what it dropped. No row is ever deleted. // // Usage: // -// audit-history-copy [-through YYYY-MM] [-verify-only] [-confirm-drop -expected-partitions-sha256 HEX] [-json] +// audit-history-copy [-through YYYY-MM] [-verify-only] [-confirm-drop -expected-partitions-sha256 HEX] [-batch-size N] [-json] // -// It reads the same AUDIT_* settings as the accounts service and runs only -// under a swap value: run it with the service's own environment, after the -// service has switched, so nothing writes audit_events any more. The -// connection principal must be a member of app_control_plane, the authority -// retention runs under. +// -through names the last month copied and must be a completed month (UTC): the +// current month and later ones are refused. It reads the same AUDIT_* settings +// as the accounts service and runs only under a swap value: run it with the +// service's own environment, after the service has switched, so nothing writes +// audit_events any more. The connection principal must be a member of +// app_control_plane, the authority retention runs under. // -// It is resumable: a run reads back what the store holds before writing, so a -// run after an interrupted one copies only what is missing. -verify-only -// copies nothing, for a re-check once the store has caught up. +// It is resumable: a run reads each window's rows, reads back what the store +// holds, appends only the events missing from it, reads the window back again +// and verifies every event, so a run after an interrupted one copies only what +// is missing. -verify-only copies nothing, for a re-check once the store has +// caught up. // -// Exit status: 0 done; 1 failed; 2 usage; 3 verification failed, nothing -// dropped. +// Exit status: +// +// 0 done +// 1 failed (configuration, connection, warehouse or database error) +// 2 usage (a flag or its value is invalid) +// 3 verification failed: an event is missing or differs, or events older +// than the cutoff sit outside the verified partitions; nothing dropped +// 4 drop refused: what was verified no longer holds, or the approval does +// not match; nothing dropped +// 5 nothing to verify: no monthly partition ends at or before the cutoff and +// nothing older than it is held elsewhere; nothing claimed, nothing dropped package auditops import ( @@ -35,6 +49,7 @@ import ( "io" "regexp" "sort" + "strings" "time" "accounts/pkg/auditstore/auditsink" @@ -57,23 +72,23 @@ type options struct { // parse reads the command line and the sink settings, refusing everything a // run cannot start from. -func parse(args []string, getenv func(string) string, stderr io.Writer) (options, *auditsink.Swap, int) { +func parse(args []string, getenv func(string) string, stderr io.Writer, now time.Time) (options, *auditsink.Swap, int) { fs := flag.NewFlagSet("audit-history-copy", flag.ContinueOnError) fs.SetOutput(stderr) databaseURL := getenv("DATABASE_URL") machine := fs.Bool("json", false, "emit a sanitized machine receipt") capabilities := fs.Bool("capabilities-json", false, "report supported command schemas and destination without connecting") expected := fs.String("expected-partitions-sha256", "", "digest from prior verification, required for drop") - through := fs.String("through", "", "copy only the partitions up to and including this month, YYYY-MM (default every partition)") + through := fs.String("through", "", "copy only the partitions up to and including this completed month (UTC), YYYY-MM (default every partition)") verifyOnly := fs.Bool("verify-only", false, "copy nothing; verify what the store already holds") confirmDrop := fs.Bool("confirm-drop", false, "after a verification pass, drop the copied partitions") batchSize := fs.Int("batch-size", business.DefaultAuditRelayBatchSize, "events per copied batch") if err := fs.Parse(args); err != nil { - return options{}, nil, 2 + return options{}, nil, exitUsage } if fs.NArg() > 0 { line(stderr, "audit-history-copy: unexpected arguments:", fs.Args()) - return options{}, nil, 2 + return options{}, nil, exitUsage } opts := options{ databaseURL: databaseURL, @@ -85,30 +100,39 @@ func parse(args []string, getenv func(string) string, stderr io.Writer) (options monthStart, err := time.Parse("2006-01", *through) if err != nil { line(stderr, "audit-history-copy: -through must be a month, YYYY-MM; got", *through) - return options{}, nil, 2 + return options{}, nil, exitUsage } end := monthStart.AddDate(0, 1, 0) + if end.After(now.UTC()) { + line(stderr, fmt.Sprintf("audit-history-copy: -through must be a completed month (UTC); %s ends %s", + *through, end.Format(time.RFC3339))) + return options{}, nil, exitUsage + } opts.through = &end } + if *batchSize < 1 || *batchSize > business.MaxAuditRelayBatchSize { + line(stderr, fmt.Sprintf("audit-history-copy: -batch-size must be between 1 and %d; got %d", business.MaxAuditRelayBatchSize, *batchSize)) + return options{}, nil, exitUsage + } if *confirmDrop && (*through == "" || !*verifyOnly || !regexp.MustCompile(`^[a-f0-9]{64}$`).MatchString(*expected)) { line(stderr, "audit-history-copy: drop requires -through YYYY-MM, -verify-only and -expected-partitions-sha256") - return options{}, nil, 2 + return options{}, nil, exitUsage } sink, err := auditsink.Load(getenv) if err != nil { line(stderr, "audit-history-copy: invalid audit destination configuration") - return options{}, nil, 1 + return options{}, nil, exitFailed } if sink.Swap == nil { line(stderr, fmt.Sprintf("audit-history-copy: AUDIT_SINK is %s; the history copy runs only under a swap value, "+ "after the service has switched, so nothing writes audit_events any more", sink.Mode)) - return options{}, nil, 1 + return options{}, nil, exitFailed } return opts, sink.Swap, 0 } func RunHistory(args []string, getenv func(string) string, stdout, stderr io.Writer) int { - opts, swap, code := parse(args, getenv, stderr) + opts, swap, code := parse(args, getenv, stderr, time.Now()) if swap == nil { return code } @@ -156,6 +180,16 @@ func RunHistory(args []string, getenv func(string) string, stdout, stderr io.Wri return summarize(stdout, stderr, report, err) } +// The exit statuses of audit-history-copy, documented in the command header and +// in module/AUDIT_OPERATIONS.md. +const ( + exitFailed = 1 + exitUsage = 2 + exitUnverified = 3 + exitDropRefused = 4 + exitNothingToVerify = 5 +) + // summarize prints what the run found and maps it to the exit status. Pure over // its inputs so the mapping is testable without a database or a warehouse. func summarize(stdout, stderr io.Writer, report business.AuditHistoryReport, err error) int { @@ -180,15 +214,27 @@ func summarize(stdout, stderr io.Writer, report business.AuditHistoryReport, err line(stderr, fmt.Sprintf(" %s: and %d more", partition.Partition.Name, hidden)) } } + for _, table := range report.Uncovered { + line(stderr, fmt.Sprintf(" %s holds %d events older than %s that no verified partition covers", + table.Table, table.Rows, report.Cutoff.UTC().Format(time.RFC3339))) + } switch { case errors.Is(err, business.ErrAuditHistoryUnverified): line(stderr, "audit-history-copy:", err) - return 3 + return exitUnverified + case errors.Is(err, business.ErrAuditHistoryNothingToVerify): + line(stderr, fmt.Sprintf("audit-history-copy: nothing to verify: no audit_events partition ends at or before %s, "+ + "and no event older than it is held elsewhere; nothing was verified or dropped", report.Cutoff.UTC().Format(time.RFC3339))) + return exitNothingToVerify + case errors.Is(err, business.ErrAuditHistoryDropRefused): + line(stderr, "audit-history-copy: drop refused: "+report.DropRefused+"; nothing was dropped") + return exitDropRefused case err != nil: line(stderr, "audit-history-copy: operation_failed") - return 1 + return exitFailed case report.Dropped > 0: - line(stdout, fmt.Sprintf("verified; dropped %d partitions ending at or before %s", report.Dropped, report.Cutoff.UTC().Format(time.RFC3339))) + line(stdout, fmt.Sprintf("verified; dropped %d partitions ending at or before %s: %s", report.Dropped, + report.Cutoff.UTC().Format(time.RFC3339), strings.Join(report.DroppedNames, ", "))) default: line(stdout, "verified") } @@ -209,7 +255,17 @@ type HistoryReceipt struct { Partitions []HistoryPartition `json:"partitions"` Dropped []string `json:"dropped"` Cutoff time.Time `json:"cutoff"` - ErrorCode string `json:"error_code"` + // Uncovered are the tables holding events older than the cutoff that no + // verified partition covers; any makes the run unverified. + Uncovered []HistoryUncovered `json:"uncovered"` + // DropRefused is why a confirmed drop did not happen; it names partitions + // and row counts only. + DropRefused string `json:"drop_refused"` + ErrorCode string `json:"error_code"` +} +type HistoryUncovered struct { + Table string `json:"table"` + Rows int64 `json:"rows"` } type HistoryPartition struct { Name string `json:"name"` @@ -230,17 +286,20 @@ func writeHistory(out io.Writer, opts options, swap *auditsink.Swap, report busi if receipt.Dropped == nil { receipt.Dropped = []string{} } - code := 0 - if err != nil { - code = 1 - receipt.ErrorCode = "operation_failed" + receipt.Uncovered = []HistoryUncovered{} + for _, table := range report.Uncovered { + receipt.Uncovered = append(receipt.Uncovered, HistoryUncovered{Table: table.Table, Rows: table.Rows}) } - if errors.Is(err, business.ErrAuditHistoryUnverified) { - code = 3 - receipt.ErrorCode = "verification_failed" - } - if report.DropRefused != "" { - receipt.ErrorCode = "drop_refused" + code := 0 + switch { + case errors.Is(err, business.ErrAuditHistoryUnverified): + code, receipt.ErrorCode = exitUnverified, "verification_failed" + case errors.Is(err, business.ErrAuditHistoryNothingToVerify): + code, receipt.ErrorCode = exitNothingToVerify, "nothing_to_verify" + case errors.Is(err, business.ErrAuditHistoryDropRefused): + code, receipt.ErrorCode, receipt.DropRefused = exitDropRefused, "drop_refused", report.DropRefused + case err != nil: + code, receipt.ErrorCode = exitFailed, "operation_failed" } for _, p := range report.Partitions { part := HistoryPartition{Name: p.Partition.Name, From: p.Partition.From, To: p.Partition.To, Failures: p.Failures, EventsSHA256: p.EventsSHA256, Orgs: []HistoryOrg{}} @@ -256,7 +315,7 @@ func writeHistory(out io.Writer, opts options, swap *auditsink.Swap, report busi receipt.Partitions = append(receipt.Partitions, part) } if json.NewEncoder(out).Encode(receipt) != nil { - return 1 + return exitFailed } return code } @@ -265,5 +324,5 @@ func historyFailure(out, stderr io.Writer, opts options, swap *auditsink.Swap, c return writeHistory(out, opts, swap, business.AuditHistoryReport{}, errors.New(code)) } line(stderr, "audit-history-copy:", code) - return 1 + return exitFailed } diff --git a/module/services/accounts/code/pkg/auditops/history_test.go b/module/services/accounts/code/pkg/auditops/history_test.go index da0e31b28..12825a705 100644 --- a/module/services/accounts/code/pkg/auditops/history_test.go +++ b/module/services/accounts/code/pkg/auditops/history_test.go @@ -2,7 +2,9 @@ package auditops import ( "bytes" + "encoding/json" "errors" + "fmt" "testing" "time" @@ -25,18 +27,22 @@ var swapEnv = map[string]string{ "AUDIT_CONTENT_RETENTION_DAYS": "90", } +// historyNow is after July 2026 and inside October, so -through 2026-09 is the +// last completed month and 2026-10 the current one. +var historyNow = time.Date(2026, 10, 5, 9, 30, 0, 0, time.UTC) + func TestParseRunsOnlyUnderASwapValue(t *testing.T) { for _, sink := range []string{"", "postgres", "both"} { values := map[string]string{"DATABASE_URL": "postgres://example.invalid/accounts", "AUDIT_SINK": sink, "AUDIT_EXTERNAL_URL": "https://audit.example/ingest"} var stderr bytes.Buffer - _, swap, code := parse(nil, env(values), &stderr) + _, swap, code := parse(nil, env(values), &stderr, historyNow) require.Nil(t, swap, sink) require.Equal(t, 1, code, sink) require.Contains(t, stderr.String(), "runs only under a swap value", sink) } var stderr bytes.Buffer - opts, swap, code := parse([]string{"-through", "2026-06", "-confirm-drop", "-verify-only", "-expected-partitions-sha256", "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", "-batch-size", "200"}, env(swapEnv), &stderr) + opts, swap, code := parse([]string{"-through", "2026-06", "-confirm-drop", "-verify-only", "-expected-partitions-sha256", "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", "-batch-size", "200"}, env(swapEnv), &stderr, historyNow) require.Equal(t, 0, code, stderr.String()) require.NotNil(t, swap) require.Equal(t, "deployment-1", swap.DeploymentID) @@ -54,13 +60,18 @@ func TestParseRefusesWhatARunCannotStartFrom(t *testing.T) { want string }{ "a malformed month": {args: []string{"-through", "June"}, values: swapEnv, code: 2, want: "YYYY-MM"}, + "the current month": {args: []string{"-through", "2026-10"}, values: swapEnv, code: 2, want: "completed month"}, + "a future month": {args: []string{"-through", "2027-01"}, values: swapEnv, code: 2, want: "completed month"}, + "a batch size of zero": {args: []string{"-batch-size", "0"}, values: swapEnv, code: 2, want: "-batch-size must be between 1 and"}, + "a negative batch": {args: []string{"-batch-size", "-5"}, values: swapEnv, code: 2, want: "-batch-size must be between 1 and"}, + "a batch past the cap": {args: []string{"-batch-size", fmt.Sprint(business.MaxAuditRelayBatchSize + 1)}, values: swapEnv, code: 2, want: "-batch-size must be between 1 and"}, "a stray argument": {args: []string{"now"}, values: swapEnv, code: 2, want: "unexpected arguments"}, "unbounded drop": {args: []string{"-confirm-drop"}, values: swapEnv, code: 2, want: "drop requires"}, "a swap missing parts": {values: map[string]string{"DATABASE_URL": "postgres://example.invalid/a", "AUDIT_SINK": "bigquery"}, code: 1, want: "invalid audit destination configuration"}, } { t.Run(name, func(t *testing.T) { var stderr bytes.Buffer - _, swap, code := parse(tc.args, env(tc.values), &stderr) + _, swap, code := parse(tc.args, env(tc.values), &stderr, historyNow) require.Nil(t, swap) require.Equal(t, tc.code, code) require.Contains(t, stderr.String(), tc.want) @@ -68,6 +79,19 @@ func TestParseRefusesWhatARunCannotStartFrom(t *testing.T) { } } +func TestParseAcceptsTheLastCompletedMonth(t *testing.T) { + var stderr bytes.Buffer + opts, swap, code := parse([]string{"-through", "2026-09", "-batch-size", fmt.Sprint(business.MaxAuditRelayBatchSize)}, env(swapEnv), &stderr, historyNow) + require.Equal(t, 0, code, stderr.String()) + require.NotNil(t, swap) + require.Equal(t, time.Date(2026, 10, 1, 0, 0, 0, 0, time.UTC), *opts.through) + // The month ends the instant it ends: at that instant it is complete. + _, _, code = parse([]string{"-through", "2026-09"}, env(swapEnv), &stderr, time.Date(2026, 10, 1, 0, 0, 0, 0, time.UTC)) + require.Equal(t, 0, code) + _, _, code = parse([]string{"-through", "2026-09"}, env(swapEnv), &stderr, time.Date(2026, 9, 30, 23, 59, 59, 0, time.UTC)) + require.Equal(t, 2, code) +} + func TestSummarizeExitStatus(t *testing.T) { partition := business.AuditHistoryPartitionReport{ Partition: business.AuditHistoryPartition{Name: "audit_events_2026_08"}, @@ -89,8 +113,120 @@ func TestSummarizeExitStatus(t *testing.T) { require.Contains(t, stdout.String(), "verified") stdout.Reset() cutoff := time.Date(2026, 9, 1, 0, 0, 0, 0, time.UTC) - require.Equal(t, 0, summarize(&stdout, &stderr, business.AuditHistoryReport{Verified: true, Dropped: 2, Cutoff: cutoff}, nil)) - require.Contains(t, stdout.String(), "dropped 2 partitions ending at or before 2026-09-01T00:00:00Z") + require.Equal(t, 0, summarize(&stdout, &stderr, business.AuditHistoryReport{ + Verified: true, Dropped: 2, Cutoff: cutoff, DroppedNames: []string{"audit_events_2026_07", "audit_events_2026_08"}, + }, nil)) + require.Contains(t, stdout.String(), "dropped 2 partitions ending at or before 2026-09-01T00:00:00Z: audit_events_2026_07, audit_events_2026_08") +} + +// A refused drop, a run with nothing to verify and a run that found events +// outside the verified partitions each say what they found and exit with a +// status of their own, so a pipeline can tell them from a failure. +func TestSummarizeDistinguishesRefusedNothingToVerifyAndUncovered(t *testing.T) { + cutoff := time.Date(2026, 9, 1, 0, 0, 0, 0, time.UTC) + + var stdout, stderr bytes.Buffer + refused := business.AuditHistoryReport{Verified: true, Cutoff: cutoff, DropRefused: "partition audit_events_2026_08 holds 6 rows, 5 were verified"} + require.Equal(t, 4, summarize(&stdout, &stderr, refused, fmt.Errorf("%w: %s", business.ErrAuditHistoryDropRefused, refused.DropRefused))) + require.Contains(t, stderr.String(), "drop refused: partition audit_events_2026_08 holds 6 rows, 5 were verified") + require.Contains(t, stderr.String(), "nothing was dropped") + require.NotContains(t, stderr.String(), "operation_failed") + + stdout.Reset() + stderr.Reset() + require.Equal(t, 5, summarize(&stdout, &stderr, business.AuditHistoryReport{Cutoff: cutoff}, business.ErrAuditHistoryNothingToVerify)) + require.Contains(t, stderr.String(), "nothing to verify") + require.Contains(t, stderr.String(), "2026-09-01T00:00:00Z") + require.NotContains(t, stdout.String(), "verified\n", "a run with nothing to verify does not print the success line") + + stdout.Reset() + stderr.Reset() + uncovered := business.AuditHistoryReport{Cutoff: cutoff, Uncovered: []business.AuditHistoryTableRows{{Table: "audit_events_default", Rows: 7}}} + require.Equal(t, 3, summarize(&stdout, &stderr, uncovered, business.ErrAuditHistoryUnverified)) + require.Contains(t, stderr.String(), "audit_events_default holds 7 events older than 2026-09-01T00:00:00Z that no verified partition covers") +} + +func TestHistoryReceiptCarriesTheSameOutcomes(t *testing.T) { + var stderr bytes.Buffer + _, swap, code := parse(nil, env(swapEnv), &stderr, historyNow) + require.Zero(t, code) + cutoff := time.Date(2026, 9, 1, 0, 0, 0, 0, time.UTC) + for name, tc := range map[string]struct { + report business.AuditHistoryReport + err error + code int + want string + }{ + "verified": {report: business.AuditHistoryReport{Verified: true, Cutoff: cutoff}, code: 0, want: ""}, + "unverified": {report: business.AuditHistoryReport{Cutoff: cutoff}, err: business.ErrAuditHistoryUnverified, code: 3, want: "verification_failed"}, + "nothing": {report: business.AuditHistoryReport{Cutoff: cutoff}, err: business.ErrAuditHistoryNothingToVerify, code: 5, want: "nothing_to_verify"}, + "refused": {report: business.AuditHistoryReport{Verified: true, Cutoff: cutoff, DropRefused: "partition audit_events_2026_08 holds 6 rows, 5 were verified"}, err: fmt.Errorf("%w: x", business.ErrAuditHistoryDropRefused), code: 4, want: "drop_refused"}, + "another failure": {report: business.AuditHistoryReport{}, err: errors.New("warehouse unavailable"), code: 1, want: "operation_failed"}, + } { + t.Run(name, func(t *testing.T) { + var out bytes.Buffer + require.Equal(t, tc.code, writeHistory(&out, options{json: true}, swap, tc.report, tc.err)) + var receipt HistoryReceipt + require.NoError(t, json.Unmarshal(out.Bytes(), &receipt)) + require.Equal(t, tc.want, receipt.ErrorCode) + require.Equal(t, tc.report.Verified, receipt.Verified) + require.Equal(t, tc.report.DropRefused, receipt.DropRefused) + require.NotNil(t, receipt.Uncovered, "present as an empty list when there is none") + }) + } + + var out bytes.Buffer + uncovered := business.AuditHistoryReport{Cutoff: cutoff, Uncovered: []business.AuditHistoryTableRows{{Table: "audit_events_default", Rows: 7}}} + require.Equal(t, 3, writeHistory(&out, options{json: true}, swap, uncovered, business.ErrAuditHistoryUnverified)) + var receipt HistoryReceipt + require.NoError(t, json.Unmarshal(out.Bytes(), &receipt)) + require.Equal(t, []HistoryUncovered{{Table: "audit_events_default", Rows: 7}}, receipt.Uncovered) +} + +// A receipt identifies the destination by what is not secret. A ClickHouse store +// is its database and cluster, which the DSN's host and password are not. +func TestReceiptsIdentifyAClickHouseDestinationWithoutItsDSN(t *testing.T) { + values := map[string]string{ + "DATABASE_URL": "postgres://example.invalid/accounts", + "AUDIT_SINK": "clickhouse", + "AUDIT_CLICKHOUSE_DSN": "clickhouse://audit_writer:hunter2-secret@clickhouse.internal.example:9000/audit_db", + "AUDIT_CLICKHOUSE_CLUSTER": "audit_cluster", + "AUDIT_EVENTS_RETENTION_DAYS": "400", + "AUDIT_ARCHIVE_URL": "gs://example-audit-archive", + "AUDIT_DEPLOYMENT_ID": "deployment-1", + "AUDIT_CONTENT_RETENTION_DAYS": "90", + } + var stderr bytes.Buffer + _, swap, code := parse(nil, env(values), &stderr, historyNow) + require.Zero(t, code, stderr.String()) + + var history bytes.Buffer + require.Equal(t, 0, writeHistory(&history, options{json: true}, swap, business.AuditHistoryReport{Verified: true}, nil)) + var capabilities bytes.Buffer + require.Equal(t, 0, RunHistory([]string{"-capabilities-json"}, env(values), &capabilities, &stderr)) + for _, body := range []string{history.String(), capabilities.String()} { + var receipt struct { + Configuration map[string]string `json:"configuration"` + } + require.NoError(t, json.Unmarshal([]byte(body), &receipt)) + require.Equal(t, "audit_db", receipt.Configuration["AUDIT_CLICKHOUSE_DATABASE"]) + require.Equal(t, "audit_cluster", receipt.Configuration["AUDIT_CLICKHOUSE_CLUSTER"]) + require.Equal(t, "400", receipt.Configuration["AUDIT_EVENTS_RETENTION_DAYS"]) + for _, secret := range []string{"hunter2-secret", "audit_writer", "clickhouse.internal.example", "clickhouse://", "AUDIT_CLICKHOUSE_DSN"} { + require.NotContains(t, body, secret) + } + } + + // A ClickHouse store with no cluster says so; two stores that differ only in + // the database are told apart. + delete(values, "AUDIT_CLICKHOUSE_CLUSTER") + values["AUDIT_CLICKHOUSE_DSN"] = "clickhouse://audit_writer:hunter2-secret@clickhouse.internal.example:9000/other_db" + _, swap, code = parse(nil, env(values), &stderr, historyNow) + require.Zero(t, code) + configuration := safeConfiguration(swap) + require.Equal(t, "other_db", configuration["AUDIT_CLICKHOUSE_DATABASE"]) + require.Contains(t, configuration, "AUDIT_CLICKHOUSE_CLUSTER") + require.Empty(t, configuration["AUDIT_CLICKHOUSE_CLUSTER"]) } func TestCapabilitiesAndHistoryMachineOutputExcludeSecretsAndProviderErrors(t *testing.T) { @@ -101,7 +237,7 @@ func TestCapabilitiesAndHistoryMachineOutputExcludeSecretsAndProviderErrors(t *t require.NotContains(t, out.String(), "DATABASE_URL") require.NotContains(t, out.String(), "postgres://") out.Reset() - _, swap, code := parse(nil, env(swapEnv), &stderr) + _, swap, code := parse(nil, env(swapEnv), &stderr, historyNow) require.Zero(t, code) require.Equal(t, 1, writeHistory(&out, options{json: true}, swap, business.AuditHistoryReport{}, errors.New("provider URL containing password=private"))) require.NotContains(t, out.String(), "private") @@ -113,7 +249,7 @@ func TestCapabilitiesAndHistoryMachineOutputExcludeSecretsAndProviderErrors(t *t values[k] = v } delete(values, "DATABASE_URL") - opts, swap, code := parse(nil, env(values), &stderr) + opts, swap, code := parse(nil, env(values), &stderr, historyNow) require.Zero(t, code) require.NotNil(t, swap) require.Empty(t, opts.databaseURL) diff --git a/module/services/accounts/code/pkg/auditops/qualification_test.go b/module/services/accounts/code/pkg/auditops/qualification_test.go index 93a1fdadb..e848062e5 100644 --- a/module/services/accounts/code/pkg/auditops/qualification_test.go +++ b/module/services/accounts/code/pkg/auditops/qualification_test.go @@ -104,7 +104,7 @@ func (s *probeWarehouse) LatestSourceSyncEvents(_ context.Context, scope busines func probeReceipt(t *testing.T, o qualificationOptions, records []business.AuditRecord) QualificationReceipt { t.Helper() var stderr bytes.Buffer - _, swap, code := parse(nil, env(swapEnv), &stderr) + _, swap, code := parse(nil, env(swapEnv), &stderr, probeNow) require.Zero(t, code) return newQualificationReceipt(o, swap, records) } From 64b3cbdfe06b3fc477fad3337506e4d0ce60fcaf Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Mon, 5 Oct 2026 17:01:49 -0400 Subject: [PATCH 24/69] fix(audit): make postgres_no_new_rows check the interval it reports The check was vacuous at prepare and partial at verify. Prepare counted audit_events rows in a five-minute window that began at its own start and ran five minutes into the future, so it could only ever see the few milliseconds before the probe. Verify counted the same window, and marked all of it passed while only the part up to its own run time had happened; rows created after the window closed and before verify ran were never looked at. The probe now reads exactly [window_from, now): prepare starts the interval and checks it up to the end of its fixture transaction, and verify checks from that same start to its own run time, whether inside the event window or long after. The receipt carries the interval it read as postgres_checked_from and postgres_checked_to, so a verify run inside the window claims only the part of it that has happened. An interval with no width fails rather than passing on nothing. Tests: prepare's interval and its fixture abort, the half-open boundaries, a clock that has not moved, verify after the window (a row at +10 minutes now fails it), and verify inside the window. Co-Authored-By: Claude Sonnet 5.5 --- .../code/pkg/auditops/qualification.go | 78 ++++++++++++---- .../code/pkg/auditops/qualification_test.go | 92 ++++++++++++++++++- .../pkg/infra/audit_qualification_test.go | 10 +- 3 files changed, 158 insertions(+), 22 deletions(-) diff --git a/module/services/accounts/code/pkg/auditops/qualification.go b/module/services/accounts/code/pkg/auditops/qualification.go index 11abee841..c9194c405 100644 --- a/module/services/accounts/code/pkg/auditops/qualification.go +++ b/module/services/accounts/code/pkg/auditops/qualification.go @@ -20,17 +20,25 @@ import ( // actually observed. Gateway, archive, alert and controlled outage acceptance // need separate observers and remain pending here. type QualificationReceipt struct { - Schema string `json:"schema"` - DeploymentID string `json:"deployment_id"` - Sink string `json:"sink"` - Configuration map[string]string `json:"configuration"` - RunID string `json:"run_id"` - ObservedAt time.Time `json:"observed_at"` - WindowFrom time.Time `json:"window_from"` - WindowTo time.Time `json:"window_to"` - Events []QualificationEvent `json:"events"` - Checks map[string]string `json:"checks"` - ErrorCode string `json:"error_code"` + Schema string `json:"schema"` + DeploymentID string `json:"deployment_id"` + Sink string `json:"sink"` + Configuration map[string]string `json:"configuration"` + RunID string `json:"run_id"` + ObservedAt time.Time `json:"observed_at"` + WindowFrom time.Time `json:"window_from"` + WindowTo time.Time `json:"window_to"` + // PostgresCheckedFrom and PostgresCheckedTo are the interval of + // audit_events.created_at the postgres_no_new_rows check covered: rows with + // created_at in [from, to), where to is the instant the check ran. Prepare + // starts the interval (window_from) and verify extends it to its own run + // time, so a verify run after prepare covers everything in between. They are + // absent until the check has run. + PostgresCheckedFrom *time.Time `json:"postgres_checked_from,omitempty"` + PostgresCheckedTo *time.Time `json:"postgres_checked_to,omitempty"` + Events []QualificationEvent `json:"events"` + Checks map[string]string `json:"checks"` + ErrorCode string `json:"error_code"` } type QualificationEvent struct { ID string `json:"id"` @@ -56,6 +64,16 @@ type qualificationOptions struct { prepare bool from, to time.Time timeout time.Duration + // clock reads the time the Postgres check ends its interval at; nil is + // time.Now. A seam for tests. + clock func() time.Time +} + +func (o qualificationOptions) now() time.Time { + if o.clock != nil { + return o.clock() + } + return time.Now() } func parseQualification(args []string, stderr io.Writer, now time.Time) (qualificationOptions, bool, error) { @@ -234,17 +252,33 @@ type qualificationDB interface { CountAuditHistory(context.Context, time.Time, time.Time) (map[string]int64, error) } -func postgresProbe(ctx context.Context, o qualificationOptions, db qualificationDB) error { - counts, err := db.CountAuditHistory(ctx, o.from, o.to) +// postgresProbe counts the audit_events rows whose created_at lies in +// [o.from, now): the interval it reports, and the only one it vouches for. Rows +// are created at their transaction's start, so a row from a transaction still +// open at the probe, or one written with an explicit earlier created_at, is not +// in what it saw; the receipt says what interval was read, not that no row can +// exist. +func postgresProbe(ctx context.Context, o qualificationOptions, db qualificationDB) (checkedTo time.Time, err error) { + checkedTo = o.now().UTC().Truncate(time.Microsecond) + if !checkedTo.After(o.from) { + return checkedTo, errors.New("postgres probe interval is empty") + } + counts, err := db.CountAuditHistory(ctx, o.from, checkedTo) if err != nil { - return err + return checkedTo, err } for _, n := range counts { if n != 0 { - return errors.New("postgres received new audit rows") + return checkedTo, errors.New("postgres received new audit rows") } } - return nil + return checkedTo, nil +} + +// recordPostgresCheck puts the interval a probe covered on the receipt. +func recordPostgresCheck(r *QualificationReceipt, from, to time.Time) { + from, to = from.UTC(), to.UTC() + r.PostgresCheckedFrom, r.PostgresCheckedTo = &from, &to } func prepareQualification(ctx context.Context, o qualificationOptions, db qualificationDB, recorder business.AuditRecorder, records []business.AuditRecord, r *QualificationReceipt) error { err := db.WithControlPlane(ctx, func(ctx context.Context) error { @@ -259,7 +293,9 @@ func prepareQualification(ctx context.Context, o qualificationOptions, db qualif return err } } - return postgresProbe(ctx, o, db) + checkedTo, err := postgresProbe(ctx, o, db) + recordPostgresCheck(r, o.from, checkedTo) + return err }) if err != nil { r.Checks["transaction_queue"] = "failed" @@ -319,7 +355,13 @@ func verifyQualification(ctx context.Context, o qualificationOptions, db qualifi case <-time.After(2 * time.Second): } } - if err := db.WithControlPlane(ctx, func(ctx context.Context) error { return postgresProbe(ctx, o, db) }); err != nil { + // From the start prepare recorded to now, not the five-minute window: the + // check covers everything created since prepare began, and says so. + if err := db.WithControlPlane(ctx, func(ctx context.Context) error { + checkedTo, err := postgresProbe(ctx, o, db) + recordPostgresCheck(r, o.from, checkedTo) + return err + }); err != nil { r.Checks["postgres_no_new_rows"] = "failed" return err } diff --git a/module/services/accounts/code/pkg/auditops/qualification_test.go b/module/services/accounts/code/pkg/auditops/qualification_test.go index e848062e5..54b736e97 100644 --- a/module/services/accounts/code/pkg/auditops/qualification_test.go +++ b/module/services/accounts/code/pkg/auditops/qualification_test.go @@ -17,12 +17,17 @@ import ( var probeNow = time.Date(2026, 10, 5, 12, 0, 0, 0, time.UTC) func probeOptions() qualificationOptions { - return qualificationOptions{orgs: organizations{"aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa", "bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb"}, runID: uuid.MustParse("cccccccc-cccc-4ccc-8ccc-cccccccccccc"), from: probeNow, to: probeNow.Add(5 * time.Minute), timeout: time.Minute} + return qualificationOptions{orgs: organizations{"aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa", "bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb"}, runID: uuid.MustParse("cccccccc-cccc-4ccc-8ccc-cccccccccccc"), from: probeNow, to: probeNow.Add(5 * time.Minute), timeout: time.Minute, + clock: func() time.Time { return probeNow.Add(2 * time.Second) }} } type probeDB struct { exists bool count int64 + // rows are the created_at of the audit_events rows the database holds; the + // count of an interval is how many fall in [from, to). + rows []time.Time + calls [][2]time.Time writes []business.AuditEntry } @@ -35,8 +40,15 @@ func (d *probeDB) WithControlPlane(ctx context.Context, fn func(context.Context) return err } func (d *probeDB) OrganizationIDExists(context.Context, string) (bool, error) { return d.exists, nil } -func (d *probeDB) CountAuditHistory(context.Context, time.Time, time.Time) (map[string]int64, error) { - return map[string]int64{"": d.count}, nil +func (d *probeDB) CountAuditHistory(_ context.Context, from, to time.Time) (map[string]int64, error) { + d.calls = append(d.calls, [2]time.Time{from, to}) + n := d.count + for _, at := range d.rows { + if !at.Before(from) && at.Before(to) { + n++ + } + } + return map[string]int64{"": n}, nil } func (d *probeDB) RecordTx(_ context.Context, e business.AuditEntry) error { d.writes = append(d.writes, e) @@ -126,6 +138,80 @@ func TestQualificationCommitsOnlyIntoExistingOrganizationsAndRefusesNewPostgresR require.Equal(t, "passed", r.Checks["transaction_queue"]) require.Equal(t, "pending", r.Checks["warehouse_readback"]) } + +// postgres_no_new_rows vouches for one interval of audit_events.created_at and +// the receipt names it. Prepare starts the interval and checks it up to the end +// of its fixture transaction; verify checks from that same start to its own run +// time, wherever that falls against the five-minute event window. +func TestQualificationPostgresCheckCoversExactlyTheIntervalItReports(t *testing.T) { + prepare := func(t *testing.T, db *probeDB, o qualificationOptions) (QualificationReceipt, error) { + records, err := qualificationRecords(o) + require.NoError(t, err) + r := probeReceipt(t, o, records) + return r, prepareQualification(context.Background(), o, db, db, records, &r) + } + t.Run("prepare checks from its start to the end of its fixture transaction", func(t *testing.T) { + db := &probeDB{exists: true} + o := probeOptions() + r, err := prepare(t, db, o) + require.NoError(t, err) + require.Equal(t, "passed", r.Checks["postgres_no_new_rows"]) + require.Equal(t, [][2]time.Time{{probeNow, probeNow.Add(2 * time.Second)}}, db.calls, + "the interval read is the one from the start to now, not the five-minute window ahead") + require.Equal(t, probeNow, *r.PostgresCheckedFrom) + require.Equal(t, probeNow.Add(2*time.Second), *r.PostgresCheckedTo) + }) + t.Run("a row created in that interval fails prepare and aborts its fixtures", func(t *testing.T) { + db := &probeDB{exists: true, rows: []time.Time{probeNow.Add(time.Second)}} + r, err := prepare(t, db, probeOptions()) + require.Error(t, err) + require.Equal(t, "failed", r.Checks["transaction_queue"]) + require.Empty(t, db.writes) + }) + t.Run("the interval is half open: before its start and at its end are outside it", func(t *testing.T) { + db := &probeDB{exists: true, rows: []time.Time{probeNow.Add(-time.Second), probeNow.Add(2 * time.Second)}} + _, err := prepare(t, db, probeOptions()) + require.NoError(t, err) + }) + t.Run("a clock that has not moved leaves nothing to check and fails", func(t *testing.T) { + o := probeOptions() + o.clock = func() time.Time { return probeNow } + _, err := prepare(t, &probeDB{exists: true}, o) + require.ErrorContains(t, err, "interval is empty") + }) + + records, err := qualificationRecords(probeOptions()) + require.NoError(t, err) + verify := func(t *testing.T, db *probeDB, at time.Duration) (QualificationReceipt, error) { + o := probeOptions() + o.clock = func() time.Time { return probeNow.Add(at) } + store := &probeWarehouse{records: records} + r := probeReceipt(t, o, records) + return r, verifyQualification(context.Background(), o, db, store, store, records, &r) + } + t.Run("verify after the event window checks all the way to its own run time", func(t *testing.T) { + db := &probeDB{exists: true} + r, err := verify(t, db, 20*time.Minute) + require.NoError(t, err) + require.Equal(t, probeNow, *r.PostgresCheckedFrom) + require.Equal(t, probeNow.Add(20*time.Minute), *r.PostgresCheckedTo, "past window_to, and reported as such") + require.True(t, r.PostgresCheckedTo.After(r.WindowTo)) + }) + t.Run("a row after the five-minute window but before verify ran fails verify", func(t *testing.T) { + r, err := verify(t, &probeDB{exists: true, rows: []time.Time{probeNow.Add(10 * time.Minute)}}, 20*time.Minute) + require.Error(t, err) + require.Equal(t, "failed", r.Checks["postgres_no_new_rows"]) + require.Equal(t, probeNow.Add(20*time.Minute), *r.PostgresCheckedTo) + }) + t.Run("verify inside the window claims only the part of it that has happened", func(t *testing.T) { + r, err := verify(t, &probeDB{exists: true}, time.Minute) + require.NoError(t, err) + require.Equal(t, "passed", r.Checks["postgres_no_new_rows"]) + require.Equal(t, probeNow.Add(time.Minute), *r.PostgresCheckedTo) + require.True(t, r.PostgresCheckedTo.Before(r.WindowTo), "the receipt does not claim the window's future") + }) +} + func TestQualificationChecksHashesAndOrganizationIsolationWithoutClaimingOtherAcceptance(t *testing.T) { o := probeOptions() records, err := qualificationRecords(o) diff --git a/module/services/accounts/code/pkg/infra/audit_qualification_test.go b/module/services/accounts/code/pkg/infra/audit_qualification_test.go index 390e03bc8..4614d6e93 100644 --- a/module/services/accounts/code/pkg/infra/audit_qualification_test.go +++ b/module/services/accounts/code/pkg/infra/audit_qualification_test.go @@ -35,8 +35,16 @@ func TestAuditQualificationPrepareUsesRuntimeAuthorityAndTransactionalQueue(t *t var n int require.NoError(t, pool.QueryRow(testCtx, `SELECT count(*) FROM audit_event_queue WHERE id=ANY($1::uuid[])`, ids).Scan(&n)) require.Equal(t, 6, n) + // The Postgres check names the interval it read: from the start prepare + // recorded to the end of the fixture transaction, not the five minutes ahead. + require.Equal(t, "passed", receipt.Checks["postgres_no_new_rows"]) + require.NotNil(t, receipt.PostgresCheckedFrom) + require.NotNil(t, receipt.PostgresCheckedTo) + require.Equal(t, receipt.WindowFrom, *receipt.PostgresCheckedFrom) + require.True(t, receipt.PostgresCheckedTo.After(receipt.WindowFrom)) + require.True(t, receipt.PostgresCheckedTo.Before(receipt.WindowTo), "a prepare run has not yet lived through the window") require.NoError(t, testStore.WithControlPlane(testCtx, func(ctx context.Context) error { - counts, err := testStore.CountAuditHistory(ctx, receipt.WindowFrom, receipt.WindowTo) + counts, err := testStore.CountAuditHistory(ctx, *receipt.PostgresCheckedFrom, *receipt.PostgresCheckedTo) require.NoError(t, err) for _, count := range counts { require.Zero(t, count) From 8c517c4212175e2e52ce704cb2856f7048d84937 Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Mon, 5 Oct 2026 17:02:17 -0400 Subject: [PATCH 25/69] docs(audit): say what the history copy and the qualification checks actually prove The operations guide claimed existing copies are verified before missing rows are appended; the code appends what is missing, reads the window back, and then verifies every row, so copies are verified after the append in the same pass. It also documented only exit 3, though the command can end in 0, 1, 2 and now 4 and 5. It now describes how partitions are chosen (by their bounds, never their names) and what the run does with events outside them, the drop as one locked and recounted database call whose returned names must equal the verified set, every exit status of audit-history-copy and audit-qualify with the receipt error_code beside it, the ClickHouse database and cluster in receipt configuration, and that copies are matched by id while the source key is (id, created_at): a duplicate id in one day-long window fails verification closed, and across windows each copy matches its own row. postgres_no_new_rows is documented as one interval of audit_events.created_at, [postgres_checked_from, postgres_checked_to), started by prepare and extended by verify to its own run time; it does not prove that no row can exist, since a row from a transaction still open at the check, or one written with an earlier created_at, is outside it. Co-Authored-By: Claude Sonnet 5.5 --- module/AUDIT_OPERATIONS.md | 134 +++++++++++++++++++++++++++++++++---- 1 file changed, 120 insertions(+), 14 deletions(-) diff --git a/module/AUDIT_OPERATIONS.md b/module/AUDIT_OPERATIONS.md index 684a01694..a4a992f40 100644 --- a/module/AUDIT_OPERATIONS.md +++ b/module/AUDIT_OPERATIONS.md @@ -24,14 +24,66 @@ command. The map deliberately excludes a ClickHouse DSN and all database secrets ## History copy, verification and removal Run `./app audit-history-copy -json -through YYYY-MM` to copy through the selected -month into the warehouse and archive. Copy is resumable; existing copies are -verified before missing rows are appended. `-verify-only` writes nothing. +month into the warehouse and archive. `-through` names the last month copied and +must be a **completed month in UTC**: the current month and later ones are refused +as a usage error. Without it the run covers every partition, which can verify but +cannot drop. `-verify-only` writes nothing. + +The run is resumable. For each day-long window of each partition it reads the +source rows, reads back what the warehouse holds for that window, appends only the +events missing from it, reads the window back again, and then verifies every source +row against every stored copy of its id. Existing copies are therefore verified +**after** the append, in the same pass; an earlier run's copies are not verified +before new rows are written. A verification failure leaves the partitions in place. + +### Which partitions + +The run verifies the monthly partitions whose **bounds as PostgreSQL holds them** +end at or before the cutoff (the first instant after the `-through` month, UTC). It +does not go by the month in a partition's name. A partition created from a session +in another time zone has bounds on that zone's midnights, so `audit_events_2026_08` +created from New York ends at 04:00 UTC on 1 September, after the cutoff of +`-through 2026-08`, so it is **not** in the run and is never dropped by it. Events +it holds from before the cutoff are then outside every verified partition, which +fails the run as described next. + +Whatever the partitions are, the run also counts, by table, the events older than +the cutoff that **no verified partition covers**: in the default partition, in a +partition under another name, in a partition that straddles the cutoff, or in +`audit_events` itself when it is not partitioned. Any such event fails the +verification (`uncovered` in the receipt lists the table and the count), so a +mis-provisioned table cannot pass by having its odd corner left out of the plan. A +run with no partition to verify and nothing older than the cutoff elsewhere has +verified nothing; it reports **nothing to verify** (exit 5, `nothing_to_verify`) +and never `verified`. + +### Copies are keyed by id + +The copy and its read-back match events by `id` within a window, while the source +key is `(id, created_at)`. Two source rows may in principle share an id at different +instants. When both fall in the same UTC-day window, each is compared with both +stored copies, one differs from it, and verification fails closed with "stored +envelope differs from the row"; nothing is dropped. When they fall on different days +each copy matches its own row and verification passes, leaving two warehouse rows +with one id, as the source holds. Deployments that generate ids as UUIDs do not +produce either. + +### Receipt and digest A `codefly/audit-history/v1` receipt includes `deployment_id`, `sink`, `configuration`, `through`, `observed_at`, `verified`, `partitions_sha256`, -`partitions`, `dropped`, `cutoff`, and `error_code` (empty on success). Each -partition names `name`, `from`, `to`, per-organization `orgs` counts, -`events_sha256`, and `failures`. Payloads and provider error messages are excluded. +`partitions`, `dropped`, `cutoff`, `uncovered`, `drop_refused`, and `error_code` +(empty on success). Each partition names `name`, `from`, `to`, per-organization +`orgs` counts, `events_sha256`, and `failures`. `uncovered` lists +`{table, rows}` for events outside the verified partitions. `drop_refused` is the +reason a confirmed drop did not happen. `error_code` is `verification_failed`, +`drop_refused`, `nothing_to_verify`, or `operation_failed`. Payloads and provider +error messages are excluded. + +`configuration` identifies the destination without a credential: the BigQuery +project and dataset, or for ClickHouse `AUDIT_CLICKHOUSE_DATABASE` and +`AUDIT_CLICKHOUSE_CLUSTER` (empty when there is no cluster), plus the archive, +deployment and retention. It never carries a DSN, host or password. `partitions_sha256` binds the deployment, cutoff, partition names and ranges, per-organization counts, and a canonical source stream of every event envelope, @@ -39,14 +91,53 @@ retention class and details hash. Run timing and the number copied by a particul attempt are excluded, so successful copy and verification runs agree. Equal counts with changed event content produce a different authorization digest. +### Removal + Removing partitions requires an explicit month, a verification-only pass, and `-confirm-drop -expected-partitions-sha256 `. -The copier verifies again and compares the approved digest before any DDL. It also -rechecks the source partition set and counts in the control-plane drop transaction. -An unverified copy, missing confirmation or changed digest drops nothing. The -operator must separately establish complete cutover acceptance, authenticate the +Copy, verify and drop are separate operations: the confirming run verifies again, +compares the approved digest, and refuses before any DDL when the digest differs, +the cutoff has not passed, a verified partition changed or went away, or the +cutoff would reach a partition it did not verify. + +The drop itself is one database call, `audit_events_drop_verified_partitions`, that +takes the **explicit list of verified partition names** with the number of rows +each held when verified. In one transaction it locks `audit_events` and each named +partition so no insert can route into one, confirms from the catalog that each is a +monthly partition of `audit_events` whose upper bound is at or before the cutoff, +recounts each and compares the count with the verified one, and only then drops +exactly those partitions. Any mismatch raises and rolls the transaction back with +nothing dropped. The command refuses unless the names the database returns are +exactly the verified ones, before the transaction commits, and the receipt's +`dropped` is what the database reported dropping, not what was planned. A writer +that arrives while the drop holds the locks waits, and is told there is no partition +for its row once the drop commits; a writer already in flight is waited for, and if it +commits, its row makes the recount disagree, so the drop is refused instead of +destroying the row. The +lock wait is bounded (10 seconds); a writer that does not finish is a refusal. + +The older `audit_events_drop_partitions_before` (month taken from the partition +name, no recount) remains for retention and is not called by this command. + +An unverified copy, missing confirmation, changed digest, or refusal drops nothing. +The operator must separately establish complete cutover acceptance, authenticate the receipt and pin the running image, destination and pod identity before removal. +### Exit status of `audit-history-copy` + +| Status | Meaning | Receipt `error_code` | +|---|---|---| +| 0 | done: copied and verified, and dropped when confirmed | empty | +| 1 | failed: configuration, connection, warehouse or database error | `operation_failed` and the other safe failure codes | +| 2 | usage: a flag or value is invalid (malformed, current or future `-through`; `-batch-size` outside 1 to the relay maximum; a drop without `-through`, `-verify-only` and the digest) | none (nothing is written) | +| 3 | verification failed: an event is missing or differs, or events older than the cutoff sit outside the verified partitions; nothing dropped | `verification_failed` | +| 4 | drop refused: the approval does not match, the cutoff has not passed, or what was verified no longer holds (non-JSON mode prints the reason); nothing dropped | `drop_refused` | +| 5 | nothing to verify: no partition ends at or before the cutoff and nothing older is held elsewhere; nothing claimed, nothing dropped | `nothing_to_verify` | + +`audit-qualify` exits 0 when its probes passed, 1 when a probe or the destination +failed (a receipt with an `error_code` is written to standard output), and 2 for +invalid options. + ## Bounded live qualification A prepare phase is an explicit audit mutation. Run `./app audit-qualify -json @@ -54,11 +145,13 @@ A prepare phase is an explicit audit mutation. Run `./app audit-qualify -json `. It uses `RecordTx` under the queued mode to commit six fixtures together: security and content events for both organizations and the platform. No domain events or webhook jobs are published. If an organization is -absent, or Postgres receives rows in `audit_events` in the qualification window, -the fixture transaction aborts. Use a fresh run ID for every prepare attempt. +absent, or Postgres created `audit_events` rows between the moment prepare started +and the end of the fixture transaction, the fixture transaction aborts. Use a fresh +run ID for every prepare attempt. Verify with `-verify` and the same run ID and organizations, plus `-window-from` -and `-window-to` copied from the prepare receipt. `-timeout` defaults to two minutes +and `-window-to` copied from the prepare receipt (the window must be recent: it +starts within the last 24 hours). `-timeout` defaults to two minutes and cannot exceed five. Verify emits nothing and waits for the service's normal relay; it does not create a second worker or interfere with a running relay. @@ -67,8 +160,21 @@ run ID, observation time, five-minute event window, six event IDs/scopes/classes and content hashes, and a `checks` map with `passed`, `failed` or `pending` values. Successful probes verify every stored copy's envelope, class and details hash; scoped list, aggregation, export and readable-source results; organization-scope -refusals; zero Postgres audit rows in the window; and an actually observed empty -cell queue. Warehouse scans refuse more than 20,000 events in the bounded window. +refusals; zero Postgres audit rows in the interval named below; and an actually +observed empty cell queue. Warehouse scans refuse more than 20,000 events in the +bounded window. + +`postgres_no_new_rows` proves one thing and the receipt names it: +`postgres_checked_from` and `postgres_checked_to` bound the interval of +`audit_events.created_at` it read, `[from, to)`. Prepare starts the interval +(`from` is `window_from`) and checks it up to the end of its fixture transaction. +Verify checks from the same `window_from` to **its own run time**, whether that +falls inside the five-minute event window or long after it, so a verify run after +prepare covers everything created in between and a verify run inside the window +claims only the part that has happened. It does not prove that no row can exist: a +row whose transaction was still open at the check, or that was written with an +earlier explicit `created_at`, is outside what it read. The five-minute +`window_from`/`window_to` is the warehouse event window, not the Postgres interval. `deployment_isolation`, archive object/lock enforcement, gateway authorization, controlled relay interruption/recovery and alert delivery remain **pending** until From df109793e4dda6a534b6ab1ca303cf90671b4958 Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Mon, 5 Oct 2026 16:38:58 -0400 Subject: [PATCH 26/69] fix(audit): read an organization id in its canonical form Postgres reads an organization id of any spelling of a uuid (uppercase, braces, no hyphens) as the one value it names, and the stores keep the lowercase form. The matcher validated the id with uuid.Parse but the BigQuery row restriction and the ClickHouse query parameter used the caller's text, so a non-canonical spelling matched no row and the read silently returned nothing. The matcher now holds the scope in canonical form (auditeval.CanonicalScope), which every store reads through Matcher.Scope(). The readable-source query, which never built a matcher, canonicalizes through the same function and now refuses an id that is no uuid, as the other reads and Postgres do. Co-Authored-By: Claude Sonnet 5.5 --- .../code/pkg/auditstore/auditeval/eval.go | 33 +++++++++++--- .../pkg/auditstore/auditeval/eval_test.go | 32 ++++++++++++++ .../pkg/auditstore/bigquerystore/reader.go | 5 ++- .../auditstore/bigquerystore/reader_test.go | 43 +++++++++++++++++++ .../clickhousestore/clickhousestore_test.go | 27 ++++++++++++ .../pkg/auditstore/clickhousestore/reader.go | 5 ++- .../auditstore/clickhousestore/server_test.go | 39 +++++++++++++++++ 7 files changed, 173 insertions(+), 11 deletions(-) diff --git a/module/services/accounts/code/pkg/auditstore/auditeval/eval.go b/module/services/accounts/code/pkg/auditstore/auditeval/eval.go index 04a33e1c8..27d0c0378 100644 --- a/module/services/accounts/code/pkg/auditstore/auditeval/eval.go +++ b/module/services/accounts/code/pkg/auditstore/auditeval/eval.go @@ -95,14 +95,19 @@ type Matcher struct { // NewMatcher compiles a read. It refuses a read without a scope, a query that // reaches past its scope, a category or namespace filter without the // event-type index, and an organization or actor id that is not a uuid — -// which Postgres refuses too, as invalid input for a uuid column. +// which Postgres refuses too, as invalid input for a uuid column. The scope +// and the actor it holds are in canonical form (CanonicalScope). func NewMatcher(read business.AuditRead) (*Matcher, error) { if err := read.Validate(); err != nil { return nil, err } q := read.Query + scope, err := CanonicalScope(read.Scope) + if err != nil { + return nil, err + } m := &Matcher{ - scope: read.Scope, + scope: scope, eventType: q.EventType, category: q.Category, namespace: q.Namespace, @@ -111,11 +116,6 @@ func NewMatcher(read business.AuditRead) (*Matcher, error) { resourceID: q.ResourceID, clientID: q.ClientID, } - if org := read.Scope.OrgID(); org != "" { - if _, err := uuid.Parse(org); err != nil { - return nil, fmt.Errorf("audit read: organization %q is not a uuid", org) - } - } if q.ActorID != "" { actor, err := uuid.Parse(q.ActorID) if err != nil { @@ -146,6 +146,25 @@ func NewMatcher(read business.AuditRead) (*Matcher, error) { return m, nil } +// CanonicalScope is scope with its organization in the canonical lowercase +// form the stores keep. Postgres reads an organization id of any spelling of a +// uuid as the one value it names, so a store that compares or restricts on the +// text must be handed that form, never the caller's spelling; an id that is no +// uuid is refused, as Postgres refuses it. The platform scope has none. +func CanonicalScope(scope business.AuditReadScope) (business.AuditReadScope, error) { + if err := scope.Validate(); err != nil { + return business.AuditReadScope{}, err + } + if scope.Platform() { + return scope, nil + } + org, err := uuid.Parse(scope.OrgID()) + if err != nil { + return business.AuditReadScope{}, fmt.Errorf("audit read: organization %q is not a uuid", scope.OrgID()) + } + return business.OrganizationAuditScope(org.String()), nil +} + // MicrosecondBound is a query's time bound at the precision Postgres compares // it at: a bound parameter is sent in whole microseconds, truncated. func MicrosecondBound(t *time.Time) *time.Time { diff --git a/module/services/accounts/code/pkg/auditstore/auditeval/eval_test.go b/module/services/accounts/code/pkg/auditstore/auditeval/eval_test.go index fc2eb067d..5faca4a29 100644 --- a/module/services/accounts/code/pkg/auditstore/auditeval/eval_test.go +++ b/module/services/accounts/code/pkg/auditstore/auditeval/eval_test.go @@ -2,6 +2,7 @@ package auditeval import ( "fmt" + "strings" "testing" "time" @@ -172,3 +173,34 @@ func TestMatcherRefusals(t *testing.T) { require.NoError(t, err) require.Equal(t, "bbbbbbbb-0000-4000-8000-000000000002", m.ActorID(), "an actor id compares as the uuid it names") } + +// Postgres reads an organization id of any spelling of a uuid as the one value +// it names; the stores keep it in the canonical lowercase form, so every read +// carries that form, whatever the caller wrote. +func TestMatcherReadsAnOrganizationInItsCanonicalForm(t *testing.T) { + canonical := "aaaaaaaa-0000-4000-8000-0000000000ab" + for _, spelling := range []string{ + strings.ToUpper(canonical), + "{" + canonical + "}", + strings.ReplaceAll(canonical, "-", ""), + } { + scope := business.OrganizationAuditScope(spelling) + m, err := NewMatcher(business.AuditRead{Scope: scope, Query: business.AuditQuery{OrgID: spelling}}) + require.NoError(t, err, spelling) + require.Equal(t, canonical, m.Scope().OrgID(), spelling) + require.True(t, m.InScope(business.AuditEntry{OrgID: canonical}), spelling) + require.False(t, m.InScope(business.AuditEntry{OrgID: "aaaaaaaa-0000-4000-8000-0000000000ac"}), spelling) + + canonicalScope, err := CanonicalScope(scope) + require.NoError(t, err, spelling) + require.Equal(t, canonical, canonicalScope.OrgID(), spelling) + } + + platform, err := CanonicalScope(business.PlatformAuditScope()) + require.NoError(t, err) + require.True(t, platform.Platform()) + _, err = CanonicalScope(business.OrganizationAuditScope("org-1")) + require.ErrorContains(t, err, "not a uuid") + _, err = CanonicalScope(business.AuditReadScope{}) + require.ErrorIs(t, err, business.ErrAuditReadUnscoped) +} diff --git a/module/services/accounts/code/pkg/auditstore/bigquerystore/reader.go b/module/services/accounts/code/pkg/auditstore/bigquerystore/reader.go index 110bc8361..51fbff967 100644 --- a/module/services/accounts/code/pkg/auditstore/bigquerystore/reader.go +++ b/module/services/accounts/code/pkg/auditstore/bigquerystore/reader.go @@ -463,7 +463,8 @@ func (r *Reader) ExportAuditEvents(ctx context.Context, read business.AuditRead) // windows first, until every source has its newest request or the history // runs out. func (r *Reader) LatestSourceSyncEvents(ctx context.Context, scope business.AuditReadScope, sources []string) (map[string]business.AuditSourceSyncEvent, error) { - if err := scope.Validate(); err != nil { + scope, err := auditeval.CanonicalScope(scope) + if err != nil { return nil, err } if scope.Platform() { @@ -483,7 +484,7 @@ func (r *Reader) LatestSourceSyncEvents(ctx context.Context, scope business.Audi } newestIDs := map[string]string{} eventType := string(business.EventDatasourceSourceSynced) - err := r.newestFirst(nil, nil, func(w window) (bool, error) { + err = r.newestFirst(nil, nil, func(w window) (bool, error) { where := r.scope(scope) w.restrict(where) where.eq("event_type", eventType) diff --git a/module/services/accounts/code/pkg/auditstore/bigquerystore/reader_test.go b/module/services/accounts/code/pkg/auditstore/bigquerystore/reader_test.go index bc53ff3d9..7170330ce 100644 --- a/module/services/accounts/code/pkg/auditstore/bigquerystore/reader_test.go +++ b/module/services/accounts/code/pkg/auditstore/bigquerystore/reader_test.go @@ -272,3 +272,46 @@ func TestRestrictionLiteralsCannotEscapeTheirQuotes(t *testing.T) { require.Contains(t, restriction, `resource_id = "x\" OR TRUE OR \"\U0000000A\U000000E9\\"`) require.False(t, strings.Contains(restriction, "\n")) } + +// Postgres reads an organization id of any spelling of a uuid as the one value +// it names, and the warehouse keeps the canonical form: a read of the same +// organization written otherwise must restrict on the canonical one. +func TestAnOrganizationIdIsReadInItsCanonicalForm(t *testing.T) { + w := newWarehouse(t) + w.append(t, deployment, + event(t, 1, orgA, business.EventAuthLogin, business.RetentionSecurity, now.Add(-time.Hour), nil), + event(t, 2, orgB, business.EventAuthLogin, business.RetentionSecurity, now.Add(-time.Hour), nil), + event(t, 3, orgA, business.EventDatasourceSourceSynced, business.RetentionContent, now.Add(-time.Hour), map[string]any{"n": 3}), + ) + ctx := context.Background() + for _, spelling := range []string{strings.ToUpper(orgA), "{" + orgA + "}", strings.ReplaceAll(orgA, "-", "")} { + read := business.AuditRead{Scope: business.OrganizationAuditScope(spelling), Query: business.AuditQuery{OrgID: spelling}} + entries, _, err := w.reader.ListAuditEvents(ctx, read) + require.NoError(t, err, spelling) + require.Equal(t, []string{"003", "001"}, ids(entries), spelling) + + exported, err := w.reader.ExportAuditEvents(ctx, read) + require.NoError(t, err, spelling) + require.Equal(t, []string{"003", "001"}, ids(exported), spelling) + + buckets, err := w.reader.AggregateAuditEvents(ctx, read, business.AuditAggregationSpec{}) + require.NoError(t, err, spelling) + var counted int64 + for _, bucket := range buckets { + counted += bucket.Count + } + require.Equal(t, int64(2), counted, spelling) + + synced, err := w.reader.LatestSourceSyncEvents(ctx, business.OrganizationAuditScope(spelling), []string{"source-0"}) + require.NoError(t, err, spelling) + require.Len(t, synced, 1, spelling) + } + for _, session := range w.fake.Sessions() { + restriction := session.GetReadSession().GetReadOptions().GetRowRestriction() + require.Contains(t, restriction, `org_id = "`+orgA+`"`) + require.NotContains(t, strings.ToLower(restriction), strings.ToUpper(orgA[:8])) + } + + _, err := w.reader.LatestSourceSyncEvents(ctx, business.OrganizationAuditScope("org-1"), []string{"source-0"}) + require.ErrorContains(t, err, "not a uuid", "the readable-source query refuses what the other reads refuse, as Postgres does") +} diff --git a/module/services/accounts/code/pkg/auditstore/clickhousestore/clickhousestore_test.go b/module/services/accounts/code/pkg/auditstore/clickhousestore/clickhousestore_test.go index 29c77b442..c0eceb850 100644 --- a/module/services/accounts/code/pkg/auditstore/clickhousestore/clickhousestore_test.go +++ b/module/services/accounts/code/pkg/auditstore/clickhousestore/clickhousestore_test.go @@ -346,3 +346,30 @@ func decode(t *testing.T, text string) map[string]any { require.NoError(t, json.Unmarshal([]byte(text), &out)) return out } + +// Postgres reads an organization id of any spelling of a uuid as the one value +// it names, and the store keeps the canonical form: the statement of a read +// written otherwise must bind the canonical one. +func TestAnOrganizationIdIsBoundInItsCanonicalForm(t *testing.T) { + store, err := New(panicConn{}, validConfig()) + require.NoError(t, err) + canonical := "22222222-aaaa-4000-8000-0000000000ab" + for _, spelling := range []string{strings.ToUpper(canonical), "{" + canonical + "}", strings.ReplaceAll(canonical, "-", "")} { + read := business.AuditRead{Scope: business.OrganizationAuditScope(spelling), Query: business.AuditQuery{OrgID: spelling}} + m, err := auditeval.NewMatcher(read) + require.NoError(t, err, spelling) + p := &sqlParams{} + w, ok := store.eventsWhere(p, newPlan(m, read.Query)) + require.True(t, ok) + require.Contains(t, w.String(), "org_id = ") + var bound []any + for _, arg := range p.args { + bound = append(bound, arg.(driver.NamedValue).Value) + } + require.Contains(t, bound, canonical, spelling) + require.NotContains(t, bound, spelling, spelling) + } + + _, err = store.LatestSourceSyncEvents(context.Background(), business.OrganizationAuditScope("org-1"), []string{"source-1"}) + require.ErrorContains(t, err, "not a uuid", "the readable-source query refuses what the other reads refuse, as Postgres does") +} diff --git a/module/services/accounts/code/pkg/auditstore/clickhousestore/reader.go b/module/services/accounts/code/pkg/auditstore/clickhousestore/reader.go index d05619786..5bc37e321 100644 --- a/module/services/accounts/code/pkg/auditstore/clickhousestore/reader.go +++ b/module/services/accounts/code/pkg/auditstore/clickhousestore/reader.go @@ -427,7 +427,8 @@ func (s *Store) aggregateInClickHouse(ctx context.Context, pl plan, spec busines // source, its newest sync request in the scope's organization — by // occurrence, and by event id between two at one instant — in one GROUP BY. func (s *Store) LatestSourceSyncEvents(ctx context.Context, scope business.AuditReadScope, sources []string) (map[string]business.AuditSourceSyncEvent, error) { - if err := scope.Validate(); err != nil { + scope, err := auditeval.CanonicalScope(scope) + if err != nil { return nil, err } if scope.Platform() { @@ -452,7 +453,7 @@ func (s *Store) LatestSourceSyncEvents(ctx context.Context, scope business.Audit w.and("has(" + p.texts(ids) + ", resource_id)") statement := "SELECT resource_id, max(occurred_at), argMax(actor_id, (occurred_at, event_id)) FROM " + EventsTable + " WHERE " + w.String() + " GROUP BY resource_id" - err := s.scan(ctx, statement, p.args, func(rows driver.Rows) (bool, error) { + err = s.scan(ctx, statement, p.args, func(rows driver.Rows) (bool, error) { var source, actor string var at time.Time if err := rows.Scan(&source, &at, &actor); err != nil { diff --git a/module/services/accounts/code/pkg/auditstore/clickhousestore/server_test.go b/module/services/accounts/code/pkg/auditstore/clickhousestore/server_test.go index 83bada483..50703e8ab 100644 --- a/module/services/accounts/code/pkg/auditstore/clickhousestore/server_test.go +++ b/module/services/accounts/code/pkg/auditstore/clickhousestore/server_test.go @@ -385,6 +385,45 @@ func TestServerLatestSourceSyncEvents(t *testing.T) { }, got, "another organization's and another deployment's requests are outside the read") } +// An organization id is any spelling of a uuid to Postgres; the store keeps +// the canonical lowercase form, so a read written otherwise still finds it. +func TestServerReadsAnOrganizationOfAnySpelling(t *testing.T) { + conn, database := serverDatabase(t) + ctx := context.Background() + store := serverStore(t, conn, database) + require.NoError(t, store.Ensure(ctx)) + + org, other, actor := uuid.NewString(), uuid.NewString(), uuid.NewString() + base := time.Now().UTC().Truncate(time.Microsecond).Add(-time.Minute) + synced := newRecord(t, org, actor, business.EventDatasourceSourceSynced, "source-1", base, business.RetentionContent, map[string]any{"n": 1}) + appendBatch(t, store, testDeployment, + synced, + newRecord(t, org, actor, business.EventAuthLogin, "", base.Add(-time.Hour), business.RetentionSecurity, map[string]any{"method": "password"}), + newRecord(t, other, actor, business.EventAuthLogin, "", base.Add(-time.Hour), business.RetentionSecurity, nil), + ) + for _, spelling := range []string{strings.ToUpper(org), "{" + org + "}", strings.ReplaceAll(org, "-", "")} { + read := business.AuditRead{Scope: business.OrganizationAuditScope(spelling), Query: business.AuditQuery{OrgID: spelling}} + listed, _, err := store.ListAuditEvents(ctx, read) + require.NoError(t, err, spelling) + require.Len(t, listed, 2, spelling) + exported, err := store.ExportAuditEvents(ctx, read) + require.NoError(t, err, spelling) + require.Len(t, exported, 2, spelling) + buckets, err := store.AggregateAuditEvents(ctx, read, business.AuditAggregationSpec{}) + require.NoError(t, err, spelling) + var counted int64 + for _, bucket := range buckets { + counted += bucket.Count + } + require.Equal(t, int64(2), counted, spelling) + latest, err := store.LatestSourceSyncEvents(ctx, business.OrganizationAuditScope(spelling), []string{"source-1"}) + require.NoError(t, err, spelling) + require.Equal(t, map[string]business.AuditSourceSyncEvent{ + "source-1": {RequestedAt: time.UnixMicro(base.UnixMicro()), ActorID: actor}, + }, latest, spelling) + } +} + // Decimal texts whose nearest double a fast parser misses: ClickHouse reads // each — as a JSON number and as a JSON string — to the double Postgres's // float8in reads, so the metrics over them run in ClickHouse and agree. From de07a78717465c1f95f09e99f0a1f6e2a40e21a8 Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Mon, 5 Oct 2026 16:39:55 -0400 Subject: [PATCH 27/69] fix(audit): clamp a ClickHouse time bound to the column's range A time bound is bound as a DateTime64(6, 'UTC') query parameter. The Go zero time (the year 1, which a zero protobuf timestamp becomes) was refused by the driver as an invalid NamedDateValue, and the server has no value before 1900, so a list, export or aggregate carrying such a bound failed where Postgres treats it as one more bound. A bound beyond the column's range (1900 to 2299) is now bound as the edge it clamps to. Every stored value lies inside the range, so each comparison keeps its answer: a bound below the range admits everything from below, one above admits everything from above, and a "to" before the range or a "from" after it still matches nothing. The service evaluates the original bound again on every row it reads, so the clamp can only widen what the server returns, never what the read returns. The keyset page position and the history read-back share the same bound path. Co-Authored-By: Claude Sonnet 5.5 --- .../clickhousestore/clickhousestore_test.go | 25 +++++++ .../pkg/auditstore/clickhousestore/query.go | 30 ++++++++- .../auditstore/clickhousestore/server_test.go | 67 +++++++++++++++++++ 3 files changed, 121 insertions(+), 1 deletion(-) diff --git a/module/services/accounts/code/pkg/auditstore/clickhousestore/clickhousestore_test.go b/module/services/accounts/code/pkg/auditstore/clickhousestore/clickhousestore_test.go index c0eceb850..6dcedd5b2 100644 --- a/module/services/accounts/code/pkg/auditstore/clickhousestore/clickhousestore_test.go +++ b/module/services/accounts/code/pkg/auditstore/clickhousestore/clickhousestore_test.go @@ -373,3 +373,28 @@ func TestAnOrganizationIdIsBoundInItsCanonicalForm(t *testing.T) { _, err = store.LatestSourceSyncEvents(context.Background(), business.OrganizationAuditScope("org-1"), []string{"source-1"}) require.ErrorContains(t, err, "not a uuid", "the readable-source query refuses what the other reads refuse, as Postgres does") } + +// A time bound is a query parameter of the column's own type, whose range is +// 1900 to 2299; the driver refuses the Go zero time outright and the server +// has no value for a year before the range. A bound outside the range is +// carried as the edge it clamps to: every stored value lies inside the range, +// so every comparison against it keeps its answer. +func TestATimeBoundOutsideTheColumnRangeIsClamped(t *testing.T) { + inside := time.Date(2026, 10, 3, 12, 0, 0, 123456000, time.UTC) + for name, tc := range map[string]struct{ at, want time.Time }{ + "in range": {inside, inside}, + "zero time": {time.Time{}, time.Date(1900, 1, 1, 0, 0, 0, 0, time.UTC)}, + "year 1000": {time.Date(1000, 1, 1, 0, 0, 0, 0, time.UTC), time.Date(1900, 1, 1, 0, 0, 0, 0, time.UTC)}, + "unix epoch": {time.Unix(0, 0), time.Unix(0, 0)}, + "year 9999": {time.Date(9999, 12, 31, 23, 59, 59, 0, time.UTC), time.Date(2299, 12, 31, 23, 59, 59, 999999000, time.UTC)}, + "far future": {time.Date(2300, 1, 1, 0, 0, 0, 0, time.UTC), time.Date(2299, 12, 31, 23, 59, 59, 999999000, time.UTC)}, + "other zone": {inside.In(time.FixedZone("x", 3600)), inside}, + "column edge": {time.Date(1900, 1, 1, 0, 0, 0, 0, time.UTC), time.Date(1900, 1, 1, 0, 0, 0, 0, time.UTC)}, + } { + p := &sqlParams{} + require.Equal(t, "{p0:DateTime64(6, 'UTC')}", p.instant(tc.at), name) + bound := p.args[0].(driver.NamedDateValue) + require.False(t, bound.Value.IsZero(), "%s: the driver refuses a zero time", name) + require.True(t, bound.Value.Equal(tc.want), "%s: got %s, want %s", name, bound.Value, tc.want) + } +} diff --git a/module/services/accounts/code/pkg/auditstore/clickhousestore/query.go b/module/services/accounts/code/pkg/auditstore/clickhousestore/query.go index 2b76a47b3..fde8056be 100644 --- a/module/services/accounts/code/pkg/auditstore/clickhousestore/query.go +++ b/module/services/accounts/code/pkg/auditstore/clickhousestore/query.go @@ -52,9 +52,37 @@ func (p *sqlParams) texts(values []string) string { return "{" + name + ":Array(String)}" } +// The range of the DateTime64(6, 'UTC') columns: the earliest and latest +// instant ClickHouse can store. A time bound outside it has no value of the +// column's type — the driver refuses the Go zero time (the year 1, which a +// zero protobuf timestamp becomes) and the server has none before 1900 — yet +// to Postgres it is a bound like any other. +var ( + columnMin = time.Date(1900, 1, 1, 0, 0, 0, 0, time.UTC) + columnMax = time.Date(2299, 12, 31, 23, 59, 59, 999999000, time.UTC) +) + +// clampToColumn is at, or the edge of the column's range it lies beyond. +// Every stored value lies inside the range, so a comparison against the edge +// answers as one against the bound itself did: a bound below the range admits +// every row from below, and one above it admits every row from above. The +// service evaluates the original bound again on every row it reads. +func clampToColumn(at time.Time) time.Time { + switch { + case at.Before(columnMin): + return columnMin + case at.After(columnMax): + return columnMax + default: + return at + } +} + // instant binds a time at the microsecond precision occurred_at keeps; a finer -// part is truncated, as Postgres truncates a bound parameter. +// part is truncated, as Postgres truncates a bound parameter. A time outside +// the column's range is bound as the edge it clamps to (clampToColumn). func (p *sqlParams) instant(at time.Time) string { + at = clampToColumn(at) name := p.name() p.args = append(p.args, clickhouse.DateNamed(name, at, clickhouse.MicroSeconds)) return "{" + name + ":DateTime64(6, 'UTC')}" diff --git a/module/services/accounts/code/pkg/auditstore/clickhousestore/server_test.go b/module/services/accounts/code/pkg/auditstore/clickhousestore/server_test.go index 50703e8ab..03d1530c9 100644 --- a/module/services/accounts/code/pkg/auditstore/clickhousestore/server_test.go +++ b/module/services/accounts/code/pkg/auditstore/clickhousestore/server_test.go @@ -424,6 +424,73 @@ func TestServerReadsAnOrganizationOfAnySpelling(t *testing.T) { } } +// A zero or far-future bound — a Go zero time through a protobuf timestamp is +// the year 1 — is no bound at all to Postgres; here it must read the same. +func TestServerReadsATimeBoundOutsideTheColumnRange(t *testing.T) { + conn, database := serverDatabase(t) + ctx := context.Background() + store := serverStore(t, conn, database) + require.NoError(t, store.Ensure(ctx)) + + org, actor := uuid.NewString(), uuid.NewString() + base := time.Now().UTC().Truncate(time.Microsecond).Add(-time.Minute) + f := &fixture{org: org} + f.records = []business.AuditRecord{ + newRecord(t, org, actor, business.EventAuthLogin, "", base, business.RetentionSecurity, map[string]any{"method": "password"}), + newRecord(t, org, actor, business.EventDocumentRead, "doc-1", base.Add(-time.Hour), business.RetentionContent, map[string]any{"boundary": "b-1", "count": 3}), + newRecord(t, org, actor, business.EventDocumentRead, "doc-2", base.Add(-48*time.Hour), business.RetentionContent, map[string]any{"boundary": "b-1"}), + } + appendBatch(t, store, testDeployment, f.records...) + + zero, farFuture := time.Time{}, time.Date(9999, 12, 31, 23, 59, 59, 0, time.UTC) + between := base.Add(-2 * time.Hour) + reads := map[string]struct { + from, to *time.Time + want int + }{ + "zero from": {from: &zero, want: 3}, + "far-future to": {to: &farFuture, want: 3}, + "both": {from: &zero, to: &farFuture, want: 3}, + "zero from, real to": {from: &zero, to: &between, want: 1}, + "real from, far to": {from: &between, to: &farFuture, want: 2}, + "to before every event": {to: &zero, want: 0}, + "from after every event": {from: &farFuture, want: 0}, + } + specs := map[string]business.AuditAggregationSpec{ + "default": {}, + "payload": {GroupBy: []string{"payload:boundary"}}, + } + for name, tc := range reads { + q := business.AuditQuery{OrgID: org, From: tc.from, To: tc.to} + read := business.AuditRead{Scope: business.OrganizationAuditScope(org), Query: q} + listed, _, err := store.ListAuditEvents(ctx, read) + require.NoError(t, err, name) + require.Len(t, listed, tc.want, name) + want, _ := f.referenceList(t, read) + require.Equal(t, want, listed, name) + exported, err := store.ExportAuditEvents(ctx, read) + require.NoError(t, err, name) + require.Equal(t, want, exported, name) + for specName, spec := range specs { + got, err := store.AggregateAuditEvents(ctx, read, spec) + require.NoError(t, err, "%s / %s", name, specName) + require.Equal(t, f.referenceAggregate(t, read, spec), got, "%s / %s", name, specName) + } + payload := q + payload.PayloadContains = map[string]any{"boundary": "b-1"} + read.Query = payload + listed, _, err = store.ListAuditEvents(ctx, read) + require.NoError(t, err, name) + want, _ = f.referenceList(t, read) + require.Equal(t, want, listed, name+" with a payload filter") + } + + // The history read-back takes the same bounds. + copies := 0 + require.NoError(t, store.ReadStoredAuditEvents(ctx, zero, farFuture, func(business.StoredAuditEvent) error { copies++; return nil })) + require.Equal(t, 3, copies) +} + // Decimal texts whose nearest double a fast parser misses: ClickHouse reads // each — as a JSON number and as a JSON string — to the double Postgres's // float8in reads, so the metrics over them run in ClickHouse and agree. From 0d370783ff7b0293286ac18f79230423f320ba40 Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Mon, 5 Oct 2026 16:42:56 -0400 Subject: [PATCH 28/69] fix(audit): make a ClickHouse cluster insert wait for a quorum On a cluster an insert is acknowledged by the one replica that took it unless told to wait, and the relay deletes its queue rows on that acknowledgement. A replica lost before the others copied the block would take events the queue no longer holds. Reads had the mirror problem: any replica could answer, including one that had not caught up. When a cluster is configured every insert now carries insert_quorum=auto (a majority of the replicas), insert_quorum_parallel=0 and an insert_quorum_timeout below the relay's attempt timeout, and every read carries select_sequential_consistency=1, which only works against sequential quorum inserts. A replica that has not caught up fails the read instead of serving a stale page. One server sends neither setting. Checked against a two-replica cluster: the same quorum insert fails with TOO_FEW_LIVE_REPLICAS while one replica is down, where the default insert the store sent before was acknowledged by the surviving replica alone. The cluster test (AUDIT_CLICKHOUSE_TEST_CLUSTER) asserts from system.query_log that the inserts and reads reached the server with those settings. Co-Authored-By: Claude Sonnet 5.5 --- .../clickhousestore/clickhousestore.go | 46 ++++++++++- .../clickhousestore/clickhousestore_test.go | 27 +++++++ .../clickhousestore/cluster_test.go | 76 +++++++++++++++++++ 3 files changed, 145 insertions(+), 4 deletions(-) create mode 100644 module/services/accounts/code/pkg/auditstore/clickhousestore/cluster_test.go diff --git a/module/services/accounts/code/pkg/auditstore/clickhousestore/clickhousestore.go b/module/services/accounts/code/pkg/auditstore/clickhousestore/clickhousestore.go index 8b7923ded..c24456393 100644 --- a/module/services/accounts/code/pkg/auditstore/clickhousestore/clickhousestore.go +++ b/module/services/accounts/code/pkg/auditstore/clickhousestore/clickhousestore.go @@ -21,7 +21,9 @@ // // Inserts are batched: one insert per table per relay batch, never a row at a // time (a single-row insert creates a data part per insert and ends in -// too-many-parts errors). Reads run in ClickHouse wherever ClickHouse computes +// too-many-parts errors). On a cluster an insert returns only once a majority +// of the replicas hold it, because the relay deletes its queue rows on the +// acknowledgement. Reads run in ClickHouse wherever ClickHouse computes // exactly what the Postgres reads compute, and in the service (auditeval) // where it does not; reader.go says which. package clickhousestore @@ -63,7 +65,10 @@ type Config struct { Database string // Cluster, when set, creates both tables ON CLUSTER as ReplicatedMergeTree, // coordinated by ClickHouse Keeper under the servers' default replica path - // and name. Empty creates them as MergeTree on the one server. + // and name. Every insert then waits for a majority of the replicas before + // it is acknowledged, and every read is served only by a replica that holds + // all of them (insert_quorum, select_sequential_consistency). Empty creates + // the tables as MergeTree on the one server and sends neither setting. Cluster string // DeploymentID confines every read to this deployment's events. DeploymentID string @@ -92,6 +97,9 @@ type Store struct { eventsDays int detailDays int readSettings clickhouse.Settings + // writeSettings are sent with every insert: on a cluster, the quorum that + // makes an acknowledgement durable. + writeSettings clickhouse.Settings // inService, when set, is told each time an aggregation is evaluated in the // service rather than in ClickHouse; tests use it to prove which ran. inService func() @@ -119,7 +127,7 @@ func New(conn Conn, cfg Config) (*Store, error) { if err != nil { return nil, err } - return &Store{ + store := &Store{ conn: conn, database: cfg.Database, cluster: cfg.Cluster, @@ -138,9 +146,36 @@ func New(conn Conn, cfg Config) (*Store, error) { // An unmatched details row reads as '' — no details — not NULL. "join_use_nulls": 0, }, - }, nil + } + if cfg.Cluster != "" { + // An insert is acknowledged by the replica that took it unless told to + // wait, and the relay deletes its queue rows on the acknowledgement: a + // replica lost before the others copied the block would take events + // the queue no longer holds. So an insert waits for a majority of the + // replicas, one at a time (a parallel quorum insert may reach a + // different majority than the one before it, which no single replica + // can then be read as complete against), and gives up before the + // relay's own attempt does. A retried block is the same block, which a + // replicated table recognizes and does not store twice. + store.writeSettings = clickhouse.Settings{ + "insert_quorum": "auto", + "insert_quorum_parallel": 0, + "insert_quorum_timeout": insertQuorumTimeoutMillis, + } + // A read is served only by a replica that holds every quorum insert, so + // an event the relay was told is stored is never missing from a read, + // and one that never reached the quorum is never in it. A replica that + // has not caught up fails the read, which the caller retries. + store.readSettings["select_sequential_consistency"] = 1 + } + return store, nil } +// insertQuorumTimeoutMillis is how long an insert waits for its quorum: less +// than the relay's two-minute attempt, so the server gives up before the +// caller's deadline does and the relay retries the block whole. +const insertQuorumTimeoutMillis = 60_000 + func wholeDays(name string, window time.Duration) (int, error) { const day = 24 * time.Hour if window < day || window%day != 0 { @@ -469,6 +504,9 @@ func (s *Store) insert(ctx context.Context, table string, columns []Column, rows if len(rows) == 0 { return nil } + if len(s.writeSettings) > 0 { + ctx = clickhouse.Context(ctx, clickhouse.WithSettings(s.writeSettings)) + } batch, err := s.conn.PrepareBatch(ctx, insertStatement(table, columns)) if err != nil { return fmt.Errorf("clickhouse audit store: append to %s: %w", table, err) diff --git a/module/services/accounts/code/pkg/auditstore/clickhousestore/clickhousestore_test.go b/module/services/accounts/code/pkg/auditstore/clickhousestore/clickhousestore_test.go index 6dcedd5b2..905c74a14 100644 --- a/module/services/accounts/code/pkg/auditstore/clickhousestore/clickhousestore_test.go +++ b/module/services/accounts/code/pkg/auditstore/clickhousestore/clickhousestore_test.go @@ -398,3 +398,30 @@ func TestATimeBoundOutsideTheColumnRangeIsClamped(t *testing.T) { require.True(t, bound.Value.Equal(tc.want), "%s: got %s, want %s", name, bound.Value, tc.want) } } + +// On a cluster an insert acknowledged by one replica can be lost with it +// before the others copy it, and the relay deletes its queue rows on the +// acknowledgement; so every insert waits for a majority of replicas, and every +// read refuses a replica that has not caught up with the quorum. On one +// server neither setting means anything and neither is sent. +func TestAClusterInsertsWithAQuorumAndReadsSequentially(t *testing.T) { + single, err := New(panicConn{}, validConfig()) + require.NoError(t, err) + require.Empty(t, single.writeSettings) + require.NotContains(t, single.readSettings, "select_sequential_consistency") + + cfg := validConfig() + cfg.Cluster = "audit_cluster" + clustered, err := New(panicConn{}, cfg) + require.NoError(t, err) + require.Equal(t, "auto", clustered.writeSettings["insert_quorum"], "a majority of the replicas") + require.Equal(t, 0, clustered.writeSettings["insert_quorum_parallel"], "sequential quorum inserts, which sequential reads need") + require.Equal(t, 1, clustered.readSettings["select_sequential_consistency"]) + timeout, ok := clustered.writeSettings["insert_quorum_timeout"].(int) + require.True(t, ok) + require.Positive(t, timeout) + require.Less(t, time.Duration(timeout)*time.Millisecond, 2*time.Minute, "the quorum gives up before the relay's own attempt does") + for name, value := range single.readSettings { + require.Equal(t, value, clustered.readSettings[name], "the pinned reading settings are the same on a cluster: %s", name) + } +} diff --git a/module/services/accounts/code/pkg/auditstore/clickhousestore/cluster_test.go b/module/services/accounts/code/pkg/auditstore/clickhousestore/cluster_test.go new file mode 100644 index 000000000..0655fe57b --- /dev/null +++ b/module/services/accounts/code/pkg/auditstore/clickhousestore/cluster_test.go @@ -0,0 +1,76 @@ +package clickhousestore + +import ( + "context" + "os" + "strings" + "testing" + "time" + + "accounts/pkg/business" + + "github.com/ClickHouse/clickhouse-go/v2" + "github.com/google/uuid" + "github.com/stretchr/testify/require" +) + +// The adapter against a ClickHouse cluster, when one is reachable: the same +// server as server_test.go with a cluster of at least two replicas, declared +// in its remote_servers and coordinated by Keeper, and its name in +// +// AUDIT_CLICKHOUSE_TEST_CLUSTER= +// +// It is skipped otherwise. The database is created and dropped on the cluster. + +func TestServerClusterInsertsWaitForAQuorumAndReadsAreSequential(t *testing.T) { + cluster := strings.TrimSpace(os.Getenv("AUDIT_CLICKHOUSE_TEST_CLUSTER")) + dsn := strings.TrimSpace(os.Getenv("AUDIT_CLICKHOUSE_TEST_DSN")) + if cluster == "" || dsn == "" { + t.Skip("AUDIT_CLICKHOUSE_TEST_CLUSTER and AUDIT_CLICKHOUSE_TEST_DSN are not both set; see the comment above") + } + ctx := context.Background() + options, err := clickhouse.ParseDSN(dsn) + require.NoError(t, err) + admin, err := clickhouse.Open(options) + require.NoError(t, err) + t.Cleanup(func() { _ = admin.Close() }) + database := "audit_" + strings.ReplaceAll(uuid.NewString(), "-", "")[:12] + require.NoError(t, admin.Exec(ctx, "CREATE DATABASE "+database+" ON CLUSTER '"+cluster+"'")) + t.Cleanup(func() { _ = admin.Exec(ctx, "DROP DATABASE IF EXISTS "+database+" ON CLUSTER '"+cluster+"' SYNC") }) + scoped := *options + scoped.Auth.Database = database + conn, err := clickhouse.Open(&scoped) + require.NoError(t, err) + t.Cleanup(func() { _ = conn.Close() }) + + store := serverStore(t, conn, database, func(cfg *Config) { cfg.Cluster = cluster }) + require.NoError(t, store.Ensure(ctx)) + for _, table := range []string{EventsTable, DetailsTable} { + var engine string + require.NoError(t, store.queryRow(ctx, "SELECT engine FROM system.tables WHERE database = currentDatabase() AND name = '"+table+"'", nil, &engine)) + require.Equal(t, "ReplicatedMergeTree", engine, table) + } + + org, actor := uuid.NewString(), uuid.NewString() + at := time.Now().UTC().Truncate(time.Microsecond).Add(-time.Minute) + appendBatch(t, store, testDeployment, + newRecord(t, org, actor, business.EventAuthLogin, "", at, business.RetentionSecurity, map[string]any{"method": "password"}), + newRecord(t, org, actor, business.EventDocumentRead, "doc-1", at.Add(-time.Hour), business.RetentionContent, map[string]any{"boundary": "b-1"}), + ) + read := business.AuditRead{Scope: business.OrganizationAuditScope(org), Query: business.AuditQuery{OrgID: org}} + listed, _, err := store.ListAuditEvents(ctx, read) + require.NoError(t, err) + require.Len(t, listed, 2) + require.NotNil(t, listed[1].Payload, "the content-class event's details are joined from the replicated details table") + + // The server saw the quorum on the inserts, and the sequential reading on + // the reads, whatever the user's profile says. + require.NoError(t, admin.Exec(ctx, "SYSTEM FLUSH LOGS")) + var inserts, reads uint64 + require.NoError(t, store.queryRow(ctx, "SELECT count() FROM system.query_log WHERE type = 'QueryFinish' AND query_kind = 'Insert' "+ + "AND current_database = currentDatabase() AND Settings['insert_quorum'] = 'auto' AND Settings['insert_quorum_parallel'] = '0'", nil, &inserts)) + require.EqualValues(t, 2, inserts, "one quorum insert per table") + require.NoError(t, store.queryRow(ctx, "SELECT count() FROM system.query_log WHERE type = 'QueryFinish' AND query_kind = 'Select' "+ + "AND current_database = currentDatabase() AND Settings['select_sequential_consistency'] = '1' AND query LIKE '%"+EventsTable+"%'", nil, &reads)) + require.Positive(t, reads) +} From 7062bff21b69303e36f253960e035aabe9c974c0 Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Mon, 5 Oct 2026 16:43:56 -0400 Subject: [PATCH 29/69] fix(audit): reset a BigQuery stream's retry budget when it makes progress readStream counted transient failures across the whole stream and gave up after five, so a long stream that broke every so often, each time further on, ran out of retries however much it had read in between. A failure that follows rows read since the last one now starts the count and the backoff over. A stream that fails repeatedly at one place still stops after five attempts; both cases are tested against the fake. Co-Authored-By: Claude Sonnet 5.5 --- .../pkg/auditstore/bigquerystore/reader.go | 12 ++++- .../auditstore/bigquerystore/reader_test.go | 51 +++++++++++++++++++ 2 files changed, 61 insertions(+), 2 deletions(-) diff --git a/module/services/accounts/code/pkg/auditstore/bigquerystore/reader.go b/module/services/accounts/code/pkg/auditstore/bigquerystore/reader.go index 51fbff967..a42caf247 100644 --- a/module/services/accounts/code/pkg/auditstore/bigquerystore/reader.go +++ b/module/services/accounts/code/pkg/auditstore/bigquerystore/reader.go @@ -623,18 +623,26 @@ func (r *Reader) scan(ctx context.Context, table string, fields []string, where } // readStream reads one stream to its end, reopening it at the offset reached -// after a transient failure. +// after a transient failure. The retries are for a stream that makes no +// headway: a failure that follows rows read since the last one starts the +// count, and the backoff, over, so a long stream that breaks now and then is +// never out of retries while it keeps moving. func (r *Reader) readStream(ctx context.Context, table, stream string, schema []byte, visit func(arrowRow) error) error { var offset int64 - backoff := gax.Backoff{Initial: 100 * time.Millisecond, Max: 5 * time.Second} + newBackoff := func() gax.Backoff { return gax.Backoff{Initial: 100 * time.Millisecond, Max: 5 * time.Second} } + backoff := newBackoff() failures := 0 for { + reached := offset rows, err := r.client.ReadRows(ctx, &storagepb.ReadRowsRequest{ReadStream: stream, Offset: offset}) if err == nil { offset, err = r.consume(rows, schema, offset, visit) if err == nil { return nil } + if offset > reached { + failures, backoff = 0, newBackoff() + } } var decodeErr decodeError if errors.As(err, &decodeErr) || !retryableRead(err) || ctx.Err() != nil { diff --git a/module/services/accounts/code/pkg/auditstore/bigquerystore/reader_test.go b/module/services/accounts/code/pkg/auditstore/bigquerystore/reader_test.go index 7170330ce..af1c33913 100644 --- a/module/services/accounts/code/pkg/auditstore/bigquerystore/reader_test.go +++ b/module/services/accounts/code/pkg/auditstore/bigquerystore/reader_test.go @@ -315,3 +315,54 @@ func TestAnOrganizationIdIsReadInItsCanonicalForm(t *testing.T) { _, err := w.reader.LatestSourceSyncEvents(ctx, business.OrganizationAuditScope("org-1"), []string{"source-0"}) require.ErrorContains(t, err, "not a uuid", "the readable-source query refuses what the other reads refuse, as Postgres does") } + +// The budget of retries is for a stream that makes no headway. A long stream +// that breaks now and then, each time further on, is not out of retries: it +// has never failed twice at one place. +func TestAStreamThatKeepsMakingProgressIsNeverOutOfRetries(t *testing.T) { + w := newWarehouse(t) + w.fake.StreamsPerSession, w.fake.RowsPerBatch = 1, 1 + const events = 12 + for i := 1; i <= events; i++ { + w.append(t, deployment, event(t, i, orgA, business.EventAuthLogin, business.RetentionSecurity, now.Add(-time.Duration(i)*time.Minute), nil)) + } + var mu sync.Mutex + failures := 0 + failedAt := map[int64]bool{} + w.fake.FailReadRows = func(_ string, offset int64) error { + mu.Lock() + defer mu.Unlock() + if offset > 0 && !failedAt[offset] { + failedAt[offset] = true + failures++ + return status.Error(codes.Unavailable, "transient") + } + return nil + } + entries, _, err := w.reader.ListAuditEvents(context.Background(), orgRead(business.AuditQuery{PageSize: events})) + require.NoError(t, err) + require.Len(t, entries, events, "nothing lost") + require.Equal(t, events-1, failures, "every row boundary broke the stream once, far more than the retries one stream without progress gets") +} + +func TestAStreamThatFailsAtOnePlaceStopsAfterItsRetries(t *testing.T) { + w := newWarehouse(t) + w.fake.StreamsPerSession, w.fake.RowsPerBatch = 1, 1 + for i := 1; i <= 3; i++ { + w.append(t, deployment, event(t, i, orgA, business.EventAuthLogin, business.RetentionSecurity, now.Add(-time.Duration(i)*time.Minute), nil)) + } + var mu sync.Mutex + attempts := 0 + w.fake.FailReadRows = func(_ string, offset int64) error { + mu.Lock() + defer mu.Unlock() + if offset == 1 { + attempts++ + return status.Error(codes.Unavailable, "transient") + } + return nil + } + _, _, err := w.reader.ListAuditEvents(context.Background(), orgRead(business.AuditQuery{})) + require.ErrorContains(t, err, "after 5 attempts") + require.Equal(t, 5, attempts) +} From 0edbabb3c4f946c87f3763254f48f91acbcc45a9 Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Mon, 5 Oct 2026 16:44:43 -0400 Subject: [PATCH 30/69] fix(audit): anchor BigQuery read windows at the present, not at a far-future bound The newest-first windows were anchored at the read's upper bound. With a "to" in the far future every bounded window was empty and the last one, which takes everything older than them, read the whole table. The windows are now anchored at the present plus a five-minute allowance for a replica whose clock runs ahead, or at the bound when that is earlier. The first window still reaches up to the bound itself, so an event stamped past the allowance is not missed. A time bound beyond the range of a TIMESTAMP is also written as the edge of the range instead of a literal BigQuery cannot cast. Co-Authored-By: Claude Sonnet 5.5 --- .../bigquerystore/bigquerystore_test.go | 19 ++++++++++ .../pkg/auditstore/bigquerystore/reader.go | 15 ++++++-- .../auditstore/bigquerystore/reader_test.go | 35 +++++++++++++++++++ .../auditstore/bigquerystore/restriction.go | 19 ++++++++-- 4 files changed, 84 insertions(+), 4 deletions(-) diff --git a/module/services/accounts/code/pkg/auditstore/bigquerystore/bigquerystore_test.go b/module/services/accounts/code/pkg/auditstore/bigquerystore/bigquerystore_test.go index 32677ac9f..8324585bf 100644 --- a/module/services/accounts/code/pkg/auditstore/bigquerystore/bigquerystore_test.go +++ b/module/services/accounts/code/pkg/auditstore/bigquerystore/bigquerystore_test.go @@ -211,6 +211,25 @@ func TestNewValidatesItsConfiguration(t *testing.T) { } } +// A time bound beyond the range of a TIMESTAMP is written as the edge of the +// range, never as a literal BigQuery cannot cast. +func TestATimeBoundBeyondTheTimestampRangeIsClamped(t *testing.T) { + for name, tc := range map[string]struct { + at time.Time + want string + }{ + "in range": {occurredAt, `occurred_at >= CAST("2026-10-02 14:30:05.123456+00:00" AS TIMESTAMP)`}, + "year 0": {time.Date(0, 6, 1, 0, 0, 0, 0, time.UTC), `occurred_at >= CAST("0001-01-01 00:00:00.000000+00:00" AS TIMESTAMP)`}, + "year -300": {time.Date(-300, 6, 1, 0, 0, 0, 0, time.UTC), `occurred_at >= CAST("0001-01-01 00:00:00.000000+00:00" AS TIMESTAMP)`}, + "year 10000": {time.Date(10000, 1, 1, 0, 0, 0, 0, time.UTC), `occurred_at >= CAST("9999-12-31 23:59:59.999999+00:00" AS TIMESTAMP)`}, + "zero time": {time.Time{}, `occurred_at >= CAST("0001-01-01 00:00:00.000000+00:00" AS TIMESTAMP)`}, + } { + where := &restriction{} + where.timeBound("occurred_at", ">=", tc.at) + require.Equal(t, tc.want, where.String(), name) + } +} + func ExampleEventRow() { r, _ := business.NewAuditRecord(business.AuditEntry{ ID: "00000000-0000-0000-0000-000000000001", ActorType: "system", EventType: business.EventRoleCreated, diff --git a/module/services/accounts/code/pkg/auditstore/bigquerystore/reader.go b/module/services/accounts/code/pkg/auditstore/bigquerystore/reader.go index a42caf247..b92cd5a62 100644 --- a/module/services/accounts/code/pkg/auditstore/bigquerystore/reader.go +++ b/module/services/accounts/code/pkg/auditstore/bigquerystore/reader.go @@ -130,12 +130,23 @@ type window struct { hiExclusive bool } +// readClockSkew is how far ahead of this replica's clock an event another +// replica wrote may be stamped, and so how far past the present the newest +// window reaches. +const readClockSkew = 5 * time.Minute + // newestFirst yields the windows a newest-first read walks, from upper // (inclusive, or open when nil) down to floor (inclusive, or open when nil), // until visit says the answer is complete. +// +// The windows are anchored at the present (plus readClockSkew), or at upper +// when that is earlier. An upper bound in the far future is no place to start +// walking: anchored there, every bounded window would be empty and the last, +// which takes everything older, would read the whole table. The first window +// still reaches up to upper itself, so nothing at or before it is missed. func (r *Reader) newestFirst(upper, floor *time.Time, visit func(window) (bool, error)) error { - anchor := r.now() - if upper != nil { + anchor := r.now().Add(readClockSkew) + if upper != nil && upper.Before(anchor) { anchor = *upper } current := window{hi: upper} diff --git a/module/services/accounts/code/pkg/auditstore/bigquerystore/reader_test.go b/module/services/accounts/code/pkg/auditstore/bigquerystore/reader_test.go index af1c33913..942a28a07 100644 --- a/module/services/accounts/code/pkg/auditstore/bigquerystore/reader_test.go +++ b/module/services/accounts/code/pkg/auditstore/bigquerystore/reader_test.go @@ -366,3 +366,38 @@ func TestAStreamThatFailsAtOnePlaceStopsAfterItsRetries(t *testing.T) { require.ErrorContains(t, err, "after 5 attempts") require.Equal(t, 5, attempts) } + +// A "to" in the far future is no reason to start reading there: the windows +// are anchored at the present (with a little allowance for a clock that runs +// ahead), so the newest day is still the first read and the last window never +// spans a whole table. +func TestAFarFutureUpperBoundStillReadsTheNewestDayFirst(t *testing.T) { + w := newWarehouse(t) + ages := []time.Duration{time.Hour, 2 * time.Hour, 72 * time.Hour, 100 * 24 * time.Hour} + for i, age := range ages { + w.append(t, deployment, event(t, i+1, orgA, business.EventAuthLogin, business.RetentionSecurity, now.Add(-age), nil)) + } + // An event a few minutes ahead of this replica's clock, written by another. + w.append(t, deployment, event(t, 5, orgA, business.EventAuthLogin, business.RetentionSecurity, now.Add(2*time.Minute), nil)) + + for name, to := range map[string]time.Time{ + "year 9999": time.Date(9999, 12, 31, 23, 59, 59, 0, time.UTC), + "next century": now.Add(100 * 365 * 24 * time.Hour), + } { + before := len(w.fake.Sessions()) + entries, next, err := w.reader.ListAuditEvents(context.Background(), orgRead(business.AuditQuery{To: &to, PageSize: 2})) + require.NoError(t, err, name) + require.Equal(t, []string{"005", "001"}, ids(entries), name) + require.NotEmpty(t, next, name) + sessions := w.fake.Sessions()[before:] + require.Len(t, sessions, 1, "%s: the newest day held a page and the proof of another", name) + require.Contains(t, sessions[0].GetReadSession().GetReadOptions().GetRowRestriction(), + `occurred_at >= CAST("2026-10-02 12:05:00.000000+00:00" AS TIMESTAMP)`, "%s: anchored at the present plus the allowance, not at the bound", name) + + // And the rest of the history still comes, windows reaching back from the present. + entries, next, err = w.reader.ListAuditEvents(context.Background(), orgRead(business.AuditQuery{To: &to, PageSize: 10, PageToken: next})) + require.NoError(t, err, name) + require.Equal(t, []string{"002", "003", "004"}, ids(entries), name) + require.Empty(t, next, name) + } +} diff --git a/module/services/accounts/code/pkg/auditstore/bigquerystore/restriction.go b/module/services/accounts/code/pkg/auditstore/bigquerystore/restriction.go index c851a4e48..2c747dabe 100644 --- a/module/services/accounts/code/pkg/auditstore/bigquerystore/restriction.go +++ b/module/services/accounts/code/pkg/auditstore/bigquerystore/restriction.go @@ -45,9 +45,24 @@ func (r *restriction) in(field string, values []string) { r.add(field + " IN (" + strings.Join(literals, ", ") + ")") } -// timeBound adds field op timestamp, for op one of >=, <, <=. +// The range of a BigQuery TIMESTAMP: 0001-01-01 to 9999-12-31, at microseconds. +var ( + timestampMin = time.Date(1, 1, 1, 0, 0, 0, 0, time.UTC) + timestampMax = time.Date(9999, 12, 31, 23, 59, 59, 999999000, time.UTC) +) + +// timeBound adds field op timestamp, for op one of >=, <, <=. A time beyond +// the range of a TIMESTAMP is the edge it clamps to, which no stored value lies +// beyond, so the comparison keeps its answer. func (r *restriction) timeBound(field, op string, at time.Time) { - r.add(fmt.Sprintf("%s %s CAST(%q AS TIMESTAMP)", field, op, at.UTC().Format(restrictionTimestampLayout))) + at = at.UTC() + switch { + case at.Before(timestampMin): + at = timestampMin + case at.After(timestampMax): + at = timestampMax + } + r.add(fmt.Sprintf("%s %s CAST(%q AS TIMESTAMP)", field, op, at.Format(restrictionTimestampLayout))) } func (r *restriction) fail(field string, err error) { From fd670ebc118f5d7a1ea9d0b8d856bfad1349af20 Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Mon, 5 Oct 2026 16:45:45 -0400 Subject: [PATCH 31/69] fix(audit): split BigQuery appends by request size, not only row count Appends were split at 500 rows, which says nothing about bytes: a few hundred content-class events with large details are many megabytes, and BigQuery refuses a request (and a row) over 10 MB. Such a batch would fail on every retry. Each table's rows are now split into requests of at most 500 rows and 8 MiB as BigQuery encodes them, leaving room for the envelope under the limit. A row that cannot fit a request alone comes back as a *RowTooLargeError (errors.Is ErrRowTooLarge) naming its event and table, so a caller that delivers batches in order can set that event aside instead of retrying the whole batch forever. Every request is planned before the first is sent, so such a row fails the batch before anything of it is written. Co-Authored-By: Claude Sonnet 5.5 --- .../auditstore/bigquerystore/bigquerystore.go | 104 +++++++++++++++--- .../bigquerystore/bigquerystore_test.go | 78 +++++++++++++ 2 files changed, 169 insertions(+), 13 deletions(-) diff --git a/module/services/accounts/code/pkg/auditstore/bigquerystore/bigquerystore.go b/module/services/accounts/code/pkg/auditstore/bigquerystore/bigquerystore.go index 6c633487b..7676acb48 100644 --- a/module/services/accounts/code/pkg/auditstore/bigquerystore/bigquerystore.go +++ b/module/services/accounts/code/pkg/auditstore/bigquerystore/bigquerystore.go @@ -14,10 +14,10 @@ package bigquerystore import ( "context" + "encoding/json" "errors" "fmt" "net/http" - "slices" "strings" "time" @@ -37,6 +37,42 @@ const ( // batch larger than it is sent as several requests. const maxRowsPerInsert = 500 +// maxRequestBytes bounds the rows one request carries, as BigQuery encodes +// them. BigQuery refuses a request over 10 MB (and, with the same limit, a +// single row over it), and a row count says nothing of that: a few hundred +// content-class events with large details are many megabytes. The bound sits +// under the limit to leave room for the request's own envelope and for the +// difference between this estimate and the encoding. +const maxRequestBytes = 8 << 20 + +// rowEnvelopeBytes is what a row adds to a request beyond its values and its +// insert id: the keys and punctuation around them. +const rowEnvelopeBytes = 64 + +// ErrRowTooLarge is the error of a row that cannot be sent in any request. +var ErrRowTooLarge = errors.New("bigquery audit store: row exceeds the request size limit") + +// RowTooLargeError names the event whose row cannot be sent: alone it is more +// than a request carries. Retrying cannot help, so a caller that delivers +// batches in order sets that event aside rather than retry the batch forever. +type RowTooLargeError struct { + // Table is the table the row was for. + Table string + // EventID is the event the row is of. + EventID string + // Size is the row's size, as BigQuery encodes it, and Limit is what one + // request carries. + Size, Limit int +} + +func (e *RowTooLargeError) Error() string { + return fmt.Sprintf("bigquery audit store: append to %s: the row of event %s is %d bytes, over the %d bytes one request carries", + e.Table, e.EventID, e.Size, e.Limit) +} + +// Unwrap makes errors.Is(err, ErrRowTooLarge) true. +func (e *RowTooLargeError) Unwrap() error { return ErrRowTooLarge } + // timestampLayout is BigQuery's canonical TIMESTAMP text at its microsecond // precision, which is also Postgres's. const timestampLayout = "2006-01-02T15:04:05.000000Z" @@ -297,29 +333,71 @@ func BatchRows(batch business.AuditBatch) (events, details []Row) { } // AppendAuditBatch implements business.AuditStoreWriter: the batch's rows -// (BatchRows) streamed to the events and details tables. +// (BatchRows) streamed to the events and details tables, each table in as few +// requests as hold at most maxRowsPerInsert rows and maxRequestBytes bytes. +// Every request is planned before the first is sent, so a row too large for a +// request (a *RowTooLargeError, naming its event) fails the batch before +// anything of it is written. func (s *Store) AppendAuditBatch(ctx context.Context, batch business.AuditBatch) error { eventRows, detailRows := BatchRows(batch) - events := make([]bigquery.ValueSaver, len(eventRows)) - for i, row := range eventRows { - events[i] = row + events, err := planRequests(EventsTable, eventRows) + if err != nil { + return err } - details := make([]bigquery.ValueSaver, len(detailRows)) - for i, row := range detailRows { - details[i] = row + details, err := planRequests(DetailsTable, detailRows) + if err != nil { + return err } - if err := putInChunks(ctx, s.events, events); err != nil { + if err := put(ctx, s.events, events); err != nil { return fmt.Errorf("bigquery audit store: append to %s: %w", EventsTable, err) } - if err := putInChunks(ctx, s.details, details); err != nil { + if err := put(ctx, s.details, details); err != nil { return fmt.Errorf("bigquery audit store: append to %s: %w", DetailsTable, err) } return nil } -func putInChunks(ctx context.Context, inserter rowInserter, rows []bigquery.ValueSaver) error { - for chunk := range slices.Chunk(rows, maxRowsPerInsert) { - if err := inserter.Put(ctx, chunk); err != nil { +// rowBytes is the size a row adds to a request: its values as BigQuery encodes +// them, its insert id, and the envelope around them. +func rowBytes(row Row) (int, error) { + encoded, err := json.Marshal(row.Values) + if err != nil { + return 0, fmt.Errorf("encode the row of event %s: %w", row.InsertID, err) + } + return len(encoded) + len(row.InsertID) + rowEnvelopeBytes, nil +} + +// planRequests splits rows, in order, into requests of at most maxRowsPerInsert +// rows and maxRequestBytes bytes each. A row that does not fit a request alone +// is a *RowTooLargeError. +func planRequests(table string, rows []Row) ([][]bigquery.ValueSaver, error) { + var requests [][]bigquery.ValueSaver + var current []bigquery.ValueSaver + currentBytes := 0 + for _, row := range rows { + size, err := rowBytes(row) + if err != nil { + return nil, fmt.Errorf("bigquery audit store: append to %s: %w", table, err) + } + if size > maxRequestBytes { + return nil, &RowTooLargeError{Table: table, EventID: row.InsertID, Size: size, Limit: maxRequestBytes} + } + if len(current) == maxRowsPerInsert || currentBytes+size > maxRequestBytes { + requests = append(requests, current) + current, currentBytes = nil, 0 + } + current = append(current, row) + currentBytes += size + } + if len(current) > 0 { + requests = append(requests, current) + } + return requests, nil +} + +func put(ctx context.Context, inserter rowInserter, requests [][]bigquery.ValueSaver) error { + for _, request := range requests { + if err := inserter.Put(ctx, request); err != nil { return describePutError(err) } } diff --git a/module/services/accounts/code/pkg/auditstore/bigquerystore/bigquerystore_test.go b/module/services/accounts/code/pkg/auditstore/bigquerystore/bigquerystore_test.go index 8324585bf..76153aa7b 100644 --- a/module/services/accounts/code/pkg/auditstore/bigquerystore/bigquerystore_test.go +++ b/module/services/accounts/code/pkg/auditstore/bigquerystore/bigquerystore_test.go @@ -2,8 +2,10 @@ package bigquerystore import ( "context" + "encoding/json" "errors" "fmt" + "strings" "testing" "time" @@ -144,6 +146,82 @@ func TestAppendAuditBatchSendsAtMostFiveHundredRowsPerRequest(t *testing.T) { } } +// largeRecord is a content-class record whose details are about size bytes. +func largeRecord(t *testing.T, size int) business.AuditRecord { + t.Helper() + r, err := business.NewAuditRecord(business.AuditEntry{ + ID: business.NewIDString(), ActorType: business.ActorTypeUser, EventType: business.EventAuthLogin, SchemaVersion: 1, + Resource: "session", CreatedAt: occurredAt, + Payload: map[string]any{"blob": strings.Repeat("x", size)}, + }, business.RetentionContent) + require.NoError(t, err) + return r +} + +// requestBytes is what a request carries: its rows as the streaming insert +// encodes them. +func requestBytes(t *testing.T, request []Row) int { + t.Helper() + total := 0 + for _, row := range request { + encoded, err := json.Marshal(row.Values) + require.NoError(t, err) + total += len(encoded) + len(row.InsertID) + } + return total +} + +func TestAppendAuditBatchSplitsRequestsBySizeAsWellAsByRowCount(t *testing.T) { + events, details := &fakeInserter{}, &fakeInserter{} + store := &Store{events: events, details: details} + batch := business.AuditBatch{DeploymentID: "deployment-1"} + // Thirty rows of about 600 KB: 18 MB, over one request however few the rows. + for range 30 { + batch.Records = append(batch.Records, largeRecord(t, 600<<10)) + } + require.NoError(t, store.AppendAuditBatch(context.Background(), batch)) + + require.Greater(t, len(details.requests), 1, "the batch is too large for one request") + var sent []string + for _, request := range details.requests { + require.LessOrEqual(t, requestBytes(t, request), maxRequestBytes) + for _, row := range request { + sent = append(sent, row.InsertID) + } + } + var want []string + for _, record := range batch.Records { + want = append(want, record.Entry.ID) + } + require.Equal(t, want, sent, "every row once, in batch order") + require.Equal(t, want, events.ids(), "the events table carries no content details and still holds every row") + require.Len(t, events.requests, 1) +} + +func TestAppendAuditBatchRefusesARowThatCannotFitARequestAndSendsNothing(t *testing.T) { + events, details := &fakeInserter{}, &fakeInserter{} + store := &Store{events: events, details: details} + small, huge := largeRecord(t, 1<<10), largeRecord(t, maxRequestBytes) + + err := store.AppendAuditBatch(context.Background(), business.AuditBatch{ + DeploymentID: "deployment-1", Records: []business.AuditRecord{small, huge, small}, + }) + require.ErrorIs(t, err, ErrRowTooLarge) + require.ErrorContains(t, err, huge.Entry.ID, "the error names the event whose row cannot be sent") + var tooLarge *RowTooLargeError + require.ErrorAs(t, err, &tooLarge) + require.Equal(t, huge.Entry.ID, tooLarge.EventID) + require.Equal(t, DetailsTable, tooLarge.Table) + require.Greater(t, tooLarge.Size, maxRequestBytes) + require.Empty(t, events.requests, "nothing is sent of a batch with a row that cannot be, so a retry of the rest never half-writes it") + require.Empty(t, details.requests) + + // Alone, the rest of the batch goes through. + require.NoError(t, store.AppendAuditBatch(context.Background(), business.AuditBatch{ + DeploymentID: "deployment-1", Records: []business.AuditRecord{small}, + })) +} + func TestAppendAuditBatchFailsWholeAndNamesTheRejectedRow(t *testing.T) { rejected := bigquery.PutMultiError{{InsertID: "event-7", RowIndex: 7, Errors: bigquery.MultiError{errors.New("no such field: extra")}}} events, details := &fakeInserter{fail: rejected}, &fakeInserter{} From fff787c2cb9cbdb88898e54485b91751d01d24ed Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Mon, 5 Oct 2026 16:53:49 -0400 Subject: [PATCH 32/69] fix(audit): bound the memory of a warehouse read by a budget, not the history A payload-filtered list, a payload aggregate and an export loaded every content-detail row of their window into a map before scanning events; the list with no "from" read "everything older" as its last window, and the export also held every event and its dedupe set. A million content events was a gigabyte or more, held for one request. A BigQuery read now walks time in windows, newest first, and counts what a window retains (the details it joins; for an aggregate or an export, the events it buffers and the set that dedupes them) against a byte budget, 64 MiB by default (ReadConfig.WindowBytes). A window that passes it is abandoned and read again as two halves of its span, newer first, so a day of a million events is read as the pieces that fit; a single instant with more than a window of events fails with ErrReadTooDense. Copies of an event share its occurrence, so deduplicating within a window is deduplicating the read. What outlives a window is the answer alone: a page and the proof of the next, an aggregate's buckets, an export's events. An export is one unary response, so its answer is the one thing that grows with the history. It now gives up with business.ErrAuditExportTooLarge when the events gathered pass 32 MiB of text (AuditExportMaxBytes), in BigQuery and ClickHouse alike, and the service returns that as ResourceExhausted with the way past it (narrow by actor or event type) rather than an internal error. Postgres is not changed. ClickHouse already streams and holds no more than the export itself. Kept events are detached from the Arrow batch they were decoded from (Event.Detach), which a string decoded out of one otherwise keeps alive in full. Tests read a month of redelivered content events through budgets of tens of KiB and hold the answer equal to an unbounded read's and the most a window counted to the budget plus one row. Co-Authored-By: Claude Sonnet 5.5 --- .../code/pkg/auditstore/auditeval/eval.go | 34 +- .../pkg/auditstore/auditeval/eval_test.go | 45 +++ .../code/pkg/auditstore/auditeval/page.go | 18 +- .../pkg/auditstore/bigquerystore/reader.go | 365 ++++++++++++++---- .../bigquerystore/reader_bound_test.go | 310 +++++++++++++++ .../auditstore/bigquerystore/reader_test.go | 59 ++- .../clickhousestore/clickhousestore.go | 25 +- .../pkg/auditstore/clickhousestore/reader.go | 9 +- .../auditstore/clickhousestore/server_test.go | 36 ++ .../code/pkg/business/audit_export.go | 20 + .../code/pkg/business/audit_export_test.go | 41 ++ 11 files changed, 859 insertions(+), 103 deletions(-) create mode 100644 module/services/accounts/code/pkg/auditstore/bigquerystore/reader_bound_test.go diff --git a/module/services/accounts/code/pkg/auditstore/auditeval/eval.go b/module/services/accounts/code/pkg/auditstore/auditeval/eval.go index 27d0c0378..c3d4afcca 100644 --- a/module/services/accounts/code/pkg/auditstore/auditeval/eval.go +++ b/module/services/accounts/code/pkg/auditstore/auditeval/eval.go @@ -15,6 +15,7 @@ import ( "encoding/json" "errors" "fmt" + "strings" "time" "accounts/pkg/business" @@ -40,6 +41,37 @@ type Event struct { decodedSet bool } +// eventOverheadBytes is what an event holds beyond its text: the struct, the +// entry's fixed-size fields and the headers of its strings. +const eventOverheadBytes = 512 + +// Size approximates the bytes the event holds in memory: a fixed overhead and +// the text of its envelope and details. A store that buffers events counts +// them by it against a budget, so its memory is a number it chose, not a +// function of how much history it reads. +func (e *Event) Size() int { + entry := &e.Entry + return eventOverheadBytes + len(entry.ID) + len(entry.OrgID) + len(entry.ActorID) + len(entry.ActorType) + + len(entry.EventType) + len(entry.Resource) + len(entry.ResourceID) + len(entry.IPAddress) + + len(entry.ImpersonatedBy) + len(entry.ClientID) + len(e.Details) + len(e.DetailsSHA256) +} + +// Detach copies the text of the event, so that an event kept does not keep +// alive what it was decoded from. A store that decodes rows out of a shared +// buffer (an Arrow record batch) hands back strings that point into it: one +// event kept for the page would hold the whole batch of thousands of rows it +// came in, and a window's worth of kept events every batch of the window. +func (e *Event) Detach() { + entry := &e.Entry + for _, text := range []*string{ + &entry.ID, &entry.OrgID, &entry.ActorID, &entry.ActorType, &entry.Resource, &entry.ResourceID, + &entry.IPAddress, &entry.ImpersonatedBy, &entry.ClientID, &e.Details, &e.DetailsSHA256, + } { + *text = strings.Clone(*text) + } + entry.EventType = business.EventType(strings.Clone(string(entry.EventType))) +} + // payload is the details decoded with json.Number, or nil when there are none. func (e *Event) payload() (map[string]any, error) { if !e.HasDetails { @@ -323,6 +355,6 @@ func (d *Dedupe) First(id string) bool { if _, seen := d.other[id]; seen { return false } - d.other[id] = struct{}{} + d.other[strings.Clone(id)] = struct{}{} return true } diff --git a/module/services/accounts/code/pkg/auditstore/auditeval/eval_test.go b/module/services/accounts/code/pkg/auditstore/auditeval/eval_test.go index 5faca4a29..54c063a6d 100644 --- a/module/services/accounts/code/pkg/auditstore/auditeval/eval_test.go +++ b/module/services/accounts/code/pkg/auditstore/auditeval/eval_test.go @@ -5,6 +5,7 @@ import ( "strings" "testing" "time" + "unsafe" "accounts/pkg/business" @@ -204,3 +205,47 @@ func TestMatcherReadsAnOrganizationInItsCanonicalForm(t *testing.T) { _, err = CanonicalScope(business.AuditReadScope{}) require.ErrorIs(t, err, business.ErrAuditReadUnscoped) } + +// An event decoded out of a shared buffer holds strings that point into it; +// kept as it is, it would keep every row of the buffer alive. +func TestDetachGivesAnEventItsOwnText(t *testing.T) { + buffer := []byte("0123456789abcdefghijklmnopqrstuvwxyz-details-hash") + view := func(from, to int) string { return unsafe.String(&buffer[from], to-from) } + event := &Event{ + Entry: business.AuditEntry{ + ID: view(0, 10), OrgID: view(1, 11), ActorID: view(2, 12), ActorType: view(3, 13), EventType: business.EventType(view(4, 14)), + Resource: view(5, 15), ResourceID: view(6, 16), IPAddress: view(7, 17), ImpersonatedBy: view(8, 18), ClientID: view(9, 19), + }, + Details: view(10, 30), DetailsSHA256: view(20, 40), + } + before := *event + event.Detach() + require.Equal(t, before.Entry, event.Entry, "the same text") + require.Equal(t, before.Details, event.Details) + for name, pair := range map[string][2]string{ + "id": {before.Entry.ID, event.Entry.ID}, "org": {before.Entry.OrgID, event.Entry.OrgID}, "actor": {before.Entry.ActorID, event.Entry.ActorID}, + "type": {string(before.Entry.EventType), string(event.Entry.EventType)}, "resource": {before.Entry.Resource, event.Entry.Resource}, + "ip": {before.Entry.IPAddress, event.Entry.IPAddress}, "client": {before.Entry.ClientID, event.Entry.ClientID}, + "details": {before.Details, event.Details}, "hash": {before.DetailsSHA256, event.DetailsSHA256}, + } { + require.NotEqual(t, uintptr(unsafe.Pointer(unsafe.StringData(pair[0]))), uintptr(unsafe.Pointer(unsafe.StringData(pair[1]))), "%s: its own copy", name) + } +} + +func TestPageReportsWhetherItKeptAnEvent(t *testing.T) { + base := time.Date(2026, 10, 3, 0, 0, 0, 0, time.UTC) + page, err := NewPage(business.AuditQuery{PageSize: 1}) + require.NoError(t, err) + require.True(t, page.Offer(&Event{Entry: entryAt(2, base.Add(2*time.Minute))})) + require.True(t, page.Offer(&Event{Entry: entryAt(1, base.Add(time.Minute))}), "the proof of a next page") + require.False(t, page.Offer(&Event{Entry: entryAt(1, base.Add(time.Minute))}), "an event it holds") + require.False(t, page.Offer(&Event{Entry: entryAt(0, base)}), "older than everything kept") + require.True(t, page.Offer(&Event{Entry: entryAt(3, base.Add(3*time.Minute))}), "newer evicts the oldest") +} + +func TestEventSizeCountsItsText(t *testing.T) { + small := &Event{Entry: entryAt(1, time.Time{})} + large := &Event{Entry: entryAt(1, time.Time{}), Details: strings.Repeat("x", 1000), DetailsSHA256: strings.Repeat("h", 64)} + require.Equal(t, small.Size()+1064, large.Size()) + require.GreaterOrEqual(t, small.Size(), len(small.Entry.ID)) +} diff --git a/module/services/accounts/code/pkg/auditstore/auditeval/page.go b/module/services/accounts/code/pkg/auditstore/auditeval/page.go index 4f6465e1b..d108503a6 100644 --- a/module/services/accounts/code/pkg/auditstore/auditeval/page.go +++ b/module/services/accounts/code/pkg/auditstore/auditeval/page.go @@ -4,6 +4,7 @@ import ( "container/heap" "fmt" "sort" + "strings" "time" "accounts/pkg/business" @@ -75,24 +76,29 @@ func (p *Page) Before() *time.Time { return &at } -// Offer considers an event that matched the read. -func (p *Page) Offer(event *Event) { +// Offer considers an event that matched the read, and reports whether the page +// kept it. A store that decodes events out of a shared buffer detaches (Event. +// Detach) the ones that were kept, and only those. +func (p *Page) Offer(event *Event) bool { entry := event.Entry if p.after != nil && !newer(business.AuditEntry{CreatedAt: p.after.at, ID: p.after.id}, entry) { - return + return false } if p.keptIDs[entry.ID] { - return + return false } if len(p.kept) == p.size+1 { if !newer(entry, p.kept[0].Entry) { - return + return false } evicted := heap.Pop(&p.kept).(*Event) delete(p.keptIDs, evicted.Entry.ID) } heap.Push(&p.kept, event) - p.keptIDs[entry.ID] = true + // The key is its own copy: the event's id may point into a buffer the event + // is about to be detached from. + p.keptIDs[strings.Clone(entry.ID)] = true + return true } // Full reports whether the page holds a whole page and the proof of another: diff --git a/module/services/accounts/code/pkg/auditstore/bigquerystore/reader.go b/module/services/accounts/code/pkg/auditstore/bigquerystore/reader.go index b92cd5a62..2ab85c1f6 100644 --- a/module/services/accounts/code/pkg/auditstore/bigquerystore/reader.go +++ b/module/services/accounts/code/pkg/auditstore/bigquerystore/reader.go @@ -8,6 +8,7 @@ import ( "io" "strings" "sync" + "sync/atomic" "time" "accounts/pkg/auditstore/auditeval" @@ -37,6 +38,12 @@ import ( // (so partitions outside it are pruned) and the event types when the read // names them — and the service sorts, pages, deduplicates by event id, joins // content details and aggregates the rows it streams back (auditeval). +// +// None of that grows with the history. A read walks time in windows, newest +// first, and a window that needs the whole of itself in memory (the details a +// payload filter joins, the events an aggregation deduplicates, an export's +// events) is counted against a byte budget and read again as halves when it +// passes it; see Reader. // ReadSessionClient is the Storage Read API surface the reader uses. // *storage.BigQueryReadClient (cloud.google.com/go/bigquery/storage/apiv1) @@ -58,6 +65,14 @@ type ReadConfig struct { // MaxStreams bounds the streams one read session is split into and read in // parallel. Zero is DefaultReadStreams. MaxStreams int + // WindowBytes bounds what one window of a read holds in memory at once: the + // content details it joins and the events it buffers, counted by their + // text (Reader's doc says what the bound covers). Zero is + // DefaultReadWindowBytes. + WindowBytes int64 + // ExportMaxBytes bounds the events one export gathers, counted the same + // way. Zero is business.AuditExportMaxBytes. + ExportMaxBytes int64 // Now is a seam for tests; nil is the clock. Now func() time.Time } @@ -65,6 +80,9 @@ type ReadConfig struct { // DefaultReadStreams is the parallelism of one read session. const DefaultReadStreams = 4 +// DefaultReadWindowBytes is the default of ReadConfig.WindowBytes. +const DefaultReadWindowBytes = 64 << 20 + // Newest-first windows: the activity list and the readable-source query read // the newest day first, then windows growing eightfold (8, 64 and 512 days // before it), and stop once the answer is complete; past those, one last read @@ -76,14 +94,39 @@ const ( readRowsRetryAttempts = 5 ) +// What a window retains is counted, not measured, so the bound is one number +// the deployment chose: a joined details row costs its text and the map entry +// around it; an event the window buffers costs auditeval.Event.Size plus its +// entry in the set that deduplicates it. +const ( + detailRowOverheadBytes = 160 + dedupeEntryBytes = 64 +) + // Reader reads the events and details tables. +// +// Its memory does not grow with the history it reads. A read walks the time +// axis in windows, newest first, and holds at most WindowBytes of one window +// at a time — the details it joins to the window's events and, for a read that +// must see every event of the window before it can answer (an aggregation, an +// export), the events themselves with the set that deduplicates them. A window +// that would hold more is abandoned and read again as two halves of its span, +// newer first, so a day of a million events is read as the several pieces that +// fit. What outlives a window is the answer alone: a page and the proof of the +// next, an aggregation's buckets, an export's events (up to ExportMaxBytes). type Reader struct { - client ReadSessionClient - project string - dataset string - deploymentID string - maxStreams int - now func() time.Time + client ReadSessionClient + project string + dataset string + deploymentID string + maxStreams int + windowBytes int64 + exportMaxBytes int64 + now func() time.Time + + // windowPeak is the most bytes any one attempt at a window has counted, the + // attempt that overflowed included, since the reader was built. + windowPeak atomic.Int64 } // NewReader validates cfg and builds a reader. @@ -96,23 +139,78 @@ func NewReader(cfg ReadConfig) (*Reader, error) { return nil, fmt.Errorf("bigquery audit store: %s is required to read", name) } } + if cfg.WindowBytes < 0 || cfg.ExportMaxBytes < 0 { + return nil, errors.New("bigquery audit store: read budgets cannot be negative") + } reader := &Reader{ - client: cfg.Client, - project: cfg.Project, - dataset: cfg.Dataset, - deploymentID: cfg.DeploymentID, - maxStreams: cfg.MaxStreams, - now: cfg.Now, + client: cfg.Client, + project: cfg.Project, + dataset: cfg.Dataset, + deploymentID: cfg.DeploymentID, + maxStreams: cfg.MaxStreams, + windowBytes: cfg.WindowBytes, + exportMaxBytes: cfg.ExportMaxBytes, + now: cfg.Now, } if reader.maxStreams <= 0 { reader.maxStreams = DefaultReadStreams } + if reader.windowBytes == 0 { + reader.windowBytes = DefaultReadWindowBytes + } + if reader.exportMaxBytes == 0 { + reader.exportMaxBytes = business.AuditExportMaxBytes + } if reader.now == nil { reader.now = time.Now } return reader, nil } +// errWindowTooLarge is the signal of a window attempt that counted more than +// its budget: the window is read again as halves (settle). +var errWindowTooLarge = errors.New("bigquery audit store: window exceeds the read budget") + +// ErrReadTooDense is the failure of a read whose events at one instant alone +// pass the window budget, which no narrower window can split. +var ErrReadTooDense = errors.New("bigquery audit store: more events share one instant than a read window holds") + +// budget counts what one attempt at a window retains. +type budget struct { + limit int64 + held int64 + peak *atomic.Int64 +} + +func (r *Reader) newBudget() *budget { + return &budget{limit: r.windowBytes, peak: &r.windowPeak} +} + +// take counts n more bytes, and reports errWindowTooLarge once they pass the +// limit. A nil budget counts nothing: the caller has bounded what it retains +// some other way. +func (b *budget) take(n int) error { + if b == nil { + return nil + } + if b.held > b.limit { + // Already over: the rows other streams are still handing over are not + // counted, nor kept, so an attempt overshoots by one row at most. + return errWindowTooLarge + } + b.held += int64(n) + for { + peak := b.peak.Load() + if b.held <= peak || b.peak.CompareAndSwap(peak, b.held) { + break + } + } + if b.held > b.limit { + return errWindowTooLarge + } + return nil +} + var ( eventFields = []string{ "event_id", "deployment_id", "org_id", "actor_id", "actor_type", "event_type", "schema_version", @@ -130,6 +228,31 @@ type window struct { hiExclusive bool } +// split halves a window at the middle of its span: the newer half keeps the +// window's upper bound, the older half is closed below the middle, so the two +// cover exactly the window. An open upper bound reaches no further than +// ceiling when the middle is chosen. A window with no lower bound, or no span +// left to halve, does not split. +func (w window) split(ceiling time.Time) (newer, older window, ok bool) { + if w.lo == nil { + return window{}, window{}, false + } + top := ceiling + if w.hi != nil && w.hi.Before(top) { + top = *w.hi + } + if !top.After(*w.lo) { + return window{}, window{}, false + } + // In microseconds since the epoch, which span the whole TIMESTAMP range; a + // time.Duration saturates at about 292 years. + middle := time.UnixMicro(w.lo.UnixMicro() + (top.UnixMicro()-w.lo.UnixMicro())/2).UTC() + if !middle.After(*w.lo) { + return window{}, window{}, false + } + return window{lo: &middle, hi: w.hi, hiExclusive: w.hiExclusive}, window{lo: w.lo, hi: &middle, hiExclusive: true}, true +} + // readClockSkew is how far ahead of this replica's clock an event another // replica wrote may be stamped, and so how far past the present the newest // window reaches. @@ -137,18 +260,28 @@ const readClockSkew = 5 * time.Minute // newestFirst yields the windows a newest-first read walks, from upper // (inclusive, or open when nil) down to floor (inclusive, or open when nil), -// until visit says the answer is complete. +// until visit says the answer is complete. A visit that reports a window too +// large is given its halves instead (settle). // // The windows are anchored at the present (plus readClockSkew), or at upper // when that is earlier. An upper bound in the far future is no place to start // walking: anchored there, every bounded window would be empty and the last, // which takes everything older, would read the whole table. The first window // still reaches up to upper itself, so nothing at or before it is missed. -func (r *Reader) newestFirst(upper, floor *time.Time, visit func(window) (bool, error)) error { +// +// A bounded walk is one whose visits can overflow and be halved, so none of its +// windows is open below: the last reaches down to floor, or to the earliest +// instant a TIMESTAMP holds. +func (r *Reader) newestFirst(upper, floor *time.Time, bounded bool, visit func(window) (bool, error)) error { anchor := r.now().Add(readClockSkew) if upper != nil && upper.Before(anchor) { anchor = *upper } + oldest := floor + if oldest == nil && bounded { + earliest := timestampMin + oldest = &earliest + } current := window{hi: upper} span := firstReadWindow lo := anchor.Add(-span) @@ -158,12 +291,12 @@ func (r *Reader) newestFirst(upper, floor *time.Time, visit func(window) (bool, case floor != nil && !lo.After(*floor): current.lo, last = floor, true case step >= boundedReadWindows+1: - current.lo, last = floor, true + current.lo, last = oldest, true default: bound := lo current.lo = &bound } - done, err := visit(current) + done, err := r.settle(current, visit) if err != nil || done || last { return err } @@ -174,6 +307,24 @@ func (r *Reader) newestFirst(upper, floor *time.Time, visit func(window) (bool, } } +// settle visits a window; when the visit counts more than the window budget +// holds, it visits the window's newer half and then its older half instead, +// each of which may be halved again. +func (r *Reader) settle(w window, visit func(window) (bool, error)) (bool, error) { + done, err := visit(w) + if !errors.Is(err, errWindowTooLarge) { + return done, err + } + newer, older, ok := w.split(r.now().Add(readClockSkew)) + if !ok { + return false, ErrReadTooDense + } + if done, err := r.settle(newer, visit); done || err != nil { + return done, err + } + return r.settle(older, visit) +} + // scope adds the read's scope to a restriction: this deployment always, and // the organization unless the read is the explicit platform read. func (r *Reader) scope(scope business.AuditReadScope) *restriction { @@ -230,30 +381,14 @@ func (r *Reader) candidates(m *auditeval.Matcher, q business.AuditQuery, w windo } // readEvents streams the events in a window that the read could match, each -// joined to its content details when withDetails, and hands each to visit -// (serially). A row of another deployment is never handed on, whatever the -// session returned. -func (r *Reader) readEvents(ctx context.Context, m *auditeval.Matcher, q business.AuditQuery, w window, withDetails bool, visit func(*auditeval.Event) error) error { +// joined to its content details from details (when it is not nil) and handed to +// visit (serially). A row of another deployment is never handed on, whatever +// the session returned. +func (r *Reader) readEvents(ctx context.Context, m *auditeval.Matcher, q business.AuditQuery, w window, details map[string]storedDetail, visit func(*auditeval.Event) error) error { where, ok := r.candidates(m, q, w) if !ok { return nil } - var details map[string]storedDetail - if withDetails { - detailsWhere := r.scope(m.Scope()) - w.restrict(detailsWhere) - if types, restricted := m.EventTypes(); restricted { - if len(types) == 1 { - detailsWhere.eq("event_type", types[0]) - } else { - detailsWhere.in("event_type", types) - } - } - var err error - if details, err = r.readDetails(ctx, detailsWhere); err != nil { - return err - } - } return r.scan(ctx, EventsTable, eventFields, where, func(row arrowRow) error { stored, err := decodeEventRow(row) if err != nil { @@ -263,7 +398,7 @@ func (r *Reader) readEvents(ctx context.Context, m *auditeval.Matcher, q busines return nil } event := stored.event() - if withDetails && stored.retentionClass != string(business.RetentionSecurity) { + if details != nil && stored.retentionClass != string(business.RetentionSecurity) { if detail, ok := details[event.Entry.ID]; ok && detail.sha256 == stored.detailsSHA256 { event.Details, event.HasDetails = detail.details, true } @@ -272,6 +407,24 @@ func (r *Reader) readEvents(ctx context.Context, m *auditeval.Matcher, q busines }) } +// windowDetails reads the content details of every candidate event in a +// window, counting them against b: errWindowTooLarge when they pass it. +func (r *Reader) windowDetails(ctx context.Context, m *auditeval.Matcher, w window, b *budget) (map[string]storedDetail, error) { + where := r.scope(m.Scope()) + w.restrict(where) + if types, restricted := m.EventTypes(); restricted { + if len(types) == 0 { + return map[string]storedDetail{}, nil + } + if len(types) == 1 { + where.eq("event_type", types[0]) + } else { + where.in("event_type", types) + } + } + return r.readDetails(ctx, where, b) +} + // storedDetail is one details-table row: the canonical details of a // content-class event and their hash. type storedDetail struct { @@ -279,7 +432,10 @@ type storedDetail struct { details string } -func (r *Reader) readDetails(ctx context.Context, where *restriction) (map[string]storedDetail, error) { +// readDetails reads the details rows a restriction admits, counting each +// against b (nil counts nothing, for a read the caller has bounded by the ids +// it names). +func (r *Reader) readDetails(ctx context.Context, where *restriction, b *budget) (map[string]storedDetail, error) { details := map[string]storedDetail{} err := r.scan(ctx, DetailsTable, detailFields, where, func(row arrowRow) error { eventID, err := row.str("event_id") @@ -301,12 +457,53 @@ func (r *Reader) readDetails(ctx context.Context, where *restriction) (map[strin if err != nil { return err } - details[eventID] = storedDetail{sha256: sha, details: text} + if err := b.take(detailRowOverheadBytes + len(eventID) + len(sha) + len(text)); err != nil { + return err + } + // Copies, so what is kept is the text alone and not the record batch + // it was decoded from. + details[strings.Clone(eventID)] = storedDetail{sha256: strings.Clone(sha), details: strings.Clone(text)} return nil }) return details, err } +// loadWindow reads the events of a window that the read matches, each once by +// event id and joined to its content details when withDetails, holding no more +// than the window budget: errWindowTooLarge when the window has more. Nothing +// is handed on before the whole window is in hand, so a caller whose answer +// cannot be unwound (an aggregation) sees a window either whole or not at all. +// Duplicates share their occurrence, so one window holds all of an event's +// copies. +func (r *Reader) loadWindow(ctx context.Context, m *auditeval.Matcher, q business.AuditQuery, w window, withDetails bool) ([]*auditeval.Event, error) { + b := r.newBudget() + var details map[string]storedDetail + if withDetails { + var err error + if details, err = r.windowDetails(ctx, m, w, b); err != nil { + return nil, err + } + } + seen := auditeval.NewDedupe() + var events []*auditeval.Event + err := r.readEvents(ctx, m, q, w, details, func(event *auditeval.Event) error { + ok, err := m.Match(event) + if err != nil || !ok || !seen.First(event.Entry.ID) { + return err + } + if err := b.take(event.Size() + dedupeEntryBytes); err != nil { + return err + } + event.Detach() + events = append(events, event) + return nil + }) + if err != nil { + return nil, err + } + return events, nil +} + // joinDetails reads the details of the given events that have none yet — the // content-class events of a list page that did not need them to match — in // one session bounded by their occurrences and ids. An event whose details are @@ -333,7 +530,8 @@ func (r *Reader) joinDetails(ctx context.Context, scope business.AuditReadScope, where := r.scope(scope) window{lo: &lo, hi: &hi}.restrict(where) where.in("event_id", ids) - details, err := r.readDetails(ctx, where) + // Bounded by the ids it names, which are a page's. + details, err := r.readDetails(ctx, where, nil) if err != nil { return err } @@ -363,16 +561,29 @@ func (r *Reader) ListAuditEvents(ctx context.Context, read business.AuditRead) ( if before := page.Before(); before != nil && (upper == nil || before.Before(*upper)) { upper = before } - // A payload filter needs every candidate's details to decide; otherwise - // only the settled page's content details are read, afterwards. + // A payload filter needs every candidate's details to decide, so the walk is + // one whose windows hold them (and are halved when they hold too many); + // otherwise only the settled page's content details are read, afterwards, + // and nothing but the page is held at all. withDetails := m.NeedsPayload() - err = r.newestFirst(upper, m.From(), func(w window) (bool, error) { - err := r.readEvents(ctx, m, read.Query, w, withDetails, func(event *auditeval.Event) error { + err = r.newestFirst(upper, m.From(), withDetails, func(w window) (bool, error) { + var details map[string]storedDetail + if withDetails { + var err error + if details, err = r.windowDetails(ctx, m, w, r.newBudget()); err != nil { + return false, err + } + } + // Offering an event twice changes nothing, so a window abandoned after + // part of it was offered is read again from its halves without harm. + err := r.readEvents(ctx, m, read.Query, w, details, func(event *auditeval.Event) error { ok, err := m.Match(event) if err != nil || !ok { return err } - page.Offer(event) + if page.Offer(event) { + event.Detach() + } return nil }) return page.Full(), err @@ -419,8 +630,10 @@ func needsPayload(m *auditeval.Matcher, spec business.AuditAggregationSpec) bool return false } -// AggregateAuditEvents implements business.AuditReader: one read of the -// query's window, deduplicated and aggregated in the service. +// AggregateAuditEvents implements business.AuditReader: the query's span read +// window by window, each window deduplicated and aggregated whole before the +// next is read. Copies of an event share its occurrence, so a window holds all +// of them and the deduplication needs no memory of the windows before it. func (r *Reader) AggregateAuditEvents(ctx context.Context, read business.AuditRead, spec business.AuditAggregationSpec) ([]business.AuditAggregateBucket, error) { m, err := auditeval.NewMatcher(read) if err != nil { @@ -430,14 +643,18 @@ func (r *Reader) AggregateAuditEvents(ctx context.Context, read business.AuditRe if err != nil { return nil, err } - seen := auditeval.NewDedupe() - w := window{lo: m.From(), hi: m.To()} - err = r.readEvents(ctx, m, read.Query, w, needsPayload(m, spec), func(event *auditeval.Event) error { - ok, err := m.Match(event) - if err != nil || !ok || !seen.First(event.Entry.ID) { - return err + withDetails := needsPayload(m, spec) + err = r.newestFirst(m.To(), m.From(), true, func(w window) (bool, error) { + events, err := r.loadWindow(ctx, m, read.Query, w, withDetails) + if err != nil { + return false, err } - return aggregator.Add(event) + for _, event := range events { + if err := aggregator.Add(event); err != nil { + return false, err + } + } + return false, nil }) if err != nil { return nil, err @@ -446,28 +663,37 @@ func (r *Reader) AggregateAuditEvents(ctx context.Context, read business.AuditRe } // ExportAuditEvents implements business.AuditReader: every matching event of -// the query's window, once each, newest first, with its payload. +// the query's span, once each, newest first, with its payload. The windows are +// read newest first and each is sorted whole, so their events follow one +// another in order. The export is the one read whose answer is as large as the +// history it matches; it gives up with business.ErrAuditExportTooLarge when the +// events gathered pass the reader's export bound, never holding more than that +// and one window. func (r *Reader) ExportAuditEvents(ctx context.Context, read business.AuditRead) ([]business.AuditEntry, error) { m, err := auditeval.NewMatcher(read) if err != nil { return nil, err } - seen := auditeval.NewDedupe() - var events []*auditeval.Event - w := window{lo: m.From(), hi: m.To()} - err = r.readEvents(ctx, m, read.Query, w, true, func(event *auditeval.Event) error { - ok, err := m.Match(event) - if err != nil || !ok || !seen.First(event.Entry.ID) { - return err + var out []business.AuditEntry + var gathered int64 + err = r.newestFirst(m.To(), m.From(), true, func(w window) (bool, error) { + events, err := r.loadWindow(ctx, m, read.Query, w, true) + if err != nil { + return false, err } - events = append(events, event) - return nil + auditeval.SortNewestFirst(events) + for _, event := range events { + if gathered += int64(event.Size()); gathered > r.exportMaxBytes { + return false, business.ErrAuditExportTooLarge + } + } + out = append(out, results(events)...) + return false, nil }) if err != nil { return nil, err } - auditeval.SortNewestFirst(events) - return results(events), nil + return out, nil } // LatestSourceSyncEvents implements business.AuditSourceSyncReader: the newest @@ -495,7 +721,7 @@ func (r *Reader) LatestSourceSyncEvents(ctx context.Context, scope business.Audi } newestIDs := map[string]string{} eventType := string(business.EventDatasourceSourceSynced) - err = r.newestFirst(nil, nil, func(w window) (bool, error) { + err = r.newestFirst(nil, nil, false, func(w window) (bool, error) { where := r.scope(scope) w.restrict(where) where.eq("event_type", eventType) @@ -514,8 +740,9 @@ func (r *Reader) LatestSourceSyncEvents(ctx context.Context, scope business.Audi current, found := out[entry.ResourceID] if !found || entry.CreatedAt.After(current.RequestedAt) || (entry.CreatedAt.Equal(current.RequestedAt) && entry.ID > newestIDs[entry.ResourceID]) { - out[entry.ResourceID] = business.AuditSourceSyncEvent{RequestedAt: entry.CreatedAt, ActorID: entry.ActorID} - newestIDs[entry.ResourceID] = entry.ID + // Copies of the text kept, not what it was decoded from. + out[entry.ResourceID] = business.AuditSourceSyncEvent{RequestedAt: entry.CreatedAt, ActorID: strings.Clone(entry.ActorID)} + newestIDs[entry.ResourceID] = strings.Clone(entry.ID) } return nil }) diff --git a/module/services/accounts/code/pkg/auditstore/bigquerystore/reader_bound_test.go b/module/services/accounts/code/pkg/auditstore/bigquerystore/reader_bound_test.go new file mode 100644 index 000000000..734433806 --- /dev/null +++ b/module/services/accounts/code/pkg/auditstore/bigquerystore/reader_bound_test.go @@ -0,0 +1,310 @@ +package bigquerystore + +import ( + "context" + "fmt" + "slices" + "strings" + "testing" + "time" + + "accounts/pkg/auditstore/bigqueryfake" + "accounts/pkg/business" + + "github.com/stretchr/testify/require" +) + +// What a read holds in memory is a number the deployment chose, not a function +// of the history it reads. These tests read a month of content-class events — +// megabytes of details — through readers whose window budget is a few +// kilobytes, and hold two things to account: the answer is the one a reader +// with no practical bound gives, and the most any window counted never passed +// the budget (plus the one row that overflowed it). + +const ( + boundProject = "test-project" + boundDataset = "audit" + boundDeployment = "deployment-1" + boundOrg = "aaaaaaaa-0000-4000-8000-000000000001" + boundActor = "cccccccc-0000-4000-8000-000000000003" +) + +var boundNow = time.Date(2026, 10, 3, 12, 0, 0, 0, time.UTC) + +type boundWarehouse struct { + fake *bigqueryfake.Server +} + +func newBoundWarehouse(t *testing.T) *boundWarehouse { + t.Helper() + fake := bigqueryfake.New() + fake.CreateTable(bigqueryfake.TablePath(boundProject, boundDataset, EventsTable), EventsSchema()) + fake.CreateTable(bigqueryfake.TablePath(boundProject, boundDataset, DetailsTable), DetailsSchema()) + return &boundWarehouse{fake: fake} +} + +func (w *boundWarehouse) reader(t *testing.T, windowBytes, exportMaxBytes int64) *Reader { + t.Helper() + reader, err := NewReader(ReadConfig{ + Client: w.fake, Project: boundProject, Dataset: boundDataset, DeploymentID: boundDeployment, + WindowBytes: windowBytes, ExportMaxBytes: exportMaxBytes, Now: func() time.Time { return boundNow }, + }) + require.NoError(t, err) + return reader +} + +func (w *boundWarehouse) append(t *testing.T, records ...business.AuditRecord) { + t.Helper() + events, details := BatchRows(business.AuditBatch{ID: "batch", DeploymentID: boundDeployment, Records: records}) + for _, row := range events { + require.NoError(t, w.fake.Insert(bigqueryfake.TablePath(boundProject, boundDataset, EventsTable), row.Values)) + } + for _, row := range details { + require.NoError(t, w.fake.Insert(bigqueryfake.TablePath(boundProject, boundDataset, DetailsTable), row.Values)) + } +} + +func boundEvent(t *testing.T, n int, at time.Time, payload map[string]any) business.AuditRecord { + t.Helper() + record, err := business.NewAuditRecord(business.AuditEntry{ + ID: fmt.Sprintf("00000000-0000-4000-8000-%012d", n), OrgID: boundOrg, ActorID: boundActor, ActorType: business.ActorTypeUser, + EventType: business.EventDocumentRead, SchemaVersion: 1, Resource: "document", ResourceID: fmt.Sprintf("doc-%d", n%5), + Payload: payload, CreatedAt: at, + }, business.RetentionContent) + require.NoError(t, err) + return record +} + +// month fills the warehouse with a month of content-class events, forty a day, +// each with about 900 bytes of details; every hundredth is "rare" and each has a +// "group" of seven. Every event is delivered twice, as a redelivered relay +// batch would leave it. It returns the events, newest first. +func (w *boundWarehouse) month(t *testing.T) []business.AuditRecord { + t.Helper() + pad := strings.Repeat("p", 900) + var records []business.AuditRecord + for i := 0; i < 1200; i++ { + at := boundNow.Add(-time.Hour - time.Duration(i)*36*time.Minute) + payload := map[string]any{"group": i % 7, "pad": pad} + if i%100 == 99 { + payload["rare"] = true + } + records = append(records, boundEvent(t, i+1, at, payload)) + } + w.append(t, records...) + w.append(t, records...) + return records +} + +func entryIDs(entries []business.AuditEntry) []string { + out := make([]string, len(entries)) + for i, entry := range entries { + out[i] = entry.ID + } + return out +} + +func orgScoped(q business.AuditQuery) business.AuditRead { + q.OrgID = boundOrg + return business.AuditRead{Scope: business.OrganizationAuditScope(boundOrg), Query: q} +} + +// oneRow is the most one row can add to a window past the budget: the row that +// overflows it is counted before the window is abandoned. +const oneRow = 4 << 10 + +// everyPage reads a list to its last page. +func everyPage(t *testing.T, reader *Reader, q business.AuditQuery) (ids []string, pages int) { + t.Helper() + for { + entries, next, err := reader.ListAuditEvents(context.Background(), orgScoped(q)) + require.NoError(t, err) + ids = append(ids, entryIDs(entries)...) + pages++ + if next == "" { + return ids, pages + } + q.PageToken = next + } +} + +func TestAPayloadFilteredListHoldsOneBoundedWindowOfDetailsNotTheHistory(t *testing.T) { + w := newBoundWarehouse(t) + records := w.month(t) + var want []string + for _, record := range records { + if record.Entry.Payload["rare"] == true { + want = append(want, record.Entry.ID) + } + } + require.Len(t, want, 12) + + const budget = 64 << 10 + small, large := w.reader(t, budget, 0), w.reader(t, 1<<40, 0) + q := business.AuditQuery{PageSize: 5, PayloadContains: map[string]any{"rare": true}} + + got, pages := everyPage(t, small, q) + require.Equal(t, want, got, "every rare event, once each, newest first, across the pages") + require.Equal(t, 3, pages) + reference, _ := everyPage(t, large, q) + require.Equal(t, reference, got, "the answer does not depend on the budget") + + require.LessOrEqual(t, small.windowPeak.Load(), int64(budget+oneRow), "no window counted more than its budget") + require.Greater(t, large.windowPeak.Load(), int64(4*budget), "unbounded, the same read holds a month of details at once") +} + +func TestAPayloadAggregationHoldsOneBoundedWindowAndCountsEachEventOnce(t *testing.T) { + w := newBoundWarehouse(t) + w.month(t) + spec := business.AuditAggregationSpec{GroupBy: []string{"payload:group"}} + read := orgScoped(business.AuditQuery{}) + + const budget = 96 << 10 + small, large := w.reader(t, budget, 0), w.reader(t, 1<<40, 0) + before := len(w.fake.Sessions()) + got, err := small.AggregateAuditEvents(context.Background(), read, spec) + require.NoError(t, err) + halved := len(w.fake.Sessions()) - before + before = len(w.fake.Sessions()) + reference, err := large.AggregateAuditEvents(context.Background(), read, spec) + require.NoError(t, err) + require.Equal(t, reference, got, "the answer does not depend on the budget") + require.Greater(t, halved, len(w.fake.Sessions())-before, "the bound costs reads, in windows halved to fit, never memory") + + counts := map[string]int64{} + var total int64 + for _, bucket := range got { + counts[bucket.Key] = bucket.Count + total += bucket.Count + } + require.Equal(t, int64(1200), total, "1200 events, each delivered twice, counted once each") + require.Equal(t, int64(172), counts["0"], "1200 events in seven groups: 172 or 171") + require.Equal(t, int64(171), counts["6"]) + + require.LessOrEqual(t, small.windowPeak.Load(), int64(budget+oneRow)) + require.Greater(t, large.windowPeak.Load(), int64(4*budget)) +} + +func TestAnExportHoldsOneBoundedWindowAndTheEventsItReturns(t *testing.T) { + w := newBoundWarehouse(t) + records := w.month(t) + var want []string + for _, record := range records { + want = append(want, record.Entry.ID) + } + read := orgScoped(business.AuditQuery{}) + + const budget = 96 << 10 + small, large := w.reader(t, budget, 0), w.reader(t, 1<<40, 0) + got, err := small.ExportAuditEvents(context.Background(), read) + require.NoError(t, err) + require.Equal(t, want, entryIDs(got), "every event once, newest first, across every window") + reference, err := large.ExportAuditEvents(context.Background(), read) + require.NoError(t, err) + require.Equal(t, reference, got, "the answer does not depend on the budget") + require.NotNil(t, got[0].Payload, "with its payload") + + require.LessOrEqual(t, small.windowPeak.Load(), int64(budget+oneRow)) + require.Greater(t, large.windowPeak.Load(), int64(4*budget)) +} + +func TestAnExportGivesUpPastItsBoundInsteadOfHoldingTheHistory(t *testing.T) { + w := newBoundWarehouse(t) + w.month(t) + read := orgScoped(business.AuditQuery{}) + + // The month is about 1.2 MB of events; the bound is a tenth of it. + bounded := w.reader(t, 96<<10, 128<<10) + before := len(w.fake.Sessions()) + _, err := bounded.ExportAuditEvents(context.Background(), read) + require.ErrorIs(t, err, business.ErrAuditExportTooLarge) + stopped := len(w.fake.Sessions()) - before + + before = len(w.fake.Sessions()) + _, err = w.reader(t, 96<<10, 0).ExportAuditEvents(context.Background(), read) + require.NoError(t, err) + require.Less(t, stopped, len(w.fake.Sessions())-before, "it stopped reading as soon as the bound was passed") + + // A narrower export fits the same bound. + recent := boundNow.Add(-24 * time.Hour) + events, err := bounded.ExportAuditEvents(context.Background(), orgScoped(business.AuditQuery{From: &recent})) + require.NoError(t, err) + require.NotEmpty(t, events) +} + +func TestAWindowThatCannotBeHalvedFailsInsteadOfOverflowing(t *testing.T) { + w := newBoundWarehouse(t) + at := boundNow.Add(-time.Hour) + pad := strings.Repeat("p", 900) + var records []business.AuditRecord + for i := 1; i <= 80; i++ { + records = append(records, boundEvent(t, i, at, map[string]any{"group": 1, "pad": pad})) + } + w.append(t, records...) + + reader := w.reader(t, 32<<10, 0) + _, err := reader.AggregateAuditEvents(context.Background(), orgScoped(business.AuditQuery{}), business.AuditAggregationSpec{GroupBy: []string{"payload:group"}}) + require.ErrorIs(t, err, ErrReadTooDense, "eighty events at one microsecond are more than a window holds") + require.LessOrEqual(t, reader.windowPeak.Load(), int64(32<<10+oneRow)) +} + +func TestHalvingAWindowCoversItExactly(t *testing.T) { + lo := time.Date(2026, 10, 1, 0, 0, 0, 0, time.UTC) + hi := lo.Add(48 * time.Hour) + ceiling := lo.Add(1000 * time.Hour) + for name, w := range map[string]window{ + "closed": {lo: &lo, hi: &hi}, + "exclusive": {lo: &lo, hi: &hi, hiExclusive: true}, + "open": {lo: &lo}, + } { + newer, older, ok := w.split(ceiling) + require.True(t, ok, name) + require.Equal(t, w.hi, newer.hi, "%s: the newer half keeps the upper bound", name) + require.Equal(t, w.hiExclusive, newer.hiExclusive, name) + require.Equal(t, w.lo, older.lo, "%s: the older half keeps the lower bound", name) + require.True(t, older.hiExclusive, "%s: the older half stops short of the middle", name) + require.True(t, newer.lo.Equal(*older.hi), "%s: the halves meet at the middle, which only the newer holds", name) + require.True(t, newer.lo.After(*w.lo) && newer.lo.Before(ceiling), name) + } + + mid, _, _ := window{lo: &lo, hi: &hi}.split(ceiling) + require.True(t, mid.lo.Equal(lo.Add(24*time.Hour))) + _, _, ok := window{lo: &lo}.split(lo) + require.False(t, ok, "no span below the ceiling") + two := lo.Add(time.Microsecond) + _, _, ok = window{lo: &lo, hi: &two}.split(ceiling) + require.False(t, ok, "a microsecond is the least a window spans") + _, _, ok = window{hi: &hi}.split(ceiling) + require.False(t, ok, "an open lower bound has no middle") + + // A window that spans centuries is halved at its middle too, not at the + // 146 years a saturated duration would give. + earliest := time.Date(1, 1, 1, 0, 0, 0, 0, time.UTC) + today := time.Date(2026, 10, 3, 0, 0, 0, 0, time.UTC) + newerHalf, olderHalf, ok := window{lo: &earliest, hi: &today, hiExclusive: true}.split(today) + require.True(t, ok) + require.True(t, newerHalf.lo.After(time.Date(1000, 1, 1, 0, 0, 0, 0, time.UTC)) && newerHalf.lo.Before(time.Date(1100, 1, 1, 0, 0, 0, 0, time.UTC)), + "the middle of the years 1 and 2026: %s", newerHalf.lo) + require.True(t, olderHalf.hi.Equal(*newerHalf.lo)) + + // And walking down it reaches every microsecond of the window exactly once. + spans := []window{{lo: &lo, hi: &hi}} + var leaves []window + for len(spans) > 0 && len(leaves) < 64 { + w := spans[0] + spans = spans[1:] + newer, older, ok := w.split(ceiling) + if !ok || w.hi.Sub(*w.lo) < time.Hour { + leaves = append(leaves, w) + continue + } + spans = append(spans, newer, older) + } + slices.SortFunc(leaves, func(a, b window) int { return a.lo.Compare(*b.lo) }) + require.True(t, leaves[0].lo.Equal(lo)) + for i := 1; i < len(leaves); i++ { + require.True(t, leaves[i-1].hi.Equal(*leaves[i].lo), "adjacent leaves meet") + require.True(t, leaves[i-1].hiExclusive, "and the older leaf excludes the instant they meet at") + } + require.True(t, leaves[len(leaves)-1].hi.Equal(hi)) +} diff --git a/module/services/accounts/code/pkg/auditstore/bigquerystore/reader_test.go b/module/services/accounts/code/pkg/auditstore/bigquerystore/reader_test.go index 942a28a07..e1e88f438 100644 --- a/module/services/accounts/code/pkg/auditstore/bigquerystore/reader_test.go +++ b/module/services/accounts/code/pkg/auditstore/bigquerystore/reader_test.go @@ -128,27 +128,46 @@ func TestEveryEventsReadIsBoundedInTimeWhenTheQueryIs(t *testing.T) { from, to := now.Add(-48*time.Hour), now.Add(-time.Hour) fromBound := `occurred_at >= CAST("2026-10-01 12:00:00.000000+00:00" AS TIMESTAMP)` toBound := `occurred_at <= CAST("2026-10-03 11:00:00.000000+00:00" AS TIMESTAMP)` - _, err := w.reader.AggregateAuditEvents(context.Background(), orgRead(business.AuditQuery{From: &from, To: &to}), business.AuditAggregationSpec{}) - require.NoError(t, err) - _, err = w.reader.ExportAuditEvents(context.Background(), orgRead(business.AuditQuery{From: &from, To: &to})) - require.NoError(t, err) - for _, session := range w.fake.Sessions() { - restriction := session.GetReadSession().GetReadOptions().GetRowRestriction() - require.Contains(t, restriction, fromBound, "partitions before the window are pruned") - require.Contains(t, restriction, toBound, "partitions after the window are pruned") + q := business.AuditQuery{From: &from, To: &to} + + // Every read walks the span in windows, newest first: the newest day, then + // the rest of it. The first window ends at the query's end, the last begins + // at its beginning, and each is bounded on both sides. + reads := map[string]func() error{ + "list": func() error { + _, _, err := w.reader.ListAuditEvents(context.Background(), orgRead(q)) + return err + }, + "aggregate": func() error { + _, err := w.reader.AggregateAuditEvents(context.Background(), orgRead(q), business.AuditAggregationSpec{}) + return err + }, + "export": func() error { + _, err := w.reader.ExportAuditEvents(context.Background(), orgRead(q)) + return err + }, } - - sessions := len(w.fake.Sessions()) - _, _, err = w.reader.ListAuditEvents(context.Background(), orgRead(business.AuditQuery{From: &from, To: &to})) - require.NoError(t, err) - list := w.fake.Sessions()[sessions:] - require.Len(t, list, 2, "the newest day, then the rest of the window") - require.Contains(t, list[0].GetReadSession().GetReadOptions().GetRowRestriction(), toBound) - require.Contains(t, list[1].GetReadSession().GetReadOptions().GetRowRestriction(), fromBound) - for _, session := range list { - restriction := session.GetReadSession().GetReadOptions().GetRowRestriction() - require.Contains(t, restriction, "occurred_at >=") - require.Contains(t, restriction, "occurred_at <") + // The tables a read touches: the events always, and the content details too + // when it needs them (an export carries every event's payload). + tables := map[string][]string{"list": {bigquerystore.EventsTable}, "aggregate": {bigquerystore.EventsTable}, "export": {bigquerystore.EventsTable, bigquerystore.DetailsTable}} + for name, read := range reads { + before := len(w.fake.Sessions()) + require.NoError(t, read(), name) + for _, table := range tables[name] { + var sessions []string + for _, session := range w.fake.Sessions()[before:] { + if strings.HasSuffix(session.GetReadSession().GetTable(), "/"+table) { + sessions = append(sessions, session.GetReadSession().GetReadOptions().GetRowRestriction()) + } + } + require.Len(t, sessions, 2, "%s %s: the newest day, then the rest of the window", name, table) + require.Contains(t, sessions[0], toBound, "%s %s: partitions after the window are pruned", name, table) + require.Contains(t, sessions[1], fromBound, "%s %s: partitions before the window are pruned", name, table) + for _, restriction := range sessions { + require.Contains(t, restriction, "occurred_at >=", name) + require.Regexp(t, `occurred_at <=? CAST`, restriction, name) + } + } } } diff --git a/module/services/accounts/code/pkg/auditstore/clickhousestore/clickhousestore.go b/module/services/accounts/code/pkg/auditstore/clickhousestore/clickhousestore.go index c24456393..43872a52c 100644 --- a/module/services/accounts/code/pkg/auditstore/clickhousestore/clickhousestore.go +++ b/module/services/accounts/code/pkg/auditstore/clickhousestore/clickhousestore.go @@ -78,6 +78,9 @@ type Config struct { // ContentDetailRetention is the details table's TTL: how long a // content-class event's full details are kept, in whole days. ContentDetailRetention time.Duration + // ExportMaxBytes bounds the events one export gathers, counted by their + // text (auditeval.Event.Size). Zero is business.AuditExportMaxBytes. + ExportMaxBytes int64 } // clusterPattern is a cluster name as remote_servers declares one, or a macro @@ -97,6 +100,8 @@ type Store struct { eventsDays int detailDays int readSettings clickhouse.Settings + // exportMaxBytes is the most an export gathers before it gives up. + exportMaxBytes int64 // writeSettings are sent with every insert: on a cluster, the quorum that // makes an acknowledgement durable. writeSettings clickhouse.Settings @@ -127,13 +132,21 @@ func New(conn Conn, cfg Config) (*Store, error) { if err != nil { return nil, err } + if cfg.ExportMaxBytes < 0 { + return nil, errors.New("clickhouse audit store: export bound cannot be negative") + } + exportMaxBytes := cfg.ExportMaxBytes + if exportMaxBytes == 0 { + exportMaxBytes = business.AuditExportMaxBytes + } store := &Store{ - conn: conn, - database: cfg.Database, - cluster: cfg.Cluster, - deploymentID: cfg.DeploymentID, - eventsDays: eventsDays, - detailDays: detailDays, + conn: conn, + database: cfg.Database, + cluster: cfg.Cluster, + deploymentID: cfg.DeploymentID, + eventsDays: eventsDays, + detailDays: detailDays, + exportMaxBytes: exportMaxBytes, // Pinned per read, whatever the user's profile says, because each one // changes an answer. readSettings: clickhouse.Settings{ diff --git a/module/services/accounts/code/pkg/auditstore/clickhousestore/reader.go b/module/services/accounts/code/pkg/auditstore/clickhousestore/reader.go index 5bc37e321..976595506 100644 --- a/module/services/accounts/code/pkg/auditstore/clickhousestore/reader.go +++ b/module/services/accounts/code/pkg/auditstore/clickhousestore/reader.go @@ -295,14 +295,21 @@ func (s *Store) joined(ctx context.Context, m *auditeval.Matcher, pl plan, order } // ExportAuditEvents implements business.AuditReader: every matching event, -// once each, newest first, with its payload. +// once each, newest first, with its payload. ClickHouse streams the events +// and the service holds none but those it returns; it gives up with +// business.ErrAuditExportTooLarge as soon as they pass the export bound, so +// an export's memory is the bound and not the history. func (s *Store) ExportAuditEvents(ctx context.Context, read business.AuditRead) ([]business.AuditEntry, error) { m, err := auditeval.NewMatcher(read) if err != nil { return nil, err } var events []*auditeval.Event + var gathered int64 err = s.joined(ctx, m, newPlan(m, read.Query), true, func(event *auditeval.Event) error { + if gathered += int64(event.Size()); gathered > s.exportMaxBytes { + return business.ErrAuditExportTooLarge + } events = append(events, event) return nil }) diff --git a/module/services/accounts/code/pkg/auditstore/clickhousestore/server_test.go b/module/services/accounts/code/pkg/auditstore/clickhousestore/server_test.go index 03d1530c9..78f03a684 100644 --- a/module/services/accounts/code/pkg/auditstore/clickhousestore/server_test.go +++ b/module/services/accounts/code/pkg/auditstore/clickhousestore/server_test.go @@ -491,6 +491,42 @@ func TestServerReadsATimeBoundOutsideTheColumnRange(t *testing.T) { require.Equal(t, 3, copies) } +// An export is as large as the history it matches, and the one read that +// holds its answer: past the bound it gives up instead of holding the history. +func TestServerExportGivesUpPastItsBound(t *testing.T) { + conn, database := serverDatabase(t) + ctx := context.Background() + org, actor := uuid.NewString(), uuid.NewString() + base := time.Now().UTC().Truncate(time.Microsecond).Add(-time.Minute) + var records []business.AuditRecord + for i := range 200 { + records = append(records, newRecord(t, org, actor, business.EventDocumentRead, "doc", base.Add(-time.Duration(i)*time.Minute), + business.RetentionContent, map[string]any{"pad": strings.Repeat("p", 900), "n": i})) + } + store := serverStore(t, conn, database) + require.NoError(t, store.Ensure(ctx)) + appendBatch(t, store, testDeployment, records...) + read := business.AuditRead{Scope: business.OrganizationAuditScope(org), Query: business.AuditQuery{OrgID: org}} + + all, err := store.ExportAuditEvents(ctx, read) + require.NoError(t, err) + require.Len(t, all, 200) + + bounded := serverStore(t, conn, database, func(cfg *Config) { cfg.ExportMaxBytes = 64 << 10 }) + _, err = bounded.ExportAuditEvents(ctx, read) + require.ErrorIs(t, err, business.ErrAuditExportTooLarge, "200 events of about 1.5 KB are over 64 KiB") + recent := base.Add(-10 * time.Minute) + read.Query.From = &recent + few, err := bounded.ExportAuditEvents(ctx, read) + require.NoError(t, err) + require.Len(t, few, 11, "a narrower export fits the same bound") + require.Equal(t, all[:11], few) + + _, err = New(conn, Config{Database: database, DeploymentID: testDeployment, EventsRetention: 2555 * 24 * time.Hour, + ContentDetailRetention: 30 * 24 * time.Hour, ExportMaxBytes: -1}) + require.ErrorContains(t, err, "export bound cannot be negative") +} + // Decimal texts whose nearest double a fast parser misses: ClickHouse reads // each — as a JSON number and as a JSON string — to the double Postgres's // float8in reads, so the metrics over them run in ClickHouse and agree. diff --git a/module/services/accounts/code/pkg/business/audit_export.go b/module/services/accounts/code/pkg/business/audit_export.go index ea9d09771..7fa90eb7f 100644 --- a/module/services/accounts/code/pkg/business/audit_export.go +++ b/module/services/accounts/code/pkg/business/audit_export.go @@ -5,13 +5,30 @@ import ( "context" "encoding/csv" "encoding/json" + "errors" "fmt" "strconv" "time" "github.com/codefly-dev/core/wool" + "google.golang.org/grpc/codes" + "google.golang.org/grpc/status" ) +// AuditExportMaxBytes bounds the events one export of a store of record +// holds: the text of their envelopes and details, 32 MiB — some tens of +// thousands of events. The export is one unary response, so the whole of it +// is in memory twice over (the events, then the file serialized from them) +// however the store reads it; without a bound a long history is a pod's +// memory, and the pod is every tenant's. A store reads in windows and gives +// up as soon as the events it has gathered pass the bound. +const AuditExportMaxBytes = 32 << 20 + +// ErrAuditExportTooLarge is the refusal of an export whose events pass +// AuditExportMaxBytes. Narrowing the export (an actor, an event type) is the +// way past it. +var ErrAuditExportTooLarge = errors.New("audit export: more events match than one download holds; narrow it by actor or event type") + // ExportAuditLog queries all matching audit events and serializes them to CSV or // JSON. eventTypes is the set form of eventType, applied together with it, so a // download can carry the same family-of-types filter a summary tile opens. @@ -36,6 +53,9 @@ func (s *Service) ExportAuditLog(ctx context.Context, orgID, format, actorID, ev return nil, "", "", w.Wrapf(err, "query audit log for export") } all, err := s.auditReads().ExportAuditEvents(ctx, read) + if errors.Is(err, ErrAuditExportTooLarge) { + return nil, "", "", status.Error(codes.ResourceExhausted, ErrAuditExportTooLarge.Error()) + } if err != nil { return nil, "", "", w.Wrapf(err, "query audit log for export") } diff --git a/module/services/accounts/code/pkg/business/audit_export_test.go b/module/services/accounts/code/pkg/business/audit_export_test.go index 4c97ea908..7a16aa002 100644 --- a/module/services/accounts/code/pkg/business/audit_export_test.go +++ b/module/services/accounts/code/pkg/business/audit_export_test.go @@ -1,11 +1,16 @@ package business import ( + "context" "encoding/csv" "encoding/json" + "fmt" "strings" "testing" "time" + + "google.golang.org/grpc/codes" + "google.golang.org/grpc/status" ) // The downloadable export is the surface a compliance review actually leaves @@ -70,3 +75,39 @@ func TestAuditExportCarriesTheClientTheCallCameThrough(t *testing.T) { t.Errorf("a call from the host's own session must name no client, got %v", decoded[1]["client_id"]) } } + +// exportingStore is a store of record whose export is whatever the test says. +type exportingStore struct { + AuditStore + entries []AuditEntry + err error +} + +func (s exportingStore) ExportAuditEvents(context.Context, AuditRead) ([]AuditEntry, error) { + return s.entries, s.err +} + +// A store that gives up an export for its size is a request the caller can +// narrow, not a server fault: the refusal reaches the caller as such, and an +// export within the bound is served as before. +func TestAuditExportTooLargeIsARefusalNotAFault(t *testing.T) { + service := &Service{auditStore: exportingStore{err: fmt.Errorf("read: %w", ErrAuditExportTooLarge)}} + _, _, _, err := service.ExportAuditLog(t.Context(), "org-1", "json", "", "", nil) + if status.Code(err) != codes.ResourceExhausted { + t.Fatalf("code = %v (%v), want ResourceExhausted", status.Code(err), err) + } + if !strings.Contains(err.Error(), "narrow it by actor or event type") { + t.Errorf("the refusal says how to get past it, got %q", err) + } + + service = &Service{auditStore: exportingStore{entries: []AuditEntry{ + {ID: "evt-1", EventType: EventTeamCreated, SchemaVersion: 1, OrgID: "org-1", CreatedAt: time.Unix(0, 0).UTC()}, + }}} + body, contentType, _, err := service.ExportAuditLog(t.Context(), "org-1", "json", "", "", nil) + if err != nil { + t.Fatalf("ExportAuditLog: %v", err) + } + if contentType != "application/json" || !strings.Contains(string(body), "evt-1") { + t.Errorf("export = %s (%s)", body, contentType) + } +} From e24ab16f704634dcf7d097aafe4ac9b145415eb6 Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Mon, 5 Oct 2026 17:13:00 -0400 Subject: [PATCH 33/69] fix(audit): register the quarantine relation in the authority inventory Migration 21 creates audit_event_quarantine, which no request role may reach; only the audit relay's role reads or writes it. Record it where the database authority is declared so the inventory gates hold it to that: the relation catalog, the tenant-scope and grants tables in DATABASE_AUTHORITY.md, and the hardening suite's expected privileges for the request roles (none) and the control plane (none). Co-Authored-By: Claude Sonnet 5.5 --- module/DATABASE_AUTHORITY.md | 4 ++-- .../accounts/code/pkg/infra/postgres_role_hardening_test.go | 6 ++++++ .../services/accounts/code/pkg/relationcatalog/catalog.go | 4 ++++ 3 files changed, 12 insertions(+), 2 deletions(-) diff --git a/module/DATABASE_AUTHORITY.md b/module/DATABASE_AUTHORITY.md index 62aebad21..d22275807 100644 --- a/module/DATABASE_AUTHORITY.md +++ b/module/DATABASE_AUTHORITY.md @@ -502,7 +502,7 @@ All tenant/user rows remain constrained by their forced RLS policies. | select, insert | `gdpr_requests` (accepted by request traffic, transitioned only by the leased privacy worker under the control plane) | | select, insert, update, delete | `mfa_backup_codes`, `mfa_devices`, `notifications`, `user_identities` | | insert only | `audit_event_queue` (read and deleted only by the audit relay, as `app_job_worker`), `client_authorization_codes`, `mfa_login_transactions` | -| no tenant relation authority | `job_attempts`, `job_messages`, `job_state_transitions`, `magic_links`, `membership_integrity_findings` | +| no tenant relation authority | `audit_event_quarantine` (written and read only by the audit relay, as `app_job_worker`; nothing deletes it), `job_attempts`, `job_messages`, `job_state_transitions`, `magic_links`, `membership_integrity_findings` | Retention deletes and token-based pre-auth reads/updates use the audited control-plane boundary; they are intentionally not granted to `app_tenant`. @@ -532,7 +532,7 @@ executable inventory and this table in the same change. | Scope | Relations | Required database boundary | |---|---|---| | `global` | `audit_event_types`, `bootstrap_state`, `data_retention_policies`, `datasource_credential_budgets`, `email_templates`, `feature_flags`, `identity_providers`, `plan_entitlements`, `plans`, `platform_admins`, `solution_registrations` | No RLS; exact grants | -| `tenant` | `actor_chain_journal`, `actor_chain_revocations`, `api_keys`, `approval_decisions`, `approval_requests`, `audit_event_idempotency`, `audit_event_queue`, `audit_events`, `connector_credentials`, `dashboards`, `datasource_account_links`, `datasource_domains`, `datasource_group_bindings`, `datasource_sources`, `delegation_grants`, `domain_events`, `entitlement_overrides`, `github_app_installations`, `github_app_setups`, `installations`, `invitations`, `membership_integrity_findings`, `org_generic_settings`, `org_identity_providers`, `org_settings`, `organization_activations`, `organization_authorization_revisions`, `organization_members`, `organizations`, `principal_authorization_revisions`, `principals`, `record_shares`, `role_assignments`, `role_permissions`, `roles`, `scope_grants`, `scope_nodes`, `source_delegations`, `source_read_revisions`, `subscriptions`, `team_members`, `team_membership_quarantine`, `teams`, `usage_events`, `usage_totals`, `webhook_deliveries`, `webhook_subscriptions`, `work_context_replay` | Enabled and forced RLS with at least one policy | +| `tenant` | `actor_chain_journal`, `actor_chain_revocations`, `api_keys`, `approval_decisions`, `approval_requests`, `audit_event_idempotency`, `audit_event_quarantine`, `audit_event_queue`, `audit_events`, `connector_credentials`, `dashboards`, `datasource_account_links`, `datasource_domains`, `datasource_group_bindings`, `datasource_sources`, `delegation_grants`, `domain_events`, `entitlement_overrides`, `github_app_installations`, `github_app_setups`, `installations`, `invitations`, `membership_integrity_findings`, `org_generic_settings`, `org_identity_providers`, `org_settings`, `organization_activations`, `organization_authorization_revisions`, `organization_members`, `organizations`, `principal_authorization_revisions`, `principals`, `record_shares`, `role_assignments`, `role_permissions`, `roles`, `scope_grants`, `scope_nodes`, `source_delegations`, `source_read_revisions`, `subscriptions`, `team_members`, `team_membership_quarantine`, `teams`, `usage_events`, `usage_totals`, `webhook_deliveries`, `webhook_subscriptions`, `work_context_replay` | Enabled and forced RLS with at least one policy | | `user` | `client_authorization_codes`, `gdpr_requests`, `mfa_backup_codes`, `mfa_devices`, `mfa_login_transactions`, `notifications`, `onboarding_progress`, `resource_follows`, `sessions`, `user_consent_events`, `user_consent_preferences`, `user_identities`, `users`, `webauthn_ceremonies`, `webauthn_credentials` | Enabled and forced RLS with at least one policy | | `pre_auth` | `magic_links`, `waitlist_entries` | Enabled and forced RLS; fail-closed request policy, accessed only by the control-plane role | | `job` | `job_messages` | Enabled and forced RLS with at least one policy; no request relation grant — function-only scoped enqueue plus exact job-worker grants | diff --git a/module/services/accounts/code/pkg/infra/postgres_role_hardening_test.go b/module/services/accounts/code/pkg/infra/postgres_role_hardening_test.go index bc68d4ad9..119c829ac 100644 --- a/module/services/accounts/code/pkg/infra/postgres_role_hardening_test.go +++ b/module/services/accounts/code/pkg/infra/postgres_role_hardening_test.go @@ -62,6 +62,7 @@ var appTenantRelationPrivileges = map[string]relationPrivileges{ "approval_decisions": {selectRows: true, insertRows: true}, // append-only, like actor_chain_journal "approval_requests": {selectRows: true, insertRows: true, updateRows: true}, "audit_event_idempotency": {selectRows: true, insertRows: true}, // append-only guard, like audit_events + "audit_event_quarantine": {}, // only the audit relay's role reaches it; no request-traffic authority at all "audit_event_queue": {insertRows: true}, // a writer never reads, changes or removes a queued event "audit_events": {selectRows: true, insertRows: true}, "connector_credentials": {selectRows: true, insertRows: true, updateRows: true, deleteRows: true}, @@ -367,6 +368,11 @@ func TestControlPlaneRelationGrantsAreExact(t *testing.T) { if relation == "audit_event_queue" { want = relationPrivileges{insertRows: true} } + // audit_event_quarantine holds the audit events the relay set aside; only + // the relay's role, as the job worker, reads or writes it. + if relation == "audit_event_quarantine" { + want = relationPrivileges{} + } // actor_chain_journal / actor_chain_revocations are append-only // like audit_events: the control plane reads and inserts but never // updates or deletes (an immutable trigger rejects those anyway). diff --git a/module/services/accounts/code/pkg/relationcatalog/catalog.go b/module/services/accounts/code/pkg/relationcatalog/catalog.go index d8cc76ac7..502c5e13c 100644 --- a/module/services/accounts/code/pkg/relationcatalog/catalog.go +++ b/module/services/accounts/code/pkg/relationcatalog/catalog.go @@ -130,6 +130,10 @@ var authorities = map[string]Authority{ Scope: ScopeTenant, PolicyShape: ShapePolymorphic, ScopeColumn: "org_id", Notes: "The transactional audit queue under a swap value (ADR 0009): writers insert only, under audit_events' tenant policy; the relay reads and deletes as the job worker.", }, + "audit_event_quarantine": { + Scope: ScopeTenant, PolicyShape: ShapeDirect, ScopeColumn: "org_id", + Notes: "Audit events the relay set aside because the warehouse refused them (ADR 0009), moved whole from the queue. Request traffic holds no grant; only the relay's role reads or writes it, and nothing deletes a row. A platform event, with no organization, matches no tenant.", + }, "audit_events": { Scope: ScopeTenant, PolicyShape: ShapePolymorphic, ScopeColumn: "org_id", Notes: "NULL org_id rows (system events) are visible only under the control-plane role. Monthly partitions inherit the partitioned parent's policy.", From 5db2d00618fcec51ac49029b64c18f8a878ff0b3 Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Mon, 5 Oct 2026 17:16:31 -0400 Subject: [PATCH 34/69] docs(audit): make the docs say what the audit relay, importer and exports now do Bring the prose in line with the code the relay, history-removal and warehouse-read changes landed together: - migration 18 and AuditBatch no longer say a queued event can never be overtaken: the relay's gate is transaction visibility, not sequence order, so delivery order is not guaranteed and nothing downstream depends on it (comment only; no statement changes) - AUDIT_RELAY_MAX_WAIT is capped at 60s - the role catalog import refuses an unset AUDIT_SINK, so a deployment that relies on the service's postgres default must set AUDIT_SINK=postgres explicitly (AUTHZ.md, deployment README) - the history copy drops through audit_events_drop_verified_partitions, not the function retention uses, and its exit statuses are documented - a warehouse export is bounded at 32 MiB (ErrAuditExportTooLarge), a ClickHouse cluster insert waits for a quorum and a read for a caught-up replica, a BigQuery append is split by request size, and a row the warehouse refuses is set aside in audit_event_quarantine - the job worker role's authority lists the relay's queue and quarantine Co-Authored-By: Claude Sonnet 5.5 --- AUTHZ.md | 9 +++++++-- module/DATABASE_AUTHORITY.md | 2 +- module/FEATURES.md | 10 +++++----- module/deployment/README.md | 4 +++- module/services/accounts/AGENTS.md | 8 ++++++-- .../accounts/code/pkg/business/audit_store.go | 6 +++--- .../store/migrations/18_audit_event_queue.up.sql | 13 ++++++++----- 7 files changed, 33 insertions(+), 19 deletions(-) diff --git a/AUTHZ.md b/AUTHZ.md index 5c921afaf..80b41e67b 100644 --- a/AUTHZ.md +++ b/AUTHZ.md @@ -912,8 +912,13 @@ The importer records one audit event per changed role through the same emitter the service uses, so it must run under the deployment's `AUDIT_SINK` (`-audit-sink`, defaulting to `$AUDIT_SINK`): under a swap value (ADR 0009) its events go to the transactional queue the accounts relay delivers, not to -`audit_events`. The deploy step below declares it (`service_environment: -[AUDIT_SINK]`), so the driver gives the Job the accounts service's own value. +`audit_events`. Unlike the service, the importer has no default: an unset (or +blank) `AUDIT_SINK` makes it exit before touching the store, because guessing +`postgres` would write its events where a warehouse deployment never reads +them. A deployment that relies on the service's `postgres` default must +therefore set `AUDIT_SINK=postgres` explicitly. The deploy step below declares +it (`service_environment: [AUDIT_SINK]`), so the driver gives the Job the +accounts service's own value, which is why the service must have one. Domain core is `pkg/rolecatalog` (parse + diff + deterministic plan, no DB); `pkg/infra` snapshots current state and applies the plan in one transaction. diff --git a/module/DATABASE_AUTHORITY.md b/module/DATABASE_AUTHORITY.md index d22275807..c36a9d4b5 100644 --- a/module/DATABASE_AUTHORITY.md +++ b/module/DATABASE_AUTHORITY.md @@ -429,7 +429,7 @@ pools' generic refusal. | `app_control_plane` | no | yes | Audited pre-auth, bootstrap, platform administration, retention, and cross-tenant account operations | | `app_billing_worker` | no | yes | Stripe subscription catalog reads and reconciliation writes only | | `app_webhook_worker` | no | yes | Webhook subscription reads and delivery-history projection only | -| `app_job_worker` | no | yes | Product-neutral job messages, attempts, transitions, and scoped enqueue operation only | +| `app_job_worker` | no | yes | Product-neutral job messages, attempts, transitions, and scoped enqueue operation, plus the audit relay's own relations: reads and deletes `audit_event_queue`, reads and inserts `audit_event_quarantine` | All application roles are `NOLOGIN`, `NOINHERIT`, `NOSUPERUSER`, `NOCREATEDB`, `NOCREATEROLE`, and `NOREPLICATION`. The tenant role cannot diff --git a/module/FEATURES.md b/module/FEATURES.md index 94b396880..e4e7a1ffa 100644 --- a/module/FEATURES.md +++ b/module/FEATURES.md @@ -395,10 +395,10 @@ see `JOBS.md` for the exact boundary and sequencing. | Multi-field filter | ✅ | By org, actor, action, resource, time range | | Cursor pagination | ✅ | Stable across writes | | Retention purge job | 🟡 | Configurable database purge policies exist; provider data, backups, holds, receipts, and production execution evidence do not | -| Export (JSON/CSV) | ✅ | `audit_export.go` | +| Export (JSON/CSV) | ✅ | `audit_export.go`; under a warehouse store of record one export holds at most 32 MiB of event text, past which it fails with `ErrAuditExportTooLarge` (`ResourceExhausted`): narrow it by actor or event type | | Impersonation tracking | ✅ | Records both real actor + viewed-as user | | Replay / event sourcing| ❌ | Audit log is read-only history; not used to reconstruct state | -| Warehouse store of record | 🟡 | `AUDIT_SINK=bigquery` or `AUDIT_SINK=clickhouse` (ADR 0009): each record commits to a Postgres queue in the mutation's transaction, then a relay delivers it to the warehouse and a locked GCS or S3 archive. Reads are scoped to the organization and deployment and deduplicated by event id; BigQuery uses the Storage Read API without query jobs, and ClickHouse uses SQL with service evaluation where needed. The shared parity suite compares both with Postgres. `audit-history-copy` verifies a switching deployment's copied history before a confirmed monthly-partition drop. No Azure Blob archive writer exists yet. | +| Warehouse store of record | 🟡 | `AUDIT_SINK=bigquery` or `AUDIT_SINK=clickhouse` (ADR 0009): each record commits to a Postgres queue in the mutation's transaction, then a relay delivers it to the warehouse and a locked GCS or S3 archive. Reads are scoped to the organization and deployment and deduplicated by event id; BigQuery uses the Storage Read API without query jobs, and ClickHouse uses SQL with service evaluation where needed. A BigQuery read holds a bounded amount in memory whatever the history: a window that needs more than its byte budget is read again as halves. A row the warehouse refuses on its own is set aside, whole, in `audit_event_quarantine` instead of holding every organization's events behind it (`saas.audit_queue.quarantined`; replaying quarantined rows is not built yet). The shared parity suite compares both with Postgres. `audit-history-copy` verifies a switching deployment's copied history before a confirmed monthly-partition drop. No Azure Blob archive writer exists yet. | ### Admin panel @@ -655,14 +655,14 @@ Environment variables consumed by the api: | `POSTHOG_HOST` | Explicit HTTPS or local capture origin | | `POSTHOG_API_HOST` | Separate PostHog management/deletion origin | | `AUDIT_SINK` | `postgres` (default), `both` (tee to `AUDIT_EXTERNAL_URL`, optional `AUDIT_EXTERNAL_TOKEN`), or `bigquery` / `clickhouse` (swap the audit store of record, ADR 0009); `external` is refused | -| `AUDIT_BIGQUERY_PROJECT`, `AUDIT_BIGQUERY_DATASET` | Required under `bigquery`: the store of record. The dataset must exist; the `audit_events` and `audit_event_details` tables are created if missing. Credentials are Application Default Credentials only. The service's identity holds no `bigquery.jobs.create`: it writes with streaming inserts and reads through the Storage Read API (`bigquery.tables.getData`, `bigquery.readsessions.create`/`getData`/`update`), because append plus job creation would permit DML | +| `AUDIT_BIGQUERY_PROJECT`, `AUDIT_BIGQUERY_DATASET` | Required under `bigquery`: the store of record. The dataset must exist; the `audit_events` and `audit_event_details` tables are created if missing. Credentials are Application Default Credentials only. The service's identity holds no `bigquery.jobs.create`: it writes with streaming inserts (a batch is split into requests of at most 500 rows and 8 MiB; a single row too large for any request is refused by naming its event, and is handled as any row the warehouse refuses) and reads through the Storage Read API (`bigquery.tables.getData`, `bigquery.readsessions.create`/`getData`/`update`), because append plus job creation would permit DML | | `AUDIT_CLICKHOUSE_DSN` | Required under `clickhouse`: the store of record, as `clickhouse://:@:9000/` (add `?secure=true` for TLS), from the deployment's secret — never logged. The database must exist; the `audit_events` and `audit_event_details` tables are created if missing (partitioned by month, a TTL per tier) and refused if their columns or TTL differ. The user needs `CREATE TABLE`, `INSERT` and `SELECT` on that database, and no `ALTER`, `DELETE`, `TRUNCATE` or `DROP`. Tested against ClickHouse 25.8 | -| `AUDIT_CLICKHOUSE_CLUSTER` | Optional under `clickhouse`: a cluster name or macro (`{cluster}`); the tables are then created `ON CLUSTER` as `ReplicatedMergeTree`, coordinated by ClickHouse Keeper under the servers' default replica path and name | +| `AUDIT_CLICKHOUSE_CLUSTER` | Optional under `clickhouse`: a cluster name or macro (`{cluster}`); the tables are then created `ON CLUSTER` as `ReplicatedMergeTree`, coordinated by ClickHouse Keeper under the servers' default replica path and name. On a cluster every insert waits for a majority of the replicas (`insert_quorum=auto`, one quorum insert at a time, a 60-second timeout) before the relay deletes its queue row, and every read is served only by a replica that holds all of them (`select_sequential_consistency`); a replica that has not caught up fails the read, which the caller retries. Without a cluster neither setting is sent | | `AUDIT_EVENTS_RETENTION_DAYS` | Required under `clickhouse`: the compliance window in days — the events table's TTL — at least `AUDIT_CONTENT_RETENTION_DAYS` | | `AUDIT_ARCHIVE_URL` | Required under a swap value: the locked archive, as `gs://` or `s3://` (the scheme picks the writer; any warehouse pairs with either). The bucket carries a retention lock the deployment sets (GCS Bucket Lock, S3 Object Lock default retention) and receives one create-only object per relay batch. S3 takes credentials and region from the AWS default chain and `AWS_REGION`, and an S3-compatible store's endpoint from `AWS_ENDPOINT_URL_S3` (addressed path-style); the writer needs `s3:PutObject` only | | `AUDIT_DEPLOYMENT_ID` | Required under a swap value: stamped on every record and the archive path | | `AUDIT_CONTENT_RETENTION_DAYS` | Required under a swap value: days a content-class event's full details are kept (the details table's partition expiration on BigQuery, its TTL on ClickHouse) | -| `AUDIT_RELAY_BATCH_SIZE`, `AUDIT_RELAY_MAX_WAIT` | Optional under a swap value: events per delivery (default 500, at most 5000) and how long a partial batch waits (default `5s`) | +| `AUDIT_RELAY_BATCH_SIZE`, `AUDIT_RELAY_MAX_WAIT` | Optional under a swap value: events per delivery (default 500, at most 5000) and how long a partial batch waits (default `5s`, at most `60s`: a longer wait would reach the five-minute relay-lag alert on a healthy relay, so a larger value is refused) | | `ERROR_TRACKING_MODE` | Explicit `disabled` or `sentry`; rejects partial config | | `SENTRY_DSN` | Server Sentry DSN, required in Sentry mode | | `OBSERVABILITY_EXPORTER` | Required in-graph collector output: `debug` or `otlphttp`; the collector refuses to start when it is unset | diff --git a/module/deployment/README.md b/module/deployment/README.md index c8d082f12..5bb02a376 100644 --- a/module/deployment/README.md +++ b/module/deployment/README.md @@ -81,7 +81,9 @@ driver mounts the catalog, connects the target, sets those variables, runs `command`, and fails the promotion if it exits non-zero. `role-catalog-import` inherits `AUDIT_SINK`: it records its audit events through the same emitter as `accounts`, so under a swap value (ADR 0009) they reach the queue the accounts -relay delivers instead of `audit_events`. Re-running an unchanged catalog is an empty +relay delivers instead of `audit_events`. Unlike `accounts`, the importer refuses +an unset `AUDIT_SINK` rather than default to `postgres`, so a deployment that +relies on that default must set `AUDIT_SINK=postgres` explicitly. Re-running an unchanged catalog is an empty no-op, so the step is idempotent. Generation rejects a deploy Job whose catalog artifact is absent, whose write target is not a declared dependency of the running service, that references an undeclared service, or that writes to a diff --git a/module/services/accounts/AGENTS.md b/module/services/accounts/AGENTS.md index 0bdbd2685..9104879f2 100644 --- a/module/services/accounts/AGENTS.md +++ b/module/services/accounts/AGENTS.md @@ -113,8 +113,12 @@ survives a restart and reaches every replica, and is only served when it is live event — and verifies them by reading them back. It reads back before it writes, so a rerun finishes an interrupted run. Only with `-confirm-drop`, after a verification pass in the same run and a recount showing nothing - changed, does it drop the copied partitions, through the function retention - uses; it never deletes a row. + changed, does it drop the copied partitions, named one by one through + `audit_events_drop_verified_partitions` (migration 20), which recounts each + under a lock and drops exactly those; retention's + `audit_events_drop_partitions_before` is not used. It never deletes a row. + Its exit statuses (2 usage, 3 unverified, 4 drop refused, 5 nothing to verify) + are documented in `module/AUDIT_OPERATIONS.md`. - A composed **module** with no frontend half declares its own audit event types through `ModuleCapabilitiesService.DeclareAuditEventTypes` (`pkg/business/module_audit_declarations.go`): the same validator diff --git a/module/services/accounts/code/pkg/business/audit_store.go b/module/services/accounts/code/pkg/business/audit_store.go index 57f4f5685..de373790b 100644 --- a/module/services/accounts/code/pkg/business/audit_store.go +++ b/module/services/accounts/code/pkg/business/audit_store.go @@ -102,9 +102,9 @@ type AuditArchive interface { WriteAuditBatch(ctx context.Context, batch AuditBatch) error } -// AuditBatch is one delivery: the records of a run of queued events, in queue -// order — oldest first, which keeps every organization's events (and the -// platform's, which have none) in the order they were queued. +// AuditBatch is one delivery: the records of a run of queued events, oldest +// first. The order is not a guarantee across batches (see AuditQueue.Drain): a +// store keys a record by event id and a read orders by event time. type AuditBatch struct { // ID is unique to this delivery attempt; the archive names its object by it. ID string diff --git a/module/services/store/migrations/18_audit_event_queue.up.sql b/module/services/store/migrations/18_audit_event_queue.up.sql index 403e03750..a6ef0a030 100644 --- a/module/services/store/migrations/18_audit_event_queue.up.sql +++ b/module/services/store/migrations/18_audit_event_queue.up.sql @@ -11,13 +11,16 @@ -- touched, and under a swap value it simply receives no new rows. Under the -- default (postgres) and the tee (both) nothing writes this table. CREATE TABLE public.audit_event_queue ( - -- Insertion order. The relay delivers in this order, which keeps every - -- organization's events — and the platform's, which have no organization — - -- in the order they were queued. + -- Insertion order. A batch is read oldest first, but delivery order is not + -- guaranteed: see xact_id below. Nothing downstream depends on it, because + -- both stores key a record by event id and every read orders by event time. seq bigint NOT NULL, -- The writing transaction. The relay reads a row only once every - -- transaction older than the oldest one still running has finished, so a - -- row committed late can never be overtaken by one queued after it. + -- transaction older than the oldest one still running has finished, so it + -- never reads a row whose transaction has not committed. The gate is + -- transaction visibility, not sequence order: a row whose transaction took + -- its id earlier but its sequence number later can be delivered ahead of a + -- lower sequence number still to commit. xact_id xid8 DEFAULT pg_current_xact_id() NOT NULL, -- The event, column for column as audit_events keeps it. id uuid NOT NULL, From 666525504594d7b22ad3b07aca2e8f9595e5fe7b Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Tue, 6 Oct 2026 15:08:02 -0400 Subject: [PATCH 35/69] fix(audit): let each store name the rows it refuses for good The relay decided which rows to set aside from timing: a row that failed alone, then failed again after another write succeeded. A flapping warehouse therefore quarantined good rows, and enough bad rows exhausted the relay's search and stalled the queue. The store is the only party that knows why a write failed, so it now says so. business.PermanentRowRejection names the event whose row a store refused for that row's own content. Everything else a store returns is retryable. BigQuery: a row insertAll lists with the reason "invalid", and every row too large for a request (all of them, found before anything is sent) or whose values cannot be encoded. Rows listed as "stopped", request-level failures (5xx, 429, quota, timeouts) and any other reason are not refusals. ClickHouse: a value the driver cannot encode (named before anything is sent), and a block the server refuses with a code that means a row's content is malformed (parsing 6, 26, 27, 38, 41, 72, 117; type and range 53, 70, 321; size 131; table CHECK 469). Such a block is split down to single rows to find the row. Network, quorum, replica, timeout, memory, too-many-parts, Keeper, access, missing-table codes and any unlisted code are never searched and never refuse a row. The relay does not read the new error yet; that follows. Co-Authored-By: Claude Sonnet 5.5 --- .../auditstore/bigquerystore/bigquerystore.go | 113 +++++-- .../bigquerystore/rejection_test.go | 197 ++++++++++++ .../clickhousestore/clickhousestore.go | 113 ++++++- .../auditstore/clickhousestore/rejection.go | 81 +++++ .../clickhousestore/rejection_server_test.go | 55 ++++ .../clickhousestore/rejection_test.go | 291 ++++++++++++++++++ .../code/pkg/business/audit_row_rejection.go | 62 ++++ .../pkg/business/audit_row_rejection_test.go | 27 ++ .../accounts/code/pkg/business/audit_store.go | 10 + 9 files changed, 917 insertions(+), 32 deletions(-) create mode 100644 module/services/accounts/code/pkg/auditstore/bigquerystore/rejection_test.go create mode 100644 module/services/accounts/code/pkg/auditstore/clickhousestore/rejection.go create mode 100644 module/services/accounts/code/pkg/auditstore/clickhousestore/rejection_server_test.go create mode 100644 module/services/accounts/code/pkg/auditstore/clickhousestore/rejection_test.go create mode 100644 module/services/accounts/code/pkg/business/audit_row_rejection.go create mode 100644 module/services/accounts/code/pkg/business/audit_row_rejection_test.go diff --git a/module/services/accounts/code/pkg/auditstore/bigquerystore/bigquerystore.go b/module/services/accounts/code/pkg/auditstore/bigquerystore/bigquerystore.go index 7676acb48..2dd246677 100644 --- a/module/services/accounts/code/pkg/auditstore/bigquerystore/bigquerystore.go +++ b/module/services/accounts/code/pkg/auditstore/bigquerystore/bigquerystore.go @@ -53,8 +53,9 @@ const rowEnvelopeBytes = 64 var ErrRowTooLarge = errors.New("bigquery audit store: row exceeds the request size limit") // RowTooLargeError names the event whose row cannot be sent: alone it is more -// than a request carries. Retrying cannot help, so a caller that delivers -// batches in order sets that event aside rather than retry the batch forever. +// than a request carries. Retrying cannot help, so AppendAuditBatch returns it +// inside a business.PermanentRowRejection, which is what the relay sets a row +// aside on. type RowTooLargeError struct { // Table is the table the row was for. Table string @@ -335,18 +336,29 @@ func BatchRows(batch business.AuditBatch) (events, details []Row) { // AppendAuditBatch implements business.AuditStoreWriter: the batch's rows // (BatchRows) streamed to the events and details tables, each table in as few // requests as hold at most maxRowsPerInsert rows and maxRequestBytes bytes. -// Every request is planned before the first is sent, so a row too large for a -// request (a *RowTooLargeError, naming its event) fails the batch before -// anything of it is written. +// +// What a failure says about the batch is what the relay acts on, so the error +// is classified here, where BigQuery's answer is known: +// +// - A row that cannot be sent in any request (a *RowTooLargeError), or whose +// values cannot be encoded, is returned as a business.PermanentRowRejection +// naming its event. Every request is planned before the first is sent, so +// such a row fails the batch before anything of it is written. +// - A row BigQuery reports back with the reason "invalid" (insertAll lists +// each row it refused, with the reason) is returned the same way: the value +// is wrong for its column, and sending it again gives the same answer. The +// batch's other rows, which BigQuery reports as "stopped", are not refused; +// the relay sends them again without the refused rows. +// - Every other failure is returned as it came — a request that failed (any +// HTTP error, a throttle, a quota, a timeout, a dead backend), a row +// reported with any other reason, an error of a kind not known here — and +// the relay retries the batch. func (s *Store) AppendAuditBatch(ctx context.Context, batch business.AuditBatch) error { eventRows, detailRows := BatchRows(batch) - events, err := planRequests(EventsTable, eventRows) - if err != nil { - return err - } - details, err := planRequests(DetailsTable, detailRows) - if err != nil { - return err + events, eventsRefused := planRequests(EventsTable, eventRows) + details, detailsRefused := planRequests(DetailsTable, detailRows) + if refused := append(eventsRefused, detailsRefused...); len(refused) > 0 { + return errors.Join(refused...) } if err := put(ctx, s.events, events); err != nil { return fmt.Errorf("bigquery audit store: append to %s: %w", EventsTable, err) @@ -369,18 +381,26 @@ func rowBytes(row Row) (int, error) { // planRequests splits rows, in order, into requests of at most maxRowsPerInsert // rows and maxRequestBytes bytes each. A row that does not fit a request alone -// is a *RowTooLargeError. -func planRequests(table string, rows []Row) ([][]bigquery.ValueSaver, error) { - var requests [][]bigquery.ValueSaver +// (a *RowTooLargeError) and a row whose values cannot be encoded are returned +// as refusals of that row, and left out of every request. +func planRequests(table string, rows []Row) (requests [][]bigquery.ValueSaver, refused []error) { var current []bigquery.ValueSaver currentBytes := 0 for _, row := range rows { - size, err := rowBytes(row) - if err != nil { - return nil, fmt.Errorf("bigquery audit store: append to %s: %w", table, err) + size, encodeErr := rowBytes(row) + if encodeErr != nil { + refused = append(refused, &business.PermanentRowRejection{ + EventID: row.InsertID, + Cause: fmt.Errorf("bigquery audit store: append to %s: %w", table, encodeErr), + }) + continue } if size > maxRequestBytes { - return nil, &RowTooLargeError{Table: table, EventID: row.InsertID, Size: size, Limit: maxRequestBytes} + refused = append(refused, &business.PermanentRowRejection{ + EventID: row.InsertID, + Cause: &RowTooLargeError{Table: table, EventID: row.InsertID, Size: size, Limit: maxRequestBytes}, + }) + continue } if len(current) == maxRowsPerInsert || currentBytes+size > maxRequestBytes { requests = append(requests, current) @@ -392,18 +412,69 @@ func planRequests(table string, rows []Row) ([][]bigquery.ValueSaver, error) { if len(current) > 0 { requests = append(requests, current) } - return requests, nil + return requests, refused } func put(ctx context.Context, inserter rowInserter, requests [][]bigquery.ValueSaver) error { for _, request := range requests { if err := inserter.Put(ctx, request); err != nil { - return describePutError(err) + return classifyPutError(err) } } return nil } +// reasonInvalid is the reason BigQuery gives a streamed row whose own content +// it refuses: a value that does not fit its column. The other reasons a row +// carries — "stopped" for a row that was fine but sent beside an invalid one, +// "timeout", "backendError", "internalError", "rateLimitExceeded", +// "quotaExceeded" — say nothing against the row. +const reasonInvalid = "invalid" + +// classifyPutError turns what Put returned into what the relay needs. A +// PutMultiError lists every row BigQuery refused with the reasons it gave; each +// row it gave the reason "invalid" is a *business.PermanentRowRejection, and the +// error also carries the original, so the rows that were merely stopped, and any +// that failed for another reason, are described and retried. Every other error is +// the request's, not a row's, and is returned as it came. +func classifyPutError(err error) error { + var multi bigquery.PutMultiError + if !errors.As(err, &multi) || len(multi) == 0 { + return err + } + failures := []error{describePutError(err)} + var refused []error + for _, row := range multi { + if row.InsertID == "" || !rowIsInvalid(row) { + continue + } + refused = append(refused, &business.PermanentRowRejection{ + EventID: row.InsertID, + Cause: fmt.Errorf("bigquery audit store: the row at index %d was refused as invalid: %v", row.RowIndex, row.Errors), + }) + } + return errors.Join(append(refused, failures...)...) +} + +// rowIsInvalid reports whether BigQuery gave the row the reason "invalid". +func rowIsInvalid(row bigquery.RowInsertionError) bool { + for _, rowErr := range row.Errors { + var byPointer *bigquery.Error + var byValue bigquery.Error + switch { + case errors.As(rowErr, &byPointer): + if byPointer.Reason == reasonInvalid { + return true + } + case errors.As(rowErr, &byValue): + if byValue.Reason == reasonInvalid { + return true + } + } + } + return false +} + // describePutError names the first rejected row: a PutMultiError's own message // says only how many rows failed. func describePutError(err error) error { diff --git a/module/services/accounts/code/pkg/auditstore/bigquerystore/rejection_test.go b/module/services/accounts/code/pkg/auditstore/bigquerystore/rejection_test.go new file mode 100644 index 000000000..a8dc4c66e --- /dev/null +++ b/module/services/accounts/code/pkg/auditstore/bigquerystore/rejection_test.go @@ -0,0 +1,197 @@ +package bigquerystore + +import ( + "context" + "errors" + "fmt" + "testing" + + "accounts/pkg/business" + + "cloud.google.com/go/bigquery" + "github.com/stretchr/testify/require" + "google.golang.org/api/googleapi" +) + +// What AppendAuditBatch says about a failure is what the relay acts on: it sets +// a row aside only on a business.PermanentRowRejection, and retries everything +// else. These pin which BigQuery answers become one. + +// scriptedInserter answers each request with answer(request), nil to accept it. +type scriptedInserter struct { + answer func(request []Row) error + requests [][]Row +} + +func (s *scriptedInserter) Put(_ context.Context, src any) error { + savers := src.([]bigquery.ValueSaver) + request := make([]Row, len(savers)) + for i, saver := range savers { + request[i] = saver.(Row) + } + s.requests = append(s.requests, request) + if s.answer == nil { + return nil + } + return s.answer(request) +} + +// insertErrors is BigQuery's insertAll answer for a request: row i of the +// request is refused with reasons[i] (the empty reason is a row it accepted). +func insertErrors(request []Row, reasons map[int]string) error { + var multi bigquery.PutMultiError + for i, row := range request { + reason, listed := reasons[i] + if !listed { + continue + } + multi = append(multi, bigquery.RowInsertionError{ + InsertID: row.InsertID, RowIndex: i, + Errors: bigquery.MultiError{&bigquery.Error{Reason: reason, Location: "occurred_at", Message: "cannot parse " + reason}}, + }) + } + return multi +} + +func refusedIDs(err error) []string { + var ids []string + for _, rejection := range business.PermanentRowRejections(err) { + ids = append(ids, rejection.EventID) + } + return ids +} + +func batchOf(t *testing.T, n int, class business.AuditRetentionClass) ([]business.AuditRecord, []string) { + t.Helper() + records := make([]business.AuditRecord, n) + ids := make([]string, n) + for i := range records { + records[i] = record(t, class, "") + ids[i] = records[i].Entry.ID + } + return records, ids +} + +func TestARowBigQueryRefusesAsInvalidIsARefusalOfThatRowAlone(t *testing.T) { + records, ids := batchOf(t, 6, business.RetentionSecurity) + events := &scriptedInserter{answer: func(request []Row) error { + // BigQuery refuses rows 1 and 4 as invalid and stops the rest of the request. + reasons := map[int]string{} + for i := range request { + reasons[i] = "stopped" + } + reasons[1], reasons[4] = "invalid", "invalid" + return insertErrors(request, reasons) + }} + store := &Store{events: events, details: &scriptedInserter{}} + + err := store.AppendAuditBatch(context.Background(), business.AuditBatch{DeploymentID: "deployment-1", Records: records}) + + require.Error(t, err) + require.Equal(t, []string{ids[1], ids[4]}, refusedIDs(err), "the rows refused as invalid, and not the rows that were stopped") + require.ErrorContains(t, err, "stopped", "the failure of the others stays in the error, for the log") +} + +func TestOnlyTheReasonInvalidIsAPermanentRefusal(t *testing.T) { + records, _ := batchOf(t, 3, business.RetentionSecurity) + for _, reason := range []string{"stopped", "backendError", "internalError", "rateLimitExceeded", "quotaExceeded", "timeout", "notFound", "accessDenied", "", "invalidQuery"} { + events := &scriptedInserter{answer: func(request []Row) error { + return insertErrors(request, map[int]string{0: reason, 1: reason, 2: reason}) + }} + store := &Store{events: events, details: &scriptedInserter{}} + + err := store.AppendAuditBatch(context.Background(), business.AuditBatch{DeploymentID: "deployment-1", Records: records}) + + require.Error(t, err, reason) + require.Empty(t, refusedIDs(err), "a row BigQuery gives the reason %q is not refused for its content", reason) + } +} + +func TestAnInvalidRowBesideARowThatFailedForAnotherReasonIsStillRefused(t *testing.T) { + records, ids := batchOf(t, 3, business.RetentionSecurity) + events := &scriptedInserter{answer: func(request []Row) error { + return insertErrors(request, map[int]string{0: "backendError", 1: "invalid", 2: "stopped"}) + }} + store := &Store{events: events, details: &scriptedInserter{}} + + err := store.AppendAuditBatch(context.Background(), business.AuditBatch{DeploymentID: "deployment-1", Records: records}) + + require.Equal(t, []string{ids[1]}, refusedIDs(err)) +} + +func TestAFailureOfTheRequestIsNeverARefusalOfARow(t *testing.T) { + records, _ := batchOf(t, 4, business.RetentionContent) + for name, failure := range map[string]error{ + "unavailable": &googleapi.Error{Code: 503, Message: "backendError"}, + "throttled": &googleapi.Error{Code: 429, Message: "rateLimitExceeded"}, + "quota": &googleapi.Error{Code: 403, Message: "quotaExceeded", Errors: []googleapi.ErrorItem{{Reason: "quotaExceeded"}}}, + "forbidden": &googleapi.Error{Code: 403, Message: "accessDenied"}, + "bad request": &googleapi.Error{Code: 400, Message: "invalid"}, + "not found": &googleapi.Error{Code: 404, Message: "notFound"}, + "deadline": context.DeadlineExceeded, + "transport": errors.New("write tcp 10.0.0.1:443: connection reset by peer"), + "unknown": errors.New("an answer of a kind this store does not know"), + "empty multi": bigquery.PutMultiError{}, + } { + for _, table := range []string{"events", "details"} { + events, details := &scriptedInserter{}, &scriptedInserter{} + if table == "events" { + events.answer = func([]Row) error { return failure } + } else { + details.answer = func([]Row) error { return failure } + } + store := &Store{events: events, details: details} + + err := store.AppendAuditBatch(context.Background(), business.AuditBatch{DeploymentID: "deployment-1", Records: records}) + + require.Error(t, err, name) + require.Empty(t, refusedIDs(err), "%s, %s table: a failure of the request says nothing about a row", name, table) + if name != "empty multi" { + require.ErrorIs(t, err, failure, "%s: the failure is returned as it came", name) + } + } + } +} + +func TestARefusalInThePutErrorSurvivesWrapping(t *testing.T) { + records, ids := batchOf(t, 2, business.RetentionSecurity) + events := &scriptedInserter{answer: func(request []Row) error { + return fmt.Errorf("transport wrapper: %w", insertErrors(request, map[int]string{1: "invalid"})) + }} + store := &Store{events: events, details: &scriptedInserter{}} + + err := store.AppendAuditBatch(context.Background(), business.AuditBatch{DeploymentID: "deployment-1", Records: records}) + + require.Equal(t, []string{ids[1]}, refusedIDs(err)) +} + +func TestARowRefusedInTheDetailsTableIsARefusalOfItsEvent(t *testing.T) { + records, ids := batchOf(t, 3, business.RetentionContent) + details := &scriptedInserter{answer: func(request []Row) error { + return insertErrors(request, map[int]string{2: "invalid"}) + }} + events := &scriptedInserter{} + store := &Store{events: events, details: details} + + err := store.AppendAuditBatch(context.Background(), business.AuditBatch{DeploymentID: "deployment-1", Records: records}) + + require.Equal(t, []string{ids[2]}, refusedIDs(err)) + require.ErrorContains(t, err, DetailsTable) +} + +func TestEveryRowThatCannotFitARequestIsARefusalAndNothingIsSent(t *testing.T) { + events, details := &scriptedInserter{}, &scriptedInserter{} + store := &Store{events: events, details: details} + small, hugeOne, hugeTwo := largeRecord(t, 1<<10), largeRecord(t, maxRequestBytes), largeRecord(t, maxRequestBytes) + + err := store.AppendAuditBatch(context.Background(), business.AuditBatch{ + DeploymentID: "deployment-1", Records: []business.AuditRecord{hugeOne, small, hugeTwo}, + }) + + require.Equal(t, []string{hugeOne.Entry.ID, hugeTwo.Entry.ID}, refusedIDs(err), "every such row, not only the first") + require.ErrorIs(t, err, ErrRowTooLarge) + var tooLarge *RowTooLargeError + require.ErrorAs(t, err, &tooLarge) + require.Empty(t, events.requests) + require.Empty(t, details.requests) +} diff --git a/module/services/accounts/code/pkg/auditstore/clickhousestore/clickhousestore.go b/module/services/accounts/code/pkg/auditstore/clickhousestore/clickhousestore.go index 43872a52c..aba6524e5 100644 --- a/module/services/accounts/code/pkg/auditstore/clickhousestore/clickhousestore.go +++ b/module/services/accounts/code/pkg/auditstore/clickhousestore/clickhousestore.go @@ -21,7 +21,8 @@ // // Inserts are batched: one insert per table per relay batch, never a row at a // time (a single-row insert creates a data part per insert and ends in -// too-many-parts errors). On a cluster an insert returns only once a majority +// too-many-parts errors). The one exception is a block the server refuses for a +// row's content: it is split to find the row (rejection.go). On a cluster an insert returns only once a majority // of the replicas hold it, because the relay deletes its queue rows on the // acknowledgement. Reads run in ClickHouse wherever ClickHouse computes // exactly what the Postgres reads compute, and in the service (auditeval) @@ -33,6 +34,7 @@ import ( "errors" "fmt" "regexp" + "slices" "sort" "strconv" "strings" @@ -505,39 +507,128 @@ func DetailRow(deploymentID string, record business.AuditRecord) []any { // written atomically; a failure between the two leaves the events without // their details until the relay delivers the batch again, and every read // returns each event once by event id. +// +// What a failure says about the batch is what the relay acts on, so it is +// classified here (rejection.go): a row ClickHouse refuses for its own content is +// returned as a business.PermanentRowRejection naming its event, found by +// splitting the refused block down to single rows; every other failure is +// returned as it came and the relay retries the batch whole. The rows around a +// refused one are written while it is found, so the relay's retry of them is a +// redelivery, which every read discards. func (s *Store) AppendAuditBatch(ctx context.Context, batch business.AuditBatch) error { events, details := BatchRows(batch) - if err := s.insert(ctx, EventsTable, EventsColumns(), events); err != nil { - return err + refused, err := s.insert(ctx, EventsTable, EventsColumns(), events) + if err != nil { + return failure(refused, err) + } + // An event whose events row was refused is not given a details row: the relay + // sets it aside, and its details would outlive the event they belong to. + details = withoutRefused(details, refused) + more, err := s.insert(ctx, DetailsTable, DetailsColumns(), details) + return failure(append(refused, more...), err) +} + +// failure is what an append returns: the refusals it found beside the failure +// that stopped it, either of which may be absent. +func failure(refused []error, err error) error { + return errors.Join(append(refused, err)...) +} + +// withoutRefused is rows less those of the events refused. Every row's first +// value is its event id. +func withoutRefused(rows [][]any, refused []error) [][]any { + if len(refused) == 0 { + return rows + } + gone := map[string]bool{} + for _, rejection := range business.PermanentRowRejections(errors.Join(refused...)) { + gone[rejection.EventID] = true + } + var kept [][]any + for _, row := range rows { + if id, _ := row[0].(string); !gone[id] { + kept = append(kept, row) + } } - return s.insert(ctx, DetailsTable, DetailsColumns(), details) + return kept } -func (s *Store) insert(ctx context.Context, table string, columns []Column, rows [][]any) error { +// insert writes rows to table. It returns the rows ClickHouse refused for their +// own content, as business.PermanentRowRejection, and the failure that stopped +// it when the rest could not be written; the rows it did not refuse are +// written when it returns no failure. +func (s *Store) insert(ctx context.Context, table string, columns []Column, rows [][]any) ([]error, error) { if len(rows) == 0 { - return nil + return nil, nil } if len(s.writeSettings) > 0 { ctx = clickhouse.Context(ctx, clickhouse.WithSettings(s.writeSettings)) } + return s.isolate(ctx, table, columns, rows) +} + +// isolate sends rows as one block. When the block is refused it finds the rows +// to blame: a row the driver cannot encode names itself (the block is sent +// again without it); a block the server refuses with a code that says a row's +// content is malformed (rowDefectCodes) is split in halves, and so on down to +// single rows, and a single row refused that way is the one. Any other failure +// — a network error, a quorum or replica failure, a timeout, a code not in the +// list — stops the search and is returned, so a store that is merely failing is +// never searched for rows to blame. +func (s *Store) isolate(ctx context.Context, table string, columns []Column, rows [][]any) ([]error, error) { + var refused []error + for len(rows) > 0 { + bad, err := s.send(ctx, table, columns, rows) + switch { + case err == nil: + return refused, nil + case bad >= 0: + refused = append(refused, refusal(table, rows[bad], err)) + rows = slices.Delete(slices.Clone(rows), bad, bad+1) + continue + case !isRowDefect(err): + return refused, err + case len(rows) == 1: + return append(refused, refusal(table, rows[0], err)), nil + } + middle := len(rows) / 2 + for _, half := range [][][]any{rows[:middle], rows[middle:]} { + found, err := s.isolate(ctx, table, columns, half) + refused = append(refused, found...) + if err != nil { + return refused, err + } + } + return refused, nil + } + return refused, nil +} + +// send writes rows as one block. When the driver cannot encode a row it returns +// that row's index, with nothing sent; otherwise the index is -1. +func (s *Store) send(ctx context.Context, table string, columns []Column, rows [][]any) (int, error) { batch, err := s.conn.PrepareBatch(ctx, insertStatement(table, columns)) if err != nil { - return fmt.Errorf("clickhouse audit store: append to %s: %w", table, err) + return -1, fmt.Errorf("clickhouse audit store: append to %s: %w", table, err) } defer func() { if !batch.IsSent() { _ = batch.Abort() } }() - for _, row := range rows { + for i, row := range rows { if err := batch.Append(row...); err != nil { - return fmt.Errorf("clickhouse audit store: append to %s: event %v: %w", table, row[0], err) + err = fmt.Errorf("clickhouse audit store: append to %s: event %v: %w", table, row[0], err) + if isUnencodableRow(err) { + return i, err + } + return -1, err } } if err := batch.Send(); err != nil { - return fmt.Errorf("clickhouse audit store: append to %s: %w", table, err) + return -1, fmt.Errorf("clickhouse audit store: append to %s: %w", table, err) } - return nil + return -1, nil } // queryRow runs a statement expected to return at most one row and scans it. diff --git a/module/services/accounts/code/pkg/auditstore/clickhousestore/rejection.go b/module/services/accounts/code/pkg/auditstore/clickhousestore/rejection.go new file mode 100644 index 000000000..1dcccd855 --- /dev/null +++ b/module/services/accounts/code/pkg/auditstore/clickhousestore/rejection.go @@ -0,0 +1,81 @@ +package clickhousestore + +import ( + "errors" + "fmt" + + "accounts/pkg/business" + + "github.com/ClickHouse/clickhouse-go/v2" + "github.com/ClickHouse/clickhouse-go/v2/lib/proto" +) + +// rowDefectCodes are the server exception codes that mean the content of a row +// is malformed for its column, so that a row refused alone with one of them is +// refused every time. Each name is as ClickHouse's errorCodeToName gives it. +// +// Parsing: a value the server cannot read as the column's type. +// +// 6 CANNOT_PARSE_TEXT 26 CANNOT_PARSE_QUOTED_STRING +// 27 CANNOT_PARSE_INPUT_ASSERTION_FAILED 38 CANNOT_PARSE_DATE +// 41 CANNOT_PARSE_DATETIME 72 CANNOT_PARSE_NUMBER +// 117 INCORRECT_DATA +// +// Type and range: a value that is not, or cannot become, one the column holds. +// +// 53 TYPE_MISMATCH 70 CANNOT_CONVERT_TYPE +// 321 VALUE_IS_OUT_OF_RANGE_OF_DATA_TYPE +// +// Size: a value over what the column holds. +// +// 131 TOO_LARGE_STRING_SIZE +// +// Constraint: a row that a CHECK constraint the deployment put on the table +// refuses. +// +// 469 VIOLATED_CONSTRAINT +// +// Every other code is retryable, and never makes a row refused, whatever the +// number of times it repeats: network and socket errors (210, 209), timeouts +// (159), memory (241), a read-only server or table (164, 242), no space (243), +// too many parts or queries (252, 202), the quorum and replica codes (285, 286, +// 319), Keeper (999), a missing table (60), access and authentication (497, 516), +// and the limits on a whole block (158, 396), which are the block's, not a row's. +// A code that is not known here is retryable too: the cost of a wrong guess in +// that direction is a queue that waits, never an event set aside. +var rowDefectCodes = map[int32]string{ + 6: "CANNOT_PARSE_TEXT", 26: "CANNOT_PARSE_QUOTED_STRING", 27: "CANNOT_PARSE_INPUT_ASSERTION_FAILED", + 38: "CANNOT_PARSE_DATE", 41: "CANNOT_PARSE_DATETIME", 72: "CANNOT_PARSE_NUMBER", 117: "INCORRECT_DATA", + 53: "TYPE_MISMATCH", 70: "CANNOT_CONVERT_TYPE", 321: "VALUE_IS_OUT_OF_RANGE_OF_DATA_TYPE", + 131: "TOO_LARGE_STRING_SIZE", + 469: "VIOLATED_CONSTRAINT", +} + +// isRowDefect reports whether err is a server exception whose code says a row's +// content is malformed. +func isRowDefect(err error) bool { + var exception *clickhouse.Exception + if !errors.As(err, &exception) { + return false + } + _, defect := rowDefectCodes[exception.Code] + return defect +} + +// isUnencodableRow reports whether err is the driver's refusal to encode a value +// of the row being appended as its column's type (clickhouse-go's AppendRow +// error). It is raised before anything is sent, and names the row because the +// store appends rows one at a time. A wrong number of values (the driver's +// Append error) is the store's own bug, not a row's content, and is not this. +func isUnencodableRow(err error) bool { + var block *proto.BlockError + return errors.As(err, &block) && block.Op == "AppendRow" +} + +// refusal is the refusal of the row, whose first value is its event id. +func refusal(table string, row []any, cause error) error { + return &business.PermanentRowRejection{ + EventID: fmt.Sprint(row[0]), + Cause: fmt.Errorf("clickhouse audit store: %s refused the row: %w", table, cause), + } +} diff --git a/module/services/accounts/code/pkg/auditstore/clickhousestore/rejection_server_test.go b/module/services/accounts/code/pkg/auditstore/clickhousestore/rejection_server_test.go new file mode 100644 index 000000000..9c53f7c2b --- /dev/null +++ b/module/services/accounts/code/pkg/auditstore/clickhousestore/rejection_server_test.go @@ -0,0 +1,55 @@ +package clickhousestore + +import ( + "context" + "testing" + "time" + + "accounts/pkg/business" + + "github.com/ClickHouse/clickhouse-go/v2" + "github.com/google/uuid" + "github.com/stretchr/testify/require" +) + +// A row a real server refuses for its content, and a failure of another kind, +// against the exception codes and types the driver really returns. Skipped +// without a server; see server_test.go. + +func TestServerIsolatesARowTheTableRefusesAndNothingElse(t *testing.T) { + ctx := context.Background() + conn, database := serverDatabase(t) + store := serverStore(t, conn, database) + require.NoError(t, store.Ensure(ctx)) + // A deployment's own CHECK constraint: the one way this schema has a row's + // content refused by the server. + require.NoError(t, conn.Exec(ctx, "ALTER TABLE "+EventsTable+" ADD CONSTRAINT no_forbidden CHECK resource_id != 'forbidden'")) + + org, actor := uuid.NewString(), uuid.NewString() + at := time.Now().UTC().Truncate(time.Microsecond).Add(-time.Minute) + var records []business.AuditRecord + for _, resourceID := range []string{"a", "b", "forbidden", "c", "d", "forbidden", "e"} { + records = append(records, newRecord(t, org, actor, business.EventAuthLogin, resourceID, at, business.RetentionContent, map[string]any{"k": "v"})) + } + + err := store.AppendAuditBatch(ctx, business.AuditBatch{ID: uuid.NewString(), DeploymentID: testDeployment, ComposedAt: time.Now(), Records: records}) + + require.ElementsMatch(t, []string{records[2].Entry.ID, records[5].Entry.ID}, refusedIDs(err)) + var exception *clickhouse.Exception + require.ErrorAs(t, err, &exception) + require.EqualValues(t, 469, exception.Code, "the code the server really gives a refused CHECK") + var events, details uint64 + require.NoError(t, store.queryRow(ctx, "SELECT count() FROM "+EventsTable, nil, &events)) + require.NoError(t, store.queryRow(ctx, "SELECT count() FROM "+DetailsTable, nil, &details)) + require.EqualValues(t, 5, events, "the rows around the refused ones are written") + require.EqualValues(t, 5, details) + + // A failure that is not about a row: a table that is gone refuses nothing. + require.NoError(t, conn.Exec(ctx, "DROP TABLE "+DetailsTable)) + err = store.AppendAuditBatch(ctx, business.AuditBatch{ID: uuid.NewString(), DeploymentID: testDeployment, ComposedAt: time.Now(), + Records: []business.AuditRecord{records[0], records[1]}}) + require.Error(t, err) + require.Empty(t, refusedIDs(err)) + require.ErrorAs(t, err, &exception) + require.EqualValues(t, 60, exception.Code, "UNKNOWN_TABLE, as the server gives it") +} diff --git a/module/services/accounts/code/pkg/auditstore/clickhousestore/rejection_test.go b/module/services/accounts/code/pkg/auditstore/clickhousestore/rejection_test.go new file mode 100644 index 000000000..e3ef4315d --- /dev/null +++ b/module/services/accounts/code/pkg/auditstore/clickhousestore/rejection_test.go @@ -0,0 +1,291 @@ +package clickhousestore + +import ( + "context" + "errors" + "fmt" + "slices" + "sort" + "testing" + + "accounts/pkg/business" + + "github.com/ClickHouse/clickhouse-go/v2" + "github.com/ClickHouse/clickhouse-go/v2/lib/driver" + "github.com/ClickHouse/clickhouse-go/v2/lib/proto" + "github.com/stretchr/testify/require" +) + +// What AppendAuditBatch says about a failure is what the relay acts on: it sets +// a row aside only on a business.PermanentRowRejection, and retries everything +// else. These pin which ClickHouse answers become one, against a connection that +// answers as a server does: a block is refused whole, with a code, and says +// nothing of which row is to blame. + +// blameConn is a server that refuses any block holding a row for which +// answer(table, rowID) returns an error, naming nothing of the row. +type blameConn struct { + Conn + answer func(table, eventID string) error + // stored is the event ids each table accepted, and sends counts the blocks + // the server was sent. + stored map[string][]string + sends int +} + +func (c *blameConn) PrepareBatch(_ context.Context, query string, _ ...driver.PrepareBatchOption) (driver.Batch, error) { + table := EventsTable + if query == insertStatement(DetailsTable, DetailsColumns()) { + table = DetailsTable + } + return &blameBatch{conn: c, table: table}, nil +} + +func (c *blameConn) ids(table string) []string { + ids := slices.Clone(c.stored[table]) + sort.Strings(ids) + return slices.Compact(ids) +} + +type blameBatch struct { + driver.Batch + conn *blameConn + table string + rows [][]any + sent bool + // unencodable is the event ids Append refuses, as the driver refuses a value + // it cannot convert to the column's type. + unencodable map[string]bool +} + +func (b *blameBatch) Append(values ...any) error { + if b.unencodable[values[0].(string)] { + return &proto.BlockError{Op: "AppendRow", ColumnName: "schema_version", Err: errors.New("converting string to Int64 is unsupported")} + } + b.rows = append(b.rows, values) + return nil +} + +func (b *blameBatch) Send() error { + b.conn.sends++ + for _, row := range b.rows { + if b.conn.answer == nil { + continue + } + if err := b.conn.answer(b.table, row[0].(string)); err != nil { + return err + } + } + if b.conn.stored == nil { + b.conn.stored = map[string][]string{} + } + for _, row := range b.rows { + b.conn.stored[b.table] = append(b.conn.stored[b.table], row[0].(string)) + } + b.sent = true + return nil +} + +func (b *blameBatch) Abort() error { b.sent = true; return nil } +func (b *blameBatch) IsSent() bool { return b.sent } + +func exception(code int32) error { + return &clickhouse.Exception{Code: code, Name: "DB::Exception", Message: fmt.Sprintf("code %d", code)} +} + +func recordsOf(t *testing.T, n int, class business.AuditRetentionClass) ([]business.AuditRecord, []string) { + t.Helper() + records := make([]business.AuditRecord, n) + ids := make([]string, n) + for i := range records { + records[i] = record(t, class, "") + ids[i] = records[i].Entry.ID + } + return records, ids +} + +func refusedIDs(err error) []string { + var ids []string + for _, rejection := range business.PermanentRowRejections(err) { + ids = append(ids, rejection.EventID) + } + return ids +} + +func TestEveryRowDefectCodeIsAKnownErrorAndFindsTheRowToBlame(t *testing.T) { + for code, name := range rowDefectCodes { + records, ids := recordsOf(t, 9, business.RetentionSecurity) + conn := &blameConn{answer: func(_, id string) error { + if id == ids[4] { + return exception(code) + } + return nil + }} + store, err := New(conn, validConfig()) + require.NoError(t, err) + + err = store.AppendAuditBatch(context.Background(), business.AuditBatch{DeploymentID: "deployment-1", Records: records}) + + require.Equal(t, []string{ids[4]}, refusedIDs(err), "%d %s", code, name) + want := slices.Clone(ids) + want = slices.Delete(want, 4, 5) + sort.Strings(want) + require.Equal(t, want, conn.ids(EventsTable), "%d %s: every other row is written while the row to blame is found", code, name) + require.Less(t, conn.sends, 2*len(records), "%d %s: the search is a bisection, not a write per row", code, name) + } +} + +func TestSeveralRowsRefusedForTheirContentAreAllFound(t *testing.T) { + records, ids := recordsOf(t, 20, business.RetentionContent) + bad := []string{ids[0], ids[7], ids[8], ids[19]} + conn := &blameConn{answer: func(table, id string) error { + if slices.Contains(bad, id) { + return exception(41) + } + return nil + }} + store, err := New(conn, validConfig()) + require.NoError(t, err) + + err = store.AppendAuditBatch(context.Background(), business.AuditBatch{DeploymentID: "deployment-1", Records: records}) + + require.ElementsMatch(t, bad, refusedIDs(err)) + require.Len(t, conn.ids(EventsTable), 16) + require.Len(t, conn.ids(DetailsTable), 16, "the details of the rows that were not refused are written") + for _, id := range bad { + require.NotContains(t, conn.ids(DetailsTable), id, "an event refused in the events table is not given details") + } +} + +func TestACodeThatIsNotARowDefectNeverSearchesAndNeverRefusesARow(t *testing.T) { + for name, failure := range map[string]error{ + "network": exception(210), + "socket timeout": exception(209), + "timeout exceeded": exception(159), + "memory": exception(241), + "readonly": exception(164), + "table readonly": exception(242), + "no space": exception(243), + "too many parts": exception(252), + "too many queries": exception(202), + "too few live replicas": exception(285), + "quorum previous write": exception(286), + "unknown insert status": exception(319), + "keeper": exception(999), + "unknown table": exception(60), + "access denied": exception(497), + "authentication": exception(516), + "block row limit": exception(158), + "block byte limit": exception(396), + "a code nobody listed": exception(9999), + "deadline": context.DeadlineExceeded, + "closed connection": errors.New("write tcp 10.0.0.2:9000: broken pipe"), + "driver batch error": clickhouse.ErrBatchInvalid, + "wrong value count": &proto.BlockError{Op: "Append", Err: errors.New("expected 17 arguments, got 5")}, + } { + for _, size := range []int{1, 12} { + records, _ := recordsOf(t, size, business.RetentionSecurity) + conn := &blameConn{answer: func(string, string) error { return failure }} + store, err := New(conn, validConfig()) + require.NoError(t, err) + + err = store.AppendAuditBatch(context.Background(), business.AuditBatch{DeploymentID: "deployment-1", Records: records}) + + require.Error(t, err, name) + require.Empty(t, refusedIDs(err), "%s, batch of %d: a failure that is not about a row's content refuses no row", name, size) + require.Equal(t, 1, conn.sends, "%s, batch of %d: and nothing is split to look for one", name, size) + require.ErrorIs(t, err, failure) + } + } +} + +// A single row that fails with a code that is not a defect is not refused even +// when a defect failed the block it was in: the search stops at the first +// failure that says nothing of a row. +func TestASearchStopsAtAFailureThatIsNotARowDefect(t *testing.T) { + records, ids := recordsOf(t, 8, business.RetentionSecurity) + conn := &blameConn{answer: func(_, id string) error { + switch id { + case ids[1]: + return exception(41) + case ids[6]: + return exception(285) + } + return nil + }} + store, err := New(conn, validConfig()) + require.NoError(t, err) + + err = store.AppendAuditBatch(context.Background(), business.AuditBatch{DeploymentID: "deployment-1", Records: records}) + + require.Equal(t, []string{ids[1]}, refusedIDs(err), "the row refused for its content before the failure is still named") + require.ErrorContains(t, err, "code 285", "and the failure that stopped the search is returned for the relay to retry") +} + +// A block that is refused as a whole for a reason of its own — its size — is +// not blamed on a row when every half of it is accepted. +func TestABlockRefusedWithADefectCodeWhoseRowsAreAllAcceptedAloneRefusesNothing(t *testing.T) { + records, ids := recordsOf(t, 10, business.RetentionSecurity) + conn := &blameConn{} + store, err := New(&sizeLimitedConn{blameConn: conn, limit: 3}, validConfig()) + require.NoError(t, err) + + require.NoError(t, store.AppendAuditBatch(context.Background(), business.AuditBatch{DeploymentID: "deployment-1", Records: records})) + + sort.Strings(ids) + require.Equal(t, ids, conn.ids(EventsTable), "every row is written") +} + +// sizeLimitedConn refuses a block of more than limit rows with TOO_LARGE_STRING_SIZE. +type sizeLimitedConn struct { + *blameConn + limit int +} + +func (c *sizeLimitedConn) PrepareBatch(ctx context.Context, query string, opts ...driver.PrepareBatchOption) (driver.Batch, error) { + inner, err := c.blameConn.PrepareBatch(ctx, query, opts...) + if err != nil { + return nil, err + } + return &limitedBatch{blameBatch: inner.(*blameBatch), limit: c.limit}, nil +} + +type limitedBatch struct { + *blameBatch + limit int +} + +func (b *limitedBatch) Send() error { + if len(b.rows) > b.limit { + b.conn.sends++ + return exception(131) + } + return b.blameBatch.Send() +} + +func TestARowTheDriverCannotEncodeIsRefusedBeforeAnythingIsSent(t *testing.T) { + records, ids := recordsOf(t, 6, business.RetentionContent) + conn := &encodingConn{blameConn: &blameConn{}, unencodable: map[string]bool{ids[2]: true, ids[3]: true}} + store, err := New(conn, validConfig()) + require.NoError(t, err) + + err = store.AppendAuditBatch(context.Background(), business.AuditBatch{DeploymentID: "deployment-1", Records: records}) + + require.ElementsMatch(t, []string{ids[2], ids[3]}, refusedIDs(err), "the driver's error names the row being appended") + require.Len(t, conn.ids(EventsTable), 4, "the others are sent, in a block without them") + require.Len(t, conn.ids(DetailsTable), 4) +} + +type encodingConn struct { + *blameConn + unencodable map[string]bool +} + +func (c *encodingConn) PrepareBatch(ctx context.Context, query string, opts ...driver.PrepareBatchOption) (driver.Batch, error) { + inner, err := c.blameConn.PrepareBatch(ctx, query, opts...) + if err != nil { + return nil, err + } + inner.(*blameBatch).unencodable = c.unencodable + return inner, nil +} diff --git a/module/services/accounts/code/pkg/business/audit_row_rejection.go b/module/services/accounts/code/pkg/business/audit_row_rejection.go new file mode 100644 index 000000000..db475e5e5 --- /dev/null +++ b/module/services/accounts/code/pkg/business/audit_row_rejection.go @@ -0,0 +1,62 @@ +package business + +import ( + "fmt" +) + +// PermanentRowRejection is the error a store of record returns for a row it +// refused for the row's own content and will refuse every time it is sent: a +// value outside a column's range, a row over the size a request carries, text +// the store cannot hold. It names the event so the relay can set that one row +// aside and deliver the rest. +// +// It is the only evidence the relay accepts that a row is undeliverable. A +// store returns it only when the store itself attributed the refusal to the +// row; a transport error, a server error, a throttle, a quota, a timeout, a +// replica or quorum failure, and any error the store cannot classify are not +// this error, and the relay retries them with backoff and sets nothing aside. +type PermanentRowRejection struct { + // EventID is the event whose row was refused. + EventID string + // Cause is what the store reported about the row. + Cause error +} + +func (e *PermanentRowRejection) Error() string { + if e.Cause == nil { + return fmt.Sprintf("audit store refused the row of event %s for good", e.EventID) + } + return fmt.Sprintf("audit store refused the row of event %s for good: %v", e.EventID, e.Cause) +} + +// Unwrap returns what the store reported. +func (e *PermanentRowRejection) Unwrap() error { return e.Cause } + +// PermanentRowRejections returns every row err says the store refused for good, +// one per event id, in the order err names them. err may be one rejection, +// wrapped, or several joined with errors.Join, with or without a retryable +// cause beside them. It returns nil when err names none — which is the case for +// every error that is not a refusal of a row. +func PermanentRowRejections(err error) []*PermanentRowRejection { + var found []*PermanentRowRejection + seen := map[string]bool{} + var walk func(error) + walk = func(err error) { + switch e := err.(type) { + case nil: + case *PermanentRowRejection: + if e != nil && e.EventID != "" && !seen[e.EventID] { + seen[e.EventID] = true + found = append(found, e) + } + case interface{ Unwrap() []error }: + for _, inner := range e.Unwrap() { + walk(inner) + } + case interface{ Unwrap() error }: + walk(e.Unwrap()) + } + } + walk(err) + return found +} diff --git a/module/services/accounts/code/pkg/business/audit_row_rejection_test.go b/module/services/accounts/code/pkg/business/audit_row_rejection_test.go new file mode 100644 index 000000000..9b209356b --- /dev/null +++ b/module/services/accounts/code/pkg/business/audit_row_rejection_test.go @@ -0,0 +1,27 @@ +package business + +import ( + "errors" + "fmt" + "testing" + + "github.com/stretchr/testify/require" +) + +func TestPermanentRowRejectionsReadsWhatTheStoreAttributed(t *testing.T) { + one := &PermanentRowRejection{EventID: "event-1", Cause: errors.New("invalid")} + two := &PermanentRowRejection{EventID: "event-2", Cause: errors.New("too large")} + retryable := errors.New("429") + + require.Nil(t, PermanentRowRejections(nil)) + require.Nil(t, PermanentRowRejections(retryable)) + require.Nil(t, PermanentRowRejections(&PermanentRowRejection{Cause: errors.New("no event")}), "a refusal that names no event names no row") + + require.Equal(t, []*PermanentRowRejection{one}, PermanentRowRejections(one)) + require.Equal(t, []*PermanentRowRejection{one}, PermanentRowRejections(fmt.Errorf("append: %w", one))) + require.Equal(t, []*PermanentRowRejection{one, two}, PermanentRowRejections(errors.Join(one, retryable, fmt.Errorf("again: %w", two), one)), + "several, beside a retryable cause, each once") + require.ErrorContains(t, one, "event-1") + require.ErrorContains(t, one, "invalid") + require.ErrorIs(t, one, one.Cause) +} diff --git a/module/services/accounts/code/pkg/business/audit_store.go b/module/services/accounts/code/pkg/business/audit_store.go index de373790b..c7167bc96 100644 --- a/module/services/accounts/code/pkg/business/audit_store.go +++ b/module/services/accounts/code/pkg/business/audit_store.go @@ -80,6 +80,16 @@ type AuditStoreWriter interface { // id and may hold an event more than once after a redelivery; every read of // the store returns each event once by event id, exactly as every reader of // the archive does. + // + // A failure is retryable unless it says otherwise. A store says otherwise + // only for a row it itself refused for that row's own content, one it will + // refuse every time, by returning a *PermanentRowRejection naming the event + // (alone, wrapped, or joined with others and with the failure of the rest). + // The relay sets exactly those rows aside and appends the others again, so + // the store makes no claim about the other rows of the batch. A transport + // error, a server error, a throttle, a quota, a timeout, a quorum or replica + // failure, and any error the store cannot classify are not a rejection: the + // relay retries them and sets nothing aside. AppendAuditBatch(ctx context.Context, batch AuditBatch) error } From f323ab914bb2979c70a481079aef3a3b8469abaa Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Tue, 6 Oct 2026 15:08:11 -0400 Subject: [PATCH 36/69] fix(audit): quarantine only the rows a store reports it refused for good Four defects in how the relay set rows aside. A batch size of one stalled the relay: the single-row path returned before any quarantine decision, so a refused first row held every event behind it, and a lone leftover row in a larger batch waited for another event. Every batch size now takes the same path. Truncating the error text at byte 1000 could cut a multibyte character, and a NUL byte or invalid UTF-8 has the same effect: PostgreSQL rejects the text, the quarantine transaction rolls back with the delivered rows, and it repeats on every pass and every restart. The reason is now valid text with no NUL, cut between characters; the queue enforces the same bound itself. The quarantine decision came from timing, not from the store: a flapping store quarantined good rows and dense bad rows stalled the queue. The relay now sets a row aside only on a PermanentRowRejection naming it, writes the rest again, and treats every other failure as retryable. The witness write and the probe limit are gone: nothing is inferred from what else was accepted. Every unserializable row was quarantined with the first row's error. Each now keeps its own, and a lone unserializable row is no longer held back for lack of a composable neighbour. Such a row cannot be archived (the archive line carries the hash of its canonical details), so the quarantine holds its only copy. A reason now starts with its class, warehouse_rejected_archived or unserializable_not_archived, so a responder can tell the two apart. Co-Authored-By: Claude Sonnet 5.5 --- module/FEATURES.md | 2 +- module/MEASUREMENT_RUNBOOKS.md | 59 +++- module/services/accounts/AGENTS.md | 15 +- .../accounts/code/pkg/business/audit_relay.go | 258 ++++++++---------- .../code/pkg/business/audit_relay_metrics.go | 2 +- .../business/audit_relay_quarantine_test.go | 120 ++++---- .../business/audit_relay_rejection_test.go | 251 +++++++++++++++++ .../code/pkg/business/audit_relay_test.go | 26 +- .../code/pkg/infra/postgres_audit_queue.go | 6 +- .../pkg/infra/postgres_audit_relay_test.go | 132 +++++++++ 10 files changed, 652 insertions(+), 219 deletions(-) create mode 100644 module/services/accounts/code/pkg/business/audit_relay_rejection_test.go create mode 100644 module/services/accounts/code/pkg/infra/postgres_audit_relay_test.go diff --git a/module/FEATURES.md b/module/FEATURES.md index e4e7a1ffa..de965ea79 100644 --- a/module/FEATURES.md +++ b/module/FEATURES.md @@ -398,7 +398,7 @@ see `JOBS.md` for the exact boundary and sequencing. | Export (JSON/CSV) | ✅ | `audit_export.go`; under a warehouse store of record one export holds at most 32 MiB of event text, past which it fails with `ErrAuditExportTooLarge` (`ResourceExhausted`): narrow it by actor or event type | | Impersonation tracking | ✅ | Records both real actor + viewed-as user | | Replay / event sourcing| ❌ | Audit log is read-only history; not used to reconstruct state | -| Warehouse store of record | 🟡 | `AUDIT_SINK=bigquery` or `AUDIT_SINK=clickhouse` (ADR 0009): each record commits to a Postgres queue in the mutation's transaction, then a relay delivers it to the warehouse and a locked GCS or S3 archive. Reads are scoped to the organization and deployment and deduplicated by event id; BigQuery uses the Storage Read API without query jobs, and ClickHouse uses SQL with service evaluation where needed. A BigQuery read holds a bounded amount in memory whatever the history: a window that needs more than its byte budget is read again as halves. A row the warehouse refuses on its own is set aside, whole, in `audit_event_quarantine` instead of holding every organization's events behind it (`saas.audit_queue.quarantined`; replaying quarantined rows is not built yet). The shared parity suite compares both with Postgres. `audit-history-copy` verifies a switching deployment's copied history before a confirmed monthly-partition drop. No Azure Blob archive writer exists yet. | +| Warehouse store of record | 🟡 | `AUDIT_SINK=bigquery` or `AUDIT_SINK=clickhouse` (ADR 0009): each record commits to a Postgres queue in the mutation's transaction, then a relay delivers it to the warehouse and a locked GCS or S3 archive. Reads are scoped to the organization and deployment and deduplicated by event id; BigQuery uses the Storage Read API without query jobs, and ClickHouse uses SQL with service evaluation where needed. A BigQuery read holds a bounded amount in memory whatever the history: a window that needs more than its byte budget is read again as halves. A row the warehouse itself reports it refuses for good (never one that merely failed while the warehouse was flaky) is set aside, whole, in `audit_event_quarantine` instead of holding every organization's events behind it (`saas.audit_queue.quarantined`; replaying quarantined rows is not built yet). The shared parity suite compares both with Postgres. `audit-history-copy` verifies a switching deployment's copied history before a confirmed monthly-partition drop. No Azure Blob archive writer exists yet. | ### Admin panel diff --git a/module/MEASUREMENT_RUNBOOKS.md b/module/MEASUREMENT_RUNBOOKS.md index bd48fdef2..725e1be92 100644 --- a/module/MEASUREMENT_RUNBOOKS.md +++ b/module/MEASUREMENT_RUNBOOKS.md @@ -60,21 +60,58 @@ readers deduplicate by event id. ### A row the warehouse refuses (`saas.audit_queue.quarantined`) One row the warehouse rejects for its own content must not hold every other -organization's events behind it. When a batch write fails, the relay splits the -batch and retries the halves, down to single rows. A single row is set aside only -when the warehouse refused it alone, accepted other rows afterwards, and refused -it again on a second try; during an outage every write fails and nothing is set -aside. The archive is written once, whole, before any of this, so a set-aside row -is already in the archive. +organization's events behind it, and a warehouse that is merely failing must not +cost a good row its place. So the relay sets a row aside only on what the +warehouse says about that row: a permanent refusal the adapter attributes to the +event. Every other failure is retried with backoff and sets nothing aside: a +transport error, a server error, a throttle (429), a quota, a timeout, a +ClickHouse quorum or replica failure, and any error the adapter does not +recognize. A flapping or unreachable warehouse grows `saas.audit_queue.depth` and +`oldest_age`, never the quarantine. The relay draws no conclusion from what else +the warehouse accepted or from how often a row failed. + +What the adapters report as a permanent refusal: + +- **BigQuery**: a row that `insertAll` lists with the reason `invalid` (a value + that does not fit its column), and a row too large for any request, which is + found before anything is sent. A row listed as `stopped` (fine, but sent beside + an invalid one), and every failure of the request as a whole, are retried. +- **ClickHouse**: a value the driver cannot encode as its column's type, named + before anything is sent, and a block the server refuses with an exception code + that means a row's content is malformed (the parsing codes 6, 26, 27, 38, 41, 72 + and 117; the type and range codes 53, 70 and 321; 131 for a string over what the + column holds; and 469 for a table CHECK constraint). The adapter splits a block + refused that way down to single rows to find the row; the rows around it are + written meanwhile. Network, socket, timeout, memory, too-many-parts, quorum, + replica, Keeper, access and missing-table codes, and any code not listed, are + retried and never searched. + +When a write names refused rows, the relay sets exactly those aside and writes the +rest again, so a batch of one is judged as a batch of five thousand. The archive +is written once, whole, before the warehouse is tried, so a row the warehouse +refused is already in the archive. A row whose own details cannot be serialized is +the exception: the archive's line for an event carries the hash of its canonical +details, which such a row does not have, so it is never archived and the +quarantine holds its only copy. A set-aside row moves, whole and with the error, into `audit_event_quarantine` -in the same transaction that deletes the delivered rows. The relay logs each at -error level (event id, type and queue sequence number, never the payload). -`saas.audit_queue.quarantined` counts the rows there and the -`audit_relay_quarantine` alert fires while it is above zero. The kit never +in the same transaction that deletes the delivered rows. The `error` column starts +with the class, then the cause (bounded to 1000 bytes, valid text): + +| `error` starts with | what it means | where else the row is | +|---|---|---| +| `warehouse_rejected_archived:` | the warehouse refused the row for good | the archive, whole | +| `unserializable_not_archived:` | the row's details cannot be serialized | nowhere: the quarantine holds the only copy | + +The relay logs each at error level (event id, type, queue sequence number and +class, never the payload). `saas.audit_queue.quarantined` counts the rows there +and the `audit_relay_quarantine` alert fires while it is above zero. The kit never deletes a quarantined row. Replaying or resolving quarantined rows is not built yet: until it is, read the table as the database owner (the relay's role may read -it), fix the cause, and keep the rows. +it), fix the cause, and keep the rows. Read the causes before concluding that the +rows are at fault: reasons that agree across events of different organizations +point at the warehouse's table, which someone changed under the relay, not at the +events. ### Switching `AUDIT_SINK` back to `postgres` (or `both`) diff --git a/module/services/accounts/AGENTS.md b/module/services/accounts/AGENTS.md index 9104879f2..88d37b7d4 100644 --- a/module/services/accounts/AGENTS.md +++ b/module/services/accounts/AGENTS.md @@ -94,11 +94,16 @@ survives a restart and reaches every replica, and is only served when it is read dedupes by event id. Four rules a change must keep. (1) The archive object is written once per batch, whole, under a name no other set of rows ever uses — the object-store writers take a precondition failure on a name to mean "this - batch, already written" — so only the *warehouse* write is split when a batch - fails. (2) A row is set aside into `audit_event_quarantine` only after the - warehouse refused it alone, accepted another row afterwards, and refused it - again; an outage sets nothing aside. The move from queue to quarantine is one - statement, and nothing in the kit ever deletes a queue or quarantine row (the + batch, already written" — so only the *warehouse* write is repeated, without + the rows it refused for good, when a batch fails. (2) A row is set aside into `audit_event_quarantine` only when the + warehouse adapter reports a permanent refusal of that row + (`business.PermanentRowRejection`, carrying the event id) or the row's details + cannot be serialized; every other failure — transport, 5xx, 429, quota, + timeout, quorum or replica, or an error the adapter cannot classify — is + retried and sets nothing aside, and no decision is drawn from what else the + warehouse accepted. The reason starts with its class + (`warehouse_rejected_archived` or `unserializable_not_archived`) and is bounded + valid text. The move from queue to quarantine is one statement, and nothing in the kit ever deletes a queue or quarantine row (the down migrations refuse while either holds rows). (3) The queue is observed under every `AUDIT_SINK` (`saas.audit_queue.{depth,oldest_age,quarantined, snapshot_errors}`): a failed read makes the gauges absent, never stale, and diff --git a/module/services/accounts/code/pkg/business/audit_relay.go b/module/services/accounts/code/pkg/business/audit_relay.go index 38fa1864c..03ba8bcd4 100644 --- a/module/services/accounts/code/pkg/business/audit_relay.go +++ b/module/services/accounts/code/pkg/business/audit_relay.go @@ -8,6 +8,7 @@ import ( "strings" "sync" "time" + "unicode/utf8" "github.com/codefly-dev/core/wool" "github.com/google/uuid" @@ -35,9 +36,10 @@ type AuditDeliveryOutcome struct { // Delivered are the rows the archive and the store both acknowledged. The // queue deletes them. Delivered []int64 - // Quarantined are the rows the relay set aside because the store refused - // them while it was accepting others. The queue moves them, whole, into the - // quarantine; it never deletes them. + // Quarantined are the rows the relay set aside because they can never be + // delivered: the store reported a permanent refusal of the row + // (PermanentRowRejection), or its details cannot be serialized. The queue + // moves them, whole, into the quarantine; it never deletes them. Quarantined []QuarantinedAuditEvent // Pending is why any other row stays queued, nil when none does. Pending error @@ -93,17 +95,31 @@ const ( // writes are done. It runs on a context of its own so a delivery that used // its whole batch budget still gets to record what it delivered. AuditRelayBookkeepingTimeout = 30 * time.Second - // auditRelayMaxSplitWrites bounds the store writes one batch spends - // isolating the rows a refused batch write holds. - auditRelayMaxSplitWrites = 64 - // auditRelayProbeWrites is how many store writes in a row may fail, with none - // accepted, before the relay concludes that the store is unreachable rather - // than refusing rows, and stops searching. - auditRelayProbeWrites = 16 - // auditRelayMaxReasonLength bounds the error text kept with a quarantined row. + // auditRelayMaxAppendWrites bounds the store writes one batch spends setting + // aside the rows the store reports it refused for good: each write that + // reports refusals removes at least one row, and a store that reports every + // refused row of a write needs a few. A batch that still holds rows after + // that stays queued and continues on the next pass. + auditRelayMaxAppendWrites = 64 + // auditRelayMaxReasonLength bounds, in bytes, the error text kept with a + // quarantined row. auditRelayMaxReasonLength = 1000 ) +// The class a quarantined row's reason starts with, as ": ", so a +// responder can tell what the quarantine holds without reading the cause. +const ( + // QuarantineWarehouseRejectedArchived is a row the store refused for good. It + // was written to the archive, whole, before the store was tried: the archive + // holds a copy of it, and the quarantine holds another. + QuarantineWarehouseRejectedArchived = "warehouse_rejected_archived" + // QuarantineUnserializableNotArchived is a row whose details cannot be + // serialized. The archive records each event with the SHA-256 of its + // canonical details, which such a row does not have, so it was never + // archived: the quarantine holds the only copy, payload included. + QuarantineUnserializableNotArchived = "unserializable_not_archived" +) + // AuditRelayConfig wires a relay. type AuditRelayConfig struct { Queue AuditQueue @@ -137,15 +153,21 @@ type AuditRelayConfig struct { // repeats the archive write, not the store's. The archive object is written // whole and once: an archive treats a precondition failure on an object name as // "this batch, already written", so a name must never be written again with -// different rows. Only the store write is split when the store refuses a batch; -// it keys each record by event id and has no use for a batch's name. +// different rows. Only the store write is repeated without the rows the store +// refused for good; it keys each record by event id and has no use for a +// batch's name. // -// A row the store refuses must not hold the rest of the queue behind it. When a -// batch write fails the relay splits the batch and retries the halves, down to -// single rows. A single row is set aside — moved whole to the quarantine, -// never deleted — only when the store refused it alone, accepted another row -// afterwards, and refused it again on a second try: during an outage every -// write fails and nothing is set aside. +// A row the store refuses for good must not hold the rest of the queue behind +// it, and a store that is merely failing must not cost a row its place. The +// relay sets a row aside — moves it whole to the quarantine, never deletes it — +// only when the store itself reports that it refused that row for the row's own +// content (a PermanentRowRejection naming the event). Every other failure is +// retried with backoff and sets nothing aside: a transport error, a server +// error, a throttle, a quota, a timeout, a failed quorum, and any error the +// store did not classify. The relay never infers a refusal from what else a +// store accepted, or from how often a row failed. When a write reports +// refusals, the relay sets those rows aside and writes the rest again, so a +// batch of one is judged exactly as a batch of five thousand. // // While it runs, the relay remembers what each queued row has had acknowledged // and what it decided about it, so retrying a batch during an outage of one @@ -365,21 +387,17 @@ func oldestEnqueued(events []QueuedAuditEvent) time.Time { // compose classifies and hashes each row it has not composed. A registry that // cannot be read is an error, retried with the batch — never mistaken for an -// unregistered type. A row whose own details cannot be serialized is set aside, -// but only when another row composed: with none composing, nothing shows that -// the rows, not the relay, are at fault. +// unregistered type. A row whose own details cannot be serialized is set aside +// with its own error: serializing is a function of the row alone, so the same +// row fails the same way every time, whatever else is queued with it. It cannot +// be archived either, since the archive's line for an event carries the hash of +// its canonical details; the quarantine holds its only copy, and its reason says +// so (QuarantineUnserializableNotArchived). func (r *AuditRelay) compose(ctx context.Context, rows []*queuedRow) error { resolver := NewAuditEventResolver(r.types) - composed := false - var unreadable []*queuedRow - var first error for _, row := range rows { state := row.state - if state.refused != "" { - continue - } - if state.composed { - composed = true + if state.refused != "" || state.composed { continue } resolved, err := resolver.Resolve(ctx, row.event.Entry.EventType) @@ -388,23 +406,10 @@ func (r *AuditRelay) compose(ctx context.Context, rows []*queuedRow) error { } record, err := NewAuditRecord(row.event.Entry, resolved.RetentionClass()) if err != nil { - if first == nil { - first = err - } - unreadable = append(unreadable, row) + r.setAside(ctx, row, QuarantineUnserializableNotArchived, err) continue } state.record, state.composed = record, true - composed = true - } - if len(unreadable) == 0 { - return nil - } - if !composed { - return first - } - for _, row := range unreadable { - r.setAside(ctx, row, first) } return nil } @@ -446,126 +451,99 @@ func (r *AuditRelay) archiveRows(ctx context.Context, rows []*queuedRow) error { return nil } -// appendRows writes the archived rows the store has not acknowledged. The -// whole run goes first; when the store refuses it, the run is split and the -// halves retried, so a row the store cannot take does not hold the rest back. -func (r *AuditRelay) appendRows(ctx context.Context, rows []*queuedRow) error { +// awaitingAppend is the archived rows the store has neither acknowledged nor +// refused for good. +func awaitingAppend(rows []*queuedRow) []*queuedRow { var todo []*queuedRow for _, row := range rows { if row.state.archive != nil && !row.state.appended && row.state.refused == "" { todo = append(todo, row) } } - if len(todo) == 0 { - return nil - } + return todo +} - writes := 0 - write := func(run []*queuedRow) error { - writes++ - first := run[0].state.archive - batch := AuditBatch{ID: first.id, DeploymentID: r.deploymentID, ComposedAt: first.composedAt, Records: make([]AuditRecord, len(run))} - for i, row := range run { - batch.Records[i] = row.state.record - } - if err := r.store.AppendAuditBatch(ctx, batch); err != nil { - return err +// appendRows writes the archived rows the store has not acknowledged, whole. +// When the store reports rows it refused for good, those are set aside and the +// rest is written again; when it fails any other way, nothing is set aside and +// the rows stay queued for the next attempt. Rows an earlier write of this call +// may have stored are written again: the store keys a record by event id. +func (r *AuditRelay) appendRows(ctx context.Context, rows []*queuedRow) error { + var failure error + for writes := 0; writes < auditRelayMaxAppendWrites; writes++ { + todo := awaitingAppend(rows) + if len(todo) == 0 { + return nil } - for _, row := range run { - row.state.appended = true + if err := ctx.Err(); err != nil { + return fmt.Errorf("audit relay: append %d rows of batch %s: %w", len(todo), todo[0].state.archive.id, err) } - return nil - } - - err := write(todo) - if err == nil { - return nil - } - failure := fmt.Errorf("audit relay: append %d rows of batch %s: %w", len(todo), todo[0].state.archive.id, err) - if len(todo) == 1 { - return failure - } - - var alone []*queuedRow - accepted := 0 - // Search breadth-first, so the first accepted run turns up within a write or - // two when only one row is bad, and an outage ends after a bounded number of - // writes instead of after one per row. - failed := [][]*queuedRow{todo} -search: - for len(failed) > 0 { - var next [][]*queuedRow - for _, run := range failed { - middle := len(run) / 2 - for _, half := range [][]*queuedRow{run[:middle], run[middle:]} { - if ctx.Err() != nil || writes >= auditRelayMaxSplitWrites || (accepted == 0 && writes >= auditRelayProbeWrites) { - break search - } - switch err := write(half); { - case err == nil: - accepted++ - case len(half) == 1: - alone = append(alone, half[0]) - default: - next = append(next, half) - } - } + first := todo[0].state.archive + batch := AuditBatch{ID: first.id, DeploymentID: r.deploymentID, ComposedAt: first.composedAt, Records: make([]AuditRecord, len(todo))} + for i, row := range todo { + batch.Records[i] = row.state.record } - failed = next - } - - // A row is set aside only on this chain of evidence: the store refused it - // alone; afterwards the store took a row it had already acknowledged, again - // (so it was reachable after the refusal, and a duplicate is something every - // read already discards); and it refuses the row alone once more. Without a - // row to write as that witness nothing is set aside. - if witness := lastAppended(rows); len(alone) > 0 && witness != nil && ctx.Err() == nil { - if write([]*queuedRow{witness}) == nil { - for _, refused := range alone { - if ctx.Err() != nil { - break - } - if err := write([]*queuedRow{refused}); err != nil && ctx.Err() == nil { - r.setAside(ctx, refused, err) - } + err := r.store.AppendAuditBatch(ctx, batch) + if err == nil { + for _, row := range todo { + row.state.appended = true } + return nil } - } - - unresolved := 0 - for _, row := range todo { - if !row.state.appended && row.state.refused == "" { - unresolved++ + failure = fmt.Errorf("audit relay: append %d rows of batch %s: %w", len(todo), first.id, err) + if !r.setAsideRefused(ctx, todo, err) { + return failure } } - if unresolved == 0 { + if len(awaitingAppend(rows)) == 0 { return nil } return failure } -// lastAppended is the last row the store has acknowledged, nil when none has. -func lastAppended(rows []*queuedRow) *queuedRow { - for i := len(rows) - 1; i >= 0; i-- { - if rows[i].state.appended { - return rows[i] +// setAsideRefused sets aside every row of todo that err says the store refused +// for good, and reports whether it set any aside. A refusal that names an event +// that is not among todo is not evidence about any of them. +func (r *AuditRelay) setAsideRefused(ctx context.Context, todo []*queuedRow, err error) bool { + byEvent := make(map[string][]*queuedRow, len(todo)) + for _, row := range todo { + byEvent[row.event.Entry.ID] = append(byEvent[row.event.Entry.ID], row) + } + set := false + for _, rejection := range PermanentRowRejections(err) { + for _, row := range byEvent[rejection.EventID] { + r.setAside(ctx, row, QuarantineWarehouseRejectedArchived, rejection) + set = true } } - return nil + return set } -// setAside records the decision that row cannot be delivered. The decision is -// kept until the row leaves the queue, so a lost commit does not turn it back -// into doubt. -func (r *AuditRelay) setAside(ctx context.Context, row *queuedRow, cause error) { - reason := cause.Error() - if len(reason) > auditRelayMaxReasonLength { - reason = reason[:auditRelayMaxReasonLength] - } - row.state.refused = reason - wool.Get(ctx).In("audit.relay").Error("audit event set aside: the store refused it while accepting others; it stays in the quarantine and is not delivered", +// AuditQuarantineReason is text as the quarantine keeps it: valid UTF-8 with no +// NUL, because that is all PostgreSQL stores in a text column and a store's +// error can hold anything; and at most 1000 bytes, cut between characters. An +// invalid sequence or a NUL becomes U+FFFD. +func AuditQuarantineReason(text string) string { + text = strings.ToValidUTF8(text, "\uFFFD") + text = strings.ReplaceAll(text, "\x00", "\uFFFD") + if len(text) <= auditRelayMaxReasonLength { + return text + } + cut := auditRelayMaxReasonLength + for cut > 0 && !utf8.RuneStart(text[cut]) { + cut-- + } + return text[:cut] +} + +// setAside records the decision that row cannot be delivered, with its class +// and the error that decided it. The decision is kept until the row leaves the +// queue, so a lost commit does not turn it back into doubt. +func (r *AuditRelay) setAside(ctx context.Context, row *queuedRow, class string, cause error) { + row.state.refused = AuditQuarantineReason(class + ": " + cause.Error()) + wool.Get(ctx).In("audit.relay").Error("audit event set aside: it cannot be delivered now or later; it stays in the quarantine", wool.Field("event_id", row.event.Entry.ID), wool.Field("event_type", string(row.event.Entry.EventType)), - wool.Field("queue_seq", strconv.FormatInt(row.event.Seq, 10)), wool.ErrField(cause)) + wool.Field("queue_seq", strconv.FormatInt(row.event.Seq, 10)), wool.Field("reason_class", class), wool.ErrField(cause)) } // pollInterval is how often the loop looks for due events: a fifth of MaxWait, diff --git a/module/services/accounts/code/pkg/business/audit_relay_metrics.go b/module/services/accounts/code/pkg/business/audit_relay_metrics.go index 4957f8556..1f1bfb186 100644 --- a/module/services/accounts/code/pkg/business/audit_relay_metrics.go +++ b/module/services/accounts/code/pkg/business/audit_relay_metrics.go @@ -74,7 +74,7 @@ func NewAuditRelayMonitor(source AuditQueueMetricsSource, meter metric.Meter, in return nil, err } if m.quarantined, err = meter.Int64ObservableGauge("saas.audit_queue.quarantined", - metric.WithDescription("Audit events the relay set aside because the store refused them. Nothing deletes them. Absent while the queue cannot be read.")); err != nil { + metric.WithDescription("Audit events the relay set aside because the store refused them for good or their details cannot be serialized. Nothing deletes them. Absent while the queue cannot be read.")); err != nil { return nil, err } if m.snapshotErrors, err = meter.Int64Counter("saas.audit_queue.snapshot_errors", diff --git a/module/services/accounts/code/pkg/business/audit_relay_quarantine_test.go b/module/services/accounts/code/pkg/business/audit_relay_quarantine_test.go index 5dcb09bd7..bc6c59d06 100644 --- a/module/services/accounts/code/pkg/business/audit_relay_quarantine_test.go +++ b/module/services/accounts/code/pkg/business/audit_relay_quarantine_test.go @@ -3,7 +3,6 @@ package business import ( "context" "errors" - "fmt" "slices" "sync" "testing" @@ -13,21 +12,9 @@ import ( ) // A queued row the store refuses for good must not hold every other tenant's -// events behind it (ADR 0009). The relay isolates it by splitting a refused -// batch, and sets it aside only on proof that the store itself was reachable. - -// rejects makes a store write fail while the batch holds any of ids, the way a -// warehouse refuses a row it cannot accept. -func rejects(ids ...string) func(AuditBatch) error { - return func(batch AuditBatch) error { - for _, record := range batch.Records { - if slices.Contains(ids, record.Entry.ID) { - return fmt.Errorf("row %s rejected: invalid value", record.Entry.ID) - } - } - return nil - } -} +// events behind it (ADR 0009). The store says which row it refused, and the relay +// sets exactly that row aside; audit_relay_rejection_test.go pins the rule from the other +// side — a store that merely fails sets nothing aside. func queuedIDs(rows []QueuedAuditEvent) []string { ids := make([]string, len(rows)) @@ -41,7 +28,7 @@ func TestAuditRelaySetsAsideARowTheStoreRefusesAndDeliversTheRest(t *testing.T) f := newRelayFixture(t, 10, time.Second, nil) ids := f.enqueue(time.Minute, EventSessionRevoked, repeatOrg(teeOrgID, 10)...) poison := ids[3] - f.warehouse.fail = rejects(poison) + f.warehouse.fail = refusesForGood(poison) delivered, err := f.relay.DrainOnce(context.Background()) require.NoError(t, err, "a refused row that is set aside is not a failed delivery") @@ -70,63 +57,78 @@ func TestAuditRelayNeverSetsAsideAnythingDuringAnOutage(t *testing.T) { require.ErrorContains(t, err, "warehouse unavailable") require.Zero(t, delivered) require.Len(t, f.queue.remaining(), 16, "everything stays queued") - require.Empty(t, f.queue.quarantined(), "a failure that no other row's success contradicts proves nothing about the row") - require.LessOrEqual(t, writes, 16, "and the search for a refused row gives up instead of writing every subrange") + require.Empty(t, f.queue.quarantined(), "a failure the store did not attribute to a row proves nothing about any row") + require.Equal(t, 1, writes, "an outage costs one write a pass, not a search through the batch") } -func TestAuditRelayDoesNotSetAsideRowsThatFailAfterTheStoreWentDown(t *testing.T) { +// A store that refuses one row and also fails the rest of the write has still +// attributed that one row: it is set aside, and the others wait. +func TestAuditRelaySetsAsideWhatTheStoreRefusedWhenItFailedTheRestToo(t *testing.T) { f := newRelayFixture(t, 8, time.Second, nil) ids := f.enqueue(time.Minute, EventSessionRevoked, repeatOrg(teeOrgID, 8)...) - // The whole batch fails, the first half is accepted, then the store is gone. - calls := 0 - f.warehouse.fail = func(AuditBatch) error { - calls++ - if calls == 2 { - return nil - } - return errors.New("warehouse unavailable") + refuse := refusesForGood(ids[2]) + f.warehouse.fail = func(batch AuditBatch) error { + return errors.Join(refuse(batch), errors.New("warehouse unavailable")) } delivered, err := f.relay.DrainOnce(context.Background()) - require.Error(t, err) - require.Equal(t, 4, delivered, "what the store accepted is delivered") - require.Empty(t, f.queue.quarantined(), "rows that failed after the last accepted write were not refused for their content") - require.Equal(t, ids[4:], queuedIDs(f.queue.remaining())) + + require.ErrorContains(t, err, "warehouse unavailable") + require.Zero(t, delivered) + require.Equal(t, ids[2:3], queuedIDs(f.queue.quarantined()), "the refused row is set aside") + require.Equal(t, append(slices.Clone(ids[:2]), ids[3:]...), queuedIDs(f.queue.remaining()), "and no other row is judged") } -func TestAuditRelayRetriesATransientRefusalBeforeSettingAnythingAside(t *testing.T) { +// A failure that is not a refusal is retried whole, however it reads. +func TestAuditRelayRetriesAFailureThatIsNotARefusalWhole(t *testing.T) { f := newRelayFixture(t, 4, time.Second, nil) f.enqueue(time.Minute, EventSessionRevoked, repeatOrg(teeOrgID, 4)...) - // The whole batch, then the first half and the first row alone fail; every - // other write is accepted, the witness and the first row's second try - // included. - script := []bool{false, false, true, false, true} calls := 0 f.warehouse.fail = func(AuditBatch) error { calls++ - if calls <= len(script) && !script[calls-1] { - return errors.New("blip") + if calls == 1 { + return errors.New("row 2 rejected: invalid value") } return nil } + _, err := f.relay.DrainOnce(context.Background()) + require.Error(t, err) + require.Len(t, f.queue.remaining(), 4) + delivered, err := f.relay.DrainOnce(context.Background()) require.NoError(t, err) require.Equal(t, 4, delivered) require.Empty(t, f.queue.remaining()) - require.Empty(t, f.queue.quarantined(), "a row the store accepts on a second try was never refused") - require.Len(t, f.warehouse.byID, 4) + require.Empty(t, f.queue.quarantined(), "an error that is not the store's refusal of a named row never sets a row aside") + require.Len(t, f.archive.order, 1, "and the archive holds the batch once") } -func TestAuditRelayRefusalOfEveryRowIsIndistinguishableFromAnOutage(t *testing.T) { +func TestAuditRelayRefusalOfEveryRowSetsEveryRowAside(t *testing.T) { f := newRelayFixture(t, 4, time.Second, nil) ids := f.enqueue(time.Minute, EventSessionRevoked, repeatOrg(teeOrgID, 4)...) - f.warehouse.fail = rejects(ids...) + f.warehouse.fail = refusesForGood(ids...) + + delivered, err := f.relay.DrainOnce(context.Background()) + require.NoError(t, err) + require.Zero(t, delivered) + require.Empty(t, f.queue.remaining()) + require.ElementsMatch(t, ids, queuedIDs(f.queue.quarantined()), "the store said so about every row") +} + +// A refusal that names an event the batch does not hold is no evidence about +// the rows it does hold. +func TestAuditRelayIgnoresARefusalOfAnEventNotInTheBatch(t *testing.T) { + f := newRelayFixture(t, 3, time.Second, nil) + f.enqueue(time.Minute, EventSessionRevoked, repeatOrg(teeOrgID, 3)...) + f.warehouse.fail = func(AuditBatch) error { + return &PermanentRowRejection{EventID: NewIDString(), Cause: errors.New("invalid")} + } _, err := f.relay.DrainOnce(context.Background()) require.Error(t, err) - require.Len(t, f.queue.remaining(), 4) - require.Empty(t, f.queue.quarantined(), "with no row accepted there is no evidence the store was reachable") + require.Empty(t, f.queue.quarantined()) + require.Len(t, f.queue.remaining(), 3) } // An archive treats a precondition failure on an object name as "this batch, @@ -137,7 +139,7 @@ func TestAuditRelayNeverWritesDifferentRowsUnderAnArchiveName(t *testing.T) { archive := &namedArchive{} f.relay.archive = archive ids := f.enqueue(time.Minute, EventSessionRevoked, repeatOrg(teeOrgID, 8)...) - f.warehouse.fail = rejects(ids[5]) + f.warehouse.fail = refusesForGood(ids[5]) _, err := f.relay.DrainOnce(context.Background()) require.NoError(t, err) @@ -212,7 +214,7 @@ func TestAuditRelayKeepsItsVerdictWhenTheCommitIsLost(t *testing.T) { f := newRelayFixture(t, 6, time.Second, nil) ids := f.enqueue(time.Minute, EventSessionRevoked, repeatOrg(teeOrgID, 6)...) writes := 0 - refuse := rejects(ids[2]) + refuse := refusesForGood(ids[2]) f.warehouse.fail = func(batch AuditBatch) error { writes++ return refuse(batch) @@ -235,7 +237,7 @@ func TestAuditRelayKeepsItsVerdictWhenTheCommitIsLost(t *testing.T) { // unencodable is a payload value no store can serialize. func unencodable() map[string]any { return map[string]any{"bad": make(chan int)} } -func TestAuditRelaySetsAsideARowItCannotComposeOnlyWhenOthersCan(t *testing.T) { +func TestAuditRelaySetsAsideEveryRowItCannotCompose(t *testing.T) { f := newRelayFixture(t, 3, time.Second, nil) ids := f.enqueue(time.Minute, EventSessionRevoked, repeatOrg(teeOrgID, 3)...) f.queue.rows[1].Entry.Payload = unencodable() @@ -246,15 +248,21 @@ func TestAuditRelaySetsAsideARowItCannotComposeOnlyWhenOthersCan(t *testing.T) { require.Equal(t, []string{ids[1]}, queuedIDs(f.queue.quarantined())) require.Empty(t, f.queue.remaining()) - g := newRelayFixture(t, 2, time.Second, nil) - g.enqueue(time.Minute, EventSessionRevoked, repeatOrg(teeOrgID, 2)...) - for i := range g.queue.rows { - g.queue.rows[i].Entry.Payload = unencodable() + // Serializing a row is a function of that row alone, so a batch in which no + // row composes holds only rows that cannot be delivered, and a batch of one + // is judged the same way as a batch of three. + for _, size := range []int{1, 2} { + g := newRelayFixture(t, size, time.Second, nil) + bad := g.enqueue(time.Minute, EventSessionRevoked, repeatOrg(teeOrgID, size)...) + for i := range g.queue.rows { + g.queue.rows[i].Entry.Payload = unencodable() + } + _, err = g.relay.DrainOnce(context.Background()) + require.NoError(t, err) + require.Empty(t, g.queue.remaining()) + require.ElementsMatch(t, bad, queuedIDs(g.queue.quarantined())) + require.Empty(t, g.archive.order, "nothing was composed, so nothing is archived") } - _, err = g.relay.DrainOnce(context.Background()) - require.Error(t, err) - require.Len(t, g.queue.remaining(), 2, "when no row composes, nothing shows the rows rather than the relay at fault") - require.Empty(t, g.queue.quarantined()) } // Each batch is bounded on its own: a delivery that is working must not be cut diff --git a/module/services/accounts/code/pkg/business/audit_relay_rejection_test.go b/module/services/accounts/code/pkg/business/audit_relay_rejection_test.go new file mode 100644 index 000000000..e16d23f74 --- /dev/null +++ b/module/services/accounts/code/pkg/business/audit_relay_rejection_test.go @@ -0,0 +1,251 @@ +package business + +import ( + "context" + "errors" + "fmt" + "math/rand" + "slices" + "strings" + "testing" + "time" + "unicode/utf8" + + "github.com/stretchr/testify/require" +) + +// The relay sets a row aside only on what the store reports about that row: a +// *PermanentRowRejection naming its event. Everything else — however often it +// repeats, whatever else the store accepts in between — is a retryable failure +// that keeps the batch queued. These tests pin that rule from both sides: a +// store that is merely unreliable quarantines nothing, and a store that refuses +// rows quarantines exactly those rows however many there are and however small +// the batch. + +// refusesForGood makes a store write fail while the batch holds any of ids, +// reporting every such row at once, as a warehouse that lists each invalid row +// of a request does. +func refusesForGood(ids ...string) func(AuditBatch) error { + return func(batch AuditBatch) error { + var refused []error + for _, record := range batch.Records { + if slices.Contains(ids, record.Entry.ID) { + refused = append(refused, &PermanentRowRejection{ + EventID: record.Entry.ID, + Cause: fmt.Errorf("value of row %s is invalid", record.Entry.ID), + }) + } + } + return errors.Join(refused...) + } +} + +// refusesOneAtATime is refusesForGood for a store that reports only the first +// refused row of a write. +func refusesOneAtATime(ids ...string) func(AuditBatch) error { + all := refusesForGood(ids...) + return func(batch AuditBatch) error { + err := all(batch) + if err == nil { + return nil + } + return PermanentRowRejections(err)[0] + } +} + +// flaky makes a share of the writes that reach inner fail with an error that +// says nothing of any row, as a throttled or half-dead warehouse does. The +// failure does not look at the batch, so it never tells the relay about a row. +func flaky(seed int64, share float64, inner func(AuditBatch) error) func(AuditBatch) error { + random := rand.New(rand.NewSource(seed)) + return func(batch AuditBatch) error { + if random.Float64() < share { + return errors.New("googleapi: Error 429: rateLimitExceeded") + } + if inner == nil { + return nil + } + return inner(batch) + } +} + +// drainUntilEmpty runs passes, as the relay loop does, until the queue is empty. +func drainUntilEmpty(t *testing.T, f *relayFixture, limit int) (passes int) { + t.Helper() + for passes < limit { + if len(f.queue.remaining()) == 0 { + return passes + } + passes++ + _, _ = f.relay.DrainOnce(context.Background()) + } + require.Fail(t, "the relay did not empty the queue", "%d rows remain after %d passes", len(f.queue.remaining()), passes) + return passes +} + +func TestAuditRelayQuarantinesNothingWhileTheStoreIsMerelyUnreliable(t *testing.T) { + for _, seed := range []int64{1, 2, 3} { + f := newRelayFixture(t, 20, time.Second, nil) + ids := f.enqueue(time.Minute, EventSessionRevoked, repeatOrg(teeOrgID, 6000)...) + f.warehouse.fail = flaky(seed, 0.5, nil) + + drainUntilEmpty(t, f, 100000) + + require.Empty(t, f.queue.quarantined(), "seed %d: a write that fails says nothing about the rows it held", seed) + require.Len(t, f.warehouse.byID, len(ids), "seed %d: every row is delivered", seed) + } +} + +func TestAuditRelayRetriesEveryFailureThatIsNotARowRefusal(t *testing.T) { + for name, failure := range map[string]error{ + "transport": errors.New("Post \"https://bigquery.googleapis.com/…\": connection reset by peer"), + "server": errors.New("googleapi: Error 503: backendError"), + "throttled": errors.New("googleapi: Error 429: rateLimitExceeded"), + "quota": errors.New("googleapi: Error 403: quotaExceeded"), + "timeout": context.DeadlineExceeded, + "quorum": errors.New("code: 285, message: Number of alive replicas is less than requested quorum"), + "unclassified": errors.New("something the store did not explain"), + "says invalid": errors.New("row 3 rejected: invalid value"), + "empty refusal": &PermanentRowRejection{}, + } { + for _, size := range []int{1, 8} { + f := newRelayFixture(t, size, time.Second, nil) + f.enqueue(time.Minute, EventSessionRevoked, repeatOrg(teeOrgID, size)...) + f.warehouse.fail = func(AuditBatch) error { return failure } + + delivered, err := f.relay.DrainOnce(context.Background()) + + require.Error(t, err, "%s, batch of %d", name, size) + require.Zero(t, delivered) + require.Empty(t, f.queue.quarantined(), "%s, batch of %d: only a refusal that names a row sets it aside", name, size) + require.Len(t, f.queue.remaining(), size, "%s, batch of %d: every row stays queued", name, size) + } + } +} + +func TestAuditRelayDeliversEveryGoodRowAmongDenseRefusedRows(t *testing.T) { + const rows = 500 + badEvery := map[string]int{"20 of 500": 25, "15 of 500": 33, "50 of 500": 10} + stores := map[string]func(bad ...string) func(AuditBatch) error{ + "all reported at once": refusesForGood, + "one reported a time": refusesOneAtATime, + "with a flaky store": func(bad ...string) func(AuditBatch) error { + return flaky(7, 0.3, refusesForGood(bad...)) + }, + } + for density, every := range badEvery { + for storeName, store := range stores { + f := newRelayFixture(t, rows, time.Second, nil) + ids := f.enqueue(time.Minute, EventSessionRevoked, repeatOrg(teeOrgID, rows)...) + var bad, good []string + for i, id := range ids { + if i%every == 0 { + bad = append(bad, id) + } else { + good = append(good, id) + } + } + f.warehouse.fail = store(bad...) + label := density + ", " + storeName + + drainUntilEmpty(t, f, 1000) + + require.ElementsMatch(t, bad, queuedIDs(f.queue.quarantined()), label+": exactly the refused rows are set aside") + require.Len(t, f.warehouse.byID, len(good), label+": every other row is delivered") + for _, id := range bad { + require.NotContains(t, f.warehouse.byID, id, label) + reason := f.queue.reasonOf(id) + require.Contains(t, reason, "value of row "+id+" is invalid", label+": each keeps the store's own reason") + require.True(t, strings.HasPrefix(reason, QuarantineWarehouseRejectedArchived+": "), label+": and says the archive holds it") + } + require.Len(t, f.archive.order, 1, label+": the archive holds the batch once, whole") + require.ElementsMatch(t, ids, f.archive.eventIDs(), label) + } + } +} + +func TestAuditRelayBatchOfOneDeliversTheRowsBehindARefusedFirstRow(t *testing.T) { + f := newRelayFixture(t, 1, time.Second, nil) + ids := f.enqueue(time.Minute, EventSessionRevoked, repeatOrg(teeOrgID, 4)...) + f.warehouse.fail = refusesForGood(ids[0]) + + delivered, err := f.relay.DrainOnce(context.Background()) + + require.NoError(t, err, "a refused row that is set aside is not a failed delivery") + require.Equal(t, 3, delivered, "the rows behind the refused one are delivered") + require.Equal(t, ids[:1], queuedIDs(f.queue.quarantined())) + require.Empty(t, f.queue.remaining()) + require.ElementsMatch(t, ids[1:], mapKeys(f.warehouse.byID)) +} + +func TestAuditRelaySetsAsideALoneRefusedRowWithoutWaitingForAnotherEvent(t *testing.T) { + f := newRelayFixture(t, 10, time.Second, nil) + ids := f.enqueue(time.Minute, EventSessionRevoked, teeOrgID) + f.warehouse.fail = refusesForGood(ids[0]) + + _, err := f.relay.DrainOnce(context.Background()) + + require.NoError(t, err) + require.Equal(t, ids, queuedIDs(f.queue.quarantined())) + require.Empty(t, f.queue.remaining()) +} + +func TestAuditRelayGivesEveryUnserializableRowItsOwnReason(t *testing.T) { + f := newRelayFixture(t, 5, time.Second, nil) + ids := f.enqueue(time.Minute, EventSessionRevoked, repeatOrg(teeOrgID, 5)...) + for _, i := range []int{0, 2, 3} { + f.queue.rows[i].Entry.Payload = unencodable() + } + + delivered, err := f.relay.DrainOnce(context.Background()) + + require.NoError(t, err) + require.Equal(t, 2, delivered) + require.ElementsMatch(t, []string{ids[0], ids[2], ids[3]}, queuedIDs(f.queue.quarantined())) + require.ElementsMatch(t, []string{ids[1], ids[4]}, f.archive.eventIDs(), "a row that cannot be serialized cannot be archived: the quarantine holds its only copy") + for _, i := range []int{0, 2, 3} { + reason := f.queue.reasonOf(ids[i]) + require.True(t, strings.HasPrefix(reason, QuarantineUnserializableNotArchived+": "), "the reason says the row was never archived") + require.Contains(t, reason, ids[i], "the reason names the row's own event") + for _, j := range []int{0, 2, 3} { + if j != i { + require.NotContains(t, reason, ids[j], "and no other row's") + } + } + } +} + +func mapKeys(byID map[string]AuditRecord) []string { + keys := make([]string, 0, len(byID)) + for id := range byID { + keys = append(keys, id) + } + return keys +} + +func TestAuditQuarantineReasonIsBoundedValidTextWithoutNul(t *testing.T) { + require.Equal(t, "", AuditQuarantineReason("")) + require.Equal(t, "plain error", AuditQuarantineReason("plain error")) + exactly := strings.Repeat("a", 1000) + require.Equal(t, exactly, AuditQuarantineReason(exactly), "text at the bound is kept whole") + + // A cut at byte 1000 must land between characters, whatever width they have + // and wherever they start. + for _, glyph := range []string{"é", "€", "😀"} { + for lead := 0; lead < 4; lead++ { + text := strings.Repeat("a", lead) + strings.Repeat(glyph, 600) + got := AuditQuarantineReason(text) + require.True(t, utf8.ValidString(got), "%q glyph after %d bytes", glyph, lead) + require.LessOrEqual(t, len(got), 1000) + require.Greater(t, len(got), 1000-len(glyph), "the cut is no earlier than the last whole character") + require.True(t, strings.HasPrefix(text, got), "what is kept is a prefix, not a rewrite") + } + } + + require.Equal(t, "before\uFFFDafter", AuditQuarantineReason("before\x00after"), "a NUL is replaced") + require.Equal(t, "bad \uFFFD bytes", AuditQuarantineReason("bad \xff\xfe bytes"), "invalid UTF-8 is replaced") + got := AuditQuarantineReason(strings.Repeat("\x00", 2000)) + require.True(t, utf8.ValidString(got)) + require.NotContains(t, got, "\x00") + require.LessOrEqual(t, len(got), 1000, "replacement is counted before the bound, not after it") +} diff --git a/module/services/accounts/code/pkg/business/audit_relay_test.go b/module/services/accounts/code/pkg/business/audit_relay_test.go index ecb4602bb..90f4ea821 100644 --- a/module/services/accounts/code/pkg/business/audit_relay_test.go +++ b/module/services/accounts/code/pkg/business/audit_relay_test.go @@ -22,10 +22,12 @@ import ( // relay delivers is deleted, what it sets aside moves to the quarantine, and // the rest stays. type memQueue struct { - mu sync.Mutex - rows []QueuedAuditEvent - held []QueuedAuditEvent // the quarantine - leased bool // another relay holds the lease + mu sync.Mutex + rows []QueuedAuditEvent + held []QueuedAuditEvent // the quarantine + // reasons is the error kept with each quarantined row, by sequence number. + reasons map[int64]string + leased bool // another relay holds the lease // crashes makes the next n removals fail before their commit, the way a // crash or a lost connection between the writes and the delete would. crashes int @@ -72,6 +74,10 @@ func (q *memQueue) Drain(ctx context.Context, limit int, deliver func(context.Co } for _, set := range outcome.Quarantined { gone[set.Seq] = true + if q.reasons == nil { + q.reasons = map[int64]string{} + } + q.reasons[set.Seq] = set.Reason } var kept []QueuedAuditEvent for _, row := range q.rows { @@ -92,6 +98,18 @@ func (q *memQueue) quarantined() []QueuedAuditEvent { return slices.Clone(q.held) } +// reasonOf is the error kept with the quarantined row of the event. +func (q *memQueue) reasonOf(eventID string) string { + q.mu.Lock() + defer q.mu.Unlock() + for _, row := range q.held { + if row.Entry.ID == eventID { + return q.reasons[row.Seq] + } + } + return "" +} + func (q *memQueue) deadlines() []time.Time { q.mu.Lock() defer q.mu.Unlock() diff --git a/module/services/accounts/code/pkg/infra/postgres_audit_queue.go b/module/services/accounts/code/pkg/infra/postgres_audit_queue.go index d603196b5..7b177eb07 100644 --- a/module/services/accounts/code/pkg/infra/postgres_audit_queue.go +++ b/module/services/accounts/code/pkg/infra/postgres_audit_queue.go @@ -194,7 +194,11 @@ func (q *PostgresAuditQueue) Drain( seqs := make([]int64, len(outcome.Quarantined)) reasons := make([]string, len(outcome.Quarantined)) for i, set := range outcome.Quarantined { - seqs[i], reasons[i] = set.Seq, set.Reason + // The relay bounds a reason already; the queue holds the constraint + // (text without NUL, valid UTF-8), so it does not trust any caller to, + // and a reason it cannot store would roll back the delivered rows too, + // on every pass. + seqs[i], reasons[i] = set.Seq, business.AuditQuarantineReason(set.Reason) } tag, err := tx.Exec(bookkeeping, auditQueueQuarantineSQL, seqs, reasons) if err != nil { diff --git a/module/services/accounts/code/pkg/infra/postgres_audit_relay_test.go b/module/services/accounts/code/pkg/infra/postgres_audit_relay_test.go new file mode 100644 index 000000000..427e3fca4 --- /dev/null +++ b/module/services/accounts/code/pkg/infra/postgres_audit_relay_test.go @@ -0,0 +1,132 @@ +//go:build !pure + +package infra_test + +import ( + "context" + "errors" + "strings" + "sync" + "testing" + "time" + "unicode/utf8" + + "github.com/stretchr/testify/require" + + "accounts/pkg/business" + "accounts/pkg/infra" +) + +// The relay against the real queue and quarantine: the relay's decisions have +// to be something Postgres can store, or the transaction that carries them +// rolls back together with the rows it delivered, and does so on every pass. + +// refusingWarehouse is a store that refuses, for good, the rows it was told to. +type refusingWarehouse struct { + mu sync.Mutex + refuse map[string]error + stored map[string]bool +} + +func (w *refusingWarehouse) AppendAuditBatch(_ context.Context, batch business.AuditBatch) error { + w.mu.Lock() + defer w.mu.Unlock() + var refused []error + for _, record := range batch.Records { + if cause, ok := w.refuse[record.Entry.ID]; ok { + refused = append(refused, &business.PermanentRowRejection{EventID: record.Entry.ID, Cause: cause}) + } + } + if len(refused) > 0 { + return errors.Join(refused...) + } + if w.stored == nil { + w.stored = map[string]bool{} + } + for _, record := range batch.Records { + w.stored[record.Entry.ID] = true + } + return nil +} + +type discardingArchive struct{} + +func (discardingArchive) WriteAuditBatch(context.Context, business.AuditBatch) error { return nil } + +// A store's error text is the store's own: it can be long, hold any character, +// and hold bytes that are not text at all. The reason kept with a quarantined +// row is cut to a bound, and the cut must not leave a half character, nor may a +// NUL or an invalid byte reach a text column. +func TestAuditRelayQuarantinesARowWhoseErrorIsLongMultibyteOrNotText(t *testing.T) { + pool := auditRelayPool(t) + emptyAuditQueue(t, pool) + orgID := seedOrg(t, seedUser(t)) + queue, err := infra.NewPostgresAuditQueue(pool) + require.NoError(t, err) + entries := enqueueN(t, orgID, 6) + + // The two long causes start one byte apart, so whatever the text before + // them, the 1000-byte cut falls inside a two-byte character for one of them. + causes := map[int]string{ + 1: strings.Repeat("é", 600), + 2: "x" + strings.Repeat("é", 600), + 3: "before\x00after", + 4: "bad \xff\xfe bytes", + } + warehouse := &refusingWarehouse{refuse: map[string]error{}} + for i, cause := range causes { + warehouse.refuse[entries[i].ID] = errors.New(cause) + } + relay, err := business.NewAuditRelay(business.AuditRelayConfig{ + Queue: queue, Store: warehouse, Archive: discardingArchive{}, DeploymentID: "deployment-1", + BatchSize: len(entries), MaxWait: time.Second, + }) + require.NoError(t, err) + + delivered, err := relay.DrainOnce(testCtx) + require.NoError(t, err, "a reason Postgres cannot store must not roll back the delivery") + require.Equal(t, 2, delivered) + require.Empty(t, drainAllWithoutDeleting(t, queue), "every row left the queue: delivered or quarantined") + require.Len(t, warehouse.stored, 2) + + for i := range causes { + var why string + require.NoError(t, pool.QueryRow(testCtx, `SELECT error FROM audit_event_quarantine WHERE id = $1`, entries[i].ID).Scan(&why), + "row %d is in the quarantine", i) + require.True(t, utf8.ValidString(why), "row %d: the reason is text", i) + require.NotContains(t, why, "\x00", "row %d", i) + require.LessOrEqual(t, len(why), 1000, "row %d: the reason stays bounded", i) + require.Contains(t, why, entries[i].ID, "row %d: and still says which event it is about", i) + require.True(t, strings.HasPrefix(why, business.QuarantineWarehouseRejectedArchived+": "), "row %d: and what the archive holds", i) + } + var why string + require.NoError(t, pool.QueryRow(testCtx, `SELECT error FROM audit_event_quarantine WHERE id = $1`, entries[3].ID).Scan(&why)) + require.Contains(t, why, "before") + require.Contains(t, why, "after") +} + +// The queue enforces the constraint itself, whoever built the outcome. +func TestAuditQueueStoresAReasonItWouldOtherwiseRefuse(t *testing.T) { + pool := auditRelayPool(t) + emptyAuditQueue(t, pool) + orgID := seedOrg(t, seedUser(t)) + queue, err := infra.NewPostgresAuditQueue(pool) + require.NoError(t, err) + entries := enqueueN(t, orgID, 2) + + _, err = queue.Drain(testCtx, 10, func(_ context.Context, events []business.QueuedAuditEvent) (business.AuditDeliveryOutcome, error) { + return business.AuditDeliveryOutcome{Quarantined: []business.QuarantinedAuditEvent{ + {Seq: events[0].Seq, Reason: "nul \x00 byte"}, + {Seq: events[1].Seq, Reason: strings.Repeat("é", 2000) + "\xff"}, + }}, nil + }) + require.NoError(t, err) + require.Empty(t, drainAllWithoutDeleting(t, queue)) + + for _, entry := range entries { + var why string + require.NoError(t, pool.QueryRow(testCtx, `SELECT error FROM audit_event_quarantine WHERE id = $1`, entry.ID).Scan(&why)) + require.True(t, utf8.ValidString(why)) + require.LessOrEqual(t, len(why), 1000) + } +} From c20b198e5d4ea0af2ad90cbfe5f993c9e05fe441 Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Tue, 6 Oct 2026 15:17:28 -0400 Subject: [PATCH 37/69] docs(adr): match ADR 0009 to the relay, grants and read bounds as built State what the audit relay, the BigQuery and ClickHouse adapters and the operator tools do, so the decision record and the code agree: - Delivery is not ordered. The queue gate is transaction visibility, not sequence number; both stores key a record by event id and reads sort by event time. - The relay keeps per-row acknowledgements. A row is quarantined only when the store reports it refused that row for good, or its details cannot be serialized; every other failure is retried with backoff. The quarantine reason starts with its class, and replay is not built. - The monitored series include the quarantine size, and all three go absent when the queue cannot be read. - The BigQuery identity also holds tables.create and tables.updateData, and the events table is refused at startup if its partitions expire. - ClickHouse on a cluster uses insert_quorum=auto, insert_quorum_parallel=0 and select_sequential_consistency=1; async_insert must not be enabled; a two-replica cluster stops appending with one replica down. - The role-catalog importer reads AUDIT_SINK through RequireMode and refuses an unset value, so a deployment on the postgres default sets it. The resolver records through RecordTx rather than raw SQL. - What is bounded is a BigQuery read window (WindowBytes, 64 MiB): a read with one instant over the budget fails, an export over 32 MiB is refused and has no time range, and the history copy reads a day outside the budget. - Migrations 18 to 21 are listed with their grants and down guards. Co-Authored-By: Claude Sonnet 5.5 --- module/docs/adr/0009-audit-store-swap.md | 218 +++++++++++++++-------- 1 file changed, 146 insertions(+), 72 deletions(-) diff --git a/module/docs/adr/0009-audit-store-swap.md b/module/docs/adr/0009-audit-store-swap.md index 4b7b4dea8..1251d51a9 100644 --- a/module/docs/adr/0009-audit-store-swap.md +++ b/module/docs/adr/0009-audit-store-swap.md @@ -41,8 +41,8 @@ for years, locked, and queried heavily: the job platform admits a global-scope job only from the privileged worker pool, which cannot enqueue on the caller's transaction, so `work.go` warns that these stay Postgres-only. It also lacks the rows the auth resolver - writes with raw SQL (`pkg/auth/pg/resolver.go`), which never pass through the - emitter. Postgres lacks everything older than its window. Every reader has + records on its resolution transaction (`pkg/auth/pg/resolver.go`), which carry + no tee job. Postgres lacks everything older than its window. Every reader has to know which copy answers which question. The requirement ADR 0006 refused to trade away still stands: no state change @@ -68,9 +68,14 @@ Postgres transaction. - `AUDIT_CONTENT_RETENTION_DAYS` — the details window of item 5; - `AUDIT_DEPLOYMENT_ID`, stamped on every record; - optionally, the relay's tuning: `AUDIT_RELAY_BATCH_SIZE` (the most events - one delivery carries; default 500) and `AUDIT_RELAY_MAX_WAIT` (how long a - partial batch waits for more events before it is delivered anyway; - default 5s, at most 60s). + one delivery carries; default 500, at most 5000) and `AUDIT_RELAY_MAX_WAIT` + (how long a partial batch waits for more events before it is delivered + anyway; default 5s, at most 60s). + + The role-catalog importer, a deploy step that records audit events apart from + the service, is the one process with no default: it refuses to run when + `AUDIT_SINK` is unset or blank. A deployment on the `postgres` default + therefore sets `AUDIT_SINK=postgres` explicitly wherever the importer runs. A missing required setting fails startup. `AUDIT_EVENTS_RETENTION_DAYS` is a ClickHouse setting only — BigQuery's events table has no expiry the kit sets @@ -103,32 +108,55 @@ Postgres transaction. is what keeps the append-only guarantee intact for every `postgres` and `both` deployment: no trigger is relaxed to make room for a delete. - A relay drains the queue: - - in **batches**; + A relay drains the queue, one relay at a time under a lease: + - in **batches**, taken oldest queue row first; - **at-least-once, keyed by event id** — a batch redelivered after a crash may leave an event in the warehouse twice, as it may in the archive, and every read returns each event once by event id (item 4); - - **ordered per organization** (platform-level events form one more ordering - key); + - **in no guaranteed order.** The relay reads a row only when its writing + transaction took its id before the oldest transaction still running, so it + never reads a row that has not committed. That gate is transaction + visibility, not sequence number: a row can be delivered ahead of a lower + sequence number whose transaction has yet to commit. Nothing depends on + delivery order: both stores key a record by event id, and every read sorts + by event time; - each batch is **written as one object to the locked archive first, then appended to the warehouse** (item 5) — the archive copy is written within the batch, not filled in later, and a failure between the two writes - repeats the archive write, not the warehouse's. The relay remembers which - writes the batch at the head of the queue has had acknowledged, so an - outage of one side does not write that batch to the other side again; only - a restart forgets, which is the redelivery above; + repeats the archive write, not the warehouse's. The relay keeps, for each + queued row, which of the two writes has acknowledged it and what it decided + about it, so an outage of one side does not write that row to the other side + again; only a restart forgets, which is the redelivery above; - the batch's queue rows are **deleted only after both writes are acknowledged**; - - a row the warehouse **permanently rejects** — one that fails alone while - the other rows of the same attempt succeed — does not stall the queue. It - moves, in one transaction, to an `audit_event_quarantine` table beside the - queue and is counted in `saas.audit_queue.quarantined`. The kit never - deletes a quarantined row; replaying one is future work. - - The queue is monitored: `saas.audit_queue.depth` and - `saas.audit_queue.oldest_age` report how much is waiting and for how long. - When the queue cannot be read they go absent rather than staying at their - last value, and each failed read counts in + - a row is **set aside only on the store's own word.** The relay quarantines + a row only when the store reports that it permanently refused that row + (`business.PermanentRowRejection`), or when the row's details cannot be + serialized. BigQuery reports a row it lists as `invalid` in a streaming + insert, a row too large for any request, and a value that cannot be + encoded; ClickHouse reports a value its driver cannot encode, and a row + refused with an error code that says the row's own content is malformed, + found by splitting the refused block down to single rows. Every other + failure — a transport or server error, a throttle, a quota, a timeout, a + failed quorum, an error the store does not recognize — is retried with + backoff and sets nothing aside, and the relay infers nothing from how often a + row failed or from what else the store accepted. After a write that names + refused rows, the relay sets those aside and writes the rest again; + - a set-aside row does not stall the queue. It moves, whole, in the same + transaction that deletes the delivered rows, to an `audit_event_quarantine` + table beside the queue, and is counted in `saas.audit_queue.quarantined`. + The reason kept with it starts with a class: `warehouse_rejected_archived:` + (the store refused the row; the archive already holds a copy, and so does the + quarantine) or `unserializable_not_archived:` (the details cannot be + serialized, so the archive's line for the event cannot be written and the + quarantine holds the only copy). The kit never deletes a quarantined row; + replaying one is not built. + + The queue is monitored: `saas.audit_queue.depth`, `saas.audit_queue.oldest_age` + and `saas.audit_queue.quarantined` report how much is waiting, for how long, + and how many rows are set aside. When the queue cannot be read — the latest + read failed, or none succeeded within twice the read interval — the three go + absent rather than staying at their last value, and each failed read counts in `saas.audit_queue.snapshot_errors`. The monitoring runs under every sink, not only a swap value: under a non-warehouse sink a non-empty queue — rows left by an earlier swap, which nothing drains — is logged at startup. @@ -140,8 +168,8 @@ Postgres transaction. one `AuditStore` interface carrying both halves: - **write** — append a batch; - **read** — list (keyset-paginated, as `QueryAuditLog` is), aggregate (as - `AggregateAuditLog`), export, and the readable-source queries - (`LatestSourceSyncRequests`). + `AggregateAuditLog`), export, and the readable-source query + (`LatestSourceSyncEvents`, which feeds `LatestSourceSyncRequests`). Two adapters ship: **BigQuery** (managed, Google Cloud) and **ClickHouse** (self-hosted or managed, any cloud or on-premises). The existing Postgres @@ -165,19 +193,38 @@ Postgres transaction. the per-event hash makes the copies provably identical. BigQuery reads run no query job. They go through the BigQuery Storage Read - API. Each read session carries a server-side row restriction: the - deployment, the caller's organization (unless the read is the platform - read), the time window, so partitions outside it are pruned, and the event - types when the read names them. The service then sorts, pages, + API. Each read session carries a server-side row restriction: the deployment, + the caller's organization (unless the read is the platform read), the time + window, so partitions outside it are pruned, and the envelope filters the read + names — event types, actor, resource, client. The service then sorts, pages, deduplicates, joins content details and aggregates the rows it streams back, - with the semantics of the Postgres reads, in bounded memory whatever the size - of the history in the window. + with the semantics of the Postgres reads. + + What is bounded is a read window, not a read. A read walks time in windows, + newest first, and counts what one window holds in memory — the content details + it joins and, for a read that must see every event of the window before it can + answer (an aggregation, an export), the events and the set that deduplicates + them — against a byte budget, `ReadConfig.WindowBytes` (64 MiB by default; the + service exposes no setting for it). A window that passes the budget is read + again as two halves. What outlives a window is the answer: a page, an + aggregation's buckets, an export's events. Three limits remain: + - A read whose events at a single instant alone pass the budget cannot be + split further and fails (`ErrReadTooDense`). + - An export is the one answer as large as the history it matches. One export + gathers at most 32 MiB of event text from either warehouse + (`AuditExportMaxBytes`) and is refused with `ResourceExhausted` past it. The + export request carries no time range, so an actor or an event type is all + that narrows it. + - The history copy (item 7) reads and holds a whole window of the store at a + time outside this budget. The reason is the grants. The service that reads is the service that appends, so it holds `bigquery.tables.updateData`. With `bigquery.jobs.create` beside it, that identity could run DELETE, UPDATE and MERGE against the store of record. So the identity is granted - `bigquery.datasets.get`, `bigquery.tables.get`, `bigquery.tables.getData`, + `bigquery.datasets.get`, `bigquery.tables.create` (the adapter creates its two + tables when they are missing), `bigquery.tables.get`, + `bigquery.tables.updateData` (the append), `bigquery.tables.getData`, `bigquery.tables.list`, and `bigquery.readsessions.create`, `.getData` and `.update`, and no job permission at all. @@ -211,8 +258,9 @@ Postgres transaction. is today: in the event type's name (`saas.approval.denied`) or in the payload's `outcome` field. How long the table keeps a row depends on the warehouse. On ClickHouse, `AUDIT_EVENTS_RETENTION_DAYS` sets the table's - TTL. On BigQuery the kit sets no expiry on the events table; a deployment - that wants one sets a table or partition expiration on its dataset; + TTL. On BigQuery the kit creates the events table with no partition + expiration, and refuses at startup an events table whose partitions expire, + since its identity holds no `bigquery.tables.update` to change one; - a **details** table holding the full details of content-class types, kept for the shorter content window, `AUDIT_CONTENT_RETENTION_DAYS` — a partition expiration on BigQuery, a table TTL on ClickHouse. @@ -228,8 +276,9 @@ Postgres transaction. still need a writer; an on-premises S3-compatible lock must be qualified before use. - Archive objects are named per batch. A retried batch writes a new object, so - the archive may hold an event more than once. Every archive reader — + Archive objects are named per batch and created only if absent. A batch + delivered again after a restart is composed anew and written as a new object, + so the archive may hold an event more than once. Every archive reader — including the history-copy verification (item 7) and any compliance export — deduplicates by event id, and the per-event hash makes duplicates provably identical. @@ -238,12 +287,28 @@ Postgres transaction. period of the archive bucket, and the events TTL on ClickHouse. 6. **ClickHouse operating requirements.** - - Inserts are batched — the relay's batches, `async_insert`, or both. A - single-row insert creates a data part per insert and ends in - too-many-parts errors. - - A single-node deployment uses MergeTree. When - `AUDIT_CLICKHOUSE_CLUSTER` is set, tables use `ReplicatedMergeTree` with - ClickHouse Keeper; the cluster must exist before the adapter starts. + - Inserts are batched: one insert per table per relay batch. A single-row + insert creates a data part per insert and ends in too-many-parts errors. + `async_insert` is not used and must not be enabled for the adapter's user or + profile: the relay deletes a queue row on the insert's acknowledgement, and + an insert acknowledged before its data is stored would let the relay delete + the only copy. + - A single-node deployment uses MergeTree and sends neither the quorum nor + the sequential-read setting below. When `AUDIT_CLICKHOUSE_CLUSTER` is set, + tables are created `ON CLUSTER` as `ReplicatedMergeTree` with ClickHouse + Keeper; the cluster must exist before the adapter starts. Every insert then + carries `insert_quorum=auto` (a majority of the replicas must hold it), + `insert_quorum_parallel=0` (one quorum insert at a time) and a 60-second + `insert_quorum_timeout`, and every read carries + `select_sequential_consistency=1`, so a read is served only by a replica that + holds every acknowledged insert; a replica that has not caught up fails the + read, which the caller retries. A majority of two replicas is both of them, + so on a two-replica cluster one replica down stops appends: the insert fails + at the quorum timeout, the relay retries with backoff, and the queue grows + until the replica returns. + - The adapter's ClickHouse user holds `CREATE TABLE`, `INSERT` and `SELECT` on + the database and no `ALTER`, `DELETE`, `UPDATE`, `TRUNCATE` or `DROP`, so it + cannot rewrite the store of record. - Tables are partitioned by month, and a table TTL expires each table at its window: `AUDIT_EVENTS_RETENTION_DAYS` for events, `AUDIT_CONTENT_RETENTION_DAYS` for details. @@ -259,16 +324,19 @@ Postgres transaction. database function (migration 20) that takes an explicit list of the verified partitions, locks them, re-counts each against the count that was verified, and refuses on any mismatch — a partition that gained a row since - verification is never dropped. `--through`, the cutoff of the copy, accepts - completed months only. From then on `audit_events` receives no new rows, and - Postgres holds only the queue. + verification is never dropped. `-through`, the cutoff of the copy, accepts + completed months (UTC) only. The copy and its read-back hold one UTC day of + the deployment's events in memory at a time, outside the read budget of + item 4. From then on `audit_events` receives no new rows, and Postgres holds + only the queue. 8. **Conformance.** Every adapter passes one shared test suite: read deduplication (after a batch is appended twice, every read returns each - event once by event id), batch append, list and aggregate parity with the Postgres implementation over the - same fixture, refusal of a query without tenant scope, and retention-class - routing (a content-class event's full details land only in the details table, - and only its content-free form in the archive). + event once by event id), batch append, list, aggregate, export and + readable-source parity with the Postgres implementation over the same + fixture, refusal of a query without tenant scope, and retention-class routing + (a content-class event's full details land only in the details table, and + only its content-free form in the archive). ## Alternatives considered @@ -282,7 +350,7 @@ The queue between the transaction and the warehouse (item 2): | Change data capture on the Postgres WAL | Still needs the Postgres row in order to capture it, and replication slots plus a connector are heavier to run than a relay over a table. | | Queue rows in `audit_events`, with the no-delete trigger relaxed for them | Weakens the append-only guarantee on the table every `postgres` and `both` deployment keeps its history in, to serve a mode those deployments do not use. | | Fill the archive from the warehouse later, on a schedule | Opens a window in which the compliance copy lags the store of record, and adds a second job to watch. | -| Conditional create-if-absent archive object names, so a retry cannot duplicate | Needs the same batch composition on every retry — more moving parts for no compliance gain. | +| Archive object names fixed by a batch's content, so a redelivery after a restart cannot duplicate | Needs the same batch composition on every redelivery — more moving parts for no compliance gain. | | Committed write streams with offsets, so a redelivery cannot duplicate in the warehouse | Stream state the relay must carry across restarts and recover on every one — more machinery for no compliance gain, since every read already deduplicates by event id, as archive readers do. | Where the adapters live (item 3): @@ -313,10 +381,13 @@ The sink shape itself: `auditsink.Open` builds the adapter and the archive writer; `audit_sink.go` starts the relay on them. The `external` refusal stays; its message, which today says Postgres is the source of truth under every value, is reworded. - The one-time history copy and the role-catalog importer resolve the sink - through the same `auditsink.Load` as the service, so a deployment configures - the swap once and neither tool writes to a different store than the service - reads; the importer never falls back silently to `audit_events`. + The one-time history copy resolves the sink through the same + `auditsink.Load` as the service, so a deployment configures the swap once and + the copy never writes to a different store than the service reads. The + role-catalog importer reads `AUDIT_SINK` alone, through + `auditsink.RequireMode`, and refuses an unset or blank value rather than + defaulting to `postgres`: under a swap value a default would write its events + to `audit_events`, where nothing reads them. - `pkg/business/audit.go` — under a swap value, the emitter's write inserts into the queue table in place of `audit_events`, for org-scoped and platform-level entries alike. `EmitTx`'s transaction contract, `VerifyAuditWiring` and the @@ -324,10 +395,11 @@ The sink shape itself: reader's grant check and the aggregate in one tenant transaction today; under a swap value it becomes grant check in Postgres, then the aggregate in the warehouse — two steps, no shared snapshot. -- `pkg/auth/pg/resolver.go` — `emitSsoJitAudit` and `emitUserRegistered` insert - into `audit_events` with raw SQL on the resolution transaction, bypassing the - emitter. Under a swap value they must go through the queue, or the swap loses - them. +- `pkg/auth/pg/resolver.go` — `emitSsoJitAudit` and `emitUserRegistered` record + their events on the resolution transaction through the emitter's `RecordTx` + rather than inserting into `audit_events` themselves. `RecordTx` writes the + record alone — the `audit_events` row, or under a swap value the queue row — + with no domain event and no tee job, so the swap does not lose them. - `pkg/infra/postgres_audit.go` — `QueryAuditLog` and `AggregateAuditLog` become the Postgres implementation of `AuditStore`. The aggregate's category dimension reads `audit_event_types` in SQL; an adapter takes it from the @@ -346,17 +418,19 @@ The sink shape itself: as well, through its own verified-partition function. Under a swap value, expiry belongs to the warehouse where it sets one — the details table on both, the events table on ClickHouse only — and to the archive's lock. -- `store/migrations` — a new queue table, under the same tenant row-level - security as `audit_events` (`audit_events_tenant`, in `1_baseline.up.sql`), - with delete granted to the relay; its down migration refuses to drop a - non-empty queue, since the rows in it are audit records no warehouse holds - yet. A second migration (21) adds the `audit_event_quarantine` table of item - 2, and migration 20 the partition-removal function of item 7. `audit_events` - itself is untouched: its - `audit_events_no_delete` and `audit_events_no_update` triggers and its - `SELECT`/`INSERT`-only grants stay as they are. Stored history in the - warehouse is guarded by the service-side scope of item 4, not by these - policies. +- `store/migrations` — migration 18 adds the queue table, under the same tenant + row-level security as `audit_events` (`audit_events_tenant`, in + `1_baseline.up.sql`), with insert granted to the writers and select and delete + to the relay; its down migration refuses to drop a non-empty queue, since the + rows in it are audit records no warehouse holds yet. Migration 19 adds the + retention class to `audit_event_types`. Migration 20 adds the + verified-partition removal function of item 7. Migration 21 adds the + `audit_event_quarantine` table of item 2, which only the relay's role can read + or insert into; its down migration refuses to drop a non-empty quarantine. + `audit_events` itself is untouched: its `audit_events_no_delete` and + `audit_events_no_update` triggers and its `SELECT`/`INSERT`-only grants stay as + they are. Stored history in the warehouse is guarded by the service-side scope + of item 4, not by these policies. **What stays:** @@ -376,8 +450,8 @@ The sink shape itself: delivered it, not when its mutation commits. - A deployment that opts in runs two more systems — the warehouse and the archive bucket — and a relay whose lag and quarantine table need watching. -- On BigQuery the events table is never expired by the kit and grows until the - deployment sets an expiry on its dataset; the locked archive, not that table, - is the compliance record. +- On BigQuery the kit never expires the events table, and refuses to start on + one whose partitions expire, so the table grows with the history; the locked + archive, not that table, is the compliance record. - Each adapter must stay at parity with the Postgres implementation; the conformance suite is what holds it there. From 7806c048eccdd6e501a6b047461432a10125c2ca Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Tue, 6 Oct 2026 15:35:29 -0400 Subject: [PATCH 38/69] docs(adr): renumber the audit migrations ADR 0009 cites to 19 through 22 Main took migration 18 for resource indicators, so the four audit migrations the store change adds are now 19 (queue), 20 (retention class), 21 (verified-partition removal) and 22 (quarantine). ADR 0009 names them by number in the history-migration item and in the store/migrations paragraph; both now match the numbers the migrations will land under. Co-Authored-By: Claude Sonnet 5.5 --- module/docs/adr/0009-audit-store-swap.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/module/docs/adr/0009-audit-store-swap.md b/module/docs/adr/0009-audit-store-swap.md index 1251d51a9..391d72424 100644 --- a/module/docs/adr/0009-audit-store-swap.md +++ b/module/docs/adr/0009-audit-store-swap.md @@ -321,7 +321,7 @@ Postgres transaction. per-row hashes. Once verified, the copied monthly partitions are removed with `DROP TABLE`, as `RunRetention` drops expired ones, so no row is deleted and the append-only triggers stay as they are. The removal is one - database function (migration 20) that takes an explicit list of the + database function (migration 21) that takes an explicit list of the verified partitions, locks them, re-counts each against the count that was verified, and refuses on any mismatch — a partition that gained a row since verification is never dropped. `-through`, the cutoff of the copy, accepts @@ -418,13 +418,13 @@ The sink shape itself: as well, through its own verified-partition function. Under a swap value, expiry belongs to the warehouse where it sets one — the details table on both, the events table on ClickHouse only — and to the archive's lock. -- `store/migrations` — migration 18 adds the queue table, under the same tenant +- `store/migrations` — migration 19 adds the queue table, under the same tenant row-level security as `audit_events` (`audit_events_tenant`, in `1_baseline.up.sql`), with insert granted to the writers and select and delete to the relay; its down migration refuses to drop a non-empty queue, since the - rows in it are audit records no warehouse holds yet. Migration 19 adds the - retention class to `audit_event_types`. Migration 20 adds the - verified-partition removal function of item 7. Migration 21 adds the + rows in it are audit records no warehouse holds yet. Migration 20 adds the + retention class to `audit_event_types`. Migration 21 adds the + verified-partition removal function of item 7. Migration 22 adds the `audit_event_quarantine` table of item 2, which only the relay's role can read or insert into; its down migration refuses to drop a non-empty quarantine. `audit_events` itself is untouched: its `audit_events_no_delete` and From 5196a0590c9890e0678d1b87cb56d9077428a77f Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Tue, 6 Oct 2026 16:28:58 -0400 Subject: [PATCH 39/69] chore(frontend): bump the published kit version for the regenerated SDK contract The audit store branch regenerated the published SDK contract under saas-sdk/generated, and its own bump to 0.3.16 collided with the 0.3.16 that main already published for an earlier contract change. The kit-version gate compares against the newest release tag and rejects changed bytes under a version the registry already serves, so @codefly-dev/saas-sdk moves to 0.3.17 in its manifest, the workspace pin and the lockfile (regenerated with npm, no other lockfile change). Co-Authored-By: Claude Sonnet 5.5 --- module/services/frontend/code/package-lock.json | 4 ++-- module/services/frontend/code/package.json | 2 +- module/services/frontend/code/packages/saas-sdk/package.json | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/module/services/frontend/code/package-lock.json b/module/services/frontend/code/package-lock.json index 9c90c5b25..19018ccd9 100644 --- a/module/services/frontend/code/package-lock.json +++ b/module/services/frontend/code/package-lock.json @@ -13,7 +13,7 @@ "dependencies": { "@base-ui/react": "^1.3.0", "@bufbuild/protobuf": "^2.11.0", - "@codefly-dev/saas-sdk": "0.3.16", + "@codefly-dev/saas-sdk": "0.3.17", "@codefly-dev/saas-ui": "0.12.4", "@codefly-dev/ui": "0.12.4", "@codefly/saas-plugin-contract": "2.4.1", @@ -15356,7 +15356,7 @@ }, "packages/saas-sdk": { "name": "@codefly-dev/saas-sdk", - "version": "0.3.16", + "version": "0.3.17", "license": "Apache-2.0", "dependencies": { "@bufbuild/protobuf": "^2.11.0", diff --git a/module/services/frontend/code/package.json b/module/services/frontend/code/package.json index cebc2ab3d..96c305773 100644 --- a/module/services/frontend/code/package.json +++ b/module/services/frontend/code/package.json @@ -36,7 +36,7 @@ "dependencies": { "@base-ui/react": "^1.3.0", "@bufbuild/protobuf": "^2.11.0", - "@codefly-dev/saas-sdk": "0.3.16", + "@codefly-dev/saas-sdk": "0.3.17", "@codefly-dev/saas-ui": "0.12.4", "@codefly-dev/ui": "0.12.4", "@codefly/saas-plugin-contract": "2.4.1", diff --git a/module/services/frontend/code/packages/saas-sdk/package.json b/module/services/frontend/code/packages/saas-sdk/package.json index 55dd02b7f..0d611fe4a 100644 --- a/module/services/frontend/code/packages/saas-sdk/package.json +++ b/module/services/frontend/code/packages/saas-sdk/package.json @@ -1,6 +1,6 @@ { "name": "@codefly-dev/saas-sdk", - "version": "0.3.16", + "version": "0.3.17", "description": "Typed Connect client, gateway-bound facades, and audit data-graph tooling for the saas-starter public API.", "repository": { "type": "git", From d0b7e4d2984f847988196d0bef647ff66093e0a4 Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Tue, 6 Oct 2026 16:43:36 -0400 Subject: [PATCH 40/69] feat(accounts,auth-gateway): resolve the cell's collector from the observability group (Refs #1022) accounts and auth-gateway used to reach an in-graph OTLP gateway through `Service("telemetry")`, with the service name typed as a Go literal and a constant `WithInsecure()`. They now read where the cell's collector is from the `observability` configuration group, which the platform delivers as `TELEMETRY_STATE` (`available` or `absent`, never empty), with `OTEL_EXPORTER_OTLP_ENDPOINT` when available and `TELEMETRY_ABSENT_REASON` when absent. The group's three keys resolve to one of three outcomes: - available with an endpoint: traces and the OTLP metric push go to it, and the transport is derived from the endpoint's scheme (`http://` is plaintext because the mesh supplies mTLS; `https://` takes the exporter's TLS default). - absent with a reason: no OTLP exporters, the reason is logged once, boot continues. A local run uses wool's own stdout tracer for traces. - anything else (state missing or unknown, available without an endpoint, absent without a reason, both set, a malformed endpoint): refuse to start, naming what is wrong. A group that did not arrive is not a cell without a collector, and reading it as one would leave a deployment converging green while exporting nothing. accounts checks this before it touches the database or Vault. The group is read only through the SDK's workspace lookup, with no fallback to a bare process variable: an ambient OTEL_EXPORTER_OTLP_ENDPOINT is conventionally the OTLP/HTTP address, not the cell's gRPC collector. Metrics now leave by one path, OTLP push. The Prometheus scrape exporter, its /metrics mount on both services and the accounts gateway-route exemption that existed only for it are removed, with their dependencies. service.name is no longer a literal. OTEL_SERVICE_NAME or OTEL_RESOURCE_ATTRIBUTES wins, and otherwise the service identity the SDK's provider holds (the one wool stamps on log lines) is used, read from the provider rather than the runtime's environment carrier. wool's tracer cannot dial TLS: wool/otel.Enable always sets an insecure transport and has no option to turn it off. An https:// endpoint is therefore refused at startup instead of being dialed in the clear and reported as TLS, until wool can dial it. This is a stopgap in this repository; the fix belongs in wool. The resolution is two small mirrored copies, one per service, because they are independent Go modules with no shared package; a module/tools test holds the copies byte-identical. `configurations/local/observability.env` and the local-dogfood example now say `TELEMETRY_STATE=absent` with a reason, and OBSERVABILITY_EXPORTER is gone. The dogfood secret example held only the collector's external headers, which nothing reads any more, so it is removed with its ignore entry. Tested: unit tests for the three outcomes in both services, including a loopback OTLP receiver that checks exported spans and metrics carry the resolved service.name. The real auth-gateway binary refuses with no group or an unknown state, and passes the telemetry step with `absent` (then stops at the unrelated missing runtime endpoint, since no Codefly runtime was present). Not exercised against a real cell collector. Co-Authored-By: Claude Opus 5.5 --- .gitignore | 1 - .../local-dogfood/observability.env.example | 24 +- .../observability.secret.env.example | 3 - module/configurations/local/observability.env | 19 +- module/services/accounts/code/go.mod | 9 +- module/services/accounts/code/go.sum | 18 - .../accounts/code/telemetry_destination.go | 267 ++++++++++++ .../code/telemetry_destination_test.go | 397 ++++++++++++++++++ .../accounts/code/telemetry_metrics.go | 49 +-- .../accounts/code/telemetry_metrics_test.go | 22 +- module/services/accounts/code/work.go | 80 ++-- .../services/accounts/code/work_auth_test.go | 19 - .../auth-gateway/code/configuration.go | 5 - .../auth-gateway/code/configuration_test.go | 19 - module/services/auth-gateway/code/go.mod | 9 +- module/services/auth-gateway/code/go.sum | 20 - module/services/auth-gateway/code/main.go | 37 +- .../code/telemetry_destination.go | 267 ++++++++++++ .../code/telemetry_destination_test.go | 376 +++++++++++++++++ .../auth-gateway/code/telemetry_metrics.go | 56 +-- .../code/telemetry_metrics_test.go | 33 +- .../telemetry_destination_mirror_test.go | 40 ++ 22 files changed, 1489 insertions(+), 281 deletions(-) delete mode 100644 configurations/local-dogfood/observability.secret.env.example create mode 100644 module/services/accounts/code/telemetry_destination.go create mode 100644 module/services/accounts/code/telemetry_destination_test.go create mode 100644 module/services/auth-gateway/code/telemetry_destination.go create mode 100644 module/services/auth-gateway/code/telemetry_destination_test.go create mode 100644 module/tools/telemetry_destination_mirror_test.go diff --git a/.gitignore b/.gitignore index f49d97580..ae9ebe8da 100644 --- a/.gitignore +++ b/.gitignore @@ -46,7 +46,6 @@ module/services/*/nix/ /configurations/local-dogfood/error-tracking.env /configurations/local-dogfood/error-tracking.secret.env /configurations/local-dogfood/observability.env -/configurations/local-dogfood/observability.secret.env /configurations/local-dogfood/abuse-protection.env /configurations/local-dogfood/abuse-protection.secret.env /configurations/local-dogfood/legal.env diff --git a/configurations/local-dogfood/observability.env.example b/configurations/local-dogfood/observability.env.example index 2ad133ad0..73ad6b219 100644 --- a/configurations/local-dogfood/observability.env.example +++ b/configurations/local-dogfood/observability.env.example @@ -1,6 +1,18 @@ -# Copy to observability.env and fill it in. SigNoz is the -# designated OTLP backend; this remains application instrumentation config. -# "debug" keeps telemetry local; "otlphttp" forwards through the collector. -OBSERVABILITY_EXPORTER=debug -OTEL_EXPORTER_OTLP_ENDPOINT= -OTEL_EXPORTER_OTLP_HEADERS= +# Copy to observability.env and fill it in. This module owns no collector: the +# cell's collector is where traces and metrics go, and the platform says where it is. +# +# A dogfood run on a laptop is not a cell, so by default it has none. The default +# below is what a plain local run uses: wool's stdout tracer stands in for traces +# and metrics are not exported. +TELEMETRY_STATE=absent +TELEMETRY_ABSENT_REASON=A local dogfood run has no cell collector; traces go to the stdout tracer and metrics are not exported. + +# To export to a collector this machine can reach (for example a cell's, through a +# port-forward), replace the two lines above with: +# +# TELEMETRY_STATE=available +# OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 +# +# The endpoint is the collector's OTLP/gRPC address. http:// is plaintext, which +# only belongs on loopback or inside the mesh; https:// is TLS. Set the endpoint +# when the state is available and the reason when it is absent, never both. diff --git a/configurations/local-dogfood/observability.secret.env.example b/configurations/local-dogfood/observability.secret.env.example deleted file mode 100644 index 87476f787..000000000 --- a/configurations/local-dogfood/observability.secret.env.example +++ /dev/null @@ -1,3 +0,0 @@ -# Copy to observability.secret.env. This file is ignored by Git. -# Example: Authorization=Basic BASE64_INSTANCE_ID_API_TOKEN -OTEL_EXPORTER_OTLP_HEADERS= diff --git a/module/configurations/local/observability.env b/module/configurations/local/observability.env index de82d1d00..ed8a6da2c 100644 --- a/module/configurations/local/observability.env +++ b/module/configurations/local/observability.env @@ -1,3 +1,16 @@ -OBSERVABILITY_EXPORTER=debug -OTEL_EXPORTER_OTLP_ENDPOINT= -OTEL_EXPORTER_OTLP_HEADERS= +# Where accounts and auth-gateway send traces and metrics, or that they send none. +# The platform delivers exactly one of two answers, and a process that gets neither +# refuses to start rather than guess: +# +# TELEMETRY_STATE=available the cell has a collector. Also sets +# OTEL_EXPORTER_OTLP_ENDPOINT to its OTLP/gRPC address, +# e.g. http://otel-collector.otel-collector.svc.cluster.local:4317 +# (http:// is plaintext because the mesh supplies mTLS; +# https:// is TLS). +# TELEMETRY_STATE=absent the cell has none. Also sets TELEMETRY_ABSENT_REASON to +# say why. +# +# A local run has no cell, so it has no collector: traces go to wool's stdout tracer +# and metrics are not exported. Logs always go to stdout. +TELEMETRY_STATE=absent +TELEMETRY_ABSENT_REASON=A local run has no cell collector; traces go to the stdout tracer and metrics are not exported. diff --git a/module/services/accounts/code/go.mod b/module/services/accounts/code/go.mod index 1baf6311e..a8374adaf 100644 --- a/module/services/accounts/code/go.mod +++ b/module/services/accounts/code/go.mod @@ -19,7 +19,6 @@ require ( github.com/google/uuid v1.6.0 github.com/grpc-ecosystem/grpc-gateway/v2 v2.30.0 github.com/jackc/pgx/v5 v5.10.0 - github.com/prometheus/client_golang v1.24.1 github.com/redis/go-redis/v9 v9.22.0 github.com/rs/cors v1.11.1 github.com/stretchr/testify v1.12.1 @@ -27,7 +26,6 @@ require ( go.opentelemetry.io/contrib/instrumentation/runtime v0.71.0 go.opentelemetry.io/otel v1.46.0 go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc v1.46.0 - go.opentelemetry.io/otel/exporters/prometheus v0.68.0 go.opentelemetry.io/otel/metric v1.46.0 go.opentelemetry.io/otel/sdk v1.46.0 go.opentelemetry.io/otel/sdk/metric v1.46.0 @@ -54,7 +52,6 @@ require ( github.com/Masterminds/semver/v3 v3.5.0 // indirect github.com/VividCortex/ewma v1.2.0 // indirect github.com/antlr4-go/antlr/v4 v4.13.1 // indirect - github.com/beorn7/perks v1.0.1 // indirect github.com/blang/semver v3.5.1+incompatible // indirect github.com/cenkalti/backoff/v5 v5.0.3 // indirect github.com/cespare/xxhash/v2 v2.3.0 // indirect @@ -98,16 +95,11 @@ require ( github.com/mattn/go-isatty v0.0.24 // indirect github.com/mattn/go-runewidth v0.0.28 // indirect github.com/mitchellh/mapstructure v1.5.0 // indirect - github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect github.com/oklog/ulid/v2 v2.1.2 // indirect github.com/philhofer/fwd v1.2.0 // indirect github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c // indirect github.com/pkg/errors v0.9.1 // indirect github.com/power-devops/perfstat v0.0.0-20260805114148-88456608a4f6 // indirect - github.com/prometheus/client_model v0.6.3 // indirect - github.com/prometheus/common v0.71.0 // indirect - github.com/prometheus/otlptranslator v1.0.0 // indirect - github.com/prometheus/procfs v0.22.0 // indirect github.com/santhosh-tekuri/jsonschema/v6 v6.0.3 // indirect github.com/shirou/gopsutil/v3 v3.24.5 // indirect github.com/shoenig/go-m1cpu v0.2.2 // indirect @@ -124,6 +116,7 @@ require ( go.opentelemetry.io/otel/exporters/stdout/stdouttrace v1.46.0 // indirect go.opentelemetry.io/otel/trace v1.46.0 // indirect go.uber.org/atomic v1.11.0 // indirect + go.yaml.in/yaml/v2 v2.4.4 // indirect go.yaml.in/yaml/v3 v3.0.5 // indirect golang.org/x/exp v0.0.0-20260824195058-e88cd73687aa // indirect golang.org/x/term v0.46.0 // indirect diff --git a/module/services/accounts/code/go.sum b/module/services/accounts/code/go.sum index 8addb8a9b..bb1012c57 100644 --- a/module/services/accounts/code/go.sum +++ b/module/services/accounts/code/go.sum @@ -36,8 +36,6 @@ github.com/VividCortex/ewma v1.2.0 h1:f58SaIzcDXrSy3kWaHNvuJgJ3Nmz59Zji6XoJR/q1o github.com/VividCortex/ewma v1.2.0/go.mod h1:nz4BbCtbLyFDeC9SUHbtcT5644juEuWfUAUnGx7j5l4= github.com/antlr4-go/antlr/v4 v4.13.1 h1:SqQKkuVZ+zWkMMNkjy5FZe5mr5WURWnlpmOuzYWrPrQ= github.com/antlr4-go/antlr/v4 v4.13.1/go.mod h1:GKmUxMtwp6ZgGwZSva4eWPC5mS6vUAmOABFgjdkM7Nw= -github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM= -github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw= github.com/blang/semver v3.5.1+incompatible h1:cQNTCjp13qL8KC3Nbxr/y2Bqb63oX6wdnnjpJbkM4JQ= github.com/blang/semver v3.5.1+incompatible/go.mod h1:kRBLl5iJ+tD4TcOOxsy/0fnwebNt5EWlYSAyrTnjyyk= github.com/blang/semver/v4 v4.0.0 h1:1PFHFE6yCCTv8C1TeyNNarDzntLi7wMI5i/pzqYIsAM= @@ -216,8 +214,6 @@ github.com/kevinburke/ssh_config v1.6.0 h1:J1FBfmuVosPHf5GRdltRLhPJtJpTlMdKTBjRg github.com/kevinburke/ssh_config v1.6.0/go.mod h1:q2RIzfka+BXARoNexmF9gkxEX7DmvbW9P4hIVx2Kg4M= github.com/keybase/go-keychain v0.0.1 h1:way+bWYa6lDppZoZcgMbYsvC7GxljxrskdNInRtuthU= github.com/keybase/go-keychain v0.0.1/go.mod h1:PdEILRW3i9D8JcdM+FmY6RwkHGnhHxXwkPPMeUgOK1k= -github.com/klauspost/compress v1.19.1 h1:VsB4HPswih7mmZ8WleSFQ75c/Ui1M4trX5oAsJnhSlk= -github.com/klauspost/compress v1.19.1/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ= github.com/klauspost/cpuid/v2 v2.3.0 h1:S4CRMLnYUhGeDFDqkGriYKdfoFlDnMtqTiI/sFzhA9Y= github.com/klauspost/cpuid/v2 v2.3.0/go.mod h1:hqwkgyIinND0mEev00jJYCxPNVRVXFQeu1XKlok6oO0= github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE= @@ -242,8 +238,6 @@ github.com/moby/patternmatcher v0.6.1 h1:qlhtafmr6kgMIJjKJMDmMWq7WLkKIo23hsrpR3x github.com/moby/patternmatcher v0.6.1/go.mod h1:hDPoyOpDY7OrrMDLaYoY3hf52gNCR/YOUYxkhApJIxc= github.com/monochromegane/go-gitignore v0.0.0-20200626010858-205db1a8cc00 h1:n6/2gBQ3RWajuToeY6ZtZTIKv2v7ThUy5KKusIT0yc0= github.com/monochromegane/go-gitignore v0.0.0-20200626010858-205db1a8cc00/go.mod h1:Pm3mSP3c5uWn86xMLZ5Sa7JB9GsEZySvHYXCTK4E9q4= -github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA= -github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ= github.com/oklog/ulid/v2 v2.1.2 h1:IEclFb9JNvzYA6MW2SCxbLzcHTVsfqm3PrqGQJH5zec= github.com/oklog/ulid/v2 v2.1.2/go.mod h1:rcEKHmBBKfef9DhnvX7y1HZBYxjXb0cP5ExxNsTT1QQ= github.com/opencontainers/go-digest v1.0.0 h1:apOUWs51W5PlhuyGyz9FCeeBIOUDA/6nW8Oi/yOhh5U= @@ -262,16 +256,6 @@ github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINE github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= github.com/power-devops/perfstat v0.0.0-20260805114148-88456608a4f6 h1:jL3a8soXdzuTCcRnKhOmtcsVOObdDTFf4O2B403HPRU= github.com/power-devops/perfstat v0.0.0-20260805114148-88456608a4f6/go.mod h1:OmDBASR4679mdNQnz2pUhc2G8CO2JrUAVFDRBDP/hJE= -github.com/prometheus/client_golang v1.24.1 h1:JnJkREXzWxUdCuPFpIWZiPispT9xVV59uiuyR2bPlnU= -github.com/prometheus/client_golang v1.24.1/go.mod h1:F+oSRECHg4sse5ucfYpYDeIv/hu68Zo0uoHKetWnzcE= -github.com/prometheus/client_model v0.6.3 h1:O0jaTVAYNxTHYInEPFJt5I3+sN8zqBtVMPTB1qyxiEo= -github.com/prometheus/client_model v0.6.3/go.mod h1:gpN5P9S7Rr6Yr92PiQ+Ixvhf6JZEkF1dnxsYL2aPBEM= -github.com/prometheus/common v0.71.0 h1:9KDAKb7Mj3HEVKyFCK6Dc/HIwlBzZIN2l7/lrHl3KK8= -github.com/prometheus/common v0.71.0/go.mod h1:CLJ5H8TEsGX8bl31BdMkfhIZ+QmZ9tBPPotUxUbfcmk= -github.com/prometheus/otlptranslator v1.0.0 h1:s0LJW/iN9dkIH+EnhiD3BlkkP5QVIUVEoIwkU+A6qos= -github.com/prometheus/otlptranslator v1.0.0/go.mod h1:vRYWnXvI6aWGpsdY/mOT/cbeVRBlPWtBNDb7kGR3uKM= -github.com/prometheus/procfs v0.22.0 h1:6q9+/JL9IKAPbCmBrv9n5O5Ty3NKnciV5X7YGw0oics= -github.com/prometheus/procfs v0.22.0/go.mod h1:CvmFr/GVhIjIvWJZW3tgkODBQMRIf0EyWMQLHCHab58= github.com/redis/go-redis/v9 v9.22.0 h1:laDvpYXTJtZLloinw1fA5Kqd6HAEH2XKxOkG/PDq2F0= github.com/redis/go-redis/v9 v9.22.0/go.mod h1:y2g0Wj8rQvuK0ELM+oxSudcLtC09JScs98I/X9gRWY4= github.com/rodaine/protogofakeit v0.1.1 h1:ZKouljuRM3A+TArppfBqnH8tGZHOwM/pjvtXe9DaXH8= @@ -333,8 +317,6 @@ go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.46.0 h1:OFnwLJr+pF3iHrlGSzb go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.46.0/go.mod h1:716wFneO0ov19A2beH5hjfh9AK5z/VWNAtDijp1Y0/g= go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.46.0 h1:w53CDeOA/Kurp7yRsegSr6pbbr759dOvJ+yNmWM6Hxs= go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.46.0/go.mod h1:BOmGMCbAtvcJiSJ+hLuhgPLdDbimnraSl8irz3iY8sY= -go.opentelemetry.io/otel/exporters/prometheus v0.68.0 h1:QOf2IftqQwITVRJpnn0M7M9ZCbgWfxz4P7i9C9yc2N4= -go.opentelemetry.io/otel/exporters/prometheus v0.68.0/go.mod h1:bgSvqu2TWGXiz7yr5UTMfObH8oqxJWHTnubQ3ef9BO4= go.opentelemetry.io/otel/exporters/stdout/stdouttrace v1.46.0 h1:KdRxPiAoMptR3vfWzvjjvutTsSiwbC2uG0496rzZNfo= go.opentelemetry.io/otel/exporters/stdout/stdouttrace v1.46.0/go.mod h1:K/qSA+3G7Eovxi4K09wzrAgkWRnosS0DAOZeEpve7sM= go.opentelemetry.io/otel/metric v1.46.0 h1:yBnkXvgV7AXFILZc5K6IZe/CBFF3OS7BJ8ov6/lj0K8= diff --git a/module/services/accounts/code/telemetry_destination.go b/module/services/accounts/code/telemetry_destination.go new file mode 100644 index 000000000..e3296805d --- /dev/null +++ b/module/services/accounts/code/telemetry_destination.go @@ -0,0 +1,267 @@ +package main + +import ( + "context" + "errors" + "fmt" + "net" + "net/url" + "strings" + + "github.com/codefly-dev/core/wool" + wooltel "github.com/codefly-dev/core/wool/otel" + codefly "github.com/codefly-dev/sdk-go" + "go.opentelemetry.io/otel/attribute" + "go.opentelemetry.io/otel/sdk/resource" +) + +// Where this process sends traces and metrics — or that it sends none. +// +// The platform tells a workload through the `observability` configuration +// group, and the group always says one of two things: +// +// TELEMETRY_STATE=available with OTEL_EXPORTER_OTLP_ENDPOINT, the cell's +// collector (OTLP/gRPC), e.g. +// http://otel-collector.otel-collector.svc.cluster.local:4317 +// TELEMETRY_STATE=absent with TELEMETRY_ABSENT_REASON, why the cell has no +// collector +// +// `http://` is plaintext on the wire because the service mesh supplies mTLS +// between workloads; `https://` is TLS, which the metrics exporter dials but +// wool's tracer cannot yet (see errWoolCannotDialTLS). This module owns no +// collector: the cell does, and a workload that cannot find it does not +// substitute one. +// +// A group that did not arrive is not a cell without a backend. Reading the first +// as the second would leave a deployment that converges green and exports +// nothing, so anything that is not exactly one of the two answers above refuses +// to start, naming what is missing. +// +// This file exists byte for byte in services/accounts/code and +// services/auth-gateway/code: the two services are independent Go modules with +// no shared package, and module/tools/telemetry_destination_mirror_test.go holds +// the copies identical. Change one, copy it over the other. + +const ( + telemetryStateKey = "TELEMETRY_STATE" + telemetryEndpointKey = "OTEL_EXPORTER_OTLP_ENDPOINT" + telemetryAbsentReasonKey = "TELEMETRY_ABSENT_REASON" + + telemetryStateAvailable = "available" + telemetryStateAbsent = "absent" + + // maxAbsentReasonLength bounds the operator-written reason before it reaches + // a log line: an unbounded value from configuration would otherwise let + // whoever writes the group forge log entries around it. + maxAbsentReasonLength = 300 +) + +// telemetryDestination is the resolved answer. Exactly one of Endpoint and +// AbsentReason is set. +type telemetryDestination struct { + // Endpoint is the collector's OTLP/gRPC address as host:port, the form the + // exporters take. Empty when the cell has no collector. + Endpoint string + // Insecure is true when the endpoint was http://, which the mesh protects. + // It is derived from the endpoint's scheme and never assumed. + Insecure bool + // AbsentReason says why the cell has no collector. Empty when Endpoint is set. + AbsentReason string +} + +// Available reports whether the cell has a collector to export to. +func (d telemetryDestination) Available() bool { + return d.Endpoint != "" +} + +// observabilityValue reads one key of the `observability` group through the +// SDK. It deliberately has no fallback to a bare process variable: an ambient +// OTEL_EXPORTER_OTLP_ENDPOINT in a shell or a pod is conventionally the +// OTLP/HTTP address, not the cell's gRPC collector, and reading it here would +// export to the wrong place without a word. +func observabilityValue(key string) string { + value, err := codefly.For(codefly.Context()).WorkspaceValue("observability", key) + if err != nil { + return "" + } + return value +} + +// configuredTelemetryDestination resolves the destination from the +// `observability` group. +func configuredTelemetryDestination() (telemetryDestination, error) { + return resolveTelemetryDestination(observabilityValue) +} + +// resolveTelemetryDestination is the whole decision, over any reader of the +// group's keys so a test can reach every outcome with nothing running. +func resolveTelemetryDestination(read func(key string) string) (telemetryDestination, error) { + state := strings.TrimSpace(read(telemetryStateKey)) + endpoint := strings.TrimSpace(read(telemetryEndpointKey)) + reason := boundedLine(read(telemetryAbsentReasonKey), maxAbsentReasonLength) + + if state == "" { + return telemetryDestination{}, errors.New( + "observability: " + telemetryStateKey + " is not set in the `observability` configuration group. " + + "The platform delivers `" + telemetryStateAvailable + "` (with " + telemetryEndpointKey + ") or `" + + telemetryStateAbsent + "` (with " + telemetryAbsentReasonKey + "); a group that did not arrive is " + + "not a cell without a collector, so this process does not start without it") + } + if state != telemetryStateAvailable && state != telemetryStateAbsent { + return telemetryDestination{}, fmt.Errorf( + "observability: %s is %q in the `observability` configuration group; it must be `%s` or `%s`", + telemetryStateKey, state, telemetryStateAvailable, telemetryStateAbsent) + } + if endpoint != "" && reason != "" { + return telemetryDestination{}, fmt.Errorf( + "observability: the `observability` configuration group sets both %s and %s; "+ + "a cell either has a collector or says why it has none", + telemetryEndpointKey, telemetryAbsentReasonKey) + } + + if state == telemetryStateAbsent { + if reason == "" { + return telemetryDestination{}, fmt.Errorf( + "observability: %s is `%s` but %s is not set in the `observability` configuration group; "+ + "an absent collector must say why", + telemetryStateKey, telemetryStateAbsent, telemetryAbsentReasonKey) + } + return telemetryDestination{AbsentReason: reason}, nil + } + + if endpoint == "" { + return telemetryDestination{}, fmt.Errorf( + "observability: %s is `%s` but %s is not set in the `observability` configuration group", + telemetryStateKey, telemetryStateAvailable, telemetryEndpointKey) + } + hostPort, insecure, err := parseCollectorEndpoint(endpoint) + if err != nil { + return telemetryDestination{}, fmt.Errorf("observability: %s: %w", telemetryEndpointKey, err) + } + return telemetryDestination{Endpoint: hostPort, Insecure: insecure}, nil +} + +// parseCollectorEndpoint turns the delivered URL into the host:port the OTLP +// gRPC exporters take, and reads the transport off its scheme. The URL is never +// echoed back: a malformed one may carry credentials. +func parseCollectorEndpoint(raw string) (hostPort string, insecure bool, err error) { + parsed, parseErr := url.Parse(raw) + if parseErr != nil { + return "", false, errors.New("is not a URL; expected http://host:port or https://host:port") + } + switch parsed.Scheme { + case "http": + insecure = true + case "https": + default: + return "", false, fmt.Errorf("must start with http:// or https://, got scheme %q", parsed.Scheme) + } + if parsed.Hostname() == "" { + return "", false, errors.New("has no host") + } + if parsed.User != nil { + return "", false, errors.New("must not carry credentials") + } + // OTLP/gRPC addresses a collector, not a path: a path or a query is the + // OTLP/HTTP form (/v1/traces), which this exporter does not speak. + if (parsed.Path != "" && parsed.Path != "/") || parsed.RawQuery != "" || parsed.Fragment != "" { + return "", false, errors.New("must be the collector's origin only, with no path, query or fragment") + } + port := parsed.Port() + if port == "" { + port = "80" + if !insecure { + port = "443" + } + } + return net.JoinHostPort(parsed.Hostname(), port), insecure, nil +} + +// boundedLine collapses a free-text value to one bounded line. +func boundedLine(value string, limit int) string { + line := strings.Join(strings.Fields(value), " ") + if runes := []rune(line); len(runes) > limit { + return string(runes[:limit]) + "…" + } + return line +} + +// telemetryResource is the one resource both signals carry. service.name is +// whatever the environment says — OTEL_SERVICE_NAME, or service.name in +// OTEL_RESOURCE_ATTRIBUTES — and otherwise the service identity the Codefly SDK +// holds. No name is typed here: a literal would override the environment, so the +// platform's identity could never reach the backend. +func telemetryResource(ctx context.Context, identity string) (*resource.Resource, error) { + options := []resource.Option{resource.WithService()} + if identity != "" { + options = append(options, resource.WithAttributes(attribute.String("service.name", identity))) + } + // Last, so what the environment declares wins over the identity above. + options = append(options, resource.WithFromEnv()) + return resource.New(ctx, options...) +} + +// currentTelemetryResource is telemetryResource for this process. +func currentTelemetryResource(ctx context.Context) (*resource.Resource, error) { + return telemetryResource(ctx, codeflyServiceName(codefly.Context())) +} + +// codeflyServiceName is the service identity the Codefly SDK holds: the +// `module/service` identity of the provider it built at Init, which wool stamps on +// every log line from this process. It is empty when the SDK has not been +// initialised. The SDK exposes no accessor of its own, and the environment +// carrier it reads this from is not product code's to read, so this asks the +// provider. +func codeflyServiceName(ctx context.Context) string { + provider, ok := ctx.Value(wool.ProviderKey).(*wool.Provider) + if !ok || provider == nil { + return "" + } + source := provider.Get(ctx).Source() + if source == nil { + return "" + } + // An identity with no module reads `/service`; with neither, `/`. + return strings.Trim(strings.TrimSpace(source.Unique), "/") +} + +// telemetryServiceName is the service.name the resource resolved to. +func telemetryServiceName(res *resource.Resource) string { + if value, ok := res.Set().Value(attribute.Key("service.name")); ok { + return value.AsString() + } + return "" +} + +// errWoolCannotDialTLS names the one transport the tracer cannot serve. +var errWoolCannotDialTLS = errors.New( + "the collector endpoint is https://, but wool's OTLP tracer cannot dial TLS: wool/otel.Enable " + + "always sets an insecure transport and offers no option to turn it off. Deliver an http:// " + + "endpoint (in-mesh, where the mesh supplies mTLS) until wool can dial TLS") + +// enableTracing registers wool's tracer for the destination: OTLP/gRPC to the +// cell's collector when there is one; wool's own stdout tracer when there is +// none and this is a local run; nothing otherwise. A nil provider means nothing +// was registered. +func enableTracing(ctx context.Context, destination telemetryDestination, local bool) (*wooltel.Provider, error) { + res, err := currentTelemetryResource(ctx) + if err != nil { + return nil, fmt.Errorf("resolve telemetry resource: %w", err) + } + options := []wooltel.Option{wooltel.WithServiceName(telemetryServiceName(res))} + switch { + case destination.Available(): + // wool dials plaintext unless told otherwise and cannot be told + // otherwise, so an https:// endpoint is refused rather than dialed in + // the clear and reported as TLS. + if !destination.Insecure { + return nil, errWoolCannotDialTLS + } + options = append(options, wooltel.WithEndpoint(destination.Endpoint), wooltel.WithInsecure()) + case local: + options = append(options, wooltel.WithStdout()) + default: + return nil, nil + } + return wooltel.Enable(options...) +} diff --git a/module/services/accounts/code/telemetry_destination_test.go b/module/services/accounts/code/telemetry_destination_test.go new file mode 100644 index 000000000..c2d1e4d24 --- /dev/null +++ b/module/services/accounts/code/telemetry_destination_test.go @@ -0,0 +1,397 @@ +package main + +import ( + "context" + "net" + "strings" + "testing" + "time" + + "github.com/codefly-dev/core/wool" + "github.com/stretchr/testify/require" + "go.opentelemetry.io/otel" + collectortracev1 "go.opentelemetry.io/proto/otlp/collector/trace/v1" + "google.golang.org/grpc" +) + +// group is a reader over a fixed set of `observability` keys. +func group(values map[string]string) func(string) string { + return func(key string) string { return values[key] } +} + +const collector = "http://otel-collector.otel-collector.svc.cluster.local:4317" + +// The first outcome: the platform delivered a collector. +func TestTelemetryDestinationAvailableExportsToTheEndpoint(t *testing.T) { + destination, err := resolveTelemetryDestination(group(map[string]string{ + "TELEMETRY_STATE": "available", + "OTEL_EXPORTER_OTLP_ENDPOINT": collector, + })) + require.NoError(t, err) + require.True(t, destination.Available()) + require.Equal(t, "otel-collector.otel-collector.svc.cluster.local:4317", destination.Endpoint) + require.True(t, destination.Insecure, "http:// is plaintext on the wire; the mesh supplies mTLS") + require.Empty(t, destination.AbsentReason) +} + +// The transport comes from the endpoint's scheme, never from a constant. +func TestTelemetryDestinationReadsTransportFromTheScheme(t *testing.T) { + for endpoint, want := range map[string]struct { + hostPort string + insecure bool + }{ + "http://collector.example:4317": {"collector.example:4317", true}, + "https://collector.example:4317": {"collector.example:4317", false}, + "https://collector.example": {"collector.example:443", false}, + "http://collector.example": {"collector.example:80", true}, + "http://collector.example:4317/": {"collector.example:4317", true}, + " http://collector.example:4317 ": {"collector.example:4317", true}, + "http://[2001:db8::1]:4317": {"[2001:db8::1]:4317", true}, + "https://10.0.0.7:4317": {"10.0.0.7:4317", false}, + "http://otel-collector.ns.svc:4317/": {"otel-collector.ns.svc:4317", true}, + } { + destination, err := resolveTelemetryDestination(group(map[string]string{ + "TELEMETRY_STATE": "available", + "OTEL_EXPORTER_OTLP_ENDPOINT": endpoint, + })) + require.NoError(t, err, endpoint) + require.Equal(t, want.hostPort, destination.Endpoint, endpoint) + require.Equal(t, want.insecure, destination.Insecure, endpoint) + } +} + +// The second outcome: the cell has no collector, and says why. The process boots +// and exports nothing. +func TestTelemetryDestinationAbsentCarriesTheReason(t *testing.T) { + destination, err := resolveTelemetryDestination(group(map[string]string{ + "TELEMETRY_STATE": "absent", + "TELEMETRY_ABSENT_REASON": "This cell's bucket has no ops cell, so no collector is rendered.", + })) + require.NoError(t, err) + require.False(t, destination.Available()) + require.Empty(t, destination.Endpoint) + require.Equal(t, "This cell's bucket has no ops cell, so no collector is rendered.", destination.AbsentReason) +} + +// The reason is operator-written free text that reaches a log line. +func TestTelemetryDestinationAbsentReasonIsOneBoundedLine(t *testing.T) { + destination, err := resolveTelemetryDestination(group(map[string]string{ + "TELEMETRY_STATE": "absent", + "TELEMETRY_ABSENT_REASON": "no collector\n2026-01-01 FORGED entry\r\n" + strings.Repeat("x", 1000), + })) + require.NoError(t, err) + require.NotContains(t, destination.AbsentReason, "\n") + require.NotContains(t, destination.AbsentReason, "\r") + require.LessOrEqual(t, len([]rune(destination.AbsentReason)), maxAbsentReasonLength+1) +} + +// The third outcome: anything else refuses to start, naming what is wrong. +// Configuration that did not arrive is never read as "no collector". +func TestTelemetryDestinationRefusesEverythingElse(t *testing.T) { + for name, tc := range map[string]struct { + values map[string]string + want string + }{ + "state missing": { + values: map[string]string{}, + want: "TELEMETRY_STATE is not set", + }, + "state missing though an ambient endpoint is present": { + values: map[string]string{"OTEL_EXPORTER_OTLP_ENDPOINT": collector}, + want: "TELEMETRY_STATE is not set", + }, + "state missing though a reason is present": { + values: map[string]string{"TELEMETRY_ABSENT_REASON": "no collector"}, + want: "TELEMETRY_STATE is not set", + }, + "state blank": { + values: map[string]string{"TELEMETRY_STATE": " "}, + want: "TELEMETRY_STATE is not set", + }, + "state unknown": { + values: map[string]string{"TELEMETRY_STATE": "disabled", "TELEMETRY_ABSENT_REASON": "no collector"}, + want: `TELEMETRY_STATE is "disabled"`, + }, + "state is not matched case-insensitively": { + values: map[string]string{"TELEMETRY_STATE": "Available", "OTEL_EXPORTER_OTLP_ENDPOINT": collector}, + want: `TELEMETRY_STATE is "Available"`, + }, + "available without an endpoint": { + values: map[string]string{"TELEMETRY_STATE": "available"}, + want: "OTEL_EXPORTER_OTLP_ENDPOINT is not set", + }, + "available with only a reason": { + values: map[string]string{"TELEMETRY_STATE": "available", "TELEMETRY_ABSENT_REASON": "no collector"}, + want: "OTEL_EXPORTER_OTLP_ENDPOINT is not set", + }, + "absent without a reason": { + values: map[string]string{"TELEMETRY_STATE": "absent"}, + want: "TELEMETRY_ABSENT_REASON is not set", + }, + "absent with a blank reason": { + values: map[string]string{"TELEMETRY_STATE": "absent", "TELEMETRY_ABSENT_REASON": " \n "}, + want: "TELEMETRY_ABSENT_REASON is not set", + }, + "absent with only an endpoint": { + values: map[string]string{"TELEMETRY_STATE": "absent", "OTEL_EXPORTER_OTLP_ENDPOINT": collector}, + want: "TELEMETRY_ABSENT_REASON is not set", + }, + "available with both an endpoint and a reason": { + values: map[string]string{ + "TELEMETRY_STATE": "available", "OTEL_EXPORTER_OTLP_ENDPOINT": collector, + "TELEMETRY_ABSENT_REASON": "no collector", + }, + want: "sets both OTEL_EXPORTER_OTLP_ENDPOINT and TELEMETRY_ABSENT_REASON", + }, + "absent with both an endpoint and a reason": { + values: map[string]string{ + "TELEMETRY_STATE": "absent", "OTEL_EXPORTER_OTLP_ENDPOINT": collector, + "TELEMETRY_ABSENT_REASON": "no collector", + }, + want: "sets both OTEL_EXPORTER_OTLP_ENDPOINT and TELEMETRY_ABSENT_REASON", + }, + "endpoint without a scheme": { + values: map[string]string{"TELEMETRY_STATE": "available", "OTEL_EXPORTER_OTLP_ENDPOINT": "otel-collector.ns.svc:4317"}, + want: "must start with http:// or https://", + }, + "endpoint with another scheme": { + values: map[string]string{"TELEMETRY_STATE": "available", "OTEL_EXPORTER_OTLP_ENDPOINT": "grpc://otel-collector.ns.svc:4317"}, + want: `got scheme "grpc"`, + }, + "endpoint without a host": { + values: map[string]string{"TELEMETRY_STATE": "available", "OTEL_EXPORTER_OTLP_ENDPOINT": "http://:4317"}, + want: "has no host", + }, + "endpoint naming an OTLP/HTTP path": { + values: map[string]string{"TELEMETRY_STATE": "available", "OTEL_EXPORTER_OTLP_ENDPOINT": "http://collector.example:4318/v1/traces"}, + want: "origin only", + }, + "endpoint with a query": { + values: map[string]string{"TELEMETRY_STATE": "available", "OTEL_EXPORTER_OTLP_ENDPOINT": "http://collector.example:4317?x=1"}, + want: "origin only", + }, + } { + t.Run(name, func(t *testing.T) { + destination, err := resolveTelemetryDestination(group(tc.values)) + require.Error(t, err) + require.ErrorContains(t, err, tc.want) + require.False(t, destination.Available(), "a refusal must never read as a usable destination") + require.Empty(t, destination.AbsentReason, "a refusal must never read as an absent collector") + }) + } +} + +// A refusal quotes no part of an endpoint, which may carry credentials. +func TestTelemetryDestinationRefusalDoesNotEchoCredentials(t *testing.T) { + _, err := resolveTelemetryDestination(group(map[string]string{ + "TELEMETRY_STATE": "available", + "OTEL_EXPORTER_OTLP_ENDPOINT": "http://user:hunter2@collector.example:4317", + })) + require.ErrorContains(t, err, "must not carry credentials") + require.NotContains(t, err.Error(), "hunter2") +} + +// The group is read through the SDK's workspace-configuration lookup, by the +// names the platform delivers them under. A bare process variable of the same +// name is not the cell's answer. +func TestConfiguredTelemetryDestinationReadsTheObservabilityGroup(t *testing.T) { + const prefix = "CODEFLY__WORKSPACE_CONFIGURATION__OBSERVABILITY__" + for _, key := range []string{"TELEMETRY_STATE", "OTEL_EXPORTER_OTLP_ENDPOINT", "TELEMETRY_ABSENT_REASON"} { + t.Setenv(key, "") + t.Setenv(prefix+key, "") + } + + _, err := configuredTelemetryDestination() + require.ErrorContains(t, err, "TELEMETRY_STATE is not set") + + // Ambient variables a shell or a pod may carry are not the group. + t.Setenv("TELEMETRY_STATE", "available") + t.Setenv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://ambient.example:4318") + _, err = configuredTelemetryDestination() + require.ErrorContains(t, err, "TELEMETRY_STATE is not set") + t.Setenv("TELEMETRY_STATE", "") + t.Setenv("OTEL_EXPORTER_OTLP_ENDPOINT", "") + + t.Setenv(prefix+"TELEMETRY_STATE", "available") + t.Setenv(prefix+"OTEL_EXPORTER_OTLP_ENDPOINT", collector) + destination, err := configuredTelemetryDestination() + require.NoError(t, err) + require.Equal(t, "otel-collector.otel-collector.svc.cluster.local:4317", destination.Endpoint) + require.True(t, destination.Insecure) + + t.Setenv(prefix+"TELEMETRY_STATE", "absent") + t.Setenv(prefix+"OTEL_EXPORTER_OTLP_ENDPOINT", "") + t.Setenv(prefix+"TELEMETRY_ABSENT_REASON", "a local run has no cell collector") + destination, err = configuredTelemetryDestination() + require.NoError(t, err) + require.False(t, destination.Available()) + require.Equal(t, "a local run has no cell collector", destination.AbsentReason) +} + +// Configuration that cannot work is refused before the process acquires +// anything, including in a local run: a group that did not arrive is not a cell +// without a collector anywhere. +func TestStartupRefusesAnObservabilityGroupThatDidNotArrive(t *testing.T) { + const prefix = "CODEFLY__WORKSPACE_CONFIGURATION__OBSERVABILITY__" + for _, key := range []string{"TELEMETRY_STATE", "OTEL_EXPORTER_OTLP_ENDPOINT", "TELEMETRY_ABSENT_REASON"} { + t.Setenv(prefix+key, "") + } + // The key-custody requirement answers first outside a local run; satisfy it + // so the observability requirement is what is under test. + t.Setenv("CODEFLY__WORKSPACE_CONFIGURATION__VAULT__VAULT_KEY_CUSTODY", "seed-signing-key") + for _, local := range []bool{true, false} { + require.ErrorContains(t, requireStartupConfiguration(local), "TELEMETRY_STATE is not set", "local=%v", local) + } + + t.Setenv(prefix+"TELEMETRY_STATE", "absent") + t.Setenv(prefix+"TELEMETRY_ABSENT_REASON", "a local run has no cell collector") + require.NoError(t, requireStartupConfiguration(true)) + require.NoError(t, requireStartupConfiguration(false)) +} + +// service.name is never a literal typed here: the environment wins, then the +// service's own Codefly identity. +func TestTelemetryResourceServiceName(t *testing.T) { + resolve := func(t *testing.T, identity string) string { + t.Helper() + res, err := telemetryResource(t.Context(), identity) + require.NoError(t, err) + return telemetryServiceName(res) + } + + t.Run("falls back to the Codefly service identity", func(t *testing.T) { + t.Setenv("OTEL_SERVICE_NAME", "") + t.Setenv("OTEL_RESOURCE_ATTRIBUTES", "") + require.Equal(t, "saas-starter/accounts", resolve(t, "saas-starter/accounts")) + }) + t.Run("OTEL_SERVICE_NAME wins over the identity", func(t *testing.T) { + t.Setenv("OTEL_SERVICE_NAME", "named-by-the-platform") + t.Setenv("OTEL_RESOURCE_ATTRIBUTES", "") + require.Equal(t, "named-by-the-platform", resolve(t, "saas-starter/accounts")) + }) + t.Run("OTEL_RESOURCE_ATTRIBUTES can set it", func(t *testing.T) { + t.Setenv("OTEL_SERVICE_NAME", "") + t.Setenv("OTEL_RESOURCE_ATTRIBUTES", "service.name=attributed-by-the-platform,deployment.environment=test") + require.Equal(t, "attributed-by-the-platform", resolve(t, "saas-starter/accounts")) + }) + t.Run("without either, the SDK default rather than a literal", func(t *testing.T) { + t.Setenv("OTEL_SERVICE_NAME", "") + t.Setenv("OTEL_RESOURCE_ATTRIBUTES", "") + require.True(t, strings.HasPrefix(resolve(t, ""), "unknown_service"), resolve(t, "")) + }) +} + +// The identity is the one the SDK built its provider with, the same one wool +// stamps on this process's log lines — read from the provider, not from the +// runtime's environment carrier. +func TestCodeflyServiceNameIsTheSDKProviderIdentity(t *testing.T) { + carrying := func(unique string) context.Context { + return wool.New(t.Context(), &wool.Resource{Kind: "service", Unique: unique}).Inject(t.Context()) + } + require.Equal(t, "saas-starter/accounts", codeflyServiceName(carrying("saas-starter/accounts"))) + require.Equal(t, "accounts", codeflyServiceName(carrying("/accounts")), "an identity with no module") + require.Empty(t, codeflyServiceName(carrying("/")), "an identity with neither") + require.Empty(t, codeflyServiceName(t.Context()), "no provider: the SDK was not initialised") +} + +// wool's tracer speaks plaintext only. An https:// endpoint is refused rather +// than dialed in the clear and reported as TLS. +func TestEnableTracingRefusesATransportWoolCannotDial(t *testing.T) { + provider, err := enableTracing(t.Context(), + telemetryDestination{Endpoint: "collector.example:4317", Insecure: false}, false) + require.ErrorIs(t, err, errWoolCannotDialTLS) + require.Nil(t, provider) +} + +func TestEnableTracingRegistersOnlyWhatTheDestinationCalls(t *testing.T) { + restore := func(t *testing.T) { + t.Helper() + previous := otel.GetTracerProvider() + t.Cleanup(func() { + wool.RegisterTelemetry(nil) + otel.SetTracerProvider(previous) + }) + } + absent := telemetryDestination{AbsentReason: "a local run has no cell collector"} + + t.Run("absent outside a local run registers no tracer", func(t *testing.T) { + restore(t) + provider, err := enableTracing(t.Context(), absent, false) + require.NoError(t, err) + require.Nil(t, provider) + require.False(t, wool.TelemetryEnabled()) + }) + t.Run("absent in a local run uses wool's stdout tracer", func(t *testing.T) { + restore(t) + provider, err := enableTracing(t.Context(), absent, true) + require.NoError(t, err) + require.NotNil(t, provider) + require.True(t, wool.TelemetryEnabled()) + shutdown(t, provider) + }) + t.Run("available over http exports spans to the collector, named by the environment", func(t *testing.T) { + restore(t) + t.Setenv("OTEL_SERVICE_NAME", "named-by-the-platform") + t.Setenv("OTEL_RESOURCE_ATTRIBUTES", "") + endpoint, capture := startTraceCapture(t) + provider, err := enableTracing(t.Context(), telemetryDestination{Endpoint: endpoint, Insecure: true}, false) + require.NoError(t, err) + require.NotNil(t, provider) + require.True(t, wool.TelemetryEnabled()) + + _, span := otel.Tracer("test").Start(t.Context(), "operation") + span.End() + shutdown(t, provider) // flushes the batch + + select { + case request := <-capture.requests: + var name string + for _, resourceSpans := range request.GetResourceSpans() { + for _, attribute := range resourceSpans.GetResource().GetAttributes() { + if attribute.GetKey() == "service.name" { + name = attribute.GetValue().GetStringValue() + } + } + } + require.Equal(t, "named-by-the-platform", name) + case <-time.After(5 * time.Second): + t.Fatal("timed out waiting for OTLP spans") + } + }) +} + +type traceCapture struct { + collectortracev1.UnimplementedTraceServiceServer + requests chan *collectortracev1.ExportTraceServiceRequest +} + +func (c *traceCapture) Export( + _ context.Context, + request *collectortracev1.ExportTraceServiceRequest, +) (*collectortracev1.ExportTraceServiceResponse, error) { + c.requests <- request + return &collectortracev1.ExportTraceServiceResponse{}, nil +} + +// startTraceCapture serves an in-process OTLP/gRPC trace receiver on loopback. +func startTraceCapture(t *testing.T) (string, *traceCapture) { + t.Helper() + listener, err := net.Listen("tcp", "127.0.0.1:0") + require.NoError(t, err) + capture := &traceCapture{requests: make(chan *collectortracev1.ExportTraceServiceRequest, 4)} + server := grpc.NewServer() + collectortracev1.RegisterTraceServiceServer(server, capture) + go func() { _ = server.Serve(listener) }() + t.Cleanup(func() { + server.Stop() + _ = listener.Close() + }) + return listener.Addr().String(), capture +} + +func shutdown(t *testing.T, provider interface{ Shutdown(context.Context) error }) { + t.Helper() + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + require.NoError(t, provider.Shutdown(ctx)) +} diff --git a/module/services/accounts/code/telemetry_metrics.go b/module/services/accounts/code/telemetry_metrics.go index b088b2346..9ab41d0ea 100644 --- a/module/services/accounts/code/telemetry_metrics.go +++ b/module/services/accounts/code/telemetry_metrics.go @@ -2,59 +2,41 @@ package main import ( "context" - "net/http" runtimemetrics "runtime/metrics" "time" - "github.com/prometheus/client_golang/prometheus" - "github.com/prometheus/client_golang/prometheus/promhttp" otelruntime "go.opentelemetry.io/contrib/instrumentation/runtime" "go.opentelemetry.io/otel" - "go.opentelemetry.io/otel/attribute" "go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc" - otelprometheus "go.opentelemetry.io/otel/exporters/prometheus" otelmetric "go.opentelemetry.io/otel/metric" "go.opentelemetry.io/otel/sdk/metric" - "go.opentelemetry.io/otel/sdk/resource" ) +// otelMetrics owns the process's MeterProvider. Metrics leave by one path only: +// an OTLP push to the cell's collector, which is the record for traces and +// metrics alike. There is no scrape endpoint to mount, expose or exempt. type otelMetrics struct { provider *metric.MeterProvider - handler http.Handler } -func enableOTELMetrics(ctx context.Context, serviceName, endpoint string) (*otelMetrics, error) { - exporter, err := otlpmetricgrpc.New( - ctx, - otlpmetricgrpc.WithEndpoint(endpoint), - otlpmetricgrpc.WithInsecure(), - ) - if err != nil { - return nil, err +func enableOTELMetrics(ctx context.Context, destination telemetryDestination) (*otelMetrics, error) { + options := []otlpmetricgrpc.Option{otlpmetricgrpc.WithEndpoint(destination.Endpoint)} + if destination.Insecure { + // http://: plaintext on the wire, because the mesh supplies mTLS. An + // https:// endpoint takes the exporter's default, TLS. + options = append(options, otlpmetricgrpc.WithInsecure()) } - registry := prometheus.NewRegistry() - scrapeExporter, err := otelprometheus.New( - otelprometheus.WithRegisterer(registry), - otelprometheus.WithResourceAsConstantLabels( - attribute.NewAllowKeysFilter("service.name"), - ), - ) + exporter, err := otlpmetricgrpc.New(ctx, options...) if err != nil { return nil, err } - res, err := resource.New( - ctx, - resource.WithService(), - resource.WithFromEnv(), - resource.WithAttributes(attribute.String("service.name", serviceName)), - ) + res, err := currentTelemetryResource(ctx) if err != nil { return nil, err } provider := metric.NewMeterProvider( metric.WithResource(res), metric.WithReader(metric.NewPeriodicReader(exporter, metric.WithInterval(30*time.Second))), - metric.WithReader(scrapeExporter), ) if err := otelruntime.Start(otelruntime.WithMeterProvider(provider)); err != nil { _ = provider.Shutdown(ctx) @@ -65,10 +47,7 @@ func enableOTELMetrics(ctx context.Context, serviceName, endpoint string) (*otel return nil, err } otel.SetMeterProvider(provider) - return &otelMetrics{ - provider: provider, - handler: promhttp.HandlerFor(registry, promhttp.HandlerOpts{}), - }, nil + return &otelMetrics{provider: provider}, nil } func startGCActivityMetrics(provider *metric.MeterProvider) error { @@ -102,10 +81,6 @@ func startGCActivityMetrics(provider *metric.MeterProvider) error { return err } -func (m *otelMetrics) Handler() http.Handler { - return m.handler -} - func (m *otelMetrics) Shutdown(ctx context.Context) error { return m.provider.Shutdown(ctx) } diff --git a/module/services/accounts/code/telemetry_metrics_test.go b/module/services/accounts/code/telemetry_metrics_test.go index 0429dcd90..fd7fd94af 100644 --- a/module/services/accounts/code/telemetry_metrics_test.go +++ b/module/services/accounts/code/telemetry_metrics_test.go @@ -60,6 +60,7 @@ func (versionConnectHandler) Version( func TestTelemetryMetricsExportRuntimeAndUnsampledRED(t *testing.T) { endpoint, capture := startMetricCapture(t) t.Setenv("OTEL_RESOURCE_ATTRIBUTES", "service.instance.id=test-instance") + t.Setenv("OTEL_SERVICE_NAME", "named-by-the-platform") previousMeterProvider := otel.GetMeterProvider() previousTracerProvider := otel.GetTracerProvider() @@ -70,7 +71,7 @@ func TestTelemetryMetricsExportRuntimeAndUnsampledRED(t *testing.T) { ) otel.SetTracerProvider(tracerProvider) - metrics, err := enableOTELMetrics(t.Context(), "test-service", endpoint) + metrics, err := enableOTELMetrics(t.Context(), telemetryDestination{Endpoint: endpoint, Insecure: true}) require.NoError(t, err) require.NotNil(t, metrics) t.Cleanup(func() { @@ -89,6 +90,7 @@ func TestTelemetryMetricsExportRuntimeAndUnsampledRED(t *testing.T) { require.NoError(t, metrics.provider.ForceFlush(t.Context())) request := receiveMetrics(t, capture.requests) require.Equal(t, "test-instance", resourceAttribute(request, "service.instance.id")) + require.Equal(t, "named-by-the-platform", resourceAttribute(request, "service.name")) metricNames := exportedMetricNames(request) require.Contains(t, metricNames, "go.goroutine.count") require.Contains(t, metricNames, "go.memory.allocated") @@ -107,24 +109,6 @@ func TestTelemetryMetricsExportRuntimeAndUnsampledRED(t *testing.T) { require.Equal(t, uint64(1), connectCounts["permission_denied"]) require.Empty(t, spanExporter.GetSpans()) - - recorder := httptest.NewRecorder() - metrics.Handler().ServeHTTP( - recorder, - httptest.NewRequest(http.MethodGet, "http://accounts.internal/metrics", nil), - ) - require.Equal(t, http.StatusOK, recorder.Code) - require.Contains(t, recorder.Header().Get("Content-Type"), "text/plain") - body := recorder.Body.String() - require.Contains(t, body, "go_goroutine_count") - require.Contains(t, body, "go_memory_allocated") - require.Contains(t, body, "go_memory_gc_goal") - require.Contains(t, body, "go_gc_cycle_count") - require.Contains(t, body, "go_gc_pause_cpu_time") - require.Contains(t, body, "rpc_server_call_duration_seconds_count") - require.Contains(t, body, `rpc_method="grpc.health.v1.Health/Check"`) - require.Contains(t, body, `rpc_response_status_code="OK"`) - require.Contains(t, body, `service_name="test-service"`) } func startMetricCapture(t *testing.T) (string, *metricCapture) { diff --git a/module/services/accounts/code/work.go b/module/services/accounts/code/work.go index 3006cc8d0..b27571e53 100644 --- a/module/services/accounts/code/work.go +++ b/module/services/accounts/code/work.go @@ -38,7 +38,6 @@ import ( "time" "github.com/codefly-dev/core/wool" - wooltel "github.com/codefly-dev/core/wool/otel" codefly "github.com/codefly-dev/sdk-go" "github.com/google/uuid" "go.opentelemetry.io/otel" @@ -83,33 +82,22 @@ func doWork(ctx context.Context) (Clean, error) { return nil, err } - // When external observability is configured, OpenTelemetry always targets - // the in-graph collector. Codefly owns its host and port; the accounts - // process never reads or hardcodes a collector address. - var otelProvider *wooltel.Provider + // Traces and metrics go to the cell's collector, whose address the platform + // delivers in the `observability` configuration group — or the group says the + // cell has none. Either answer was already required before anything was + // acquired (requireStartupConfiguration); this reads it again to act on it. + telemetryDestination, err := configuredTelemetryDestination() + if err != nil { + return nil, err + } + // A local run with no cell collector gets wool's own stdout tracer instead. + otelProvider, err := enableTracing(ctx, telemetryDestination, codefly.IsLocal()) + if err != nil { + return nil, fmt.Errorf("configure OTEL tracing: %w", err) + } var otelMetricProvider *otelMetrics - if observabilityEnabled() { - collectorNetwork, err := codefly.For(ctx). - Service("telemetry"). - Endpoint("grpc"). - API("grpc"). - ResolveNetworkInstance() - if err != nil { - return nil, fmt.Errorf("resolve telemetry collector through Codefly: %w", err) - } - p, oerr := wooltel.Enable( - wooltel.WithServiceName("saas-starter-api"), - wooltel.WithEndpoint(collectorNetwork.Host), - wooltel.WithInsecure(), - ) - if oerr != nil { - return nil, fmt.Errorf("configure OTEL tracing: %w", oerr) - } - otelProvider = p - w.Info("OTEL enabled", - wool.Field("endpoint", collectorNetwork.Host), - wool.Field("service.name", "saas-starter-api")) - metricProvider, oerr := enableOTELMetrics(ctx, "saas-starter-api", collectorNetwork.Host) + if telemetryDestination.Available() { + metricProvider, oerr := enableOTELMetrics(ctx, telemetryDestination) if oerr != nil { shutdownCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second) _ = otelProvider.Shutdown(shutdownCtx) @@ -117,8 +105,12 @@ func doWork(ctx context.Context) (Clean, error) { return nil, fmt.Errorf("configure OTEL metrics: %w", oerr) } otelMetricProvider = metricProvider - // codefly:gateway-route-exempt the OTEL scrape endpoint; the collector reaches it over the mesh and it must never be public - adapters.RegisterHTTPRoute("/metrics", otelMetricProvider.Handler()) + w.Info("OTEL enabled", + wool.Field("endpoint", telemetryDestination.Endpoint), + wool.Field("insecure", telemetryDestination.Insecure)) + } else { + w.Info("OTEL export disabled: the cell has no collector", + wool.Field("reason", telemetryDestination.AbsentReason)) } store, err := infra.NewPostgresStore(ctx) @@ -171,10 +163,10 @@ func doWork(ctx context.Context) (Clean, error) { } jobStore := infra.NewPostgresJobStore(jobWorkerPool) // Durable job-operations metrics are enabled only when OTEL metrics are, i.e. - // otelMetricProvider != nil (observabilityEnabled()). With observability off - // the global meter is a no-op, so building the monitor would poll the job - // store every interval to feed instruments nothing can read; keep it nil and - // let the Start/Shutdown nil-checks below skip it entirely. + // otelMetricProvider != nil (the cell has a collector). Without one the global + // meter is a no-op, so building the monitor would poll the job store every + // interval to feed instruments nothing can read; keep it nil and let the + // Start/Shutdown nil-checks below skip it entirely. jobOperationsMonitor, err := newDurableJobMetricsMonitor(otelMetricProvider != nil, jobStore) if err != nil { return nil, fmt.Errorf("configure durable job metrics: %w", err) @@ -1664,13 +1656,9 @@ func workspaceEnv(configuration, key string) string { return os.Getenv(key) } -func observabilityEnabled() bool { - return strings.TrimSpace(workspaceEnv("observability", "OTEL_EXPORTER_OTLP_ENDPOINT")) != "" -} - // newDurableJobMetricsMonitor builds the durable job-operations metrics monitor // only when OTEL metrics are enabled (metricsEnabled mirrors a non-nil -// otelMetricProvider, i.e. observabilityEnabled()). When metrics are disabled the +// otelMetricProvider, i.e. the cell has a collector). When metrics are disabled the // global meter is a no-op, so a monitor would poll the job store every interval // only to record into instruments nothing can read; returning a nil monitor lets // the caller's Start/Shutdown nil-checks skip that background work entirely. @@ -2231,7 +2219,21 @@ func loadSigningKey(ctx context.Context, allowEphemeral, isLocal bool) (ed25519c // deployment can still be fixed rather than discovered from a crash loop — and // so a test can reach all of them with nothing running. func requireStartupConfiguration(isLocal bool) error { - return requireKeyCustody(isLocal) + if err := requireKeyCustody(isLocal); err != nil { + return err + } + return requireTelemetryConfiguration() +} + +// requireTelemetryConfiguration refuses to start unless the `observability` +// group says where the cell's collector is, or that the cell has none. It only +// checks: doWork reads the same answer again to act on it. A group that did not +// arrive is refused here, before the database or Vault is touched, because +// reading it as "no collector" would leave a cell exporting nothing while it +// converges green. +func requireTelemetryConfiguration() error { + _, err := configuredTelemetryDestination() + return err } // requireKeyCustody refuses to start outside the local environment without the diff --git a/module/services/accounts/code/work_auth_test.go b/module/services/accounts/code/work_auth_test.go index 57f6e0d4d..dc6bc8160 100644 --- a/module/services/accounts/code/work_auth_test.go +++ b/module/services/accounts/code/work_auth_test.go @@ -77,25 +77,6 @@ func TestWorkspaceEnvPreservesExactCodeflyConfigurationName(t *testing.T) { require.Equal(t, "legacy-normalized", workspaceEnv("internal-auth", key)) } -func TestObservabilityEnabledRequiresExternalOTLPEndpoint(t *testing.T) { - const key = "OTEL_EXPORTER_OTLP_ENDPOINT" - configurationKey := "CODEFLY__WORKSPACE_CONFIGURATION__OBSERVABILITY__" + key - secretKey := "CODEFLY__WORKSPACE_SECRET_CONFIGURATION__OBSERVABILITY__" + key - t.Setenv(key, "") - t.Setenv(configurationKey, "") - t.Setenv(secretKey, "") - require.False(t, observabilityEnabled()) - - t.Setenv(key, " ") - require.False(t, observabilityEnabled()) - t.Setenv(key, "https://otel.example") - require.True(t, observabilityEnabled()) - - t.Setenv(key, "") - t.Setenv(configurationKey, "https://workspace-otel.example") - require.True(t, observabilityEnabled()) -} - func TestConfiguredMFAStepUpMaxAge(t *testing.T) { t.Setenv("CODEFLY__WORKSPACE_CONFIGURATION__SECURITY__MFA_STEP_UP_MAX_AGE", "") t.Setenv("CODEFLY__WORKSPACE_SECRET_CONFIGURATION__SECURITY__MFA_STEP_UP_MAX_AGE", "") diff --git a/module/services/auth-gateway/code/configuration.go b/module/services/auth-gateway/code/configuration.go index 45b601c36..b557942fe 100644 --- a/module/services/auth-gateway/code/configuration.go +++ b/module/services/auth-gateway/code/configuration.go @@ -2,15 +2,10 @@ package main import ( "os" - "strings" codefly "github.com/codefly-dev/sdk-go" ) -func observabilityEnabled() bool { - return strings.TrimSpace(workspaceEnv("observability", "OTEL_EXPORTER_OTLP_ENDPOINT")) != "" -} - // workspaceEnv resolves Codefly workspace configuration and secret values // before falling back to a raw environment variable. Services must use this // boundary for declared workspace-configuration-dependencies; Codefly keeps diff --git a/module/services/auth-gateway/code/configuration_test.go b/module/services/auth-gateway/code/configuration_test.go index 8c66bd328..513f79245 100644 --- a/module/services/auth-gateway/code/configuration_test.go +++ b/module/services/auth-gateway/code/configuration_test.go @@ -27,22 +27,3 @@ func TestWorkspaceEnvResolvesConfigurationSecretAndFallback(t *testing.T) { t.Setenv(configurationKey, "configuration") require.Equal(t, "configuration", workspaceEnv("internal-auth", key)) } - -func TestObservabilityEnabledRequiresExternalOTLPEndpoint(t *testing.T) { - const key = "OTEL_EXPORTER_OTLP_ENDPOINT" - configurationKey := "CODEFLY__WORKSPACE_CONFIGURATION__OBSERVABILITY__" + key - secretKey := "CODEFLY__WORKSPACE_SECRET_CONFIGURATION__OBSERVABILITY__" + key - t.Setenv(key, "") - t.Setenv(configurationKey, "") - t.Setenv(secretKey, "") - require.False(t, observabilityEnabled()) - - t.Setenv(key, " ") - require.False(t, observabilityEnabled()) - t.Setenv(key, "https://otel.example") - require.True(t, observabilityEnabled()) - - t.Setenv(key, "") - t.Setenv(configurationKey, "https://workspace-otel.example") - require.True(t, observabilityEnabled()) -} diff --git a/module/services/auth-gateway/code/go.mod b/module/services/auth-gateway/code/go.mod index 35024443d..d9c26a210 100644 --- a/module/services/auth-gateway/code/go.mod +++ b/module/services/auth-gateway/code/go.mod @@ -11,14 +11,12 @@ require ( github.com/golang-jwt/jwt/v5 v5.3.1 github.com/google/uuid v1.6.0 github.com/grpc-ecosystem/grpc-gateway/v2 v2.30.0 - github.com/prometheus/client_golang v1.24.1 github.com/redis/go-redis/v9 v9.22.0 github.com/stretchr/testify v1.12.1 go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.71.0 go.opentelemetry.io/contrib/instrumentation/runtime v0.71.0 go.opentelemetry.io/otel v1.46.0 go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc v1.46.0 - go.opentelemetry.io/otel/exporters/prometheus v0.68.0 go.opentelemetry.io/otel/metric v1.46.0 go.opentelemetry.io/otel/sdk v1.46.0 go.opentelemetry.io/otel/sdk/metric v1.46.0 @@ -38,7 +36,6 @@ require ( github.com/Masterminds/semver/v3 v3.5.0 // indirect github.com/VividCortex/ewma v1.2.0 // indirect github.com/antlr4-go/antlr/v4 v4.13.1 // indirect - github.com/beorn7/perks v1.0.1 // indirect github.com/blang/semver v3.5.1+incompatible // indirect github.com/cenkalti/backoff/v5 v5.0.3 // indirect github.com/cespare/xxhash/v2 v2.3.0 // indirect @@ -78,15 +75,10 @@ require ( github.com/mattn/go-isatty v0.0.24 // indirect github.com/mattn/go-runewidth v0.0.27 // indirect github.com/mitchellh/mapstructure v1.5.0 // indirect - github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect github.com/oklog/ulid/v2 v2.1.2 // indirect github.com/pkg/errors v0.9.1 // indirect github.com/planetscale/vtprotobuf v0.6.1-0.20240319094008-0393e58bdf10 // indirect github.com/power-devops/perfstat v0.0.0-20240221224432-82ca36839d55 // indirect - github.com/prometheus/client_model v0.6.3 // indirect - github.com/prometheus/common v0.71.0 // indirect - github.com/prometheus/otlptranslator v1.0.0 // indirect - github.com/prometheus/procfs v0.22.0 // indirect github.com/santhosh-tekuri/jsonschema/v6 v6.0.3 // indirect github.com/shirou/gopsutil/v3 v3.24.5 // indirect github.com/shoenig/go-m1cpu v0.2.1 // indirect @@ -101,6 +93,7 @@ require ( go.opentelemetry.io/otel/exporters/stdout/stdouttrace v1.46.0 // indirect go.opentelemetry.io/otel/trace v1.46.0 // indirect go.uber.org/atomic v1.11.0 // indirect + go.yaml.in/yaml/v2 v2.4.4 // indirect go.yaml.in/yaml/v3 v3.0.5 // indirect golang.org/x/exp v0.0.0-20260824195058-e88cd73687aa // indirect golang.org/x/net v0.59.0 // indirect diff --git a/module/services/auth-gateway/code/go.sum b/module/services/auth-gateway/code/go.sum index 1162c0299..6be460d10 100644 --- a/module/services/auth-gateway/code/go.sum +++ b/module/services/auth-gateway/code/go.sum @@ -20,8 +20,6 @@ github.com/VividCortex/ewma v1.2.0 h1:f58SaIzcDXrSy3kWaHNvuJgJ3Nmz59Zji6XoJR/q1o github.com/VividCortex/ewma v1.2.0/go.mod h1:nz4BbCtbLyFDeC9SUHbtcT5644juEuWfUAUnGx7j5l4= github.com/antlr4-go/antlr/v4 v4.13.1 h1:SqQKkuVZ+zWkMMNkjy5FZe5mr5WURWnlpmOuzYWrPrQ= github.com/antlr4-go/antlr/v4 v4.13.1/go.mod h1:GKmUxMtwp6ZgGwZSva4eWPC5mS6vUAmOABFgjdkM7Nw= -github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM= -github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw= github.com/blang/semver v3.5.1+incompatible h1:cQNTCjp13qL8KC3Nbxr/y2Bqb63oX6wdnnjpJbkM4JQ= github.com/blang/semver v3.5.1+incompatible/go.mod h1:kRBLl5iJ+tD4TcOOxsy/0fnwebNt5EWlYSAyrTnjyyk= github.com/blang/semver/v4 v4.0.0 h1:1PFHFE6yCCTv8C1TeyNNarDzntLi7wMI5i/pzqYIsAM= @@ -175,16 +173,12 @@ github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99 h1:BQSFePA1RWJOl github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99/go.mod h1:1lJo3i6rXxKeerYnT8Nvf0QmHCRC1n8sfWVwXF2Frvo= github.com/kevinburke/ssh_config v1.6.0 h1:J1FBfmuVosPHf5GRdltRLhPJtJpTlMdKTBjRgTaQBFY= github.com/kevinburke/ssh_config v1.6.0/go.mod h1:q2RIzfka+BXARoNexmF9gkxEX7DmvbW9P4hIVx2Kg4M= -github.com/klauspost/compress v1.19.1 h1:VsB4HPswih7mmZ8WleSFQ75c/Ui1M4trX5oAsJnhSlk= -github.com/klauspost/compress v1.19.1/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ= github.com/klauspost/cpuid/v2 v2.3.0 h1:S4CRMLnYUhGeDFDqkGriYKdfoFlDnMtqTiI/sFzhA9Y= github.com/klauspost/cpuid/v2 v2.3.0/go.mod h1:hqwkgyIinND0mEev00jJYCxPNVRVXFQeu1XKlok6oO0= github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE= github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk= github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY= github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE= -github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc= -github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw= github.com/lufia/plan9stats v0.0.0-20260330125221-c963978e514e h1:Q6MvJtQK/iRcRtzAscm/zF23XxJlbECiGPyRicsX+Ak= github.com/lufia/plan9stats v0.0.0-20260330125221-c963978e514e/go.mod h1:autxFIvghDt3jPTLoqZ9OZ7s9qTGNAWmYCjVFWPX/zg= github.com/mattn/go-colorable v0.1.15 h1:+u9SLTRGnXv73cEsnsmoZBom+dMU88B2M0aDcWy0/jY= @@ -201,8 +195,6 @@ github.com/moby/patternmatcher v0.6.1 h1:qlhtafmr6kgMIJjKJMDmMWq7WLkKIo23hsrpR3x github.com/moby/patternmatcher v0.6.1/go.mod h1:hDPoyOpDY7OrrMDLaYoY3hf52gNCR/YOUYxkhApJIxc= github.com/monochromegane/go-gitignore v0.0.0-20200626010858-205db1a8cc00 h1:n6/2gBQ3RWajuToeY6ZtZTIKv2v7ThUy5KKusIT0yc0= github.com/monochromegane/go-gitignore v0.0.0-20200626010858-205db1a8cc00/go.mod h1:Pm3mSP3c5uWn86xMLZ5Sa7JB9GsEZySvHYXCTK4E9q4= -github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA= -github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ= github.com/oklog/ulid/v2 v2.1.2 h1:IEclFb9JNvzYA6MW2SCxbLzcHTVsfqm3PrqGQJH5zec= github.com/oklog/ulid/v2 v2.1.2/go.mod h1:rcEKHmBBKfef9DhnvX7y1HZBYxjXb0cP5ExxNsTT1QQ= github.com/opencontainers/go-digest v1.0.0 h1:apOUWs51W5PlhuyGyz9FCeeBIOUDA/6nW8Oi/yOhh5U= @@ -218,16 +210,6 @@ github.com/planetscale/vtprotobuf v0.6.1-0.20240319094008-0393e58bdf10 h1:GFCKgm github.com/planetscale/vtprotobuf v0.6.1-0.20240319094008-0393e58bdf10/go.mod h1:t/avpk3KcrXxUnYOhZhMXJlSEyie6gQbtLq5NM3loB8= github.com/power-devops/perfstat v0.0.0-20240221224432-82ca36839d55 h1:o4JXh1EVt9k/+g42oCprj/FisM4qX9L3sZB3upGN2ZU= github.com/power-devops/perfstat v0.0.0-20240221224432-82ca36839d55/go.mod h1:OmDBASR4679mdNQnz2pUhc2G8CO2JrUAVFDRBDP/hJE= -github.com/prometheus/client_golang v1.24.1 h1:JnJkREXzWxUdCuPFpIWZiPispT9xVV59uiuyR2bPlnU= -github.com/prometheus/client_golang v1.24.1/go.mod h1:F+oSRECHg4sse5ucfYpYDeIv/hu68Zo0uoHKetWnzcE= -github.com/prometheus/client_model v0.6.3 h1:O0jaTVAYNxTHYInEPFJt5I3+sN8zqBtVMPTB1qyxiEo= -github.com/prometheus/client_model v0.6.3/go.mod h1:gpN5P9S7Rr6Yr92PiQ+Ixvhf6JZEkF1dnxsYL2aPBEM= -github.com/prometheus/common v0.71.0 h1:9KDAKb7Mj3HEVKyFCK6Dc/HIwlBzZIN2l7/lrHl3KK8= -github.com/prometheus/common v0.71.0/go.mod h1:CLJ5H8TEsGX8bl31BdMkfhIZ+QmZ9tBPPotUxUbfcmk= -github.com/prometheus/otlptranslator v1.0.0 h1:s0LJW/iN9dkIH+EnhiD3BlkkP5QVIUVEoIwkU+A6qos= -github.com/prometheus/otlptranslator v1.0.0/go.mod h1:vRYWnXvI6aWGpsdY/mOT/cbeVRBlPWtBNDb7kGR3uKM= -github.com/prometheus/procfs v0.22.0 h1:6q9+/JL9IKAPbCmBrv9n5O5Ty3NKnciV5X7YGw0oics= -github.com/prometheus/procfs v0.22.0/go.mod h1:CvmFr/GVhIjIvWJZW3tgkODBQMRIf0EyWMQLHCHab58= github.com/redis/go-redis/v9 v9.22.0 h1:laDvpYXTJtZLloinw1fA5Kqd6HAEH2XKxOkG/PDq2F0= github.com/redis/go-redis/v9 v9.22.0/go.mod h1:y2g0Wj8rQvuK0ELM+oxSudcLtC09JScs98I/X9gRWY4= github.com/rodaine/protogofakeit v0.1.1 h1:ZKouljuRM3A+TArppfBqnH8tGZHOwM/pjvtXe9DaXH8= @@ -278,8 +260,6 @@ go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.46.0 h1:OFnwLJr+pF3iHrlGSzb go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.46.0/go.mod h1:716wFneO0ov19A2beH5hjfh9AK5z/VWNAtDijp1Y0/g= go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.46.0 h1:w53CDeOA/Kurp7yRsegSr6pbbr759dOvJ+yNmWM6Hxs= go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.46.0/go.mod h1:BOmGMCbAtvcJiSJ+hLuhgPLdDbimnraSl8irz3iY8sY= -go.opentelemetry.io/otel/exporters/prometheus v0.68.0 h1:QOf2IftqQwITVRJpnn0M7M9ZCbgWfxz4P7i9C9yc2N4= -go.opentelemetry.io/otel/exporters/prometheus v0.68.0/go.mod h1:bgSvqu2TWGXiz7yr5UTMfObH8oqxJWHTnubQ3ef9BO4= go.opentelemetry.io/otel/exporters/stdout/stdouttrace v1.46.0 h1:KdRxPiAoMptR3vfWzvjjvutTsSiwbC2uG0496rzZNfo= go.opentelemetry.io/otel/exporters/stdout/stdouttrace v1.46.0/go.mod h1:K/qSA+3G7Eovxi4K09wzrAgkWRnosS0DAOZeEpve7sM= go.opentelemetry.io/otel/metric v1.46.0 h1:yBnkXvgV7AXFILZc5K6IZe/CBFF3OS7BJ8ov6/lj0K8= diff --git a/module/services/auth-gateway/code/main.go b/module/services/auth-gateway/code/main.go index a1ebb1a3e..33a03c8bb 100644 --- a/module/services/auth-gateway/code/main.go +++ b/module/services/auth-gateway/code/main.go @@ -33,6 +33,14 @@ func main() { panic(fmt.Sprintf("codefly init failed: %v", err)) } + // The cell's collector is delivered in the `observability` configuration + // group, or the group says the cell has none. A group that did not arrive is + // neither, so it is refused before anything else starts. + telemetryDestination, err := configuredTelemetryDestination() + if err != nil { + panic(err) + } + grpcPort := codefly.For(ctx).WithDefaultNetwork().API(standards.GRPC).NetworkInstance().Port var httpPort uint16 httpNet, httpNetErr := codefly.For(ctx).API(standards.REST).ResolveNetworkInstance() @@ -40,32 +48,23 @@ func main() { panic(fmt.Sprintf("Codefly REST endpoint is unavailable: %v", httpNetErr)) } httpPort = httpNet.Port - var otelProvider *wooltel.Provider + // A local run with no cell collector gets wool's own stdout tracer instead. + otelProvider, err := enableTracing(ctx, telemetryDestination, codefly.IsLocal()) + if err != nil { + panic(fmt.Sprintf("configure OTEL tracing: %v", err)) + } var otelMetricProvider *otelMetrics - if observabilityEnabled() { - collectorNetwork, err := codefly.For(ctx). - Service("telemetry"). - Endpoint("grpc"). - API("grpc"). - ResolveNetworkInstance() - if err != nil { - panic(fmt.Sprintf("resolve telemetry collector through Codefly: %v", err)) - } - otelProvider, err = wooltel.Enable( - wooltel.WithServiceName("saas-starter-auth-gateway"), - wooltel.WithEndpoint(collectorNetwork.Host), - wooltel.WithInsecure(), - ) - if err != nil { - panic(fmt.Sprintf("configure OTEL tracing: %v", err)) - } - otelMetricProvider, err = enableOTELMetrics(ctx, "saas-starter-auth-gateway", collectorNetwork.Host) + if telemetryDestination.Available() { + otelMetricProvider, err = enableOTELMetrics(ctx, telemetryDestination) if err != nil { shutdownCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second) _ = otelProvider.Shutdown(shutdownCtx) cancel() panic(fmt.Sprintf("configure OTEL metrics: %v", err)) } + log.Printf("OTEL enabled: endpoint=%s insecure=%t", telemetryDestination.Endpoint, telemetryDestination.Insecure) + } else { + log.Printf("OTEL export disabled: the cell has no collector: %s", telemetryDestination.AbsentReason) } defer func() { shutdownCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second) diff --git a/module/services/auth-gateway/code/telemetry_destination.go b/module/services/auth-gateway/code/telemetry_destination.go new file mode 100644 index 000000000..e3296805d --- /dev/null +++ b/module/services/auth-gateway/code/telemetry_destination.go @@ -0,0 +1,267 @@ +package main + +import ( + "context" + "errors" + "fmt" + "net" + "net/url" + "strings" + + "github.com/codefly-dev/core/wool" + wooltel "github.com/codefly-dev/core/wool/otel" + codefly "github.com/codefly-dev/sdk-go" + "go.opentelemetry.io/otel/attribute" + "go.opentelemetry.io/otel/sdk/resource" +) + +// Where this process sends traces and metrics — or that it sends none. +// +// The platform tells a workload through the `observability` configuration +// group, and the group always says one of two things: +// +// TELEMETRY_STATE=available with OTEL_EXPORTER_OTLP_ENDPOINT, the cell's +// collector (OTLP/gRPC), e.g. +// http://otel-collector.otel-collector.svc.cluster.local:4317 +// TELEMETRY_STATE=absent with TELEMETRY_ABSENT_REASON, why the cell has no +// collector +// +// `http://` is plaintext on the wire because the service mesh supplies mTLS +// between workloads; `https://` is TLS, which the metrics exporter dials but +// wool's tracer cannot yet (see errWoolCannotDialTLS). This module owns no +// collector: the cell does, and a workload that cannot find it does not +// substitute one. +// +// A group that did not arrive is not a cell without a backend. Reading the first +// as the second would leave a deployment that converges green and exports +// nothing, so anything that is not exactly one of the two answers above refuses +// to start, naming what is missing. +// +// This file exists byte for byte in services/accounts/code and +// services/auth-gateway/code: the two services are independent Go modules with +// no shared package, and module/tools/telemetry_destination_mirror_test.go holds +// the copies identical. Change one, copy it over the other. + +const ( + telemetryStateKey = "TELEMETRY_STATE" + telemetryEndpointKey = "OTEL_EXPORTER_OTLP_ENDPOINT" + telemetryAbsentReasonKey = "TELEMETRY_ABSENT_REASON" + + telemetryStateAvailable = "available" + telemetryStateAbsent = "absent" + + // maxAbsentReasonLength bounds the operator-written reason before it reaches + // a log line: an unbounded value from configuration would otherwise let + // whoever writes the group forge log entries around it. + maxAbsentReasonLength = 300 +) + +// telemetryDestination is the resolved answer. Exactly one of Endpoint and +// AbsentReason is set. +type telemetryDestination struct { + // Endpoint is the collector's OTLP/gRPC address as host:port, the form the + // exporters take. Empty when the cell has no collector. + Endpoint string + // Insecure is true when the endpoint was http://, which the mesh protects. + // It is derived from the endpoint's scheme and never assumed. + Insecure bool + // AbsentReason says why the cell has no collector. Empty when Endpoint is set. + AbsentReason string +} + +// Available reports whether the cell has a collector to export to. +func (d telemetryDestination) Available() bool { + return d.Endpoint != "" +} + +// observabilityValue reads one key of the `observability` group through the +// SDK. It deliberately has no fallback to a bare process variable: an ambient +// OTEL_EXPORTER_OTLP_ENDPOINT in a shell or a pod is conventionally the +// OTLP/HTTP address, not the cell's gRPC collector, and reading it here would +// export to the wrong place without a word. +func observabilityValue(key string) string { + value, err := codefly.For(codefly.Context()).WorkspaceValue("observability", key) + if err != nil { + return "" + } + return value +} + +// configuredTelemetryDestination resolves the destination from the +// `observability` group. +func configuredTelemetryDestination() (telemetryDestination, error) { + return resolveTelemetryDestination(observabilityValue) +} + +// resolveTelemetryDestination is the whole decision, over any reader of the +// group's keys so a test can reach every outcome with nothing running. +func resolveTelemetryDestination(read func(key string) string) (telemetryDestination, error) { + state := strings.TrimSpace(read(telemetryStateKey)) + endpoint := strings.TrimSpace(read(telemetryEndpointKey)) + reason := boundedLine(read(telemetryAbsentReasonKey), maxAbsentReasonLength) + + if state == "" { + return telemetryDestination{}, errors.New( + "observability: " + telemetryStateKey + " is not set in the `observability` configuration group. " + + "The platform delivers `" + telemetryStateAvailable + "` (with " + telemetryEndpointKey + ") or `" + + telemetryStateAbsent + "` (with " + telemetryAbsentReasonKey + "); a group that did not arrive is " + + "not a cell without a collector, so this process does not start without it") + } + if state != telemetryStateAvailable && state != telemetryStateAbsent { + return telemetryDestination{}, fmt.Errorf( + "observability: %s is %q in the `observability` configuration group; it must be `%s` or `%s`", + telemetryStateKey, state, telemetryStateAvailable, telemetryStateAbsent) + } + if endpoint != "" && reason != "" { + return telemetryDestination{}, fmt.Errorf( + "observability: the `observability` configuration group sets both %s and %s; "+ + "a cell either has a collector or says why it has none", + telemetryEndpointKey, telemetryAbsentReasonKey) + } + + if state == telemetryStateAbsent { + if reason == "" { + return telemetryDestination{}, fmt.Errorf( + "observability: %s is `%s` but %s is not set in the `observability` configuration group; "+ + "an absent collector must say why", + telemetryStateKey, telemetryStateAbsent, telemetryAbsentReasonKey) + } + return telemetryDestination{AbsentReason: reason}, nil + } + + if endpoint == "" { + return telemetryDestination{}, fmt.Errorf( + "observability: %s is `%s` but %s is not set in the `observability` configuration group", + telemetryStateKey, telemetryStateAvailable, telemetryEndpointKey) + } + hostPort, insecure, err := parseCollectorEndpoint(endpoint) + if err != nil { + return telemetryDestination{}, fmt.Errorf("observability: %s: %w", telemetryEndpointKey, err) + } + return telemetryDestination{Endpoint: hostPort, Insecure: insecure}, nil +} + +// parseCollectorEndpoint turns the delivered URL into the host:port the OTLP +// gRPC exporters take, and reads the transport off its scheme. The URL is never +// echoed back: a malformed one may carry credentials. +func parseCollectorEndpoint(raw string) (hostPort string, insecure bool, err error) { + parsed, parseErr := url.Parse(raw) + if parseErr != nil { + return "", false, errors.New("is not a URL; expected http://host:port or https://host:port") + } + switch parsed.Scheme { + case "http": + insecure = true + case "https": + default: + return "", false, fmt.Errorf("must start with http:// or https://, got scheme %q", parsed.Scheme) + } + if parsed.Hostname() == "" { + return "", false, errors.New("has no host") + } + if parsed.User != nil { + return "", false, errors.New("must not carry credentials") + } + // OTLP/gRPC addresses a collector, not a path: a path or a query is the + // OTLP/HTTP form (/v1/traces), which this exporter does not speak. + if (parsed.Path != "" && parsed.Path != "/") || parsed.RawQuery != "" || parsed.Fragment != "" { + return "", false, errors.New("must be the collector's origin only, with no path, query or fragment") + } + port := parsed.Port() + if port == "" { + port = "80" + if !insecure { + port = "443" + } + } + return net.JoinHostPort(parsed.Hostname(), port), insecure, nil +} + +// boundedLine collapses a free-text value to one bounded line. +func boundedLine(value string, limit int) string { + line := strings.Join(strings.Fields(value), " ") + if runes := []rune(line); len(runes) > limit { + return string(runes[:limit]) + "…" + } + return line +} + +// telemetryResource is the one resource both signals carry. service.name is +// whatever the environment says — OTEL_SERVICE_NAME, or service.name in +// OTEL_RESOURCE_ATTRIBUTES — and otherwise the service identity the Codefly SDK +// holds. No name is typed here: a literal would override the environment, so the +// platform's identity could never reach the backend. +func telemetryResource(ctx context.Context, identity string) (*resource.Resource, error) { + options := []resource.Option{resource.WithService()} + if identity != "" { + options = append(options, resource.WithAttributes(attribute.String("service.name", identity))) + } + // Last, so what the environment declares wins over the identity above. + options = append(options, resource.WithFromEnv()) + return resource.New(ctx, options...) +} + +// currentTelemetryResource is telemetryResource for this process. +func currentTelemetryResource(ctx context.Context) (*resource.Resource, error) { + return telemetryResource(ctx, codeflyServiceName(codefly.Context())) +} + +// codeflyServiceName is the service identity the Codefly SDK holds: the +// `module/service` identity of the provider it built at Init, which wool stamps on +// every log line from this process. It is empty when the SDK has not been +// initialised. The SDK exposes no accessor of its own, and the environment +// carrier it reads this from is not product code's to read, so this asks the +// provider. +func codeflyServiceName(ctx context.Context) string { + provider, ok := ctx.Value(wool.ProviderKey).(*wool.Provider) + if !ok || provider == nil { + return "" + } + source := provider.Get(ctx).Source() + if source == nil { + return "" + } + // An identity with no module reads `/service`; with neither, `/`. + return strings.Trim(strings.TrimSpace(source.Unique), "/") +} + +// telemetryServiceName is the service.name the resource resolved to. +func telemetryServiceName(res *resource.Resource) string { + if value, ok := res.Set().Value(attribute.Key("service.name")); ok { + return value.AsString() + } + return "" +} + +// errWoolCannotDialTLS names the one transport the tracer cannot serve. +var errWoolCannotDialTLS = errors.New( + "the collector endpoint is https://, but wool's OTLP tracer cannot dial TLS: wool/otel.Enable " + + "always sets an insecure transport and offers no option to turn it off. Deliver an http:// " + + "endpoint (in-mesh, where the mesh supplies mTLS) until wool can dial TLS") + +// enableTracing registers wool's tracer for the destination: OTLP/gRPC to the +// cell's collector when there is one; wool's own stdout tracer when there is +// none and this is a local run; nothing otherwise. A nil provider means nothing +// was registered. +func enableTracing(ctx context.Context, destination telemetryDestination, local bool) (*wooltel.Provider, error) { + res, err := currentTelemetryResource(ctx) + if err != nil { + return nil, fmt.Errorf("resolve telemetry resource: %w", err) + } + options := []wooltel.Option{wooltel.WithServiceName(telemetryServiceName(res))} + switch { + case destination.Available(): + // wool dials plaintext unless told otherwise and cannot be told + // otherwise, so an https:// endpoint is refused rather than dialed in + // the clear and reported as TLS. + if !destination.Insecure { + return nil, errWoolCannotDialTLS + } + options = append(options, wooltel.WithEndpoint(destination.Endpoint), wooltel.WithInsecure()) + case local: + options = append(options, wooltel.WithStdout()) + default: + return nil, nil + } + return wooltel.Enable(options...) +} diff --git a/module/services/auth-gateway/code/telemetry_destination_test.go b/module/services/auth-gateway/code/telemetry_destination_test.go new file mode 100644 index 000000000..4d81a7698 --- /dev/null +++ b/module/services/auth-gateway/code/telemetry_destination_test.go @@ -0,0 +1,376 @@ +package main + +import ( + "context" + "net" + "strings" + "testing" + "time" + + "github.com/codefly-dev/core/wool" + "github.com/stretchr/testify/require" + "go.opentelemetry.io/otel" + collectortracev1 "go.opentelemetry.io/proto/otlp/collector/trace/v1" + "google.golang.org/grpc" +) + +// group is a reader over a fixed set of `observability` keys. +func group(values map[string]string) func(string) string { + return func(key string) string { return values[key] } +} + +const collector = "http://otel-collector.otel-collector.svc.cluster.local:4317" + +// The first outcome: the platform delivered a collector. +func TestTelemetryDestinationAvailableExportsToTheEndpoint(t *testing.T) { + destination, err := resolveTelemetryDestination(group(map[string]string{ + "TELEMETRY_STATE": "available", + "OTEL_EXPORTER_OTLP_ENDPOINT": collector, + })) + require.NoError(t, err) + require.True(t, destination.Available()) + require.Equal(t, "otel-collector.otel-collector.svc.cluster.local:4317", destination.Endpoint) + require.True(t, destination.Insecure, "http:// is plaintext on the wire; the mesh supplies mTLS") + require.Empty(t, destination.AbsentReason) +} + +// The transport comes from the endpoint's scheme, never from a constant. +func TestTelemetryDestinationReadsTransportFromTheScheme(t *testing.T) { + for endpoint, want := range map[string]struct { + hostPort string + insecure bool + }{ + "http://collector.example:4317": {"collector.example:4317", true}, + "https://collector.example:4317": {"collector.example:4317", false}, + "https://collector.example": {"collector.example:443", false}, + "http://collector.example": {"collector.example:80", true}, + "http://collector.example:4317/": {"collector.example:4317", true}, + " http://collector.example:4317 ": {"collector.example:4317", true}, + "http://[2001:db8::1]:4317": {"[2001:db8::1]:4317", true}, + "https://10.0.0.7:4317": {"10.0.0.7:4317", false}, + "http://otel-collector.ns.svc:4317/": {"otel-collector.ns.svc:4317", true}, + } { + destination, err := resolveTelemetryDestination(group(map[string]string{ + "TELEMETRY_STATE": "available", + "OTEL_EXPORTER_OTLP_ENDPOINT": endpoint, + })) + require.NoError(t, err, endpoint) + require.Equal(t, want.hostPort, destination.Endpoint, endpoint) + require.Equal(t, want.insecure, destination.Insecure, endpoint) + } +} + +// The second outcome: the cell has no collector, and says why. The process boots +// and exports nothing. +func TestTelemetryDestinationAbsentCarriesTheReason(t *testing.T) { + destination, err := resolveTelemetryDestination(group(map[string]string{ + "TELEMETRY_STATE": "absent", + "TELEMETRY_ABSENT_REASON": "This cell's bucket has no ops cell, so no collector is rendered.", + })) + require.NoError(t, err) + require.False(t, destination.Available()) + require.Empty(t, destination.Endpoint) + require.Equal(t, "This cell's bucket has no ops cell, so no collector is rendered.", destination.AbsentReason) +} + +// The reason is operator-written free text that reaches a log line. +func TestTelemetryDestinationAbsentReasonIsOneBoundedLine(t *testing.T) { + destination, err := resolveTelemetryDestination(group(map[string]string{ + "TELEMETRY_STATE": "absent", + "TELEMETRY_ABSENT_REASON": "no collector\n2026-01-01 FORGED entry\r\n" + strings.Repeat("x", 1000), + })) + require.NoError(t, err) + require.NotContains(t, destination.AbsentReason, "\n") + require.NotContains(t, destination.AbsentReason, "\r") + require.LessOrEqual(t, len([]rune(destination.AbsentReason)), maxAbsentReasonLength+1) +} + +// The third outcome: anything else refuses to start, naming what is wrong. +// Configuration that did not arrive is never read as "no collector". +func TestTelemetryDestinationRefusesEverythingElse(t *testing.T) { + for name, tc := range map[string]struct { + values map[string]string + want string + }{ + "state missing": { + values: map[string]string{}, + want: "TELEMETRY_STATE is not set", + }, + "state missing though an ambient endpoint is present": { + values: map[string]string{"OTEL_EXPORTER_OTLP_ENDPOINT": collector}, + want: "TELEMETRY_STATE is not set", + }, + "state missing though a reason is present": { + values: map[string]string{"TELEMETRY_ABSENT_REASON": "no collector"}, + want: "TELEMETRY_STATE is not set", + }, + "state blank": { + values: map[string]string{"TELEMETRY_STATE": " "}, + want: "TELEMETRY_STATE is not set", + }, + "state unknown": { + values: map[string]string{"TELEMETRY_STATE": "disabled", "TELEMETRY_ABSENT_REASON": "no collector"}, + want: `TELEMETRY_STATE is "disabled"`, + }, + "state is not matched case-insensitively": { + values: map[string]string{"TELEMETRY_STATE": "Available", "OTEL_EXPORTER_OTLP_ENDPOINT": collector}, + want: `TELEMETRY_STATE is "Available"`, + }, + "available without an endpoint": { + values: map[string]string{"TELEMETRY_STATE": "available"}, + want: "OTEL_EXPORTER_OTLP_ENDPOINT is not set", + }, + "available with only a reason": { + values: map[string]string{"TELEMETRY_STATE": "available", "TELEMETRY_ABSENT_REASON": "no collector"}, + want: "OTEL_EXPORTER_OTLP_ENDPOINT is not set", + }, + "absent without a reason": { + values: map[string]string{"TELEMETRY_STATE": "absent"}, + want: "TELEMETRY_ABSENT_REASON is not set", + }, + "absent with a blank reason": { + values: map[string]string{"TELEMETRY_STATE": "absent", "TELEMETRY_ABSENT_REASON": " \n "}, + want: "TELEMETRY_ABSENT_REASON is not set", + }, + "absent with only an endpoint": { + values: map[string]string{"TELEMETRY_STATE": "absent", "OTEL_EXPORTER_OTLP_ENDPOINT": collector}, + want: "TELEMETRY_ABSENT_REASON is not set", + }, + "available with both an endpoint and a reason": { + values: map[string]string{ + "TELEMETRY_STATE": "available", "OTEL_EXPORTER_OTLP_ENDPOINT": collector, + "TELEMETRY_ABSENT_REASON": "no collector", + }, + want: "sets both OTEL_EXPORTER_OTLP_ENDPOINT and TELEMETRY_ABSENT_REASON", + }, + "absent with both an endpoint and a reason": { + values: map[string]string{ + "TELEMETRY_STATE": "absent", "OTEL_EXPORTER_OTLP_ENDPOINT": collector, + "TELEMETRY_ABSENT_REASON": "no collector", + }, + want: "sets both OTEL_EXPORTER_OTLP_ENDPOINT and TELEMETRY_ABSENT_REASON", + }, + "endpoint without a scheme": { + values: map[string]string{"TELEMETRY_STATE": "available", "OTEL_EXPORTER_OTLP_ENDPOINT": "otel-collector.ns.svc:4317"}, + want: "must start with http:// or https://", + }, + "endpoint with another scheme": { + values: map[string]string{"TELEMETRY_STATE": "available", "OTEL_EXPORTER_OTLP_ENDPOINT": "grpc://otel-collector.ns.svc:4317"}, + want: `got scheme "grpc"`, + }, + "endpoint without a host": { + values: map[string]string{"TELEMETRY_STATE": "available", "OTEL_EXPORTER_OTLP_ENDPOINT": "http://:4317"}, + want: "has no host", + }, + "endpoint naming an OTLP/HTTP path": { + values: map[string]string{"TELEMETRY_STATE": "available", "OTEL_EXPORTER_OTLP_ENDPOINT": "http://collector.example:4318/v1/traces"}, + want: "origin only", + }, + "endpoint with a query": { + values: map[string]string{"TELEMETRY_STATE": "available", "OTEL_EXPORTER_OTLP_ENDPOINT": "http://collector.example:4317?x=1"}, + want: "origin only", + }, + } { + t.Run(name, func(t *testing.T) { + destination, err := resolveTelemetryDestination(group(tc.values)) + require.Error(t, err) + require.ErrorContains(t, err, tc.want) + require.False(t, destination.Available(), "a refusal must never read as a usable destination") + require.Empty(t, destination.AbsentReason, "a refusal must never read as an absent collector") + }) + } +} + +// A refusal quotes no part of an endpoint, which may carry credentials. +func TestTelemetryDestinationRefusalDoesNotEchoCredentials(t *testing.T) { + _, err := resolveTelemetryDestination(group(map[string]string{ + "TELEMETRY_STATE": "available", + "OTEL_EXPORTER_OTLP_ENDPOINT": "http://user:hunter2@collector.example:4317", + })) + require.ErrorContains(t, err, "must not carry credentials") + require.NotContains(t, err.Error(), "hunter2") +} + +// The group is read through the SDK's workspace-configuration lookup, by the +// names the platform delivers them under. A bare process variable of the same +// name is not the cell's answer. +func TestConfiguredTelemetryDestinationReadsTheObservabilityGroup(t *testing.T) { + const prefix = "CODEFLY__WORKSPACE_CONFIGURATION__OBSERVABILITY__" + for _, key := range []string{"TELEMETRY_STATE", "OTEL_EXPORTER_OTLP_ENDPOINT", "TELEMETRY_ABSENT_REASON"} { + t.Setenv(key, "") + t.Setenv(prefix+key, "") + } + + _, err := configuredTelemetryDestination() + require.ErrorContains(t, err, "TELEMETRY_STATE is not set") + + // Ambient variables a shell or a pod may carry are not the group. + t.Setenv("TELEMETRY_STATE", "available") + t.Setenv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://ambient.example:4318") + _, err = configuredTelemetryDestination() + require.ErrorContains(t, err, "TELEMETRY_STATE is not set") + t.Setenv("TELEMETRY_STATE", "") + t.Setenv("OTEL_EXPORTER_OTLP_ENDPOINT", "") + + t.Setenv(prefix+"TELEMETRY_STATE", "available") + t.Setenv(prefix+"OTEL_EXPORTER_OTLP_ENDPOINT", collector) + destination, err := configuredTelemetryDestination() + require.NoError(t, err) + require.Equal(t, "otel-collector.otel-collector.svc.cluster.local:4317", destination.Endpoint) + require.True(t, destination.Insecure) + + t.Setenv(prefix+"TELEMETRY_STATE", "absent") + t.Setenv(prefix+"OTEL_EXPORTER_OTLP_ENDPOINT", "") + t.Setenv(prefix+"TELEMETRY_ABSENT_REASON", "a local run has no cell collector") + destination, err = configuredTelemetryDestination() + require.NoError(t, err) + require.False(t, destination.Available()) + require.Equal(t, "a local run has no cell collector", destination.AbsentReason) +} + +// service.name is never a literal typed here: the environment wins, then the +// service's own Codefly identity. +func TestTelemetryResourceServiceName(t *testing.T) { + resolve := func(t *testing.T, identity string) string { + t.Helper() + res, err := telemetryResource(t.Context(), identity) + require.NoError(t, err) + return telemetryServiceName(res) + } + + t.Run("falls back to the Codefly service identity", func(t *testing.T) { + t.Setenv("OTEL_SERVICE_NAME", "") + t.Setenv("OTEL_RESOURCE_ATTRIBUTES", "") + require.Equal(t, "saas-starter/accounts", resolve(t, "saas-starter/accounts")) + }) + t.Run("OTEL_SERVICE_NAME wins over the identity", func(t *testing.T) { + t.Setenv("OTEL_SERVICE_NAME", "named-by-the-platform") + t.Setenv("OTEL_RESOURCE_ATTRIBUTES", "") + require.Equal(t, "named-by-the-platform", resolve(t, "saas-starter/accounts")) + }) + t.Run("OTEL_RESOURCE_ATTRIBUTES can set it", func(t *testing.T) { + t.Setenv("OTEL_SERVICE_NAME", "") + t.Setenv("OTEL_RESOURCE_ATTRIBUTES", "service.name=attributed-by-the-platform,deployment.environment=test") + require.Equal(t, "attributed-by-the-platform", resolve(t, "saas-starter/accounts")) + }) + t.Run("without either, the SDK default rather than a literal", func(t *testing.T) { + t.Setenv("OTEL_SERVICE_NAME", "") + t.Setenv("OTEL_RESOURCE_ATTRIBUTES", "") + require.True(t, strings.HasPrefix(resolve(t, ""), "unknown_service"), resolve(t, "")) + }) +} + +// The identity is the one the SDK built its provider with, the same one wool +// stamps on this process's log lines — read from the provider, not from the +// runtime's environment carrier. +func TestCodeflyServiceNameIsTheSDKProviderIdentity(t *testing.T) { + carrying := func(unique string) context.Context { + return wool.New(t.Context(), &wool.Resource{Kind: "service", Unique: unique}).Inject(t.Context()) + } + require.Equal(t, "saas-starter/accounts", codeflyServiceName(carrying("saas-starter/accounts"))) + require.Equal(t, "accounts", codeflyServiceName(carrying("/accounts")), "an identity with no module") + require.Empty(t, codeflyServiceName(carrying("/")), "an identity with neither") + require.Empty(t, codeflyServiceName(t.Context()), "no provider: the SDK was not initialised") +} + +// wool's tracer speaks plaintext only. An https:// endpoint is refused rather +// than dialed in the clear and reported as TLS. +func TestEnableTracingRefusesATransportWoolCannotDial(t *testing.T) { + provider, err := enableTracing(t.Context(), + telemetryDestination{Endpoint: "collector.example:4317", Insecure: false}, false) + require.ErrorIs(t, err, errWoolCannotDialTLS) + require.Nil(t, provider) +} + +func TestEnableTracingRegistersOnlyWhatTheDestinationCalls(t *testing.T) { + restore := func(t *testing.T) { + t.Helper() + previous := otel.GetTracerProvider() + t.Cleanup(func() { + wool.RegisterTelemetry(nil) + otel.SetTracerProvider(previous) + }) + } + absent := telemetryDestination{AbsentReason: "a local run has no cell collector"} + + t.Run("absent outside a local run registers no tracer", func(t *testing.T) { + restore(t) + provider, err := enableTracing(t.Context(), absent, false) + require.NoError(t, err) + require.Nil(t, provider) + require.False(t, wool.TelemetryEnabled()) + }) + t.Run("absent in a local run uses wool's stdout tracer", func(t *testing.T) { + restore(t) + provider, err := enableTracing(t.Context(), absent, true) + require.NoError(t, err) + require.NotNil(t, provider) + require.True(t, wool.TelemetryEnabled()) + shutdown(t, provider) + }) + t.Run("available over http exports spans to the collector, named by the environment", func(t *testing.T) { + restore(t) + t.Setenv("OTEL_SERVICE_NAME", "named-by-the-platform") + t.Setenv("OTEL_RESOURCE_ATTRIBUTES", "") + endpoint, capture := startTraceCapture(t) + provider, err := enableTracing(t.Context(), telemetryDestination{Endpoint: endpoint, Insecure: true}, false) + require.NoError(t, err) + require.NotNil(t, provider) + require.True(t, wool.TelemetryEnabled()) + + _, span := otel.Tracer("test").Start(t.Context(), "operation") + span.End() + shutdown(t, provider) // flushes the batch + + select { + case request := <-capture.requests: + var name string + for _, resourceSpans := range request.GetResourceSpans() { + for _, attribute := range resourceSpans.GetResource().GetAttributes() { + if attribute.GetKey() == "service.name" { + name = attribute.GetValue().GetStringValue() + } + } + } + require.Equal(t, "named-by-the-platform", name) + case <-time.After(5 * time.Second): + t.Fatal("timed out waiting for OTLP spans") + } + }) +} + +type traceCapture struct { + collectortracev1.UnimplementedTraceServiceServer + requests chan *collectortracev1.ExportTraceServiceRequest +} + +func (c *traceCapture) Export( + _ context.Context, + request *collectortracev1.ExportTraceServiceRequest, +) (*collectortracev1.ExportTraceServiceResponse, error) { + c.requests <- request + return &collectortracev1.ExportTraceServiceResponse{}, nil +} + +// startTraceCapture serves an in-process OTLP/gRPC trace receiver on loopback. +func startTraceCapture(t *testing.T) (string, *traceCapture) { + t.Helper() + listener, err := net.Listen("tcp", "127.0.0.1:0") + require.NoError(t, err) + capture := &traceCapture{requests: make(chan *collectortracev1.ExportTraceServiceRequest, 4)} + server := grpc.NewServer() + collectortracev1.RegisterTraceServiceServer(server, capture) + go func() { _ = server.Serve(listener) }() + t.Cleanup(func() { + server.Stop() + _ = listener.Close() + }) + return listener.Addr().String(), capture +} + +func shutdown(t *testing.T, provider interface{ Shutdown(context.Context) error }) { + t.Helper() + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + require.NoError(t, provider.Shutdown(ctx)) +} diff --git a/module/services/auth-gateway/code/telemetry_metrics.go b/module/services/auth-gateway/code/telemetry_metrics.go index ad2e7f8e0..bcebfed60 100644 --- a/module/services/auth-gateway/code/telemetry_metrics.go +++ b/module/services/auth-gateway/code/telemetry_metrics.go @@ -6,56 +6,39 @@ import ( runtimemetrics "runtime/metrics" "time" - "github.com/prometheus/client_golang/prometheus" - "github.com/prometheus/client_golang/prometheus/promhttp" "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp" otelruntime "go.opentelemetry.io/contrib/instrumentation/runtime" "go.opentelemetry.io/otel" - "go.opentelemetry.io/otel/attribute" "go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc" - otelprometheus "go.opentelemetry.io/otel/exporters/prometheus" otelmetric "go.opentelemetry.io/otel/metric" "go.opentelemetry.io/otel/sdk/metric" - "go.opentelemetry.io/otel/sdk/resource" ) +// otelMetrics owns the process's MeterProvider. Metrics leave by one path only: +// an OTLP push to the cell's collector, which is the record for traces and +// metrics alike. There is no scrape endpoint to mount, expose or exempt. type otelMetrics struct { provider *metric.MeterProvider - handler http.Handler } -func enableOTELMetrics(ctx context.Context, serviceName, endpoint string) (*otelMetrics, error) { - exporter, err := otlpmetricgrpc.New( - ctx, - otlpmetricgrpc.WithEndpoint(endpoint), - otlpmetricgrpc.WithInsecure(), - ) - if err != nil { - return nil, err +func enableOTELMetrics(ctx context.Context, destination telemetryDestination) (*otelMetrics, error) { + options := []otlpmetricgrpc.Option{otlpmetricgrpc.WithEndpoint(destination.Endpoint)} + if destination.Insecure { + // http://: plaintext on the wire, because the mesh supplies mTLS. An + // https:// endpoint takes the exporter's default, TLS. + options = append(options, otlpmetricgrpc.WithInsecure()) } - registry := prometheus.NewRegistry() - scrapeExporter, err := otelprometheus.New( - otelprometheus.WithRegisterer(registry), - otelprometheus.WithResourceAsConstantLabels( - attribute.NewAllowKeysFilter("service.name"), - ), - ) + exporter, err := otlpmetricgrpc.New(ctx, options...) if err != nil { return nil, err } - res, err := resource.New( - ctx, - resource.WithService(), - resource.WithFromEnv(), - resource.WithAttributes(attribute.String("service.name", serviceName)), - ) + res, err := currentTelemetryResource(ctx) if err != nil { return nil, err } provider := metric.NewMeterProvider( metric.WithResource(res), metric.WithReader(metric.NewPeriodicReader(exporter, metric.WithInterval(30*time.Second))), - metric.WithReader(scrapeExporter), ) if err := otelruntime.Start(otelruntime.WithMeterProvider(provider)); err != nil { _ = provider.Shutdown(ctx) @@ -66,10 +49,7 @@ func enableOTELMetrics(ctx context.Context, serviceName, endpoint string) (*otel return nil, err } otel.SetMeterProvider(provider) - return &otelMetrics{ - provider: provider, - handler: promhttp.HandlerFor(registry, promhttp.HandlerOpts{}), - }, nil + return &otelMetrics{provider: provider}, nil } func startGCActivityMetrics(provider *metric.MeterProvider) error { @@ -103,18 +83,14 @@ func startGCActivityMetrics(provider *metric.MeterProvider) error { return err } +// newGatewayHTTPHandler instruments the gateway's HTTP listener when metrics are +// on. It adds no route of its own: the listener serves the gateway's routes and +// nothing else, with or without telemetry. func newGatewayHTTPHandler(gateway http.Handler, metrics *otelMetrics) http.Handler { if metrics == nil { return gateway } - mux := http.NewServeMux() - mux.Handle("GET /metrics", metrics.Handler()) - mux.Handle("/", otelhttp.NewHandler(gateway, "auth-gateway.gateway")) - return mux -} - -func (m *otelMetrics) Handler() http.Handler { - return m.handler + return otelhttp.NewHandler(gateway, "auth-gateway.gateway") } func (m *otelMetrics) Shutdown(ctx context.Context) error { diff --git a/module/services/auth-gateway/code/telemetry_metrics_test.go b/module/services/auth-gateway/code/telemetry_metrics_test.go index 6e461dae4..701c6c851 100644 --- a/module/services/auth-gateway/code/telemetry_metrics_test.go +++ b/module/services/auth-gateway/code/telemetry_metrics_test.go @@ -39,8 +39,9 @@ func (c *authMetricCapture) Export( func TestAuthGatewayTelemetryExportsGatewayAndGRPCRED(t *testing.T) { endpoint, capture := startAuthMetricCapture(t) t.Setenv("OTEL_RESOURCE_ATTRIBUTES", "service.instance.id=auth-test-instance") + t.Setenv("OTEL_SERVICE_NAME", "named-by-the-platform") previousMeterProvider := otel.GetMeterProvider() - metrics, err := enableOTELMetrics(t.Context(), "test-auth-gateway", endpoint) + metrics, err := enableOTELMetrics(t.Context(), telemetryDestination{Endpoint: endpoint, Insecure: true}) require.NoError(t, err) require.NotNil(t, metrics) t.Cleanup(func() { @@ -57,19 +58,10 @@ func TestAuthGatewayTelemetryExportsGatewayAndGRPCRED(t *testing.T) { require.Equal(t, http.StatusNotFound, getStatus(t, httpServer.URL+"/missing")) runtime.GC() - response, err := http.Get(httpServer.URL + "/metrics") - require.NoError(t, err) - t.Cleanup(func() { _ = response.Body.Close() }) - require.Equal(t, http.StatusOK, response.StatusCode) - prometheusBody, err := io.ReadAll(response.Body) - require.NoError(t, err) - require.Contains(t, string(prometheusBody), "http_server_request_duration_seconds_count") - require.Contains(t, string(prometheusBody), `http_route="/health"`) - require.Contains(t, string(prometheusBody), `service_name="test-auth-gateway"`) - require.NoError(t, metrics.provider.ForceFlush(t.Context())) request := receiveAuthMetrics(t, capture.requests) require.Equal(t, "auth-test-instance", authResourceAttribute(request, "service.instance.id")) + require.Equal(t, "named-by-the-platform", authResourceAttribute(request, "service.name")) metricNames := authExportedMetricNames(request) require.Contains(t, metricNames, "go.goroutine.count") require.Contains(t, metricNames, "go.gc.cycle.count") @@ -88,19 +80,26 @@ func TestAuthGatewayTelemetryExportsGatewayAndGRPCRED(t *testing.T) { require.Equal(t, uint64(2), httpRoutes["/health"]) } -func TestGatewayHandlerHasNoMetricsRouteWhenTelemetryIsDisabled(t *testing.T) { +// Metrics leave by OTLP push alone: the gateway's listener serves its own routes +// and no scrape endpoint, whether or not telemetry is on. +func TestGatewayHandlerHasNoMetricsRoute(t *testing.T) { matcher := NewRouteMatcher([]*RouteEntry{{ Service: "self", Method: http.MethodGet, Path: "/health", }}, nil) gateway := NewGateway(nil, matcher, nil, nil, newFakeSolutionRegistry(), newFakeClientRegistry()) - request := httptest.NewRequest(http.MethodGet, "/metrics", nil) - recorder := httptest.NewRecorder() - newGatewayHTTPHandler(gateway, nil).ServeHTTP(recorder, request) - - require.Equal(t, http.StatusNotFound, recorder.Code) + for name, metrics := range map[string]*otelMetrics{ + "telemetry disabled": nil, + "telemetry enabled": {}, + } { + t.Run(name, func(t *testing.T) { + recorder := httptest.NewRecorder() + newGatewayHTTPHandler(gateway, metrics).ServeHTTP(recorder, httptest.NewRequest(http.MethodGet, "/metrics", nil)) + require.Equal(t, http.StatusNotFound, recorder.Code) + }) + } } func authInt64SumValue(request *collectormetricsv1.ExportMetricsServiceRequest, name string) int64 { diff --git a/module/tools/telemetry_destination_mirror_test.go b/module/tools/telemetry_destination_mirror_test.go new file mode 100644 index 000000000..e7310335e --- /dev/null +++ b/module/tools/telemetry_destination_mirror_test.go @@ -0,0 +1,40 @@ +package tools + +import ( + "os" + "path/filepath" + "testing" +) + +// telemetryDestinationCopies are source files that exist in both Go services, +// keyed by the copy and valued by the file it must match. +// +// accounts and auth-gateway both resolve where the cell's collector is from the +// `observability` configuration group: the same three keys, the same three +// outcomes, the same refusal to start on anything else. They are separate Go +// modules with no go.work, so neither can import the other, and a shared module +// would be a third thing to version for one small decision. The copy is the +// price; this test keeps the price from being paid twice, because two producers +// that read the group differently would let one service export while the other +// refuses to start on the same configuration. +var telemetryDestinationCopies = map[string]string{ + "services/auth-gateway/code/telemetry_destination.go": "services/accounts/code/telemetry_destination.go", +} + +func TestTelemetryDestinationCopiesMatchTheirSource(t *testing.T) { + moduleDir := findModuleDir(t) + for copyPath, sourcePath := range telemetryDestinationCopies { + copyBody, err := os.ReadFile(filepath.Join(moduleDir, filepath.FromSlash(copyPath))) + if err != nil { + t.Fatalf("read telemetry destination copy %s: %v", copyPath, err) + } + sourceBody, err := os.ReadFile(filepath.Join(moduleDir, filepath.FromSlash(sourcePath))) + if err != nil { + t.Fatalf("read telemetry destination source %s: %v", sourcePath, err) + } + if string(copyBody) != string(sourceBody) { + t.Errorf("%s has diverged from %s; change the source and copy it over the other so both services resolve the collector the same way", + copyPath, sourcePath) + } + } +} From 7e0ee42a036b78d032a03812dc6c611704a70531 Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Tue, 6 Oct 2026 16:44:54 -0400 Subject: [PATCH 41/69] feat(module)!: remove the telemetry service; the cell's collector replaces it (Refs #1022) `module/services/telemetry` was a hand-written in-graph OTLP gateway: a go-grpc service that registered the three upstream collector services and either logged each signal or re-marshalled it and POSTed it on. Every other upstream in this module is a published Codefly agent with a declarative spec; this one existed only because no collector agent did, and it was a worse collector than the one the cell already runs. An inline POST in the gRPC handler has no batching, no queue, no retry and no sampling, and a failed forward reaches the client as an unretryable `Unknown`, so every forward failure dropped that batch. LogsService was dead surface. accounts and auth-gateway now resolve the cell's collector from the `observability` group (previous commit), so the service has no caller. Removed: - the service: manifest, code with its own go.mod, proto and its SchemaMarker. - its dependency blocks in accounts and auth-gateway, its entry in module.codefly.yaml and in module.package.codefly.yaml. - the generated Codefly client stubs under `code/external/.../telemetry` in both services. `codefly sync service` does not prune an orphaned stub; with the dependency gone and the stubs deleted, a sync is a no-op in both services, which is what a from-scratch run produces. Regenerated, never edited, with `go generate ./pkg/cataloggen`: the topology loses the service, both dependency edges and the telemetry public-egress row, and the network-policy golden loses the five rules that named it. auth-gateway still has no public egress, which is now asserted: it reached a backend only by talking in-mesh to the gateway, and exporting to the cell's in-mesh collector needs no 443 allowance on the one service whose job is to be the perimeter. The topology and golden tests move with it, including the strict-spec cases that borrowed this service's manifest as a fixture. The module-package version is bumped 0.1.0 -> 0.2.0. Removing a service and making `TELEMETRY_STATE` mandatory is a breaking change for anyone composing the package and, at 0.x, that is a minor bump; the manifest records it in `breaking-changes`. `contracts/api/catalog.codefly.json` carries the package version, so it is regenerated with `codefly generate contracts saas-starter` (a one-line change) and the check passes. The reference composition's accepted range and the pin examples move to the 0.2 line, and a composition test fixture now takes the package version from the manifest instead of a literal. Counted prose corrected: seven services, twelve endpoints, six dependency edges, six interface endpoints, three public-egress grants and eighteen NetworkPolicy documents (the topology doc's endpoint, interface and policy counts were already stale against its own test); five Go modules by the enumeration given; the service graph and dependency tables; the collector-resolution paragraphs in the module, topology, analytics, SigNoz, features, local-dogfood and dogfooding docs and the run-locally skill. The metrics decision is recorded: OTLP push only, 2026-10-06. Left alone on purpose: the job-worker and analytics-outbox telemetry files and the synthetic `telemetry` fixture in the root module's mesh-policy test, which share only a word with this service. The point-in-time security review records still describe the former /metrics route as filed. Not done here: this does not bump the pinned codefly core, so the log/trace correlation fix is not yet consumed. Tested: the Go suites in accounts (pure), auth-gateway, module/tools, the root module and qualification/module-authority, the cataloggen parity tests, and the repository's node gates. Two module/tools tests fail identically on an untouched checkout in this environment (a missing `buf` binary, and a cache rename refused on macOS) and were not run to success. Co-Authored-By: Claude Opus 5.5 --- .../skills/run-the-starter-locally/SKILL.md | 11 +- AGENTS.md | 6 +- CLAIM_INVENTORY.md | 2 +- DOGFOODING.md | 6 +- LOCAL_DOGFOODING.md | 26 +- MODULE.md | 9 +- README.md | 2 +- RELEASE_GATES.md | 6 +- module/AGENTS.md | 14 +- module/DEPLOYMENT_TOPOLOGY.md | 27 +- module/FEATURES.md | 35 ++- module/PRODUCT_ANALYTICS.md | 7 +- module/SIGNOZ_PROVISIONING.md | 11 +- module/contracts/api/catalog.codefly.json | 2 +- .../contracts/reference/module.codefly.yaml | 2 +- .../generated/service-topology.json | 29 --- module/docs/composition-v2-migration.md | 4 +- module/module.codefly.yaml | 1 - module/module.package.codefly.yaml | 5 +- module/services/README.md | 2 +- module/services/accounts/TESTING.md | 4 +- .../saas-starter_telemetry_grpc.pb.go | 60 ----- .../code/pkg/auth/pg/session_store_test.go | 2 +- .../code/pkg/billing/pg/store_test.go | 2 +- .../code/pkg/business/service_test.go | 2 +- .../cataloggen/deployment_topology_test.go | 57 +++-- .../code/pkg/cataloggen/test_targets_test.go | 1 - .../testdata/network-policy.golden.yaml | 93 ------- .../code/pkg/infra/postgres_webhooks_test.go | 2 +- .../code/pkg/redisintegration/main_test.go | 2 +- module/services/accounts/service.codefly.yaml | 3 - .../saas-starter_telemetry_grpc.pb.go | 60 ----- .../auth-gateway/service.codefly.yaml | 3 - .../frontend/code/src/test/pipeline-setup.ts | 6 +- .../telemetry/code/collector/collector.go | 218 ---------------- .../code/collector/collector_test.go | 134 ---------- module/services/telemetry/code/go.mod | 58 ----- module/services/telemetry/code/go.sum | 235 ------------------ module/services/telemetry/code/main.go | 128 ---------- module/services/telemetry/code/main_test.go | 98 -------- .../gen/saas/telemetry/v1/schema_marker.pb.go | 117 --------- module/services/telemetry/proto/README.md | 9 - module/services/telemetry/proto/buf.gen.yaml | 11 - module/services/telemetry/proto/buf.yaml | 6 - .../saas/telemetry/v1/schema_marker.proto | 7 - .../services/telemetry/service.codefly.yaml | 26 -- module/tools/composition/composition_test.go | 9 +- .../tools/modulepackage/modulepackage_test.go | 6 +- 48 files changed, 154 insertions(+), 1412 deletions(-) delete mode 100644 module/services/accounts/code/external/saas-starter/telemetry/saas-starter_telemetry_grpc.pb.go delete mode 100644 module/services/auth-gateway/code/external/saas-starter/telemetry/saas-starter_telemetry_grpc.pb.go delete mode 100644 module/services/telemetry/code/collector/collector.go delete mode 100644 module/services/telemetry/code/collector/collector_test.go delete mode 100644 module/services/telemetry/code/go.mod delete mode 100644 module/services/telemetry/code/go.sum delete mode 100644 module/services/telemetry/code/main.go delete mode 100644 module/services/telemetry/code/main_test.go delete mode 100644 module/services/telemetry/code/pkg/gen/saas/telemetry/v1/schema_marker.pb.go delete mode 100644 module/services/telemetry/proto/README.md delete mode 100644 module/services/telemetry/proto/buf.gen.yaml delete mode 100644 module/services/telemetry/proto/buf.yaml delete mode 100644 module/services/telemetry/proto/saas/telemetry/v1/schema_marker.proto delete mode 100644 module/services/telemetry/service.codefly.yaml diff --git a/.claude/skills/run-the-starter-locally/SKILL.md b/.claude/skills/run-the-starter-locally/SKILL.md index 2c6a92afc..1f96ddc95 100644 --- a/.claude/skills/run-the-starter-locally/SKILL.md +++ b/.claude/skills/run-the-starter-locally/SKILL.md @@ -5,8 +5,8 @@ description: Boot this module's whole service graph on a developer machine — w # Running the starter locally -The graph is vault → store → cache → telemetry → accounts → auth-gateway → -frontend, plus marketing. One command boots all of it with a fake-auth fixture: +The graph is vault → store → cache → accounts → auth-gateway → frontend, plus +marketing. One command boots all of it with a fake-auth fixture: ```bash codefly run service --fixture dev-admin @@ -18,6 +18,13 @@ codefly run service --fixture dev-admin - No TTY (CI, pipes, MCP) auto-enables `--headless`. - Docker must be running. `codefly doctor` checks prerequisites; `codefly clear` reaps stray processes and containers between runs. +- A local run has no cell collector, and the module ships none. The `observability` + group (`configurations/local/observability.env`) says so — `TELEMETRY_STATE=absent` + with a reason — and accounts and auth-gateway log the reason once, send traces to + wool's stdout tracer and export no metrics. They refuse to start, naming + `TELEMETRY_STATE`, when that group did not arrive: a composing workspace that + overrode it without the keys, or a `local-dogfood` run without its copied + `observability.env`. For a real external identity provider and the production-grade provider stack, use the `local-dogfood` environment and its configuration groups: diff --git a/AGENTS.md b/AGENTS.md index f4c2e0bb6..a285ee2f3 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -11,7 +11,7 @@ in `.claude/skills/`. See [Where the depth lives](#where-the-depth-lives). `saas-starter` is a Codefly **module**: a multi-tenant SaaS backend (three-layer authorization, Postgres RLS, RBAC, impersonation, audit), an authenticated Next.js product, a separately deployable marketing site, and the -supporting cache/store/vault/telemetry services. It is published as an +supporting cache/store/vault services. It is published as an immutable module package that downstream workspaces **compose** (never fork). - Architecture, service graph, and capability ownership: [MODULE.md](./MODULE.md) @@ -156,7 +156,7 @@ Depth sits with the code that owns it: | File | Covers | | --- | --- | -| [module/AGENTS.md](./module/AGENTS.md) | the shipped tree: generated vs authored files, the six Go modules and how to test each, configuration groups | +| [module/AGENTS.md](./module/AGENTS.md) | the shipped tree: generated vs authored files, the five Go modules and how to test each, configuration groups | | [module/deployment/AGENTS.md](./module/deployment/AGENTS.md) | the manifests as the model, `spec.deployment`, and moving an agent version | | [module/services/auth-gateway/AGENTS.md](./module/services/auth-gateway/AGENTS.md) | upstream registration, the composed-module REST prefix, the credentials both take | | [module/services/accounts/AGENTS.md](./module/services/accounts/AGENTS.md) | the durable registry record, module principals, minting a module Work Context, mesh reachability | @@ -183,7 +183,7 @@ Skills in `.claude/skills/`, loaded when the task calls for them: such a list with a count: nothing enforces one, so two changes that each add a step and each bump the same number merge cleanly into a total that is silently wrong. -- Go suites live in six independent Go modules with no `go.work` +- Go suites live in five independent Go modules with no `go.work` ([module/AGENTS.md](./module/AGENTS.md#go-suites)). - Vulnerability policy: the complete audit runs non-blocking so vendor-image findings stay in the evidence report, while a separate fail-closed step enforces diff --git a/CLAIM_INVENTORY.md b/CLAIM_INVENTORY.md index 182d441b9..5d4c9a1a0 100644 --- a/CLAIM_INVENTORY.md +++ b/CLAIM_INVENTORY.md @@ -69,7 +69,7 @@ must not be inferred from a row here. See | The documented repository-specific gate list is the enforced one | RELEASE_GATES.md § Repository-specific gates; AGENTS.md | `release-gates.test.mjs` compares the table and both prose counts against `REQUIRED_GATES` | implemented | #541 | | The complete audit runs `--fail-on-vuln=false`; a separate step enforces first-party findings fail-closed | RELEASE_GATES.md § Vulnerability policy | `.github/workflows/ci.yml` `codefly-supply-chain` | configured | Codefly + this repo | | No artifact-writing job runs unless every mandatory gate actually succeeded | RELEASE_GATES.md § Publication gating | `scripts/ci/release-gates.mjs check` + `decide`, `release-gates.test.mjs` | implemented (#535) | this repo | -| Root `go test ./...` does not cover nested service modules | RELEASE_GATES.md, AGENTS.md | six independent `go.mod` files; there is no `go.work` | implemented (documented) | #541 | +| Root `go test ./...` does not cover nested service modules | RELEASE_GATES.md, AGENTS.md | independent `go.mod` files (the root, `module/tools`, one per Go service); there is no `go.work` | implemented (documented) | #541 | | `main` here is the only default branch in the organization merging through a queue, and `secure-saas-platform` the only other requiring any check | RELEASE_GATES.md § The fleet beyond this repository | `scripts/ci/release-gates.mjs fleet` — operator-run: reading branch configuration needs `Administration: read`, which no workflow token can hold, so no gate can carry this | reviewed narrative | #617 | ## Public registry contract diff --git a/DOGFOODING.md b/DOGFOODING.md index 5869f606d..f6dc63229 100644 --- a/DOGFOODING.md +++ b/DOGFOODING.md @@ -24,7 +24,7 @@ cd ~/Development/deus/codefly/module-saas-starter codefly run service --fixture dev-admin ``` -Expected graph: `vault + store + cache + telemetry → accounts → +Expected graph: `vault + store + cache → accounts → auth-gateway → frontend` (plus marketing). Independent services may start concurrently. If any service hangs at "waiting for ready", check that step's `--debug` output. @@ -219,7 +219,7 @@ the same Codefly-managed graph. For each of `email` (Resend), `configurations/local-dogfood/.env.example` and `.secret.env.example` to their Git-ignored names and fill them in ([LOCAL_DOGFOODING.md](./LOCAL_DOGFOODING.md#configure-providers-through-codefly)). -Keep `observability.env.example` as it is for local (`debug`) telemetry. +Keep `observability.env.example` as it is for a local run with no collector (`TELEMETRY_STATE=absent`). Turnstile's published test keys drive the three abuse-protection cases: | Case | `NEXT_PUBLIC_TURNSTILE_SITE_KEY` | `TURNSTILE_SECRET_KEY` | @@ -239,7 +239,7 @@ codefly run service --env local-dogfood - [ ] Grant analytics consent, navigate through onboarding, and verify bounded browser plus durable backend events in PostHog. - [ ] Withdraw analytics consent or log out. Browser capture stops immediately and identity resets. - [ ] Trigger controlled browser and backend errors. Sentry correlates release and environment without creating performance transactions. -- [ ] With the `debug` exporter, exercise login/onboarding and observe trace/metric/log summaries from the in-graph telemetry service. +- [ ] With `TELEMETRY_STATE=absent`, check that accounts and auth-gateway log the absent reason once at startup, then exercise login/onboarding and observe wool's stdout trace output (metrics are not exported; logs are on stdout). Remove `TELEMETRY_STATE` and confirm both services refuse to start, naming it. - [ ] Switch Turnstile to the **fail** keys. Registration and waitlist submission fail without database writes. - [ ] Switch Turnstile to the **replay** keys. The first deterministic verification follows Cloudflare's fixture behavior; replay rejection leaves state unchanged. diff --git a/LOCAL_DOGFOODING.md b/LOCAL_DOGFOODING.md index d2c758ccb..a08cab495 100644 --- a/LOCAL_DOGFOODING.md +++ b/LOCAL_DOGFOODING.md @@ -11,7 +11,7 @@ Two Codefly inputs are intentionally independent: | Input | Authority | Purpose | |---|---|---| -| `--env local-dogfood` | `configurations/local-dogfood/*` | Select independently configured identity, billing, email, analytics, error, telemetry, and abuse adapters | +| `--env local-dogfood` | `configurations/local-dogfood/*` | Select independently configured identity, billing, email, analytics, error, and abuse adapters, and the telemetry destination | | optional `--fixture ` | `module/fixtures/.yaml` | Seed a useful starting product state | A fixture never changes the identity provider. The ordinary `local` @@ -33,8 +33,8 @@ production/background fallback; a local port never belongs in product config. ## Configure providers through Codefly WorkOS is the identity adapter. The production-grade dogfood stack also -includes Stripe, Resend, PostHog, Sentry, the in-graph OpenTelemetry gateway, -and Cloudflare Turnstile. Each one is a Codefly configuration group on this +includes Stripe, Resend, PostHog, Sentry, the OpenTelemetry collector +destination, and Cloudflare Turnstile. Each one is a Codefly configuration group on this profile: copy the committed `configurations/local-dogfood/.env.example` (and `.secret.env.example`, where there is one) to the Git-ignored `.env` / `.secret.env`, and fill in the values. Every group is @@ -50,19 +50,21 @@ plugin, which validates the account, observes the webhook and projects the | `email` | Resend | | `product-analytics` | PostHog | | `error-tracking` | Sentry | -| `observability` | the in-graph OpenTelemetry gateway | +| `observability` | the OpenTelemetry collector the cell provides, or none | | `abuse-protection` | Cloudflare Turnstile | Validate the result with `codefly doctor workspace --env local-dogfood`. -`observability` is the one group this profile cannot skip. The `telemetry` -service declares it as a workspace-configuration-dependency, and on -`local-dogfood` the real `observability.env` / `observability.secret.env` are -Git-ignored, so there is no committed default to fall back on. Codefly skips a -declared group whose files are absent rather than failing, and the collector -refuses to start without an explicit `OBSERVABILITY_EXPORTER` — it will not -silently downgrade itself to `debug` and drop every span. Copy -`observability.env.example` as it is to keep telemetry local. +`observability` is the one group this profile cannot skip. `accounts` and +`auth-gateway` declare it as a workspace-configuration-dependency, and on +`local-dogfood` the real `observability.env` is Git-ignored, so there is no +committed default to fall back on. Codefly skips a declared group whose files are +absent rather than failing, and both services refuse to start without an explicit +`TELEMETRY_STATE` — a group that did not arrive is not a cell without a collector, +and they will not guess. Copy `observability.env.example` as it is to run with +none: `TELEMETRY_STATE=absent` and its reason, traces on wool's stdout tracer, +metrics not exported. To export to a collector this machine can reach, set +`TELEMETRY_STATE=available` and `OTEL_EXPORTER_OTLP_ENDPOINT` as the example says. `email` is required because this profile is not the `local` environment, and accounts refuses to start there without an explicit `EMAIL_PROVIDER`: `resend` diff --git a/MODULE.md b/MODULE.md index f5757d539..e4ce70545 100644 --- a/MODULE.md +++ b/MODULE.md @@ -100,7 +100,7 @@ capability; it does not claim that every planned provider is released yet. | Product analytics | PostHog | The `product-analytics@1` browser/server APIs may capture registered events and perform consent-gated identify, alias, organization grouping, and privacy suppression. An event may carry a variant already evaluated by Unleash for experiment analysis. | Flag definition or evaluation, exception capture, and traces, metrics, or logs | | Session replay | PostHog | A browser-only recorder may start after separate replay consent and redaction policy resolve to allow. It has no server SDK or implicit analytics-consent fallback. | Flags, errors, and APM signals | | Error tracking | Sentry | Browser/server exception events, release/environment tags, source-map upload, and the error-issue workflow. Provisioning credentials remain build/provider-only. | Performance transactions, profiles, replay, logs, metrics, and feature flags | -| APM traces, metrics, and logs | SigNoz | Applications use OpenTelemetry SDKs and standard OTLP configuration through the in-graph collector. SigNoz receives those signals as the OTLP backend. | Product analytics, replay, feature flags, and the Sentry error-issue contract | +| APM traces, metrics, and logs | SigNoz | Applications use OpenTelemetry SDKs and export traces and metrics over OTLP/gRPC to the cell's collector, whose address the platform delivers in the `observability` configuration group (or the group says the cell has none); logs go to stdout. SigNoz receives those signals as the OTLP backend. | Product analytics, replay, feature flags, and the Sentry error-issue contract | Provider manifests are allowlists. `provider-posthog` may project only `product-analytics@1`; its browser initialization must disable flag remote @@ -111,8 +111,9 @@ Sentry trace sampling at zero and installs no Sentry tracing integration. `provider-signoz`, if API qualification succeeds, may manage exactly owned dashboards and alerts but projects no application runtime configuration. In particular, dashboard provisioning is not an application telemetry contract: -application OTLP endpoints remain instrumentation configuration resolved -through the telemetry service. +the application OTLP endpoint remains instrumentation configuration, delivered +as `OTEL_EXPORTER_OTLP_ENDPOINT` in the `observability` group when the cell has a +collector. The starter keeps the capability configurations independent. Selecting PostHog never selects flags, errors, or observability; selecting Sentry never @@ -585,7 +586,7 @@ names the source repository and a package version rather than a path: modules: - name: saas-starter source: codefly-dev/module-saas-starter - version: "0.1.0" + version: "0.2.0" ``` Resolution fails closed, so the workspace must also say who it trusts to have diff --git a/README.md b/README.md index 6bb76e61f..df0633c3e 100644 --- a/README.md +++ b/README.md @@ -73,7 +73,7 @@ not this list, is the enumeration. Beyond the API, the module ships the authenticated product frontend (whose routes and navigation modules contribute to), a separately deployable public -marketing site, and the cache, store, vault, telemetry, and gateway services +marketing site, and the cache, store, vault, and gateway services the above runs on. ## What it does not own diff --git a/RELEASE_GATES.md b/RELEASE_GATES.md index fdf3509ef..412f1be86 100644 --- a/RELEASE_GATES.md +++ b/RELEASE_GATES.md @@ -193,7 +193,7 @@ Agent-generated `module/services/*/builder/Dockerfile` recipes must **not** be configured in Dependabot: agents replace them before building. The coverage gate rejects those entries. `scripts/ci/build-images.json` records the expected build images and the owning agent source for Go, -Next.js and the Postgres migration builder (including telemetry's generated Go +Next.js and the Postgres migration builder (including the Go services' generated recipe, which is not checked in). The weekly `build-images.yml` monitor compares registry digests for the effective @@ -441,8 +441,8 @@ reverse, fails the `release-contract` job. | `scoped-pools` | the authority of the pools the store's logins reach: the scoped reader, the scoped writer and the request pool, judged against a native PostgreSQL the suite provisions and widens one grant at a time | `python3 scripts/qualify-scoped-pools.py --postgres-bin "$(pg_config --bindir)"`, which runs `go test -race ./qualification/scopedpools` and fails unless every declared test in that package ran and passed | Two of those `go test ./...` invocations are worth reading carefully. This -repository holds **six independent Go modules** — root, `module/tools`, and one -per Go service — with no `go.work`, so `go test ./...` covers only the module it +repository holds **five independent Go modules** — root, `module/tools`, and one +per Go service (`accounts`, `auth-gateway`, `store`) — with no `go.work`, so `go test ./...` covers only the module it is run in. The `sdk-boundary` invocation runs in the root module (the module agent, the host, the generated reference composition) and the `module-package` one runs in `module/tools`; neither compiles, let alone tests, diff --git a/module/AGENTS.md b/module/AGENTS.md index f89e7041a..475eafbf1 100644 --- a/module/AGENTS.md +++ b/module/AGENTS.md @@ -13,7 +13,7 @@ Read the repository-root `AGENTS.md` first for ownership, boundaries and the behavioural rules (in a consumer, read your own). [SERVICE_CATALOG.md](./SERVICE_CATALOG.md) and [DEPLOYMENT_TOPOLOGY.md](./DEPLOYMENT_TOPOLOGY.md) describe the graph; -[services/README.md](./services/README.md) introduces the eight services. +[services/README.md](./services/README.md) introduces the seven services. ## Generated files are never edited by hand @@ -60,11 +60,13 @@ adding a REST-enabled RPC means editing it. Do not go looking for a generator. ## Go suites Every Go service here carries **its own `go.mod`** — `accounts`, `auth-gateway`, -`store`, `telemetry` — and `tools` is another, with **no `go.work`** tying them -together. So `go test ./...` only ever covers the module you run it in, and a -service's suite is outside whatever you run from the tree root. The canonical -repository adds its own root module (the module agent, the host, and the generated -reference composition), for six in total; a consumer's root module is its own. +`store` — and `tools` is another, with **no `go.work`** tying them together. So +`go test ./...` only ever covers the module you run it in, and a service's suite +is outside whatever you run from the tree root. The canonical repository adds its +own root module (the module agent, the host, and the generated reference +composition), for five in total; a consumer's root module is its own. +`libraries/source-read-sdk/go` and `qualification/module-authority` are two more +modules, outside this tree, that CI exercises in steps of their own. To exercise a service, run its suite from its own directory — its DB-backed suites need Codefly and Docker — or let `codefly ci run` do it. diff --git a/module/DEPLOYMENT_TOPOLOGY.md b/module/DEPLOYMENT_TOPOLOGY.md index 90a39df12..569713648 100644 --- a/module/DEPLOYMENT_TOPOLOGY.md +++ b/module/DEPLOYMENT_TOPOLOGY.md @@ -21,8 +21,8 @@ anywhere else. | `services/accounts/code/pkg/cataloggen/testdata/mesh-policy.golden.yaml` | Test-only mesh-policy golden (STRICT mTLS + the internal-authority and internal-HTTP AuthorizationPolicies + waypoint); mirrors those resources from the per-environment GitOps mesh baseline, not the whole of it — the namespace `default-deny` and the L4 internal policies are rendered there only. | | `services/accounts/code/pkg/cataloggen/deployment_topology.go` | Strict compiler, semantic validator, and renderers. | -The normalized inventory currently contains eight services, twelve endpoints, -eight dependency edges, four module-interface endpoints, and four explicit +The normalized inventory currently contains seven services, twelve endpoints, +six dependency edges, six module-interface endpoints, and three explicit public-egress grants (`deployment_topology_test.go` pins the counts). The accounts descriptor catalog is an input: if its RPCs use gRPC, Connect, or REST without a corresponding accounts endpoint, generation fails. @@ -33,7 +33,6 @@ generation fails. | --- | --- | --- | | `accounts` | `cache/read`, `cache/write` | TCP 6379 | | `accounts` | `store/tcp` | TCP 5432 | -| `accounts` | `telemetry/grpc` | Codefly-assigned OTLP gRPC port | | `accounts` | `vault/http` | TCP 8200 | | `auth-gateway` | `accounts/connect`, `accounts/rest`, `accounts/grpc` | TCP 8080, 9090 | | `auth-gateway` | `cache/write` | TCP 6379 | @@ -42,21 +41,21 @@ generation fails. The Codefly module interface exposes the public `frontend/http` and `marketing/http` endpoints; the auth-gateway gRPC ext-authz endpoint has module visibility. Istio routes apex/`www`/docs hosts to `marketing/http` and `app` to -`frontend/http`. Accounts, frontend, marketing, and telemetry may reach -public IP space only over TCP 443. The public rules exclude private, loopback, +`frontend/http`. Accounts, frontend, and marketing may reach +public IP space only over TCP 443; `auth-gateway` has no public egress. The public rules exclude private, loopback, link-local, metadata, documentation, benchmark, multicast, and other special-purpose IPv4/IPv6 ranges. Temporal's gRPC frontend and HTTP UI remain module-visible without a public ingress route. ## Network-policy model -The topology-policy golden contains 25 `NetworkPolicy` resources: +The topology-policy golden contains 18 `NetworkPolicy` resources: - one namespace-wide ingress/egress default deny; - DNS and Istio control-plane egress for all injected workloads; - Istio ingress only to the public frontend and marketing HTTP ports; - target ingress and caller egress policies for every declared dependency; -- HTTPS public egress only for accounts, frontend, marketing, and telemetry. +- HTTPS public egress only for accounts, frontend, and marketing. There is no `allow-intra-namespace` rule. Adding a service dependency or port requires changing the topology binding and reviewing both generated directions @@ -71,10 +70,14 @@ topology binding in the same change; pinned deployment schema/render validation is tracked by `P1-CI-004`. The generated Codefly dependency declarations contain exact endpoint -references. Accounts resolves `telemetry/grpc` through the SDK and passes that -exact Codefly-owned address to both tracing and metrics; no product -configuration owns a local collector port. The generated NetworkPolicy remains -the hard endpoint/port enforcement boundary. +references. The module owns no collector: accounts and auth-gateway export traces +and metrics to the cell's, whose address the platform delivers in the +`observability` configuration group (`TELEMETRY_STATE`, with +`OTEL_EXPORTER_OTLP_ENDPOINT` when the cell has a collector and +`TELEMETRY_ABSENT_REASON` when it does not), so neither service needs public +egress to reach a telemetry backend and no product configuration owns a collector +port. The generated NetworkPolicy remains the hard endpoint/port enforcement +boundary for the services the module declares. The AWS overlay replaces stateful services with managed dependencies. A pod-selector rule cannot authorize an RDS, ElastiCache, external Vault, or S3 @@ -271,7 +274,7 @@ exports, and missing descriptor-required accounts protocols. Parity tests build every artifact twice, compare all checked-in outputs, parse the generated files through Codefly's resource model, and strictly inspect all -25 NetworkPolicy golden documents. After the module generator creates the +18 NetworkPolicy golden documents. After the module generator creates the consumer-owned GitOps tree, render an environment with: ```sh diff --git a/module/FEATURES.md b/module/FEATURES.md index 7753e98bf..29566a9fc 100644 --- a/module/FEATURES.md +++ b/module/FEATURES.md @@ -49,8 +49,8 @@ the auth-gateway in front so this code path isn't reached. | Test infra | Playwright e2e against the real stack via `withDependencies` | Everything is orchestrated by Codefly: `codefly run service --fixture -dev-admin` resolves the module's service graph and brings up all eight -services—Postgres + Vault + Redis + object storage + telemetry + Accounts + +dev-admin` resolves the module's service graph and brings up its +services—Postgres + Vault + Redis + object storage + Accounts + the private auth gateway + frontend—with seed data in one command. Marketing remains a separate runtime and may be started, deployed, rolled back, or disabled without changing the authenticated product. @@ -495,8 +495,8 @@ extraction contracts. |----------------|--------|-----------------------------------------------------------------| | Structured logs| ✅ | `wool` everywhere; user/org/action context auto-attached | | Audit trail | ✅ | Separate from app logs; queryable | -| Metrics | ✅ | Job-worker OTel instruments, durable queue projections, and an in-graph OTLP gateway | -| Tracing | ✅ | Accounts resolves the Codefly collector endpoint and exports OTLP for the designated SigNoz backend | +| Metrics | ✅ | Job-worker OTel instruments, durable queue projections, and Go runtime and request metrics pushed over OTLP to the cell's collector | +| Tracing | ✅ | Accounts and auth-gateway export OTLP to the cell's collector, whose address the platform delivers in the `observability` group, for the designated SigNoz backend | | Error tracking | ✅ | Explicit fail-closed Sentry mode for server/browser errors; trace sampling is fixed at zero | | Dashboards | 🟡 | Versioned provider-neutral business dashboard pack; [SigNoz provisioning remains unsupported pending a pinned service qualification](SIGNOZ_PROVISIONING.md) | @@ -655,20 +655,29 @@ Environment variables consumed by the api: | `POSTHOG_API_HOST` | Separate PostHog management/deletion origin | | `ERROR_TRACKING_MODE` | Explicit `disabled` or `sentry`; rejects partial config | | `SENTRY_DSN` | Server Sentry DSN, required in Sentry mode | -| `OBSERVABILITY_EXPORTER` | Required in-graph collector output: `debug` or `otlphttp`; the collector refuses to start when it is unset | -| `OTEL_EXPORTER_OTLP_ENDPOINT` | External OTLP/HTTP destination used only by the collector | -| `OTEL_EXPORTER_OTLP_HEADERS` | Secret external collector headers | +| `TELEMETRY_STATE` | `observability` group, delivered by the platform: `available` or `absent`, never empty; accounts and auth-gateway refuse to start without it | +| `OTEL_EXPORTER_OTLP_ENDPOINT` | `observability` group: the cell collector's OTLP/gRPC address, present only when `available`; `http://` is plaintext on the wire because the mesh supplies mTLS, `https://` is TLS | +| `TELEMETRY_ABSENT_REASON` | `observability` group: why the cell has no collector, present only when `absent`; logged once at startup | | `ABUSE_PROTECTION_MODE` | Explicit `disabled` or `turnstile` | | `TURNSTILE_SECRET_KEY` | Server-only Siteverify credential | | `TURNSTILE_ALLOWED_HOSTNAMES` | Exact accepted Turnstile response hostnames | | `CODEFLY__FIXTURE` | Loads fixture YAML (e.g. `dev-admin`); FE login picker too | -When `OTEL_EXPORTER_OTLP_ENDPOINT` is configured, the accounts service and the -auth-gateway export unsampled request and Go runtime metrics through the in-graph -OpenTelemetry collector. The auth-gateway covers both its HTTP gateway and gRPC ext_authz -authorization service. Prometheus can alternatively scrape `/metrics` on each -service's private REST endpoint; neither route is a module or public interface -endpoint. +When `TELEMETRY_STATE` is `available`, the accounts service and the auth-gateway +export traces and unsampled request and Go runtime metrics over OTLP/gRPC to the +cell's collector at `OTEL_EXPORTER_OTLP_ENDPOINT`. The auth-gateway covers both its +HTTP gateway and gRPC ext_authz authorization service. When it is `absent`, they +export nothing over OTLP, log `TELEMETRY_ABSENT_REASON` once and boot; a local run +then uses wool's stdout tracer for traces. Anything else — the state missing or +unknown, `available` without an endpoint, `absent` without a reason, or both +set — refuses to start, because a group that did not arrive is not a cell without +a collector. An `https://` endpoint is refused at startup for now: wool's OTLP +tracer dials plaintext only, so it would be sent in the clear and reported as TLS. +Metrics leave by OTLP push alone — decided 2026-10-06, because the cell's +collector is the record for traces and metrics take the same path — so neither +service serves a scrape endpoint. `service.name` comes from `OTEL_SERVICE_NAME` or +`OTEL_RESOURCE_ATTRIBUTES` when the platform sets it, else the service's own +Codefly identity (`module/service`, the one wool puts on its log lines). Frontend browser configuration. The server reads each value from its Codefly group per request (`identity`, `legal`, `product-features`, `product-analytics`, `error-tracking`, diff --git a/module/PRODUCT_ANALYTICS.md b/module/PRODUCT_ANALYTICS.md index 886315b33..20798f901 100644 --- a/module/PRODUCT_ANALYTICS.md +++ b/module/PRODUCT_ANALYTICS.md @@ -200,9 +200,10 @@ active organization. ## Operational metrics, alerts, and recovery SigNoz is the single backend for application traces, metrics, and logs. The -application surface remains standard OpenTelemetry/OTLP through the in-graph -collector; an optional SigNoz provider may provision dashboards and alerts but -does not project an application SDK, endpoint, or credential. +application surface remains standard OpenTelemetry/OTLP to the cell's collector, +whose address the platform delivers in the `observability` configuration group; an +optional SigNoz provider may provision dashboards and alerts but does not project +an application SDK, endpoint, or credential. Job workers emit OpenTelemetry `saas.jobs.polls`, `claimed`, `active`, `completed`, and `duration` instruments. Labels are limited to queue plus diff --git a/module/SIGNOZ_PROVISIONING.md b/module/SIGNOZ_PROVISIONING.md index f1bdb6f5f..e6fa41412 100644 --- a/module/SIGNOZ_PROVISIONING.md +++ b/module/SIGNOZ_PROVISIONING.md @@ -11,11 +11,12 @@ deployment against which to prove the lifecycle. Creating a provider before that boundary exists would turn an untested upstream API into a claimed Codefly contract. -This decision does not affect application telemetry. Applications continue to -send OTLP to the in-graph collector endpoint resolved through Codefly; they do -not read or hard-code that address. `OTEL_EXPORTER_OTLP_ENDPOINT` remains only -the collector's external OTLP/HTTP destination. Dashboard provisioning is not -an application runtime capability. +This decision does not affect application telemetry. Applications export +OTLP/gRPC to the cell's collector, at the address the platform delivers as +`OTEL_EXPORTER_OTLP_ENDPOINT` in the `observability` configuration group (or the +group says the cell has none); they do not hard-code it, and the module ships no +collector of its own. Dashboard provisioning is not an application runtime +capability. ## Qualification result diff --git a/module/contracts/api/catalog.codefly.json b/module/contracts/api/catalog.codefly.json index 96565e7e1..5d35b09f0 100644 --- a/module/contracts/api/catalog.codefly.json +++ b/module/contracts/api/catalog.codefly.json @@ -1,7 +1,7 @@ { "schema": "codefly/module-api-contracts/v1", "package": "codefly/saas-starter", - "version": "0.1.0", + "version": "0.2.0", "endpoints": [ { "service": "accounts", diff --git a/module/contracts/reference/module.codefly.yaml b/module/contracts/reference/module.codefly.yaml index 3cb683573..c5bb83ce7 100644 --- a/module/contracts/reference/module.codefly.yaml +++ b/module/contracts/reference/module.codefly.yaml @@ -2,7 +2,7 @@ kind: composed-module name: reference-product base: id: codefly/saas-starter - version: ">=0.1.0 <0.2.0" + version: ">=0.2.0 <0.3.0" contributions: frontend: - path: frontend diff --git a/module/deployment/generated/service-topology.json b/module/deployment/generated/service-topology.json index b40e3cafd..6164033d8 100644 --- a/module/deployment/generated/service-topology.json +++ b/module/deployment/generated/service-topology.json @@ -44,12 +44,6 @@ "tcp" ] }, - { - "service": "telemetry", - "endpoints": [ - "grpc" - ] - }, { "service": "vault", "endpoints": [ @@ -88,12 +82,6 @@ "endpoints": [ "write" ] - }, - { - "service": "telemetry", - "endpoints": [ - "grpc" - ] } ] }, @@ -155,17 +143,6 @@ } ] }, - { - "name": "telemetry", - "endpoints": [ - { - "name": "grpc", - "api": "CODEFLY_API_GRPC", - "visibility": "ENDPOINT_VISIBILITY_PRIVATE", - "port": 9090 - } - ] - }, { "name": "vault", "endpoints": [ @@ -228,12 +205,6 @@ "ports": [ 443 ] - }, - { - "service": "telemetry", - "ports": [ - 443 - ] } ] } diff --git a/module/docs/composition-v2-migration.md b/module/docs/composition-v2-migration.md index 9a376f607..e61b14c03 100644 --- a/module/docs/composition-v2-migration.md +++ b/module/docs/composition-v2-migration.md @@ -11,7 +11,7 @@ contract. The copied tree is removed only after the v1 and v2 projections produc the same service catalog, endpoint graph, frontend inventory, permissions, settings, fixtures, and runtime smoke results. -The `0.1.x` package line supports migration from Starter `0.0.32` and later. Earlier +The `0.1.x` and `0.2.x` package lines support migration from Starter `0.0.32` and later. Earlier copies must first update to a release the package line supports. The v2 descriptor points directly at contribution sources; Starter does not @@ -23,7 +23,7 @@ kind: composed-module name: my-product base: id: codefly/saas-starter - version: ">=0.1.0 <0.2.0" + version: ">=0.2.0 <0.3.0" contributions: frontend: - {path: frontend, export: productPlugin} diff --git a/module/module.codefly.yaml b/module/module.codefly.yaml index a67b091fe..1140227aa 100644 --- a/module/module.codefly.yaml +++ b/module/module.codefly.yaml @@ -29,5 +29,4 @@ services: - name: frontend - name: marketing - name: store - - name: telemetry - name: vault diff --git a/module/module.package.codefly.yaml b/module/module.package.codefly.yaml index 9f1fb55a7..452f16bad 100644 --- a/module/module.package.codefly.yaml +++ b/module/module.package.codefly.yaml @@ -1,7 +1,7 @@ schema: codefly/module-package/v2 kind: module-package id: codefly/saas-starter -version: 0.1.0 +version: 0.2.0 minimum-codefly-version: ">=0.3.32 <1.0.0" artifact-roots: - contracts @@ -40,8 +40,6 @@ services: endpoints: [http] - name: store endpoints: [tcp] - - name: telemetry - endpoints: [grpc] - name: vault endpoints: [http] fixtures: @@ -100,6 +98,7 @@ migrations: breaking-changes: - Consumers resolve the immutable package into a disposable projection instead of editing Starter-owned files. - "Audit event types are namespaced under `saas.*` (`auth.login` -> `saas.auth.login`). The wire value of `audit_events.event_type`, the `X-Webhook-Event` header, and webhook routing keys all change; migration 116 rewrites stored history and existing subscriptions in place, so a live subscriber keeps firing, but a consumer that re-creates a subscription from a hardcoded pre-namespace name, or emits one through `ModuleEmitAuditEvent`, is now rejected. No alias layer." + - "The `telemetry` service is removed: the module no longer ships an OTLP gateway, and the cell's collector takes its place. `accounts` and `auth-gateway` read where it is from the `observability` configuration group — `TELEMETRY_STATE` is `available` with `OTEL_EXPORTER_OTLP_ENDPOINT` (an `http://` endpoint is plaintext on the wire because the mesh supplies mTLS; `https://` is TLS), or `absent` with `TELEMETRY_ABSENT_REASON` — and refuse to start when the group does not say which. `OBSERVABILITY_EXPORTER` and `OTEL_EXPORTER_OTLP_HEADERS` no longer mean anything, the `/metrics` scrape endpoint of both services is gone (metrics leave by OTLP push only), and `service.name` is no longer fixed in code: `OTEL_SERVICE_NAME` or `OTEL_RESOURCE_ATTRIBUTES` sets it, else the service's own Codefly identity. A consumer that depended on the `telemetry` service, or that overrides the `observability` group, must move to the keys above." reserved-namespaces: - "@codefly/saas-" - saas diff --git a/module/services/README.md b/module/services/README.md index 09ba17c49..383ea5d0f 100644 --- a/module/services/README.md +++ b/module/services/README.md @@ -1,6 +1,6 @@ # Module services -The module contains eight first-class Codefly services. `marketing` and +The module contains seven first-class Codefly services. `marketing` and `frontend` are separate Next.js applications: marketing owns public apex/`www` content, while frontend remains the authenticated product behind `auth-gateway`. diff --git a/module/services/accounts/TESTING.md b/module/services/accounts/TESTING.md index 3456af2ab..bd83c7f5a 100644 --- a/module/services/accounts/TESTING.md +++ b/module/services/accounts/TESTING.md @@ -23,8 +23,8 @@ The default `go test ./...` and `codefly ci run` retain all existing default tes The pure command is an additional fast check, not a replacement release gate. Database targets also retain their package's pure tests. They serialize setup, execution and teardown through the existing package lock. Business tests use -Vault for hashing and encrypted credentials. None of them uses telemetry, and -only the Redis target starts the `cache` service: it proves the org-membership +Vault for hashing and encrypted credentials. Only the Redis target starts the +`cache` service: it proves the org-membership cache stack across two instances and the Redis-held revocation, nonce and rate-limit state against the real server. Dependency exclusions prevent every other service from building or starting. diff --git a/module/services/accounts/code/external/saas-starter/telemetry/saas-starter_telemetry_grpc.pb.go b/module/services/accounts/code/external/saas-starter/telemetry/saas-starter_telemetry_grpc.pb.go deleted file mode 100644 index 689bf2157..000000000 --- a/module/services/accounts/code/external/saas-starter/telemetry/saas-starter_telemetry_grpc.pb.go +++ /dev/null @@ -1,60 +0,0 @@ -// Code generated by protoc-gen-go. DO NOT EDIT. -// versions: -// protoc-gen-go v1.36.11 -// protoc (unknown) -// source: saas-starter_telemetry_grpc.proto - -package telemetry - -import ( - reflect "reflect" - unsafe "unsafe" - - protoreflect "google.golang.org/protobuf/reflect/protoreflect" - protoimpl "google.golang.org/protobuf/runtime/protoimpl" -) - -const ( - // Verify that this generated code is sufficiently up-to-date. - _ = protoimpl.EnforceVersion(20 - protoimpl.MinVersion) - // Verify that runtime/protoimpl is sufficiently up-to-date. - _ = protoimpl.EnforceVersion(protoimpl.MaxVersion - 20) -) - -var File_saas_starter_telemetry_grpc_proto protoreflect.FileDescriptor - -const file_saas_starter_telemetry_grpc_proto_rawDesc = "" + - "\n" + - "!saas-starter_telemetry_grpc.protoBhB\x1dSaasStarterTelemetryGrpcProtoP\x01ZEgithub.com/codefly-dev/cli/pkg/builder/clients/saas-starter/telemetry" - -var file_saas_starter_telemetry_grpc_proto_goTypes = []any{} -var file_saas_starter_telemetry_grpc_proto_depIdxs = []int32{ - 0, // [0:0] is the sub-list for method output_type - 0, // [0:0] is the sub-list for method input_type - 0, // [0:0] is the sub-list for extension type_name - 0, // [0:0] is the sub-list for extension extendee - 0, // [0:0] is the sub-list for field type_name -} - -func init() { file_saas_starter_telemetry_grpc_proto_init() } -func file_saas_starter_telemetry_grpc_proto_init() { - if File_saas_starter_telemetry_grpc_proto != nil { - return - } - type x struct{} - out := protoimpl.TypeBuilder{ - File: protoimpl.DescBuilder{ - GoPackagePath: reflect.TypeOf(x{}).PkgPath(), - RawDescriptor: unsafe.Slice(unsafe.StringData(file_saas_starter_telemetry_grpc_proto_rawDesc), len(file_saas_starter_telemetry_grpc_proto_rawDesc)), - NumEnums: 0, - NumMessages: 0, - NumExtensions: 0, - NumServices: 0, - }, - GoTypes: file_saas_starter_telemetry_grpc_proto_goTypes, - DependencyIndexes: file_saas_starter_telemetry_grpc_proto_depIdxs, - }.Build() - File_saas_starter_telemetry_grpc_proto = out.File - file_saas_starter_telemetry_grpc_proto_goTypes = nil - file_saas_starter_telemetry_grpc_proto_depIdxs = nil -} diff --git a/module/services/accounts/code/pkg/auth/pg/session_store_test.go b/module/services/accounts/code/pkg/auth/pg/session_store_test.go index ea72e651a..a85061641 100644 --- a/module/services/accounts/code/pkg/auth/pg/session_store_test.go +++ b/module/services/accounts/code/pkg/auth/pg/session_store_test.go @@ -64,7 +64,7 @@ func runSessionStoreTests(m *testing.M) int { deps, err := sdk.WithDependencies(ctx, sdk.WithDebug(), sdk.WithSharedControlChannel(), - sdk.WithExcludedDependencies("cache", "vault", "telemetry"), + sdk.WithExcludedDependencies("cache", "vault"), sdk.WithNamingScope("pgauth-test"), sdk.WithTimeout(120*time.Second), sdk.WithSilence("store"), diff --git a/module/services/accounts/code/pkg/billing/pg/store_test.go b/module/services/accounts/code/pkg/billing/pg/store_test.go index 549ea891a..db2686f5c 100644 --- a/module/services/accounts/code/pkg/billing/pg/store_test.go +++ b/module/services/accounts/code/pkg/billing/pg/store_test.go @@ -45,7 +45,7 @@ func runBillingStoreTests(m *testing.M) int { deps, err := sdk.WithDependencies(ctx, sdk.WithDebug(), sdk.WithSharedControlChannel(), - sdk.WithExcludedDependencies("cache", "vault", "telemetry"), + sdk.WithExcludedDependencies("cache", "vault"), sdk.WithNamingScope("pgbilling-test"), sdk.WithTimeout(120*time.Second), sdk.WithSilence("store"), diff --git a/module/services/accounts/code/pkg/business/service_test.go b/module/services/accounts/code/pkg/business/service_test.go index 800624ed8..b792dbabd 100644 --- a/module/services/accounts/code/pkg/business/service_test.go +++ b/module/services/accounts/code/pkg/business/service_test.go @@ -109,7 +109,7 @@ func runBusinessTests(m *testing.M) int { deps, err := sdk.WithDependencies(ctx, sdk.WithDebug(), sdk.WithSharedControlChannel(), - sdk.WithExcludedDependencies("cache", "telemetry"), + sdk.WithExcludedDependencies("cache"), // Keep this package-owned integration stack distinct from the outer // service runtime and the other package TestMain stacks. The SDK gives // each short-lived flow temporary host ports; the scope also isolates diff --git a/module/services/accounts/code/pkg/cataloggen/deployment_topology_test.go b/module/services/accounts/code/pkg/cataloggen/deployment_topology_test.go index 2e37d274c..50394ce47 100644 --- a/module/services/accounts/code/pkg/cataloggen/deployment_topology_test.go +++ b/module/services/accounts/code/pkg/cataloggen/deployment_topology_test.go @@ -71,16 +71,16 @@ func TestDeploymentTopologyIsDeterministicAndCurrent(t *testing.T) { require.Equal(t, string(readFixture(t, "testdata/network-policy.golden.yaml")), string(first.NetworkPolicy), "run: go generate ./pkg/cataloggen") require.Equal(t, string(readFixture(t, "testdata/mesh-policy.golden.yaml")), string(first.MeshPolicy), "run: go generate ./pkg/cataloggen") - require.Len(t, first.Catalog.GetServices(), 8) + require.Len(t, first.Catalog.GetServices(), 7) require.Len(t, first.Catalog.GetInterfaceEndpoints(), 6) - require.Len(t, first.Catalog.GetPublicEgress(), 4) + require.Len(t, first.Catalog.GetPublicEgress(), 3) endpointCount, dependencyCount := 0, 0 for _, service := range first.Catalog.GetServices() { endpointCount += len(service.GetEndpoints()) dependencyCount += len(service.GetDependencies()) } - require.Equal(t, 13, endpointCount) - require.Equal(t, 8, dependencyCount) + require.Equal(t, 12, endpointCount) + require.Equal(t, 6, dependencyCount) // The accounts REST surface is reachable only through the gateway; the // gateway's REST surface is module-visible because composed modules and // solutions use it (Work Context minting, solution registration) and the @@ -90,7 +90,6 @@ func TestDeploymentTopologyIsDeterministicAndCurrent(t *testing.T) { "auth-gateway": catalogv1.EndpointVisibility_ENDPOINT_VISIBILITY_MODULE, } privateREST := map[string]bool{"accounts": false, "auth-gateway": false} - authGatewayTelemetry := false for _, service := range first.Catalog.GetServices() { if _, ok := privateREST[service.GetName()]; !ok { continue @@ -101,17 +100,23 @@ func TestDeploymentTopologyIsDeterministicAndCurrent(t *testing.T) { privateREST[service.GetName()] = true } } - if service.GetName() == "auth-gateway" { - for _, dependency := range service.GetDependencies() { - if dependency.GetService() == "telemetry" { - require.Equal(t, []string{"grpc"}, dependency.GetEndpoints()) - authGatewayTelemetry = true - } - } - } } require.Equal(t, map[string]bool{"accounts": true, "auth-gateway": true}, privateREST) - require.True(t, authGatewayTelemetry) + // The module owns no collector: telemetry is exported to the cell's, which is + // reached in-mesh, so no service has a telemetry dependency and the gateway, + // whose job is to be the perimeter, has no public egress to reach a backend. + for _, service := range first.Catalog.GetServices() { + require.NotEqual(t, "telemetry", service.GetName()) + for _, dependency := range service.GetDependencies() { + require.NotEqual(t, "telemetry", dependency.GetService(), "%s depends on a telemetry service", service.GetName()) + } + } + egress := map[string]bool{} + for _, grant := range first.Catalog.GetPublicEgress() { + egress[grant.GetService()] = true + } + require.Equal(t, map[string]bool{"accounts": true, "frontend": true, "marketing": true}, egress) + require.False(t, egress["auth-gateway"], "auth-gateway must have no public egress") accountsConnectExposed, accountsAuthorityExposed, gatewayRESTExposed := false, false, false for _, endpoint := range first.Catalog.GetInterfaceEndpoints() { if endpoint.GetService() == "accounts" && endpoint.GetEndpoint() == "connect" { @@ -131,16 +136,18 @@ func TestDeploymentTopologyIsDeterministicAndCurrent(t *testing.T) { require.True(t, accountsConnectExposed) require.True(t, accountsAuthorityExposed) require.True(t, gatewayRESTExposed) - require.Equal(t, 20, strings.Count(string(first.NetworkPolicy), "\nkind: NetworkPolicy\n")) + require.Equal(t, 18, strings.Count(string(first.NetworkPolicy), "\nkind: NetworkPolicy\n")) require.NotContains(t, string(first.NetworkPolicy), "allow-intra-namespace") for _, name := range []string{ "allow-accounts-from-dependents", "allow-auth-gateway-from-dependents", "allow-auth-gateway-to-dependencies", "allow-frontend-to-dependencies", "allow-store-from-bootstrap", "allow-store-bootstrap-to-store", - "allow-frontend-public-egress", "allow-marketing-public-egress", "allow-telemetry-from-dependents", - "allow-telemetry-public-egress", "allow-istio-ingress-to-marketing", "allow-istio-ingress-to-frontend", + "allow-frontend-public-egress", "allow-marketing-public-egress", + "allow-istio-ingress-to-marketing", "allow-istio-ingress-to-frontend", } { require.Contains(t, string(first.NetworkPolicy), "name: "+name) } + require.NotContains(t, string(first.NetworkPolicy), "telemetry") + require.NotContains(t, string(first.NetworkPolicy), "name: allow-auth-gateway-public-egress") require.NotContains(t, string(first.NetworkPolicy), "name: allow-istio-ingress-to-auth-gateway") require.Equal(t, 2, strings.Count(string(first.NetworkPolicy), "codefly.dev/bootstrap-service: store")) require.Contains(t, string(first.NetworkPolicy), "192.175.48.0/24") @@ -165,11 +172,11 @@ func TestDeploymentSpecIsStrictAndComplete(t *testing.T) { require.ErrorContains(t, err, `names unknown endpoint "ghost"`) _, err = cataloggen.BuildDeploymentArtifacts(serviceCatalog, - withService(t, documents, "telemetry", " deployment:\n", " deployment-typo:\n")) + withService(t, documents, "vault", " deployment:\n", " deployment-typo:\n")) require.ErrorContains(t, err, "has no spec.deployment block") _, err = cataloggen.BuildDeploymentArtifacts(serviceCatalog, - withServiceDocument(documents, "ghost", documents.Services["telemetry"])) + withServiceDocument(documents, "ghost", documents.Services["vault"])) require.ErrorContains(t, err, "not declared by module.codefly.yaml") _, err = cataloggen.BuildDeploymentArtifacts(serviceCatalog, @@ -184,7 +191,7 @@ func TestGeneratedCodeflyAndNetworkManifestsParseStrictly(t *testing.T) { require.NoError(t, loadedModule.ValidateInterface(ctx)) loadedServices, err := loadedModule.LoadServices(ctx) require.NoError(t, err) - require.Len(t, loadedServices, 8) + require.Len(t, loadedServices, 7) moduleDocument := readFixture(t, "../../../../../module.codefly.yaml") var moduleEntry struct { @@ -196,7 +203,7 @@ func TestGeneratedCodeflyAndNetworkManifestsParseStrictly(t *testing.T) { require.NoError(t, yaml.Unmarshal(moduleDocument, &module)) _, err = module.Proto(ctx) require.NoError(t, err) - require.Len(t, module.ServiceReferences, 8) + require.Len(t, module.ServiceReferences, 7) for _, reference := range module.ServiceReferences { document := readFixture(t, filepath.Join("../../../../../services", reference.Name, "service.codefly.yaml")) @@ -238,14 +245,14 @@ func TestGeneratedCodeflyAndNetworkManifestsParseStrictly(t *testing.T) { require.False(t, names[document.Metadata.Name], "duplicate NetworkPolicy %s", document.Metadata.Name) names[document.Metadata.Name] = true } - require.Len(t, names, 20) + require.Len(t, names, 18) require.True(t, names["allow-istio-ingress-to-marketing"]) require.True(t, names["allow-istio-ingress-to-frontend"]) require.False(t, names["allow-istio-ingress-to-auth-gateway"]) require.True(t, names["allow-store-from-bootstrap"]) require.True(t, names["allow-store-bootstrap-to-store"]) - require.True(t, names["allow-telemetry-from-dependents"]) - require.True(t, names["allow-telemetry-public-egress"]) + require.False(t, names["allow-telemetry-from-dependents"]) + require.False(t, names["allow-telemetry-public-egress"]) } func TestGeneratedMeshPolicyGatesInternalAuthorityByCallerIdentity(t *testing.T) { @@ -727,7 +734,7 @@ func TestDeploymentTopologyPodPortIsTheAllocationUnlessDeclared(t *testing.T) { } for endpoint, want := range map[string]uint32{ "accounts/connect": 8081, "accounts/grpc": 9090, "accounts/rest": 8080, "accounts/authority": 0, - "auth-gateway/grpc": 9090, "auth-gateway/rest": 8080, "telemetry/grpc": 9090, + "auth-gateway/grpc": 9090, "auth-gateway/rest": 8080, "cache/read": 6379, "cache/write": 6379, "store/tcp": 5432, "vault/http": 8200, "frontend/http": 3000, "marketing/http": 3000, } { diff --git a/module/services/accounts/code/pkg/cataloggen/test_targets_test.go b/module/services/accounts/code/pkg/cataloggen/test_targets_test.go index 93bcbfef3..0340ff899 100644 --- a/module/services/accounts/code/pkg/cataloggen/test_targets_test.go +++ b/module/services/accounts/code/pkg/cataloggen/test_targets_test.go @@ -110,7 +110,6 @@ func TestAccountsTargetMetadataRetainsConservativeInputs(t *testing.T) { require.Contains(t, target.Command, "-tags=pure") } else { require.Contains(t, target.RuntimeServices, "store") - require.NotContains(t, target.RuntimeServices, "telemetry") if startsCache[target.Suite] { require.Contains(t, target.RuntimeServices, "cache") } else { diff --git a/module/services/accounts/code/pkg/cataloggen/testdata/network-policy.golden.yaml b/module/services/accounts/code/pkg/cataloggen/testdata/network-policy.golden.yaml index f2c9e20af..da814e95a 100644 --- a/module/services/accounts/code/pkg/cataloggen/testdata/network-policy.golden.yaml +++ b/module/services/accounts/code/pkg/cataloggen/testdata/network-policy.golden.yaml @@ -196,33 +196,6 @@ spec: --- apiVersion: networking.k8s.io/v1 kind: NetworkPolicy -metadata: - name: allow-telemetry-from-dependents - namespace: saas-starter -spec: - podSelector: - matchLabels: - app: telemetry - policyTypes: - - Ingress - ingress: - - from: - - podSelector: - matchLabels: - app: accounts - ports: - - protocol: TCP - port: 9090 - - from: - - podSelector: - matchLabels: - app: auth-gateway - ports: - - protocol: TCP - port: 9090 ---- -apiVersion: networking.k8s.io/v1 -kind: NetworkPolicy metadata: name: allow-vault-from-dependents namespace: saas-starter @@ -267,13 +240,6 @@ spec: ports: - protocol: TCP port: 5432 - - to: - - podSelector: - matchLabels: - app: telemetry - ports: - - protocol: TCP - port: 9090 - to: - podSelector: matchLabels: @@ -312,13 +278,6 @@ spec: ports: - protocol: TCP port: 6379 - - to: - - podSelector: - matchLabels: - app: telemetry - ports: - - protocol: TCP - port: 9090 --- apiVersion: networking.k8s.io/v1 kind: NetworkPolicy @@ -537,55 +496,3 @@ spec: ports: - protocol: TCP port: 443 ---- -apiVersion: networking.k8s.io/v1 -kind: NetworkPolicy -metadata: - name: allow-telemetry-public-egress - namespace: saas-starter -spec: - podSelector: - matchLabels: - app: telemetry - policyTypes: - - Egress - egress: - - to: - - ipBlock: - cidr: 0.0.0.0/0 - except: - - 0.0.0.0/8 - - 10.0.0.0/8 - - 100.64.0.0/10 - - 127.0.0.0/8 - - 169.254.0.0/16 - - 172.16.0.0/12 - - 192.0.0.0/24 - - 192.0.2.0/24 - - 192.31.196.0/24 - - 192.52.193.0/24 - - 192.88.99.0/24 - - 192.168.0.0/16 - - 192.175.48.0/24 - - 198.18.0.0/15 - - 198.51.100.0/24 - - 203.0.113.0/24 - - 224.0.0.0/4 - - 240.0.0.0/4 - - ipBlock: - cidr: ::/0 - except: - - ::/128 - - ::1/128 - - 64:ff9b::/96 - - 64:ff9b:1::/48 - - 100::/64 - - 2001::/23 - - 2001:db8::/32 - - 2002::/16 - - fc00::/7 - - fe80::/10 - - ff00::/8 - ports: - - protocol: TCP - port: 443 diff --git a/module/services/accounts/code/pkg/infra/postgres_webhooks_test.go b/module/services/accounts/code/pkg/infra/postgres_webhooks_test.go index 89e35cf10..723c709e4 100644 --- a/module/services/accounts/code/pkg/infra/postgres_webhooks_test.go +++ b/module/services/accounts/code/pkg/infra/postgres_webhooks_test.go @@ -102,7 +102,7 @@ func runPostgresInfraTests(m *testing.M) int { deps, err := sdk.WithDependencies(ctx, sdk.WithDebug(), sdk.WithSharedControlChannel(), - sdk.WithExcludedDependencies("cache", "vault", "telemetry"), + sdk.WithExcludedDependencies("cache", "vault"), sdk.WithNamingScope("test-infra"), sdk.WithTimeout(90*time.Second), sdk.WithSilence("store"), diff --git a/module/services/accounts/code/pkg/redisintegration/main_test.go b/module/services/accounts/code/pkg/redisintegration/main_test.go index 904e6345d..9784c936e 100644 --- a/module/services/accounts/code/pkg/redisintegration/main_test.go +++ b/module/services/accounts/code/pkg/redisintegration/main_test.go @@ -47,7 +47,7 @@ func runRedisIntegrationTests(m *testing.M) int { deps, err := sdk.WithDependencies(ctx, sdk.WithDebug(), sdk.WithSharedControlChannel(), - sdk.WithExcludedDependencies("vault", "telemetry"), + sdk.WithExcludedDependencies("vault"), sdk.WithNamingScope("redis-integration"), sdk.WithTimeout(120*time.Second), sdk.WithSilence("store", "cache"), diff --git a/module/services/accounts/service.codefly.yaml b/module/services/accounts/service.codefly.yaml index 511b10963..25c211b2f 100644 --- a/module/services/accounts/service.codefly.yaml +++ b/module/services/accounts/service.codefly.yaml @@ -18,9 +18,6 @@ service-dependencies: - name: store endpoints: - name: tcp - - name: telemetry - endpoints: - - name: grpc - name: vault endpoints: - name: http diff --git a/module/services/auth-gateway/code/external/saas-starter/telemetry/saas-starter_telemetry_grpc.pb.go b/module/services/auth-gateway/code/external/saas-starter/telemetry/saas-starter_telemetry_grpc.pb.go deleted file mode 100644 index 689bf2157..000000000 --- a/module/services/auth-gateway/code/external/saas-starter/telemetry/saas-starter_telemetry_grpc.pb.go +++ /dev/null @@ -1,60 +0,0 @@ -// Code generated by protoc-gen-go. DO NOT EDIT. -// versions: -// protoc-gen-go v1.36.11 -// protoc (unknown) -// source: saas-starter_telemetry_grpc.proto - -package telemetry - -import ( - reflect "reflect" - unsafe "unsafe" - - protoreflect "google.golang.org/protobuf/reflect/protoreflect" - protoimpl "google.golang.org/protobuf/runtime/protoimpl" -) - -const ( - // Verify that this generated code is sufficiently up-to-date. - _ = protoimpl.EnforceVersion(20 - protoimpl.MinVersion) - // Verify that runtime/protoimpl is sufficiently up-to-date. - _ = protoimpl.EnforceVersion(protoimpl.MaxVersion - 20) -) - -var File_saas_starter_telemetry_grpc_proto protoreflect.FileDescriptor - -const file_saas_starter_telemetry_grpc_proto_rawDesc = "" + - "\n" + - "!saas-starter_telemetry_grpc.protoBhB\x1dSaasStarterTelemetryGrpcProtoP\x01ZEgithub.com/codefly-dev/cli/pkg/builder/clients/saas-starter/telemetry" - -var file_saas_starter_telemetry_grpc_proto_goTypes = []any{} -var file_saas_starter_telemetry_grpc_proto_depIdxs = []int32{ - 0, // [0:0] is the sub-list for method output_type - 0, // [0:0] is the sub-list for method input_type - 0, // [0:0] is the sub-list for extension type_name - 0, // [0:0] is the sub-list for extension extendee - 0, // [0:0] is the sub-list for field type_name -} - -func init() { file_saas_starter_telemetry_grpc_proto_init() } -func file_saas_starter_telemetry_grpc_proto_init() { - if File_saas_starter_telemetry_grpc_proto != nil { - return - } - type x struct{} - out := protoimpl.TypeBuilder{ - File: protoimpl.DescBuilder{ - GoPackagePath: reflect.TypeOf(x{}).PkgPath(), - RawDescriptor: unsafe.Slice(unsafe.StringData(file_saas_starter_telemetry_grpc_proto_rawDesc), len(file_saas_starter_telemetry_grpc_proto_rawDesc)), - NumEnums: 0, - NumMessages: 0, - NumExtensions: 0, - NumServices: 0, - }, - GoTypes: file_saas_starter_telemetry_grpc_proto_goTypes, - DependencyIndexes: file_saas_starter_telemetry_grpc_proto_depIdxs, - }.Build() - File_saas_starter_telemetry_grpc_proto = out.File - file_saas_starter_telemetry_grpc_proto_goTypes = nil - file_saas_starter_telemetry_grpc_proto_depIdxs = nil -} diff --git a/module/services/auth-gateway/service.codefly.yaml b/module/services/auth-gateway/service.codefly.yaml index f7eb093c3..de214971f 100644 --- a/module/services/auth-gateway/service.codefly.yaml +++ b/module/services/auth-gateway/service.codefly.yaml @@ -20,9 +20,6 @@ service-dependencies: - name: cache endpoints: - name: write - - name: telemetry - endpoints: - - name: grpc workspace-configuration-dependencies: # APP_BASE_URL: this host's own public origin. The gateway publishes it in # the RFC 9728 document for a solution's MCP endpoint and requires a token's diff --git a/module/services/frontend/code/src/test/pipeline-setup.ts b/module/services/frontend/code/src/test/pipeline-setup.ts index 08746fad9..85d2f0d0f 100644 --- a/module/services/frontend/code/src/test/pipeline-setup.ts +++ b/module/services/frontend/code/src/test/pipeline-setup.ts @@ -15,8 +15,8 @@ // workers, so starting the graph per file would start one graph per file. // // withDependencies passes --exclude-root: it brings up the frontend's -// dependencies (auth-gateway, Accounts, Postgres, Vault, Redis, storage, -// telemetry) without the Next server the frontend itself would run. The +// dependencies (auth-gateway, Accounts, Postgres, Vault, Redis, storage) +// without the Next server the frontend itself would run. The // pipeline tier addresses the gateway directly, so the root service is dead // weight — and leaving it out means these tests can run alongside a // `codefly run service` dev stack instead of fighting it for the Next dev lock. @@ -47,7 +47,7 @@ export default async function setup(): Promise<(() => Promise) | void> { readyService: "auth-gateway", scope, fixture: "dev-admin", - silents: ["store", "cache", "telemetry"], + silents: ["store", "cache"], echo: process.env.CODEFLY_TEST_ECHO === "1", }); } catch (error) { diff --git a/module/services/telemetry/code/collector/collector.go b/module/services/telemetry/code/collector/collector.go deleted file mode 100644 index ed9196499..000000000 --- a/module/services/telemetry/code/collector/collector.go +++ /dev/null @@ -1,218 +0,0 @@ -package collector - -import ( - "bytes" - "context" - "errors" - "fmt" - "io" - "log/slog" - "net/http" - "net/url" - "strings" - "time" - - collectorlogsv1 "go.opentelemetry.io/proto/otlp/collector/logs/v1" - collectormetricsv1 "go.opentelemetry.io/proto/otlp/collector/metrics/v1" - collectortracev1 "go.opentelemetry.io/proto/otlp/collector/trace/v1" - "google.golang.org/protobuf/proto" -) - -type Config struct { - Exporter string - Endpoint string - Headers string - HTTPClient *http.Client - Logger *slog.Logger -} - -type Collector struct { - exporter string - endpoint string - headers http.Header - client *http.Client - logger *slog.Logger -} - -type TraceService struct { - collectortracev1.UnimplementedTraceServiceServer - collector *Collector -} - -type MetricsService struct { - collectormetricsv1.UnimplementedMetricsServiceServer - collector *Collector -} - -type LogsService struct { - collectorlogsv1.UnimplementedLogsServiceServer - collector *Collector -} - -func (c *Collector) TraceService() *TraceService { return &TraceService{collector: c} } - -func (c *Collector) MetricsService() *MetricsService { - return &MetricsService{collector: c} -} - -func (c *Collector) LogsService() *LogsService { return &LogsService{collector: c} } - -func New(config Config) (*Collector, error) { - exporter := strings.ToLower(strings.TrimSpace(config.Exporter)) - // Fail closed on an empty exporter instead of defaulting to "debug". - // An empty value never meant "the operator chose debug" — every committed - // observability.env sets this key explicitly, and so does - // configurations/local-dogfood/observability.env.example — it meant the configuration never reached the process. - // Defaulting it to debug turned that into SILENT trace loss: the collector - // logged "received OTLP signal" and dropped every span while its own - // ConfigMap said otlphttp. Refusing to start surfaces the same fault in one - // line of pod logs. - if exporter == "" { - return nil, errors.New( - "telemetry: OBSERVABILITY_EXPORTER is required and must be debug or otlphttp; " + - "the observability workspace configuration did not reach this process " + - "(on the local-dogfood profile, copy configurations/local-dogfood/observability.env.example to observability.env)") - } - switch exporter { - case "debug": - if strings.TrimSpace(config.Endpoint) != "" || strings.TrimSpace(config.Headers) != "" { - return nil, errors.New("telemetry: debug exporter cannot have an external endpoint or headers") - } - case "otlphttp": - endpoint, err := url.Parse(strings.TrimRight(strings.TrimSpace(config.Endpoint), "/")) - if err != nil || endpoint.Scheme == "" || endpoint.Host == "" { - return nil, errors.New("telemetry: OTLP/HTTP endpoint must be absolute") - } - // Plaintext is allowed only to loopback, which never leaves the pod and so - // is governed by no network policy. Everything else must be HTTPS, because - // that is the only egress this module actually grants telemetry: - // services/telemetry/service.codefly.yaml declares spec.deployment.public-egress-ports - // [443], which renders allow-telemetry-public-egress — TCP 443 to public IP - // space with 10/8, 172.16/12, 192.168/16 and fc00::/7 excepted — on top of a - // namespace-wide default-deny that has no allow-intra-namespace rule. - // - // A cluster-internal plaintext address (a *.svc name on 4318) is therefore - // denied at the network layer no matter what this check says; accepting it - // here would only move the failure from a startup error to a 10s export - // timeout per batch. Reaching an in-cluster collector is a topology change — - // a declared dependency edge with its regenerated NetworkPolicy — not a - // transport exemption in application code. - local := endpoint.Hostname() == "localhost" || endpoint.Hostname() == "127.0.0.1" - if endpoint.Scheme != "https" && (endpoint.Scheme != "http" || !local) { - return nil, errors.New("telemetry: OTLP/HTTP endpoint must use HTTPS") - } - config.Endpoint = endpoint.String() - default: - return nil, errors.New("telemetry: OBSERVABILITY_EXPORTER must be debug or otlphttp") - } - headers, err := parseHeaders(config.Headers) - if err != nil { - return nil, err - } - client := config.HTTPClient - if client == nil { - client = &http.Client{Timeout: 10 * time.Second} - } - logger := config.Logger - if logger == nil { - logger = slog.Default() - } - return &Collector{ - exporter: exporter, - endpoint: strings.TrimRight(config.Endpoint, "/"), - headers: headers, - client: client, - logger: logger, - }, nil -} - -func (s *TraceService) Export( - ctx context.Context, - request *collectortracev1.ExportTraceServiceRequest, -) (*collectortracev1.ExportTraceServiceResponse, error) { - if err := s.collector.deliver(ctx, "traces", request, len(request.GetResourceSpans())); err != nil { - return nil, err - } - return &collectortracev1.ExportTraceServiceResponse{}, nil -} - -func (s *MetricsService) Export( - ctx context.Context, - request *collectormetricsv1.ExportMetricsServiceRequest, -) (*collectormetricsv1.ExportMetricsServiceResponse, error) { - if err := s.collector.deliver(ctx, "metrics", request, len(request.GetResourceMetrics())); err != nil { - return nil, err - } - return &collectormetricsv1.ExportMetricsServiceResponse{}, nil -} - -func (s *LogsService) Export( - ctx context.Context, - request *collectorlogsv1.ExportLogsServiceRequest, -) (*collectorlogsv1.ExportLogsServiceResponse, error) { - if err := s.collector.deliver(ctx, "logs", request, len(request.GetResourceLogs())); err != nil { - return nil, err - } - return &collectorlogsv1.ExportLogsServiceResponse{}, nil -} - -func (c *Collector) deliver(ctx context.Context, signal string, message proto.Message, resources int) error { - if c.exporter == "debug" { - c.logger.Info("received OTLP signal", "signal", signal, "resources", resources) - return nil - } - body, err := proto.Marshal(message) - if err != nil { - return fmt.Errorf("telemetry: encode %s: %w", signal, err) - } - request, err := http.NewRequestWithContext( - ctx, - http.MethodPost, - c.endpoint+"/v1/"+signal, - bytes.NewReader(body), - ) - if err != nil { - return fmt.Errorf("telemetry: create %s export: %w", signal, err) - } - for name, values := range c.headers { - for _, value := range values { - request.Header.Add(name, value) - } - } - request.Header.Set("Content-Type", "application/x-protobuf") - response, err := c.client.Do(request) - if err != nil { - return fmt.Errorf("telemetry: export %s: %w", signal, err) - } - defer func() { _ = response.Body.Close() }() - _, _ = io.Copy(io.Discard, io.LimitReader(response.Body, 64*1024)) - if response.StatusCode < http.StatusOK || response.StatusCode >= http.StatusMultipleChoices { - return fmt.Errorf("telemetry: export %s returned HTTP %d", signal, response.StatusCode) - } - return nil -} - -func parseHeaders(raw string) (http.Header, error) { - headers := make(http.Header) - raw = strings.TrimSpace(raw) - if raw == "" { - return headers, nil - } - values, err := url.ParseQuery(strings.ReplaceAll(raw, ",", "&")) - if err != nil { - return nil, errors.New("telemetry: OTLP headers must use URL-encoded key=value pairs") - } - for name, entries := range values { - name = strings.TrimSpace(name) - if name == "" { - return nil, errors.New("telemetry: OTLP header name is required") - } - for _, value := range entries { - if strings.ContainsAny(value, "\r\n") { - return nil, errors.New("telemetry: OTLP header value contains a newline") - } - headers.Add(name, value) - } - } - return headers, nil -} diff --git a/module/services/telemetry/code/collector/collector_test.go b/module/services/telemetry/code/collector/collector_test.go deleted file mode 100644 index dc121fbb4..000000000 --- a/module/services/telemetry/code/collector/collector_test.go +++ /dev/null @@ -1,134 +0,0 @@ -package collector_test - -import ( - "context" - "io" - "net" - "net/http" - "net/http/httptest" - "sync" - "testing" - - "telemetry/collector" - - collectortracev1 "go.opentelemetry.io/proto/otlp/collector/trace/v1" - tracev1 "go.opentelemetry.io/proto/otlp/trace/v1" - "google.golang.org/grpc" - "google.golang.org/grpc/credentials/insecure" - "google.golang.org/grpc/test/bufconn" - "google.golang.org/protobuf/proto" - - "github.com/stretchr/testify/require" -) - -func TestCollectorReceivesGRPCAndForwardsOTLPHTTP(t *testing.T) { - var mu sync.Mutex - var forwarded collectortracev1.ExportTraceServiceRequest - upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - require.Equal(t, "/v1/traces", r.URL.Path) - require.Equal(t, "Bearer test", r.Header.Get("Authorization")) - body, err := io.ReadAll(r.Body) - require.NoError(t, err) - mu.Lock() - defer mu.Unlock() - require.NoError(t, proto.Unmarshal(body, &forwarded)) - w.WriteHeader(http.StatusOK) - })) - defer upstream.Close() - - sink, err := collector.New(collector.Config{ - Exporter: "otlphttp", - Endpoint: upstream.URL, - Headers: "Authorization=Bearer+test", - }) - require.NoError(t, err) - listener := bufconn.Listen(1024 * 1024) - server := grpc.NewServer() - collectortracev1.RegisterTraceServiceServer(server, sink.TraceService()) - go func() { _ = server.Serve(listener) }() - t.Cleanup(server.Stop) - - connection, err := grpc.NewClient( - "passthrough:///bufconn", - grpc.WithTransportCredentials(insecure.NewCredentials()), - grpc.WithContextDialer(func(context.Context, string) (net.Conn, error) { - return listener.Dial() - }), - ) - require.NoError(t, err) - defer func() { _ = connection.Close() }() - client := collectortracev1.NewTraceServiceClient(connection) - _, err = client.Export(t.Context(), &collectortracev1.ExportTraceServiceRequest{ - ResourceSpans: []*tracev1.ResourceSpans{{}}, - }) - require.NoError(t, err) - mu.Lock() - defer mu.Unlock() - require.Len(t, forwarded.GetResourceSpans(), 1) -} - -func TestCollectorConfigurationFailsClosed(t *testing.T) { - _, err := collector.New(collector.Config{ - Exporter: "debug", Endpoint: "https://collector.example", - }) - require.ErrorContains(t, err, "cannot have") - _, err = collector.New(collector.Config{ - Exporter: "otlphttp", Endpoint: "http://collector.example", - }) - require.ErrorContains(t, err, "HTTPS") -} - -// TestCollectorRequiresExplicitExporter pins the fail-closed behaviour: an empty -// OBSERVABILITY_EXPORTER used to fall back to "debug", which turned a broken -// configuration carrier into silent trace loss instead of a startup failure. -func TestCollectorRequiresExplicitExporter(t *testing.T) { - _, err := collector.New(collector.Config{}) - require.ErrorContains(t, err, "OBSERVABILITY_EXPORTER is required") - - _, err = collector.New(collector.Config{Exporter: " "}) - require.ErrorContains(t, err, "OBSERVABILITY_EXPORTER is required") -} - -// TestCollectorEndpointTransportPolicy pins which OTLP/HTTP destinations may be -// reached in plaintext: loopback only, because loopback never leaves the pod. -// -// The cluster-internal cases are the regression. Exempting *.svc / -// *.svc.cluster.local from the HTTPS requirement looks safe — those names -// resolve only inside the cluster — but this module grants telemetry exactly one -// egress path: public_egress_ports [443] in -// services/telemetry/service.codefly.yaml (spec.deployment), rendered as -// allow-telemetry-public-egress (TCP 443 to public IP space, private ranges -// excepted) over a namespace-wide default-deny with no allow-intra-namespace -// rule. A ClusterIP on 4318 is denied there no matter what this package accepts, -// so accepting it only converts a startup error into a 10s export timeout per -// batch. Reaching an in-cluster collector is a topology change, not a transport -// exemption in application code. -func TestCollectorEndpointTransportPolicy(t *testing.T) { - for _, testCase := range []struct { - name string - endpoint string - accepted bool - }{ - {"loopback name", "http://localhost:4318", true}, - {"loopback address", "http://127.0.0.1:4318", true}, - {"external TLS", "https://example.com", true}, - {"in-cluster collector FQDN is not reachable and not exempt", - "http://otel-collector.otel-collector.svc.cluster.local:4318", false}, - {"in-cluster collector short form", "http://otel-collector.otel-collector.svc:4318", false}, - {"external plaintext", "http://example.com:4318", false}, - {"external plaintext subdomain", "http://otel.internal.ops.example.com:4318", false}, - {"host merely containing svc", "http://svc.example.com:4318", false}, - } { - t.Run(testCase.name, func(t *testing.T) { - _, err := collector.New(collector.Config{ - Exporter: "otlphttp", - Endpoint: testCase.endpoint, - }) - if testCase.accepted { - require.NoError(t, err) - return - } - require.ErrorContains(t, err, "HTTPS") - }) - } -} diff --git a/module/services/telemetry/code/go.mod b/module/services/telemetry/code/go.mod deleted file mode 100644 index 263d1c6b5..000000000 --- a/module/services/telemetry/code/go.mod +++ /dev/null @@ -1,58 +0,0 @@ -module telemetry - -go 1.27.0 - -require ( - github.com/codefly-dev/sdk-go v0.2.1-0.20260926212413-b91ce91eb23e - github.com/stretchr/testify v1.12.1 - go.opentelemetry.io/proto/otlp v1.11.0 - google.golang.org/grpc v1.83.2 - google.golang.org/protobuf v1.36.12 -) - -require ( - buf.build/gen/go/bufbuild/protovalidate/protocolbuffers/go v1.36.12-20260825204119-511051f7f437.2 // indirect - buf.build/go/protovalidate v1.4.0 // indirect - cel.dev/cel-go v0.32.0 // indirect - cel.dev/expr v0.25.3 // indirect - github.com/Masterminds/semver v1.5.0 // indirect - github.com/Masterminds/semver/v3 v3.5.0 // indirect - github.com/antlr4-go/antlr/v4 v4.13.1 // indirect - github.com/codefly-dev/core v0.5.11-0.20260926231217-d481a2d49683 // indirect - github.com/go-openapi/analysis v1.0.0 // indirect - github.com/go-openapi/errors v0.22.8 // indirect - github.com/go-openapi/jsonpointer v1.0.1 // indirect - github.com/go-openapi/jsonreference v1.0.2 // indirect - github.com/go-openapi/loads v0.25.3 // indirect - github.com/go-openapi/spec v1.0.1 // indirect - github.com/go-openapi/strfmt v0.27.2 // indirect - github.com/go-openapi/swag/conv v0.29.1 // indirect - github.com/go-openapi/swag/jsonutils v0.29.1 // indirect - github.com/go-openapi/swag/loading v0.29.1 // indirect - github.com/go-openapi/swag/mangling v0.29.1 // indirect - github.com/go-openapi/swag/pools v0.29.1 // indirect - github.com/go-openapi/swag/stringutils v0.29.1 // indirect - github.com/go-openapi/swag/typeutils v0.29.1 // indirect - github.com/go-openapi/swag/yamlutils v0.29.1 // indirect - github.com/go-viper/mapstructure/v2 v2.5.0 // indirect - github.com/gofrs/flock v0.13.1 // indirect - github.com/google/go-github/v89 v89.0.0 // indirect - github.com/google/go-querystring v1.2.0 // indirect - github.com/google/uuid v1.6.0 // indirect - github.com/grpc-ecosystem/grpc-gateway/v2 v2.30.0 // indirect - github.com/hashicorp/errwrap v1.1.0 // indirect - github.com/hashicorp/go-multierror v1.1.1 // indirect - github.com/mitchellh/mapstructure v1.5.0 // indirect - github.com/oklog/ulid/v2 v2.1.2 // indirect - github.com/pkg/errors v0.9.1 // indirect - github.com/yoheimuta/go-protoparser/v4 v4.14.2 // indirect - go.yaml.in/yaml/v3 v3.0.5 // indirect - golang.org/x/exp v0.0.0-20260824195058-e88cd73687aa // indirect - golang.org/x/net v0.59.0 // indirect - golang.org/x/sys v0.48.0 // indirect - golang.org/x/term v0.46.0 // indirect - golang.org/x/text v0.42.0 // indirect - google.golang.org/genproto/googleapis/api v0.0.0-20260831171406-18b4a7587f8a // indirect - google.golang.org/genproto/googleapis/rpc v0.0.0-20260831171406-18b4a7587f8a // indirect - gopkg.in/yaml.v3 v3.0.1 // indirect -) diff --git a/module/services/telemetry/code/go.sum b/module/services/telemetry/code/go.sum deleted file mode 100644 index a94236ffd..000000000 --- a/module/services/telemetry/code/go.sum +++ /dev/null @@ -1,235 +0,0 @@ -buf.build/gen/go/bufbuild/protovalidate/protocolbuffers/go v1.36.12-20260825204119-511051f7f437.2 h1:NbnlmV26O7oZ1iM5tsCI+GEx+3ZSdrhvnKQ/eWSrLiY= -buf.build/gen/go/bufbuild/protovalidate/protocolbuffers/go v1.36.12-20260825204119-511051f7f437.2/go.mod h1:TCt1lluMFnctISJXvkIQ4x3ABrPuUKCWKyjKdkJNBpw= -buf.build/go/protovalidate v1.4.0 h1:UjLrYbt5VX7+TMOs2+pG5FhZhIG1mSfK4EIopbb4LcM= -buf.build/go/protovalidate v1.4.0/go.mod h1:8vJfzNT6NIG2qm3uFsJDXMlRmG+bQJzbcIn1Aa0vPGs= -cel.dev/cel-go v0.32.0 h1:irvpFKr5EuGPyxeME03ERh0rii1TX+BDAnB9eL3IvNk= -cel.dev/cel-go v0.32.0/go.mod h1:DnVip7tpJSsgZymwfT+m1tnEVy3ivAjSMXPx12YrMkU= -cel.dev/expr v0.25.3 h1:A2jO8jwOugrrovveCWfj0KEZOfqiLgAcwjpHPhzIGw0= -cel.dev/expr v0.25.3/go.mod h1:hrXvqGP6G6gyx8UAHSHJ5RGk//1Oj5nXQ2NI02Nrsg4= -github.com/Masterminds/semver v1.5.0 h1:H65muMkzWKEuNDnfl9d70GUjFniHKHRbFPGBuZ3QEww= -github.com/Masterminds/semver v1.5.0/go.mod h1:MB6lktGJrhw8PrUyiEoblNEGEQ+RzHPF078ddwwvV3Y= -github.com/Masterminds/semver/v3 v3.5.0 h1:kQceYJfbupGfZOKZQg0kou0DgAKhzDg2NZPAwZ/2OOE= -github.com/Masterminds/semver/v3 v3.5.0/go.mod h1:4V+yj/TJE1HU9XfppCwVMZq3I84lprf4nC11bSS5beM= -github.com/Microsoft/go-winio v0.6.2 h1:F2VQgta7ecxGYO8k3ZZz3RS8fVIXVxONVUPlNERoyfY= -github.com/Microsoft/go-winio v0.6.2/go.mod h1:yd8OoFMLzJbo9gZq8j5qaps8bJ9aShtEA8Ipt1oGCvU= -github.com/VividCortex/ewma v1.2.0 h1:f58SaIzcDXrSy3kWaHNvuJgJ3Nmz59Zji6XoJR/q1ow= -github.com/VividCortex/ewma v1.2.0/go.mod h1:nz4BbCtbLyFDeC9SUHbtcT5644juEuWfUAUnGx7j5l4= -github.com/antlr4-go/antlr/v4 v4.13.1 h1:SqQKkuVZ+zWkMMNkjy5FZe5mr5WURWnlpmOuzYWrPrQ= -github.com/antlr4-go/antlr/v4 v4.13.1/go.mod h1:GKmUxMtwp6ZgGwZSva4eWPC5mS6vUAmOABFgjdkM7Nw= -github.com/blang/semver v3.5.1+incompatible h1:cQNTCjp13qL8KC3Nbxr/y2Bqb63oX6wdnnjpJbkM4JQ= -github.com/blang/semver v3.5.1+incompatible/go.mod h1:kRBLl5iJ+tD4TcOOxsy/0fnwebNt5EWlYSAyrTnjyyk= -github.com/blang/semver/v4 v4.0.0 h1:1PFHFE6yCCTv8C1TeyNNarDzntLi7wMI5i/pzqYIsAM= -github.com/blang/semver/v4 v4.0.0/go.mod h1:IbckMUScFkM3pff0VJDNKRiT6TG/YpiHIM2yvyW5YoQ= -github.com/brianvoe/gofakeit/v6 v6.28.0 h1:Xib46XXuQfmlLS2EXRuJpqcw8St6qSZz75OUo0tgAW4= -github.com/brianvoe/gofakeit/v6 v6.28.0/go.mod h1:Xj58BMSnFqcn/fAQeSK+/PLtC5kSb7FJIq4JyGa8vEs= -github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs= -github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= -github.com/cheggaaa/pb/v3 v3.2.1 h1:aprZbFRG+B7+ug76S8QZ6Y1PW168UHzOmbC3wa+aU6I= -github.com/cheggaaa/pb/v3 v3.2.1/go.mod h1:U9hSVxoKqJrZIE3PkFG1xXXNaK/Ilzg+EF/scpXzEdw= -github.com/clipperhouse/uax29/v2 v2.7.0 h1:+gs4oBZ2gPfVrKPthwbMzWZDaAFPGYK72F0NJv2v7Vk= -github.com/clipperhouse/uax29/v2 v2.7.0/go.mod h1:EFJ2TJMRUaplDxHKj1qAEhCtQPW2tJSwu5BF98AuoVM= -github.com/codefly-dev/core v0.5.11-0.20260926231217-d481a2d49683 h1:/vzymsu5B9NWwdhWRtlU+68VcVEVDmeZNChRCXAUwlE= -github.com/codefly-dev/core v0.5.11-0.20260926231217-d481a2d49683/go.mod h1:SVF+t040X+9pAXAqBnNE/JkXwmw9peVuKnHKFSOzyzk= -github.com/codefly-dev/gortk v0.2.0 h1:7bOlS5valYz2zil+fZctQNcPCYBcPj86abcw9N8h1hQ= -github.com/codefly-dev/gortk v0.2.0/go.mod h1:dDWUMFgAP063OCGhTEpV7FFUj9+zTcxxO/nV80VKEwg= -github.com/codefly-dev/sdk-go v0.2.1-0.20260926212413-b91ce91eb23e h1:V8YSUvAm0QQgHY9V0QmHiHuvf+beHnI10D6cJLWB2eI= -github.com/codefly-dev/sdk-go v0.2.1-0.20260926212413-b91ce91eb23e/go.mod h1:X0JAk/gnr6ZcjuPsM+5jvisqY4iyTC1SeVac93iWX8A= -github.com/containerd/errdefs v1.0.0 h1:tg5yIfIlQIrxYtu9ajqY42W3lpS19XqdxRQeEwYG8PI= -github.com/containerd/errdefs v1.0.0/go.mod h1:+YBYIdtsnF4Iw6nWZhJcqGSg/dwvV7tyJ/kCkyJ2k+M= -github.com/containerd/errdefs/pkg v0.3.0 h1:9IKJ06FvyNlexW690DXuQNx2KA2cUJXx151Xdx3ZPPE= -github.com/containerd/errdefs/pkg v0.3.0/go.mod h1:NJw6s9HwNuRhnjJhM7pylWwMyAkmCQvQ4GpJHEqRLVk= -github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1VwoXQT9A3Wy9MM3WgvqSxFWenqJduM= -github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= -github.com/distribution/reference v0.6.0 h1:0IXCQ5g4/QMHHkarYzh5l+u8T3t73zM5QvfrDyIgxBk= -github.com/distribution/reference v0.6.0/go.mod h1:BbU0aIcezP1/5jX/8MP0YiH4SdvB5Y4f/wlDRiLyi3E= -github.com/docker/docker v28.5.2+incompatible h1:DBX0Y0zAjZbSrm1uzOkdr1onVghKaftjlSWt4AFexzM= -github.com/docker/docker v28.5.2+incompatible/go.mod h1:eEKB0N0r5NX/I1kEveEz05bcu8tLC/8azJZsviup8Sk= -github.com/docker/go-connections v0.8.1 h1:JibmG5hULs5qXSr/cp/w3Pw5fZuStt4MOHMUExb29/M= -github.com/docker/go-connections v0.8.1/go.mod h1:no1qkHdjq7kLMGUXYAduOhYPSJxxvgWBh7ogVvptn3Q= -github.com/docker/go-units v0.5.0 h1:69rxXcBk27SvSaaxTtLh/8llcHD8vYHT7WSdRZ/jvr4= -github.com/docker/go-units v0.5.0/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDDbaIK4Dk= -github.com/fatih/color v1.19.0 h1:Zp3PiM21/9Ld6FzSKyL5c/BULoe/ONr9KlbYVOfG8+w= -github.com/fatih/color v1.19.0/go.mod h1:zNk67I0ZUT1bEGsSGyCZYZNrHuTkJJB+r6Q9VuMi0LE= -github.com/felixge/httpsnoop v1.1.0 h1:3YtUj32ZZkqZtt3sZZsClsymw/QDuVfpNhoA31zeORc= -github.com/felixge/httpsnoop v1.1.0/go.mod h1:Zqxgdd+1Rkcz8euOqdr7lqgCRJztwr5hp9vDSi5UZCE= -github.com/fsnotify/fsnotify v1.10.1 h1:b0/UzAf9yR5rhf3RPm9gf3ehBPpf0oZKIjtpKrx59Ho= -github.com/fsnotify/fsnotify v1.10.1/go.mod h1:TLheqan6HD6GBK6PrDWyDPBaEV8LspOxvPSjC+bVfgo= -github.com/go-errors/errors v1.4.2 h1:J6MZopCL4uSllY1OfXM374weqZFFItUbrImctkmUxIA= -github.com/go-errors/errors v1.4.2/go.mod h1:sIVyrIiJhuEF+Pj9Ebtd6P/rEYROXFi3BopGUQ5a5Og= -github.com/go-logr/logr v1.4.4 h1:tG4xh9yMsRCAiodLVTxyrkzSZ9+o0L1Kg/+cPVcbP/8= -github.com/go-logr/logr v1.4.4/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY= -github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag= -github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE= -github.com/go-ole/go-ole v1.3.0 h1:Dt6ye7+vXGIKZ7Xtk4s6/xVdGDQynvom7xCFEdWr6uE= -github.com/go-ole/go-ole v1.3.0/go.mod h1:5LS6F96DhAwUc7C+1HLexzMXY1xGRSryjyPPKW6zv78= -github.com/go-openapi/analysis v1.0.0 h1:sNvbAGCJqUTqIAodr9IVqJMmuZas3YS9ms1dGK9yiJ4= -github.com/go-openapi/analysis v1.0.0/go.mod h1:NhYjJ57fnE+bcE7UwrJyMkhWA3Dfz7TdiBfTVAnos4Y= -github.com/go-openapi/errors v0.22.8 h1:oP7sW7TWc3wFFjrzzj0nI83H2qMBkNjNfSd+XRejk/I= -github.com/go-openapi/errors v0.22.8/go.mod h1:BuUoHcYrU6E7V9gfj1I5wLQqgtIHnup/alXZ8KdgQ0w= -github.com/go-openapi/jsonpointer v1.0.1 h1:2KxywRmNwJkT/FMBa3iRNHEaAxSJvjqoufQZy3au1Mg= -github.com/go-openapi/jsonpointer v1.0.1/go.mod h1:wI7ZYsFmbIi9nBXOZqgDaS/bqOchRGZjqxFli7FBYxY= -github.com/go-openapi/jsonreference v1.0.2 h1:oS4et8FOf3p3UQxEo4Xt0esijmBUM+F259Xl72OSZsc= -github.com/go-openapi/jsonreference v1.0.2/go.mod h1:TbUNSOo+fcorZjFaNoiSDSoaNnnZtqJtLGR1PuvE/Cs= -github.com/go-openapi/loads v0.25.3 h1:V+jKy/thXWdLJUuYC8sZX2dICyAa8M3DokF70jj34P0= -github.com/go-openapi/loads v0.25.3/go.mod h1:LgLyCSOLBL2Qnj0Ps1oo2YH8jZoEKsMeVZCc+ALIxFo= -github.com/go-openapi/spec v1.0.1 h1:lj2vdGpNDcVgwRc6qXdw6qt/KQpCtSa9tnUH6vpDPDk= -github.com/go-openapi/spec v1.0.1/go.mod h1:M//GWQGtDUAjnP37gE6fInLgaczB+FatoipV3H1fYw8= -github.com/go-openapi/strfmt v0.27.2 h1:SG32SlbwNy92s0KJiVxt2joJeFdqIYHvwrA0OU6HqzQ= -github.com/go-openapi/strfmt v0.27.2/go.mod h1:M4CKsMO0Fb8qR10+1Ra75wCKNNquy+Vj+4LWZrhTo2E= -github.com/go-openapi/swag v0.28.0 h1:xkgbOSKj6DZziNpyqRRAOt3GJGtgjgsd2RoyT30VWuw= -github.com/go-openapi/swag v0.28.0/go.mod h1:4qYnT3Cqr1p1VknOdPo70evN4rgQnAg6jwApHyxSGIg= -github.com/go-openapi/swag/cmdutils v0.28.0 h1:7TOeNtkYru1SG8Y34tDh9WBbLsMqGnptuxWiHREPZ4Q= -github.com/go-openapi/swag/cmdutils v0.28.0/go.mod h1:Sm1MVFMkF6guJJ+pQqHnQA3N0j9qALV3NxzDSv6bETM= -github.com/go-openapi/swag/conv v0.29.1 h1:AC4Eh/5c/eUDOUCzzsRC9ghmFgOSBHeRMGIngY0ZUGA= -github.com/go-openapi/swag/conv v0.29.1/go.mod h1:S1X7/ZrBEZOC0Wc8AGxjbcGS92l3WEjA7aPtpl+RaqM= -github.com/go-openapi/swag/fileutils v0.28.0 h1:Z04XWQD7R8Eq+7GnOrjovBxPPmZzsS4gt2H2GPGIViU= -github.com/go-openapi/swag/fileutils v0.28.0/go.mod h1:VvJFZLTZS0AI854gEQz5tk7dBESdLjiNUMSZ/th2ry8= -github.com/go-openapi/swag/jsonutils v0.29.1 h1:AFCxs0eQZ24/QyfhVHM2t49rMz7Vv3XCsZQI6yrNy+c= -github.com/go-openapi/swag/jsonutils v0.29.1/go.mod h1:u3+sCfJpttDpcmS5kpm0yxL6GK0eWgODsx8Yw8fcqNM= -github.com/go-openapi/swag/jsonutils/fixtures_test v0.29.1 h1:BiiXE31Bx9SfpsMmOQj5KYpUhTZBpLVriVhJDuLuY2o= -github.com/go-openapi/swag/jsonutils/fixtures_test v0.29.1/go.mod h1:julgTUKZ9/D0j6O7GKajmRs+812FWxQg/mMpGunWSjg= -github.com/go-openapi/swag/loading v0.29.1 h1:FCv5fG8UhTdDJa2R7w+5O9Ekpcbw7tt0nFWvmDKGBjc= -github.com/go-openapi/swag/loading v0.29.1/go.mod h1:N0ESuem4p2oedKal8EJhciqnJ9Q9Wmt83L1CRB3Fouw= -github.com/go-openapi/swag/mangling v0.29.1 h1:lHALtvYCdxVnRl4GrHmFPwfBTZYIObqdGNSKyu/8D6I= -github.com/go-openapi/swag/mangling v0.29.1/go.mod h1:SAop9pB7PUjQ/CGCNf/JmCKTRK+GDO+RqE9UHqC/N6s= -github.com/go-openapi/swag/netutils v0.28.0 h1:YXN6TALEi2pzts8/8GNm6T61HTAZsieukGZidap989k= -github.com/go-openapi/swag/netutils v0.28.0/go.mod h1:J+WYyFMLtvtCGqa6jLv+YNUmIKI3ZRQRrvfNDMoQoEQ= -github.com/go-openapi/swag/pools v0.29.1 h1:NRogYxdEW9SjRM4mkAOji9iefO4MRXq3p/ZJcoQbUKg= -github.com/go-openapi/swag/pools v0.29.1/go.mod h1:leDcaghjkRAhCuCRv9NfJU5f0mjoU3cT/XZObhMk3pc= -github.com/go-openapi/swag/stringutils v0.29.1 h1:1ykunK7iJQk1uOO7+oUH1ukbsK85fFCOiCFMOVSY+F0= -github.com/go-openapi/swag/stringutils v0.29.1/go.mod h1:7fSqZ+z8Qc0tOfAAK0jVa5qFGrnIlRi6n7NeGGrr1vc= -github.com/go-openapi/swag/typeutils v0.29.1 h1:Nzv9nhnlLCRBPQqfOX+7lB6Guju370or8StT+lIOf6M= -github.com/go-openapi/swag/typeutils v0.29.1/go.mod h1:hxpgDZJVBkBsi/d3MIUosafoFdE5exaQRmVp0zwu3YE= -github.com/go-openapi/swag/yamlutils v0.29.1 h1:69w3tsBajm7MR/fejLy7HD/3J68Ys1SeeZMEzZ3w2sk= -github.com/go-openapi/swag/yamlutils v0.29.1/go.mod h1:rgsp3vT/QdWzKwn43CigDwjOGIenPyTZMKnxEM8jZOA= -github.com/go-openapi/testify/enable/yaml/v2 v2.7.0 h1:wPW6YRgx3+SID1yUy/Xwa17L8kFEaEKod2VRbJDZNUs= -github.com/go-openapi/testify/enable/yaml/v2 v2.7.0/go.mod h1:mI1M88etYbc3PhgHsWQK2kwvNwW5aGFqMPbmib+SGIs= -github.com/go-openapi/testify/v2 v2.7.0 h1:bycOreEj6wfBvijg3YFogZ/sFjTCDmQnwSodSzHa3X8= -github.com/go-openapi/testify/v2 v2.7.0/go.mod h1:SgsVHtfooshd0tublTtJ50FPKhujf47YRqauXXOUxfw= -github.com/go-viper/mapstructure/v2 v2.5.0 h1:vM5IJoUAy3d7zRSVtIwQgBj7BiWtMPfmPEgAXnvj1Ro= -github.com/go-viper/mapstructure/v2 v2.5.0/go.mod h1:oJDH3BJKyqBA2TXFhDsKDGDTlndYOZ6rGS0BRZIxGhM= -github.com/gofrs/flock v0.13.1 h1:jjREztyBeSKBZYAC+mgc1laB+xsgy4kYMf3FbKF2UBo= -github.com/gofrs/flock v0.13.1/go.mod h1:sf4BFiHwnvgxa25DlQoDqXQnwRMEOwqxRq37P6MzzmE= -github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek= -github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps= -github.com/google/gnostic-models v0.6.9 h1:MU/8wDLif2qCXZmzncUQ/BOfxWfthHi63KqpoNbWqVw= -github.com/google/gnostic-models v0.6.9/go.mod h1:CiWsm0s6BSQd1hRn8/QmxqB6BesYcbSZxsz9b0KuDBw= -github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY= -github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8= -github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU= -github.com/google/go-github/v89 v89.0.0 h1:35bEK5XoEcF3PZrlVbl9XN63f5BcJRA/UGkxeC9xPg0= -github.com/google/go-github/v89 v89.0.0/go.mod h1:QLcbU0ipeAqQuR5KSg8c2lql4Qk1EwJ2dWz/0rP4Nho= -github.com/google/go-querystring v1.2.0 h1:yhqkPbu2/OH+V9BfpCVPZkNmUXhb2gBxJArfhIxNtP0= -github.com/google/go-querystring v1.2.0/go.mod h1:8IFJqpSRITyJ8QhQ13bmbeMBDfmeEJZD5A0egEOmkqU= -github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0= -github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= -github.com/grpc-ecosystem/grpc-gateway/v2 v2.30.0 h1:/Tnpcb2E0Pz/tN9s3bfEY2Q8ePCEX9iuS+cneUwncnw= -github.com/grpc-ecosystem/grpc-gateway/v2 v2.30.0/go.mod h1:zOBXOsUaBSjKgmH4OGzV1esUpR3oUSCPYVd2cUBjKYY= -github.com/hashicorp/errwrap v1.0.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4= -github.com/hashicorp/errwrap v1.1.0 h1:OxrOeh75EUXMY8TBjag2fzXGZ40LB6IKw45YeGUDY2I= -github.com/hashicorp/errwrap v1.1.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4= -github.com/hashicorp/go-multierror v1.1.1 h1:H5DkEtf6CXdFp0N0Em5UCwQpXMWke8IA0+lD48awMYo= -github.com/hashicorp/go-multierror v1.1.1/go.mod h1:iw975J/qwKPdAO1clOe2L8331t/9/fmwbPZ6JB6eMoM= -github.com/lufia/plan9stats v0.0.0-20260330125221-c963978e514e h1:Q6MvJtQK/iRcRtzAscm/zF23XxJlbECiGPyRicsX+Ak= -github.com/lufia/plan9stats v0.0.0-20260330125221-c963978e514e/go.mod h1:autxFIvghDt3jPTLoqZ9OZ7s9qTGNAWmYCjVFWPX/zg= -github.com/mattn/go-colorable v0.1.15 h1:+u9SLTRGnXv73cEsnsmoZBom+dMU88B2M0aDcWy0/jY= -github.com/mattn/go-colorable v0.1.15/go.mod h1:6LmQG8QLFO4G5z1gPvYEzlUgJ2wF+stgPZH1UqBm1s8= -github.com/mattn/go-isatty v0.0.24 h1:tGZZoVgT/KiqK1c8ocVLeDS8BSWMRd47J3Lbz7vsReI= -github.com/mattn/go-isatty v0.0.24/go.mod h1:nMCL3Zebbrt45jsMDgnfIwz6ydEQApk5oEI3HqDio6A= -github.com/mattn/go-runewidth v0.0.27 h1:Feg/Oou5zI/wnpgDF6omIU0OokC9GxLC/WRknhVlIR0= -github.com/mattn/go-runewidth v0.0.27/go.mod h1:3qAiGCV4Koz/yuveO58qUefmUTRm8r0IGEXZ9jeHp/8= -github.com/mitchellh/mapstructure v1.5.0 h1:jeMsZIYE/09sWLaz43PL7Gy6RuMjD2eJVyuac5Z2hdY= -github.com/mitchellh/mapstructure v1.5.0/go.mod h1:bFUtVrKA4DC2yAKiSyO/QUcy7e+RRV2QTWOzhPopBRo= -github.com/moby/docker-image-spec v1.3.1 h1:jMKff3w6PgbfSa69GfNg+zN/XLhfXJGnEx3Nl2EsFP0= -github.com/moby/docker-image-spec v1.3.1/go.mod h1:eKmb5VW8vQEh/BAr2yvVNvuiJuY6UIocYsFu/DxxRpo= -github.com/moby/patternmatcher v0.6.1 h1:qlhtafmr6kgMIJjKJMDmMWq7WLkKIo23hsrpR3x084U= -github.com/moby/patternmatcher v0.6.1/go.mod h1:hDPoyOpDY7OrrMDLaYoY3hf52gNCR/YOUYxkhApJIxc= -github.com/monochromegane/go-gitignore v0.0.0-20200626010858-205db1a8cc00 h1:n6/2gBQ3RWajuToeY6ZtZTIKv2v7ThUy5KKusIT0yc0= -github.com/monochromegane/go-gitignore v0.0.0-20200626010858-205db1a8cc00/go.mod h1:Pm3mSP3c5uWn86xMLZ5Sa7JB9GsEZySvHYXCTK4E9q4= -github.com/oklog/ulid/v2 v2.1.2 h1:IEclFb9JNvzYA6MW2SCxbLzcHTVsfqm3PrqGQJH5zec= -github.com/oklog/ulid/v2 v2.1.2/go.mod h1:rcEKHmBBKfef9DhnvX7y1HZBYxjXb0cP5ExxNsTT1QQ= -github.com/opencontainers/go-digest v1.0.0 h1:apOUWs51W5PlhuyGyz9FCeeBIOUDA/6nW8Oi/yOhh5U= -github.com/opencontainers/go-digest v1.0.0/go.mod h1:0JzlMkj0TRzQZfJkVvzbP0HBR3IKzErnv2BNG4W4MAM= -github.com/opencontainers/image-spec v1.1.1 h1:y0fUlFfIZhPF1W537XOLg0/fcx6zcHCJwooC2xJA040= -github.com/opencontainers/image-spec v1.1.1/go.mod h1:qpqAh3Dmcf36wStyyWU+kCeDgrGnAve2nCC8+7h8Q0M= -github.com/pborman/getopt v0.0.0-20170112200414-7148bc3a4c30/go.mod h1:85jBQOZwpVEaDAr341tbn15RS4fCAsIst0qp7i8ex1o= -github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4= -github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= -github.com/power-devops/perfstat v0.0.0-20240221224432-82ca36839d55 h1:o4JXh1EVt9k/+g42oCprj/FisM4qX9L3sZB3upGN2ZU= -github.com/power-devops/perfstat v0.0.0-20240221224432-82ca36839d55/go.mod h1:OmDBASR4679mdNQnz2pUhc2G8CO2JrUAVFDRBDP/hJE= -github.com/rodaine/protogofakeit v0.1.1 h1:ZKouljuRM3A+TArppfBqnH8tGZHOwM/pjvtXe9DaXH8= -github.com/rodaine/protogofakeit v0.1.1/go.mod h1:pXn/AstBYMaSfc1/RqH3N82pBuxtWgejz1AlYpY1mI0= -github.com/santhosh-tekuri/jsonschema/v6 v6.0.3 h1:1EYB5IzjZawrrnELUi78f9fPu57HuXjmddZPjrls/28= -github.com/santhosh-tekuri/jsonschema/v6 v6.0.3/go.mod h1:JXeL+ps8p7/KNMjDQk3TCwPpBy0wYklyWTfbkIzdIFU= -github.com/shirou/gopsutil/v3 v3.24.5 h1:i0t8kL+kQTvpAYToeuiVk3TgDeKOFioZO3Ztz/iZ9pI= -github.com/shirou/gopsutil/v3 v3.24.5/go.mod h1:bsoOS1aStSs9ErQ1WWfxllSeS1K5D+U30r2NfcubMVk= -github.com/shoenig/go-m1cpu v0.2.1 h1:yqRB4fvOge2+FyRXFkXqsyMoqPazv14Yyy+iyccT2E4= -github.com/shoenig/go-m1cpu v0.2.1/go.mod h1:KkDOw6m3ZJQAPHbrzkZki4hnx+pDRR1Lo+ldA56wD5w= -github.com/stretchr/testify v1.12.1 h1:EuwCh5fleGS7H32xRwO3wRGT7DxrDhLAT6FF8MpWDWE= -github.com/stretchr/testify v1.12.1/go.mod h1:MDEgiDPPsNp5cuIrHPPCyornHKgEVbtFUmoNlxoYthg= -github.com/tklauser/go-sysconf v0.4.0 h1:7H0uAN+7RkwWRaxhYXDLqa5V3LPrJeV8wmD9dRUgPQU= -github.com/tklauser/go-sysconf v0.4.0/go.mod h1:8mTNWyog7H+MpKijp4VmKJAd2bbYQ2zuUwkYRbUArPI= -github.com/tklauser/numcpus v0.12.0 h1:NR85qdvHA9pFse3x3weVZ0r0ST8R6l5RHbZrlRaqob4= -github.com/tklauser/numcpus v0.12.0/go.mod h1:ABHeXzJnr/qqwguhClkZKT1/8VABcYrsyUiUGobwWJg= -github.com/xlab/treeprint v1.2.0 h1:HzHnuAF1plUN2zGlAFHbSQP2qJ0ZAD3XF5XD7OesXRQ= -github.com/xlab/treeprint v1.2.0/go.mod h1:gj5Gd3gPdKtR1ikdDK6fnFLdmIS0X30kTTuNd/WEJu0= -github.com/yoheimuta/go-protoparser/v4 v4.14.2 h1:/P/LlX1CF9NaTWEltGcIZVvNlPbhABuAnBtAWpb3+74= -github.com/yoheimuta/go-protoparser/v4 v4.14.2/go.mod h1:AHNNnSWnb0UoL4QgHPiOAg2BniQceFscPI5X/BZNHl8= -github.com/yusufpapurcu/wmi v1.2.4 h1:zFUKzehAFReQwLys1b/iSMl+JQGSCSjtVqQn9bBrPo0= -github.com/yusufpapurcu/wmi v1.2.4/go.mod h1:SBZ9tNy3G9/m5Oi98Zks0QjeHVDvuK0qfxQmPyzfmi0= -go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ64= -go.opentelemetry.io/auto/sdk v1.2.1/go.mod h1:KRTj+aOaElaLi+wW1kO/DZRXwkF4C5xPbEe3ZiIhN7Y= -go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.71.0 h1:B2h3uqicet1CT2N5TOFhS+Gq++9i0/CLmaxvhmhtP5s= -go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.71.0/go.mod h1:dylvB+ZiiwMvsDij9O84Uy7SijLgHMX4mbkncds+4Sw= -go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.70.0 h1:LMuyCAyfalSjDyjdC65nK6N0zoTT63+E/u95X0JovZI= -go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.70.0/go.mod h1:085m8qbm4hgc8rZWGDEa4vmyyo2c3nPxUslYUKUIU04= -go.opentelemetry.io/otel v1.46.0 h1:FHt5/CDyVxi/8IM1CH7VE/rRgq3kLHa2mSTVMO8AWyc= -go.opentelemetry.io/otel v1.46.0/go.mod h1:Gj3SEScelsNC45tp4nSxRYlS+f5iez7W8XPMCt905kE= -go.opentelemetry.io/otel/metric v1.46.0 h1:yBnkXvgV7AXFILZc5K6IZe/CBFF3OS7BJ8ov6/lj0K8= -go.opentelemetry.io/otel/metric v1.46.0/go.mod h1:iPmdWqifKUdzziPkvvzIJXITl56fQx2mGM/DHLB3/2o= -go.opentelemetry.io/otel/sdk v1.46.0 h1:h5CNQQjEbuQXY/JfZtgt3i7HVFV3aHPO2OAwO2eTYPI= -go.opentelemetry.io/otel/sdk v1.46.0/go.mod h1:GAERFXFt5SYCEB+YiKUbMBeza6UaDH7GmGOZEfh2gSM= -go.opentelemetry.io/otel/sdk/metric v1.44.0 h1:3LlKgI+VjbVsjNRFZJZAJ30WjXC5VkNRks6si09iEfI= -go.opentelemetry.io/otel/sdk/metric v1.44.0/go.mod h1:5B5pMARnXxKhltooO4xUuCBorl65a4EpnTalObqOigA= -go.opentelemetry.io/otel/trace v1.46.0 h1:OULy7ccdJnZtJ0UDYFOIGaCmiWzJ8Vi2G/Rsu60qs1c= -go.opentelemetry.io/otel/trace v1.46.0/go.mod h1:J7GAXweO77XSFkB/rmAqk9D6ihszhFjLU+d9WuUxDLI= -go.opentelemetry.io/proto/otlp v1.11.0 h1:5rrYs0Ykyj50sdU/JU0x8etU+LubXWb+gED6TbEdMIk= -go.opentelemetry.io/proto/otlp v1.11.0/go.mod h1:SmVizdCOAm3XBtG1g1NnOdhW6jtddT72hLMhv8VwA8E= -go.yaml.in/yaml/v2 v2.4.2 h1:DzmwEr2rDGHl7lsFgAHxmNz/1NlQ7xLIrlN2h5d1eGI= -go.yaml.in/yaml/v2 v2.4.2/go.mod h1:081UH+NErpNdqlCXm3TtEran0rJZGxAYx9hb/ELlsPU= -go.yaml.in/yaml/v3 v3.0.5 h1:N6y/pJk8buWs9NY5ERU2HSMfm+IuD/OtfdAnq6kESPw= -go.yaml.in/yaml/v3 v3.0.5/go.mod h1:HVTZu1O7/Vkt2N+BFy8Zza+lnLsABggaTM2ZpNIGuKg= -golang.org/x/exp v0.0.0-20260824195058-e88cd73687aa h1:QSyA8ishJCyT21kER9KwNt0b7BM3iRK4x9QXhjN5Fdk= -golang.org/x/exp v0.0.0-20260824195058-e88cd73687aa/go.mod h1:zeBbvyFKDaLwa7CH/zI8KXt7gTl14SF7sO08Pl5jBCM= -golang.org/x/net v0.59.0 h1:5zfYln+w5XCxwrnMMJPufRgNoXEaGxl0wo5GqPXyues= -golang.org/x/net v0.59.0/go.mod h1:2DA/G1UfVbCpQPeWTmMPGY7Cs2PkBkwu743bVX5PIVg= -golang.org/x/sys v0.48.0 h1:bbX/i/6MgT9BVLM9RT1thmxL04yeTAhbEz4SyadbXoo= -golang.org/x/sys v0.48.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og= -golang.org/x/term v0.46.0 h1:3+OXuTbaKDgwk8jTi3aSLHRlmWqHEUDUtxnbFigO4YE= -golang.org/x/term v0.46.0/go.mod h1:+K02xbkittuwc0Am4abfA3Fc+XRGXkvBXNO88NCXPoc= -golang.org/x/text v0.42.0 h1:JbOZXgfeCPU9gacVtYliJqOhD+zhrEqK4LfdpmlUZqI= -golang.org/x/text v0.42.0/go.mod h1:ojzP1Z+2QtioaF8DTtO8K5q7JWVVYwZKenzujK0Zd0E= -gonum.org/v1/gonum v0.17.0 h1:VbpOemQlsSMrYmn7T2OUvQ4dqxQXU+ouZFQsZOx50z4= -gonum.org/v1/gonum v0.17.0/go.mod h1:El3tOrEuMpv2UdMrbNlKEh9vd86bmQ6vqIcDwxEOc1E= -google.golang.org/genproto/googleapis/api v0.0.0-20260831171406-18b4a7587f8a h1:i3TAXhpKc7TUP1VAPiBBrv45kamjoizCC3rOC0cAbOs= -google.golang.org/genproto/googleapis/api v0.0.0-20260831171406-18b4a7587f8a/go.mod h1:CvYJHpbzPlT0fb/PsgtAamdwru/GVxUsomFdXTpOTI8= -google.golang.org/genproto/googleapis/rpc v0.0.0-20260831171406-18b4a7587f8a h1:3Dnd1cDaZlB68lziofO+bJXpjOy8UfRv8Unt+yH8tQ4= -google.golang.org/genproto/googleapis/rpc v0.0.0-20260831171406-18b4a7587f8a/go.mod h1:DjtHYE8FKJLivXcBEjGwndXfIC23G0VpXiXKqG179uA= -google.golang.org/grpc v1.83.2 h1:EManeRomTObA0BU7I8vXgg/78uE5MJ9M8B39EX2WscU= -google.golang.org/grpc v1.83.2/go.mod h1:YPI1hK3kDked6iHvgX3tR0y+nX/qpMFKhPgFsokw1S8= -google.golang.org/protobuf v1.36.12 h1:pJOKDDOyeXErUroCihFAd5LQuwXBSpVnKGrj5o/fwxc= -google.golang.org/protobuf v1.36.12/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco= -gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM= -gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= -gopkg.in/evanphx/json-patch.v4 v4.13.0 h1:czT3CmqEaQ1aanPc5SdlgQrrEIb8w/wwCvWWnfEbYzo= -gopkg.in/evanphx/json-patch.v4 v4.13.0/go.mod h1:p8EYWUEYMpynmqDbY58zCKCFZw8pRWMG4EsWvDvM72M= -gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= -gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= -k8s.io/kube-openapi v0.0.0-20241212222426-2c72e554b1e7 h1:hcha5B1kVACrLujCKLbr8XWMxCxzQx42DY8QKYJrDLg= -k8s.io/kube-openapi v0.0.0-20241212222426-2c72e554b1e7/go.mod h1:GewRfANuJ70iYzvn+i4lezLDAFzvjxZYK1gn1lWcfas= -sigs.k8s.io/kustomize/api v0.21.1 h1:lzqbzvz2CSvsjIUZUBNFKtIMsEw7hVLJp0JeSIVmuJs= -sigs.k8s.io/kustomize/api v0.21.1/go.mod h1:f3wkKByTrgpgltLgySCntrYoq5d3q7aaxveSagwTlwI= -sigs.k8s.io/kustomize/kyaml v0.21.1 h1:IVlbmhC076nf6foyL6Taw4BkrLuEsXUXNpsE+ScX7fI= -sigs.k8s.io/kustomize/kyaml v0.21.1/go.mod h1:hmxADesM3yUN2vbA5z1/YTBnzLJ1dajdqpQonwBL1FQ= -sigs.k8s.io/yaml v1.5.0 h1:M10b2U7aEUY6hRtU870n2VTPgR5RZiL/I6Lcc2F4NUQ= -sigs.k8s.io/yaml v1.5.0/go.mod h1:wZs27Rbxoai4C0f8/9urLZtZtF3avA3gKvGyPdDqTO4= diff --git a/module/services/telemetry/code/main.go b/module/services/telemetry/code/main.go deleted file mode 100644 index f4d0ee527..000000000 --- a/module/services/telemetry/code/main.go +++ /dev/null @@ -1,128 +0,0 @@ -package main - -import ( - "context" - "fmt" - "log" - "net" - "os" - "os/signal" - "strconv" - "syscall" - - "telemetry/collector" - - codefly "github.com/codefly-dev/sdk-go" - collectorlogsv1 "go.opentelemetry.io/proto/otlp/collector/logs/v1" - collectormetricsv1 "go.opentelemetry.io/proto/otlp/collector/metrics/v1" - collectortracev1 "go.opentelemetry.io/proto/otlp/collector/trace/v1" - "google.golang.org/grpc" -) - -// observabilityConfiguration is the workspace configuration group that carries -// the collector's upstream-exporter settings. Values arrive from either half of -// the group — configurations//observability.env or its .secret.env -// sibling, which is where OTEL_EXPORTER_OTLP_HEADERS lives — because -// WorkspaceValue consults the public namespace first and then the secret one. -const observabilityConfiguration = "observability" - -// exporterKeys are the three settings that make up one exporter configuration. -// They are resolved together, never individually; see resolveExporterConfig. -var exporterKeys = struct{ exporter, endpoint, headers string }{ - exporter: "OBSERVABILITY_EXPORTER", - endpoint: "OTEL_EXPORTER_OTLP_ENDPOINT", - headers: "OTEL_EXPORTER_OTLP_HEADERS", -} - -// workspaceExporterConfig reads all three settings from the observability -// workspace configuration group, reporting whether the group delivered any of -// them. -// -// Reading the group is load-bearing, not cosmetic. telemetry declares -// `observability` as a workspace-configuration-dependency, and a declared group -// reaches a deployed pod ONLY under prefixed names — -// CODEFLY__WORKSPACE_CONFIGURATION__OBSERVABILITY__, via envFrom a -// ConfigMap — while codefly.Init builds an in-process map without ever mutating -// the process environment. A bare os.Getenv("OBSERVABILITY_EXPORTER") therefore -// resolved to "" wherever the configuration arrives that way; collector.New -// defaulted that empty value to "debug", and the collector logged and dropped -// every span it received while its own ConfigMap said otlphttp. -func workspaceExporterConfig(ctx context.Context) (collector.Config, bool) { - value := func(key string) string { - resolved, err := codefly.For(ctx).WorkspaceValue(observabilityConfiguration, key) - if err != nil { - return "" - } - return resolved - } - config := collector.Config{ - Exporter: value(exporterKeys.exporter), - Endpoint: value(exporterKeys.endpoint), - Headers: value(exporterKeys.headers), - } - return config, config.Exporter != "" || config.Endpoint != "" || config.Headers != "" -} - -// resolveExporterConfig picks ONE authority for the whole exporter -// configuration: the workspace group if it delivered anything at all, otherwise -// the bare process environment, so a local or non-Codefly run still works. -// -// The choice is deliberately made for the tuple, not per key. collector.New -// cross-validates these three values (debug rejects an endpoint; otlphttp -// requires one), so mixing sources produces configurations no operator wrote. -// Resolving key by key meant a developer who exported OBSERVABILITY_EXPORTER -// and OTEL_EXPORTER_OTLP_ENDPOINT in their shell could no -// longer start the collector: the committed local group supplied -// OBSERVABILITY_EXPORTER=debug while its empty endpoint fell through to the -// shell's, and debug+endpoint is refused. The same split let an ambient -// OTEL_EXPORTER_OTLP_HEADERS — the credential, and a name the OpenTelemetry -// spec makes ubiquitous — attach itself to a workspace-configured endpoint it -// was never issued for, whenever the observability.secret.env half was not -// provisioned. One authority for all three makes both states unrepresentable. -func resolveExporterConfig(ctx context.Context) collector.Config { - if config, delivered := workspaceExporterConfig(ctx); delivered { - return config - } - return collector.Config{ - Exporter: os.Getenv(exporterKeys.exporter), - Endpoint: os.Getenv(exporterKeys.endpoint), - Headers: os.Getenv(exporterKeys.headers), - } -} - -func main() { - ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM) - defer stop() - provider, err := codefly.Init(ctx) - if err != nil { - log.Fatal(err) - } - ctx = provider.Inject(ctx) - defer codefly.CatchPanic(ctx) - - port := codefly.For(ctx).WithDefaultNetwork().API("grpc").NetworkInstance().Port - if port == 0 { - log.Fatal("telemetry: Codefly did not inject the collector gRPC port") - } - sink, err := collector.New(resolveExporterConfig(ctx)) - if err != nil { - log.Fatal(err) - } - listener, err := net.Listen("tcp", net.JoinHostPort("", strconv.Itoa(int(port)))) - if err != nil { - log.Fatal(err) - } - server := grpc.NewServer() - collectortracev1.RegisterTraceServiceServer(server, sink.TraceService()) - collectormetricsv1.RegisterMetricsServiceServer(server, sink.MetricsService()) - collectorlogsv1.RegisterLogsServiceServer(server, sink.LogsService()) - go func() { - if err := server.Serve(listener); err != nil { - log.Printf("telemetry: serve: %v", err) - stop() - } - }() - fmt.Printf("OpenTelemetry collector listening on Codefly gRPC endpoint %d\n", port) - <-ctx.Done() - server.GracefulStop() -} diff --git a/module/services/telemetry/code/main_test.go b/module/services/telemetry/code/main_test.go deleted file mode 100644 index d7002508f..000000000 --- a/module/services/telemetry/code/main_test.go +++ /dev/null @@ -1,98 +0,0 @@ -package main - -import ( - "context" - "testing" - - "telemetry/collector" - - codefly "github.com/codefly-dev/sdk-go" - "github.com/stretchr/testify/require" -) - -// clearWorkspaceObservability removes every carrier the observability group can -// arrive under, so a test that means "no workspace configuration" actually gets -// it. Without this a test passes only because a neighbouring test happened to -// restore these first, and fails wherever they are genuinely exported — which is -// any process started by `codefly run`. -func clearWorkspaceObservability(t *testing.T) { - t.Helper() - for _, prefix := range []string{ - "CODEFLY__WORKSPACE_CONFIGURATION__OBSERVABILITY__", - "CODEFLY__WORKSPACE_SECRET_CONFIGURATION__OBSERVABILITY__", - } { - for _, key := range []string{ - "OBSERVABILITY_EXPORTER", - "OTEL_EXPORTER_OTLP_ENDPOINT", - "OTEL_EXPORTER_OTLP_HEADERS", - } { - t.Setenv(prefix+key, "") - } - } -} - -// TestResolveExporterConfigPrefersWorkspaceValue is the regression test for the -// boundary that broke in deployment: workspace configuration reaches the pod -// under prefixed names only, so reading the bare name returned "" and the -// collector silently downgraded itself to the debug exporter. -func TestResolveExporterConfigPrefersWorkspaceValue(t *testing.T) { - clearWorkspaceObservability(t) - t.Setenv("CODEFLY__WORKSPACE_CONFIGURATION__OBSERVABILITY__OBSERVABILITY_EXPORTER", "otlphttp") - t.Setenv("CODEFLY__WORKSPACE_CONFIGURATION__OBSERVABILITY__OTEL_EXPORTER_OTLP_ENDPOINT", - "https://collector.example") - t.Setenv("CODEFLY__WORKSPACE_SECRET_CONFIGURATION__OBSERVABILITY__OTEL_EXPORTER_OTLP_HEADERS", - "Authorization=Bearer+workspace") - // The bare names are what main.go used to read; they must lose to the - // prefixed workspace values above — including the header, which is the value - // that carries the credential. - t.Setenv("OBSERVABILITY_EXPORTER", "debug") - t.Setenv("OTEL_EXPORTER_OTLP_ENDPOINT", "https://wrong-backend.example") - t.Setenv("OTEL_EXPORTER_OTLP_HEADERS", "Authorization=Bearer+ambient") - require.NoError(t, codefly.LoadEnvironmentVariables()) - - config := resolveExporterConfig(context.Background()) - require.Equal(t, "otlphttp", config.Exporter) - require.Equal(t, "https://collector.example", config.Endpoint) - require.Equal(t, "Authorization=Bearer+workspace", config.Headers) -} - -// TestResolveExporterConfigTakesWholeTupleFromOneAuthority is the regression -// test for per-key resolution. The workspace group selects debug and says -// nothing about an endpoint or headers; the process environment carries both. -// Resolving key by key produced debug+endpoint — a combination collector.New -// refuses — so a developer who had exported these in their shell could no -// longer start the collector at all. It also -// stopped an ambient OTEL_EXPORTER_OTLP_HEADERS credential from being attached -// to an endpoint it was never issued for. -func TestResolveExporterConfigTakesWholeTupleFromOneAuthority(t *testing.T) { - clearWorkspaceObservability(t) - t.Setenv("CODEFLY__WORKSPACE_CONFIGURATION__OBSERVABILITY__OBSERVABILITY_EXPORTER", "debug") - t.Setenv("OTEL_EXPORTER_OTLP_ENDPOINT", "https://shell-backend.example") - t.Setenv("OTEL_EXPORTER_OTLP_HEADERS", "Authorization=Bearer+ambient") - require.NoError(t, codefly.LoadEnvironmentVariables()) - - config := resolveExporterConfig(context.Background()) - require.Equal(t, "debug", config.Exporter) - require.Empty(t, config.Endpoint, "endpoint must not be mixed in from the process environment") - require.Empty(t, config.Headers, "an ambient credential must not reach a workspace-selected exporter") - - // The consequence the mixing produced: the collector refused to start. - _, err := collector.New(config) - require.NoError(t, err) -} - -// TestResolveExporterConfigFallsBackToProcessEnvironment keeps a local or -// non-Codefly run working: with no workspace configuration injected, the bare -// process variables are still honoured — as a complete tuple. -func TestResolveExporterConfigFallsBackToProcessEnvironment(t *testing.T) { - clearWorkspaceObservability(t) - t.Setenv("OBSERVABILITY_EXPORTER", "otlphttp") - t.Setenv("OTEL_EXPORTER_OTLP_ENDPOINT", "https://collector.example") - t.Setenv("OTEL_EXPORTER_OTLP_HEADERS", "Authorization=Bearer+ambient") - require.NoError(t, codefly.LoadEnvironmentVariables()) - - config := resolveExporterConfig(context.Background()) - require.Equal(t, "otlphttp", config.Exporter) - require.Equal(t, "https://collector.example", config.Endpoint) - require.Equal(t, "Authorization=Bearer+ambient", config.Headers) -} diff --git a/module/services/telemetry/code/pkg/gen/saas/telemetry/v1/schema_marker.pb.go b/module/services/telemetry/code/pkg/gen/saas/telemetry/v1/schema_marker.pb.go deleted file mode 100644 index 6821690f7..000000000 --- a/module/services/telemetry/code/pkg/gen/saas/telemetry/v1/schema_marker.pb.go +++ /dev/null @@ -1,117 +0,0 @@ -// Code generated by protoc-gen-go. DO NOT EDIT. -// versions: -// protoc-gen-go v1.36.11 -// protoc (unknown) -// source: saas/telemetry/v1/schema_marker.proto - -package telemetryv1 - -import ( - reflect "reflect" - sync "sync" - unsafe "unsafe" - - protoreflect "google.golang.org/protobuf/reflect/protoreflect" - protoimpl "google.golang.org/protobuf/runtime/protoimpl" -) - -const ( - // Verify that this generated code is sufficiently up-to-date. - _ = protoimpl.EnforceVersion(20 - protoimpl.MinVersion) - // Verify that runtime/protoimpl is sufficiently up-to-date. - _ = protoimpl.EnforceVersion(protoimpl.MaxVersion - 20) -) - -// SchemaMarker keeps the Codefly protobuf sync input valid without duplicating -// the standard OpenTelemetry Protocol service definitions. -type SchemaMarker struct { - state protoimpl.MessageState `protogen:"open.v1"` - unknownFields protoimpl.UnknownFields - sizeCache protoimpl.SizeCache -} - -func (x *SchemaMarker) Reset() { - *x = SchemaMarker{} - mi := &file_saas_telemetry_v1_schema_marker_proto_msgTypes[0] - ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) - ms.StoreMessageInfo(mi) -} - -func (x *SchemaMarker) String() string { - return protoimpl.X.MessageStringOf(x) -} - -func (*SchemaMarker) ProtoMessage() {} - -func (x *SchemaMarker) ProtoReflect() protoreflect.Message { - mi := &file_saas_telemetry_v1_schema_marker_proto_msgTypes[0] - if x != nil { - ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) - if ms.LoadMessageInfo() == nil { - ms.StoreMessageInfo(mi) - } - return ms - } - return mi.MessageOf(x) -} - -// Deprecated: Use SchemaMarker.ProtoReflect.Descriptor instead. -func (*SchemaMarker) Descriptor() ([]byte, []int) { - return file_saas_telemetry_v1_schema_marker_proto_rawDescGZIP(), []int{0} -} - -var File_saas_telemetry_v1_schema_marker_proto protoreflect.FileDescriptor - -const file_saas_telemetry_v1_schema_marker_proto_rawDesc = "" + - "\n" + - "%saas/telemetry/v1/schema_marker.proto\x12\x11saas.telemetry.v1\"\x0e\n" + - "\fSchemaMarkerB\xc1\x01\n" + - "\x15com.saas.telemetry.v1B\x11SchemaMarkerProtoP\x01Z/telemetry/pkg/gen/saas/telemetry/v1;telemetryv1\xa2\x02\x03STX\xaa\x02\x11Saas.Telemetry.V1\xca\x02\x11Saas\\Telemetry\\V1\xe2\x02\x1dSaas\\Telemetry\\V1\\GPBMetadata\xea\x02\x13Saas::Telemetry::V1b\x06proto3" - -var ( - file_saas_telemetry_v1_schema_marker_proto_rawDescOnce sync.Once - file_saas_telemetry_v1_schema_marker_proto_rawDescData []byte -) - -func file_saas_telemetry_v1_schema_marker_proto_rawDescGZIP() []byte { - file_saas_telemetry_v1_schema_marker_proto_rawDescOnce.Do(func() { - file_saas_telemetry_v1_schema_marker_proto_rawDescData = protoimpl.X.CompressGZIP(unsafe.Slice(unsafe.StringData(file_saas_telemetry_v1_schema_marker_proto_rawDesc), len(file_saas_telemetry_v1_schema_marker_proto_rawDesc))) - }) - return file_saas_telemetry_v1_schema_marker_proto_rawDescData -} - -var file_saas_telemetry_v1_schema_marker_proto_msgTypes = make([]protoimpl.MessageInfo, 1) -var file_saas_telemetry_v1_schema_marker_proto_goTypes = []any{ - (*SchemaMarker)(nil), // 0: saas.telemetry.v1.SchemaMarker -} -var file_saas_telemetry_v1_schema_marker_proto_depIdxs = []int32{ - 0, // [0:0] is the sub-list for method output_type - 0, // [0:0] is the sub-list for method input_type - 0, // [0:0] is the sub-list for extension type_name - 0, // [0:0] is the sub-list for extension extendee - 0, // [0:0] is the sub-list for field type_name -} - -func init() { file_saas_telemetry_v1_schema_marker_proto_init() } -func file_saas_telemetry_v1_schema_marker_proto_init() { - if File_saas_telemetry_v1_schema_marker_proto != nil { - return - } - type x struct{} - out := protoimpl.TypeBuilder{ - File: protoimpl.DescBuilder{ - GoPackagePath: reflect.TypeOf(x{}).PkgPath(), - RawDescriptor: unsafe.Slice(unsafe.StringData(file_saas_telemetry_v1_schema_marker_proto_rawDesc), len(file_saas_telemetry_v1_schema_marker_proto_rawDesc)), - NumEnums: 0, - NumMessages: 1, - NumExtensions: 0, - NumServices: 0, - }, - GoTypes: file_saas_telemetry_v1_schema_marker_proto_goTypes, - DependencyIndexes: file_saas_telemetry_v1_schema_marker_proto_depIdxs, - MessageInfos: file_saas_telemetry_v1_schema_marker_proto_msgTypes, - }.Build() - File_saas_telemetry_v1_schema_marker_proto = out.File - file_saas_telemetry_v1_schema_marker_proto_goTypes = nil - file_saas_telemetry_v1_schema_marker_proto_depIdxs = nil -} diff --git a/module/services/telemetry/proto/README.md b/module/services/telemetry/proto/README.md deleted file mode 100644 index e70d7c6dc..000000000 --- a/module/services/telemetry/proto/README.md +++ /dev/null @@ -1,9 +0,0 @@ -# Telemetry protobuf input - -The telemetry gateway implements the standard OpenTelemetry Protocol services -from `go.opentelemetry.io/proto/otlp`; it does not duplicate those upstream -service definitions. - -The local `SchemaMarker` is deliberately code-generation-free. It keeps the -Codefly Go gRPC agent's required protobuf sync input valid for a service that -depends exclusively on upstream wire schemas. diff --git a/module/services/telemetry/proto/buf.gen.yaml b/module/services/telemetry/proto/buf.gen.yaml deleted file mode 100644 index 63afd5b44..000000000 --- a/module/services/telemetry/proto/buf.gen.yaml +++ /dev/null @@ -1,11 +0,0 @@ -version: v2 -clean: true -managed: - enabled: true - override: - - file_option: go_package_prefix - value: telemetry/pkg/gen -plugins: - - local: protoc-gen-go - out: ../code/pkg/gen - opt: paths=source_relative diff --git a/module/services/telemetry/proto/buf.yaml b/module/services/telemetry/proto/buf.yaml deleted file mode 100644 index cb3b14b9f..000000000 --- a/module/services/telemetry/proto/buf.yaml +++ /dev/null @@ -1,6 +0,0 @@ -version: v2 -modules: - - path: . -lint: - use: - - STANDARD diff --git a/module/services/telemetry/proto/saas/telemetry/v1/schema_marker.proto b/module/services/telemetry/proto/saas/telemetry/v1/schema_marker.proto deleted file mode 100644 index ad48f2919..000000000 --- a/module/services/telemetry/proto/saas/telemetry/v1/schema_marker.proto +++ /dev/null @@ -1,7 +0,0 @@ -syntax = "proto3"; - -package saas.telemetry.v1; - -// SchemaMarker keeps the Codefly protobuf sync input valid without duplicating -// the standard OpenTelemetry Protocol service definitions. -message SchemaMarker {} diff --git a/module/services/telemetry/service.codefly.yaml b/module/services/telemetry/service.codefly.yaml deleted file mode 100644 index b4e307607..000000000 --- a/module/services/telemetry/service.codefly.yaml +++ /dev/null @@ -1,26 +0,0 @@ -name: telemetry -version: 0.0.0 -description: | - In-graph OTLP gateway. Accounts resolves this endpoint exclusively - through the Codefly SDK; the collector either logs locally or forwards - trace, metric, and log protobufs to a configured OTLP/HTTP destination. -agent: - kind: codefly:service - name: go-grpc - version: 0.1.50 - publisher: codefly.dev -workspace-configuration-dependencies: - - observability -endpoints: - - name: grpc -spec: - hot-reload: true - protocol-output-dirs: - - code/pkg/gen - deployment: - # No endpoint-ports: go-grpc binds every endpoint on the port Codefly - # allocates for it, so each pod port is its deployed port, which the - # render takes from Codefly. endpoint-ports is only for a port the - # process binds at a fixed number the allocation does not give it. - public-egress-ports: - - 443 diff --git a/module/tools/composition/composition_test.go b/module/tools/composition/composition_test.go index cf7e16a9f..7f99bf294 100644 --- a/module/tools/composition/composition_test.go +++ b/module/tools/composition/composition_test.go @@ -329,11 +329,18 @@ permissions: func writeCoreInput(t *testing.T, fixture *coreCompositionFixture) { t.Helper() + // The input names the package release it was made for, which the generator + // holds to the package it runs inside: the version is the manifest's, not a + // literal that every release would have to chase. + manifest, err := corecomposition.LoadPackageManifest(findModuleRoot(t)) + if err != nil { + t.Fatalf("load package manifest: %v", err) + } input := corecomposition.CompositionInput{ Schema: "codefly/composition-input/v2", Module: fixture.descriptor.Name, Package: "codefly/saas-starter", - Version: "0.1.0", + Version: manifest.Version, ConsumerRoot: fixture.consumerRoot, Projection: fixture.projection, Contracts: map[string]string{ diff --git a/module/tools/modulepackage/modulepackage_test.go b/module/tools/modulepackage/modulepackage_test.go index 0a08bfe86..da79a25c8 100644 --- a/module/tools/modulepackage/modulepackage_test.go +++ b/module/tools/modulepackage/modulepackage_test.go @@ -23,11 +23,11 @@ func TestStarterManifestIsTheCoreV2Contract(t *testing.T) { if err != nil { t.Fatal(err) } - if manifest.ID != "codefly/saas-starter" || manifest.Version != "0.1.0" { + if manifest.ID != "codefly/saas-starter" || manifest.Version != "0.2.0" { t.Fatalf("unexpected package identity: %s@%s", manifest.ID, manifest.Version) } - if len(manifest.Services) != 8 { - t.Fatalf("provided service count = %d, want 8", len(manifest.Services)) + if len(manifest.Services) != 7 { + t.Fatalf("provided service count = %d, want 7", len(manifest.Services)) } for _, contract := range []string{"composition", "frontendPlugin", "settings", "permissions", "fixtures"} { if manifest.Contracts[contract] == "" { From 0d37dc7005d8f3ed1aeaed4eda91bc9496c4afd5 Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Tue, 6 Oct 2026 17:27:28 -0400 Subject: [PATCH 42/69] fix(audit): lock the audit queue and quarantine before a rollback counts them The down migrations for the audit queue and the audit quarantine counted the rows and refused when the table was not empty, but they counted before anything kept writers out. A transaction that had inserted a row and not yet committed was invisible to the count, so the count read zero. DROP TABLE then waited for that transaction, and once it committed, dropped the table beneath the row it had just written: the only copy of an audit event, destroyed by a rollback written to refuse exactly that. Both migrations now take an ACCESS EXCLUSIVE lock on the table before the count, in the same transaction as the DROP. The rollback waits for the in-flight writer, the count sees what it committed, the rollback refuses, and nothing can insert between the count and the drop. Migrations 20 and 21 do not count and then drop, so they are unchanged. The new test holds an uncommitted insert on one session, runs the down migration on another, commits the insert, and asserts the rollback refuses and the event survives. Against the previous SQL both cases fail: the down migration returns no error and drops the table. --- .../pkg/infra/postgres_audit_rollback_test.go | 156 ++++++++++++++++++ .../migrations/19_audit_event_queue.down.sql | 9 + .../22_audit_event_quarantine.down.sql | 9 + 3 files changed, 174 insertions(+) diff --git a/module/services/accounts/code/pkg/infra/postgres_audit_rollback_test.go b/module/services/accounts/code/pkg/infra/postgres_audit_rollback_test.go index 6a2c1b361..320ac5c7f 100644 --- a/module/services/accounts/code/pkg/infra/postgres_audit_rollback_test.go +++ b/module/services/accounts/code/pkg/infra/postgres_audit_rollback_test.go @@ -7,6 +7,7 @@ import ( "os" "strings" "testing" + "time" "github.com/jackc/pgx/v5" "github.com/jackc/pgx/v5/pgxpool" @@ -114,3 +115,158 @@ func TestRollingBackTheQuarantineRefusesWhileRowsAreSetAside(t *testing.T) { require.Equal(t, held+1, countQuarantine()) }) } + +// migrationSessions hands fn two independent sessions with the authority +// migrations run under: the first stands in for a request or the relay that is +// writing to the table, the second for the operator running a down migration. +// Neither is wrapped in a transaction: fn begins and ends its own. +func migrationSessions(t *testing.T, fn func(ctx context.Context, writer, roller *pgx.Conn)) { + t.Helper() + run := func(ctx context.Context, writer *pgx.Conn) { + roller, err := pgx.ConnectConfig(ctx, writer.Config()) + require.NoError(t, err) + defer roller.Close(ctx) + fn(ctx, writer, roller) + } + if raw := os.Getenv("ACCOUNTS_INSTALLER_TEST_DATABASE_URL"); raw != "" { + writer, err := pgx.Connect(testCtx, raw) + require.NoError(t, err) + defer writer.Close(testCtx) + run(testCtx, writer) + return + } + asMigrationOwner(t, func(ctx context.Context, conn *pgxpool.Conn) { run(ctx, conn.Conn()) }) +} + +// A down migration that counts the rows and then drops the table must keep +// writers out between the two. A row inserted by a transaction that has not +// committed is invisible to the count, and DROP TABLE then waits for that +// transaction and drops the table beneath the row it just committed: the only +// copy of an event, gone, by a rollback that was written to refuse exactly that. +func TestRollingBackWaitsForAnInFlightWriterBeforeItCounts(t *testing.T) { + cases := []struct { + name string + down string + table string + writerRole string + insert string + refusal string + }{ + { + name: "queue", + down: "19_audit_event_queue.down.sql", + table: "public.audit_event_queue", + writerRole: "app_control_plane", + insert: `INSERT INTO public.audit_event_queue (id, event_type, schema_version, actor_type, resource, created_at) + VALUES ($1, 'saas.session.revoked', 1, 'system', 'session', now())`, + refusal: "audit_event_queue still holds 1 queued audit events", + }, + { + name: "quarantine", + down: "22_audit_event_quarantine.down.sql", + table: "public.audit_event_quarantine", + writerRole: "app_job_worker", + insert: `INSERT INTO public.audit_event_quarantine (seq, xact_id, id, event_type, schema_version, actor_type, resource, created_at, enqueued_at, error) + VALUES (9000000001, pg_current_xact_id(), $1, 'saas.session.revoked', 1, 'system', 'session', now(), now(), 'refused')`, + refusal: "audit_event_quarantine still holds 1 audit events", + }, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + down := readMigration(t, tc.down) + eventID := business.NewIDString() + + migrationSessions(t, func(ctx context.Context, writer, roller *pgx.Conn) { + ctx, cancel := context.WithTimeout(ctx, 60*time.Second) + defer cancel() + + // The race needs an empty table, or the count refuses for the + // rows already there and proves nothing. The migration owner + // may truncate; no role the application runs as can. + empty := func() { + _, err := roller.Exec(context.WithoutCancel(ctx), "TRUNCATE "+tc.table) + require.NoError(t, err) + } + empty() + defer empty() + + // The roller opens first and is left waiting on the writer. + rolling, err := roller.Begin(ctx) + require.NoError(t, err) + var rollerPID int + require.NoError(t, rolling.QueryRow(ctx, "SELECT pg_backend_pid()").Scan(&rollerPID)) + defer func() { _ = rolling.Rollback(context.WithoutCancel(ctx)) }() + + // An insert that has not committed. + writing, err := writer.Begin(ctx) + require.NoError(t, err) + defer func() { _ = writing.Rollback(context.WithoutCancel(ctx)) }() + _, err = writing.Exec(ctx, "SET LOCAL ROLE "+tc.writerRole) + require.NoError(t, err) + _, err = writing.Exec(ctx, tc.insert, eventID) + require.NoError(t, err) + + finished := make(chan error, 1) + go func() { + _, err := rolling.Exec(ctx, down) + finished <- err + }() + // Whatever the down migration does, it is stuck behind the writer. + deadline := time.Now().Add(15 * time.Second) + for { + var waiting bool + require.NoError(t, writing.QueryRow(ctx, + `SELECT EXISTS (SELECT 1 FROM pg_locks WHERE pid = $1 AND NOT granted)`, rollerPID).Scan(&waiting)) + if waiting { + break + } + select { + case err := <-finished: + t.Fatalf("the down migration finished while an insert was uncommitted: %v", err) + default: + } + require.True(t, time.Now().Before(deadline), "the down migration never waited on the uncommitted insert") + time.Sleep(20 * time.Millisecond) + } + select { + case err := <-finished: + t.Fatalf("the down migration finished while an insert was uncommitted: %v", err) + case <-time.After(200 * time.Millisecond): + } + + require.NoError(t, writing.Commit(ctx)) + + var verdict error + select { + case verdict = <-finished: + case <-time.After(30 * time.Second): + t.Fatal("the down migration never finished after the insert committed") + } + require.Error(t, verdict, "the down migration dropped %s beneath an event that committed while it ran", tc.table) + require.Contains(t, verdict.Error(), tc.refusal) + require.NoError(t, rolling.Rollback(ctx)) + + var kept int + kept, err = countWhere(ctx, roller, tc.table, eventID) + require.NoError(t, err) + require.Equal(t, 1, kept, "the committed event is still in %s", tc.table) + }) + }) + } +} + +// countWhere counts the rows of table with the given id, as the relay's role, +// the only one both audit tables admit a read from. +func countWhere(ctx context.Context, conn *pgx.Conn, table, id string) (int, error) { + tx, err := conn.Begin(ctx) + if err != nil { + return 0, err + } + defer func() { _ = tx.Rollback(context.WithoutCancel(ctx)) }() + if _, err := tx.Exec(ctx, "SET LOCAL ROLE app_job_worker"); err != nil { + return 0, err + } + var n int + err = tx.QueryRow(ctx, "SELECT count(*) FROM "+table+" WHERE id = $1", id).Scan(&n) + return n, err +} diff --git a/module/services/store/migrations/19_audit_event_queue.down.sql b/module/services/store/migrations/19_audit_event_queue.down.sql index 822a9c4e9..9a2290c56 100644 --- a/module/services/store/migrations/19_audit_event_queue.down.sql +++ b/module/services/store/migrations/19_audit_event_queue.down.sql @@ -7,6 +7,15 @@ -- The check reads as the relay's role: the queue forces row level security, and -- that role's policy admits every row, so the count is the real one rather than -- whatever the migrating session's own policies happen to show it. +-- +-- The table is locked first, before anything is counted. A writer that has +-- inserted but not committed is invisible to the count, and DROP TABLE waits for +-- it and then drops the table beneath the row it committed. Taking the lock +-- first makes this wait for that writer instead, and the count that follows sees +-- what it committed; nothing can insert between the count and the drop, because +-- the lock is held until this transaction ends. The lock is taken as the +-- migrating role, which owns the table, before the role switch below. +LOCK TABLE public.audit_event_queue IN ACCESS EXCLUSIVE MODE; SET LOCAL ROLE app_job_worker; DO $$ DECLARE diff --git a/module/services/store/migrations/22_audit_event_quarantine.down.sql b/module/services/store/migrations/22_audit_event_quarantine.down.sql index 8a2db7c38..780836b52 100644 --- a/module/services/store/migrations/22_audit_event_quarantine.down.sql +++ b/module/services/store/migrations/22_audit_event_quarantine.down.sql @@ -3,6 +3,15 @@ -- -- The check reads as the relay's role, the only one the quarantine's row level -- security admits. +-- +-- The table is locked first, before anything is counted. A relay that has moved +-- a row in but not committed is invisible to the count, and DROP TABLE waits for +-- it and then drops the table beneath the row it committed. Taking the lock +-- first makes this wait for that transaction instead, and the count that follows +-- sees what it committed; nothing can insert between the count and the drop, +-- because the lock is held until this transaction ends. The lock is taken as the +-- migrating role, which owns the table, before the role switch below. +LOCK TABLE public.audit_event_quarantine IN ACCESS EXCLUSIVE MODE; SET LOCAL ROLE app_job_worker; DO $$ DECLARE From da9d88ab82469620735e2ce19c9d90fab0157d64 Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Tue, 6 Oct 2026 17:33:16 -0400 Subject: [PATCH 43/69] fix(audit): refuse an existing ClickHouse table that is not replicated on a cluster Ensure checked an existing audit table's columns and TTL and accepted it, whatever its engine. With AUDIT_CLICKHOUSE_CLUSTER set, a table left as a plain MergeTree (a one-server deployment that was later given a cluster, or one created by hand) lives on a single node. insert_quorum counts replicas, and a non-replicated table has none, so the insert is acknowledged and the relay deletes its queue row, with the only copy of the event on one node. describe now reads the engine from system.tables beside engine_full, and on a cluster Ensure refuses any audit table whose engine is not a Replicated*MergeTree, with an error naming the table, its engine and the cluster. Every table is checked, not only the first. Without a cluster nothing changes: MergeTree is still accepted, and so is a replicated table. Tests run against a catalogue fake, so they run everywhere: a cluster refuses MergeTree on either table, a MergeTree variant that is not replicated and a non-MergeTree engine, and accepts the Replicated engines; one server accepts MergeTree. A server test on a two-replica cluster pins the same against ClickHouse's own engine names. Against the previous code every refusal case fails: Ensure returns no error. The configuration reference and ADR 0009 now say a table already on a cluster must be replicated. --- module/FEATURES.md | 2 +- module/docs/adr/0009-audit-store-swap.md | 6 +- .../clickhousestore/clickhousestore.go | 40 +++- .../clickhousestore/clickhousestore_test.go | 175 ++++++++++++++++++ .../clickhousestore/cluster_test.go | 63 ++++++- 5 files changed, 273 insertions(+), 13 deletions(-) diff --git a/module/FEATURES.md b/module/FEATURES.md index de965ea79..930b3f22f 100644 --- a/module/FEATURES.md +++ b/module/FEATURES.md @@ -657,7 +657,7 @@ Environment variables consumed by the api: | `AUDIT_SINK` | `postgres` (default), `both` (tee to `AUDIT_EXTERNAL_URL`, optional `AUDIT_EXTERNAL_TOKEN`), or `bigquery` / `clickhouse` (swap the audit store of record, ADR 0009); `external` is refused | | `AUDIT_BIGQUERY_PROJECT`, `AUDIT_BIGQUERY_DATASET` | Required under `bigquery`: the store of record. The dataset must exist; the `audit_events` and `audit_event_details` tables are created if missing. Credentials are Application Default Credentials only. The service's identity holds no `bigquery.jobs.create`: it writes with streaming inserts (a batch is split into requests of at most 500 rows and 8 MiB; a single row too large for any request is refused by naming its event, and is handled as any row the warehouse refuses) and reads through the Storage Read API (`bigquery.tables.getData`, `bigquery.readsessions.create`/`getData`/`update`), because append plus job creation would permit DML | | `AUDIT_CLICKHOUSE_DSN` | Required under `clickhouse`: the store of record, as `clickhouse://:@:9000/` (add `?secure=true` for TLS), from the deployment's secret — never logged. The database must exist; the `audit_events` and `audit_event_details` tables are created if missing (partitioned by month, a TTL per tier) and refused if their columns or TTL differ. The user needs `CREATE TABLE`, `INSERT` and `SELECT` on that database, and no `ALTER`, `DELETE`, `TRUNCATE` or `DROP`. Tested against ClickHouse 25.8 | -| `AUDIT_CLICKHOUSE_CLUSTER` | Optional under `clickhouse`: a cluster name or macro (`{cluster}`); the tables are then created `ON CLUSTER` as `ReplicatedMergeTree`, coordinated by ClickHouse Keeper under the servers' default replica path and name. On a cluster every insert waits for a majority of the replicas (`insert_quorum=auto`, one quorum insert at a time, a 60-second timeout) before the relay deletes its queue row, and every read is served only by a replica that holds all of them (`select_sequential_consistency`); a replica that has not caught up fails the read, which the caller retries. Without a cluster neither setting is sent | +| `AUDIT_CLICKHOUSE_CLUSTER` | Optional under `clickhouse`: a cluster name or macro (`{cluster}`); the tables are then created `ON CLUSTER` as `ReplicatedMergeTree`, coordinated by ClickHouse Keeper under the servers' default replica path and name; a table already there must be a `Replicated*MergeTree` too, and startup refuses one that is not (it lives on one node, so no quorum can make an insert durable). On a cluster every insert waits for a majority of the replicas (`insert_quorum=auto`, one quorum insert at a time, a 60-second timeout) before the relay deletes its queue row, and every read is served only by a replica that holds all of them (`select_sequential_consistency`); a replica that has not caught up fails the read, which the caller retries. Without a cluster neither setting is sent | | `AUDIT_EVENTS_RETENTION_DAYS` | Required under `clickhouse`: the compliance window in days — the events table's TTL — at least `AUDIT_CONTENT_RETENTION_DAYS` | | `AUDIT_ARCHIVE_URL` | Required under a swap value: the locked archive, as `gs://` or `s3://` (the scheme picks the writer; any warehouse pairs with either). The bucket carries a retention lock the deployment sets (GCS Bucket Lock, S3 Object Lock default retention) and receives one create-only object per relay batch. S3 takes credentials and region from the AWS default chain and `AWS_REGION`, and an S3-compatible store's endpoint from `AWS_ENDPOINT_URL_S3` (addressed path-style); the writer needs `s3:PutObject` only | | `AUDIT_DEPLOYMENT_ID` | Required under a swap value: stamped on every record and the archive path | diff --git a/module/docs/adr/0009-audit-store-swap.md b/module/docs/adr/0009-audit-store-swap.md index 391d72424..a69fa2cd6 100644 --- a/module/docs/adr/0009-audit-store-swap.md +++ b/module/docs/adr/0009-audit-store-swap.md @@ -296,7 +296,11 @@ Postgres transaction. - A single-node deployment uses MergeTree and sends neither the quorum nor the sequential-read setting below. When `AUDIT_CLICKHOUSE_CLUSTER` is set, tables are created `ON CLUSTER` as `ReplicatedMergeTree` with ClickHouse - Keeper; the cluster must exist before the adapter starts. Every insert then + Keeper; the cluster must exist before the adapter starts, and a table that + already exists must be a `Replicated*MergeTree` as well: a `MergeTree` table + lives on one node, where no quorum can make an insert durable, so startup + refuses it by naming the table and its engine rather than accepting it for + its columns and TTL. Every insert then carries `insert_quorum=auto` (a majority of the replicas must hold it), `insert_quorum_parallel=0` (one quorum insert at a time) and a 60-second `insert_quorum_timeout`, and every read carries diff --git a/module/services/accounts/code/pkg/auditstore/clickhousestore/clickhousestore.go b/module/services/accounts/code/pkg/auditstore/clickhousestore/clickhousestore.go index aba6524e5..cbc5e931e 100644 --- a/module/services/accounts/code/pkg/auditstore/clickhousestore/clickhousestore.go +++ b/module/services/accounts/code/pkg/auditstore/clickhousestore/clickhousestore.go @@ -69,8 +69,10 @@ type Config struct { // coordinated by ClickHouse Keeper under the servers' default replica path // and name. Every insert then waits for a majority of the replicas before // it is acknowledged, and every read is served only by a replica that holds - // all of them (insert_quorum, select_sequential_consistency). Empty creates - // the tables as MergeTree on the one server and sends neither setting. + // all of them (insert_quorum, select_sequential_consistency). A table that + // already exists must be a Replicated*MergeTree too: Ensure refuses a + // MergeTree one, which holds its rows on one node. Empty creates the tables + // as MergeTree on the one server and sends neither setting. Cluster string // DeploymentID confines every read to this deployment's events. DeploymentID string @@ -330,7 +332,10 @@ const unknownDatabase = 81 // and refuses a table it finds that does not match: the service's user holds // no ALTER, so a mismatch is the deployment's to fix, and a table expiring at // a window other than the configured one would keep events or content longer -// or shorter than the deployment declared. +// or shorter than the deployment declared. On a cluster it also refuses a table +// that is not a Replicated*MergeTree, whatever its columns and window: a table +// on one node cannot hold a quorum insert, and the relay deletes its queue rows +// on the insert's acknowledgement. func (s *Store) Ensure(ctx context.Context) error { var current string var found uint64 @@ -361,6 +366,11 @@ func (s *Store) Ensure(ctx context.Context) error { return fmt.Errorf("clickhouse audit store: table %s.%s was not created", s.database, spec.name) } } + if s.cluster != "" { + if err := replicated(s.cluster, existing); err != nil { + return fmt.Errorf("clickhouse audit store: table %s.%s: %w", s.database, spec.name, err) + } + } if err := conforms(spec, existing); err != nil { return fmt.Errorf("clickhouse audit store: table %s.%s: %w", s.database, spec.name, err) } @@ -378,22 +388,25 @@ func (s *Store) ensureError(doing string, err error) error { return fmt.Errorf("clickhouse audit store: %s: %w", doing, err) } -// tableShape is what system.tables and system.columns say of a table. +// tableShape is what system.tables and system.columns say of a table: its +// engine (system.tables.engine), the full engine clause with its partitioning, +// ordering and TTL (engine_full), and its columns. type tableShape struct { + engine string engineFull string columns map[string]string } func (s *Store) describe(ctx context.Context, table string) (*tableShape, error) { p := &sqlParams{} - rows, err := s.conn.Query(ctx, "SELECT engine_full FROM system.tables WHERE database = currentDatabase() AND name = "+p.text(table), p.args...) + rows, err := s.conn.Query(ctx, "SELECT engine, engine_full FROM system.tables WHERE database = currentDatabase() AND name = "+p.text(table), p.args...) if err != nil { return nil, err } var shape *tableShape for rows.Next() { shape = &tableShape{columns: map[string]string{}} - if err := rows.Scan(&shape.engineFull); err != nil { + if err := rows.Scan(&shape.engine, &shape.engineFull); err != nil { _ = rows.Close() return nil, err } @@ -420,6 +433,21 @@ func (s *Store) describe(ctx context.Context, table string) (*tableShape, error) return shape, nil } +// replicated reports why an existing table cannot serve a cluster: its engine +// is not one of the Replicated*MergeTree engines (ReplicatedMergeTree, +// ReplicatedReplacingMergeTree, ...). A MergeTree table lives on the one node +// that holds it, so insert_quorum has no replicas to wait for and an +// acknowledged insert is stored once; the relay deletes its queue row on that +// acknowledgement. The service's user holds no DROP or ALTER, so the +// deployment replaces the table; the store creates it replicated when it is +// missing. +func replicated(cluster string, existing *tableShape) error { + if strings.HasPrefix(existing.engine, "Replicated") && strings.HasSuffix(existing.engine, "MergeTree") { + return nil + } + return fmt.Errorf("engine %s is not replicated, and cluster %q is configured: a cluster needs a Replicated*MergeTree table, because a quorum insert cannot be made durable on one node; replace the table with a ReplicatedMergeTree one", existing.engine, cluster) +} + // ttlClause is the TTL clause of an engine_full, up to its SETTINGS. var ttlClause = regexp.MustCompile(` TTL (.+?)(?: SETTINGS |$)`) diff --git a/module/services/accounts/code/pkg/auditstore/clickhousestore/clickhousestore_test.go b/module/services/accounts/code/pkg/auditstore/clickhousestore/clickhousestore_test.go index 905c74a14..5b8d2bc07 100644 --- a/module/services/accounts/code/pkg/auditstore/clickhousestore/clickhousestore_test.go +++ b/module/services/accounts/code/pkg/auditstore/clickhousestore/clickhousestore_test.go @@ -4,6 +4,7 @@ import ( "context" "encoding/json" "errors" + "strconv" "strings" "testing" "time" @@ -425,3 +426,177 @@ func TestAClusterInsertsWithAQuorumAndReadsSequentially(t *testing.T) { require.Equal(t, value, clustered.readSettings[name], "the pinned reading settings are the same on a cluster: %s", name) } } + +// catalogueConn answers what Ensure asks of a server's system tables from a +// fixed catalogue, so the engines Ensure accepts and refuses are pinned without +// a server: the database it reports, each table's engine and engine_full, and +// each table's columns. It records every statement Ensure executes, since a +// table found must never be created over. +type catalogueConn struct { + Conn + database string + tables map[string]catalogueTable + executed []string +} + +type catalogueTable struct { + engine, engineFull string + columns map[string]string +} + +func (c *catalogueConn) Exec(_ context.Context, query string, _ ...any) error { + c.executed = append(c.executed, query) + return nil +} + +func (c *catalogueConn) Query(_ context.Context, query string, args ...any) (driver.Rows, error) { + table := func() (catalogueTable, bool) { + if len(args) != 1 { + return catalogueTable{}, false + } + named, ok := args[0].(driver.NamedValue) + if !ok { + return catalogueTable{}, false + } + found, ok := c.tables[named.Value.(string)] + return found, ok + } + switch { + case strings.Contains(query, "system.databases"): + return &catalogueRows{rows: [][]any{{c.database, uint64(1)}}}, nil + case strings.Contains(query, "system.tables"): + found, ok := table() + if !ok { + return &catalogueRows{}, nil + } + return &catalogueRows{rows: [][]any{{found.engine, found.engineFull}}}, nil + case strings.Contains(query, "system.columns"): + found, _ := table() + var rows [][]any + for name, typ := range found.columns { + rows = append(rows, []any{name, typ}) + } + return &catalogueRows{rows: rows}, nil + } + return nil, errors.New("catalogueConn: unexpected query: " + query) +} + +// catalogueRows scans a row into its destinations from the right: system.tables +// is read as (engine, engine_full) or as engine_full alone, whichever the store +// asks for. +type catalogueRows struct { + driver.Rows + rows [][]any + next int +} + +func (r *catalogueRows) Next() bool { + r.next++ + return r.next <= len(r.rows) +} + +func (r *catalogueRows) Scan(dest ...any) error { + row := r.rows[r.next-1] + if len(dest) > len(row) { + return errors.New("catalogueRows: more destinations than columns") + } + for i, d := range dest { + value := row[len(row)-len(dest)+i] + switch d := d.(type) { + case *string: + *d = value.(string) + case *uint64: + *d = value.(uint64) + default: + return errors.New("catalogueRows: unsupported destination") + } + } + return nil +} + +func (r *catalogueRows) Err() error { return nil } +func (r *catalogueRows) Close() error { return nil } + +// existingTables is a catalogue holding both audit tables exactly as the store +// writes them, with the given engine for each. +func existingTables(store *Store, engineFor func(table string) string) *catalogueConn { + conn := &catalogueConn{database: store.database, tables: map[string]catalogueTable{}} + for _, spec := range store.tableSpecs() { + columns := map[string]string{} + for _, column := range spec.columns { + columns[column.Name] = column.Type + } + engine := engineFor(spec.name) + engineFull := engine + if strings.HasPrefix(engine, "Replicated") { + engineFull += "('/clickhouse/tables/{uuid}/{shard}', '{replica}')" + } + engineFull += " PARTITION BY toYYYYMM(occurred_at) ORDER BY (deployment_id, org_id, occurred_at, event_id) " + + "TTL occurred_at + toIntervalDay(" + strconv.Itoa(spec.days) + ") SETTINGS index_granularity = 8192" + conn.tables[spec.name] = catalogueTable{engine: engine, engineFull: engineFull, columns: columns} + } + return conn +} + +func same(engine string) func(string) string { return func(string) string { return engine } } + +// A cluster's inserts are made durable by insert_quorum, which counts replicas; +// a MergeTree table lives on one node and has none to count. The relay deletes +// its queue rows on the acknowledgement, so a table found on a cluster that is +// not a Replicated*MergeTree is refused, naming the table and its engine, rather +// than accepted for having the right columns and TTL. +func TestEnsureOnAClusterRequiresEveryTableToBeReplicated(t *testing.T) { + clustered := validConfig() + clustered.Cluster = "audit_cluster" + + for name, tc := range map[string]struct { + cfg Config + engines func(table string) string + refuse []string + }{ + "a cluster refuses a MergeTree events table": { + cfg: clustered, engines: same("MergeTree"), + refuse: []string{"table audit.audit_events", "engine MergeTree", `cluster "audit_cluster"`, "Replicated*MergeTree"}, + }, + "a cluster refuses a MergeTree details table beside a replicated events table": { + cfg: clustered, + engines: func(table string) string { + if table == DetailsTable { + return "MergeTree" + } + return "ReplicatedMergeTree" + }, + refuse: []string{"table audit.audit_event_details", "engine MergeTree"}, + }, + "a cluster refuses a MergeTree variant that is not replicated": { + cfg: clustered, engines: same("ReplacingMergeTree"), + refuse: []string{"table audit.audit_events", "engine ReplacingMergeTree"}, + }, + "a cluster refuses an engine that is not a MergeTree at all": { + cfg: clustered, engines: same("Memory"), + refuse: []string{"table audit.audit_events", "engine Memory"}, + }, + "a cluster accepts ReplicatedMergeTree": {cfg: clustered, engines: same("ReplicatedMergeTree")}, + "a cluster accepts another Replicated MergeTree": {cfg: clustered, engines: same("ReplicatedReplacingMergeTree")}, + "one server accepts MergeTree": {cfg: validConfig(), engines: same("MergeTree")}, + "one server keeps accepting a replicated table": {cfg: validConfig(), engines: same("ReplicatedMergeTree")}, + } { + t.Run(name, func(t *testing.T) { + store, err := New(panicConn{}, tc.cfg) + require.NoError(t, err) + conn := existingTables(store, tc.engines) + store.conn = conn + + err = store.Ensure(context.Background()) + if tc.refuse == nil { + require.NoError(t, err) + } else { + require.Error(t, err) + for _, want := range tc.refuse { + require.ErrorContains(t, err, want) + } + } + require.Empty(t, conn.executed, "a table that exists is never created over") + }) + } +} diff --git a/module/services/accounts/code/pkg/auditstore/clickhousestore/cluster_test.go b/module/services/accounts/code/pkg/auditstore/clickhousestore/cluster_test.go index 0655fe57b..a37d48e23 100644 --- a/module/services/accounts/code/pkg/auditstore/clickhousestore/cluster_test.go +++ b/module/services/accounts/code/pkg/auditstore/clickhousestore/cluster_test.go @@ -10,6 +10,7 @@ import ( "accounts/pkg/business" "github.com/ClickHouse/clickhouse-go/v2" + "github.com/ClickHouse/clickhouse-go/v2/lib/driver" "github.com/google/uuid" "github.com/stretchr/testify/require" ) @@ -22,8 +23,11 @@ import ( // // It is skipped otherwise. The database is created and dropped on the cluster. -func TestServerClusterInsertsWaitForAQuorumAndReadsAreSequential(t *testing.T) { - cluster := strings.TrimSpace(os.Getenv("AUDIT_CLICKHOUSE_TEST_CLUSTER")) +// clusterDatabase creates a database for one test on the cluster, connects to +// it, and returns the cluster's name with it. +func clusterDatabase(t *testing.T) (conn driver.Conn, database, cluster string) { + t.Helper() + cluster = strings.TrimSpace(os.Getenv("AUDIT_CLICKHOUSE_TEST_CLUSTER")) dsn := strings.TrimSpace(os.Getenv("AUDIT_CLICKHOUSE_TEST_DSN")) if cluster == "" || dsn == "" { t.Skip("AUDIT_CLICKHOUSE_TEST_CLUSTER and AUDIT_CLICKHOUSE_TEST_DSN are not both set; see the comment above") @@ -34,14 +38,20 @@ func TestServerClusterInsertsWaitForAQuorumAndReadsAreSequential(t *testing.T) { admin, err := clickhouse.Open(options) require.NoError(t, err) t.Cleanup(func() { _ = admin.Close() }) - database := "audit_" + strings.ReplaceAll(uuid.NewString(), "-", "")[:12] + database = "audit_" + strings.ReplaceAll(uuid.NewString(), "-", "")[:12] require.NoError(t, admin.Exec(ctx, "CREATE DATABASE "+database+" ON CLUSTER '"+cluster+"'")) t.Cleanup(func() { _ = admin.Exec(ctx, "DROP DATABASE IF EXISTS "+database+" ON CLUSTER '"+cluster+"' SYNC") }) scoped := *options scoped.Auth.Database = database - conn, err := clickhouse.Open(&scoped) + conn, err = clickhouse.Open(&scoped) require.NoError(t, err) t.Cleanup(func() { _ = conn.Close() }) + return conn, database, cluster +} + +func TestServerClusterInsertsWaitForAQuorumAndReadsAreSequential(t *testing.T) { + conn, database, cluster := clusterDatabase(t) + ctx := context.Background() store := serverStore(t, conn, database, func(cfg *Config) { cfg.Cluster = cluster }) require.NoError(t, store.Ensure(ctx)) @@ -65,7 +75,7 @@ func TestServerClusterInsertsWaitForAQuorumAndReadsAreSequential(t *testing.T) { // The server saw the quorum on the inserts, and the sequential reading on // the reads, whatever the user's profile says. - require.NoError(t, admin.Exec(ctx, "SYSTEM FLUSH LOGS")) + require.NoError(t, conn.Exec(ctx, "SYSTEM FLUSH LOGS")) var inserts, reads uint64 require.NoError(t, store.queryRow(ctx, "SELECT count() FROM system.query_log WHERE type = 'QueryFinish' AND query_kind = 'Insert' "+ "AND current_database = currentDatabase() AND Settings['insert_quorum'] = 'auto' AND Settings['insert_quorum_parallel'] = '0'", nil, &inserts)) @@ -74,3 +84,46 @@ func TestServerClusterInsertsWaitForAQuorumAndReadsAreSequential(t *testing.T) { "AND current_database = currentDatabase() AND Settings['select_sequential_consistency'] = '1' AND query LIKE '%"+EventsTable+"%'", nil, &reads)) require.Positive(t, reads) } + +// A table that is already there is accepted for its columns and window, and on +// a cluster that is not enough: a MergeTree table lives on one node, so +// insert_quorum has nothing to wait for. A table found on a cluster is refused +// unless it is replicated, each table by itself, and one server keeps +// accepting a MergeTree table. +func TestServerClusterRefusesAnExistingTableThatIsNotReplicated(t *testing.T) { + conn, database, cluster := clusterDatabase(t) + ctx := context.Background() + single := serverStore(t, conn, database) + clustered := serverStore(t, conn, database, func(cfg *Config) { cfg.Cluster = cluster }) + + // Both tables exist as MergeTree, the way a one-server deployment made them. + require.NoError(t, single.Ensure(ctx)) + require.NoError(t, single.Ensure(ctx), "one server accepts the MergeTree tables it made") + err := clustered.Ensure(ctx) + require.ErrorContains(t, err, "table "+database+".audit_events: engine MergeTree is not replicated") + require.ErrorContains(t, err, `cluster "`+cluster+`"`) + + // Replace the events table with the replicated one the store writes on a + // cluster; the details table is still a MergeTree one. The MergeTree tables + // exist on the one node the connection reached, so a drop on the cluster + // finds them missing on the others. + drop := func(table string) { + require.NoError(t, conn.Exec(ctx, "DROP TABLE IF EXISTS "+table+" ON CLUSTER '"+cluster+"' SYNC")) + } + drop(EventsTable) + statements := clustered.CreateStatements() + require.NoError(t, conn.Exec(ctx, statements[0])) + err = clustered.Ensure(ctx) + require.ErrorContains(t, err, "table "+database+".audit_event_details: engine MergeTree is not replicated") + require.NotContains(t, err.Error(), EventsTable+":", "the replicated events table is not the one refused") + + // Replaced too, every table is replicated, and the cluster accepts them. + drop(DetailsTable) + require.NoError(t, conn.Exec(ctx, statements[1])) + require.NoError(t, clustered.Ensure(ctx), "a replicated table that already exists is accepted") + for _, table := range []string{EventsTable, DetailsTable} { + var engine string + require.NoError(t, clustered.queryRow(ctx, "SELECT engine FROM system.tables WHERE database = currentDatabase() AND name = '"+table+"'", nil, &engine)) + require.Equal(t, "ReplicatedMergeTree", engine, table) + } +} From 2cafb08b41585f4dc529002680734d94a9cd5b62 Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Tue, 6 Oct 2026 17:34:35 -0400 Subject: [PATCH 44/69] fix(audit): bound the state an aggregation keeps across read windows An aggregation reads a store's history window by window, and the window budget frees each window before the next. The aggregator's own state outlived them and was counted nowhere: a float per percentile input, a string per distinct value, a bucket per distinct key. A permitted aggregation with no time range could therefore exhaust the service's memory while returning a single bucket. The aggregator now counts that state against a total budget (auditeval.StateBudget): 8 bytes a sample, a distinct value's text plus its entry (a repeated value costs nothing), and a bucket's keys plus its metrics. The default is business.AuditAggregateMaxBytes, 32 MiB as the export bound is, and it is configurable like the window and export bounds (bigquerystore.ReadConfig.AggregateMaxBytes, clickhousestore.Config.AggregateMaxBytes). Past it the read stops with business.ErrAuditAggregateTooLarge, never a partial answer, and the service reports it as ResourceExhausted, the code an oversized export carries. It applies to every path that aggregates in the service: BigQuery, and ClickHouse's service-side fallback; ClickHouse's in-engine GROUP BY counts the buckets it hands the service against the same bound. Co-Authored-By: Claude Sonnet 5.5 --- module/FEATURES.md | 2 +- module/docs/adr/0009-audit-store-swap.md | 12 +- .../adapters/audit_aggregate_bound_test.go | 46 ++++++ .../pkg/auditstore/auditeval/aggregate.go | 97 +++++++++++- .../auditeval/aggregate_bound_test.go | 142 ++++++++++++++++++ .../pkg/auditstore/auditeval/eval_test.go | 2 +- .../pkg/auditstore/bigquerystore/reader.go | 54 ++++--- .../bigquerystore/reader_bound_test.go | 100 +++++++++++- .../clickhousestore/clickhousestore.go | 29 +++- .../pkg/auditstore/clickhousestore/reader.go | 22 ++- .../auditstore/clickhousestore/server_test.go | 62 +++++++- .../accounts/code/pkg/business/audit.go | 4 +- .../pkg/business/audit_aggregate_bound.go | 40 +++++ .../business/audit_aggregate_bound_test.go | 72 +++++++++ 14 files changed, 641 insertions(+), 43 deletions(-) create mode 100644 module/services/accounts/code/pkg/adapters/audit_aggregate_bound_test.go create mode 100644 module/services/accounts/code/pkg/auditstore/auditeval/aggregate_bound_test.go create mode 100644 module/services/accounts/code/pkg/business/audit_aggregate_bound.go create mode 100644 module/services/accounts/code/pkg/business/audit_aggregate_bound_test.go diff --git a/module/FEATURES.md b/module/FEATURES.md index 930b3f22f..154320cf5 100644 --- a/module/FEATURES.md +++ b/module/FEATURES.md @@ -398,7 +398,7 @@ see `JOBS.md` for the exact boundary and sequencing. | Export (JSON/CSV) | ✅ | `audit_export.go`; under a warehouse store of record one export holds at most 32 MiB of event text, past which it fails with `ErrAuditExportTooLarge` (`ResourceExhausted`): narrow it by actor or event type | | Impersonation tracking | ✅ | Records both real actor + viewed-as user | | Replay / event sourcing| ❌ | Audit log is read-only history; not used to reconstruct state | -| Warehouse store of record | 🟡 | `AUDIT_SINK=bigquery` or `AUDIT_SINK=clickhouse` (ADR 0009): each record commits to a Postgres queue in the mutation's transaction, then a relay delivers it to the warehouse and a locked GCS or S3 archive. Reads are scoped to the organization and deployment and deduplicated by event id; BigQuery uses the Storage Read API without query jobs, and ClickHouse uses SQL with service evaluation where needed. A BigQuery read holds a bounded amount in memory whatever the history: a window that needs more than its byte budget is read again as halves. A row the warehouse itself reports it refuses for good (never one that merely failed while the warehouse was flaky) is set aside, whole, in `audit_event_quarantine` instead of holding every organization's events behind it (`saas.audit_queue.quarantined`; replaying quarantined rows is not built yet). The shared parity suite compares both with Postgres. `audit-history-copy` verifies a switching deployment's copied history before a confirmed monthly-partition drop. No Azure Blob archive writer exists yet. | +| Warehouse store of record | 🟡 | `AUDIT_SINK=bigquery` or `AUDIT_SINK=clickhouse` (ADR 0009): each record commits to a Postgres queue in the mutation's transaction, then a relay delivers it to the warehouse and a locked GCS or S3 archive. Reads are scoped to the organization and deployment and deduplicated by event id; BigQuery uses the Storage Read API without query jobs, and ClickHouse uses SQL with service evaluation where needed. A BigQuery read holds a bounded amount in memory whatever the history: a window that needs more than its byte budget is read again as halves. What an aggregation keeps across the windows (its buckets, percentile samples and distinct values) is counted too, and one that keeps more than 32 MiB fails with `ErrAuditAggregateTooLarge` (`ResourceExhausted`), never with a partial answer: narrow it by time range, actor or event type, or group by fewer dimensions. A row the warehouse itself reports it refuses for good (never one that merely failed while the warehouse was flaky) is set aside, whole, in `audit_event_quarantine` instead of holding every organization's events behind it (`saas.audit_queue.quarantined`; replaying quarantined rows is not built yet). The shared parity suite compares both with Postgres. `audit-history-copy` verifies a switching deployment's copied history before a confirmed monthly-partition drop. No Azure Blob archive writer exists yet. | ### Admin panel diff --git a/module/docs/adr/0009-audit-store-swap.md b/module/docs/adr/0009-audit-store-swap.md index a69fa2cd6..0bb2b006c 100644 --- a/module/docs/adr/0009-audit-store-swap.md +++ b/module/docs/adr/0009-audit-store-swap.md @@ -207,7 +207,7 @@ Postgres transaction. them — against a byte budget, `ReadConfig.WindowBytes` (64 MiB by default; the service exposes no setting for it). A window that passes the budget is read again as two halves. What outlives a window is the answer: a page, an - aggregation's buckets, an export's events. Three limits remain: + aggregation's buckets, an export's events. Four limits remain: - A read whose events at a single instant alone pass the budget cannot be split further and fails (`ErrReadTooDense`). - An export is the one answer as large as the history it matches. One export @@ -215,6 +215,16 @@ Postgres transaction. (`AuditExportMaxBytes`) and is refused with `ResourceExhausted` past it. The export request carries no time range, so an actor or an event type is all that narrows it. + - An aggregation keeps more than its buckets while it reads: a float per + percentile input and a string per distinct value, over every window, so a + read with no time range can outgrow a pod while it returns one bucket. The + aggregator counts that state — 8 bytes a sample, a value's text and entry, + a bucket's keys and metrics — against a bound of 32 MiB + (`AuditAggregateMaxBytes`, `ReadConfig.AggregateMaxBytes` and + `clickhousestore.Config.AggregateMaxBytes`), and the read is refused with + `ResourceExhausted` (`ErrAuditAggregateTooLarge`) as soon as it passes it, + never answered in part. A time range, an actor or an event type narrows it, + as does grouping by fewer dimensions. - The history copy (item 7) reads and holds a whole window of the store at a time outside this budget. diff --git a/module/services/accounts/code/pkg/adapters/audit_aggregate_bound_test.go b/module/services/accounts/code/pkg/adapters/audit_aggregate_bound_test.go new file mode 100644 index 000000000..b032b1b94 --- /dev/null +++ b/module/services/accounts/code/pkg/adapters/audit_aggregate_bound_test.go @@ -0,0 +1,46 @@ +package adapters + +import ( + "context" + "fmt" + "testing" + + "accounts/pkg/auth" + "accounts/pkg/business" + gen "accounts/pkg/gen/saas/accounts/v1" + + "github.com/stretchr/testify/require" + "google.golang.org/grpc/codes" + "google.golang.org/grpc/status" +) + +// tooLargeAuditStore is a store of record whose aggregation is refused for its +// size, as a warehouse store's is when what it keeps passes its bound. +type tooLargeAuditStore struct{ business.AuditStore } + +func (tooLargeAuditStore) AggregateAuditEvents(context.Context, business.AuditRead, business.AuditAggregationSpec) ([]business.AuditAggregateBucket, error) { + return []business.AuditAggregateBucket{{Key: "partial", Count: 1}}, fmt.Errorf("read: %w", business.ErrAuditAggregateTooLarge) +} + +// The refusal reaches the RPC's caller as ResourceExhausted, with no bucket, on +// both paths an aggregation takes: the organization-wide read and the one gated +// on a collection grant. +func TestAggregateAuditLogRefusedForSizeIsResourceExhausted(t *testing.T) { + store := &auditContractStore{layeredAuthzStore: layeredAuthzStore{role: gen.OrgRole_ORG_ROLE_ADMIN}, allowed: true} + installLayeredAuthzService(t, store) + service.SetAuditStore(tooLargeAuditStore{}) + previousCache := orgMembershipCache + orgMembershipCache = nil + t.Cleanup(func() { orgMembershipCache = previousCache }) + ctx := stampVerifiedIdentity(context.Background(), layeredActorID, layeredOrgID, auth.Assurance{}) + + for name, request := range map[string]*gen.AggregateAuditLogRequest{ + "organization-wide": {OrgId: layeredOrgID}, + "a collection's read": {OrgId: layeredOrgID, CollectionId: "collection-a", EventType: "saas.document.read"}, + } { + response, err := (&AuditServer{}).AggregateAuditLog(ctx, request) + require.Equal(t, codes.ResourceExhausted, status.Code(err), "%s: %v", name, err) + require.ErrorContains(t, err, "narrow it by time range, actor or event type", name) + require.Nil(t, response, name) + } +} diff --git a/module/services/accounts/code/pkg/auditstore/auditeval/aggregate.go b/module/services/accounts/code/pkg/auditstore/auditeval/aggregate.go index e651a6248..9fc47606f 100644 --- a/module/services/accounts/code/pkg/auditstore/auditeval/aggregate.go +++ b/module/services/accounts/code/pkg/auditstore/auditeval/aggregate.go @@ -30,6 +30,13 @@ import ( // // An event without details (a content-class event past the content window) // has no payload: its payload dimensions are "" and its payload metrics NULL. +// +// What an Aggregator keeps grows with the events it is offered — a bucket per +// distinct key, a float per percentile input, a string per distinct value — +// and a store offers it every event of a history, window by window, so it +// counts that state against a budget (StateBudget) and gives up with +// business.ErrAuditAggregateTooLarge when it passes. An Aggregator that has +// returned an error from Add has an incomplete state and must be discarded. type Aggregator struct { spec business.AuditAggregationSpec dims []string @@ -37,8 +44,76 @@ type Aggregator struct { aliases []string groups map[string]*group order []*group + state *StateBudget +} + +// What the state of an aggregation costs, counted rather than measured so the +// bound is one number the deployment chose. The figures err high: a bucket's +// cost includes the bucket it is answered as, and a sample's the 8 bytes of its +// float, not the capacity a slice that grows by doubling may hold past it. +const ( + // bucketBytes is a bucket's own: the group, its entries in the map and the + // ordered list, and the answer built from it. + bucketBytes = 256 + // keyBytes is what each of a bucket's dimension values costs beside its + // text, which is held twice (as the key and in the bucket's identity). + keyBytes = 32 + // metricBytes is a bucket's accumulator for one metric and its entries in + // the answer's Samples and Metrics maps. + metricBytes = 160 + // sampleBytes is a percentile input: one float64. + sampleBytes = 8 + // distinctEntryBytes is what a value a distinct count remembers costs beside + // its text: its string header and its entry in the set. + distinctEntryBytes = 32 +) + +// BucketBytes is the state of one bucket of an aggregation with metrics +// metrics, whose dimension values are keys. +func BucketBytes(keys []string, metrics int) int64 { + n := int64(bucketBytes + len(keys)*keyBytes + metrics*metricBytes) + for _, key := range keys { + n += 2 * int64(len(key)) + } + return n +} + +// SampleBytes is the state of n percentile inputs. +func SampleBytes(n int) int64 { return int64(n) * sampleBytes } + +// StateBudget counts the bytes an aggregation retains across every window it +// reads. The window budget of a store bounds what one window holds; this is +// the bound on what outlives a window. +type StateBudget struct { + limit, held int64 +} + +// NewStateBudget is a budget of limit bytes; zero is +// business.AuditAggregateMaxBytes. +func NewStateBudget(limit int64) (*StateBudget, error) { + switch { + case limit < 0: + return nil, errors.New("audit: the aggregation state budget cannot be negative") + case limit == 0: + limit = business.AuditAggregateMaxBytes + } + return &StateBudget{limit: limit}, nil +} + +// Take counts n more bytes as retained, and reports +// business.ErrAuditAggregateTooLarge once they pass the limit. The bytes are +// counted either way: a caller that gets the error stops. +func (b *StateBudget) Take(n int64) error { + b.held += n + if b.held > b.limit { + return fmt.Errorf("%w (the bound is %d bytes)", business.ErrAuditAggregateTooLarge, b.limit) + } + return nil } +// Held is the bytes counted so far. +func (b *StateBudget) Held() int64 { return b.held } + type group struct { keys []string count int64 @@ -56,7 +131,13 @@ type accumulator struct { // NewAggregator prepares spec, which the caller has validated. It refuses a // spec that groups or counts by category without the event-type index, and // any dimension or metric the Postgres aggregation would not run either. -func NewAggregator(spec business.AuditAggregationSpec, types business.AuditEventTypeIndex) (*Aggregator, error) { +// maxStateBytes is the bound on the state it keeps (StateBudget); zero is +// business.AuditAggregateMaxBytes. +func NewAggregator(spec business.AuditAggregationSpec, types business.AuditEventTypeIndex, maxStateBytes int64) (*Aggregator, error) { + state, err := NewStateBudget(maxStateBytes) + if err != nil { + return nil, err + } dims := spec.GroupBy if len(dims) == 0 { dims = []string{"event_type"} @@ -103,6 +184,7 @@ func NewAggregator(spec business.AuditAggregationSpec, types business.AuditEvent types: types, aliases: aliases, groups: map[string]*group{}, + state: state, }, nil } @@ -124,6 +206,9 @@ func (a *Aggregator) Add(event *Event) error { id := groupID(keys) g, ok := a.groups[id] if !ok { + if err := a.state.Take(BucketBytes(keys, len(a.spec.Metrics))); err != nil { + return err + } g = &group{keys: keys, metrics: make([]*accumulator, len(a.spec.Metrics))} for i := range g.metrics { g.metrics[i] = &accumulator{min: math.Inf(1), max: math.Inf(-1)} @@ -146,7 +231,12 @@ func (a *Aggregator) Add(event *Event) error { if acc.distinct == nil { acc.distinct = map[string]struct{}{} } - acc.distinct[value] = struct{}{} + if _, seen := acc.distinct[value]; !seen { + if err := a.state.Take(distinctEntryBytes + int64(len(value))); err != nil { + return err + } + acc.distinct[value] = struct{}{} + } default: key, _ := payloadKey(metric.Field) value, ok := jsonbNumeric(payload, key) @@ -158,6 +248,9 @@ func (a *Aggregator) Add(event *Event) error { acc.min = math.Min(acc.min, value) acc.max = math.Max(acc.max, value) if metric.Op == "percentile" { + if err := a.state.Take(sampleBytes); err != nil { + return err + } acc.values = append(acc.values, value) } } diff --git a/module/services/accounts/code/pkg/auditstore/auditeval/aggregate_bound_test.go b/module/services/accounts/code/pkg/auditstore/auditeval/aggregate_bound_test.go new file mode 100644 index 000000000..08d6af74a --- /dev/null +++ b/module/services/accounts/code/pkg/auditstore/auditeval/aggregate_bound_test.go @@ -0,0 +1,142 @@ +package auditeval + +import ( + "fmt" + "math" + "testing" + "time" + + "accounts/pkg/business" + + "github.com/stretchr/testify/require" +) + +// An aggregation keeps state across every window a store reads: a bucket per +// distinct key, a float per percentile input, a string per distinct value. The +// window budget of a store does not count it, so the aggregator does, and gives +// up rather than return an answer it could not finish. + +// offerWindow offers the aggregator one window of a store's events: n events +// numbered from first, all of one type (so one bucket unless a spec groups by +// a payload key), each with its own number and its own text. +func offerWindow(aggregator *Aggregator, first, n int) error { + at := time.Date(2026, 10, 3, 0, 0, 0, 0, time.UTC) + for i := first; i < first+n; i++ { + event := &Event{ + Entry: business.AuditEntry{ID: fmt.Sprintf("00000000-0000-4000-8000-%012d", i), EventType: business.EventDocumentRead, CreatedAt: at.Add(time.Duration(i) * time.Second)}, + Details: fmt.Sprintf(`{"n":%d,"uniq":"value-%06d","group":"g-%06d"}`, i, i, i), + HasDetails: true, + } + if err := aggregator.Add(event); err != nil { + return err + } + } + return nil +} + +// walk offers windows of 40 events until the aggregator gives up or it has +// offered windows of them, and reports how many it completed. +func walk(aggregator *Aggregator, windows int) (completed int, err error) { + for ; completed < windows; completed++ { + if err := offerWindow(aggregator, completed*40, 40); err != nil { + return completed, err + } + } + return completed, nil +} + +func TestAnAggregationOverManyWindowsGivesUpPastItsStateBound(t *testing.T) { + const budget = 16 << 10 + for name, spec := range map[string]business.AuditAggregationSpec{ + "a percentile": {Metrics: []business.AuditMetric{{Op: "percentile", Field: "payload:n", Percentile: 0.5}}}, + "a distinct count": {Metrics: []business.AuditMetric{{Op: "count_distinct", Field: "payload:uniq"}}}, + "both, in one bucket": {Metrics: []business.AuditMetric{ + {Op: "percentile", Field: "payload:n", Percentile: 0.9}, + {Op: "count_distinct", Field: "payload:uniq"}, + }}, + "a bucket per key": {GroupBy: []string{"payload:group"}}, + } { + aggregator, err := NewAggregator(spec, nil, budget) + require.NoError(t, err, name) + completed, err := walk(aggregator, 500) + require.ErrorIs(t, err, business.ErrAuditAggregateTooLarge, "%s: 20000 events cannot fit %d bytes", name, budget) + require.Positive(t, completed, "%s: the first window fits; it is the windows together that do not", name) + require.Less(t, completed, 500, "%s: it gave up as soon as the state passed the bound", name) + require.LessOrEqual(t, aggregator.state.Held(), int64(budget)+BucketBytes([]string{"g-000000"}, 2)+SampleBytes(1)+distinctEntryBytes+16, + "%s: and kept no more than the one entry that passed it", name) + } +} + +func TestTheStateBoundCountsWhatIsKeptNotWhatIsOffered(t *testing.T) { + // One bucket and one remembered value, whatever the windows: the same 20000 + // events keep a counter. + spec := business.AuditAggregationSpec{Metrics: []business.AuditMetric{{Op: "count"}, {Op: "count_distinct", Field: "event_type", Alias: "types"}}} + aggregator, err := NewAggregator(spec, nil, 4<<10) + require.NoError(t, err) + completed, err := walk(aggregator, 500) + require.NoError(t, err) + require.Equal(t, 500, completed) + held := aggregator.state.Held() + require.Less(t, held, int64(1<<10), "one bucket and one remembered value") + buckets := aggregator.Buckets() + require.Len(t, buckets, 1) + require.Equal(t, int64(20000), buckets[0].Count) + require.Equal(t, 1.0, buckets[0].Metrics["types"]) + + // A value seen again costs nothing. + require.NoError(t, offerWindow(aggregator, 0, 40)) + require.Equal(t, held, aggregator.state.Held(), "the same events again") +} + +func TestAnAggregationUnderTheDefaultBoundMatchesOneWithoutAnyBound(t *testing.T) { + spec := business.AuditAggregationSpec{ + GroupBy: []string{"event_type"}, + Metrics: []business.AuditMetric{ + {Op: "percentile", Field: "payload:n", Percentile: 0.5, Alias: "p50"}, + {Op: "percentile", Field: "payload:n", Percentile: 0.99, Alias: "p99"}, + {Op: "count_distinct", Field: "payload:uniq", Alias: "distinct"}, + {Op: "sum", Field: "payload:n", Alias: "total"}, + }, + } + bounded, err := NewAggregator(spec, nil, 0) + require.NoError(t, err) + unbounded, err := NewAggregator(spec, nil, math.MaxInt64) + require.NoError(t, err) + for _, aggregator := range []*Aggregator{bounded, unbounded} { + completed, err := walk(aggregator, 250) + require.NoError(t, err) + require.Equal(t, 250, completed) + } + got, want := bounded.Buckets(), unbounded.Buckets() + require.Equal(t, want, got, "10000 events are well inside the default bound") + require.Len(t, got, 1) + require.Equal(t, 4999.5, got[0].Metrics["p50"]) + require.Equal(t, 9999*0.99, got[0].Metrics["p99"]) + require.Equal(t, 10000.0, got[0].Metrics["distinct"]) + require.Equal(t, 9999.0*10000/2, got[0].Metrics["total"]) +} + +func TestTheStateBoundDefaultsAndRefusesANegativeOne(t *testing.T) { + _, err := NewAggregator(business.AuditAggregationSpec{}, nil, -1) + require.ErrorContains(t, err, "cannot be negative") + + budget, err := NewStateBudget(0) + require.NoError(t, err) + require.NoError(t, budget.Take(business.AuditAggregateMaxBytes)) + require.ErrorIs(t, budget.Take(1), business.ErrAuditAggregateTooLarge, "zero is business.AuditAggregateMaxBytes") + + explicit, err := NewStateBudget(100) + require.NoError(t, err) + require.NoError(t, explicit.Take(100)) + err = explicit.Take(1) + require.ErrorIs(t, err, business.ErrAuditAggregateTooLarge) + require.ErrorContains(t, err, "narrow it", "the refusal says how to get past it") +} + +func TestABucketsStateCountsItsKeysAndMetrics(t *testing.T) { + require.Greater(t, BucketBytes([]string{"a", "b"}, 3), BucketBytes([]string{"a"}, 3)) + require.Greater(t, BucketBytes([]string{"a"}, 3), BucketBytes([]string{"a"}, 1)) + require.Equal(t, BucketBytes([]string{"x"}, 0)+2*100, BucketBytes([]string{string(make([]byte, 101))}, 0), + "a key's text is held twice, in the bucket and in its identity") + require.Equal(t, int64(8000), SampleBytes(1000)) +} diff --git a/module/services/accounts/code/pkg/auditstore/auditeval/eval_test.go b/module/services/accounts/code/pkg/auditstore/auditeval/eval_test.go index 54c063a6d..6d624586f 100644 --- a/module/services/accounts/code/pkg/auditstore/auditeval/eval_test.go +++ b/module/services/accounts/code/pkg/auditstore/auditeval/eval_test.go @@ -165,7 +165,7 @@ func TestMatcherRefusals(t *testing.T) { require.ErrorContains(t, err, "event-type index") _, err = NewMatcher(business.AuditRead{Scope: business.OrganizationAuditScope(org), Query: business.AuditQuery{OrgID: org, CollectionID: "c"}}) require.ErrorContains(t, err, "uncompiled collection filter") - _, err = NewAggregator(business.AuditAggregationSpec{GroupBy: []string{"category"}}, nil) + _, err = NewAggregator(business.AuditAggregationSpec{GroupBy: []string{"category"}}, nil, 0) require.ErrorContains(t, err, "event-type index") m, err := NewMatcher(business.AuditRead{Scope: business.OrganizationAuditScope(org), Query: business.AuditQuery{ diff --git a/module/services/accounts/code/pkg/auditstore/bigquerystore/reader.go b/module/services/accounts/code/pkg/auditstore/bigquerystore/reader.go index 2ab85c1f6..9ecf0cf9c 100644 --- a/module/services/accounts/code/pkg/auditstore/bigquerystore/reader.go +++ b/module/services/accounts/code/pkg/auditstore/bigquerystore/reader.go @@ -73,6 +73,10 @@ type ReadConfig struct { // ExportMaxBytes bounds the events one export gathers, counted the same // way. Zero is business.AuditExportMaxBytes. ExportMaxBytes int64 + // AggregateMaxBytes bounds the state one aggregation keeps across all the + // windows it reads — its buckets, percentile samples and distinct values, + // counted by auditeval.StateBudget. Zero is business.AuditAggregateMaxBytes. + AggregateMaxBytes int64 // Now is a seam for tests; nil is the clock. Now func() time.Time } @@ -113,16 +117,18 @@ const ( // that would hold more is abandoned and read again as two halves of its span, // newer first, so a day of a million events is read as the several pieces that // fit. What outlives a window is the answer alone: a page and the proof of the -// next, an aggregation's buckets, an export's events (up to ExportMaxBytes). +// next, an aggregation's state (up to AggregateMaxBytes), an export's events +// (up to ExportMaxBytes). type Reader struct { - client ReadSessionClient - project string - dataset string - deploymentID string - maxStreams int - windowBytes int64 - exportMaxBytes int64 - now func() time.Time + client ReadSessionClient + project string + dataset string + deploymentID string + maxStreams int + windowBytes int64 + exportMaxBytes int64 + aggregateMaxBytes int64 + now func() time.Time // windowPeak is the most bytes any one attempt at a window has counted, the // attempt that overflowed included, since the reader was built. @@ -139,18 +145,19 @@ func NewReader(cfg ReadConfig) (*Reader, error) { return nil, fmt.Errorf("bigquery audit store: %s is required to read", name) } } - if cfg.WindowBytes < 0 || cfg.ExportMaxBytes < 0 { + if cfg.WindowBytes < 0 || cfg.ExportMaxBytes < 0 || cfg.AggregateMaxBytes < 0 { return nil, errors.New("bigquery audit store: read budgets cannot be negative") } reader := &Reader{ - client: cfg.Client, - project: cfg.Project, - dataset: cfg.Dataset, - deploymentID: cfg.DeploymentID, - maxStreams: cfg.MaxStreams, - windowBytes: cfg.WindowBytes, - exportMaxBytes: cfg.ExportMaxBytes, - now: cfg.Now, + client: cfg.Client, + project: cfg.Project, + dataset: cfg.Dataset, + deploymentID: cfg.DeploymentID, + maxStreams: cfg.MaxStreams, + windowBytes: cfg.WindowBytes, + exportMaxBytes: cfg.ExportMaxBytes, + aggregateMaxBytes: cfg.AggregateMaxBytes, + now: cfg.Now, } if reader.maxStreams <= 0 { reader.maxStreams = DefaultReadStreams @@ -161,6 +168,9 @@ func NewReader(cfg ReadConfig) (*Reader, error) { if reader.exportMaxBytes == 0 { reader.exportMaxBytes = business.AuditExportMaxBytes } + if reader.aggregateMaxBytes == 0 { + reader.aggregateMaxBytes = business.AuditAggregateMaxBytes + } if reader.now == nil { reader.now = time.Now } @@ -633,13 +643,17 @@ func needsPayload(m *auditeval.Matcher, spec business.AuditAggregationSpec) bool // AggregateAuditEvents implements business.AuditReader: the query's span read // window by window, each window deduplicated and aggregated whole before the // next is read. Copies of an event share its occurrence, so a window holds all -// of them and the deduplication needs no memory of the windows before it. +// of them and the deduplication needs no memory of the windows before it. What +// the aggregation keeps from one window to the next (its buckets, percentile +// samples and distinct values) is counted against the reader's aggregation +// bound, and the read gives up with business.ErrAuditAggregateTooLarge, never +// a partial answer, as soon as it passes it. func (r *Reader) AggregateAuditEvents(ctx context.Context, read business.AuditRead, spec business.AuditAggregationSpec) ([]business.AuditAggregateBucket, error) { m, err := auditeval.NewMatcher(read) if err != nil { return nil, err } - aggregator, err := auditeval.NewAggregator(spec, read.Types) + aggregator, err := auditeval.NewAggregator(spec, read.Types, r.aggregateMaxBytes) if err != nil { return nil, err } diff --git a/module/services/accounts/code/pkg/auditstore/bigquerystore/reader_bound_test.go b/module/services/accounts/code/pkg/auditstore/bigquerystore/reader_bound_test.go index 734433806..a277935fc 100644 --- a/module/services/accounts/code/pkg/auditstore/bigquerystore/reader_bound_test.go +++ b/module/services/accounts/code/pkg/auditstore/bigquerystore/reader_bound_test.go @@ -45,10 +45,17 @@ func newBoundWarehouse(t *testing.T) *boundWarehouse { func (w *boundWarehouse) reader(t *testing.T, windowBytes, exportMaxBytes int64) *Reader { t.Helper() - reader, err := NewReader(ReadConfig{ + return w.readerWith(t, func(cfg *ReadConfig) { cfg.WindowBytes, cfg.ExportMaxBytes = windowBytes, exportMaxBytes }) +} + +func (w *boundWarehouse) readerWith(t *testing.T, mutate func(*ReadConfig)) *Reader { + t.Helper() + cfg := ReadConfig{ Client: w.fake, Project: boundProject, Dataset: boundDataset, DeploymentID: boundDeployment, - WindowBytes: windowBytes, ExportMaxBytes: exportMaxBytes, Now: func() time.Time { return boundNow }, - }) + Now: func() time.Time { return boundNow }, + } + mutate(&cfg) + reader, err := NewReader(cfg) require.NoError(t, err) return reader } @@ -185,6 +192,93 @@ func TestAPayloadAggregationHoldsOneBoundedWindowAndCountsEachEventOnce(t *testi require.Greater(t, large.windowPeak.Load(), int64(4*budget)) } +// distinctMonth fills the warehouse with 600 events over fifteen days, each with +// a number and a text of its own: an aggregation over them keeps a float and a +// string per event, and a bucket per event when it groups by the text. +func (w *boundWarehouse) distinctMonth(t *testing.T) { + t.Helper() + var records []business.AuditRecord + for i := 0; i < 600; i++ { + at := boundNow.Add(-time.Hour - time.Duration(i)*36*time.Minute) + records = append(records, boundEvent(t, i+1, at, map[string]any{"n": i, "uniq": fmt.Sprintf("value-%06d", i), "pad": strings.Repeat("p", 300)})) + } + w.append(t, records...) + w.append(t, records...) +} + +// What an aggregation keeps from window to window is a number the deployment +// chose too: the window budget frees each window before the next, so it does +// not count the percentile samples, distinct values and buckets that outlive +// one. A read of a long history that needs more than the aggregation bound is +// refused, as soon as it passes it and never with a partial answer. +func TestAnAggregationOverManyWindowsGivesUpPastItsStateBound(t *testing.T) { + w := newBoundWarehouse(t) + w.distinctMonth(t) + read := orgScoped(business.AuditQuery{}) + const windowBytes = 64 << 10 + // The newest day is forty events: it fits each bound below; the fifteen days + // do not. + newestDay := boundNow.Add(-24 * time.Hour) + for name, tc := range map[string]struct { + spec business.AuditAggregationSpec + budget int64 + }{ + "a percentile": {business.AuditAggregationSpec{Metrics: []business.AuditMetric{{Op: "percentile", Field: "payload:n", Percentile: 0.5, Alias: "p50"}}}, 4 << 10}, + "a distinct count": {business.AuditAggregationSpec{Metrics: []business.AuditMetric{{Op: "count_distinct", Field: "payload:uniq", Alias: "distinct"}}}, 4 << 10}, + "a bucket per key": {business.AuditAggregationSpec{GroupBy: []string{"payload:uniq"}}, 20 << 10}, + } { + bounded := w.readerWith(t, func(cfg *ReadConfig) { cfg.WindowBytes, cfg.AggregateMaxBytes = windowBytes, tc.budget }) + before := len(w.fake.Sessions()) + _, err := bounded.AggregateAuditEvents(context.Background(), read, tc.spec) + require.ErrorIs(t, err, business.ErrAuditAggregateTooLarge, name) + stopped := len(w.fake.Sessions()) - before + + recent := orgScoped(business.AuditQuery{From: &newestDay}) + got, err := bounded.AggregateAuditEvents(context.Background(), recent, tc.spec) + require.NoError(t, err, "%s: the newest day alone fits the same bound; it is the windows together that do not", name) + require.NotEmpty(t, got, name) + + before = len(w.fake.Sessions()) + got, err = w.readerWith(t, func(cfg *ReadConfig) { cfg.WindowBytes = windowBytes }).AggregateAuditEvents(context.Background(), read, tc.spec) + require.NoError(t, err, "%s: under the default bound the same read is answered", name) + require.NotEmpty(t, got, name) + require.Less(t, stopped, len(w.fake.Sessions())-before, "%s: and the bounded read stopped as soon as the bound was passed", name) + } +} + +// A bound that is not passed changes nothing: the answer is the one a reader +// with no practical bound gives, across the many windows it is read in. +func TestAnAggregationUnderTheDefaultStateBoundMatchesOneWithoutAny(t *testing.T) { + w := newBoundWarehouse(t) + w.distinctMonth(t) + spec := business.AuditAggregationSpec{Metrics: []business.AuditMetric{ + {Op: "percentile", Field: "payload:n", Percentile: 0.5, Alias: "p50"}, + {Op: "count_distinct", Field: "payload:uniq", Alias: "distinct"}, + {Op: "sum", Field: "payload:n", Alias: "total"}, + }} + read := orgScoped(business.AuditQuery{}) + + before := len(w.fake.Sessions()) + got, err := w.readerWith(t, func(cfg *ReadConfig) { cfg.WindowBytes = 64 << 10 }).AggregateAuditEvents(context.Background(), read, spec) + require.NoError(t, err) + require.Greater(t, len(w.fake.Sessions())-before, 4, "read in many windows") + reference, err := w.readerWith(t, func(cfg *ReadConfig) { cfg.WindowBytes, cfg.AggregateMaxBytes = 1<<40, 1<<40 }).AggregateAuditEvents(context.Background(), read, spec) + require.NoError(t, err) + require.Equal(t, reference, got) + require.Len(t, got, 1) + require.Equal(t, 299.5, got[0].Metrics["p50"], "600 events of 0 to 599, each delivered twice and counted once") + require.Equal(t, 600.0, got[0].Metrics["distinct"]) + require.Equal(t, 599.0*600/2, got[0].Metrics["total"]) +} + +func TestAReadersAggregationBoundCannotBeNegative(t *testing.T) { + w := newBoundWarehouse(t) + _, err := NewReader(ReadConfig{ + Client: w.fake, Project: boundProject, Dataset: boundDataset, DeploymentID: boundDeployment, AggregateMaxBytes: -1, + }) + require.ErrorContains(t, err, "read budgets cannot be negative") +} + func TestAnExportHoldsOneBoundedWindowAndTheEventsItReturns(t *testing.T) { w := newBoundWarehouse(t) records := w.month(t) diff --git a/module/services/accounts/code/pkg/auditstore/clickhousestore/clickhousestore.go b/module/services/accounts/code/pkg/auditstore/clickhousestore/clickhousestore.go index cbc5e931e..193b9a183 100644 --- a/module/services/accounts/code/pkg/auditstore/clickhousestore/clickhousestore.go +++ b/module/services/accounts/code/pkg/auditstore/clickhousestore/clickhousestore.go @@ -85,6 +85,11 @@ type Config struct { // ExportMaxBytes bounds the events one export gathers, counted by their // text (auditeval.Event.Size). Zero is business.AuditExportMaxBytes. ExportMaxBytes int64 + // AggregateMaxBytes bounds the state one aggregation keeps in the service — + // its buckets, and the samples and distinct values of the aggregation the + // service evaluates itself — counted by auditeval.StateBudget. Zero is + // business.AuditAggregateMaxBytes. + AggregateMaxBytes int64 } // clusterPattern is a cluster name as remote_servers declares one, or a macro @@ -106,6 +111,8 @@ type Store struct { readSettings clickhouse.Settings // exportMaxBytes is the most an export gathers before it gives up. exportMaxBytes int64 + // aggregateMaxBytes is the most state an aggregation keeps before it gives up. + aggregateMaxBytes int64 // writeSettings are sent with every insert: on a cluster, the quorum that // makes an acknowledgement durable. writeSettings clickhouse.Settings @@ -143,14 +150,22 @@ func New(conn Conn, cfg Config) (*Store, error) { if exportMaxBytes == 0 { exportMaxBytes = business.AuditExportMaxBytes } + if cfg.AggregateMaxBytes < 0 { + return nil, errors.New("clickhouse audit store: aggregation bound cannot be negative") + } + aggregateMaxBytes := cfg.AggregateMaxBytes + if aggregateMaxBytes == 0 { + aggregateMaxBytes = business.AuditAggregateMaxBytes + } store := &Store{ - conn: conn, - database: cfg.Database, - cluster: cfg.Cluster, - deploymentID: cfg.DeploymentID, - eventsDays: eventsDays, - detailDays: detailDays, - exportMaxBytes: exportMaxBytes, + conn: conn, + database: cfg.Database, + cluster: cfg.Cluster, + deploymentID: cfg.DeploymentID, + eventsDays: eventsDays, + detailDays: detailDays, + exportMaxBytes: exportMaxBytes, + aggregateMaxBytes: aggregateMaxBytes, // Pinned per read, whatever the user's profile says, because each one // changes an answer. readSettings: clickhouse.Settings{ diff --git a/module/services/accounts/code/pkg/auditstore/clickhousestore/reader.go b/module/services/accounts/code/pkg/auditstore/clickhousestore/reader.go index 976595506..70c403bd3 100644 --- a/module/services/accounts/code/pkg/auditstore/clickhousestore/reader.go +++ b/module/services/accounts/code/pkg/auditstore/clickhousestore/reader.go @@ -321,7 +321,12 @@ func (s *Store) ExportAuditEvents(ctx context.Context, read business.AuditRead) // AggregateAuditEvents implements business.AuditReader: one GROUP BY in // ClickHouse when it is exact for this read (aggregate.go), and otherwise the -// deduplicated, matched events aggregated in the service. +// deduplicated, matched events aggregated in the service. Either way the state +// the service keeps for the answer (the buckets, and in the second the +// percentile samples and distinct values) is counted against the store's +// aggregation bound, and the read gives up with +// business.ErrAuditAggregateTooLarge, never a partial answer, as soon as it +// passes it. func (s *Store) AggregateAuditEvents(ctx context.Context, read business.AuditRead, spec business.AuditAggregationSpec) ([]business.AuditAggregateBucket, error) { m, err := auditeval.NewMatcher(read) if err != nil { @@ -329,7 +334,7 @@ func (s *Store) AggregateAuditEvents(ctx context.Context, read business.AuditRea } // The service's aggregator is built either way: it refuses what Postgres // refuses, and it is the fallback. - aggregator, err := auditeval.NewAggregator(spec, read.Types) + aggregator, err := auditeval.NewAggregator(spec, read.Types, s.aggregateMaxBytes) if err != nil { return nil, err } @@ -338,7 +343,11 @@ func (s *Store) AggregateAuditEvents(ctx context.Context, read business.AuditRea return nil, nil } if !m.NeedsPayload() || pl.payload != nil { - buckets, exact, err := s.aggregateInClickHouse(ctx, pl, spec, read.Types) + state, err := auditeval.NewStateBudget(s.aggregateMaxBytes) + if err != nil { + return nil, err + } + buckets, exact, err := s.aggregateInClickHouse(ctx, pl, spec, read.Types, state) if err != nil || exact { return buckets, err } @@ -355,8 +364,8 @@ func (s *Store) AggregateAuditEvents(ctx context.Context, read business.AuditRea // aggregateInClickHouse runs the aggregation statement. exact is false when a // row it grouped is one ClickHouse cannot read as Postgres does, and the -// buckets are then discarded. -func (s *Store) aggregateInClickHouse(ctx context.Context, pl plan, spec business.AuditAggregationSpec, types business.AuditEventTypeIndex) ([]business.AuditAggregateBucket, bool, error) { +// buckets are then discarded. Each bucket it keeps is counted against state. +func (s *Store) aggregateInClickHouse(ctx context.Context, pl plan, spec business.AuditAggregationSpec, types business.AuditEventTypeIndex, state *auditeval.StateBudget) ([]business.AuditAggregateBucket, bool, error) { agg, ok, err := s.buildAggregation(pl, spec, types) if err != nil || !ok { return nil, err == nil, err @@ -392,6 +401,9 @@ func (s *Store) aggregateInClickHouse(ctx context.Context, pl plan, spec busines if inexact > 0 { return false, nil } + if err := state.Take(auditeval.BucketBytes(keys, len(agg.metrics))); err != nil { + return false, err + } bucket := business.AuditAggregateBucket{ Keys: keys, Key: keys[0], diff --git a/module/services/accounts/code/pkg/auditstore/clickhousestore/server_test.go b/module/services/accounts/code/pkg/auditstore/clickhousestore/server_test.go index 78f03a684..788bcb866 100644 --- a/module/services/accounts/code/pkg/auditstore/clickhousestore/server_test.go +++ b/module/services/accounts/code/pkg/auditstore/clickhousestore/server_test.go @@ -167,7 +167,7 @@ func (f *fixture) referenceAggregate(t *testing.T, read business.AuditRead, spec t.Helper() m, err := auditeval.NewMatcher(read) require.NoError(t, err) - aggregator, err := auditeval.NewAggregator(spec, read.Types) + aggregator, err := auditeval.NewAggregator(spec, read.Types, 0) require.NoError(t, err) for _, event := range f.events() { ok, err := m.Match(event) @@ -569,3 +569,63 @@ func TestServerNumericReadingIsExact(t *testing.T) { } require.Zero(t, inService, "every one ran in ClickHouse") } + +// An aggregation keeps state in the service whichever way ClickHouse reads it: +// the buckets of its answer, and, when the service evaluates it itself, a float +// per percentile input and a string per distinct value. That state grows with +// the history, so past its bound the aggregation gives up instead. +func TestServerAggregationGivesUpPastItsStateBound(t *testing.T) { + conn, database := serverDatabase(t) + ctx := context.Background() + org, actor := uuid.NewString(), uuid.NewString() + base := time.Now().UTC().Truncate(time.Microsecond).Add(-time.Minute) + f := &fixture{org: org} + for i := range 400 { + f.records = append(f.records, newRecord(t, org, actor, business.EventDocumentRead, "doc", base.Add(-time.Duration(i)*time.Minute), + business.RetentionContent, map[string]any{"nested": map[string]any{"a": []any{"q"}}, "n": i, "uniq": "value-" + uuid.NewString()})) + } + store := serverStore(t, conn, database) + require.NoError(t, store.Ensure(ctx)) + appendBatch(t, store, testDeployment, f.records...) + + inService := 0 + store.inService = func() { inService++ } + bounded := serverStore(t, conn, database, func(cfg *Config) { cfg.AggregateMaxBytes = 2 << 10 }) + boundedInService := 0 + bounded.inService = func() { boundedInService++ } + + // A nested payload filter has no SQL form: the service aggregates the events + // itself, in one bucket that keeps a sample and a value per event. + service := business.AuditRead{Scope: business.OrganizationAuditScope(org), Query: business.AuditQuery{ + OrgID: org, PayloadContains: map[string]any{"nested": map[string]any{"a": []any{"q"}}}, + }} + // ClickHouse groups it: one bucket per distinct text. + engine := business.AuditRead{Scope: business.OrganizationAuditScope(org), Query: business.AuditQuery{OrgID: org}} + for name, tc := range map[string]struct { + read business.AuditRead + spec business.AuditAggregationSpec + inService bool + }{ + "a percentile in the service": {service, business.AuditAggregationSpec{Metrics: []business.AuditMetric{{Op: "percentile", Field: "payload:n", Percentile: 0.5, Alias: "p50"}}}, true}, + "a distinct count in the service": {service, business.AuditAggregationSpec{Metrics: []business.AuditMetric{ + {Op: "count_distinct", Field: "payload:uniq", Alias: "distinct"}}}, true}, + "a bucket per key in the service": {service, business.AuditAggregationSpec{GroupBy: []string{"payload:uniq"}}, true}, + "a bucket per key in ClickHouse": {engine, business.AuditAggregationSpec{GroupBy: []string{"payload:uniq"}}, false}, + } { + before, boundedBefore := inService, boundedInService + _, err := bounded.AggregateAuditEvents(ctx, tc.read, tc.spec) + require.ErrorIs(t, err, business.ErrAuditAggregateTooLarge, name) + require.Equal(t, tc.inService, boundedInService-boundedBefore == 1, "%s: which path ran", name) + + // Under the default bound the same read is answered, as the reference + // semantics answer it. + got, err := store.AggregateAuditEvents(ctx, tc.read, tc.spec) + require.NoError(t, err, name) + require.Equal(t, f.referenceAggregate(t, tc.read, tc.spec), got, name) + require.Equal(t, tc.inService, inService-before == 1, "%s: the same path", name) + } + + _, err := New(conn, Config{Database: database, DeploymentID: testDeployment, EventsRetention: 2555 * 24 * time.Hour, + ContentDetailRetention: 30 * 24 * time.Hour, AggregateMaxBytes: -1}) + require.ErrorContains(t, err, "aggregation bound cannot be negative") +} diff --git a/module/services/accounts/code/pkg/business/audit.go b/module/services/accounts/code/pkg/business/audit.go index dd142ce73..2eb6f2258 100644 --- a/module/services/accounts/code/pkg/business/audit.go +++ b/module/services/accounts/code/pkg/business/audit.go @@ -568,7 +568,7 @@ func (s *Service) AggregateAuditLogForReader(ctx context.Context, reader string, if err != nil { return nil, err } - return s.auditStore.AggregateAuditEvents(ctx, read, spec) + return auditAggregateResult(s.auditStore.AggregateAuditEvents(ctx, read, spec)) } var out []AuditAggregateBucket err := s.store.WithOrgTx(ctx, q.OrgID, func(ctx context.Context) error { @@ -681,7 +681,7 @@ func (s *Service) AggregateAuditLog(ctx context.Context, q AuditQuery, spec Audi if err != nil { return nil, err } - return s.auditReads().AggregateAuditEvents(ctx, read, spec) + return auditAggregateResult(s.auditReads().AggregateAuditEvents(ctx, read, spec)) } // buildAuditEntry assembles an AuditEntry. actorID is the effective subject the diff --git a/module/services/accounts/code/pkg/business/audit_aggregate_bound.go b/module/services/accounts/code/pkg/business/audit_aggregate_bound.go new file mode 100644 index 000000000..315d5e048 --- /dev/null +++ b/module/services/accounts/code/pkg/business/audit_aggregate_bound.go @@ -0,0 +1,40 @@ +package business + +import ( + "errors" + + "google.golang.org/grpc/codes" + "google.golang.org/grpc/status" +) + +// AuditAggregateMaxBytes bounds what one aggregation of a store of record keeps +// while it reads: the buckets it has opened, the samples a percentile collects +// and the values a distinct count remembers, 32 MiB as the export is. A store +// reads in windows and frees each window before the next, so the window budget +// bounds what a window holds; this bounds what lives past the windows, which +// otherwise grows with the history an aggregation without a time range reads +// (a percentile keeps a float per event, a distinct count a string per value, +// a grouping a bucket per key) however few buckets the answer has. A store +// gives up as soon as the state it has gathered passes the bound. +const AuditAggregateMaxBytes = 32 << 20 + +// ErrAuditAggregateTooLarge is the refusal of an aggregation whose state passes +// AuditAggregateMaxBytes. Narrowing the read (a time range, an event type, an +// actor) or grouping by fewer dimensions is the way past it. No partial answer +// is ever returned in its place. +var ErrAuditAggregateTooLarge = errors.New("audit aggregation: more is kept than one read holds; narrow it by time range, actor or event type, or group by fewer dimensions") + +// auditAggregateResult is what an aggregation of a store of record returns to +// its caller: the buckets, or, when the store failed, no bucket and its error, +// which for a refusal for size is reported as a request the caller can narrow, +// not a server fault. +func auditAggregateResult(buckets []AuditAggregateBucket, err error) ([]AuditAggregateBucket, error) { + switch { + case err == nil: + return buckets, nil + case errors.Is(err, ErrAuditAggregateTooLarge): + return nil, status.Error(codes.ResourceExhausted, ErrAuditAggregateTooLarge.Error()) + default: + return nil, err + } +} diff --git a/module/services/accounts/code/pkg/business/audit_aggregate_bound_test.go b/module/services/accounts/code/pkg/business/audit_aggregate_bound_test.go new file mode 100644 index 000000000..c39184774 --- /dev/null +++ b/module/services/accounts/code/pkg/business/audit_aggregate_bound_test.go @@ -0,0 +1,72 @@ +package business + +import ( + "context" + "errors" + "fmt" + "strings" + "testing" + + "google.golang.org/grpc/codes" + "google.golang.org/grpc/status" +) + +// aggregatingStore is a store of record whose aggregation is whatever the test +// says. +type aggregatingStore struct { + AuditStore + buckets []AuditAggregateBucket + err error +} + +func (s aggregatingStore) AggregateAuditEvents(context.Context, AuditRead, AuditAggregationSpec) ([]AuditAggregateBucket, error) { + return s.buckets, s.err +} + +// A store that gives up an aggregation for its size is a request the caller can +// narrow, not a server fault: the refusal reaches the caller as such, with the +// same code an export's does, and no bucket comes with it. +func TestAuditAggregateTooLargeIsARefusalNotAFault(t *testing.T) { + service := &Service{auditStore: aggregatingStore{ + buckets: []AuditAggregateBucket{{Key: "partial", Count: 1}}, + err: fmt.Errorf("read: %w", ErrAuditAggregateTooLarge), + }} + buckets, err := service.AggregateAuditLog(t.Context(), AuditQuery{OrgID: "org-1"}, AuditAggregationSpec{}) + if status.Code(err) != codes.ResourceExhausted { + t.Fatalf("code = %v (%v), want ResourceExhausted", status.Code(err), err) + } + if !strings.Contains(err.Error(), "narrow it by time range, actor or event type") { + t.Errorf("the refusal says how to get past it, got %q", err) + } + if status.Code(err) != status.Code(exportRefusal(t)) { + t.Errorf("an aggregation and an export refused for size carry different codes") + } + if len(buckets) != 0 { + t.Errorf("a refused aggregation returned %d buckets", len(buckets)) + } +} + +func exportRefusal(t *testing.T) error { + t.Helper() + service := &Service{auditStore: exportingStore{err: ErrAuditExportTooLarge}} + _, _, _, err := service.ExportAuditLog(t.Context(), "org-1", "json", "", "", nil) + return err +} + +// Only that refusal is mapped: any other failure of the store is the caller's +// to see as it came, and an aggregation within the bound is served as before. +func TestAuditAggregateRefusalMapsOnlyItself(t *testing.T) { + failure := errors.New("warehouse unreachable") + service := &Service{auditStore: aggregatingStore{err: failure}} + _, err := service.AggregateAuditLog(t.Context(), AuditQuery{OrgID: "org-1"}, AuditAggregationSpec{}) + if !errors.Is(err, failure) || status.Code(err) == codes.ResourceExhausted { + t.Fatalf("err = %v, want the store's own failure", err) + } + + want := []AuditAggregateBucket{{Key: "saas.auth.login", Count: 3}} + service = &Service{auditStore: aggregatingStore{buckets: want}} + got, err := service.AggregateAuditLog(t.Context(), AuditQuery{OrgID: "org-1"}, AuditAggregationSpec{}) + if err != nil || len(got) != 1 || got[0].Key != want[0].Key || got[0].Count != 3 { + t.Fatalf("buckets = %v, err = %v; want %v", got, err, want) + } +} From 152fd504896be70a61226e2bc0dda68ceca2499d Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Wed, 7 Oct 2026 10:55:03 -0400 Subject: [PATCH 45/69] fix(accounts,auth-gateway): the observability state decides; build the metrics exporter from the endpoint URL (Refs #1022) The `observability` group is layered: a cell's values override the module's local defaults one key at a time. A deployed producer therefore receives `TELEMETRY_STATE=available` and `OTEL_EXPORTER_OTLP_ENDPOINT` from the cell and a leftover `TELEMETRY_ABSENT_REASON` from the local profile, and the previous resolution refused that as "both set", so the service would not start on the one configuration a cell is certain to deliver. The state now decides, in both mirrored copies of telemetry_destination.go: - `available` requires a non-empty endpoint. A leftover reason is ignored. - `absent` requires a non-empty reason. A leftover endpoint is ignored, and is not parsed, so a stale one cannot stop a cell that says it has no collector. - A missing or unknown state, and the key the state requires being missing, still refuse to start. What was ignored is carried on the destination and reported once, where the process acts on the answer (accounts' doWork, auth-gateway's main), not by requireStartupConfiguration, which reads the group a second time only to refuse. The notice names the key and never its value: a leftover endpoint may carry credentials and a leftover reason is free text. The metrics exporter is now built with `otlpmetricgrpc.WithEndpointURL`, so the official exporter reads its transport off the URL's scheme instead of being given host:port and a separately derived insecure flag. The destination carries the origin as scheme://host:port with the port always explicit, because gRPC would otherwise dial 443 whatever the scheme. Traces keep wool's Enable path and its labelled https stopgap; a separate core change teaches wool TLS-from-scheme. An empty URL is refused rather than handed to WithEndpointURL, which would fall back to the exporter's own localhost:4317 and report success. The exporter used to be created before the resource, and a resource that could not be built returned without shutting it down, leaking its client connection. The resource is now built first, so there is no exporter to leak; the later failure paths already shut the provider down, which shuts the exporter down with it. Tests: both layered cases, a module default of `absent` with a reason under a cell override of `available` with an endpoint (and the reverse), a leftover endpoint that is malformed or carries credentials, blank leftovers, the same through the SDK's lookup, and the requirement that a state still supplies its own key. An https:// URL is shown to be TLS by failing to export to a plaintext receiver, and the test fails if the exporter is given a constant insecure transport. The mirror test still holds the two copies identical. FEATURES, the local profile and the local-dogfood example describe the rule. Co-Authored-By: Claude Opus 5.5 --- .../local-dogfood/observability.env.example | 3 +- module/FEATURES.md | 18 +- module/configurations/local/observability.env | 4 + .../accounts/code/telemetry_destination.go | 97 ++++++--- .../code/telemetry_destination_test.go | 187 +++++++++++++++--- .../accounts/code/telemetry_metrics.go | 19 +- .../accounts/code/telemetry_metrics_test.go | 61 +++++- module/services/accounts/code/work.go | 5 + module/services/auth-gateway/code/main.go | 4 + .../code/telemetry_destination.go | 97 ++++++--- .../code/telemetry_destination_test.go | 187 +++++++++++++++--- .../auth-gateway/code/telemetry_metrics.go | 19 +- .../code/telemetry_metrics_test.go | 61 +++++- 13 files changed, 636 insertions(+), 126 deletions(-) diff --git a/configurations/local-dogfood/observability.env.example b/configurations/local-dogfood/observability.env.example index 73ad6b219..84f833283 100644 --- a/configurations/local-dogfood/observability.env.example +++ b/configurations/local-dogfood/observability.env.example @@ -15,4 +15,5 @@ TELEMETRY_ABSENT_REASON=A local dogfood run has no cell collector; traces go to # # The endpoint is the collector's OTLP/gRPC address. http:// is plaintext, which # only belongs on loopback or inside the mesh; https:// is TLS. Set the endpoint -# when the state is available and the reason when it is absent, never both. +# when the state is available and the reason when it is absent. The state decides: the +# other key is ignored, and the process logs that once. diff --git a/module/FEATURES.md b/module/FEATURES.md index 29566a9fc..15c4b824e 100644 --- a/module/FEATURES.md +++ b/module/FEATURES.md @@ -656,8 +656,8 @@ Environment variables consumed by the api: | `ERROR_TRACKING_MODE` | Explicit `disabled` or `sentry`; rejects partial config | | `SENTRY_DSN` | Server Sentry DSN, required in Sentry mode | | `TELEMETRY_STATE` | `observability` group, delivered by the platform: `available` or `absent`, never empty; accounts and auth-gateway refuse to start without it | -| `OTEL_EXPORTER_OTLP_ENDPOINT` | `observability` group: the cell collector's OTLP/gRPC address, present only when `available`; `http://` is plaintext on the wire because the mesh supplies mTLS, `https://` is TLS | -| `TELEMETRY_ABSENT_REASON` | `observability` group: why the cell has no collector, present only when `absent`; logged once at startup | +| `OTEL_EXPORTER_OTLP_ENDPOINT` | `observability` group: the cell collector's OTLP/gRPC address, required when `available` and ignored when `absent`; `http://` is plaintext on the wire because the mesh supplies mTLS, `https://` is TLS | +| `TELEMETRY_ABSENT_REASON` | `observability` group: why the cell has no collector, required when `absent` and ignored when `available`; logged once at startup | | `ABUSE_PROTECTION_MODE` | Explicit `disabled` or `turnstile` | | `TURNSTILE_SECRET_KEY` | Server-only Siteverify credential | | `TURNSTILE_ALLOWED_HOSTNAMES` | Exact accepted Turnstile response hostnames | @@ -668,11 +668,15 @@ export traces and unsampled request and Go runtime metrics over OTLP/gRPC to the cell's collector at `OTEL_EXPORTER_OTLP_ENDPOINT`. The auth-gateway covers both its HTTP gateway and gRPC ext_authz authorization service. When it is `absent`, they export nothing over OTLP, log `TELEMETRY_ABSENT_REASON` once and boot; a local run -then uses wool's stdout tracer for traces. Anything else — the state missing or -unknown, `available` without an endpoint, `absent` without a reason, or both -set — refuses to start, because a group that did not arrive is not a cell without -a collector. An `https://` endpoint is refused at startup for now: wool's OTLP -tracer dials plaintext only, so it would be sent in the clear and reported as TLS. +then uses wool's stdout tracer for traces. The state decides: a cell's values +override the module's local defaults one key at a time, so a deployed cell that +says `available` with an endpoint can still carry the `absent` reason the local +profile left behind. The key the state does not use is ignored, and a startup +warning names it once. Anything else — the state missing or unknown, `available` +without an endpoint, or `absent` without a reason — refuses to start, because a +group that did not arrive is not a cell without a collector. An `https://` +endpoint is refused at startup for now: wool's OTLP tracer dials plaintext only, +so it would be sent in the clear and reported as TLS. Metrics leave by OTLP push alone — decided 2026-10-06, because the cell's collector is the record for traces and metrics take the same path — so neither service serves a scrape endpoint. `service.name` comes from `OTEL_SERVICE_NAME` or diff --git a/module/configurations/local/observability.env b/module/configurations/local/observability.env index ed8a6da2c..4d7ddcb97 100644 --- a/module/configurations/local/observability.env +++ b/module/configurations/local/observability.env @@ -10,6 +10,10 @@ # TELEMETRY_STATE=absent the cell has none. Also sets TELEMETRY_ABSENT_REASON to # say why. # +# The state decides. A cell's values override these defaults one key at a time, so a +# cell that sets `available` and an endpoint still carries the reason below; the +# process ignores the key its state does not use and logs that once. +# # A local run has no cell, so it has no collector: traces go to wool's stdout tracer # and metrics are not exported. Logs always go to stdout. TELEMETRY_STATE=absent diff --git a/module/services/accounts/code/telemetry_destination.go b/module/services/accounts/code/telemetry_destination.go index e3296805d..c752d82fd 100644 --- a/module/services/accounts/code/telemetry_destination.go +++ b/module/services/accounts/code/telemetry_destination.go @@ -32,10 +32,17 @@ import ( // collector: the cell does, and a workload that cannot find it does not // substitute one. // +// The state decides. The group is layered: the cell's values override the +// module's local defaults one key at a time, so a deployed workload sees the +// cell's `available` and endpoint beside the `absent` reason the module's local +// profile left behind. The key that does not belong to the state is therefore +// ignored, not refused, and reported once through IgnoredNotice. What each state +// requires is unchanged: `available` needs an endpoint, `absent` needs a reason. +// // A group that did not arrive is not a cell without a backend. Reading the first // as the second would leave a deployment that converges green and exports -// nothing, so anything that is not exactly one of the two answers above refuses -// to start, naming what is missing. +// nothing, so a missing or unknown state, or the key the state requires being +// absent, refuses to start, naming what is missing. // // This file exists byte for byte in services/accounts/code and // services/auth-gateway/code: the two services are independent Go modules with @@ -56,17 +63,28 @@ const ( maxAbsentReasonLength = 300 ) -// telemetryDestination is the resolved answer. Exactly one of Endpoint and +// telemetryDestination is the resolved answer. Exactly one of URL and // AbsentReason is set. type telemetryDestination struct { - // Endpoint is the collector's OTLP/gRPC address as host:port, the form the - // exporters take. Empty when the cell has no collector. + // URL is the collector's origin as scheme://host:port, the port always + // explicit, which is the form otlpmetricgrpc.WithEndpointURL takes: the + // exporter reads its transport off the scheme. Empty when the cell has no + // collector. + URL string + // Endpoint is the same address as host:port. wool's tracer takes it in this + // form and has no option that reads a URL. Empty when the cell has no + // collector. Endpoint string // Insecure is true when the endpoint was http://, which the mesh protects. - // It is derived from the endpoint's scheme and never assumed. + // wool's tracer needs it spelled out; it is derived from the endpoint's + // scheme and never assumed. Insecure bool - // AbsentReason says why the cell has no collector. Empty when Endpoint is set. + // AbsentReason says why the cell has no collector. Empty when URL is set. AbsentReason string + // Ignored names the key the group carried that has no meaning for the state + // that decided: the absent reason under `available`, the endpoint under + // `absent`. Empty when the group carried nothing extra. + Ignored string } // Available reports whether the cell has a collector to export to. @@ -74,6 +92,23 @@ func (d telemetryDestination) Available() bool { return d.Endpoint != "" } +// IgnoredNotice is the one line to log, once, when the group carried a key the +// state does not use. It is empty otherwise. It names the key and never its +// value: a leftover endpoint may carry credentials, and a leftover reason is +// free text. +func (d telemetryDestination) IgnoredNotice() string { + if d.Ignored == "" { + return "" + } + state := telemetryStateAbsent + if d.Available() { + state = telemetryStateAvailable + } + return fmt.Sprintf( + "observability: %s is set but %s is `%s`, which decides; %s is ignored (a leftover from a lower layer of the group)", + d.Ignored, telemetryStateKey, state, d.Ignored) +} + // observabilityValue reads one key of the `observability` group through the // SDK. It deliberately has no fallback to a bare process variable: an ambient // OTEL_EXPORTER_OTLP_ENDPOINT in a shell or a pod is conventionally the @@ -112,13 +147,11 @@ func resolveTelemetryDestination(read func(key string) string) (telemetryDestina "observability: %s is %q in the `observability` configuration group; it must be `%s` or `%s`", telemetryStateKey, state, telemetryStateAvailable, telemetryStateAbsent) } - if endpoint != "" && reason != "" { - return telemetryDestination{}, fmt.Errorf( - "observability: the `observability` configuration group sets both %s and %s; "+ - "a cell either has a collector or says why it has none", - telemetryEndpointKey, telemetryAbsentReasonKey) - } + // The state decides: the key that belongs to the other state is a leftover + // of a lower layer and is neither read nor refused. The endpoint under + // `absent` is not even parsed, so a stale one cannot stop a cell that says + // it has no collector. if state == telemetryStateAbsent { if reason == "" { return telemetryDestination{}, fmt.Errorf( @@ -126,7 +159,11 @@ func resolveTelemetryDestination(read func(key string) string) (telemetryDestina "an absent collector must say why", telemetryStateKey, telemetryStateAbsent, telemetryAbsentReasonKey) } - return telemetryDestination{AbsentReason: reason}, nil + destination := telemetryDestination{AbsentReason: reason} + if endpoint != "" { + destination.Ignored = telemetryEndpointKey + } + return destination, nil } if endpoint == "" { @@ -134,39 +171,46 @@ func resolveTelemetryDestination(read func(key string) string) (telemetryDestina "observability: %s is `%s` but %s is not set in the `observability` configuration group", telemetryStateKey, telemetryStateAvailable, telemetryEndpointKey) } - hostPort, insecure, err := parseCollectorEndpoint(endpoint) + origin, hostPort, insecure, err := parseCollectorEndpoint(endpoint) if err != nil { return telemetryDestination{}, fmt.Errorf("observability: %s: %w", telemetryEndpointKey, err) } - return telemetryDestination{Endpoint: hostPort, Insecure: insecure}, nil + destination := telemetryDestination{URL: origin, Endpoint: hostPort, Insecure: insecure} + if reason != "" { + destination.Ignored = telemetryAbsentReasonKey + } + return destination, nil } -// parseCollectorEndpoint turns the delivered URL into the host:port the OTLP -// gRPC exporters take, and reads the transport off its scheme. The URL is never -// echoed back: a malformed one may carry credentials. -func parseCollectorEndpoint(raw string) (hostPort string, insecure bool, err error) { +// parseCollectorEndpoint turns the delivered URL into the two forms the +// exporters take — the origin with an explicit port, and host:port — and reads +// the transport off its scheme. The URL is never echoed back: a malformed one +// may carry credentials. +func parseCollectorEndpoint(raw string) (origin, hostPort string, insecure bool, err error) { parsed, parseErr := url.Parse(raw) if parseErr != nil { - return "", false, errors.New("is not a URL; expected http://host:port or https://host:port") + return "", "", false, errors.New("is not a URL; expected http://host:port or https://host:port") } switch parsed.Scheme { case "http": insecure = true case "https": default: - return "", false, fmt.Errorf("must start with http:// or https://, got scheme %q", parsed.Scheme) + return "", "", false, fmt.Errorf("must start with http:// or https://, got scheme %q", parsed.Scheme) } if parsed.Hostname() == "" { - return "", false, errors.New("has no host") + return "", "", false, errors.New("has no host") } if parsed.User != nil { - return "", false, errors.New("must not carry credentials") + return "", "", false, errors.New("must not carry credentials") } // OTLP/gRPC addresses a collector, not a path: a path or a query is the // OTLP/HTTP form (/v1/traces), which this exporter does not speak. if (parsed.Path != "" && parsed.Path != "/") || parsed.RawQuery != "" || parsed.Fragment != "" { - return "", false, errors.New("must be the collector's origin only, with no path, query or fragment") + return "", "", false, errors.New("must be the collector's origin only, with no path, query or fragment") } + // The port is made explicit because gRPC would otherwise dial 443 whatever + // the scheme, and a plaintext http:// collector with no port is not on 443. port := parsed.Port() if port == "" { port = "80" @@ -174,7 +218,8 @@ func parseCollectorEndpoint(raw string) (hostPort string, insecure bool, err err port = "443" } } - return net.JoinHostPort(parsed.Hostname(), port), insecure, nil + hostPort = net.JoinHostPort(parsed.Hostname(), port) + return parsed.Scheme + "://" + hostPort, hostPort, insecure, nil } // boundedLine collapses a free-text value to one bounded line. diff --git a/module/services/accounts/code/telemetry_destination_test.go b/module/services/accounts/code/telemetry_destination_test.go index c2d1e4d24..4ac8a2b75 100644 --- a/module/services/accounts/code/telemetry_destination_test.go +++ b/module/services/accounts/code/telemetry_destination_test.go @@ -29,32 +29,39 @@ func TestTelemetryDestinationAvailableExportsToTheEndpoint(t *testing.T) { })) require.NoError(t, err) require.True(t, destination.Available()) + require.Equal(t, collector, destination.URL, "the metrics exporter reads its transport off this URL") require.Equal(t, "otel-collector.otel-collector.svc.cluster.local:4317", destination.Endpoint) require.True(t, destination.Insecure, "http:// is plaintext on the wire; the mesh supplies mTLS") require.Empty(t, destination.AbsentReason) + require.Empty(t, destination.Ignored) + require.Empty(t, destination.IgnoredNotice()) } -// The transport comes from the endpoint's scheme, never from a constant. +// The transport comes from the endpoint's scheme, never from a constant, and +// the URL handed to the metrics exporter always carries an explicit port: gRPC +// would otherwise dial 443 whatever the scheme. func TestTelemetryDestinationReadsTransportFromTheScheme(t *testing.T) { for endpoint, want := range map[string]struct { + url string hostPort string insecure bool }{ - "http://collector.example:4317": {"collector.example:4317", true}, - "https://collector.example:4317": {"collector.example:4317", false}, - "https://collector.example": {"collector.example:443", false}, - "http://collector.example": {"collector.example:80", true}, - "http://collector.example:4317/": {"collector.example:4317", true}, - " http://collector.example:4317 ": {"collector.example:4317", true}, - "http://[2001:db8::1]:4317": {"[2001:db8::1]:4317", true}, - "https://10.0.0.7:4317": {"10.0.0.7:4317", false}, - "http://otel-collector.ns.svc:4317/": {"otel-collector.ns.svc:4317", true}, + "http://collector.example:4317": {"http://collector.example:4317", "collector.example:4317", true}, + "https://collector.example:4317": {"https://collector.example:4317", "collector.example:4317", false}, + "https://collector.example": {"https://collector.example:443", "collector.example:443", false}, + "http://collector.example": {"http://collector.example:80", "collector.example:80", true}, + "http://collector.example:4317/": {"http://collector.example:4317", "collector.example:4317", true}, + " http://collector.example:4317 ": {"http://collector.example:4317", "collector.example:4317", true}, + "http://[2001:db8::1]:4317": {"http://[2001:db8::1]:4317", "[2001:db8::1]:4317", true}, + "https://10.0.0.7:4317": {"https://10.0.0.7:4317", "10.0.0.7:4317", false}, + "http://otel-collector.ns.svc:4317/": {"http://otel-collector.ns.svc:4317", "otel-collector.ns.svc:4317", true}, } { destination, err := resolveTelemetryDestination(group(map[string]string{ "TELEMETRY_STATE": "available", "OTEL_EXPORTER_OTLP_ENDPOINT": endpoint, })) require.NoError(t, err, endpoint) + require.Equal(t, want.url, destination.URL, endpoint) require.Equal(t, want.hostPort, destination.Endpoint, endpoint) require.Equal(t, want.insecure, destination.Insecure, endpoint) } @@ -69,8 +76,11 @@ func TestTelemetryDestinationAbsentCarriesTheReason(t *testing.T) { })) require.NoError(t, err) require.False(t, destination.Available()) + require.Empty(t, destination.URL) require.Empty(t, destination.Endpoint) require.Equal(t, "This cell's bucket has no ops cell, so no collector is rendered.", destination.AbsentReason) + require.Empty(t, destination.Ignored) + require.Empty(t, destination.IgnoredNotice()) } // The reason is operator-written free text that reaches a log line. @@ -86,7 +96,9 @@ func TestTelemetryDestinationAbsentReasonIsOneBoundedLine(t *testing.T) { } // The third outcome: anything else refuses to start, naming what is wrong. -// Configuration that did not arrive is never read as "no collector". +// Configuration that did not arrive is never read as "no collector". A key the +// state does not use is not a refusal — it is ignored, below — but a key the +// state requires, missing, always is. func TestTelemetryDestinationRefusesEverythingElse(t *testing.T) { for name, tc := range map[string]struct { values map[string]string @@ -136,20 +148,6 @@ func TestTelemetryDestinationRefusesEverythingElse(t *testing.T) { values: map[string]string{"TELEMETRY_STATE": "absent", "OTEL_EXPORTER_OTLP_ENDPOINT": collector}, want: "TELEMETRY_ABSENT_REASON is not set", }, - "available with both an endpoint and a reason": { - values: map[string]string{ - "TELEMETRY_STATE": "available", "OTEL_EXPORTER_OTLP_ENDPOINT": collector, - "TELEMETRY_ABSENT_REASON": "no collector", - }, - want: "sets both OTEL_EXPORTER_OTLP_ENDPOINT and TELEMETRY_ABSENT_REASON", - }, - "absent with both an endpoint and a reason": { - values: map[string]string{ - "TELEMETRY_STATE": "absent", "OTEL_EXPORTER_OTLP_ENDPOINT": collector, - "TELEMETRY_ABSENT_REASON": "no collector", - }, - want: "sets both OTEL_EXPORTER_OTLP_ENDPOINT and TELEMETRY_ABSENT_REASON", - }, "endpoint without a scheme": { values: map[string]string{"TELEMETRY_STATE": "available", "OTEL_EXPORTER_OTLP_ENDPOINT": "otel-collector.ns.svc:4317"}, want: "must start with http:// or https://", @@ -181,6 +179,129 @@ func TestTelemetryDestinationRefusesEverythingElse(t *testing.T) { } } +// The state decides. The cell's values override the module's local defaults one +// key at a time, so a deployed producer sees `available` and an endpoint from the +// cell beside the reason the local profile left behind. That is not a +// contradiction to refuse: the reason is ignored, and said so once. +func TestTelemetryDestinationAvailableIgnoresALeftoverReason(t *testing.T) { + destination, err := resolveTelemetryDestination(group(map[string]string{ + "TELEMETRY_STATE": "available", + "OTEL_EXPORTER_OTLP_ENDPOINT": collector, + "TELEMETRY_ABSENT_REASON": "A local run has no cell collector.", + })) + require.NoError(t, err) + require.True(t, destination.Available()) + require.Equal(t, collector, destination.URL) + require.Empty(t, destination.AbsentReason, "a leftover reason must not turn an available collector into an absent one") + require.Equal(t, "TELEMETRY_ABSENT_REASON", destination.Ignored) + notice := destination.IgnoredNotice() + require.Contains(t, notice, "TELEMETRY_ABSENT_REASON is set but TELEMETRY_STATE is `available`") + require.NotContains(t, notice, "A local run", "the notice names the key, never its free-text value") +} + +// ... and the other way round: a cell that says it has no collector is not +// refused for an endpoint a lower layer left behind, and that endpoint is not +// even parsed. +func TestTelemetryDestinationAbsentIgnoresALeftoverEndpoint(t *testing.T) { + for name, leftover := range map[string]string{ + "a usable endpoint": collector, + "a malformed endpoint": "otel-collector.ns.svc:4317", + "an endpoint with a user": "http://user:hunter2@collector.example:4317", + "an unreadable endpoint": "http://[::1", + } { + t.Run(name, func(t *testing.T) { + destination, err := resolveTelemetryDestination(group(map[string]string{ + "TELEMETRY_STATE": "absent", + "TELEMETRY_ABSENT_REASON": "This cell has no ops cell.", + "OTEL_EXPORTER_OTLP_ENDPOINT": leftover, + })) + require.NoError(t, err) + require.False(t, destination.Available(), "a leftover endpoint must not turn an absent collector into an available one") + require.Empty(t, destination.URL) + require.Empty(t, destination.Endpoint) + require.Equal(t, "This cell has no ops cell.", destination.AbsentReason) + require.Equal(t, "OTEL_EXPORTER_OTLP_ENDPOINT", destination.Ignored) + notice := destination.IgnoredNotice() + require.Contains(t, notice, "OTEL_EXPORTER_OTLP_ENDPOINT is set but TELEMETRY_STATE is `absent`") + require.NotContains(t, notice, "hunter2", "the notice names the key, never its value") + }) + } +} + +// A blank leftover is nothing left over. +func TestTelemetryDestinationBlankLeftoversAreNotNoticed(t *testing.T) { + available, err := resolveTelemetryDestination(group(map[string]string{ + "TELEMETRY_STATE": "available", "OTEL_EXPORTER_OTLP_ENDPOINT": collector, "TELEMETRY_ABSENT_REASON": " \n ", + })) + require.NoError(t, err) + require.Empty(t, available.IgnoredNotice()) + + absent, err := resolveTelemetryDestination(group(map[string]string{ + "TELEMETRY_STATE": "absent", "TELEMETRY_ABSENT_REASON": "no collector", "OTEL_EXPORTER_OTLP_ENDPOINT": " ", + })) + require.NoError(t, err) + require.Empty(t, absent.IgnoredNotice()) +} + +// layered is what the platform hands a workload: each layer overrides the one +// below it key by key, so a key a higher layer does not set survives from the +// lower one. +func layered(layers ...map[string]string) func(string) string { + merged := map[string]string{} + for _, layer := range layers { + for key, value := range layer { + merged[key] = value + } + } + return group(merged) +} + +// The case this exists for: the module's local profile defaults to `absent` with +// a reason, and a deployed cell overrides the state and the endpoint but not the +// reason. The cell's answer wins, whichever way it points. +func TestTelemetryDestinationTheCellOverridesTheModuleDefaults(t *testing.T) { + moduleDefault := map[string]string{ + "TELEMETRY_STATE": "absent", + "TELEMETRY_ABSENT_REASON": "A local run has no cell collector.", + } + + t.Run("a cell with a collector overrides the local absent default", func(t *testing.T) { + destination, err := resolveTelemetryDestination(layered(moduleDefault, map[string]string{ + "TELEMETRY_STATE": "available", + "OTEL_EXPORTER_OTLP_ENDPOINT": collector, + })) + require.NoError(t, err) + require.True(t, destination.Available()) + require.Equal(t, collector, destination.URL) + require.Empty(t, destination.AbsentReason) + require.Equal(t, "TELEMETRY_ABSENT_REASON", destination.Ignored) + }) + + t.Run("a cell with no collector, over a default that had an endpoint, is absent", func(t *testing.T) { + destination, err := resolveTelemetryDestination(layered( + map[string]string{"TELEMETRY_STATE": "available", "OTEL_EXPORTER_OTLP_ENDPOINT": collector}, + map[string]string{"TELEMETRY_STATE": "absent", "TELEMETRY_ABSENT_REASON": "This cell has no ops cell."}, + )) + require.NoError(t, err) + require.False(t, destination.Available()) + require.Equal(t, "This cell has no ops cell.", destination.AbsentReason) + require.Equal(t, "OTEL_EXPORTER_OTLP_ENDPOINT", destination.Ignored) + }) + + t.Run("a cell that overrides only the state still has to supply what it requires", func(t *testing.T) { + _, err := resolveTelemetryDestination(layered(moduleDefault, map[string]string{"TELEMETRY_STATE": "available"})) + require.ErrorContains(t, err, "OTEL_EXPORTER_OTLP_ENDPOINT is not set", + "the module default's reason is not an endpoint") + + _, err = resolveTelemetryDestination(layered( + map[string]string{"TELEMETRY_STATE": "available", "OTEL_EXPORTER_OTLP_ENDPOINT": collector}, + map[string]string{"TELEMETRY_STATE": "absent"}, + )) + require.ErrorContains(t, err, "TELEMETRY_ABSENT_REASON is not set", + "the module default's endpoint is not a reason") + }) +} + // A refusal quotes no part of an endpoint, which may carry credentials. func TestTelemetryDestinationRefusalDoesNotEchoCredentials(t *testing.T) { _, err := resolveTelemetryDestination(group(map[string]string{ @@ -228,6 +349,22 @@ func TestConfiguredTelemetryDestinationReadsTheObservabilityGroup(t *testing.T) require.Equal(t, "a local run has no cell collector", destination.AbsentReason) } +// Through the SDK's lookup, with the keys exactly as a deployed cell delivers +// them after the layers merge: the cell's state and endpoint, and the reason the +// module's local profile left behind. +func TestConfiguredTelemetryDestinationStateDecidesAcrossLayers(t *testing.T) { + const prefix = "CODEFLY__WORKSPACE_CONFIGURATION__OBSERVABILITY__" + t.Setenv(prefix+"TELEMETRY_STATE", "available") + t.Setenv(prefix+"OTEL_EXPORTER_OTLP_ENDPOINT", collector) + t.Setenv(prefix+"TELEMETRY_ABSENT_REASON", "A local run has no cell collector.") + + destination, err := configuredTelemetryDestination() + require.NoError(t, err) + require.True(t, destination.Available()) + require.Equal(t, collector, destination.URL) + require.NotEmpty(t, destination.IgnoredNotice()) +} + // Configuration that cannot work is refused before the process acquires // anything, including in a local run: a group that did not arrive is not a cell // without a collector anywhere. diff --git a/module/services/accounts/code/telemetry_metrics.go b/module/services/accounts/code/telemetry_metrics.go index 9ab41d0ea..b7ce54122 100644 --- a/module/services/accounts/code/telemetry_metrics.go +++ b/module/services/accounts/code/telemetry_metrics.go @@ -2,6 +2,7 @@ package main import ( "context" + "errors" runtimemetrics "runtime/metrics" "time" @@ -20,17 +21,21 @@ type otelMetrics struct { } func enableOTELMetrics(ctx context.Context, destination telemetryDestination) (*otelMetrics, error) { - options := []otlpmetricgrpc.Option{otlpmetricgrpc.WithEndpoint(destination.Endpoint)} - if destination.Insecure { - // http://: plaintext on the wire, because the mesh supplies mTLS. An - // https:// endpoint takes the exporter's default, TLS. - options = append(options, otlpmetricgrpc.WithInsecure()) + if destination.URL == "" { + // WithEndpointURL("") would fall back to the exporter's own default, + // localhost:4317: a process that exports to nowhere and reports success. + return nil, errors.New("telemetry: no collector URL to export metrics to") } - exporter, err := otlpmetricgrpc.New(ctx, options...) + // The resource comes before the exporter, so a failure here leaves nothing + // to shut down: an exporter holds a client connection that only Shutdown + // releases. + res, err := currentTelemetryResource(ctx) if err != nil { return nil, err } - res, err := currentTelemetryResource(ctx) + // The exporter reads the transport off the URL's scheme itself: http:// is + // plaintext on the wire, because the mesh supplies mTLS, and https:// is TLS. + exporter, err := otlpmetricgrpc.New(ctx, otlpmetricgrpc.WithEndpointURL(destination.URL)) if err != nil { return nil, err } diff --git a/module/services/accounts/code/telemetry_metrics_test.go b/module/services/accounts/code/telemetry_metrics_test.go index fd7fd94af..7af83c684 100644 --- a/module/services/accounts/code/telemetry_metrics_test.go +++ b/module/services/accounts/code/telemetry_metrics_test.go @@ -71,7 +71,7 @@ func TestTelemetryMetricsExportRuntimeAndUnsampledRED(t *testing.T) { ) otel.SetTracerProvider(tracerProvider) - metrics, err := enableOTELMetrics(t.Context(), telemetryDestination{Endpoint: endpoint, Insecure: true}) + metrics, err := enableOTELMetrics(t.Context(), availableAt(t, "http", endpoint)) require.NoError(t, err) require.NotNil(t, metrics) t.Cleanup(func() { @@ -111,6 +111,65 @@ func TestTelemetryMetricsExportRuntimeAndUnsampledRED(t *testing.T) { require.Empty(t, spanExporter.GetSpans()) } +// availableAt resolves the destination the way the process does, from the +// endpoint the platform would deliver, so these tests hand the exporter the URL +// it really receives rather than a hand-built struct. +func availableAt(t *testing.T, scheme, hostPort string) telemetryDestination { + t.Helper() + destination, err := resolveTelemetryDestination(group(map[string]string{ + "TELEMETRY_STATE": "available", + "OTEL_EXPORTER_OTLP_ENDPOINT": scheme + "://" + hostPort, + })) + require.NoError(t, err) + return destination +} + +// The exporter reads its transport off the URL's scheme, so an https:// URL is +// TLS and cannot talk to a plaintext receiver. A constant WithInsecure() would +// have let it. +func TestEnableOTELMetricsDerivesTransportFromTheURLScheme(t *testing.T) { + endpoint, capture := startMetricCapture(t) + previousMeterProvider := otel.GetMeterProvider() + metrics, err := enableOTELMetrics(t.Context(), availableAt(t, "https", endpoint)) + require.NoError(t, err) + require.NotNil(t, metrics) + t.Cleanup(func() { + shutdownCtx, cancel := context.WithTimeout(context.Background(), time.Second) + defer cancel() + _ = metrics.Shutdown(shutdownCtx) // the flush it attempts is the failure under test + otel.SetMeterProvider(previousMeterProvider) + }) + + flushCtx, cancel := context.WithTimeout(t.Context(), time.Second) + defer cancel() + require.Error(t, metrics.provider.ForceFlush(flushCtx), "a TLS client cannot export to a plaintext receiver") + select { + case <-capture.requests: + t.Fatal("an https:// destination reached a plaintext receiver") + default: + } +} + +// A destination with no collector URL is refused. WithEndpointURL("") would +// otherwise fall back to the exporter's own default, localhost:4317. +func TestEnableOTELMetricsRefusesADestinationWithoutAURL(t *testing.T) { + metrics, err := enableOTELMetrics(t.Context(), telemetryDestination{AbsentReason: "no collector"}) + require.ErrorContains(t, err, "no collector URL") + require.Nil(t, metrics) +} + +// A resource that cannot be built fails the whole call, and the call builds it +// before it creates the exporter: there is no exporter, and so no client +// connection, left running behind the failure. +func TestEnableOTELMetricsFailsWhenTheResourceCannotBeBuilt(t *testing.T) { + t.Setenv("OTEL_SERVICE_NAME", "") + t.Setenv("OTEL_RESOURCE_ATTRIBUTES", "an-attribute-with-no-value") + endpoint, _ := startMetricCapture(t) + metrics, err := enableOTELMetrics(t.Context(), availableAt(t, "http", endpoint)) + require.Error(t, err) + require.Nil(t, metrics) +} + func startMetricCapture(t *testing.T) (string, *metricCapture) { t.Helper() listener, err := net.Listen("tcp", "127.0.0.1:0") diff --git a/module/services/accounts/code/work.go b/module/services/accounts/code/work.go index b27571e53..34faee487 100644 --- a/module/services/accounts/code/work.go +++ b/module/services/accounts/code/work.go @@ -112,6 +112,11 @@ func doWork(ctx context.Context) (Clean, error) { w.Info("OTEL export disabled: the cell has no collector", wool.Field("reason", telemetryDestination.AbsentReason)) } + // The state decided; a key it does not use is a leftover, reported here once. + // requireStartupConfiguration reads the group too, but only to refuse. + if notice := telemetryDestination.IgnoredNotice(); notice != "" { + w.Warn(notice) + } store, err := infra.NewPostgresStore(ctx) if err != nil { diff --git a/module/services/auth-gateway/code/main.go b/module/services/auth-gateway/code/main.go index 33a03c8bb..4ea768fe6 100644 --- a/module/services/auth-gateway/code/main.go +++ b/module/services/auth-gateway/code/main.go @@ -66,6 +66,10 @@ func main() { } else { log.Printf("OTEL export disabled: the cell has no collector: %s", telemetryDestination.AbsentReason) } + // The state decided; a key it does not use is a leftover, reported here once. + if notice := telemetryDestination.IgnoredNotice(); notice != "" { + log.Print(notice) + } defer func() { shutdownCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second) defer cancel() diff --git a/module/services/auth-gateway/code/telemetry_destination.go b/module/services/auth-gateway/code/telemetry_destination.go index e3296805d..c752d82fd 100644 --- a/module/services/auth-gateway/code/telemetry_destination.go +++ b/module/services/auth-gateway/code/telemetry_destination.go @@ -32,10 +32,17 @@ import ( // collector: the cell does, and a workload that cannot find it does not // substitute one. // +// The state decides. The group is layered: the cell's values override the +// module's local defaults one key at a time, so a deployed workload sees the +// cell's `available` and endpoint beside the `absent` reason the module's local +// profile left behind. The key that does not belong to the state is therefore +// ignored, not refused, and reported once through IgnoredNotice. What each state +// requires is unchanged: `available` needs an endpoint, `absent` needs a reason. +// // A group that did not arrive is not a cell without a backend. Reading the first // as the second would leave a deployment that converges green and exports -// nothing, so anything that is not exactly one of the two answers above refuses -// to start, naming what is missing. +// nothing, so a missing or unknown state, or the key the state requires being +// absent, refuses to start, naming what is missing. // // This file exists byte for byte in services/accounts/code and // services/auth-gateway/code: the two services are independent Go modules with @@ -56,17 +63,28 @@ const ( maxAbsentReasonLength = 300 ) -// telemetryDestination is the resolved answer. Exactly one of Endpoint and +// telemetryDestination is the resolved answer. Exactly one of URL and // AbsentReason is set. type telemetryDestination struct { - // Endpoint is the collector's OTLP/gRPC address as host:port, the form the - // exporters take. Empty when the cell has no collector. + // URL is the collector's origin as scheme://host:port, the port always + // explicit, which is the form otlpmetricgrpc.WithEndpointURL takes: the + // exporter reads its transport off the scheme. Empty when the cell has no + // collector. + URL string + // Endpoint is the same address as host:port. wool's tracer takes it in this + // form and has no option that reads a URL. Empty when the cell has no + // collector. Endpoint string // Insecure is true when the endpoint was http://, which the mesh protects. - // It is derived from the endpoint's scheme and never assumed. + // wool's tracer needs it spelled out; it is derived from the endpoint's + // scheme and never assumed. Insecure bool - // AbsentReason says why the cell has no collector. Empty when Endpoint is set. + // AbsentReason says why the cell has no collector. Empty when URL is set. AbsentReason string + // Ignored names the key the group carried that has no meaning for the state + // that decided: the absent reason under `available`, the endpoint under + // `absent`. Empty when the group carried nothing extra. + Ignored string } // Available reports whether the cell has a collector to export to. @@ -74,6 +92,23 @@ func (d telemetryDestination) Available() bool { return d.Endpoint != "" } +// IgnoredNotice is the one line to log, once, when the group carried a key the +// state does not use. It is empty otherwise. It names the key and never its +// value: a leftover endpoint may carry credentials, and a leftover reason is +// free text. +func (d telemetryDestination) IgnoredNotice() string { + if d.Ignored == "" { + return "" + } + state := telemetryStateAbsent + if d.Available() { + state = telemetryStateAvailable + } + return fmt.Sprintf( + "observability: %s is set but %s is `%s`, which decides; %s is ignored (a leftover from a lower layer of the group)", + d.Ignored, telemetryStateKey, state, d.Ignored) +} + // observabilityValue reads one key of the `observability` group through the // SDK. It deliberately has no fallback to a bare process variable: an ambient // OTEL_EXPORTER_OTLP_ENDPOINT in a shell or a pod is conventionally the @@ -112,13 +147,11 @@ func resolveTelemetryDestination(read func(key string) string) (telemetryDestina "observability: %s is %q in the `observability` configuration group; it must be `%s` or `%s`", telemetryStateKey, state, telemetryStateAvailable, telemetryStateAbsent) } - if endpoint != "" && reason != "" { - return telemetryDestination{}, fmt.Errorf( - "observability: the `observability` configuration group sets both %s and %s; "+ - "a cell either has a collector or says why it has none", - telemetryEndpointKey, telemetryAbsentReasonKey) - } + // The state decides: the key that belongs to the other state is a leftover + // of a lower layer and is neither read nor refused. The endpoint under + // `absent` is not even parsed, so a stale one cannot stop a cell that says + // it has no collector. if state == telemetryStateAbsent { if reason == "" { return telemetryDestination{}, fmt.Errorf( @@ -126,7 +159,11 @@ func resolveTelemetryDestination(read func(key string) string) (telemetryDestina "an absent collector must say why", telemetryStateKey, telemetryStateAbsent, telemetryAbsentReasonKey) } - return telemetryDestination{AbsentReason: reason}, nil + destination := telemetryDestination{AbsentReason: reason} + if endpoint != "" { + destination.Ignored = telemetryEndpointKey + } + return destination, nil } if endpoint == "" { @@ -134,39 +171,46 @@ func resolveTelemetryDestination(read func(key string) string) (telemetryDestina "observability: %s is `%s` but %s is not set in the `observability` configuration group", telemetryStateKey, telemetryStateAvailable, telemetryEndpointKey) } - hostPort, insecure, err := parseCollectorEndpoint(endpoint) + origin, hostPort, insecure, err := parseCollectorEndpoint(endpoint) if err != nil { return telemetryDestination{}, fmt.Errorf("observability: %s: %w", telemetryEndpointKey, err) } - return telemetryDestination{Endpoint: hostPort, Insecure: insecure}, nil + destination := telemetryDestination{URL: origin, Endpoint: hostPort, Insecure: insecure} + if reason != "" { + destination.Ignored = telemetryAbsentReasonKey + } + return destination, nil } -// parseCollectorEndpoint turns the delivered URL into the host:port the OTLP -// gRPC exporters take, and reads the transport off its scheme. The URL is never -// echoed back: a malformed one may carry credentials. -func parseCollectorEndpoint(raw string) (hostPort string, insecure bool, err error) { +// parseCollectorEndpoint turns the delivered URL into the two forms the +// exporters take — the origin with an explicit port, and host:port — and reads +// the transport off its scheme. The URL is never echoed back: a malformed one +// may carry credentials. +func parseCollectorEndpoint(raw string) (origin, hostPort string, insecure bool, err error) { parsed, parseErr := url.Parse(raw) if parseErr != nil { - return "", false, errors.New("is not a URL; expected http://host:port or https://host:port") + return "", "", false, errors.New("is not a URL; expected http://host:port or https://host:port") } switch parsed.Scheme { case "http": insecure = true case "https": default: - return "", false, fmt.Errorf("must start with http:// or https://, got scheme %q", parsed.Scheme) + return "", "", false, fmt.Errorf("must start with http:// or https://, got scheme %q", parsed.Scheme) } if parsed.Hostname() == "" { - return "", false, errors.New("has no host") + return "", "", false, errors.New("has no host") } if parsed.User != nil { - return "", false, errors.New("must not carry credentials") + return "", "", false, errors.New("must not carry credentials") } // OTLP/gRPC addresses a collector, not a path: a path or a query is the // OTLP/HTTP form (/v1/traces), which this exporter does not speak. if (parsed.Path != "" && parsed.Path != "/") || parsed.RawQuery != "" || parsed.Fragment != "" { - return "", false, errors.New("must be the collector's origin only, with no path, query or fragment") + return "", "", false, errors.New("must be the collector's origin only, with no path, query or fragment") } + // The port is made explicit because gRPC would otherwise dial 443 whatever + // the scheme, and a plaintext http:// collector with no port is not on 443. port := parsed.Port() if port == "" { port = "80" @@ -174,7 +218,8 @@ func parseCollectorEndpoint(raw string) (hostPort string, insecure bool, err err port = "443" } } - return net.JoinHostPort(parsed.Hostname(), port), insecure, nil + hostPort = net.JoinHostPort(parsed.Hostname(), port) + return parsed.Scheme + "://" + hostPort, hostPort, insecure, nil } // boundedLine collapses a free-text value to one bounded line. diff --git a/module/services/auth-gateway/code/telemetry_destination_test.go b/module/services/auth-gateway/code/telemetry_destination_test.go index 4d81a7698..fe4f32e4a 100644 --- a/module/services/auth-gateway/code/telemetry_destination_test.go +++ b/module/services/auth-gateway/code/telemetry_destination_test.go @@ -29,32 +29,39 @@ func TestTelemetryDestinationAvailableExportsToTheEndpoint(t *testing.T) { })) require.NoError(t, err) require.True(t, destination.Available()) + require.Equal(t, collector, destination.URL, "the metrics exporter reads its transport off this URL") require.Equal(t, "otel-collector.otel-collector.svc.cluster.local:4317", destination.Endpoint) require.True(t, destination.Insecure, "http:// is plaintext on the wire; the mesh supplies mTLS") require.Empty(t, destination.AbsentReason) + require.Empty(t, destination.Ignored) + require.Empty(t, destination.IgnoredNotice()) } -// The transport comes from the endpoint's scheme, never from a constant. +// The transport comes from the endpoint's scheme, never from a constant, and +// the URL handed to the metrics exporter always carries an explicit port: gRPC +// would otherwise dial 443 whatever the scheme. func TestTelemetryDestinationReadsTransportFromTheScheme(t *testing.T) { for endpoint, want := range map[string]struct { + url string hostPort string insecure bool }{ - "http://collector.example:4317": {"collector.example:4317", true}, - "https://collector.example:4317": {"collector.example:4317", false}, - "https://collector.example": {"collector.example:443", false}, - "http://collector.example": {"collector.example:80", true}, - "http://collector.example:4317/": {"collector.example:4317", true}, - " http://collector.example:4317 ": {"collector.example:4317", true}, - "http://[2001:db8::1]:4317": {"[2001:db8::1]:4317", true}, - "https://10.0.0.7:4317": {"10.0.0.7:4317", false}, - "http://otel-collector.ns.svc:4317/": {"otel-collector.ns.svc:4317", true}, + "http://collector.example:4317": {"http://collector.example:4317", "collector.example:4317", true}, + "https://collector.example:4317": {"https://collector.example:4317", "collector.example:4317", false}, + "https://collector.example": {"https://collector.example:443", "collector.example:443", false}, + "http://collector.example": {"http://collector.example:80", "collector.example:80", true}, + "http://collector.example:4317/": {"http://collector.example:4317", "collector.example:4317", true}, + " http://collector.example:4317 ": {"http://collector.example:4317", "collector.example:4317", true}, + "http://[2001:db8::1]:4317": {"http://[2001:db8::1]:4317", "[2001:db8::1]:4317", true}, + "https://10.0.0.7:4317": {"https://10.0.0.7:4317", "10.0.0.7:4317", false}, + "http://otel-collector.ns.svc:4317/": {"http://otel-collector.ns.svc:4317", "otel-collector.ns.svc:4317", true}, } { destination, err := resolveTelemetryDestination(group(map[string]string{ "TELEMETRY_STATE": "available", "OTEL_EXPORTER_OTLP_ENDPOINT": endpoint, })) require.NoError(t, err, endpoint) + require.Equal(t, want.url, destination.URL, endpoint) require.Equal(t, want.hostPort, destination.Endpoint, endpoint) require.Equal(t, want.insecure, destination.Insecure, endpoint) } @@ -69,8 +76,11 @@ func TestTelemetryDestinationAbsentCarriesTheReason(t *testing.T) { })) require.NoError(t, err) require.False(t, destination.Available()) + require.Empty(t, destination.URL) require.Empty(t, destination.Endpoint) require.Equal(t, "This cell's bucket has no ops cell, so no collector is rendered.", destination.AbsentReason) + require.Empty(t, destination.Ignored) + require.Empty(t, destination.IgnoredNotice()) } // The reason is operator-written free text that reaches a log line. @@ -86,7 +96,9 @@ func TestTelemetryDestinationAbsentReasonIsOneBoundedLine(t *testing.T) { } // The third outcome: anything else refuses to start, naming what is wrong. -// Configuration that did not arrive is never read as "no collector". +// Configuration that did not arrive is never read as "no collector". A key the +// state does not use is not a refusal — it is ignored, below — but a key the +// state requires, missing, always is. func TestTelemetryDestinationRefusesEverythingElse(t *testing.T) { for name, tc := range map[string]struct { values map[string]string @@ -136,20 +148,6 @@ func TestTelemetryDestinationRefusesEverythingElse(t *testing.T) { values: map[string]string{"TELEMETRY_STATE": "absent", "OTEL_EXPORTER_OTLP_ENDPOINT": collector}, want: "TELEMETRY_ABSENT_REASON is not set", }, - "available with both an endpoint and a reason": { - values: map[string]string{ - "TELEMETRY_STATE": "available", "OTEL_EXPORTER_OTLP_ENDPOINT": collector, - "TELEMETRY_ABSENT_REASON": "no collector", - }, - want: "sets both OTEL_EXPORTER_OTLP_ENDPOINT and TELEMETRY_ABSENT_REASON", - }, - "absent with both an endpoint and a reason": { - values: map[string]string{ - "TELEMETRY_STATE": "absent", "OTEL_EXPORTER_OTLP_ENDPOINT": collector, - "TELEMETRY_ABSENT_REASON": "no collector", - }, - want: "sets both OTEL_EXPORTER_OTLP_ENDPOINT and TELEMETRY_ABSENT_REASON", - }, "endpoint without a scheme": { values: map[string]string{"TELEMETRY_STATE": "available", "OTEL_EXPORTER_OTLP_ENDPOINT": "otel-collector.ns.svc:4317"}, want: "must start with http:// or https://", @@ -181,6 +179,129 @@ func TestTelemetryDestinationRefusesEverythingElse(t *testing.T) { } } +// The state decides. The cell's values override the module's local defaults one +// key at a time, so a deployed producer sees `available` and an endpoint from the +// cell beside the reason the local profile left behind. That is not a +// contradiction to refuse: the reason is ignored, and said so once. +func TestTelemetryDestinationAvailableIgnoresALeftoverReason(t *testing.T) { + destination, err := resolveTelemetryDestination(group(map[string]string{ + "TELEMETRY_STATE": "available", + "OTEL_EXPORTER_OTLP_ENDPOINT": collector, + "TELEMETRY_ABSENT_REASON": "A local run has no cell collector.", + })) + require.NoError(t, err) + require.True(t, destination.Available()) + require.Equal(t, collector, destination.URL) + require.Empty(t, destination.AbsentReason, "a leftover reason must not turn an available collector into an absent one") + require.Equal(t, "TELEMETRY_ABSENT_REASON", destination.Ignored) + notice := destination.IgnoredNotice() + require.Contains(t, notice, "TELEMETRY_ABSENT_REASON is set but TELEMETRY_STATE is `available`") + require.NotContains(t, notice, "A local run", "the notice names the key, never its free-text value") +} + +// ... and the other way round: a cell that says it has no collector is not +// refused for an endpoint a lower layer left behind, and that endpoint is not +// even parsed. +func TestTelemetryDestinationAbsentIgnoresALeftoverEndpoint(t *testing.T) { + for name, leftover := range map[string]string{ + "a usable endpoint": collector, + "a malformed endpoint": "otel-collector.ns.svc:4317", + "an endpoint with a user": "http://user:hunter2@collector.example:4317", + "an unreadable endpoint": "http://[::1", + } { + t.Run(name, func(t *testing.T) { + destination, err := resolveTelemetryDestination(group(map[string]string{ + "TELEMETRY_STATE": "absent", + "TELEMETRY_ABSENT_REASON": "This cell has no ops cell.", + "OTEL_EXPORTER_OTLP_ENDPOINT": leftover, + })) + require.NoError(t, err) + require.False(t, destination.Available(), "a leftover endpoint must not turn an absent collector into an available one") + require.Empty(t, destination.URL) + require.Empty(t, destination.Endpoint) + require.Equal(t, "This cell has no ops cell.", destination.AbsentReason) + require.Equal(t, "OTEL_EXPORTER_OTLP_ENDPOINT", destination.Ignored) + notice := destination.IgnoredNotice() + require.Contains(t, notice, "OTEL_EXPORTER_OTLP_ENDPOINT is set but TELEMETRY_STATE is `absent`") + require.NotContains(t, notice, "hunter2", "the notice names the key, never its value") + }) + } +} + +// A blank leftover is nothing left over. +func TestTelemetryDestinationBlankLeftoversAreNotNoticed(t *testing.T) { + available, err := resolveTelemetryDestination(group(map[string]string{ + "TELEMETRY_STATE": "available", "OTEL_EXPORTER_OTLP_ENDPOINT": collector, "TELEMETRY_ABSENT_REASON": " \n ", + })) + require.NoError(t, err) + require.Empty(t, available.IgnoredNotice()) + + absent, err := resolveTelemetryDestination(group(map[string]string{ + "TELEMETRY_STATE": "absent", "TELEMETRY_ABSENT_REASON": "no collector", "OTEL_EXPORTER_OTLP_ENDPOINT": " ", + })) + require.NoError(t, err) + require.Empty(t, absent.IgnoredNotice()) +} + +// layered is what the platform hands a workload: each layer overrides the one +// below it key by key, so a key a higher layer does not set survives from the +// lower one. +func layered(layers ...map[string]string) func(string) string { + merged := map[string]string{} + for _, layer := range layers { + for key, value := range layer { + merged[key] = value + } + } + return group(merged) +} + +// The case this exists for: the module's local profile defaults to `absent` with +// a reason, and a deployed cell overrides the state and the endpoint but not the +// reason. The cell's answer wins, whichever way it points. +func TestTelemetryDestinationTheCellOverridesTheModuleDefaults(t *testing.T) { + moduleDefault := map[string]string{ + "TELEMETRY_STATE": "absent", + "TELEMETRY_ABSENT_REASON": "A local run has no cell collector.", + } + + t.Run("a cell with a collector overrides the local absent default", func(t *testing.T) { + destination, err := resolveTelemetryDestination(layered(moduleDefault, map[string]string{ + "TELEMETRY_STATE": "available", + "OTEL_EXPORTER_OTLP_ENDPOINT": collector, + })) + require.NoError(t, err) + require.True(t, destination.Available()) + require.Equal(t, collector, destination.URL) + require.Empty(t, destination.AbsentReason) + require.Equal(t, "TELEMETRY_ABSENT_REASON", destination.Ignored) + }) + + t.Run("a cell with no collector, over a default that had an endpoint, is absent", func(t *testing.T) { + destination, err := resolveTelemetryDestination(layered( + map[string]string{"TELEMETRY_STATE": "available", "OTEL_EXPORTER_OTLP_ENDPOINT": collector}, + map[string]string{"TELEMETRY_STATE": "absent", "TELEMETRY_ABSENT_REASON": "This cell has no ops cell."}, + )) + require.NoError(t, err) + require.False(t, destination.Available()) + require.Equal(t, "This cell has no ops cell.", destination.AbsentReason) + require.Equal(t, "OTEL_EXPORTER_OTLP_ENDPOINT", destination.Ignored) + }) + + t.Run("a cell that overrides only the state still has to supply what it requires", func(t *testing.T) { + _, err := resolveTelemetryDestination(layered(moduleDefault, map[string]string{"TELEMETRY_STATE": "available"})) + require.ErrorContains(t, err, "OTEL_EXPORTER_OTLP_ENDPOINT is not set", + "the module default's reason is not an endpoint") + + _, err = resolveTelemetryDestination(layered( + map[string]string{"TELEMETRY_STATE": "available", "OTEL_EXPORTER_OTLP_ENDPOINT": collector}, + map[string]string{"TELEMETRY_STATE": "absent"}, + )) + require.ErrorContains(t, err, "TELEMETRY_ABSENT_REASON is not set", + "the module default's endpoint is not a reason") + }) +} + // A refusal quotes no part of an endpoint, which may carry credentials. func TestTelemetryDestinationRefusalDoesNotEchoCredentials(t *testing.T) { _, err := resolveTelemetryDestination(group(map[string]string{ @@ -228,6 +349,22 @@ func TestConfiguredTelemetryDestinationReadsTheObservabilityGroup(t *testing.T) require.Equal(t, "a local run has no cell collector", destination.AbsentReason) } +// Through the SDK's lookup, with the keys exactly as a deployed cell delivers +// them after the layers merge: the cell's state and endpoint, and the reason the +// module's local profile left behind. +func TestConfiguredTelemetryDestinationStateDecidesAcrossLayers(t *testing.T) { + const prefix = "CODEFLY__WORKSPACE_CONFIGURATION__OBSERVABILITY__" + t.Setenv(prefix+"TELEMETRY_STATE", "available") + t.Setenv(prefix+"OTEL_EXPORTER_OTLP_ENDPOINT", collector) + t.Setenv(prefix+"TELEMETRY_ABSENT_REASON", "A local run has no cell collector.") + + destination, err := configuredTelemetryDestination() + require.NoError(t, err) + require.True(t, destination.Available()) + require.Equal(t, collector, destination.URL) + require.NotEmpty(t, destination.IgnoredNotice()) +} + // service.name is never a literal typed here: the environment wins, then the // service's own Codefly identity. func TestTelemetryResourceServiceName(t *testing.T) { diff --git a/module/services/auth-gateway/code/telemetry_metrics.go b/module/services/auth-gateway/code/telemetry_metrics.go index bcebfed60..114e1e107 100644 --- a/module/services/auth-gateway/code/telemetry_metrics.go +++ b/module/services/auth-gateway/code/telemetry_metrics.go @@ -2,6 +2,7 @@ package main import ( "context" + "errors" "net/http" runtimemetrics "runtime/metrics" "time" @@ -22,17 +23,21 @@ type otelMetrics struct { } func enableOTELMetrics(ctx context.Context, destination telemetryDestination) (*otelMetrics, error) { - options := []otlpmetricgrpc.Option{otlpmetricgrpc.WithEndpoint(destination.Endpoint)} - if destination.Insecure { - // http://: plaintext on the wire, because the mesh supplies mTLS. An - // https:// endpoint takes the exporter's default, TLS. - options = append(options, otlpmetricgrpc.WithInsecure()) + if destination.URL == "" { + // WithEndpointURL("") would fall back to the exporter's own default, + // localhost:4317: a process that exports to nowhere and reports success. + return nil, errors.New("telemetry: no collector URL to export metrics to") } - exporter, err := otlpmetricgrpc.New(ctx, options...) + // The resource comes before the exporter, so a failure here leaves nothing + // to shut down: an exporter holds a client connection that only Shutdown + // releases. + res, err := currentTelemetryResource(ctx) if err != nil { return nil, err } - res, err := currentTelemetryResource(ctx) + // The exporter reads the transport off the URL's scheme itself: http:// is + // plaintext on the wire, because the mesh supplies mTLS, and https:// is TLS. + exporter, err := otlpmetricgrpc.New(ctx, otlpmetricgrpc.WithEndpointURL(destination.URL)) if err != nil { return nil, err } diff --git a/module/services/auth-gateway/code/telemetry_metrics_test.go b/module/services/auth-gateway/code/telemetry_metrics_test.go index 701c6c851..692e9eb8c 100644 --- a/module/services/auth-gateway/code/telemetry_metrics_test.go +++ b/module/services/auth-gateway/code/telemetry_metrics_test.go @@ -41,7 +41,7 @@ func TestAuthGatewayTelemetryExportsGatewayAndGRPCRED(t *testing.T) { t.Setenv("OTEL_RESOURCE_ATTRIBUTES", "service.instance.id=auth-test-instance") t.Setenv("OTEL_SERVICE_NAME", "named-by-the-platform") previousMeterProvider := otel.GetMeterProvider() - metrics, err := enableOTELMetrics(t.Context(), telemetryDestination{Endpoint: endpoint, Insecure: true}) + metrics, err := enableOTELMetrics(t.Context(), availableAt(t, "http", endpoint)) require.NoError(t, err) require.NotNil(t, metrics) t.Cleanup(func() { @@ -134,6 +134,65 @@ func authFloat64SumValue(request *collectormetricsv1.ExportMetricsServiceRequest return value } +// availableAt resolves the destination the way the process does, from the +// endpoint the platform would deliver, so these tests hand the exporter the URL +// it really receives rather than a hand-built struct. +func availableAt(t *testing.T, scheme, hostPort string) telemetryDestination { + t.Helper() + destination, err := resolveTelemetryDestination(group(map[string]string{ + "TELEMETRY_STATE": "available", + "OTEL_EXPORTER_OTLP_ENDPOINT": scheme + "://" + hostPort, + })) + require.NoError(t, err) + return destination +} + +// The exporter reads its transport off the URL's scheme, so an https:// URL is +// TLS and cannot talk to a plaintext receiver. A constant WithInsecure() would +// have let it. +func TestEnableOTELMetricsDerivesTransportFromTheURLScheme(t *testing.T) { + endpoint, capture := startAuthMetricCapture(t) + previousMeterProvider := otel.GetMeterProvider() + metrics, err := enableOTELMetrics(t.Context(), availableAt(t, "https", endpoint)) + require.NoError(t, err) + require.NotNil(t, metrics) + t.Cleanup(func() { + shutdownCtx, cancel := context.WithTimeout(context.Background(), time.Second) + defer cancel() + _ = metrics.Shutdown(shutdownCtx) // the flush it attempts is the failure under test + otel.SetMeterProvider(previousMeterProvider) + }) + + flushCtx, cancel := context.WithTimeout(t.Context(), time.Second) + defer cancel() + require.Error(t, metrics.provider.ForceFlush(flushCtx), "a TLS client cannot export to a plaintext receiver") + select { + case <-capture.requests: + t.Fatal("an https:// destination reached a plaintext receiver") + default: + } +} + +// A destination with no collector URL is refused. WithEndpointURL("") would +// otherwise fall back to the exporter's own default, localhost:4317. +func TestEnableOTELMetricsRefusesADestinationWithoutAURL(t *testing.T) { + metrics, err := enableOTELMetrics(t.Context(), telemetryDestination{AbsentReason: "no collector"}) + require.ErrorContains(t, err, "no collector URL") + require.Nil(t, metrics) +} + +// A resource that cannot be built fails the whole call, and the call builds it +// before it creates the exporter: there is no exporter, and so no client +// connection, left running behind the failure. +func TestEnableOTELMetricsFailsWhenTheResourceCannotBeBuilt(t *testing.T) { + t.Setenv("OTEL_SERVICE_NAME", "") + t.Setenv("OTEL_RESOURCE_ATTRIBUTES", "an-attribute-with-no-value") + endpoint, _ := startAuthMetricCapture(t) + metrics, err := enableOTELMetrics(t.Context(), availableAt(t, "http", endpoint)) + require.Error(t, err) + require.Nil(t, metrics) +} + func startAuthMetricCapture(t *testing.T) (string, *authMetricCapture) { t.Helper() listener, err := net.Listen("tcp", "127.0.0.1:0") From 2991e7b8a40424c2a7cb7955cbb52ff15bb32aa1 Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Wed, 7 Oct 2026 11:07:54 -0400 Subject: [PATCH 46/69] refactor(accounts,auth-gateway): resolve the listener ports without a default network (Refs #1022) Both services read their own listener ports through `WithDefaultNetwork()`, which makes the SDK answer an endpoint that does not resolve with a default network instance instead of an error. The next SDK release deletes that method, and its own test holds it deleted ("legacy means delete, not deprecate"): a port the runtime did not inject is an error, never a default, and a service that serves on a port nobody gave it is reachable from nowhere and reports healthy. This lands before the dependency bump so the bump is only a bump. - auth-gateway: the gRPC port is resolved with ResolveNetworkInstance and the process refuses to start without it, exactly as it already does for its REST port on the next line. - accounts: the gRPC port is required, for the same reason: it is what the process is. REST and Connect were optional in code but, behind the default network, never actually absent; they now resolve through one helper that, when an endpoint does not resolve, logs it by name and serves no listener for it, which is how this file already treats the module authority endpoint. The comment above that endpoint no longer needs to say it is the one query that avoids the default network. Behaviour is unchanged wherever the runtime injects the endpoint (auth-gateway already refused to start on a missing REST carrier, so that path is exercised). What changes is only the broken case: a missing gRPC endpoint is a refusal to start instead of a listener on a default port, and a missing REST or Connect endpoint is a logged error and no listener instead of a listener on a default port. Not exercised against a real cell: only a simulated deployed process, with the carriers set by hand. Classified as a fix at the place that owns the behaviour: these two call sites asked the SDK to invent a value, and the SDK stopped doing that. accounts' main.go still carries the template's "do not edit" banner, but it is authored: it already holds the module authority endpoint block and `codefly sync service` leaves it untouched. Co-Authored-By: Claude Opus 5.5 --- module/services/accounts/code/main.go | 40 +++++++++++++++++------ module/services/auth-gateway/code/main.go | 8 ++++- 2 files changed, 37 insertions(+), 11 deletions(-) diff --git a/module/services/accounts/code/main.go b/module/services/accounts/code/main.go index 730776405..3a20c6f1d 100644 --- a/module/services/accounts/code/main.go +++ b/module/services/accounts/code/main.go @@ -34,6 +34,22 @@ import ( codefly "github.com/codefly-dev/sdk-go" ) +// resolvedPort is the port of one of this service's own endpoints, or nil when +// it does not resolve. A nil port serves no listener for the endpoint and says +// so loudly; it does not substitute a default one. +func resolvedPort(ctx context.Context, api string) *uint16 { + instance, err := codefly.For(ctx).API(api).ResolveNetworkInstance() + if err != nil { + wool.Get(ctx).In("main").Error( + "the endpoint did not resolve; no listener is served for it", + wool.Field("endpoint", api), + wool.ErrField(err), + ) + return nil + } + return shared.Pointer(instance.Port) +} + type Clean func() type Work func(ctx context.Context) (Clean, error) @@ -55,14 +71,19 @@ func main() { defer codefly.CatchPanic(ctx) - config := &adapters.Configuration{ - EndpointGrpcPort: codefly.For(ctx).WithDefaultNetwork().API(standards.GRPC).NetworkInstance().Port, - } - if net := codefly.For(ctx).WithDefaultNetwork().API(standards.REST).NetworkInstance(); net != nil { - config.EndpointHttpPort = shared.Pointer(net.Port) + // The SDK resolves an endpoint from the carrier the runtime injected or, in a + // local run, from the workspace's endpoint map, and never invents a port: an + // address that does not resolve is an error here, not a default. + // + // gRPC is what this process is. Without it there is nothing to start. + grpcNet, grpcErr := codefly.For(ctx).API(standards.GRPC).ResolveNetworkInstance() + if grpcErr != nil || grpcNet == nil { + panic(fmt.Sprintf("Codefly gRPC endpoint is unavailable: %v", grpcErr)) } - if net := codefly.For(ctx).WithDefaultNetwork().API(standards.CONNECT).NetworkInstance(); net != nil { - config.EndpointConnectPort = shared.Pointer(net.Port) + config := &adapters.Configuration{ + EndpointGrpcPort: grpcNet.Port, + EndpointHttpPort: resolvedPort(ctx, standards.REST), + EndpointConnectPort: resolvedPort(ctx, standards.CONNECT), } // The module authority endpoint is declared in service.codefly.yaml, and a // topology that omits it — or gives it a port another listener already @@ -76,9 +97,8 @@ func main() { // accounts would crashloop and take the tenant surfaces, the internal tier // and every service that authenticates through them down with it. // - // This query deliberately does not use WithDefaultNetwork(), so resolution - // never invents a port; the only outcomes are the injected address or this - // error path. + // Like the two above, this query never invents a port: the only outcomes are + // the injected address or this error path. if authority, resolveErr := codefly.For(ctx).Endpoint(business.ModuleAuthorityEndpoint).API(standards.GRPC).ResolveNetworkInstance(); resolveErr != nil { wool.Get(ctx).In("main").Error( "the module authority endpoint did not resolve; no module authority listener is served", diff --git a/module/services/auth-gateway/code/main.go b/module/services/auth-gateway/code/main.go index 4ea768fe6..77e0471c7 100644 --- a/module/services/auth-gateway/code/main.go +++ b/module/services/auth-gateway/code/main.go @@ -41,7 +41,13 @@ func main() { panic(err) } - grpcPort := codefly.For(ctx).WithDefaultNetwork().API(standards.GRPC).NetworkInstance().Port + // The SDK resolves an endpoint from the carrier the runtime injected or, in a + // local run, from the workspace's endpoint map, and never invents a port. + grpcNet, grpcNetErr := codefly.For(ctx).API(standards.GRPC).ResolveNetworkInstance() + if grpcNetErr != nil || grpcNet == nil { + panic(fmt.Sprintf("Codefly gRPC endpoint is unavailable: %v", grpcNetErr)) + } + grpcPort := grpcNet.Port var httpPort uint16 httpNet, httpNetErr := codefly.For(ctx).API(standards.REST).ResolveNetworkInstance() if httpNetErr != nil || httpNet == nil { From 2720aa815a72bc9f546a25b05aa46f53d97292ab Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Wed, 7 Oct 2026 11:08:38 -0400 Subject: [PATCH 47/69] chore(deps): bump codefly core to v0.13.0 and sdk-go to v0.3.1 (Refs #1022) Core v0.13.0 carries the log/trace correlation that #1022 waits on, so accounts and auth-gateway now consume it. Each Go module was moved with the repository's bump verb, `codefly agent deps --dir --pin v0.13.0` (which pins every lock the module owns and verifies the standalone build), and `go mod tidy` leaves every module unchanged afterwards. Modules that pin core and moved: the root module agent, `module/tools`, accounts, auth-gateway, and `qualification/module-authority` and the store migrator, which only carry it indirectly. `libraries/source-read-sdk/go` does not depend on core. sdk-go moves with it, v0.2.1-0.20260926212413 -> v0.3.1, in accounts, auth-gateway and the store migrator, because it is not optional: the old sdk-go uses `resources.VisibilityExternal`, which core removed before v0.13.0, so no sdk-go older than v0.3.1 builds against any core this change could pin. v0.3.1 itself requires core v0.13.0, which makes v0.13.0 the pair upstream tested. Its one source-level break for this repository, the deleted `WithDefaultNetwork`, is handled in the commit before this one. Why not v0.14.0, which was the target: v0.14.0 builds, but it refuses a module interface or endpoint that authors `allow-modules`, since an allow-list is derived from the consumers' declared dependencies and never written by the module it grants. This module authors `allow-modules: ["*"]` in `module.codefly.yaml` and in the accounts, auth-gateway and vault manifests, so `TestGeneratedCodeflyAndNetworkManifestsParseStrictly` fails to load them under v0.14.0 and passes under v0.13.0. Moving those manifests to derived allow-lists is its own change, with its own render consequences for composing workspaces, and is not made here. `sdk-go/workcontext` stays on its current pseudo-version in accounts and auth-gateway. The newer workcontext replaces the signing API this repository imports (`WorkContextSigner`, `WorkContextVerifier`, `WorkContextToken` and the header constants), a migration of its own that touches most of accounts' Work Context handling. Tested with the generated telemetry stubs present: `go vet` and `go test` in accounts (pure suite), auth-gateway, `module/tools`, the root module and `qualification/module-authority`, `go vet` in the store migrator and the source-read SDK; every generator and `codefly sync` rerun leaves the tree unchanged; the contracts check with the CI-pinned CLI 0.1.171. Not exercised: `codefly run` of the graph, a deployed cell, or the accounts database suites. Co-Authored-By: Claude Opus 5.5 --- go.mod | 2 +- go.sum | 4 ++-- module/services/accounts/code/go.mod | 4 ++-- module/services/accounts/code/go.sum | 8 ++++---- module/services/auth-gateway/code/go.mod | 4 ++-- module/services/auth-gateway/code/go.sum | 8 ++++---- module/services/store/code/go.mod | 4 ++-- module/services/store/code/go.sum | 8 ++++---- module/tools/go.mod | 2 +- module/tools/go.sum | 4 ++-- qualification/module-authority/go.mod | 2 +- qualification/module-authority/go.sum | 4 ++-- 12 files changed, 27 insertions(+), 27 deletions(-) diff --git a/go.mod b/go.mod index 575a0b1cd..12e08b125 100644 --- a/go.mod +++ b/go.mod @@ -3,7 +3,7 @@ module github.com/codefly-dev/agents/modules/saas-starter go 1.27.0 require ( - github.com/codefly-dev/core v0.5.11-0.20260926231217-d481a2d49683 + github.com/codefly-dev/core v0.13.0 gopkg.in/yaml.v3 v3.0.1 ) diff --git a/go.sum b/go.sum index d1e38c6b3..d4703ab21 100644 --- a/go.sum +++ b/go.sum @@ -28,8 +28,8 @@ github.com/cheggaaa/pb/v3 v3.2.1 h1:aprZbFRG+B7+ug76S8QZ6Y1PW168UHzOmbC3wa+aU6I= github.com/cheggaaa/pb/v3 v3.2.1/go.mod h1:U9hSVxoKqJrZIE3PkFG1xXXNaK/Ilzg+EF/scpXzEdw= github.com/clipperhouse/uax29/v2 v2.7.0 h1:+gs4oBZ2gPfVrKPthwbMzWZDaAFPGYK72F0NJv2v7Vk= github.com/clipperhouse/uax29/v2 v2.7.0/go.mod h1:EFJ2TJMRUaplDxHKj1qAEhCtQPW2tJSwu5BF98AuoVM= -github.com/codefly-dev/core v0.5.11-0.20260926231217-d481a2d49683 h1:/vzymsu5B9NWwdhWRtlU+68VcVEVDmeZNChRCXAUwlE= -github.com/codefly-dev/core v0.5.11-0.20260926231217-d481a2d49683/go.mod h1:SVF+t040X+9pAXAqBnNE/JkXwmw9peVuKnHKFSOzyzk= +github.com/codefly-dev/core v0.13.0 h1:FiphttBZRrKXEWoCZD7JpUB6oVuBCoqtAc06QeNdYWY= +github.com/codefly-dev/core v0.13.0/go.mod h1:rp4YZkCJEOufliWQAgK3sv8YlO7Xcva7+cN7hIlAR0I= github.com/codefly-dev/gortk v0.2.0 h1:7bOlS5valYz2zil+fZctQNcPCYBcPj86abcw9N8h1hQ= github.com/codefly-dev/gortk v0.2.0/go.mod h1:dDWUMFgAP063OCGhTEpV7FFUj9+zTcxxO/nV80VKEwg= github.com/containerd/errdefs v1.0.0 h1:tg5yIfIlQIrxYtu9ajqY42W3lpS19XqdxRQeEwYG8PI= diff --git a/module/services/accounts/code/go.mod b/module/services/accounts/code/go.mod index a8374adaf..d18e4ad9a 100644 --- a/module/services/accounts/code/go.mod +++ b/module/services/accounts/code/go.mod @@ -8,9 +8,9 @@ require ( connectrpc.com/connect v1.21.0 connectrpc.com/otelconnect v0.9.0 github.com/codefly-dev/cli/contracts/deployment v0.0.0-20261006151622-5824cf5342fc - github.com/codefly-dev/core v0.5.11-0.20260926231217-d481a2d49683 + github.com/codefly-dev/core v0.13.0 github.com/codefly-dev/interface-cache/go/cache v0.2.2-0.20260926205419-327a6e1d9aba - github.com/codefly-dev/sdk-go v0.2.1-0.20260926212413-b91ce91eb23e + github.com/codefly-dev/sdk-go v0.3.1 github.com/codefly-dev/sdk-go/workcontext v0.0.0-20260926212413-b91ce91eb23e github.com/codefly-dev/service-postgres/libs/go v0.0.0-20260929200903-bae6c31c7211 github.com/codefly-dev/service-redis/cache v0.2.0 diff --git a/module/services/accounts/code/go.sum b/module/services/accounts/code/go.sum index bb1012c57..3d200f682 100644 --- a/module/services/accounts/code/go.sum +++ b/module/services/accounts/code/go.sum @@ -58,14 +58,14 @@ github.com/cloudflare/circl v1.6.3 h1:9GPOhQGF9MCYUeXyMYlqTR6a5gTrgR/fBLXvUgtVcg github.com/cloudflare/circl v1.6.3/go.mod h1:2eXP6Qfat4O/Yhh8BznvKnJ+uzEoTQ6jVKJRn81BiS4= github.com/codefly-dev/cli/contracts/deployment v0.0.0-20261006151622-5824cf5342fc h1:olDnVJ1A1Retn0z9A59YMwDRmDLgkV2y1lbl+HzNrj8= github.com/codefly-dev/cli/contracts/deployment v0.0.0-20261006151622-5824cf5342fc/go.mod h1:VWPYY3FH4te7SzdN2xl9odNpWidJ1Grl/Hjs1SNoEF8= -github.com/codefly-dev/core v0.5.11-0.20260926231217-d481a2d49683 h1:/vzymsu5B9NWwdhWRtlU+68VcVEVDmeZNChRCXAUwlE= -github.com/codefly-dev/core v0.5.11-0.20260926231217-d481a2d49683/go.mod h1:SVF+t040X+9pAXAqBnNE/JkXwmw9peVuKnHKFSOzyzk= +github.com/codefly-dev/core v0.13.0 h1:FiphttBZRrKXEWoCZD7JpUB6oVuBCoqtAc06QeNdYWY= +github.com/codefly-dev/core v0.13.0/go.mod h1:rp4YZkCJEOufliWQAgK3sv8YlO7Xcva7+cN7hIlAR0I= github.com/codefly-dev/gortk v0.2.0 h1:7bOlS5valYz2zil+fZctQNcPCYBcPj86abcw9N8h1hQ= github.com/codefly-dev/gortk v0.2.0/go.mod h1:dDWUMFgAP063OCGhTEpV7FFUj9+zTcxxO/nV80VKEwg= github.com/codefly-dev/interface-cache/go/cache v0.2.2-0.20260926205419-327a6e1d9aba h1:6nFaDYsLyA5x3WiGAk6pMI3c0naqxiPeaTHMqRPvbVs= github.com/codefly-dev/interface-cache/go/cache v0.2.2-0.20260926205419-327a6e1d9aba/go.mod h1:OQDNwz4rpEGfq7VC/FjvmQGlo9H8gH8fIDcZhKIe8iQ= -github.com/codefly-dev/sdk-go v0.2.1-0.20260926212413-b91ce91eb23e h1:V8YSUvAm0QQgHY9V0QmHiHuvf+beHnI10D6cJLWB2eI= -github.com/codefly-dev/sdk-go v0.2.1-0.20260926212413-b91ce91eb23e/go.mod h1:X0JAk/gnr6ZcjuPsM+5jvisqY4iyTC1SeVac93iWX8A= +github.com/codefly-dev/sdk-go v0.3.1 h1:ljKwrfHIiESV+R/MXO3a7XTdKwFNaY6vdw0+8YAo9g8= +github.com/codefly-dev/sdk-go v0.3.1/go.mod h1:vWrT2zXNMuCrctTAHMBYWuGRkfr0OP3IXATyuJ6iYvE= github.com/codefly-dev/sdk-go/workcontext v0.0.0-20260926212413-b91ce91eb23e h1:NlcOB8YN/372aNAGGkkFnT0du++qCFyilYVZLuqkogI= github.com/codefly-dev/sdk-go/workcontext v0.0.0-20260926212413-b91ce91eb23e/go.mod h1:BbqUHTJEnmH3oxYblfjS3Oh9AUw14YaEFVV4Y19yQPY= github.com/codefly-dev/service-postgres/libs/go v0.0.0-20260929200903-bae6c31c7211 h1:4KAPx8haCA/h4Kx5mB7D0rMriwLLKiOW9G9mMWuiQDw= diff --git a/module/services/auth-gateway/code/go.mod b/module/services/auth-gateway/code/go.mod index d9c26a210..25805db5f 100644 --- a/module/services/auth-gateway/code/go.mod +++ b/module/services/auth-gateway/code/go.mod @@ -4,8 +4,8 @@ go 1.27.0 require ( buf.build/gen/go/bufbuild/protovalidate/protocolbuffers/go v1.36.12-20260825204119-511051f7f437.2 - github.com/codefly-dev/core v0.5.11-0.20260926231217-d481a2d49683 - github.com/codefly-dev/sdk-go v0.2.1-0.20260926212413-b91ce91eb23e + github.com/codefly-dev/core v0.13.0 + github.com/codefly-dev/sdk-go v0.3.1 github.com/codefly-dev/sdk-go/workcontext v0.0.0-20260926212413-b91ce91eb23e github.com/envoyproxy/go-control-plane/envoy v1.39.0 github.com/golang-jwt/jwt/v5 v5.3.1 diff --git a/module/services/auth-gateway/code/go.sum b/module/services/auth-gateway/code/go.sum index 6be460d10..2e176b4a0 100644 --- a/module/services/auth-gateway/code/go.sum +++ b/module/services/auth-gateway/code/go.sum @@ -42,12 +42,12 @@ github.com/cloudflare/circl v1.6.3 h1:9GPOhQGF9MCYUeXyMYlqTR6a5gTrgR/fBLXvUgtVcg github.com/cloudflare/circl v1.6.3/go.mod h1:2eXP6Qfat4O/Yhh8BznvKnJ+uzEoTQ6jVKJRn81BiS4= github.com/cncf/xds/go v0.0.0-20260202195803-dba9d589def2 h1:aBangftG7EVZoUb69Os8IaYg++6uMOdKK83QtkkvJik= github.com/cncf/xds/go v0.0.0-20260202195803-dba9d589def2/go.mod h1:qwXFYgsP6T7XnJtbKlf1HP8AjxZZyzxMmc+Lq5GjlU4= -github.com/codefly-dev/core v0.5.11-0.20260926231217-d481a2d49683 h1:/vzymsu5B9NWwdhWRtlU+68VcVEVDmeZNChRCXAUwlE= -github.com/codefly-dev/core v0.5.11-0.20260926231217-d481a2d49683/go.mod h1:SVF+t040X+9pAXAqBnNE/JkXwmw9peVuKnHKFSOzyzk= +github.com/codefly-dev/core v0.13.0 h1:FiphttBZRrKXEWoCZD7JpUB6oVuBCoqtAc06QeNdYWY= +github.com/codefly-dev/core v0.13.0/go.mod h1:rp4YZkCJEOufliWQAgK3sv8YlO7Xcva7+cN7hIlAR0I= github.com/codefly-dev/gortk v0.2.0 h1:7bOlS5valYz2zil+fZctQNcPCYBcPj86abcw9N8h1hQ= github.com/codefly-dev/gortk v0.2.0/go.mod h1:dDWUMFgAP063OCGhTEpV7FFUj9+zTcxxO/nV80VKEwg= -github.com/codefly-dev/sdk-go v0.2.1-0.20260926212413-b91ce91eb23e h1:V8YSUvAm0QQgHY9V0QmHiHuvf+beHnI10D6cJLWB2eI= -github.com/codefly-dev/sdk-go v0.2.1-0.20260926212413-b91ce91eb23e/go.mod h1:X0JAk/gnr6ZcjuPsM+5jvisqY4iyTC1SeVac93iWX8A= +github.com/codefly-dev/sdk-go v0.3.1 h1:ljKwrfHIiESV+R/MXO3a7XTdKwFNaY6vdw0+8YAo9g8= +github.com/codefly-dev/sdk-go v0.3.1/go.mod h1:vWrT2zXNMuCrctTAHMBYWuGRkfr0OP3IXATyuJ6iYvE= github.com/codefly-dev/sdk-go/workcontext v0.0.0-20260926212413-b91ce91eb23e h1:NlcOB8YN/372aNAGGkkFnT0du++qCFyilYVZLuqkogI= github.com/codefly-dev/sdk-go/workcontext v0.0.0-20260926212413-b91ce91eb23e/go.mod h1:BbqUHTJEnmH3oxYblfjS3Oh9AUw14YaEFVV4Y19yQPY= github.com/containerd/errdefs v1.0.0 h1:tg5yIfIlQIrxYtu9ajqY42W3lpS19XqdxRQeEwYG8PI= diff --git a/module/services/store/code/go.mod b/module/services/store/code/go.mod index 60d84e43e..05f10868b 100644 --- a/module/services/store/code/go.mod +++ b/module/services/store/code/go.mod @@ -3,7 +3,7 @@ module store-migrator go 1.27.0 require ( - github.com/codefly-dev/sdk-go v0.2.1-0.20260926212413-b91ce91eb23e + github.com/codefly-dev/sdk-go v0.3.1 github.com/golang-migrate/migrate/v4 v4.19.1 github.com/lib/pq v1.12.3 ) @@ -16,7 +16,7 @@ require ( github.com/Masterminds/semver v1.5.0 // indirect github.com/Masterminds/semver/v3 v3.5.0 // indirect github.com/antlr4-go/antlr/v4 v4.13.1 // indirect - github.com/codefly-dev/core v0.5.11-0.20260926231217-d481a2d49683 // indirect + github.com/codefly-dev/core v0.13.0 // indirect github.com/go-openapi/analysis v1.0.0 // indirect github.com/go-openapi/errors v0.22.8 // indirect github.com/go-openapi/jsonpointer v1.0.1 // indirect diff --git a/module/services/store/code/go.sum b/module/services/store/code/go.sum index 99e8ef195..adfc358a3 100644 --- a/module/services/store/code/go.sum +++ b/module/services/store/code/go.sum @@ -30,12 +30,12 @@ github.com/cheggaaa/pb/v3 v3.2.1 h1:aprZbFRG+B7+ug76S8QZ6Y1PW168UHzOmbC3wa+aU6I= github.com/cheggaaa/pb/v3 v3.2.1/go.mod h1:U9hSVxoKqJrZIE3PkFG1xXXNaK/Ilzg+EF/scpXzEdw= github.com/clipperhouse/uax29/v2 v2.7.0 h1:+gs4oBZ2gPfVrKPthwbMzWZDaAFPGYK72F0NJv2v7Vk= github.com/clipperhouse/uax29/v2 v2.7.0/go.mod h1:EFJ2TJMRUaplDxHKj1qAEhCtQPW2tJSwu5BF98AuoVM= -github.com/codefly-dev/core v0.5.11-0.20260926231217-d481a2d49683 h1:/vzymsu5B9NWwdhWRtlU+68VcVEVDmeZNChRCXAUwlE= -github.com/codefly-dev/core v0.5.11-0.20260926231217-d481a2d49683/go.mod h1:SVF+t040X+9pAXAqBnNE/JkXwmw9peVuKnHKFSOzyzk= +github.com/codefly-dev/core v0.13.0 h1:FiphttBZRrKXEWoCZD7JpUB6oVuBCoqtAc06QeNdYWY= +github.com/codefly-dev/core v0.13.0/go.mod h1:rp4YZkCJEOufliWQAgK3sv8YlO7Xcva7+cN7hIlAR0I= github.com/codefly-dev/gortk v0.2.0 h1:7bOlS5valYz2zil+fZctQNcPCYBcPj86abcw9N8h1hQ= github.com/codefly-dev/gortk v0.2.0/go.mod h1:dDWUMFgAP063OCGhTEpV7FFUj9+zTcxxO/nV80VKEwg= -github.com/codefly-dev/sdk-go v0.2.1-0.20260926212413-b91ce91eb23e h1:V8YSUvAm0QQgHY9V0QmHiHuvf+beHnI10D6cJLWB2eI= -github.com/codefly-dev/sdk-go v0.2.1-0.20260926212413-b91ce91eb23e/go.mod h1:X0JAk/gnr6ZcjuPsM+5jvisqY4iyTC1SeVac93iWX8A= +github.com/codefly-dev/sdk-go v0.3.1 h1:ljKwrfHIiESV+R/MXO3a7XTdKwFNaY6vdw0+8YAo9g8= +github.com/codefly-dev/sdk-go v0.3.1/go.mod h1:vWrT2zXNMuCrctTAHMBYWuGRkfr0OP3IXATyuJ6iYvE= github.com/containerd/errdefs v1.0.0 h1:tg5yIfIlQIrxYtu9ajqY42W3lpS19XqdxRQeEwYG8PI= github.com/containerd/errdefs v1.0.0/go.mod h1:+YBYIdtsnF4Iw6nWZhJcqGSg/dwvV7tyJ/kCkyJ2k+M= github.com/containerd/errdefs/pkg v0.3.0 h1:9IKJ06FvyNlexW690DXuQNx2KA2cUJXx151Xdx3ZPPE= diff --git a/module/tools/go.mod b/module/tools/go.mod index d694ed99c..18f97c6b8 100644 --- a/module/tools/go.mod +++ b/module/tools/go.mod @@ -4,7 +4,7 @@ go 1.27.0 require ( github.com/Masterminds/semver/v3 v3.5.0 - github.com/codefly-dev/core v0.5.11-0.20260926231217-d481a2d49683 + github.com/codefly-dev/core v0.13.0 github.com/yoheimuta/go-protoparser/v4 v4.14.2 golang.org/x/tools v0.49.0 google.golang.org/protobuf v1.36.12 diff --git a/module/tools/go.sum b/module/tools/go.sum index 9a67efddf..46f722672 100644 --- a/module/tools/go.sum +++ b/module/tools/go.sum @@ -28,8 +28,8 @@ github.com/cheggaaa/pb/v3 v3.2.1 h1:aprZbFRG+B7+ug76S8QZ6Y1PW168UHzOmbC3wa+aU6I= github.com/cheggaaa/pb/v3 v3.2.1/go.mod h1:U9hSVxoKqJrZIE3PkFG1xXXNaK/Ilzg+EF/scpXzEdw= github.com/clipperhouse/uax29/v2 v2.7.0 h1:+gs4oBZ2gPfVrKPthwbMzWZDaAFPGYK72F0NJv2v7Vk= github.com/clipperhouse/uax29/v2 v2.7.0/go.mod h1:EFJ2TJMRUaplDxHKj1qAEhCtQPW2tJSwu5BF98AuoVM= -github.com/codefly-dev/core v0.5.11-0.20260926231217-d481a2d49683 h1:/vzymsu5B9NWwdhWRtlU+68VcVEVDmeZNChRCXAUwlE= -github.com/codefly-dev/core v0.5.11-0.20260926231217-d481a2d49683/go.mod h1:SVF+t040X+9pAXAqBnNE/JkXwmw9peVuKnHKFSOzyzk= +github.com/codefly-dev/core v0.13.0 h1:FiphttBZRrKXEWoCZD7JpUB6oVuBCoqtAc06QeNdYWY= +github.com/codefly-dev/core v0.13.0/go.mod h1:rp4YZkCJEOufliWQAgK3sv8YlO7Xcva7+cN7hIlAR0I= github.com/codefly-dev/gortk v0.2.0 h1:7bOlS5valYz2zil+fZctQNcPCYBcPj86abcw9N8h1hQ= github.com/codefly-dev/gortk v0.2.0/go.mod h1:dDWUMFgAP063OCGhTEpV7FFUj9+zTcxxO/nV80VKEwg= github.com/containerd/errdefs v1.0.0 h1:tg5yIfIlQIrxYtu9ajqY42W3lpS19XqdxRQeEwYG8PI= diff --git a/qualification/module-authority/go.mod b/qualification/module-authority/go.mod index bd8a5454d..2a311147d 100644 --- a/qualification/module-authority/go.mod +++ b/qualification/module-authority/go.mod @@ -9,7 +9,7 @@ require ( require ( buf.build/gen/go/bufbuild/protovalidate/protocolbuffers/go v1.36.12-20260825204119-511051f7f437.2 // indirect - github.com/codefly-dev/core v0.3.32 // indirect + github.com/codefly-dev/core v0.13.0 // indirect github.com/codefly-dev/sdk-go/workcontext v0.0.0-20260927024316-b3fb40d43099 // indirect google.golang.org/genproto/googleapis/api v0.0.0-20260819154853-08b0e4226688 // indirect google.golang.org/genproto/googleapis/rpc v0.0.0-20260825221802-da73d73af1c5 // indirect diff --git a/qualification/module-authority/go.sum b/qualification/module-authority/go.sum index bd1f7e2a8..379eb172b 100644 --- a/qualification/module-authority/go.sum +++ b/qualification/module-authority/go.sum @@ -2,8 +2,8 @@ buf.build/gen/go/bufbuild/protovalidate/protocolbuffers/go v1.36.12-202608252041 buf.build/gen/go/bufbuild/protovalidate/protocolbuffers/go v1.36.12-20260825204119-511051f7f437.2/go.mod h1:TCt1lluMFnctISJXvkIQ4x3ABrPuUKCWKyjKdkJNBpw= connectrpc.com/connect v1.21.0 h1:LhqSJt7jHf5NJBo9Jq/t/9FjcYAideif0mg+qe2jCUs= connectrpc.com/connect v1.21.0/go.mod h1:A2ygJrukXwWy32vkCAAHNVguZrqZ+jeZ9rGRnGR4dN4= -github.com/codefly-dev/core v0.3.32 h1:zVgGAljUERRjHe5ONhI2uBuQg0eayZzkFSm0murp5NI= -github.com/codefly-dev/core v0.3.32/go.mod h1:f5LY9gRu5lpBQ0tNKujP0q6MF9K8fUnbYvItuHaUWlY= +github.com/codefly-dev/core v0.13.0 h1:FiphttBZRrKXEWoCZD7JpUB6oVuBCoqtAc06QeNdYWY= +github.com/codefly-dev/core v0.13.0/go.mod h1:rp4YZkCJEOufliWQAgK3sv8YlO7Xcva7+cN7hIlAR0I= github.com/codefly-dev/saas-sdk-go v0.0.3-0.20260927130902-fed59a51306c h1:xZHqwC4BG2+N8Rz8euD6pF60f0QuHVljHVkPXShS2zU= github.com/codefly-dev/saas-sdk-go v0.0.3-0.20260927130902-fed59a51306c/go.mod h1:o/Gkv2g9T9ObjHxg54Dafp6/yPJDEgSSrh+pyg4SD44= github.com/codefly-dev/sdk-go/workcontext v0.0.0-20260927024316-b3fb40d43099 h1:MEzUboHy6HzBex1VoAOo3JqBgjTjSYs3jQGyqIwHOhQ= From badfb6c8e3423e00017ab551e3eb77c83e7c4a65 Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Wed, 7 Oct 2026 11:08:54 -0400 Subject: [PATCH 48/69] fix(accounts,auth-gateway): restore the generated telemetry client stubs the generator owns (Refs #1022) The service-removal commit deleted `external/saas-starter/telemetry/` from accounts and auth-gateway by hand. Both files are generator output (`Code generated by protoc-gen-go. DO NOT EDIT.`), and this repository does not hand-edit generated code, deletion included. They are restored byte for byte from main, with no other change. They are orphaned: no dependency block declares the telemetry service any more, and nothing imports the package. `codefly sync service` does not prune bindings for a dependency that is no longer declared, so a sync leaves them where they are, and the tree stays clean after every generator and sync rerun with them present. With them present, `go vet` and `go test` in accounts (pure suite) and auth-gateway, the contracts check with the CI-pinned CLI 0.1.171, the module-package validation and archive determinism, and the node gates all pass, so there is no failing check to justify removing them. The generator owns them and will prune them once it learns to; a draft issue against the go-grpc agent describes the gap. Co-Authored-By: Claude Opus 5.5 --- .../saas-starter_telemetry_grpc.pb.go | 60 +++++++++++++++++++ .../saas-starter_telemetry_grpc.pb.go | 60 +++++++++++++++++++ 2 files changed, 120 insertions(+) create mode 100644 module/services/accounts/code/external/saas-starter/telemetry/saas-starter_telemetry_grpc.pb.go create mode 100644 module/services/auth-gateway/code/external/saas-starter/telemetry/saas-starter_telemetry_grpc.pb.go diff --git a/module/services/accounts/code/external/saas-starter/telemetry/saas-starter_telemetry_grpc.pb.go b/module/services/accounts/code/external/saas-starter/telemetry/saas-starter_telemetry_grpc.pb.go new file mode 100644 index 000000000..689bf2157 --- /dev/null +++ b/module/services/accounts/code/external/saas-starter/telemetry/saas-starter_telemetry_grpc.pb.go @@ -0,0 +1,60 @@ +// Code generated by protoc-gen-go. DO NOT EDIT. +// versions: +// protoc-gen-go v1.36.11 +// protoc (unknown) +// source: saas-starter_telemetry_grpc.proto + +package telemetry + +import ( + reflect "reflect" + unsafe "unsafe" + + protoreflect "google.golang.org/protobuf/reflect/protoreflect" + protoimpl "google.golang.org/protobuf/runtime/protoimpl" +) + +const ( + // Verify that this generated code is sufficiently up-to-date. + _ = protoimpl.EnforceVersion(20 - protoimpl.MinVersion) + // Verify that runtime/protoimpl is sufficiently up-to-date. + _ = protoimpl.EnforceVersion(protoimpl.MaxVersion - 20) +) + +var File_saas_starter_telemetry_grpc_proto protoreflect.FileDescriptor + +const file_saas_starter_telemetry_grpc_proto_rawDesc = "" + + "\n" + + "!saas-starter_telemetry_grpc.protoBhB\x1dSaasStarterTelemetryGrpcProtoP\x01ZEgithub.com/codefly-dev/cli/pkg/builder/clients/saas-starter/telemetry" + +var file_saas_starter_telemetry_grpc_proto_goTypes = []any{} +var file_saas_starter_telemetry_grpc_proto_depIdxs = []int32{ + 0, // [0:0] is the sub-list for method output_type + 0, // [0:0] is the sub-list for method input_type + 0, // [0:0] is the sub-list for extension type_name + 0, // [0:0] is the sub-list for extension extendee + 0, // [0:0] is the sub-list for field type_name +} + +func init() { file_saas_starter_telemetry_grpc_proto_init() } +func file_saas_starter_telemetry_grpc_proto_init() { + if File_saas_starter_telemetry_grpc_proto != nil { + return + } + type x struct{} + out := protoimpl.TypeBuilder{ + File: protoimpl.DescBuilder{ + GoPackagePath: reflect.TypeOf(x{}).PkgPath(), + RawDescriptor: unsafe.Slice(unsafe.StringData(file_saas_starter_telemetry_grpc_proto_rawDesc), len(file_saas_starter_telemetry_grpc_proto_rawDesc)), + NumEnums: 0, + NumMessages: 0, + NumExtensions: 0, + NumServices: 0, + }, + GoTypes: file_saas_starter_telemetry_grpc_proto_goTypes, + DependencyIndexes: file_saas_starter_telemetry_grpc_proto_depIdxs, + }.Build() + File_saas_starter_telemetry_grpc_proto = out.File + file_saas_starter_telemetry_grpc_proto_goTypes = nil + file_saas_starter_telemetry_grpc_proto_depIdxs = nil +} diff --git a/module/services/auth-gateway/code/external/saas-starter/telemetry/saas-starter_telemetry_grpc.pb.go b/module/services/auth-gateway/code/external/saas-starter/telemetry/saas-starter_telemetry_grpc.pb.go new file mode 100644 index 000000000..689bf2157 --- /dev/null +++ b/module/services/auth-gateway/code/external/saas-starter/telemetry/saas-starter_telemetry_grpc.pb.go @@ -0,0 +1,60 @@ +// Code generated by protoc-gen-go. DO NOT EDIT. +// versions: +// protoc-gen-go v1.36.11 +// protoc (unknown) +// source: saas-starter_telemetry_grpc.proto + +package telemetry + +import ( + reflect "reflect" + unsafe "unsafe" + + protoreflect "google.golang.org/protobuf/reflect/protoreflect" + protoimpl "google.golang.org/protobuf/runtime/protoimpl" +) + +const ( + // Verify that this generated code is sufficiently up-to-date. + _ = protoimpl.EnforceVersion(20 - protoimpl.MinVersion) + // Verify that runtime/protoimpl is sufficiently up-to-date. + _ = protoimpl.EnforceVersion(protoimpl.MaxVersion - 20) +) + +var File_saas_starter_telemetry_grpc_proto protoreflect.FileDescriptor + +const file_saas_starter_telemetry_grpc_proto_rawDesc = "" + + "\n" + + "!saas-starter_telemetry_grpc.protoBhB\x1dSaasStarterTelemetryGrpcProtoP\x01ZEgithub.com/codefly-dev/cli/pkg/builder/clients/saas-starter/telemetry" + +var file_saas_starter_telemetry_grpc_proto_goTypes = []any{} +var file_saas_starter_telemetry_grpc_proto_depIdxs = []int32{ + 0, // [0:0] is the sub-list for method output_type + 0, // [0:0] is the sub-list for method input_type + 0, // [0:0] is the sub-list for extension type_name + 0, // [0:0] is the sub-list for extension extendee + 0, // [0:0] is the sub-list for field type_name +} + +func init() { file_saas_starter_telemetry_grpc_proto_init() } +func file_saas_starter_telemetry_grpc_proto_init() { + if File_saas_starter_telemetry_grpc_proto != nil { + return + } + type x struct{} + out := protoimpl.TypeBuilder{ + File: protoimpl.DescBuilder{ + GoPackagePath: reflect.TypeOf(x{}).PkgPath(), + RawDescriptor: unsafe.Slice(unsafe.StringData(file_saas_starter_telemetry_grpc_proto_rawDesc), len(file_saas_starter_telemetry_grpc_proto_rawDesc)), + NumEnums: 0, + NumMessages: 0, + NumExtensions: 0, + NumServices: 0, + }, + GoTypes: file_saas_starter_telemetry_grpc_proto_goTypes, + DependencyIndexes: file_saas_starter_telemetry_grpc_proto_depIdxs, + }.Build() + File_saas_starter_telemetry_grpc_proto = out.File + file_saas_starter_telemetry_grpc_proto_goTypes = nil + file_saas_starter_telemetry_grpc_proto_depIdxs = nil +} From feafbdd872e9e10c7219b06cb18345b5bb9678e0 Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Wed, 7 Oct 2026 12:26:52 -0400 Subject: [PATCH 49/69] fix(audit): history copy rewrites events whose details a failed write left out Both adapters write an event's events row before its details row. When the details write failed, the next history copy run found the event id in the store, skipped the event, and verification then failed on "content details are missing inside the content window" for good: nothing ever wrote the details, so the operator could never drop the verified partitions. The copy now decides what to write from what verification would conclude, not from the id being in the store. One rule, AuditHistoryCopy.inspect, is shared by copyWindow and verify: it returns what is wrong with the stored copies of an event and whether writing the event again would remove it. - Written: the store holds none of the event, or holds it without the details of a content-class event inside the content window. The write appends an events row and a details row; both adapters attach details to every events row of the event by its id on read, so the copy already there is completed too (ClickHouse and the BigQuery fake read-back are pinned by new tests). The report counts these as Rewritten, apart from Copied. - Not written, still a verification failure: a copy under another deployment, an envelope or retention class that differs, a details hash that differs, details that do not hash to the row's hash, and a security-class copy without details. The store is append-only and every copy is verified, so a new row would leave the wrong copy in place; a security-class copy's details live in its own events row, which a new row does not complete. When copies fail differently, the failure no write removes is the one reported. - Content-class events past the content window may have no details and are not written. AUDIT_OPERATIONS.md, the accounts AGENTS.md and the command's doc comment describe the retry. Tests: a split store (events table, details table, join by event id on read) whose first details write fails after its events rows landed: the first run errors, the second writes the three lost content events again and verifies; content past the window is not written; each unfixable case is left alone and a later run appends and archives nothing; a table test of the shared rule, mixed copies included. Co-Authored-By: Claude Sonnet 5.5 --- module/AUDIT_OPERATIONS.md | 27 +- module/services/accounts/AGENTS.md | 4 +- .../accounts/code/pkg/auditops/history.go | 10 +- .../bigquerystore/history_rewrite_test.go | 57 ++++ .../history_rewrite_server_test.go | 58 ++++ .../code/pkg/business/audit_history.go | 132 ++++++-- .../code/pkg/business/audit_history_test.go | 317 ++++++++++++++++++ 7 files changed, 559 insertions(+), 46 deletions(-) create mode 100644 module/services/accounts/code/pkg/auditstore/bigquerystore/history_rewrite_test.go create mode 100644 module/services/accounts/code/pkg/auditstore/clickhousestore/history_rewrite_server_test.go diff --git a/module/AUDIT_OPERATIONS.md b/module/AUDIT_OPERATIONS.md index a4a992f40..c1c9bd238 100644 --- a/module/AUDIT_OPERATIONS.md +++ b/module/AUDIT_OPERATIONS.md @@ -31,10 +31,29 @@ cannot drop. `-verify-only` writes nothing. The run is resumable. For each day-long window of each partition it reads the source rows, reads back what the warehouse holds for that window, appends only the -events missing from it, reads the window back again, and then verifies every source -row against every stored copy of its id. Existing copies are therefore verified -**after** the append, in the same pass; an earlier run's copies are not verified -before new rows are written. A verification failure leaves the partitions in place. +events whose stored copies are not complete, reads the window back again, and then +verifies every source row against every stored copy of its id. Existing copies are +therefore verified **after** the append, in the same pass; an earlier run's copies +are not verified before new rows are written. A verification failure leaves the +partitions in place. + +An event is appended when the warehouse holds none of it, and again when it holds +it without details that verification requires and a write supplies: a content-class +event inside the content window whose details row is not there. Both adapters write +an event's events row before its details row, so a run whose details write failed +leaves exactly that behind, and the next run appends the event again, events row +and details row. The details are attached to every copy of the event by its id, so +the copy already in the warehouse is complete too. A text-mode run counts these as +`rewritten` in its progress lines, apart from the events `copied`. Content-class +events older than the content window may have no details (the warehouse expired +them) and are not appended. + +A copy that is wrong in itself is **not** appended again: the warehouse is +append-only and every copy is verified, so the wrong copy would stay. These are +verification failures for the operator to look at: a copy stored under another +deployment id, with an envelope or retention class the row does not have, with a +details hash that is not the row's, or a security-class copy without its details +(they live in that copy's own row). ### Which partitions diff --git a/module/services/accounts/AGENTS.md b/module/services/accounts/AGENTS.md index 31f463324..6d8f3f81d 100644 --- a/module/services/accounts/AGENTS.md +++ b/module/services/accounts/AGENTS.md @@ -116,7 +116,9 @@ survives a restart and reaches every replica, and is only served when it is a deployment switching to a swap value copies its `audit_events` rows into the store of record and the archive — classified and hashed as the relay does a live event — and verifies them by reading them back. It reads back before it - writes, so a rerun finishes an interrupted run. Only with `-confirm-drop`, + writes, and writes an event again when the store holds it without details a + write supplies (`inspect`, the one rule `verify` shares), so a rerun finishes + an interrupted run, a failed details write included. Only with `-confirm-drop`, after a verification pass in the same run and a recount showing nothing changed, does it drop the copied partitions, named one by one through `audit_events_drop_verified_partitions` (migration 22), which recounts each diff --git a/module/services/accounts/code/pkg/auditops/history.go b/module/services/accounts/code/pkg/auditops/history.go index ef347a79a..af89462e4 100644 --- a/module/services/accounts/code/pkg/auditops/history.go +++ b/module/services/accounts/code/pkg/auditops/history.go @@ -22,10 +22,12 @@ // app_control_plane, the authority retention runs under. // // It is resumable: a run reads each window's rows, reads back what the store -// holds, appends only the events missing from it, reads the window back again -// and verifies every event, so a run after an interrupted one copies only what -// is missing. -verify-only copies nothing, for a re-check once the store has -// caught up. +// holds, appends only the events whose stored copies are not complete — those +// the store lacks, and those it holds without details a write supplies, such as +// a content-class event whose details write failed — reads the window back again +// and verifies every event, so a run after an interrupted one finishes what it +// left. -verify-only copies nothing, for a re-check once the store has caught +// up. // // Exit status: // diff --git a/module/services/accounts/code/pkg/auditstore/bigquerystore/history_rewrite_test.go b/module/services/accounts/code/pkg/auditstore/bigquerystore/history_rewrite_test.go new file mode 100644 index 000000000..78f7c1d03 --- /dev/null +++ b/module/services/accounts/code/pkg/auditstore/bigquerystore/history_rewrite_test.go @@ -0,0 +1,57 @@ +package bigquerystore_test + +import ( + "context" + "testing" + "time" + + "accounts/pkg/auditstore/bigqueryfake" + "accounts/pkg/auditstore/bigquerystore" + "accounts/pkg/business" + + "github.com/stretchr/testify/require" +) + +// The history copy writes an event again when a failed details write left only +// its events row (business.AuditHistoryCopy). The adapter must make that write +// complete the copy already stored: the append of the event adds an events row +// and a details row, and the read-back attaches the details to every events row +// of the event by its id, so no copy is left without them. +func TestAnEventWrittenAgainAfterAFailedDetailsWriteIsCompleteInEveryCopy(t *testing.T) { + w := newWarehouse(t) + at := now.Add(-time.Hour) + content := event(t, 1, orgA, business.EventDatasourceSourceSynced, business.RetentionContent, at, map[string]any{"n": 1}) + security := event(t, 2, orgA, business.EventAuthLogin, business.RetentionSecurity, at, map[string]any{"method": "password"}) + + // The first write: the events rows landed, the details write failed. + events, _ := bigquerystore.BatchRows(business.AuditBatch{ID: "first", DeploymentID: deployment, Records: []business.AuditRecord{content, security}}) + for _, row := range events { + require.NoError(t, w.fake.Insert(bigqueryfake.TablePath(project, dataset, bigquerystore.EventsTable), row.Values)) + } + stored := w.historyReadBack(t, at) + require.Len(t, stored[content.Entry.ID], 1) + require.False(t, stored[content.Entry.ID][0].HasDetails, "the details write was lost") + require.True(t, stored[security.Entry.ID][0].HasDetails, "a security-class event carries its details in its events row") + + // The write again: the content-class event's events row and details row. + w.append(t, deployment, content) + stored = w.historyReadBack(t, at) + require.Len(t, stored[content.Entry.ID], 2, "the events row written again is another copy") + for _, got := range stored[content.Entry.ID] { + require.True(t, got.HasDetails, "the details attach to every copy of the event by its id") + require.Equal(t, content.DetailsSHA256, got.DetailsSHA256) + require.Equal(t, content.DetailsSHA256, business.AuditDetailsSHA256(got.Details)) + } + require.Len(t, stored[security.Entry.ID], 1) +} + +// historyReadBack is every stored copy of every event in the hour around at, by id. +func (w *warehouse) historyReadBack(t *testing.T, at time.Time) map[string][]business.StoredAuditEvent { + t.Helper() + out := map[string][]business.StoredAuditEvent{} + require.NoError(t, w.reader.ReadStoredAuditEvents(context.Background(), at.Add(-time.Hour), at.Add(time.Hour), func(got business.StoredAuditEvent) error { + out[got.Entry.ID] = append(out[got.Entry.ID], got) + return nil + })) + return out +} diff --git a/module/services/accounts/code/pkg/auditstore/clickhousestore/history_rewrite_server_test.go b/module/services/accounts/code/pkg/auditstore/clickhousestore/history_rewrite_server_test.go new file mode 100644 index 000000000..0c05a0967 --- /dev/null +++ b/module/services/accounts/code/pkg/auditstore/clickhousestore/history_rewrite_server_test.go @@ -0,0 +1,58 @@ +package clickhousestore + +import ( + "context" + "testing" + "time" + + "accounts/pkg/business" + + "github.com/google/uuid" + "github.com/stretchr/testify/require" +) + +// The history copy writes an event again when a failed details write left only +// its events row (business.AuditHistoryCopy). The adapter must make that write +// complete the copy already stored: the append of the event adds an events row +// and a details row, and the read-back attaches the details to every events row +// of the event by its id, so no copy is left without them. +func TestServerAnEventWrittenAgainAfterAFailedDetailsWriteIsCompleteInEveryCopy(t *testing.T) { + conn, database := serverDatabase(t) + ctx := context.Background() + store := serverStore(t, conn, database) + require.NoError(t, store.Ensure(ctx)) + + org, actor := uuid.NewString(), uuid.NewString() + at := time.Now().UTC().Truncate(time.Microsecond).Add(-time.Hour) + content := newRecord(t, org, actor, business.EventDatasourceSourceSynced, "source-1", at, business.RetentionContent, map[string]any{"n": 1}) + security := newRecord(t, org, actor, business.EventAuthLogin, "session-1", at, business.RetentionSecurity, map[string]any{"method": "password"}) + readBack := func() map[string][]business.StoredAuditEvent { + out := map[string][]business.StoredAuditEvent{} + require.NoError(t, store.ReadStoredAuditEvents(ctx, at.Add(-time.Hour), at.Add(time.Hour), func(got business.StoredAuditEvent) error { + out[got.Entry.ID] = append(out[got.Entry.ID], got) + return nil + })) + return out + } + + // The first write: the events rows landed, the details write failed. + events, _ := BatchRows(business.AuditBatch{DeploymentID: testDeployment, Records: []business.AuditRecord{content, security}}) + refused, err := store.insert(ctx, EventsTable, EventsColumns(), events) + require.NoError(t, err) + require.Empty(t, refused) + stored := readBack() + require.Len(t, stored[content.Entry.ID], 1) + require.False(t, stored[content.Entry.ID][0].HasDetails, "the details write was lost") + require.True(t, stored[security.Entry.ID][0].HasDetails, "a security-class event carries its details in its events row") + + // The write again: the content-class event's events row and details row. + appendBatch(t, store, testDeployment, content) + stored = readBack() + require.Len(t, stored[content.Entry.ID], 2, "the events row written again is another copy") + for _, got := range stored[content.Entry.ID] { + require.True(t, got.HasDetails, "the details attach to every copy of the event by its id") + require.Equal(t, content.DetailsSHA256, got.DetailsSHA256) + require.Equal(t, content.DetailsSHA256, business.AuditDetailsSHA256(got.Details)) + } + require.Len(t, stored[security.Entry.ID], 1) +} diff --git a/module/services/accounts/code/pkg/business/audit_history.go b/module/services/accounts/code/pkg/business/audit_history.go index 968cee4c3..464e76919 100644 --- a/module/services/accounts/code/pkg/business/audit_history.go +++ b/module/services/accounts/code/pkg/business/audit_history.go @@ -164,8 +164,14 @@ type AuditHistoryPartitionReport struct { Partition AuditHistoryPartition // EventsSHA256 binds the source event envelopes, classes and details hashes. EventsSHA256 string - // Copied is how many events this run wrote; the rest were already there. + // Copied is how many events this run wrote because the store held none of + // them. Copied int + // Rewritten is how many events this run wrote again because the store held + // them without details that verification requires and that a write supplies + // (a content-class event whose details write was lost); the rest were + // already complete. + Rewritten int // Orgs counts the partition's events per organization ("" is the // platform's). Orgs map[string]AuditHistoryOrgCount @@ -267,9 +273,11 @@ func NewAuditHistoryCopy(cfg AuditHistoryCopyConfig) (*AuditHistoryCopy, error) // Run copies, verifies and — when confirmed and verified — drops. It is // resumable: a run reads each window's rows, reads back what the store already -// holds, appends only the events missing from it, reads the window back again -// and verifies every event, so a run after an interrupted one finishes what -// that one started. A verification failure blocks the drop and returns +// holds, appends only the events whose stored copies are not complete — the +// events the store lacks, and those it holds without the details a write +// supplies — reads the window back again and verifies every event, so a run +// after an interrupted one finishes what that one started, a failed details +// write included. A verification failure blocks the drop and returns // ErrAuditHistoryUnverified; a run with nothing to verify returns // ErrAuditHistoryNothingToVerify; a drop that does not happen returns // ErrAuditHistoryDropRefused with the reason. @@ -445,13 +453,13 @@ func (c *AuditHistoryCopy) copyPartition(ctx context.Context, resolver *AuditEve rows += count.Postgres verified += count.Verified } - c.cfg.Progress(fmt.Sprintf("%s: %d events, %d copied by this run, %d verified, %d problems", - partition.Name, rows, report.Copied, verified, report.Failures)) + c.cfg.Progress(fmt.Sprintf("%s: %d events, %d copied and %d rewritten by this run, %d verified, %d problems", + partition.Name, rows, report.Copied, report.Rewritten, verified, report.Failures)) return report, nil } // copyWindow reads a window's rows, reads back what the store holds of it, -// writes what is missing, and verifies every row. +// writes what is missing or incomplete, and verifies every row. func (c *AuditHistoryCopy) copyWindow(ctx context.Context, resolver *AuditEventResolver, from, to time.Time, opts AuditHistoryCopyOptions, report *AuditHistoryPartitionReport, digest hash.Hash) error { records, err := c.readWindow(ctx, resolver, from, to) if err != nil { @@ -465,19 +473,33 @@ func (c *AuditHistoryCopy) copyWindow(ctx context.Context, resolver *AuditEventR return err } if !opts.VerifyOnly { - var missing []AuditRecord + // An event is written when verification would fail for want of a write: + // the store holds none of it, or holds it without details a write + // supplies. The id being in the store says nothing: both adapters write + // an event's row before its details, so a failed details write leaves + // the row behind, and skipping on its id would block verification for + // good. + var pending []AuditRecord + var rewrite []bool // parallel to pending: the store already held the event for _, record := range records { - if len(stored[record.Entry.ID]) == 0 { - missing = append(missing, record) + copies := stored[record.Entry.ID] + if problem, writable := c.inspect(record, copies); problem != "" && writable { + pending, rewrite = append(pending, record), append(rewrite, len(copies) > 0) } } - if len(missing) > 0 { - for start := 0; start < len(missing); start += c.cfg.BatchSize { - end := min(start+c.cfg.BatchSize, len(missing)) - if err := c.write(ctx, missing[start:end]); err != nil { + if len(pending) > 0 { + for start := 0; start < len(pending); start += c.cfg.BatchSize { + end := min(start+c.cfg.BatchSize, len(pending)) + if err := c.write(ctx, pending[start:end]); err != nil { return err } - report.Copied += end - start + for _, again := range rewrite[start:end] { + if again { + report.Rewritten++ + } else { + report.Copied++ + } + } } if stored, err = c.readBack(ctx, from, to); err != nil { return err @@ -582,34 +604,70 @@ func (c *AuditHistoryCopy) write(ctx context.Context, records []AuditRecord) err // event's details must be there, and a content-class event's while it is // inside the content window. func (c *AuditHistoryCopy) verify(record AuditRecord, copies []StoredAuditEvent) string { + problem, _ := c.inspect(record, copies) + return problem +} + +// inspect is the one rule for the store's copies of record, shared by what +// writes and what verifies: what verification finds wrong with them, and +// whether writing the event again would remove it. +// +// A write does when the store holds none of the event, or holds it without the +// details of a content-class event inside the content window: both adapters +// keep those in the details table and attach them to every copy of the event by +// its id, so one more details row completes the copies already there, and the +// events row the write appends beside them carries the same envelope. +// +// A write does not when a copy is wrong in itself — stored under another +// deployment, with an envelope or a retention class the row does not have, with a +// details hash that is not the row's, or with details that do not hash to it — +// because the store is append-only and every copy is verified, so the wrong copy +// would still be there, and a read of the event may return it. Nor when a +// security-class copy lacks its details: they live in that copy's own row, which +// a new row does not complete. Such a copy is a verification failure for an +// operator to look at. When copies differ in what is wrong, the reason that no +// write removes is the one returned. +func (c *AuditHistoryCopy) inspect(record AuditRecord, copies []StoredAuditEvent) (problem string, writable bool) { if len(copies) == 0 { - return "missing from the store" + return "missing from the store", true } - want := record.Entry for _, stored := range copies { - got := stored.Entry + reason, fixable := c.inspectCopy(record, stored) switch { - case stored.DeploymentID != c.cfg.DeploymentID: - return fmt.Sprintf("stored under deployment %q", stored.DeploymentID) - case got.OrgID != want.OrgID || got.ActorID != want.ActorID || got.ActorType != want.ActorType || - got.EventType != want.EventType || got.SchemaVersion != want.SchemaVersion || - got.Resource != want.Resource || got.ResourceID != want.ResourceID || got.IPAddress != want.IPAddress || - got.ImpersonatedBy != want.ImpersonatedBy || got.IsImpersonated != want.IsImpersonated || - got.ClientID != want.ClientID || !got.CreatedAt.Equal(want.CreatedAt): - return "stored envelope differs from the row" - case stored.Retention != record.Retention: - return fmt.Sprintf("stored as %s, classified %s", stored.Retention, record.Retention) - case stored.DetailsSHA256 != record.DetailsSHA256: - return "stored details hash differs from the row's" - case stored.HasDetails && AuditDetailsSHA256(stored.Details) != record.DetailsSHA256: - return "stored details do not hash to the row's hash" - case !stored.HasDetails && record.Retention == RetentionSecurity: - return "security-class details are missing" - case !stored.HasDetails && want.CreatedAt.After(c.cfg.Now().Add(-c.cfg.ContentRetention+auditHistoryDetailsMargin)): - return "content details are missing inside the content window" + case reason == "": + case !fixable: + return reason, false + case problem == "": + problem = reason } } - return "" + return problem, problem != "" +} + +// inspectCopy is inspect for one stored copy. +func (c *AuditHistoryCopy) inspectCopy(record AuditRecord, stored StoredAuditEvent) (problem string, writable bool) { + want, got := record.Entry, stored.Entry + switch { + case stored.DeploymentID != c.cfg.DeploymentID: + return fmt.Sprintf("stored under deployment %q", stored.DeploymentID), false + case got.OrgID != want.OrgID || got.ActorID != want.ActorID || got.ActorType != want.ActorType || + got.EventType != want.EventType || got.SchemaVersion != want.SchemaVersion || + got.Resource != want.Resource || got.ResourceID != want.ResourceID || got.IPAddress != want.IPAddress || + got.ImpersonatedBy != want.ImpersonatedBy || got.IsImpersonated != want.IsImpersonated || + got.ClientID != want.ClientID || !got.CreatedAt.Equal(want.CreatedAt): + return "stored envelope differs from the row", false + case stored.Retention != record.Retention: + return fmt.Sprintf("stored as %s, classified %s", stored.Retention, record.Retention), false + case stored.DetailsSHA256 != record.DetailsSHA256: + return "stored details hash differs from the row's", false + case stored.HasDetails && AuditDetailsSHA256(stored.Details) != record.DetailsSHA256: + return "stored details do not hash to the row's hash", false + case !stored.HasDetails && record.Retention == RetentionSecurity: + return "security-class details are missing", false + case !stored.HasDetails && want.CreatedAt.After(c.cfg.Now().Add(-c.cfg.ContentRetention+auditHistoryDetailsMargin)): + return "content details are missing inside the content window", true + } + return "", false } // drop checks that nothing changed since verification and drops the verified diff --git a/module/services/accounts/code/pkg/business/audit_history_test.go b/module/services/accounts/code/pkg/business/audit_history_test.go index 3b9f89358..d75747724 100644 --- a/module/services/accounts/code/pkg/business/audit_history_test.go +++ b/module/services/accounts/code/pkg/business/audit_history_test.go @@ -188,6 +188,81 @@ func (s *historyStore) copies() map[string]int { return out } +// historyDetailsRow is one row of a split store's details table. +type historyDetailsRow struct { + eventID, sha256, text string +} + +// splitHistoryStore is a store of record as both adapters keep one: an events +// row for every record — carrying its details when the event is security-class — +// and, written after the events rows, a details row for each content-class +// record. A read attaches the details to every events row of the event by its +// id, and when the details rows disagree reports one whose hash is not the +// event's, as the adapters' readers do. failDetails makes the details write of +// that (1-based) append fail after its events rows landed: the partial write +// either adapter can leave. +type splitHistoryStore struct { + events []StoredAuditEvent // a content-class row carries no details + details []historyDetailsRow + appends int + failDetails int +} + +func (s *splitHistoryStore) AppendAuditBatch(_ context.Context, batch AuditBatch) error { + s.appends++ + for _, record := range batch.Records { + entry := record.Entry + entry.Payload = nil + event := StoredAuditEvent{ + DeploymentID: batch.DeploymentID, Entry: entry, Retention: record.Retention, DetailsSHA256: record.DetailsSHA256, + } + if record.Retention == RetentionSecurity { + event.Details, event.HasDetails = record.Details, true + } + s.events = append(s.events, event) + } + if s.appends == s.failDetails { + return errors.New("details write failed") + } + for _, record := range batch.Records { + if record.Retention != RetentionSecurity { + s.details = append(s.details, historyDetailsRow{record.Entry.ID, record.DetailsSHA256, record.Details}) + } + } + return nil +} + +func (s *splitHistoryStore) ReadStoredAuditEvents(_ context.Context, from, to time.Time, visit func(StoredAuditEvent) error) error { + for _, event := range s.events { + if event.Entry.CreatedAt.Before(from) || !event.Entry.CreatedAt.Before(to) { + continue + } + if event.Retention != RetentionSecurity { + for _, row := range s.details { + if row.eventID != event.Entry.ID { + continue + } + event.Details, event.HasDetails = row.text, true + if row.sha256 != event.DetailsSHA256 { + break + } + } + } + if err := visit(event); err != nil { + return err + } + } + return nil +} + +func (s *splitHistoryStore) copies() map[string]int { + out := map[string]int{} + for _, event := range s.events { + out[event.Entry.ID]++ + } + return out +} + type historyArchive struct{ batches []AuditBatch } func (a *historyArchive) WriteAuditBatch(_ context.Context, batch AuditBatch) error { @@ -217,6 +292,9 @@ type historyFixture struct { source *historySource store *historyStore archive *historyArchive + // split, when set, is the store of record in place of store: one that keeps + // a content-class event's details in a table of their own, as the adapters do. + split *splitHistoryStore } func newHistoryFixture() *historyFixture { @@ -239,6 +317,9 @@ func newHistoryFixture() *historyFixture { func (f *historyFixture) copier(t *testing.T, cfg AuditHistoryCopyConfig) *AuditHistoryCopy { t.Helper() cfg.Source, cfg.Store, cfg.Archive, cfg.ReadBack = f.source, f.store, f.archive, f.store + if f.split != nil { + cfg.Store, cfg.ReadBack = f.split, f.split + } if cfg.Through == nil && len(f.source.partitions) > 0 { through := f.source.partitions[len(f.source.partitions)-1].To cfg.Through = &through @@ -751,3 +832,239 @@ func TestAuditHistoryDuplicateIDsFailVerificationWithinAWindow(t *testing.T) { }) } } + +// Both adapters write an event's events row before its details row, so a +// details write that fails leaves the events row behind. The copy decides what +// to write from what verification would conclude, not from the id being in the +// store: the next run writes the event again, its details land, and they attach +// to the events row already there. +func TestAuditHistoryRewritesEventsWhoseDetailsWriteFailed(t *testing.T) { + f := newHistoryFixture() + f.split = &splitHistoryStore{failDetails: 1} + _, err := f.copier(t, AuditHistoryCopyConfig{}).Run(context.Background(), AuditHistoryCopyOptions{}) + require.ErrorContains(t, err, "details write failed") + require.Empty(t, f.split.details, "the failed append left events rows and no details") + require.Len(t, f.split.events, 4, "the first batch's events rows landed") + var landed []string // the ids of the content-class events whose details are missing + for _, event := range f.split.events { + if event.Retention == RetentionContent { + landed = append(landed, event.Entry.ID) + } + } + require.Len(t, landed, 3) + + f.split.failDetails = 0 + report, err := f.copier(t, AuditHistoryCopyConfig{}).Run(context.Background(), AuditHistoryCopyOptions{}) + require.NoError(t, err) + require.True(t, report.Verified) + var copied, rewritten int + for _, partition := range report.Partitions { + copied += partition.Copied + rewritten += partition.Rewritten + require.Zero(t, partition.Failures) + } + require.Equal(t, 3, rewritten, "the content events whose details were lost are written again") + require.Equal(t, len(f.source.rows)-4, copied, "the events the first run never reached are copied") + + copies := f.split.copies() + for _, id := range landed { + require.Equal(t, 2, copies[id], "the events row of the failed write and the one written again: %s", id) + } + // Every copy a read-back returns carries its details, so every copy passes + // the verification the run just made. + require.NoError(t, f.split.ReadStoredAuditEvents(context.Background(), time.Time{}, historyNow, func(event StoredAuditEvent) error { + require.True(t, event.HasDetails, "event %s", event.Entry.ID) + require.Equal(t, event.DetailsSHA256, AuditDetailsSHA256(event.Details)) + return nil + })) + + // A run after that one finds nothing to write, and drops what was verified. + appends := f.split.appends + report, err = f.copier(t, AuditHistoryCopyConfig{}).Run(context.Background(), f.dropOptions(t, AuditHistoryCopyConfig{})) + require.NoError(t, err) + require.Equal(t, appends, f.split.appends) + require.Equal(t, int64(3), report.Dropped) + for _, partition := range report.Partitions { + require.Zero(t, partition.Copied+partition.Rewritten) + } +} + +// A content-class event past the content window may have no details — the +// warehouse expired them — so verification does not ask for them and the copy +// does not write the event again to supply them. +func TestAuditHistoryDoesNotRewriteContentEventsPastTheWindow(t *testing.T) { + f := newHistoryFixture() + f.split = &splitHistoryStore{failDetails: 1} + config := AuditHistoryCopyConfig{ContentRetention: 7 * 24 * time.Hour} + _, err := f.copier(t, config).Run(context.Background(), AuditHistoryCopyOptions{}) + require.ErrorContains(t, err, "details write failed") + + f.split.failDetails = 0 + report, err := f.copier(t, config).Run(context.Background(), AuditHistoryCopyOptions{}) + require.NoError(t, err) + require.True(t, report.Verified) + for _, partition := range report.Partitions { + require.Zero(t, partition.Rewritten) + } + for id, n := range f.split.copies() { + require.Equal(t, 1, n, "event %s is in the store once", id) + } +} + +// A write appends a copy; it removes nothing. So an event whose stored copy is +// wrong in itself is left alone — written again, the wrong copy would still be +// there, and still fail verification — and stays a verification failure. +func TestAuditHistoryDoesNotRewriteWhatAWriteCannotFix(t *testing.T) { + const content, security = 0, 2 // indexes of rows of the fixture + for name, tc := range map[string]struct { + row int + tamper func(*StoredAuditEvent) + want string + }{ + "a copy under another deployment": { + row: content, + tamper: func(e *StoredAuditEvent) { e.DeploymentID = "deployment-2" }, + want: "stored under deployment", + }, + "an envelope that differs": { + row: content, + tamper: func(e *StoredAuditEvent) { e.Entry.OrgID = "cccccccc-0000-4000-8000-000000000003" }, + want: "envelope differs", + }, + "a retention class that differs": { + row: content, + tamper: func(e *StoredAuditEvent) { e.Retention = RetentionSecurity }, + want: "stored as security, classified content", + }, + "a details hash that differs": { + row: content, + tamper: func(e *StoredAuditEvent) { e.DetailsSHA256 = AuditDetailsSHA256("{}") }, + want: "details hash differs", + }, + "details that do not hash to the row's hash": { + row: content, + tamper: func(e *StoredAuditEvent) { e.Details = `{"n":0}` }, + want: "do not hash", + }, + "details missing under a hash that differs": { + row: content, + tamper: func(e *StoredAuditEvent) { + e.DetailsSHA256, e.Details, e.HasDetails = AuditDetailsSHA256("{}"), "", false + }, + want: "details hash differs", + }, + "security-class details missing": { + row: security, + tamper: func(e *StoredAuditEvent) { e.Details, e.HasDetails = "", false }, + want: "security-class details are missing", + }, + } { + t.Run(name, func(t *testing.T) { + f := newHistoryFixture() + bad := f.source.rows[tc.row].ID + f.store.tamper = func(e *StoredAuditEvent) { + if e.Entry.ID == bad { + tc.tamper(e) + } + } + _, err := f.copier(t, AuditHistoryCopyConfig{}).Run(context.Background(), AuditHistoryCopyOptions{}) + require.ErrorIs(t, err, ErrAuditHistoryUnverified) + appends, archived := f.store.appends, len(f.archive.batches) + + // The run that follows would append a good copy beside the bad one + // if it wrote at all: the tamper is gone, and nothing is appended. + f.store.tamper = nil + report, err := f.copier(t, AuditHistoryCopyConfig{}).Run(context.Background(), AuditHistoryCopyOptions{}) + require.ErrorIs(t, err, ErrAuditHistoryUnverified) + require.False(t, report.Verified) + require.Equal(t, appends, f.store.appends, "nothing is appended") + require.Len(t, f.archive.batches, archived, "nothing is archived") + require.Equal(t, 1, f.store.copies()[bad]) + var problems []string + for _, partition := range report.Partitions { + require.Zero(t, partition.Copied+partition.Rewritten) + problems = append(problems, partition.Problems...) + } + require.Len(t, problems, 1) + require.Contains(t, problems[0], bad) + require.Contains(t, problems[0], tc.want) + }) + } +} + +// What the copy writes and what verification reports are one rule, applied to +// every copy of the event: a write is for the failure it removes, and when a +// copy is wrong in a way no write removes, that is what is reported. +func TestAuditHistoryInspectSaysWhatAWriteRemoves(t *testing.T) { + copier, err := NewAuditHistoryCopy(AuditHistoryCopyConfig{ + Source: &historySource{}, Store: &historyStore{}, Archive: &historyArchive{}, ReadBack: &historyStore{}, + DeploymentID: "deployment-1", ContentRetention: 7 * 24 * time.Hour, + Now: func() time.Time { return historyNow }, + }) + require.NoError(t, err) + recent, old := historyNow.Add(-24*time.Hour), historyNow.Add(-30*24*time.Hour) + recordOf := func(n int, eventType EventType, at time.Time) AuditRecord { + resolved, err := NewAuditEventResolver(nil).Resolve(context.Background(), eventType) + require.NoError(t, err) + record, err := NewAuditRecord(historyRow(n, "", eventType, at), resolved.RetentionClass()) + require.NoError(t, err) + return record + } + storedOf := func(record AuditRecord, change func(*StoredAuditEvent)) StoredAuditEvent { + entry := record.Entry + entry.Payload = nil + stored := StoredAuditEvent{DeploymentID: "deployment-1", Entry: entry, Retention: record.Retention, DetailsSHA256: record.DetailsSHA256} + if record.Retention == RetentionSecurity { + stored.Details, stored.HasDetails = record.Details, true + } + if change != nil { + change(&stored) + } + return stored + } + hasDetails := func(record AuditRecord) func(*StoredAuditEvent) { + return func(e *StoredAuditEvent) { e.Details, e.HasDetails = record.Details, true } + } + otherOrg := func(e *StoredAuditEvent) { e.Entry.OrgID = "cccccccc-0000-4000-8000-000000000003" } + noDetails := func(e *StoredAuditEvent) { e.Details, e.HasDetails = "", false } + + inside, outside := recordOf(1, EventDatasourceSourceSynced, recent), recordOf(2, EventDatasourceSourceSynced, old) + login := recordOf(3, EventAuthLogin, recent) + for name, tc := range map[string]struct { + record AuditRecord + copies []StoredAuditEvent + problem string + writable bool + }{ + "no copy": {record: inside, problem: "missing from the store", writable: true}, + "a complete copy": {record: inside, copies: []StoredAuditEvent{storedOf(inside, hasDetails(inside))}}, + "content details missing inside the window": { + record: inside, copies: []StoredAuditEvent{storedOf(inside, nil)}, + problem: "content details are missing inside the content window", writable: true, + }, + "content details missing past the window": {record: outside, copies: []StoredAuditEvent{storedOf(outside, nil)}}, + "security details missing": { + record: login, copies: []StoredAuditEvent{storedOf(login, noDetails)}, + problem: "security-class details are missing", + }, + "one copy missing details, the other differing, in either order": { + record: inside, copies: []StoredAuditEvent{storedOf(inside, nil), storedOf(inside, func(e *StoredAuditEvent) { otherOrg(e); hasDetails(inside)(e) })}, + problem: "stored envelope differs from the row", + }, + "one copy differing, the other missing details": { + record: inside, copies: []StoredAuditEvent{storedOf(inside, func(e *StoredAuditEvent) { otherOrg(e); hasDetails(inside)(e) }), storedOf(inside, nil)}, + problem: "stored envelope differs from the row", + }, + "every copy missing details": { + record: inside, copies: []StoredAuditEvent{storedOf(inside, nil), storedOf(inside, nil)}, + problem: "content details are missing inside the content window", writable: true, + }, + } { + t.Run(name, func(t *testing.T) { + problem, writable := copier.inspect(tc.record, tc.copies) + require.Equal(t, tc.problem, problem) + require.Equal(t, tc.writable, writable) + require.Equal(t, tc.problem, copier.verify(tc.record, tc.copies), "verification reports what inspect found") + }) + } +} From 7de65343e9ed0b4b26383e4dd197af9e49957d11 Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Wed, 7 Oct 2026 12:26:24 -0400 Subject: [PATCH 50/69] fix(audit): charge a BigQuery read once per event, not per delivered copy The relay delivers at least once, so the events and details tables can hold an event, and its details, several times over. readDetails charged the window budget for every details row it streamed while keeping one per event id, so a window whose unique details fit could still pass its budget. Every copy of an event shares its occurrence and lands in the same window, so halving the window never shed them: the read ended in ErrReadTooDense and the payload-filtered list, aggregation and export of that span failed for good. A window now counts what it keeps. The details of an event are kept once per event id and details hash; a repeat of a copy already kept is neither counted nor kept. Copies of one event under different hashes disagree, which the history verification reports; the read keeps each, counted, and joins an event to the copy under the hash its own row carries, as the ClickHouse store does, so no stream order picks the details an event shows and a copy no event row names is never shown. The join of a list page's details keeps only the id and hash each page event names, so a flood of other copies cannot grow it. The events side already deduplicated by event id before it charged, and its dedupe set only holds events that were charged; the new tests pin that too. Co-Authored-By: Claude Sonnet 5.5 --- module/docs/adr/0009-audit-store-swap.md | 9 +- .../pkg/auditstore/bigquerystore/reader.go | 102 +++++-- .../bigquerystore/reader_copies_test.go | 268 ++++++++++++++++++ 3 files changed, 345 insertions(+), 34 deletions(-) create mode 100644 module/services/accounts/code/pkg/auditstore/bigquerystore/reader_copies_test.go diff --git a/module/docs/adr/0009-audit-store-swap.md b/module/docs/adr/0009-audit-store-swap.md index 4b11640cd..87a145140 100644 --- a/module/docs/adr/0009-audit-store-swap.md +++ b/module/docs/adr/0009-audit-store-swap.md @@ -205,9 +205,12 @@ Postgres transaction. it joins and, for a read that must see every event of the window before it can answer (an aggregation, an export), the events and the set that deduplicates them — against a byte budget, `ReadConfig.WindowBytes` (64 MiB by default; the - service exposes no setting for it). A window that passes the budget is read - again as two halves. What outlives a window is the answer: a page, an - aggregation's buckets, an export's events. Four limits remain: + service exposes no setting for it). What is counted is what is kept, once per + event however many copies a redelivery left in the tables: the copies of an + event share its occurrence, so no narrower window could shed them. A window + that passes the budget is read again as two halves. What outlives a window is + the answer: a page, an aggregation's buckets, an export's events. Four limits + remain: - A read whose events at a single instant alone pass the budget cannot be split further and fails (`ErrReadTooDense`). - An export is the one answer as large as the history it matches. One export diff --git a/module/services/accounts/code/pkg/auditstore/bigquerystore/reader.go b/module/services/accounts/code/pkg/auditstore/bigquerystore/reader.go index 9ecf0cf9c..25dfcc3b2 100644 --- a/module/services/accounts/code/pkg/auditstore/bigquerystore/reader.go +++ b/module/services/accounts/code/pkg/auditstore/bigquerystore/reader.go @@ -101,7 +101,10 @@ const ( // What a window retains is counted, not measured, so the bound is one number // the deployment chose: a joined details row costs its text and the map entry // around it; an event the window buffers costs auditeval.Event.Size plus its -// entry in the set that deduplicates it. +// entry in the set that deduplicates it. What is counted is what is kept: the +// relay delivers at least once, so the tables may hold an event, and its +// details, several times over, and a copy the window already holds costs +// nothing. const ( detailRowOverheadBytes = 160 dedupeEntryBytes = 64 @@ -394,7 +397,7 @@ func (r *Reader) candidates(m *auditeval.Matcher, q business.AuditQuery, w windo // joined to its content details from details (when it is not nil) and handed to // visit (serially). A row of another deployment is never handed on, whatever // the session returned. -func (r *Reader) readEvents(ctx context.Context, m *auditeval.Matcher, q business.AuditQuery, w window, details map[string]storedDetail, visit func(*auditeval.Event) error) error { +func (r *Reader) readEvents(ctx context.Context, m *auditeval.Matcher, q business.AuditQuery, w window, details map[detailKey]string, visit func(*auditeval.Event) error) error { where, ok := r.candidates(m, q, w) if !ok { return nil @@ -409,8 +412,8 @@ func (r *Reader) readEvents(ctx context.Context, m *auditeval.Matcher, q busines } event := stored.event() if details != nil && stored.retentionClass != string(business.RetentionSecurity) { - if detail, ok := details[event.Entry.ID]; ok && detail.sha256 == stored.detailsSHA256 { - event.Details, event.HasDetails = detail.details, true + if text, ok := details[detailKey{eventID: event.Entry.ID, sha256: stored.detailsSHA256}]; ok { + event.Details, event.HasDetails = text, true } } return visit(event) @@ -419,12 +422,12 @@ func (r *Reader) readEvents(ctx context.Context, m *auditeval.Matcher, q busines // windowDetails reads the content details of every candidate event in a // window, counting them against b: errWindowTooLarge when they pass it. -func (r *Reader) windowDetails(ctx context.Context, m *auditeval.Matcher, w window, b *budget) (map[string]storedDetail, error) { +func (r *Reader) windowDetails(ctx context.Context, m *auditeval.Matcher, w window, b *budget) (map[detailKey]string, error) { where := r.scope(m.Scope()) w.restrict(where) if types, restricted := m.EventTypes(); restricted { if len(types) == 0 { - return map[string]storedDetail{}, nil + return map[detailKey]string{}, nil } if len(types) == 1 { where.eq("event_type", types[0]) @@ -432,21 +435,45 @@ func (r *Reader) windowDetails(ctx context.Context, m *auditeval.Matcher, w wind where.in("event_type", types) } } - return r.readDetails(ctx, where, b) + return r.readDetails(ctx, where, b, nil) } // storedDetail is one details-table row: the canonical details of a -// content-class event and their hash. +// content-class event and their hash. The history read-back keeps every one. type storedDetail struct { sha256 string details string } -// readDetails reads the details rows a restriction admits, counting each -// against b (nil counts nothing, for a read the caller has bounded by the ids -// it names). -func (r *Reader) readDetails(ctx context.Context, where *restriction, b *budget) (map[string]storedDetail, error) { - details := map[string]storedDetail{} +// detailKey names the details of a content-class event the way the event does: +// by its id and the hash its details carry. A read joins an event to the copy +// under its own hash, as the ClickHouse store does (it groups the details table +// by id and hash and joins on both). +type detailKey struct { + eventID string + sha256 string +} + +// readDetails reads the details rows a restriction admits and keeps one text +// per event id and hash, counting each text kept against b. A nil b counts +// nothing, and then keep must bound what is kept (a nil keep keeps every copy +// the restriction admits): it is asked of each id and hash, and a copy it +// refuses is skipped. +// +// The relay delivers at least once, so the table may hold the details of an +// event any number of times. Every copy of an event shares its occurrence, so +// they all land in one window and no narrower window can shed them; counting +// them all would make a window that fits its unique details unreadable. A copy +// of an id and hash already kept is therefore neither counted nor kept: it is +// the same text, which the hash makes provable, and the first copy stays. +// +// Two copies of one event id that carry different hashes disagree, which the +// history verification reports. Here each hash is a copy of its own, counted +// because it is kept, and the event is joined to the copy under the hash its +// own row carries. A copy no event row names is never shown, and no stream +// order decides which copy an event gets. +func (r *Reader) readDetails(ctx context.Context, where *restriction, b *budget, keep func(detailKey) bool) (map[detailKey]string, error) { + details := map[detailKey]string{} err := r.scan(ctx, DetailsTable, detailFields, where, func(row arrowRow) error { eventID, err := row.str("event_id") if err != nil { @@ -463,6 +490,10 @@ func (r *Reader) readDetails(ctx context.Context, where *restriction, b *budget) if err != nil { return err } + key := detailKey{eventID: eventID, sha256: sha} + if _, kept := details[key]; kept || (keep != nil && !keep(key)) { + return nil + } text, err := row.str("details") if err != nil { return err @@ -472,7 +503,7 @@ func (r *Reader) readDetails(ctx context.Context, where *restriction, b *budget) } // Copies, so what is kept is the text alone and not the record batch // it was decoded from. - details[strings.Clone(eventID)] = storedDetail{sha256: strings.Clone(sha), details: strings.Clone(text)} + details[detailKey{eventID: strings.Clone(key.eventID), sha256: strings.Clone(key.sha256)}] = strings.Clone(text) return nil }) return details, err @@ -487,7 +518,7 @@ func (r *Reader) readDetails(ctx context.Context, where *restriction, b *budget) // copies. func (r *Reader) loadWindow(ctx context.Context, m *auditeval.Matcher, q business.AuditQuery, w window, withDetails bool) ([]*auditeval.Event, error) { b := r.newBudget() - var details map[string]storedDetail + var details map[detailKey]string if withDetails { var err error if details, err = r.windowDetails(ctx, m, w, b); err != nil { @@ -519,7 +550,28 @@ func (r *Reader) loadWindow(ctx context.Context, m *auditeval.Matcher, q busines // one session bounded by their occurrences and ids. An event whose details are // past the content window keeps none. func (r *Reader) joinDetails(ctx context.Context, scope business.AuditReadScope, events []*auditeval.Event) error { + details, err := r.pageDetails(ctx, scope, events) + if err != nil { + return err + } + for _, event := range events { + if event.HasDetails { + continue + } + if text, ok := details[detailKey{eventID: event.Entry.ID, sha256: event.DetailsSHA256}]; ok { + event.Details, event.HasDetails = text, true + } + } + return nil +} + +// pageDetails reads the details the given events that have none yet name, each +// under the hash its event carries. What it keeps is bounded by the page, not +// by the copies the table holds: the ids it names are a page's, and a copy +// under another hash is skipped, not kept. +func (r *Reader) pageDetails(ctx context.Context, scope business.AuditReadScope, events []*auditeval.Event) (map[detailKey]string, error) { var ids []string + wanted := map[detailKey]bool{} var lo, hi time.Time for _, event := range events { if event.HasDetails { @@ -533,27 +585,15 @@ func (r *Reader) joinDetails(ctx context.Context, scope business.AuditReadScope, hi = at } ids = append(ids, event.Entry.ID) + wanted[detailKey{eventID: event.Entry.ID, sha256: event.DetailsSHA256}] = true } if len(ids) == 0 { - return nil + return nil, nil } where := r.scope(scope) window{lo: &lo, hi: &hi}.restrict(where) where.in("event_id", ids) - // Bounded by the ids it names, which are a page's. - details, err := r.readDetails(ctx, where, nil) - if err != nil { - return err - } - for _, event := range events { - if event.HasDetails { - continue - } - if detail, ok := details[event.Entry.ID]; ok && detail.sha256 == event.DetailsSHA256 { - event.Details, event.HasDetails = detail.details, true - } - } - return nil + return r.readDetails(ctx, where, nil, func(key detailKey) bool { return wanted[key] }) } // ListAuditEvents implements business.AuditReader: the newest windows first, @@ -577,7 +617,7 @@ func (r *Reader) ListAuditEvents(ctx context.Context, read business.AuditRead) ( // and nothing but the page is held at all. withDetails := m.NeedsPayload() err = r.newestFirst(upper, m.From(), withDetails, func(w window) (bool, error) { - var details map[string]storedDetail + var details map[detailKey]string if withDetails { var err error if details, err = r.windowDetails(ctx, m, w, r.newBudget()); err != nil { diff --git a/module/services/accounts/code/pkg/auditstore/bigquerystore/reader_copies_test.go b/module/services/accounts/code/pkg/auditstore/bigquerystore/reader_copies_test.go new file mode 100644 index 000000000..261fd0ad1 --- /dev/null +++ b/module/services/accounts/code/pkg/auditstore/bigquerystore/reader_copies_test.go @@ -0,0 +1,268 @@ +package bigquerystore + +import ( + "context" + "fmt" + "strings" + "testing" + "time" + + "accounts/pkg/auditstore/auditeval" + "accounts/pkg/auditstore/bigqueryfake" + "accounts/pkg/business" + + "github.com/stretchr/testify/require" +) + +// The relay delivers at least once, so the events and details tables may hold +// an event, and its details, many times over. Every copy of an event shares its +// occurrence, so all of them land in one window and no narrower window can shed +// them. What a window counts against its budget is what it keeps, once per +// event, or a window whose unique data fits would be unreadable for ever. + +// deliver appends records times times, as a relay redelivering a batch would. +func (w *boundWarehouse) deliver(t *testing.T, times int, records ...business.AuditRecord) { + t.Helper() + for i := 0; i < times; i++ { + w.append(t, records...) + } +} + +// insertDetails writes one more details-table copy of a record's event, with +// text as its details and the hash of text. +func (w *boundWarehouse) insertDetails(t *testing.T, record business.AuditRecord, text string) { + t.Helper() + row := DetailRow(boundDeployment, record) + row.Values["details"] = text + row.Values["details_sha256"] = business.AuditDetailsSHA256(text) + require.NoError(t, w.fake.Insert(bigqueryfake.TablePath(boundProject, boundDataset, DetailsTable), row.Values)) +} + +// instant is n content-class events at one instant, each with about 900 bytes +// of details; the odd ones are "rare", and each is in a "group" of two. +func instant(t *testing.T, n int, at time.Time) []business.AuditRecord { + t.Helper() + pad := strings.Repeat("p", 900) + var records []business.AuditRecord + for i := 1; i <= n; i++ { + payload := map[string]any{"group": i % 2, "pad": pad} + if i%2 == 1 { + payload["rare"] = true + } + records = append(records, boundEvent(t, i, at, payload)) + } + return records +} + +// detailsCost is what a window counts for the details of records, once each. +func detailsCost(records ...business.AuditRecord) int64 { + var cost int64 + for _, record := range records { + cost += int64(detailRowOverheadBytes + len(record.Entry.ID) + len(record.DetailsSHA256) + len(record.Details)) + } + return cost +} + +// readKinds are the reads that hold a window in memory, each returning its +// answer in a form two readers can be compared by. +var readKinds = map[string]func(*testing.T, *Reader) (any, error){ + "a payload-filtered list": func(t *testing.T, reader *Reader) (any, error) { + var ids []string + q := business.AuditQuery{PageSize: 2, PayloadContains: map[string]any{"rare": true}} + for { + entries, next, err := reader.ListAuditEvents(context.Background(), orgScoped(q)) + if err != nil { + return nil, err + } + ids = append(ids, entryIDs(entries)...) + if next == "" { + return ids, nil + } + q.PageToken = next + } + }, + "a payload aggregation": func(t *testing.T, reader *Reader) (any, error) { + return reader.AggregateAuditEvents(context.Background(), orgScoped(business.AuditQuery{}), + business.AuditAggregationSpec{GroupBy: []string{"payload:group"}}) + }, + "an aggregation of envelopes": func(t *testing.T, reader *Reader) (any, error) { + return reader.AggregateAuditEvents(context.Background(), orgScoped(business.AuditQuery{}), business.AuditAggregationSpec{}) + }, + "an export": func(t *testing.T, reader *Reader) (any, error) { + return reader.ExportAuditEvents(context.Background(), orgScoped(business.AuditQuery{})) + }, +} + +func TestAWindowOfRedeliveredCopiesIsCountedOncePerEventNotOncePerCopy(t *testing.T) { + at := boundNow.Add(-time.Hour) + records := instant(t, 6, at) + once, many := newBoundWarehouse(t), newBoundWarehouse(t) + once.deliver(t, 1, records...) + many.deliver(t, 60, records...) + require.Len(t, many.fake.Rows(bigqueryfake.TablePath(boundProject, boundDataset, EventsTable)), 360, "sixty copies of each of six events") + require.Len(t, many.fake.Rows(bigqueryfake.TablePath(boundProject, boundDataset, DetailsTable)), 360) + + // Six events and their details are about 17 KB; 360 copies would be a megabyte. + const budget = 32 << 10 + for name, read := range readKinds { + reference, copies := once.reader(t, budget, 0), many.reader(t, budget, 0) + want, err := read(t, reference) + require.NoError(t, err, name) + got, err := read(t, copies) + require.NoError(t, err, "%s: the unique data fits the window, however many copies the tables hold", name) + require.Equal(t, want, got, "%s: the same answer as a single delivery", name) + require.Equal(t, reference.windowPeak.Load(), copies.windowPeak.Load(), "%s: counted for what is kept, not for what was delivered", name) + require.Greater(t, copies.windowPeak.Load(), int64(0), name) + require.LessOrEqual(t, copies.windowPeak.Load(), int64(budget), name) + } + + // And the answers are the events, once each. + exported, err := many.reader(t, budget, 0).ExportAuditEvents(context.Background(), orgScoped(business.AuditQuery{})) + require.NoError(t, err) + require.Len(t, exported, 6) + require.NotNil(t, exported[0].Payload) + buckets, err := many.reader(t, budget, 0).AggregateAuditEvents(context.Background(), orgScoped(business.AuditQuery{}), + business.AuditAggregationSpec{GroupBy: []string{"payload:group"}}) + require.NoError(t, err) + var total int64 + for _, bucket := range buckets { + total += bucket.Count + } + require.Equal(t, int64(6), total) +} + +// The window counts exactly what it keeps: the budget that holds the unique +// details holds them whatever the copies, and one byte less does not. +func TestAWindowBudgetThatHoldsTheUniqueDetailsHoldsThemWhateverTheCopies(t *testing.T) { + at := boundNow.Add(-time.Hour) + records := instant(t, 6, at) + w := newBoundWarehouse(t) + w.deliver(t, 40, records...) + cost := detailsCost(records...) + q := business.AuditQuery{PayloadContains: map[string]any{"rare": true}} + + fits := w.reader(t, cost, 0) + entries, _, err := fits.ListAuditEvents(context.Background(), orgScoped(q)) + require.NoError(t, err) + require.Len(t, entries, 3) + require.Equal(t, cost, fits.windowPeak.Load()) + + short := w.reader(t, cost-1, 0) + _, _, err = short.ListAuditEvents(context.Background(), orgScoped(q)) + require.ErrorIs(t, err, ErrReadTooDense, "one byte short of the unique details, and the instant cannot be split") +} + +func TestUniqueDataOverTheBudgetStillFailsHoweverItIsDelivered(t *testing.T) { + at := boundNow.Add(-time.Hour) + records := instant(t, 80, at) + const budget = 32 << 10 + for _, deliveries := range []int{1, 3} { + w := newBoundWarehouse(t) + w.deliver(t, deliveries, records...) + for name, read := range readKinds { + reader := w.reader(t, budget, 0) + _, err := read(t, reader) + require.ErrorIs(t, err, ErrReadTooDense, "%s, delivered %d times: eighty events at one instant are more than a window holds", name, deliveries) + require.LessOrEqual(t, reader.windowPeak.Load(), int64(budget+oneRow), name) + } + } +} + +// Two copies of one event that carry different hashes disagree — the history +// verification reports it. A read joins the event to the copy under the hash +// its own row carries, as the ClickHouse store does, so the answer does not +// depend on the order the copies arrive in, a copy no event names is never +// shown, and each distinct copy kept is counted once. +func TestConflictingDetailsAreJoinedByHashAndEachDistinctCopyCountedOnce(t *testing.T) { + at := boundNow.Add(-time.Hour) + good := boundEvent(t, 1, at, map[string]any{"mark": "good"}) + other := boundEvent(t, 2, at, map[string]any{"mark": "other"}) + const forged = `{"mark": "forged"}` + const budget = 32 << 10 + clean := newBoundWarehouse(t) + clean.deliver(t, 1, good, other) + cleanReader := clean.reader(t, budget, 0) + _, _, err := cleanReader.ListAuditEvents(context.Background(), orgScoped(business.AuditQuery{PayloadContains: map[string]any{"mark": "good"}})) + require.NoError(t, err) + forgedCost := int64(detailRowOverheadBytes + len(good.Entry.ID) + len(business.AuditDetailsSHA256(forged)) + len(forged)) + + for name, forgedFirst := range map[string]bool{"the conflicting copy first": true, "the conflicting copy last": false} { + for _, deliveries := range []int{1, 25} { + w := newBoundWarehouse(t) + if forgedFirst { + for i := 0; i < deliveries; i++ { + w.insertDetails(t, good, forged) + } + } + w.deliver(t, deliveries, good, other) + if !forgedFirst { + for i := 0; i < deliveries; i++ { + w.insertDetails(t, good, forged) + } + } + label := fmt.Sprintf("%s, delivered %d times", name, deliveries) + + // A filter on the details the event's own row names finds it, with those details. + reader := w.reader(t, budget, 0) + entries, _, err := reader.ListAuditEvents(context.Background(), orgScoped(business.AuditQuery{PayloadContains: map[string]any{"mark": "good"}})) + require.NoError(t, err, label) + require.Equal(t, []string{good.Entry.ID}, entryIDs(entries), label) + require.Equal(t, map[string]any{"mark": "good"}, entries[0].Payload, label) + require.Equal(t, cleanReader.windowPeak.Load()+forgedCost, reader.windowPeak.Load(), + "%s: the conflicting copy is kept, so it is counted, once however often it was delivered", label) + + // The copy no event row names is never shown. + entries, _, err = w.reader(t, budget, 0).ListAuditEvents(context.Background(), orgScoped(business.AuditQuery{PayloadContains: map[string]any{"mark": "forged"}})) + require.NoError(t, err, label) + require.Empty(t, entries, label) + + // Without a filter the page's details are joined afterwards, by the same rule. + entries, _, err = w.reader(t, budget, 0).ListAuditEvents(context.Background(), orgScoped(business.AuditQuery{})) + require.NoError(t, err, label) + require.Len(t, entries, 2, label) + marks := map[string]any{} + for _, entry := range entries { + marks[entry.ID] = entry.Payload["mark"] + } + require.Equal(t, map[string]any{good.Entry.ID: "good", other.Entry.ID: "other"}, marks, label) + + // The windows an aggregation and an export hold agree. + buckets, err := w.reader(t, budget, 0).AggregateAuditEvents(context.Background(), orgScoped(business.AuditQuery{}), + business.AuditAggregationSpec{GroupBy: []string{"payload:mark"}}) + require.NoError(t, err, label) + counts := map[string]int64{} + for _, bucket := range buckets { + counts[bucket.Key] = bucket.Count + } + require.Equal(t, map[string]int64{"good": 1, "other": 1}, counts, label) + exported, err := w.reader(t, budget, 0).ExportAuditEvents(context.Background(), orgScoped(business.AuditQuery{})) + require.NoError(t, err, label) + require.Len(t, exported, 2, label) + } + } +} + +// A list page joins the details of its own events, which are a page's. What +// that read keeps is bounded by the page, not by the copies the table holds: +// of a flood of copies under other hashes it keeps none. +func TestAPagesDetailsAreBoundedByThePageNotByTheCopiesOfItsEvents(t *testing.T) { + at := boundNow.Add(-time.Hour) + good := boundEvent(t, 1, at, map[string]any{"mark": "good"}) + w := newBoundWarehouse(t) + w.deliver(t, 5, good) + for i := 0; i < 200; i++ { + w.insertDetails(t, good, fmt.Sprintf(`{"mark": "forged-%d"}`, i)) + } + reader := w.reader(t, 0, 0) + + // The event as a list page holds it: its envelope and the hash of its details. + page := []*auditeval.Event{{Entry: good.Entry, DetailsSHA256: good.DetailsSHA256}} + details, err := reader.pageDetails(context.Background(), business.OrganizationAuditScope(boundOrg), page) + require.NoError(t, err) + require.Equal(t, map[detailKey]string{{eventID: good.Entry.ID, sha256: good.DetailsSHA256}: good.Details}, details, + "the one copy the event names; the 204 others under the same id are skipped") + + require.NoError(t, reader.joinDetails(context.Background(), business.OrganizationAuditScope(boundOrg), page)) + require.True(t, page[0].HasDetails) + require.Equal(t, good.Details, page[0].Details) +} From 270df7a21f90c1d80fa025e3ea6637465563a412 Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Wed, 7 Oct 2026 12:32:37 -0400 Subject: [PATCH 51/69] fix(audit): bound ClickHouse percentile samples before they reach the service A ClickHouse percentile aggregation returned every sample of a group through groupArray. The service's state budget counted only the bucket's keys, so one large group (or many medium ones) was decoded by the driver, copied by Scan and copied again and sorted by PercentileCont before anything was refused. The statement now bounds what it returns before the driver decodes it: - each percentile array is groupArray(N+1), where N is the budget divided by what the service holds per input (the driver's block and the slice Scan copies out of it, 16 bytes), so the server never builds a group's array past what could be taken; a group with more inputs than N is refused for its exact count, never answered from a truncated array; - the statement sums what the service will charge for every group (bucket, key bytes, inputs) in a window function and, when the sum passes the budget or any row is inexact, returns the keys and arrays empty with the totals, so the service refuses (or evaluates the aggregation itself) from the first row and the history never sits in a driver block. The service counts what it receives with the same arithmetic and refuses an array that is not the inputs its count says. The percentile is taken by sorting the scanned slice in place (auditeval.PercentileContInPlace), which removes the second copy from this path and from Aggregator.Buckets. Co-Authored-By: Claude Sonnet 5.5 --- module/docs/adr/0009-audit-store-swap.md | 8 +- .../pkg/auditstore/auditeval/aggregate.go | 45 ++- .../auditeval/aggregate_bound_test.go | 40 +++ .../auditstore/clickhousestore/aggregate.go | 95 +++++- .../aggregate_bound_server_test.go | 191 ++++++++++++ .../clickhousestore/aggregate_bound_test.go | 290 ++++++++++++++++++ .../clickhousestore/clickhousestore.go | 9 +- .../pkg/auditstore/clickhousestore/reader.go | 59 +++- 8 files changed, 699 insertions(+), 38 deletions(-) create mode 100644 module/services/accounts/code/pkg/auditstore/clickhousestore/aggregate_bound_server_test.go create mode 100644 module/services/accounts/code/pkg/auditstore/clickhousestore/aggregate_bound_test.go diff --git a/module/docs/adr/0009-audit-store-swap.md b/module/docs/adr/0009-audit-store-swap.md index 87a145140..24d4fa748 100644 --- a/module/docs/adr/0009-audit-store-swap.md +++ b/module/docs/adr/0009-audit-store-swap.md @@ -226,8 +226,12 @@ Postgres transaction. (`AuditAggregateMaxBytes`, `ReadConfig.AggregateMaxBytes` and `clickhousestore.Config.AggregateMaxBytes`), and the read is refused with `ResourceExhausted` (`ErrAuditAggregateTooLarge`) as soon as it passes it, - never answered in part. A time range, an actor or an event type narrows it, - as does grouping by fewer dimensions. + never answered in part. The ClickHouse statement counts the same state + itself and returns neither keys nor percentile inputs once it would pass the + bound (the service holds each input twice while it reads one, so there it is + 16 bytes), so the service never receives what it could not keep. A time + range, an actor or an event type narrows it, as does grouping by fewer + dimensions. - The history copy (item 7) reads and holds a whole window of the store at a time outside this budget. diff --git a/module/services/accounts/code/pkg/auditstore/auditeval/aggregate.go b/module/services/accounts/code/pkg/auditstore/auditeval/aggregate.go index 9fc47606f..caf934104 100644 --- a/module/services/accounts/code/pkg/auditstore/auditeval/aggregate.go +++ b/module/services/accounts/code/pkg/auditstore/auditeval/aggregate.go @@ -48,9 +48,15 @@ type Aggregator struct { } // What the state of an aggregation costs, counted rather than measured so the -// bound is one number the deployment chose. The figures err high: a bucket's -// cost includes the bucket it is answered as, and a sample's the 8 bytes of its -// float, not the capacity a slice that grows by doubling may hold past it. +// bound is one number the deployment chose. A bucket's cost includes the bucket +// it is answered as. A sample is counted at the 8 bytes of its float, not at the +// capacity of the slice that holds a group's samples: append leaves that slice +// up to a quarter larger than what it holds once a group has a few hundred +// samples (up to twice before), and keeps the slice it replaces beside it while +// it copies, so what a percentile really holds is up to about two and a quarter +// times what is counted, and the bound is set with that in mind. A store that +// receives its samples as a block counts them at what it really holds +// (clickhousestore's scannedSampleBytes). const ( // bucketBytes is a bucket's own: the group, its entries in the map and the // ordered list, and the answer built from it. @@ -78,7 +84,9 @@ func BucketBytes(keys []string, metrics int) int64 { return n } -// SampleBytes is the state of n percentile inputs. +// SampleBytes is the state of n percentile inputs: the 8 bytes of each float64 +// in the slice that holds them. The percentile is taken by sorting that slice +// in place (PercentileContInPlace), so it costs no second copy. func SampleBytes(n int) int64 { return int64(n) * sampleBytes } // StateBudget counts the bytes an aggregation retains across every window it @@ -106,11 +114,18 @@ func NewStateBudget(limit int64) (*StateBudget, error) { func (b *StateBudget) Take(n int64) error { b.held += n if b.held > b.limit { - return fmt.Errorf("%w (the bound is %d bytes)", business.ErrAuditAggregateTooLarge, b.limit) + return b.Exceeded() } return nil } +// Exceeded is the refusal Take reports once the limit is passed. A store that +// learns the bound was passed without holding the bytes to count (a statement +// that withheld its rows because they would pass it) reports the same one. +func (b *StateBudget) Exceeded() error { + return fmt.Errorf("%w (the bound is %d bytes)", business.ErrAuditAggregateTooLarge, b.limit) +} + // Held is the bytes counted so far. func (b *StateBudget) Held() int64 { return b.held } @@ -371,7 +386,10 @@ func (a *Aggregator) Buckets() []business.AuditAggregateBucket { case "max": bucket.Metrics[alias] = acc.max case "percentile": - bucket.Metrics[alias] = PercentileCont(acc.values, metric.Percentile) + // The state is the aggregation's own and is never read in order, + // so it is sorted where it lies: a copy would hold every sample + // twice, once more than the budget counts. + bucket.Metrics[alias] = PercentileContInPlace(acc.values, metric.Percentile) } } for _, derived := range a.spec.Derived { @@ -398,12 +416,19 @@ func (a *Aggregator) Buckets() []business.AuditAggregateBucket { // does on baseline amd64, so the result matches the Postgres of the same // architecture to the last bit. Forcing either rounding would match only one. func PercentileCont(values []float64, p float64) float64 { - sorted := append([]float64(nil), values...) - sort.Float64s(sorted) - position := p * float64(len(sorted)-1) + return PercentileContInPlace(append([]float64(nil), values...), p) +} + +// PercentileContInPlace is PercentileCont for a caller that owns values and +// does not need their order: it sorts them where they lie instead of holding +// a second copy of every sample, which a budget that counts one copy would not +// have counted. values must not be empty. +func PercentileContInPlace(values []float64, p float64) float64 { + sort.Float64s(values) + position := p * float64(len(values)-1) first := math.Floor(position) second := math.Ceil(position) - low, high := sorted[int(first)], sorted[int(second)] + low, high := values[int(first)], values[int(second)] if first == second { return low } diff --git a/module/services/accounts/code/pkg/auditstore/auditeval/aggregate_bound_test.go b/module/services/accounts/code/pkg/auditstore/auditeval/aggregate_bound_test.go index 08d6af74a..abd09d07e 100644 --- a/module/services/accounts/code/pkg/auditstore/auditeval/aggregate_bound_test.go +++ b/module/services/accounts/code/pkg/auditstore/auditeval/aggregate_bound_test.go @@ -3,6 +3,7 @@ package auditeval import ( "fmt" "math" + "runtime" "testing" "time" @@ -140,3 +141,42 @@ func TestABucketsStateCountsItsKeysAndMetrics(t *testing.T) { "a key's text is held twice, in the bucket and in its identity") require.Equal(t, int64(8000), SampleBytes(1000)) } + +func TestAPercentileIsTakenWithoutACopyOfItsInputs(t *testing.T) { + // The inputs of a percentile are counted once, at 8 bytes each; taking the + // percentile must not hold them a second time. + values := []float64{9, 1, 5, 3, 7} + require.Equal(t, 5.0, PercentileCont(values, 0.5)) + require.Equal(t, []float64{9, 1, 5, 3, 7}, values, "PercentileCont leaves its argument alone") + require.Equal(t, 5.0, PercentileContInPlace(values, 0.5)) + require.Equal(t, []float64{1, 3, 5, 7, 9}, values, "PercentileContInPlace sorts where the inputs lie") + require.Equal(t, 6.0, PercentileContInPlace(values, 0.625)) + + const n = 20000 + aggregator, err := NewAggregator(business.AuditAggregationSpec{ + Metrics: []business.AuditMetric{{Op: "percentile", Field: "payload:n", Percentile: 0.5, Alias: "p50"}}, + }, nil, 0) + require.NoError(t, err) + require.NoError(t, offerWindow(aggregator, 0, n)) + held := aggregator.state.Held() + require.GreaterOrEqual(t, held, SampleBytes(n)) + + var before, after runtime.MemStats + runtime.ReadMemStats(&before) + buckets := aggregator.Buckets() + runtime.ReadMemStats(&after) + require.Equal(t, 9999.5, buckets[0].Metrics["p50"]) + require.Less(t, int64(after.TotalAlloc-before.TotalAlloc), SampleBytes(n)/4, "no copy of the %d inputs was made", n) + require.Equal(t, buckets, aggregator.Buckets(), "and the answer is the same asked again") + require.Equal(t, held, aggregator.state.Held()) +} + +func TestTheRefusalOfAStateBudgetNamesItsBound(t *testing.T) { + budget, err := NewStateBudget(1000) + require.NoError(t, err) + err = budget.Exceeded() + require.ErrorIs(t, err, business.ErrAuditAggregateTooLarge) + require.ErrorContains(t, err, "the bound is 1000 bytes") + require.Zero(t, budget.Held(), "a refusal is not a count") + require.ErrorContains(t, budget.Take(1001), "the bound is 1000 bytes", "Take reports the same one") +} diff --git a/module/services/accounts/code/pkg/auditstore/clickhousestore/aggregate.go b/module/services/accounts/code/pkg/auditstore/clickhousestore/aggregate.go index 517f6b5f5..84669a6fc 100644 --- a/module/services/accounts/code/pkg/auditstore/clickhousestore/aggregate.go +++ b/module/services/accounts/code/pkg/auditstore/clickhousestore/aggregate.go @@ -5,6 +5,7 @@ import ( "sort" "strings" + "accounts/pkg/auditstore/auditeval" "accounts/pkg/business" ) @@ -25,8 +26,9 @@ import ( // Postgres, which no plan fixes: sums of exactly representable values // agree to the bit, as two Postgres plans would; // - percentile_cont's inputs are collected per group and interpolated in -// the service (auditeval.PercentileCont), whose rounding matches -// Postgres's; +// the service (auditeval.PercentileContInPlace), whose rounding matches +// Postgres's. They are the one thing here that grows with the history, so +// the statement bounds them before they leave the server (below); // - a payload key's ->> text is exact for a string, a boolean, an integer // or no value at all. // @@ -40,9 +42,32 @@ import ( // aggregation is evaluated in the service over the deduplicated rows instead // (auditeval), so the answer is Postgres's either way. +// The service holds what a statement returns, so the statement does not return +// more than the aggregation's state budget (auditeval.StateBudget) can count. +// It computes, per group, what the service will charge for the group (a bucket's +// keys and metrics, and for each percentile input scannedSampleBytes) and sums +// it over every group in a window function. When the sum passes the budget, or +// any row is inexact, the statement withholds what grows: the keys come back +// empty and the percentile arrays empty, with the totals beside them, so the +// service refuses (or, for an inexact row, evaluates the aggregation itself) +// from the first row without a driver block ever holding the history. Below the +// budget the rows are returned whole and the service counts them again, with +// the same arithmetic. A percentile's array is also cut to one more than the +// budget could hold (groupArray(max_size)), so the server never builds a group's +// array past what the service could take: a group with more samples than that +// has a sample count (exact, from count) which passes the budget alone, never an +// array that was silently truncated into an inexact percentile. + +// scannedSampleBytes is what the service holds for one percentile input while a +// ClickHouse read returns it: the 8 bytes the driver keeps for it in the block it +// decoded, and the 8 of the slice Scan copies out of that block. The percentile +// is taken by sorting that slice in place, which adds no third. +var scannedSampleBytes = 2 * auditeval.SampleBytes(1) + // aggregation is a rendered aggregation statement and how to read its rows: -// per group, the dimensions d0..dn, the count, each metric, each metric's -// sample count, and the inexact-row count. +// per group, the dimensions, the count, each metric, each metric's sample +// count, the count of inexact rows in the whole read, and whether the read's +// state passes the budget. type aggregation struct { statement string args []any @@ -220,7 +245,9 @@ func (s *Store) buildAggregation(pl plan, spec business.AuditAggregationSpec, ty value, sample = "ifNull("+metric.Op+"("+number+"), 0)", "count("+number+")" case "percentile": number := x.number(strings.TrimPrefix(metric.Field, "payload:")) - value, sample = "groupArray("+number+")", "count("+number+")" + // One more than the budget can hold: a group that has them all is + // past the budget, and the sample count says so exactly. + value, sample = "groupArray("+p.count(s.maxSamples()+1)+")("+number+")", "count("+number+")" default: return aggregation{}, false, fmt.Errorf("audit: unhandled metric op %q", metric.Op) } @@ -238,8 +265,62 @@ func (s *Store) buildAggregation(pl plan, spec business.AuditAggregationSpec, ty if len(dims) == 1 && dims[0] == "time" { orderBy = "d0 ASC" } - statement := "SELECT " + strings.Join(selects, ", ") + " FROM (" + source + ") GROUP BY " + strings.Join(groups, ", ") + " ORDER BY " + orderBy - return aggregation{statement: statement, args: p.args, dims: len(dims), metrics: spec.Metrics, aliases: aliases}, true, nil + grouped := "SELECT " + strings.Join(selects, ", ") + " FROM (" + source + ") GROUP BY " + strings.Join(groups, ", ") + return aggregation{statement: s.withheld(p, grouped, len(dims), spec.Metrics, orderBy), args: p.args, dims: len(dims), metrics: spec.Metrics, aliases: aliases}, true, nil +} + +// maxSamples is the most percentile inputs a read can return: the budget at +// what each costs the service. +func (s *Store) maxSamples() int { + return int(s.aggregateMaxBytes / scannedSampleBytes) +} + +// withheld wraps the grouped statement (dimensions d0..dn, cnt, metrics m0.., +// sample counts s0.., inexact) in the gate described above, and orders the +// result. Its columns are k0..kn (the dimensions), cnt, v0.. (the metrics), +// s0.., gate_inexact (inexact rows in the whole read) and gate_over (the read's +// state passes the budget). The cost it sums is the arithmetic of +// auditeval.BucketBytes and of scannedSampleBytes. +func (s *Store) withheld(p *sqlParams, grouped string, dims int, metrics []business.AuditMetric, orderBy string) string { + // BucketBytes of a bucket whose keys are empty, and of one more byte of key. + fixed := auditeval.BucketBytes(make([]string, dims), len(metrics)) + perKeyByte := auditeval.BucketBytes([]string{"x"}, 0) - auditeval.BucketBytes([]string{""}, 0) + lengths := make([]string, dims) + for i := range lengths { + lengths[i] = fmt.Sprintf("length(d%d)", i) + } + cost := p.count(int(fixed)) + " + " + p.count(int(perKeyByte)) + " * (" + strings.Join(lengths, " + ") + ")" + var percentiles []string + for i, metric := range metrics { + if metric.Op == "percentile" { + percentiles = append(percentiles, fmt.Sprintf("s%d", i)) + } + } + if len(percentiles) > 0 { + cost += " + " + p.count(int(scannedSampleBytes)) + " * (" + strings.Join(percentiles, " + ") + ")" + } + + const withhold = "(gate_over OR gate_inexact > 0)" + var selects []string + for i := 0; i < dims; i++ { + selects = append(selects, fmt.Sprintf("if(%s, '', d%d) AS k%d", withhold, i, i)) + } + selects = append(selects, "cnt") + for i, metric := range metrics { + if metric.Op == "percentile" { + selects = append(selects, fmt.Sprintf("if(%s, emptyArrayFloat64(), m%d) AS v%d", withhold, i, i)) + } else { + selects = append(selects, fmt.Sprintf("m%d AS v%d", i, i)) + } + } + for i := range metrics { + selects = append(selects, fmt.Sprintf("s%d", i)) + } + selects = append(selects, + "sum(inexact) OVER () AS gate_inexact", + "(sum("+cost+") OVER () > "+p.count(int(s.aggregateMaxBytes))+") AS gate_over", + ) + return "SELECT " + strings.Join(selects, ", ") + " FROM (" + grouped + ") ORDER BY " + orderBy } // timeBucket is the time dimension: the UTC start of the event's day (the diff --git a/module/services/accounts/code/pkg/auditstore/clickhousestore/aggregate_bound_server_test.go b/module/services/accounts/code/pkg/auditstore/clickhousestore/aggregate_bound_server_test.go new file mode 100644 index 000000000..366af1c80 --- /dev/null +++ b/module/services/accounts/code/pkg/auditstore/clickhousestore/aggregate_bound_server_test.go @@ -0,0 +1,191 @@ +package clickhousestore + +import ( + "context" + "fmt" + "sync" + "testing" + "time" + + "accounts/pkg/auditstore/auditeval" + "accounts/pkg/business" + + "github.com/ClickHouse/clickhouse-go/v2/lib/driver" + "github.com/google/uuid" + "github.com/stretchr/testify/require" +) + +// meteredConn totals what an aggregation's statements return to the service: +// the percentile inputs and the bytes of the keys the driver hands to Scan. It is +// what the store would hold, seen from the service's side of the connection. +type meteredConn struct { + Conn + mu sync.Mutex + inputs int + keyBytes int +} + +func (c *meteredConn) Query(ctx context.Context, query string, args ...any) (driver.Rows, error) { + rows, err := c.Conn.Query(ctx, query, args...) + if err != nil { + return nil, err + } + return &meteredRows{Rows: rows, conn: c}, nil +} + +func (c *meteredConn) reset() { + c.mu.Lock() + defer c.mu.Unlock() + c.inputs, c.keyBytes = 0, 0 +} + +func (c *meteredConn) returned() (inputs, keyBytes int) { + c.mu.Lock() + defer c.mu.Unlock() + return c.inputs, c.keyBytes +} + +type meteredRows struct { + driver.Rows + conn *meteredConn +} + +func (r *meteredRows) Scan(dest ...any) error { + err := r.Rows.Scan(dest...) + r.conn.mu.Lock() + defer r.conn.mu.Unlock() + for _, d := range dest { + switch d := d.(type) { + case *[]float64: + r.conn.inputs += len(*d) + case *string: + r.conn.keyBytes += len(*d) + } + } + return err +} + +// stateOf is what the service counts for an answer, by the arithmetic of the +// read it came from: each bucket, and each percentile input at what it costs the +// service while it is read. +func stateOf(buckets []business.AuditAggregateBucket, spec business.AuditAggregationSpec) int64 { + var held int64 + for _, bucket := range buckets { + held += auditeval.BucketBytes(bucket.Keys, len(spec.Metrics)) + for _, metric := range spec.Metrics { + if metric.Op == "percentile" { + held += bucket.Samples[metric.ResolvedAlias()] * scannedSampleBytes + } + } + } + return held +} + +func TestServerAPercentileReadNeverReturnsMoreThanTheBound(t *testing.T) { + conn, database := serverDatabase(t) + ctx := context.Background() + meter := &meteredConn{Conn: conn} + org, actor := uuid.NewString(), uuid.NewString() + base := time.Now().UTC().Truncate(time.Microsecond).Add(-time.Minute) + f := &fixture{org: org} + const events, groups = 1000, 20 + for i := range events { + f.records = append(f.records, newRecord(t, org, actor, business.EventDocumentRead, "doc", base.Add(-time.Duration(i)*time.Second), + business.RetentionContent, map[string]any{"n": i, "group": fmt.Sprintf("group-%02d", i%groups), "key": fmt.Sprintf("key-%04d", i)})) + } + setup := serverStore(t, conn, database) + require.NoError(t, setup.Ensure(ctx)) + appendBatch(t, setup, testDeployment, f.records...) + read := business.AuditRead{Scope: business.OrganizationAuditScope(org), Query: business.AuditQuery{OrgID: org}} + + storeAt := func(limit int64) *Store { + store, err := New(meter, Config{Database: database, DeploymentID: testDeployment, EventsRetention: 2555 * 24 * time.Hour, + ContentDetailRetention: 30 * 24 * time.Hour, AggregateMaxBytes: limit}) + require.NoError(t, err) + return store + } + percentiles := []business.AuditMetric{ + {Op: "percentile", Field: "payload:n", Percentile: 0.5, Alias: "p50"}, + {Op: "percentile", Field: "payload:n", Percentile: 0.99, Alias: "p99"}, + } + + for name, tc := range map[string]struct { + spec business.AuditAggregationSpec + // inputs is how many percentile inputs a read of the whole history holds. + inputs int + }{ + "one group of every event": {business.AuditAggregationSpec{Metrics: percentiles[:1]}, events}, + "twenty groups, two percentiles each": {business.AuditAggregationSpec{GroupBy: []string{"payload:group"}, Metrics: percentiles}, 2 * events}, + "a group of its own for every event": {business.AuditAggregationSpec{GroupBy: []string{"payload:key"}, Metrics: percentiles[:1]}, events}, + "keys alone, a group for every event": {business.AuditAggregationSpec{GroupBy: []string{"payload:key"}}, 0}, + } { + want := f.referenceAggregate(t, read, tc.spec) + exact := stateOf(want, tc.spec) + + // A budget of exactly what the service counts for the answer returns it, + // as the reference semantics answer it, and the server returned every + // input to do so, and no more. + meter.reset() + got, err := storeAt(exact).AggregateAuditEvents(ctx, read, tc.spec) + require.NoError(t, err, name) + require.Equal(t, want, got, name) + inputs, _ := meter.returned() + require.Equal(t, tc.inputs, inputs, "%s: the inputs the answer is made of", name) + require.LessOrEqual(t, int64(inputs)*scannedSampleBytes, exact, "%s: and they fit the budget", name) + + // One byte less is refused, and the server returned neither an input nor + // a key: the statement withheld them, so neither the driver nor the + // service ever held what the read could not keep. + meter.reset() + _, err = storeAt(exact-1).AggregateAuditEvents(ctx, read, tc.spec) + require.ErrorIs(t, err, business.ErrAuditAggregateTooLarge, name) + inputs, keyBytes := meter.returned() + require.Zero(t, inputs, "%s: no input left the server", name) + require.Zero(t, keyBytes, "%s: no key left the server", name) + + // A budget far under it, however the history is grouped. + meter.reset() + _, err = storeAt(4<<10).AggregateAuditEvents(ctx, read, tc.spec) + require.ErrorIs(t, err, business.ErrAuditAggregateTooLarge, name) + inputs, keyBytes = meter.returned() + require.Zero(t, inputs+keyBytes, "%s: nothing left the server", name) + } + + // One group past the most inputs a budget holds is refused for its count, and + // the server never builds it into an array it returns: of the 1000 inputs + // a 4 KiB budget holds 256. + small := storeAt(4 << 10) + require.Equal(t, 256, small.maxSamples()) + meter.reset() + _, err := small.AggregateAuditEvents(ctx, read, business.AuditAggregationSpec{Metrics: percentiles[:1]}) + require.ErrorIs(t, err, business.ErrAuditAggregateTooLarge) + inputs, _ := meter.returned() + require.Zero(t, inputs) + + // Many groups, each far under the most inputs a budget holds (50 of 1024), + // are refused when their total passes it. + medium := storeAt(16 << 10) + require.Equal(t, 1024, medium.maxSamples()) + meter.reset() + spec := business.AuditAggregationSpec{GroupBy: []string{"payload:group"}, Metrics: percentiles[:1]} + _, err = medium.AggregateAuditEvents(ctx, read, spec) + require.ErrorIs(t, err, business.ErrAuditAggregateTooLarge, "20 groups of 50 inputs, 1000 inputs at 16 bytes and 20 buckets, in 16 KiB") + inputs, _ = meter.returned() + require.Zero(t, inputs) + + // An inexact payload (an integer beyond 64 bits, which the server's JSON + // parser cannot read) makes the service evaluate the aggregation itself, + // whatever the statement withheld on the way: the answer is the reference's. + odd := newRecord(t, org, actor, business.EventDocumentRead, "doc", base.Add(time.Second), + business.RetentionContent, map[string]any{"n": 7, "big": jsonNumber("123456789012345678901234567890")}) + appendBatch(t, setup, testDeployment, odd) + f.records = append(f.records, odd) + inService := 0 + roomy := storeAt(0) + roomy.inService = func() { inService++ } + spec = business.AuditAggregationSpec{GroupBy: []string{"payload:big"}, Metrics: percentiles[:1]} + got, err := roomy.AggregateAuditEvents(ctx, read, spec) + require.NoError(t, err) + require.Equal(t, f.referenceAggregate(t, read, spec), got) + require.Equal(t, 1, inService, "the service evaluated it") +} diff --git a/module/services/accounts/code/pkg/auditstore/clickhousestore/aggregate_bound_test.go b/module/services/accounts/code/pkg/auditstore/clickhousestore/aggregate_bound_test.go new file mode 100644 index 000000000..c52024ef1 --- /dev/null +++ b/module/services/accounts/code/pkg/auditstore/clickhousestore/aggregate_bound_test.go @@ -0,0 +1,290 @@ +package clickhousestore + +import ( + "context" + "fmt" + "regexp" + "strconv" + "testing" + + "accounts/pkg/auditstore/auditeval" + "accounts/pkg/business" + + "github.com/ClickHouse/clickhouse-go/v2/lib/driver" + "github.com/stretchr/testify/require" +) + +// An aggregation in ClickHouse returns, per group, the percentile's inputs, and +// the service holds them while it takes the percentile. What it holds must be +// bounded by the aggregation's state budget before the driver decodes them, not +// only counted after, so the statement is pinned here and the service's side of +// it is pinned against rows a server could return. + +const boundOrg = "22222222-2222-2222-2222-222222222222" + +func boundRead() business.AuditRead { + return business.AuditRead{Scope: business.OrganizationAuditScope(boundOrg), Query: business.AuditQuery{OrgID: boundOrg}} +} + +func boundPlan(t *testing.T) plan { + t.Helper() + read := boundRead() + m, err := auditeval.NewMatcher(read) + require.NoError(t, err) + return newPlan(m, read.Query) +} + +func boundStore(t *testing.T, conn Conn, limit int64) *Store { + t.Helper() + cfg := validConfig() + cfg.AggregateMaxBytes = limit + store, err := New(conn, cfg) + require.NoError(t, err) + return store +} + +var percentileByType = business.AuditAggregationSpec{ + GroupBy: []string{"event_type"}, + Metrics: []business.AuditMetric{{Op: "percentile", Field: "payload:n", Percentile: 0.5, Alias: "p50"}}, +} + +// params maps each bound parameter's name to the text it is sent as. +func params(t *testing.T, args []any) map[string]string { + t.Helper() + out := map[string]string{} + for _, arg := range args { + named, ok := arg.(driver.NamedValue) + require.True(t, ok) + if text, ok := named.Value.(string); ok { + out[named.Name] = text + } + } + return out +} + +func TestAPercentileStatementCapsItsArraysAndGatesItsRows(t *testing.T) { + const limit = 64 << 10 + store := boundStore(t, panicConn{}, limit) + agg, ok, err := store.buildAggregation(boundPlan(t), percentileByType, nil) + require.NoError(t, err) + require.True(t, ok) + bound := params(t, agg.args) + + // Each array is cut to one more than the budget could hold, at what each + // input costs the service: past that a group is past the budget by its count + // alone, and no array the server builds or ships is any longer. + const maxSamples = limit / 16 + require.Equal(t, maxSamples, store.maxSamples()) + sized := regexp.MustCompile(`groupArray\(\{(p\d+):UInt64\}\)\(`).FindStringSubmatch(agg.statement) + require.NotNil(t, sized, "the percentile's array has a size cap: %s", agg.statement) + require.Equal(t, strconv.Itoa(maxSamples+1), bound[sized[1]]) + + // The rows are gated on the read's whole state, as the service counts it: + // each group's bucket, key bytes and inputs, summed over every group, against + // the budget. + require.Contains(t, agg.statement, "sum(") + require.Contains(t, agg.statement, ") OVER ()") + require.Contains(t, agg.statement, "emptyArrayFloat64()") + require.Contains(t, agg.statement, "length(d0)") + counted := map[string]bool{} + for _, text := range bound { + counted[text] = true + } + for name, want := range map[string]int64{ + "the budget": limit, + "a bucket": auditeval.BucketBytes([]string{""}, 1), + "a byte of a key": auditeval.BucketBytes([]string{"x"}, 0) - auditeval.BucketBytes([]string{""}, 0), + "a percentile input": scannedSampleBytes, + } { + require.True(t, counted[strconv.FormatInt(want, 10)], "%s (%d) is bound to the statement", name, want) + } + + // An aggregation of keys alone has no array and is gated on its keys. + agg, ok, err = store.buildAggregation(boundPlan(t), business.AuditAggregationSpec{GroupBy: []string{"payload:k"}}, nil) + require.NoError(t, err) + require.True(t, ok) + require.NotContains(t, agg.statement, "groupArray") + require.Contains(t, agg.statement, ") OVER ()") + require.Contains(t, agg.statement, "length(d0)") +} + +// aggregateConn answers the aggregation statement with rows a server could +// return, in the order the statement selects them: the keys, the count, each +// metric, each metric's sample count, the inexact total, and whether the +// statement withheld its rows for the budget. +type aggregateConn struct { + Conn + statements []string + rows [][]any + served int +} + +func (c *aggregateConn) Query(_ context.Context, query string, _ ...any) (driver.Rows, error) { + c.statements = append(c.statements, query) + return &aggregateRows{conn: c}, nil +} + +type aggregateRows struct { + driver.Rows + conn *aggregateConn + next int +} + +func (r *aggregateRows) Next() bool { + r.next++ + return r.next <= len(r.conn.rows) +} + +func (r *aggregateRows) Scan(dest ...any) error { + r.conn.served++ + row := r.conn.rows[r.next-1] + if len(dest) != len(row) { + return fmt.Errorf("aggregateRows: %d destinations for %d columns", len(dest), len(row)) + } + for i, d := range dest { + switch d := d.(type) { + case *string: + *d = row[i].(string) + case *uint64: + *d = row[i].(uint64) + case *uint8: + *d = row[i].(uint8) + case *float64: + *d = row[i].(float64) + case *[]float64: + *d = append([]float64(nil), row[i].([]float64)...) + default: + return fmt.Errorf("aggregateRows: unsupported destination %T", d) + } + } + return nil +} + +func (r *aggregateRows) Err() error { return nil } +func (r *aggregateRows) Close() error { return nil } + +// inputs is n percentile inputs, 1 to n. +func inputs(n int) []float64 { + out := make([]float64, n) + for i := range out { + out[i] = float64(i + 1) + } + return out +} + +// group is the row of one group of percentileByType: a key, its count, its +// inputs and their number, and the two gate columns. +func group(key string, samples []float64, inexact uint64, over uint8) []any { + return []any{key, uint64(len(samples)), samples, uint64(len(samples)), inexact, over} +} + +func TestAReadTheStatementWithheldIsRefusedFromItsFirstRow(t *testing.T) { + // What a server returns for a read past the budget: the keys and the arrays + // empty, the counts exact, the gate set on every row. The service refuses + // on the first, without reading arrays that are not there, and without + // falling back to evaluating the events itself. + conn := &aggregateConn{rows: [][]any{ + {"", uint64(9_000_000), []float64(nil), uint64(9_000_000), uint64(0), uint8(1)}, + {"", uint64(1), []float64(nil), uint64(1), uint64(0), uint8(1)}, + }} + store := boundStore(t, conn, 64<<10) + _, err := store.AggregateAuditEvents(context.Background(), boundRead(), percentileByType) + require.ErrorIs(t, err, business.ErrAuditAggregateTooLarge) + require.ErrorContains(t, err, "the bound is 65536 bytes") + require.Equal(t, 1, conn.served, "it stopped at the first row") + require.Len(t, conn.statements, 1, "and did not aggregate the events itself") +} + +func TestAnInexactReadIsTheServicesToEvaluateWhateverTheBoundSays(t *testing.T) { + // The statement withholds for an inexact row as for the budget; the service + // then evaluates the aggregation over the events, under its own budget, and + // does not refuse on the statement's account. + conn := &aggregateConn{rows: [][]any{{"", uint64(3), []float64(nil), uint64(3), uint64(1), uint8(1)}}} + state, err := auditeval.NewStateBudget(64 << 10) + require.NoError(t, err) + store := boundStore(t, conn, 64<<10) + buckets, exact, err := store.aggregateInClickHouse(context.Background(), boundPlan(t), percentileByType, nil, state) + require.NoError(t, err) + require.False(t, exact) + require.Nil(t, buckets) + require.Zero(t, state.Held()) +} + +func TestAGroupPastTheBoundIsRefusedEvenIfTheServerReturnsIt(t *testing.T) { + // The service counts what it receives with the arithmetic the statement used, + // so a statement that returned more than it should have is refused too: by + // the count when one metric alone passes the budget, by the budget when the + // groups together do. + const limit = 16 << 10 + store := boundStore(t, panicConn{}, limit) + maxSamples := store.maxSamples() + for name, rows := range map[string][][]any{ + "one group past the most inputs the budget holds": {group("a", inputs(maxSamples+1), 0, 0)}, + "a count that overflows what it is multiplied by": {{"a", uint64(1), []float64(nil), uint64(1 << 62), uint64(0), uint8(0)}}, + } { + conn := &aggregateConn{rows: rows} + store.conn = conn + state, err := auditeval.NewStateBudget(limit) + require.NoError(t, err) + _, _, err = store.aggregateInClickHouse(context.Background(), boundPlan(t), percentileByType, nil, state) + require.ErrorIs(t, err, business.ErrAuditAggregateTooLarge, name) + require.Zero(t, state.Held(), "%s: refused by the count, before any input is counted or sorted", name) + } + + // A row whose arrays are not the inputs its counts say it has is an error, + // never a percentile of what is there. + for name, row := range map[string][]any{ + "withheld arrays, no gate": {"a", uint64(5), []float64(nil), uint64(5), uint64(0), uint8(0)}, + "a truncated array": {"a", uint64(5), inputs(4), uint64(5), uint64(0), uint8(0)}, + } { + store.conn = &aggregateConn{rows: [][]any{row}} + state, err := auditeval.NewStateBudget(limit) + require.NoError(t, err) + _, _, err = store.aggregateInClickHouse(context.Background(), boundPlan(t), percentileByType, nil, state) + require.ErrorContains(t, err, "percentile inputs returned", name) + require.NotErrorIs(t, err, business.ErrAuditAggregateTooLarge, name) + } +} + +func TestManyGroupsUnderThePerGroupBoundAreBoundedInTotal(t *testing.T) { + const ( + limit = 16 << 10 + groups = 30 + each = 100 + ) + store := boundStore(t, panicConn{}, limit) + require.Less(t, each, store.maxSamples(), "every group is far under the most inputs one metric may have") + rows := make([][]any, groups) + for i := range rows { + rows[i] = group(fmt.Sprintf("group-%02d", i), inputs(each), 0, 0) + } + one := auditeval.BucketBytes([]string{"group-00"}, 1) + auditeval.SampleBytes(each)*2 + + // Handed every group, as a server that did not gate would, the service + // refuses once their total passes the budget, and counts no more than the + // group that passed it. + conn := &aggregateConn{rows: rows} + store.conn = conn + state, err := auditeval.NewStateBudget(limit) + require.NoError(t, err) + _, _, err = store.aggregateInClickHouse(context.Background(), boundPlan(t), percentileByType, nil, state) + require.ErrorIs(t, err, business.ErrAuditAggregateTooLarge) + fits := int(limit / one) + require.Equal(t, fits+1, conn.served, "it stopped at the group that passed the bound") + require.Less(t, conn.served, groups) + require.Equal(t, int64(fits+1)*one, state.Held()) + + // The same groups, fewer of them, are answered, and every input is counted: + // the bucket's own, and what each of the inputs costs while it is read. + conn = &aggregateConn{rows: rows[:fits]} + store.conn = conn + state, err = auditeval.NewStateBudget(limit) + require.NoError(t, err) + buckets, exact, err := store.aggregateInClickHouse(context.Background(), boundPlan(t), percentileByType, nil, state) + require.NoError(t, err) + require.True(t, exact) + require.Len(t, buckets, fits) + require.Equal(t, int64(fits)*one, state.Held()) + require.Equal(t, 50.5, buckets[0].Metrics["p50"]) + require.EqualValues(t, each, buckets[0].Samples["p50"]) +} diff --git a/module/services/accounts/code/pkg/auditstore/clickhousestore/clickhousestore.go b/module/services/accounts/code/pkg/auditstore/clickhousestore/clickhousestore.go index 193b9a183..58dc84380 100644 --- a/module/services/accounts/code/pkg/auditstore/clickhousestore/clickhousestore.go +++ b/module/services/accounts/code/pkg/auditstore/clickhousestore/clickhousestore.go @@ -85,10 +85,11 @@ type Config struct { // ExportMaxBytes bounds the events one export gathers, counted by their // text (auditeval.Event.Size). Zero is business.AuditExportMaxBytes. ExportMaxBytes int64 - // AggregateMaxBytes bounds the state one aggregation keeps in the service — - // its buckets, and the samples and distinct values of the aggregation the - // service evaluates itself — counted by auditeval.StateBudget. Zero is - // business.AuditAggregateMaxBytes. + // AggregateMaxBytes bounds the state one aggregation holds in the service — + // its buckets, its percentile samples, and the distinct values of the + // aggregation the service evaluates itself — counted by + // auditeval.StateBudget. The statement ClickHouse runs for the aggregation + // returns no more than this either. Zero is business.AuditAggregateMaxBytes. AggregateMaxBytes int64 } diff --git a/module/services/accounts/code/pkg/auditstore/clickhousestore/reader.go b/module/services/accounts/code/pkg/auditstore/clickhousestore/reader.go index 70c403bd3..704585745 100644 --- a/module/services/accounts/code/pkg/auditstore/clickhousestore/reader.go +++ b/module/services/accounts/code/pkg/auditstore/clickhousestore/reader.go @@ -322,11 +322,12 @@ func (s *Store) ExportAuditEvents(ctx context.Context, read business.AuditRead) // AggregateAuditEvents implements business.AuditReader: one GROUP BY in // ClickHouse when it is exact for this read (aggregate.go), and otherwise the // deduplicated, matched events aggregated in the service. Either way the state -// the service keeps for the answer (the buckets, and in the second the -// percentile samples and distinct values) is counted against the store's -// aggregation bound, and the read gives up with -// business.ErrAuditAggregateTooLarge, never a partial answer, as soon as it -// passes it. +// the service holds for the answer (the buckets and the percentile samples, and +// in the second the distinct values) is counted against the store's aggregation +// bound, and the read gives up with business.ErrAuditAggregateTooLarge, never a +// partial answer, as soon as it passes it. In the first, ClickHouse itself +// withholds the keys and samples of a read that would pass the bound, so the +// service never receives them to count (aggregate.go). func (s *Store) AggregateAuditEvents(ctx context.Context, read business.AuditRead, spec business.AuditAggregationSpec) ([]business.AuditAggregateBucket, error) { m, err := auditeval.NewMatcher(read) if err != nil { @@ -364,21 +365,25 @@ func (s *Store) AggregateAuditEvents(ctx context.Context, read business.AuditRea // aggregateInClickHouse runs the aggregation statement. exact is false when a // row it grouped is one ClickHouse cannot read as Postgres does, and the -// buckets are then discarded. Each bucket it keeps is counted against state. +// buckets are then discarded. Each bucket it keeps, and each percentile input it +// receives, is counted against state; the statement itself returns nothing once +// the read would pass the state's limit (aggregate.go), so the service neither +// holds nor sorts more than the limit before it refuses. func (s *Store) aggregateInClickHouse(ctx context.Context, pl plan, spec business.AuditAggregationSpec, types business.AuditEventTypeIndex, state *auditeval.StateBudget) ([]business.AuditAggregateBucket, bool, error) { agg, ok, err := s.buildAggregation(pl, spec, types) if err != nil || !ok { return nil, err == nil, err } var out []business.AuditAggregateBucket - var inexact uint64 + inexact := false err = s.scan(ctx, agg.statement, agg.args, func(rows driver.Rows) (bool, error) { keys := make([]string, agg.dims) - var count, rowInexact uint64 + var count, readInexact uint64 + var over uint8 values := make([]float64, len(agg.metrics)) arrays := make([][]float64, len(agg.metrics)) samples := make([]uint64, len(agg.metrics)) - dest := make([]any, 0, agg.dims+2+2*len(agg.metrics)) + dest := make([]any, 0, agg.dims+3+2*len(agg.metrics)) for i := range keys { dest = append(dest, &keys[i]) } @@ -393,15 +398,37 @@ func (s *Store) aggregateInClickHouse(ctx context.Context, pl plan, spec busines for i := range samples { dest = append(dest, &samples[i]) } - dest = append(dest, &rowInexact) + dest = append(dest, &readInexact, &over) if err := rows.Scan(dest...); err != nil { return false, fmt.Errorf("clickhouse audit store: read aggregation: %w", err) } - inexact += rowInexact - if inexact > 0 { + // Both are totals of the whole read, repeated on every row, so the first + // row decides. An inexact row is the service's to evaluate, whatever the + // bound says of this statement's own state; the statement has withheld + // its keys and arrays in either case, and neither is read. + if readInexact > 0 { + inexact = true return false, nil } - if err := state.Take(auditeval.BucketBytes(keys, len(agg.metrics))); err != nil { + if over > 0 { + return false, state.Exceeded() + } + held := auditeval.BucketBytes(keys, len(agg.metrics)) + for i, metric := range agg.metrics { + if metric.Op != "percentile" { + continue + } + // A count past the most inputs the budget holds is past it alone, + // and keeps the products below in range. + if samples[i] > uint64(s.maxSamples()) { + return false, state.Exceeded() + } + if uint64(len(arrays[i])) != samples[i] { + return false, fmt.Errorf("clickhouse audit store: read aggregation: %d percentile inputs returned of %d counted", len(arrays[i]), samples[i]) + } + held += int64(samples[i]) * scannedSampleBytes + } + if err := state.Take(held); err != nil { return false, err } bucket := business.AuditAggregateBucket{ @@ -418,7 +445,9 @@ func (s *Store) aggregateInClickHouse(ctx context.Context, pl plan, spec busines continue } if metric.Op == "percentile" { - bucket.Metrics[alias] = auditeval.PercentileCont(arrays[i], metric.Percentile) + // Sorted where it lies: the slice is this row's own, and a copy + // would hold every input once more than state counted. + bucket.Metrics[alias] = auditeval.PercentileContInPlace(arrays[i], metric.Percentile) } else { bucket.Metrics[alias] = values[i] } @@ -436,7 +465,7 @@ func (s *Store) aggregateInClickHouse(ctx context.Context, pl plan, spec busines if err != nil { return nil, false, err } - if inexact > 0 { + if inexact { return nil, false, nil } return out, true, nil From 3532058682ac596b906a334c095e4a98d7b428a1 Mon Sep 17 00:00:00 2001 From: Cristopher Fiorini <301773486+crisfiorini-oss@users.noreply.github.com> Date: Wed, 7 Oct 2026 13:06:14 -0400 Subject: [PATCH 52/69] fix(vault): declare its endpoint private within the module --- .../generated/service-topology.json | 2 +- module/services/vault/service.codefly.yaml | 3 +- vault_binding_test.go | 42 +++++++++++++++++++ 3 files changed, 44 insertions(+), 3 deletions(-) diff --git a/module/deployment/generated/service-topology.json b/module/deployment/generated/service-topology.json index 6164033d8..360262947 100644 --- a/module/deployment/generated/service-topology.json +++ b/module/deployment/generated/service-topology.json @@ -149,7 +149,7 @@ { "name": "http", "api": "CODEFLY_API_HTTP", - "visibility": "ENDPOINT_VISIBILITY_MODULE", + "visibility": "ENDPOINT_VISIBILITY_PRIVATE", "port": 8200 } ] diff --git a/module/services/vault/service.codefly.yaml b/module/services/vault/service.codefly.yaml index 75e87fb52..12bfc39a6 100644 --- a/module/services/vault/service.codefly.yaml +++ b/module/services/vault/service.codefly.yaml @@ -11,8 +11,7 @@ secret-service-configurations: - key: vault_token endpoints: - name: http - visibility: internal - allow-modules: ["*"] + visibility: private spec: transit-key: api-keys deployment: diff --git a/vault_binding_test.go b/vault_binding_test.go index 405f57259..a3992f9ce 100644 --- a/vault_binding_test.go +++ b/vault_binding_test.go @@ -6,6 +6,8 @@ import ( "strings" "testing" + basev0 "github.com/codefly-dev/core/generated/go/codefly/base/v0" + "github.com/codefly-dev/core/resources" "gopkg.in/yaml.v3" ) @@ -31,6 +33,46 @@ const ( meshGroupDefaults = "module/configurations/local/internal-transport.env" ) +// Vault is a dependency of accounts within this module, never an exported +// endpoint. Declare that boundary on the service too: older supported CLIs +// make an omitted module-interface endpoint private without removing its +// service's allow-list, which newer SDKs correctly reject as invalid. +func TestVaultEndpointIsPrivateAndResolvesOnlyWithinItsModule(t *testing.T) { + t.Parallel() + data, err := os.ReadFile(vaultServiceManifest) + if err != nil { + t.Fatal(err) + } + var service resources.Service + if err := yaml.Unmarshal(data, &service); err != nil { + t.Fatal(err) + } + if len(service.Endpoints) != 1 || service.Endpoints[0].Name != "http" { + t.Fatalf("unexpected vault endpoints: %#v", service.Endpoints) + } + endpoint := service.Endpoints[0] + if endpoint.Visibility != resources.VisibilityPrivate || len(endpoint.AllowModules) != 0 { + t.Fatalf("vault/http must declare private visibility without allow-modules; got %q, %v", endpoint.Visibility, endpoint.AllowModules) + } + endpoint.Module, endpoint.Service = "saas-starter", "vault" + declaration, err := endpoint.Proto() + if err != nil { + t.Fatal(err) + } + dependencies := []*resources.ServiceDependency{{ + Module: "saas-starter", Name: "vault", + Endpoints: []*resources.EndpointReference{{Name: "http"}}, + }} + mappings := []*basev0.NetworkMapping{{Endpoint: declaration}} + resolved, err := resources.ResolveDependencyNetworkMappings("saas-starter", dependencies, mappings) + if err != nil || len(resolved) != 1 { + t.Fatalf("same-module vault dependency: resolved %d mappings, error %v", len(resolved), err) + } + if _, err := resources.ResolveDependencyNetworkMappings("example", dependencies, mappings); err == nil { + t.Fatal("vault/http was handed to a different module") + } +} + // vaultDurableRenderFloor is the first vault agent release whose deployed render // is durable: `vault server` with integrated raft storage on a retained // PersistentVolumeClaim, auto-unsealed by the seal the environment supplies, From 23eb5192fe637bfcaac269bcf352f87c2735b224 Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Wed, 7 Oct 2026 13:13:06 -0400 Subject: [PATCH 53/69] fix(audit): history copy sets aside an event the store refuses and keeps the details requirement for short retention The rewrite of an event held without details (feafbdd8) returned any PermanentRowRejection from the store as a fatal error. An event whose events row landed and whose details row the store refuses for good was then written again on every run: each run stopped at its window, never reached the later windows or partitions, and added an events row and a locked archive object. The copy now judges a refusal as the relay does: - write archives the batch once, then sends the store the rest of the batch again without the events the store named, until it accepts. A refusal that names no event of the batch is not evidence about any of them and fails the run like any other failure. - A refused event is that event's verification failure alone; every other event, window and partition is copied. It is counted as neither copied nor rewritten, and the store's own message is not printed (operator output carries no provider errors). - A write for missing details is made once. inspect no longer calls an event writable when the store holds more than one copy of it: a second copy is the proof that a write was made and did not supply the details (refused for good, or accepted and not readable). The failure stays, with the copy count as its reason, so a refused row costs one more events row and one archive object on the first rewrite and nothing on later runs. An event is never written twice in one run: there is one write pass, one read-back, then verification, and a problem for an event written by the run says so. The two-day margin cut the requirement for content details out of the window itself. An event in the last two days with an events row and no details was neither rewritten nor failed, while an event with no stored copy was written with its details; and a content retention of 1 or 2 days put the cutoff at or after now, so details were never required. The rule now counts from what the store does. It expires details by the event's UTC day, so it is certain to hold them until the start of that day plus the retention (detailsHeldUntil); until then an event without them fails, for any retention, so one day still requires today's events. For the margin after that instant the store may still hold them: the event is written for them and never failed. Beyond it, nothing is written or required. Tests that fail on the previous head: the reviewer's scenario (27 events, one details row refused for good, as ClickHouse isolates it and as BigQuery reports it, counting events rows and archive objects over five runs); a refused events row; details accepted and never readable; the rule at every boundary for retentions of 1, 2 and 7 days and three clock times (a margin of 0 or 3 days, a flipped or non-strict comparison and a rule without the UTC day each fail it); the margin and the short retention through whole runs. A refusal naming no event of the batch is pinned as a failed run. AUDIT_OPERATIONS.md: drop "copy under another deployment" from what an operator sees (both readers read this deployment only), say who removes a wrong copy (the warehouse administrator; the service user cannot), and that each rewrite archives the batch again, which archive readers deduplicate by event id. Co-Authored-By: Claude Sonnet 5.5 --- module/AUDIT_OPERATIONS.md | 56 ++- module/services/accounts/AGENTS.md | 4 +- .../accounts/code/pkg/auditops/history.go | 7 +- .../code/pkg/business/audit_history.go | 200 ++++++-- .../code/pkg/business/audit_history_test.go | 460 +++++++++++++++++- 5 files changed, 638 insertions(+), 89 deletions(-) diff --git a/module/AUDIT_OPERATIONS.md b/module/AUDIT_OPERATIONS.md index c1c9bd238..aefabab7b 100644 --- a/module/AUDIT_OPERATIONS.md +++ b/module/AUDIT_OPERATIONS.md @@ -38,22 +38,52 @@ are not verified before new rows are written. A verification failure leaves the partitions in place. An event is appended when the warehouse holds none of it, and again when it holds -it without details that verification requires and a write supplies: a content-class -event inside the content window whose details row is not there. Both adapters write -an event's events row before its details row, so a run whose details write failed -leaves exactly that behind, and the next run appends the event again, events row -and details row. The details are attached to every copy of the event by its id, so -the copy already in the warehouse is complete too. A text-mode run counts these as -`rewritten` in its progress lines, apart from the events `copied`. Content-class -events older than the content window may have no details (the warehouse expired -them) and are not appended. +it without the details of a content-class event. Both adapters write an event's +events row before its details row, so a run whose details write failed leaves +exactly that behind, and the next run appends the event again, events row and +details row. The details are attached to every copy of the event by its id, so the +copy already in the warehouse is complete too. A text-mode run counts these as +`rewritten` in its progress lines, apart from the events `copied`. + +The warehouse expires details by the UTC day of the event, so it is certain to hold +an event's details until the start of that day plus `AUDIT_CONTENT_RETENTION_DAYS`. +Until then an event without them fails verification, whatever the retention: with +1 day, the events of today are still required. For the two days after that +instant the warehouse may still hold them: the event is appended to supply them +and never fails for lacking them. After that it is neither appended nor required +to have them. + +An event is appended again for missing details **once**. When the warehouse holds +more than one copy of an event and still no details, an earlier write did not +supply them (the warehouse refused the row, or accepted it and does not show it), +and another would add an events row and an archive object and change nothing. The +event stays a verification failure, and the report says how many copies the +warehouse holds. An event written in a run is never written again in that run, +whatever the read-back shows. + +An event whose row the warehouse refuses for good (a row too large, a value its +column cannot hold) fails verification **by itself**: the rest of its batch and of +the run are copied, and the report names the event (the warehouse's own message is +not printed). If the events row was refused, nothing of the event is stored and +every run tries it again, archiving that event alone each time. If only the details +row was refused, the next run writes the event once more and later runs do not. +Either way the partitions are not dropped: the command has no way to skip an event. + +Each write archives its batch again: the archive never overwrites an object, so a +rewrite adds one that holds the event a second time. That is safe because archive +readers deduplicate by event id, as the warehouse's readers do. A copy that is wrong in itself is **not** appended again: the warehouse is append-only and every copy is verified, so the wrong copy would stay. These are -verification failures for the operator to look at: a copy stored under another -deployment id, with an envelope or retention class the row does not have, with a -details hash that is not the row's, or a security-class copy without its details -(they live in that copy's own row). +verification failures for the operator to look at: a copy with an envelope or +retention class the row does not have, with a details hash that is not the row's, +or a security-class copy without its details (they live in that copy's own row). A +copy stored under another deployment id fails in the same way, but both readers +read this deployment's rows only, so it does not occur with them. The service's +warehouse user cannot delete or alter rows (ClickHouse grants no `ALTER`/`DELETE`; +BigQuery no DML), so neither can this command: the warehouse administrator removes +the wrong copy with credentials of their own, and `-verify-only` then checks the +partition again. Until that is done the partition stays undroppable. ### Which partitions diff --git a/module/services/accounts/AGENTS.md b/module/services/accounts/AGENTS.md index 6d8f3f81d..bed96baf0 100644 --- a/module/services/accounts/AGENTS.md +++ b/module/services/accounts/AGENTS.md @@ -118,7 +118,9 @@ survives a restart and reaches every replica, and is only served when it is live event — and verifies them by reading them back. It reads back before it writes, and writes an event again when the store holds it without details a write supplies (`inspect`, the one rule `verify` shares), so a rerun finishes - an interrupted run, a failed details write included. Only with `-confirm-drop`, + an interrupted run, a failed details write included. That rewrite is made once + per event (a second copy in the store means it was), and an event the store + refuses for good fails verification alone, as the relay sets one row aside. Only with `-confirm-drop`, after a verification pass in the same run and a recount showing nothing changed, does it drop the copied partitions, named one by one through `audit_events_drop_verified_partitions` (migration 22), which recounts each diff --git a/module/services/accounts/code/pkg/auditops/history.go b/module/services/accounts/code/pkg/auditops/history.go index af89462e4..88fae1bd6 100644 --- a/module/services/accounts/code/pkg/auditops/history.go +++ b/module/services/accounts/code/pkg/auditops/history.go @@ -24,9 +24,10 @@ // It is resumable: a run reads each window's rows, reads back what the store // holds, appends only the events whose stored copies are not complete — those // the store lacks, and those it holds without details a write supplies, such as -// a content-class event whose details write failed — reads the window back again -// and verifies every event, so a run after an interrupted one finishes what it -// left. -verify-only copies nothing, for a re-check once the store has caught +// a content-class event whose details write failed, once — reads the window back +// again and verifies every event, so a run after an interrupted one finishes what +// it left. An event the store refuses for good fails verification alone; the rest +// is copied. -verify-only copies nothing, for a re-check once the store has caught // up. // // Exit status: diff --git a/module/services/accounts/code/pkg/business/audit_history.go b/module/services/accounts/code/pkg/business/audit_history.go index 464e76919..ec2cf875b 100644 --- a/module/services/accounts/code/pkg/business/audit_history.go +++ b/module/services/accounts/code/pkg/business/audit_history.go @@ -122,8 +122,10 @@ type AuditHistoryCopyConfig struct { Types DeclaredAuditEventTypeReader // DeploymentID stamps every copied record, as the relay's. DeploymentID string - // ContentRetention is the details window: a content-class event older than - // it has no details left to verify in the store. + // ContentRetention is the details window: the store keeps a content-class + // event's details for this long (a whole number of days, at least one). The + // copy requires the details of an event the store is certain to still hold + // when it is run (detailsHeldUntil) and tries to supply those of one it may. ContentRetention time.Duration // BatchSize is the most events one copied batch carries (default // DefaultAuditRelayBatchSize). @@ -168,9 +170,12 @@ type AuditHistoryPartitionReport struct { // them. Copied int // Rewritten is how many events this run wrote again because the store held - // them without details that verification requires and that a write supplies - // (a content-class event whose details write was lost); the rest were - // already complete. + // them without details that a write supplies (a content-class event whose + // details write was lost); the rest were already complete. An event is + // written again only while the store holds one copy of it, so a rewrite that + // does not supply the details is made once, not on every run. Events the + // store refused for good are in neither Copied nor Rewritten: they are + // problems. Rewritten int // Orgs counts the partition's events per organization ("" is the // platform's). @@ -231,10 +236,13 @@ const auditHistoryPageSize = 1000 // time: a day keeps the read-back of a busy deployment in memory. const auditHistoryWindow = 24 * time.Hour -// auditHistoryDetailsMargin is how far inside the content window a -// content-class event's details must still be in the store: the store expires -// details by whole day partitions, so the last day or two of the window may -// already be gone. +// auditHistoryDetailsMargin is how far past the instant the store is certain to +// hold an event's details the copy still tries to supply them. The store expires +// details by the day of the event (BigQuery by day partition, ClickHouse by a TTL +// merge that runs after the TTL), so for a day or two after that instant it may +// still hold them. The copy writes in that span — a missing event is written with +// its details, and so is one held without — but never fails verification there: +// the details may be legitimately gone. const auditHistoryDetailsMargin = 2 * 24 * time.Hour // AuditHistoryCopy is the one-time history copy. @@ -277,7 +285,9 @@ func NewAuditHistoryCopy(cfg AuditHistoryCopyConfig) (*AuditHistoryCopy, error) // events the store lacks, and those it holds without the details a write // supplies — reads the window back again and verifies every event, so a run // after an interrupted one finishes what that one started, a failed details -// write included. A verification failure blocks the drop and returns +// write included. An event the store refuses for good (a PermanentRowRejection) +// is a verification failure of that event alone: the rest of its batch and of the +// run carry on. A verification failure blocks the drop and returns // ErrAuditHistoryUnverified; a run with nothing to verify returns // ErrAuditHistoryNothingToVerify; a drop that does not happen returns // ErrAuditHistoryDropRefused with the reason. @@ -460,6 +470,12 @@ func (c *AuditHistoryCopy) copyPartition(ctx context.Context, resolver *AuditEve // copyWindow reads a window's rows, reads back what the store holds of it, // writes what is missing or incomplete, and verifies every row. +// +// Each event is written at most once by this call, whatever the read-back shows +// afterwards: there is one write pass, then one read-back, then verification. An +// event the store refuses for good is set aside for the rest of the call — the +// relay's rule — and reported as that event's problem; the other events of its +// batch are written without it. func (c *AuditHistoryCopy) copyWindow(ctx context.Context, resolver *AuditEventResolver, from, to time.Time, opts AuditHistoryCopyOptions, report *AuditHistoryPartitionReport, digest hash.Hash) error { records, err := c.readWindow(ctx, resolver, from, to) if err != nil { @@ -472,31 +488,39 @@ func (c *AuditHistoryCopy) copyWindow(ctx context.Context, resolver *AuditEventR if err != nil { return err } + written := map[string]bool{} // event id: written by this call + refused := map[string]bool{} // event id: the store refused it for good if !opts.VerifyOnly { - // An event is written when verification would fail for want of a write: - // the store holds none of it, or holds it without details a write - // supplies. The id being in the store says nothing: both adapters write - // an event's row before its details, so a failed details write leaves - // the row behind, and skipping on its id would block verification for - // good. + // An event is written when a write adds what verification would fail + // for, or what the store may still hold: the store holds none of it, or + // holds it without details a write supplies. The id being in the store + // says nothing: both adapters write an event's row before its details, + // so a failed details write leaves the row behind, and skipping on its + // id would block verification for good. var pending []AuditRecord var rewrite []bool // parallel to pending: the store already held the event for _, record := range records { copies := stored[record.Entry.ID] - if problem, writable := c.inspect(record, copies); problem != "" && writable { + if _, writable := c.inspect(record, copies); writable { pending, rewrite = append(pending, record), append(rewrite, len(copies) > 0) } } if len(pending) > 0 { for start := 0; start < len(pending); start += c.cfg.BatchSize { end := min(start+c.cfg.BatchSize, len(pending)) - if err := c.write(ctx, pending[start:end]); err != nil { + rejected, err := c.write(ctx, pending[start:end]) + if err != nil { return err } - for _, again := range rewrite[start:end] { - if again { + for i, record := range pending[start:end] { + id := record.Entry.ID + written[id] = true + switch { + case rejected[id]: + refused[id] = true + case rewrite[start+i]: report.Rewritten++ - } else { + default: report.Copied++ } } @@ -520,10 +544,17 @@ func (c *AuditHistoryCopy) copyWindow(ctx context.Context, resolver *AuditEventR org := record.Entry.OrgID count := report.Orgs[org] count.Postgres++ - if problem := c.verify(record, stored[record.Entry.ID]); problem != "" { + id := record.Entry.ID + if problem := c.verify(record, stored[id]); problem != "" { + switch { + case refused[id]: + problem = auditHistoryRefusal + "; " + problem + case written[id]: + problem = "written by this run, but the read-back finds it incomplete: " + problem + } report.Failures++ if len(report.Problems) < maxAuditHistoryProblems { - report.Problems = append(report.Problems, fmt.Sprintf("event %s: %s", record.Entry.ID, problem)) + report.Problems = append(report.Problems, fmt.Sprintf("event %s: %s", id, problem)) } } else { count.Verified++ @@ -580,9 +611,21 @@ func (c *AuditHistoryCopy) readBack(ctx context.Context, from, to time.Time) (ma return stored, nil } -// write delivers one batch as the relay does: the archive first, then the -// store. A failure leaves the rest of the window for the next run. -func (c *AuditHistoryCopy) write(ctx context.Context, records []AuditRecord) error { +// auditHistoryRefusal is what a problem says of an event the store refused for +// good. The store's own words are left out of it: the operator's output carries no +// provider errors. +const auditHistoryRefusal = "the store refused the event's row for good" + +// write delivers one batch as the relay does: the archive first, whole and once, +// then the store. A store that reports rows it refused for good (a +// PermanentRowRejection naming an event of the batch) is sent the rest again +// without them, as many times as it names another, so one event never holds the +// others back, and returns those events by id. The batch is archived once: the +// archive holds the refused events whole, and a store write is repeated without +// them as the relay's is. A refusal that names no event of the batch, or any +// other failure, is returned as an error and leaves the rest of the window for +// the next run. +func (c *AuditHistoryCopy) write(ctx context.Context, records []AuditRecord) (map[string]bool, error) { batch := AuditBatch{ ID: c.cfg.NewBatchID(), DeploymentID: c.cfg.DeploymentID, @@ -590,19 +633,44 @@ func (c *AuditHistoryCopy) write(ctx context.Context, records []AuditRecord) err Records: records, } if err := c.cfg.Archive.WriteAuditBatch(ctx, batch); err != nil { - return fmt.Errorf("archive batch %s: %w", batch.ID, err) + return nil, fmt.Errorf("archive batch %s: %w", batch.ID, err) } - if err := c.cfg.Store.AppendAuditBatch(ctx, batch); err != nil { - return fmt.Errorf("append batch %s: %w", batch.ID, err) + refused := map[string]bool{} + for todo := records; len(todo) > 0; { + if err := ctx.Err(); err != nil { + return nil, fmt.Errorf("append batch %s: %w", batch.ID, err) + } + attempt := batch + attempt.Records = todo + err := c.cfg.Store.AppendAuditBatch(ctx, attempt) + if err == nil { + return refused, nil + } + rejections := map[string]bool{} + for _, rejection := range PermanentRowRejections(err) { + rejections[rejection.EventID] = true + } + var kept []AuditRecord + for _, record := range todo { + if id := record.Entry.ID; rejections[id] { + refused[id] = true + } else { + kept = append(kept, record) + } + } + if len(kept) == len(todo) { + return nil, fmt.Errorf("append batch %s: %w", batch.ID, err) + } + todo = kept } - return nil + return refused, nil } // verify reports what is wrong with the store's copies of record, or "". // Every copy must carry the record's envelope, retention class and details // hash, and whatever details it carries must hash to it; a security-class -// event's details must be there, and a content-class event's while it is -// inside the content window. +// event's details must be there, and a content-class event's while the store +// is certain to hold them (detailsHeldUntil). func (c *AuditHistoryCopy) verify(record AuditRecord, copies []StoredAuditEvent) string { problem, _ := c.inspect(record, copies) return problem @@ -610,15 +678,18 @@ func (c *AuditHistoryCopy) verify(record AuditRecord, copies []StoredAuditEvent) // inspect is the one rule for the store's copies of record, shared by what // writes and what verifies: what verification finds wrong with them, and -// whether writing the event again would remove it. +// whether writing the event again would help. // -// A write does when the store holds none of the event, or holds it without the -// details of a content-class event inside the content window: both adapters -// keep those in the details table and attach them to every copy of the event by -// its id, so one more details row completes the copies already there, and the -// events row the write appends beside them carries the same envelope. +// A write helps when the store holds none of the event, or holds it without the +// details of a content-class event: both adapters keep those in the details +// table and attach them to every copy of the event by its id, so one more +// details row completes the copies already there, and the events row the write +// appends beside them carries the same envelope. The details are a verification +// failure while the store is certain to hold them (the event is before +// detailsHeldUntil); for auditHistoryDetailsMargin after that they may still be +// held, so the write is made without a failure when they are not readable. // -// A write does not when a copy is wrong in itself — stored under another +// A write does not help when a copy is wrong in itself — stored under another // deployment, with an envelope or a retention class the row does not have, with a // details hash that is not the row's, or with details that do not hash to it — // because the store is append-only and every copy is verified, so the wrong copy @@ -627,6 +698,12 @@ func (c *AuditHistoryCopy) verify(record AuditRecord, copies []StoredAuditEvent) // a new row does not complete. Such a copy is a verification failure for an // operator to look at. When copies differ in what is wrong, the reason that no // write removes is the one returned. +// +// A write for missing details is made once: when the store holds more than one +// copy of the event, a write has already been made and did not supply them +// (refused for good, or written and not readable), and another would add a copy +// and an archive object and conclude the same. The failure stays, with the count +// of copies as its reason. func (c *AuditHistoryCopy) inspect(record AuditRecord, copies []StoredAuditEvent) (problem string, writable bool) { if len(copies) == 0 { return "missing from the store", true @@ -634,17 +711,26 @@ func (c *AuditHistoryCopy) inspect(record AuditRecord, copies []StoredAuditEvent for _, stored := range copies { reason, fixable := c.inspectCopy(record, stored) switch { - case reason == "": - case !fixable: + case reason != "" && !fixable: return reason, false - case problem == "": - problem = reason + case fixable: + writable = true + if problem == "" { + problem = reason + } + } + } + if writable && len(copies) > 1 { + if problem != "" { + problem = fmt.Sprintf("%s; the store holds %d copies, so an earlier write did not supply them, and the event is not written again", problem, len(copies)) } + return problem, false } - return problem, problem != "" + return problem, writable } -// inspectCopy is inspect for one stored copy. +// inspectCopy is inspect for one stored copy. It returns writable without a +// problem for details the store may still hold: written for, never failed. func (c *AuditHistoryCopy) inspectCopy(record AuditRecord, stored StoredAuditEvent) (problem string, writable bool) { want, got := record.Entry, stored.Entry switch { @@ -664,12 +750,34 @@ func (c *AuditHistoryCopy) inspectCopy(record AuditRecord, stored StoredAuditEve return "stored details do not hash to the row's hash", false case !stored.HasDetails && record.Retention == RetentionSecurity: return "security-class details are missing", false - case !stored.HasDetails && want.CreatedAt.After(c.cfg.Now().Add(-c.cfg.ContentRetention+auditHistoryDetailsMargin)): + case stored.HasDetails: + return "", false + } + // A content-class copy without details: its row says when the store may have + // expired them. + now, heldUntil := c.cfg.Now(), c.detailsHeldUntil(want.CreatedAt) + switch { + case now.Before(heldUntil): return "content details are missing inside the content window", true + case now.Before(heldUntil.Add(auditHistoryDetailsMargin)): + return "", true } return "", false } +// detailsHeldUntil is the first instant at which the store may have expired the +// details of an event that occurred at t: before it the store holds them. The +// store expires details by the day of the event, so the count starts at the +// start of its UTC day, which no adapter beats: BigQuery drops a day partition +// once its day's start plus the retention has passed, and a ClickHouse TTL does +// not fire before the event plus the retention. It is a function of the +// retention itself, not of a fixed span cut from the window, so a short +// retention keeps its requirement: with one day, every event of today is held. +func (c *AuditHistoryCopy) detailsHeldUntil(t time.Time) time.Time { + t = t.UTC() + return time.Date(t.Year(), t.Month(), t.Day(), 0, 0, 0, 0, time.UTC).Add(c.cfg.ContentRetention) +} + // drop checks that nothing changed since verification and drops the verified // partitions — those, named one by one, and no others. It refuses, returning // why, when a verified partition is gone or has other bounds, when the cutoff diff --git a/module/services/accounts/code/pkg/business/audit_history_test.go b/module/services/accounts/code/pkg/business/audit_history_test.go index d75747724..41424b9b6 100644 --- a/module/services/accounts/code/pkg/business/audit_history_test.go +++ b/module/services/accounts/code/pkg/business/audit_history_test.go @@ -201,16 +201,51 @@ type historyDetailsRow struct { // event's, as the adapters' readers do. failDetails makes the details write of // that (1-based) append fail after its events rows landed: the partial write // either adapter can leave. +// +// It can also refuse rows for good, as both adapters do: refuseDetails names +// events whose details row it refuses (its events row has landed), refuseEvents +// those whose events row it refuses (and so their details row is never sent), each +// reported as a PermanentRowRejection beside the other rows' result. By default +// the rest of the batch is written, as ClickHouse does around a refused row; +// stopRest leaves the rest of the refused table's rows unwritten, as BigQuery +// does for the rows it reports as stopped. unreadable names events whose details +// the store accepts and never reads back, as when their partition is already gone. type splitHistoryStore struct { events []StoredAuditEvent // a content-class row carries no details details []historyDetailsRow appends int failDetails int + + refuseDetails, refuseEvents, unreadable map[string]bool + stopRest bool + // attempts counts the records of every append by event id. + attempts map[string]int +} + +func refusalOf(id, what string) error { + return &PermanentRowRejection{EventID: id, Cause: errors.New(what + " refused")} } func (s *splitHistoryStore) AppendAuditBatch(_ context.Context, batch AuditBatch) error { s.appends++ + if s.attempts == nil { + s.attempts = map[string]int{} + } + var refusals, eventRefusals []error + for _, record := range batch.Records { + s.attempts[record.Entry.ID]++ + if s.refuseEvents[record.Entry.ID] { + eventRefusals = append(eventRefusals, refusalOf(record.Entry.ID, "events row")) + } + } + refusals = append(refusals, eventRefusals...) + if s.stopRest && len(eventRefusals) > 0 { + return errors.Join(append(refusals, errors.New("events rows stopped"))...) + } for _, record := range batch.Records { + if s.refuseEvents[record.Entry.ID] { + continue + } entry := record.Entry entry.Payload = nil event := StoredAuditEvent{ @@ -224,12 +259,22 @@ func (s *splitHistoryStore) AppendAuditBatch(_ context.Context, batch AuditBatch if s.appends == s.failDetails { return errors.New("details write failed") } + var detailRefusals []error + for _, record := range batch.Records { + if record.Retention != RetentionSecurity && !s.refuseEvents[record.Entry.ID] && s.refuseDetails[record.Entry.ID] { + detailRefusals = append(detailRefusals, refusalOf(record.Entry.ID, "details row")) + } + } + refusals = append(refusals, detailRefusals...) + if s.stopRest && len(detailRefusals) > 0 { + return errors.Join(append(refusals, errors.New("details rows stopped"))...) + } for _, record := range batch.Records { - if record.Retention != RetentionSecurity { + if record.Retention != RetentionSecurity && !s.refuseEvents[record.Entry.ID] && !s.refuseDetails[record.Entry.ID] { s.details = append(s.details, historyDetailsRow{record.Entry.ID, record.DetailsSHA256, record.Details}) } } - return nil + return errors.Join(refusals...) } func (s *splitHistoryStore) ReadStoredAuditEvents(_ context.Context, from, to time.Time, visit func(StoredAuditEvent) error) error { @@ -237,7 +282,7 @@ func (s *splitHistoryStore) ReadStoredAuditEvents(_ context.Context, from, to ti if event.Entry.CreatedAt.Before(from) || !event.Entry.CreatedAt.Before(to) { continue } - if event.Retention != RetentionSecurity { + if event.Retention != RetentionSecurity && !s.unreadable[event.Entry.ID] { for _, row := range s.details { if row.eventID != event.Entry.ID { continue @@ -995,33 +1040,52 @@ func TestAuditHistoryDoesNotRewriteWhatAWriteCannotFix(t *testing.T) { // What the copy writes and what verification reports are one rule, applied to // every copy of the event: a write is for the failure it removes, and when a // copy is wrong in a way no write removes, that is what is reported. -func TestAuditHistoryInspectSaysWhatAWriteRemoves(t *testing.T) { +// historyRecordOf is the record the copy builds for a row of the fixture's kind. +func historyRecordOf(t *testing.T, n int, eventType EventType, at time.Time) AuditRecord { + t.Helper() + resolved, err := NewAuditEventResolver(nil).Resolve(context.Background(), eventType) + require.NoError(t, err) + record, err := NewAuditRecord(historyRow(n, "", eventType, at), resolved.RetentionClass()) + require.NoError(t, err) + return record +} + +// historyStoredOf is a copy of record as a store holds it right after the events +// row landed: a security-class one with its details, a content-class one without. +func historyStoredOf(record AuditRecord, change func(*StoredAuditEvent)) StoredAuditEvent { + entry := record.Entry + entry.Payload = nil + stored := StoredAuditEvent{DeploymentID: "deployment-1", Entry: entry, Retention: record.Retention, DetailsSHA256: record.DetailsSHA256} + if record.Retention == RetentionSecurity { + stored.Details, stored.HasDetails = record.Details, true + } + if change != nil { + change(&stored) + } + return stored +} + +// historyInspector is a copy with nothing behind it, for the rule alone. +func historyInspector(t *testing.T, retention time.Duration, now time.Time) *AuditHistoryCopy { + t.Helper() copier, err := NewAuditHistoryCopy(AuditHistoryCopyConfig{ Source: &historySource{}, Store: &historyStore{}, Archive: &historyArchive{}, ReadBack: &historyStore{}, - DeploymentID: "deployment-1", ContentRetention: 7 * 24 * time.Hour, - Now: func() time.Time { return historyNow }, + DeploymentID: "deployment-1", ContentRetention: retention, + Now: func() time.Time { return now }, }) require.NoError(t, err) - recent, old := historyNow.Add(-24*time.Hour), historyNow.Add(-30*24*time.Hour) + return copier +} + +func TestAuditHistoryInspectSaysWhatAWriteRemoves(t *testing.T) { + copier := historyInspector(t, 7*24*time.Hour, historyNow) + // recent is held; margin is a day the store may have expired a few hours ago + // and may still hold; old is long gone. + recent, margin, old := historyNow.Add(-24*time.Hour), historyNow.Add(-7*24*time.Hour), historyNow.Add(-30*24*time.Hour) recordOf := func(n int, eventType EventType, at time.Time) AuditRecord { - resolved, err := NewAuditEventResolver(nil).Resolve(context.Background(), eventType) - require.NoError(t, err) - record, err := NewAuditRecord(historyRow(n, "", eventType, at), resolved.RetentionClass()) - require.NoError(t, err) - return record - } - storedOf := func(record AuditRecord, change func(*StoredAuditEvent)) StoredAuditEvent { - entry := record.Entry - entry.Payload = nil - stored := StoredAuditEvent{DeploymentID: "deployment-1", Entry: entry, Retention: record.Retention, DetailsSHA256: record.DetailsSHA256} - if record.Retention == RetentionSecurity { - stored.Details, stored.HasDetails = record.Details, true - } - if change != nil { - change(&stored) - } - return stored + return historyRecordOf(t, n, eventType, at) } + storedOf := historyStoredOf hasDetails := func(record AuditRecord) func(*StoredAuditEvent) { return func(e *StoredAuditEvent) { e.Details, e.HasDetails = record.Details, true } } @@ -1029,6 +1093,7 @@ func TestAuditHistoryInspectSaysWhatAWriteRemoves(t *testing.T) { noDetails := func(e *StoredAuditEvent) { e.Details, e.HasDetails = "", false } inside, outside := recordOf(1, EventDatasourceSourceSynced, recent), recordOf(2, EventDatasourceSourceSynced, old) + inMargin := recordOf(4, EventDatasourceSourceSynced, margin) login := recordOf(3, EventAuthLogin, recent) for name, tc := range map[string]struct { record AuditRecord @@ -1043,6 +1108,11 @@ func TestAuditHistoryInspectSaysWhatAWriteRemoves(t *testing.T) { problem: "content details are missing inside the content window", writable: true, }, "content details missing past the window": {record: outside, copies: []StoredAuditEvent{storedOf(outside, nil)}}, + "content details missing in the margin: written for, not a failure": { + record: inMargin, copies: []StoredAuditEvent{storedOf(inMargin, nil)}, writable: true, + }, + "no copy in the margin": {record: inMargin, problem: "missing from the store", writable: true}, + "content details present in the margin": {record: inMargin, copies: []StoredAuditEvent{storedOf(inMargin, hasDetails(inMargin))}}, "security details missing": { record: login, copies: []StoredAuditEvent{storedOf(login, noDetails)}, problem: "security-class details are missing", @@ -1055,9 +1125,17 @@ func TestAuditHistoryInspectSaysWhatAWriteRemoves(t *testing.T) { record: inside, copies: []StoredAuditEvent{storedOf(inside, func(e *StoredAuditEvent) { otherOrg(e); hasDetails(inside)(e) }), storedOf(inside, nil)}, problem: "stored envelope differs from the row", }, - "every copy missing details": { + // A write for missing details is made once: a second copy of the event + // beside the first is the proof that one was made. + "every copy missing details, so a write was made": { record: inside, copies: []StoredAuditEvent{storedOf(inside, nil), storedOf(inside, nil)}, - problem: "content details are missing inside the content window", writable: true, + problem: "content details are missing inside the content window; the store holds 2 copies, so an earlier write did not supply them, and the event is not written again", + }, + "every copy missing details in the margin, so a write was made": { + record: inMargin, copies: []StoredAuditEvent{storedOf(inMargin, nil), storedOf(inMargin, nil)}, + }, + "two copies, the details attached": { + record: inside, copies: []StoredAuditEvent{storedOf(inside, hasDetails(inside)), storedOf(inside, hasDetails(inside))}, }, } { t.Run(name, func(t *testing.T) { @@ -1068,3 +1146,333 @@ func TestAuditHistoryInspectSaysWhatAWriteRemoves(t *testing.T) { }) } } + +// historyTotals is what a run reported across its partitions. +type historyTotals struct { + copied, rewritten, failures int + verified, postgres int64 + problems []string +} + +func totalsOf(report AuditHistoryReport) historyTotals { + var totals historyTotals + for _, partition := range report.Partitions { + totals.copied += partition.Copied + totals.rewritten += partition.Rewritten + totals.failures += partition.Failures + totals.problems = append(totals.problems, partition.Problems...) + for _, count := range partition.Orgs { + totals.verified += count.Verified + totals.postgres += count.Postgres + } + } + return totals +} + +// The store refuses one event's details row for good — its events row has +// landed — as ClickHouse does when its isolation finds the row, and as BigQuery +// does when it reports the row invalid (and the rest of that request stopped). +// That event is a verification failure of its own. It does not stop the copy of +// the others, and it is not written again on every run: a write for missing +// details is made once, and the run that finds one already made says so. +func TestAuditHistoryAPermanentlyRefusedDetailsRowStopsNothingAndIsNotRepeated(t *testing.T) { + for name, stopRest := range map[string]bool{"the rest written around the refused row": false, "the rest of the request stopped": true} { + t.Run(name, func(t *testing.T) { + f := newHistoryFixture() + bad := f.source.rows[0].ID // the first content-class event, in the first batch + require.Equal(t, RetentionContent, historyRecordOf(t, 1, f.source.rows[0].EventType, f.source.rows[0].CreatedAt).Retention) + f.split = &splitHistoryStore{refuseDetails: map[string]bool{bad: true}, stopRest: stopRest} + run := func() (AuditHistoryReport, historyTotals) { + report, err := f.copier(t, AuditHistoryCopyConfig{}).Run(context.Background(), AuditHistoryCopyOptions{}) + require.ErrorIs(t, err, ErrAuditHistoryUnverified, "a refused event is a verification failure, not a failed run") + require.False(t, report.Verified) + totals := totalsOf(report) + require.Equal(t, 1, totals.failures) + require.Len(t, totals.problems, 1) + require.Contains(t, totals.problems[0], bad) + require.Equal(t, int64(len(f.source.rows)), totals.postgres) + require.Equal(t, int64(len(f.source.rows)-1), totals.verified, "every other event is copied and verified") + return report, totals + } + + // The run reaches the end: every later window and the later partition. + report, totals := run() + require.Len(t, report.Partitions, 3) + require.Equal(t, len(f.source.rows)-1, totals.copied, "the refused event is not counted as copied") + require.Zero(t, totals.rewritten) + require.Contains(t, totals.problems[0], "the store refused the event's row for good") + require.Equal(t, 1, f.split.copies()[bad]) + require.Equal(t, 1, f.split.attempts[bad], "its batch is sent again without it, not with it") + september := report.Partitions[1] + require.NotEmpty(t, september.Orgs) + require.Zero(t, september.Failures) + events, archived := len(f.split.events), len(f.archive.batches) + + // The next run makes the one write for the details it is missing and is + // refused again; nothing else is written. + _, totals = run() + require.Zero(t, totals.copied+totals.rewritten) + require.Equal(t, 2, f.split.attempts[bad]) + require.Equal(t, 2, f.split.copies()[bad]) + require.Equal(t, events+1, len(f.split.events)) + require.Equal(t, archived+1, len(f.archive.batches)) + require.Equal(t, []string{bad}, historyRecordIDs(f.archive.batches[len(f.archive.batches)-1].Records)) + require.Contains(t, totals.problems[0], "the store refused the event's row for good") + + // Neither this run nor the ones after it write or archive anything. + appends := f.split.appends + for range 3 { + _, totals = run() + require.Zero(t, totals.copied+totals.rewritten) + require.Equal(t, appends, f.split.appends, "nothing is appended") + require.Equal(t, events+1, len(f.split.events)) + require.Len(t, f.archive.batches, archived+1, "nothing is archived") + require.Equal(t, 2, f.split.attempts[bad]) + require.Contains(t, totals.problems[0], "the store holds 2 copies") + require.Contains(t, totals.problems[0], "not written again") + } + }) + } +} + +func historyRecordIDs(records []AuditRecord) []string { + var ids []string + for _, record := range records { + ids = append(ids, record.Entry.ID) + } + return ids +} + +// An event whose events row the store refuses for good is stored nowhere, so +// nothing on the next run says a write was made: it is written again each run, and +// reported each time. The cost is one archive object holding that event alone, not +// the window's, and no row in the store. +func TestAuditHistoryAnEventWhoseEventsRowIsRefusedStopsNothing(t *testing.T) { + for name, stopRest := range map[string]bool{"the rest written around the refused row": false, "the rest of the request stopped": true} { + t.Run(name, func(t *testing.T) { + f := newHistoryFixture() + bad := f.source.rows[0].ID + f.split = &splitHistoryStore{refuseEvents: map[string]bool{bad: true}, stopRest: stopRest} + for run := range 3 { + before := len(f.archive.batches) + report, err := f.copier(t, AuditHistoryCopyConfig{}).Run(context.Background(), AuditHistoryCopyOptions{}) + require.ErrorIs(t, err, ErrAuditHistoryUnverified) + totals := totalsOf(report) + require.Equal(t, 1, totals.failures) + require.Equal(t, int64(len(f.source.rows)-1), totals.verified) + require.Len(t, totals.problems, 1) + require.Contains(t, totals.problems[0], bad) + require.Contains(t, totals.problems[0], "the store refused the event's row for good; missing from the store") + require.Zero(t, f.split.copies()[bad], "nothing of it is stored") + if run == 0 { + require.Equal(t, len(f.source.rows)-1, totals.copied) + } else { + require.Zero(t, totals.copied+totals.rewritten) + require.Len(t, f.archive.batches, before+1) + require.Equal(t, []string{bad}, historyRecordIDs(f.archive.batches[before].Records)) + } + } + }) + } +} + +// A refusal that names no event of the batch is no evidence about any of them: +// the run fails, as it does for any other failure of the store. +func TestAuditHistoryARefusalOfAnEventNotInTheBatchFailsTheRun(t *testing.T) { + f := newHistoryFixture() + store := &refusingHistoryStore{historyStore: f.store, err: &PermanentRowRejection{EventID: "someone-else", Cause: errors.New("refused")}} + copier := f.copier(t, AuditHistoryCopyConfig{}) + copier.cfg.Store = store + _, err := copier.Run(context.Background(), AuditHistoryCopyOptions{}) + require.ErrorContains(t, err, "append batch") + require.NotErrorIs(t, err, ErrAuditHistoryUnverified) + require.Equal(t, 1, store.calls, "the batch is not sent again") +} + +type refusingHistoryStore struct { + *historyStore + err error + calls int +} + +func (s *refusingHistoryStore) AppendAuditBatch(context.Context, AuditBatch) error { + s.calls++ + return s.err +} + +// A write whose details the store accepts and then never shows is made once in a +// run, once more by the next, and not again: the run that finds the second copy +// says that a write was made and did not supply them. +func TestAuditHistoryDetailsWrittenButNeverReadableAreNotWrittenAgainAndAgain(t *testing.T) { + f := newHistoryFixture() + bad := f.source.rows[0].ID + f.split = &splitHistoryStore{unreadable: map[string]bool{bad: true}} + run := func() historyTotals { + report, err := f.copier(t, AuditHistoryCopyConfig{}).Run(context.Background(), AuditHistoryCopyOptions{}) + require.ErrorIs(t, err, ErrAuditHistoryUnverified) + totals := totalsOf(report) + require.Equal(t, 1, totals.failures) + require.Equal(t, int64(len(f.source.rows)-1), totals.verified) + require.Contains(t, totals.problems[0], bad) + return totals + } + + totals := run() + require.Equal(t, len(f.source.rows), totals.copied) + require.Equal(t, 1, f.split.attempts[bad], "written once in the run, not again after the read-back") + require.Contains(t, totals.problems[0], "written by this run, but the read-back finds it incomplete: content details are missing") + + totals = run() + require.Equal(t, 1, totals.rewritten) + require.Zero(t, totals.copied) + require.Equal(t, 2, f.split.attempts[bad]) + require.Contains(t, totals.problems[0], "written by this run, but the read-back finds it incomplete") + + appends, archived := f.split.appends, len(f.archive.batches) + for range 2 { + totals = run() + require.Zero(t, totals.copied+totals.rewritten) + require.Equal(t, appends, f.split.appends) + require.Len(t, f.archive.batches, archived) + require.Contains(t, totals.problems[0], "an earlier write did not supply them") + } +} + +// The rule for the details of a content-class event, at the instants where it +// changes. The store holds the details until the start of the event's UTC day +// plus the retention, because it expires by day, so an event's details are +// required while now is before that instant — and only then. For the margin +// after it they may still be held: the copy writes them, and never fails for them. +// A short retention keeps its requirement: one day still requires the events of +// today. +func TestAuditHistoryDetailsRuleAtTheBoundaries(t *testing.T) { + const day = 24 * time.Hour + today := time.Date(2026, 11, 2, 0, 0, 0, 0, time.UTC) + nows := map[string]time.Time{ + "at midnight": today, + "at noon": today.Add(12 * time.Hour), + "just before midnight": today.Add(day - time.Nanosecond), + } + type zone int + const ( + required zone = iota // the store holds the details: a failure without them + optional // it may: written for, never a failure + neither // it does not: no failure, no write + ) + for _, retention := range []time.Duration{day, 2 * day, 7 * day} { + for nowName, now := range nows { + copier := historyInspector(t, retention, now) + first := today.Add(-retention) // the day whose details the store may expire as of today + for offset, want := range map[int]zone{3: required, 2: required, 1: required, 0: optional, -1: optional, -2: neither, -3: neither} { + for _, at := range []time.Duration{0, day - time.Nanosecond} { + created := first.Add(time.Duration(offset)*day + at) + if created.After(now) { + continue + } + name := fmt.Sprintf("retention %s %s, event %d days from the first day, %s into its day", retention, nowName, offset, at) + t.Run(name, func(t *testing.T) { + record := historyRecordOf(t, 1, EventDatasourceSourceSynced, created) + copies := []StoredAuditEvent{historyStoredOf(record, nil)} + problem, writable := copier.inspect(record, copies) + switch want { + case required: + require.Equal(t, "content details are missing inside the content window", problem) + require.True(t, writable) + case optional: + require.Empty(t, problem, "the store may have expired them") + require.True(t, writable, "and may still hold them") + default: + require.Empty(t, problem) + require.False(t, writable) + } + require.Equal(t, problem, copier.verify(record, copies)) + // The same rule for an event that is not in the store yet: it + // is written, whichever the zone. + problem, writable = copier.inspect(record, nil) + require.Equal(t, "missing from the store", problem) + require.True(t, writable) + }) + } + } + } + } +} + +// The same rule through a run, with the events of the fixture in each zone +// (retention 62 days, now 2 November at noon): 30 August is past the margin, +// 31 August and 1 September are in it, 2 September is held. The store accepts +// each event's details and never shows those of one event of each day. +func TestAuditHistoryContentDetailsInTheMarginAreWrittenForAndNeverFailed(t *testing.T) { + f := newHistoryFixture() + config := AuditHistoryCopyConfig{ContentRetention: 62 * 24 * time.Hour} + f.split = &splitHistoryStore{unreadable: map[string]bool{}} + byDay := map[int]string{} + for _, row := range f.source.rows { + if row.EventType == EventDatasourceSourceSynced && byDay[row.CreatedAt.Day()] == "" { + byDay[row.CreatedAt.Day()] = row.ID + f.split.unreadable[row.ID] = true + } + } + past, margin1, margin2, held := byDay[30], byDay[31], byDay[1], byDay[2] + require.Len(t, byDay, 4) + run := func() historyTotals { + report, err := f.copier(t, config).Run(context.Background(), AuditHistoryCopyOptions{}) + require.ErrorIs(t, err, ErrAuditHistoryUnverified) + totals := totalsOf(report) + require.Equal(t, 1, totals.failures, "only the event the store holds details of fails") + require.Contains(t, totals.problems[0], held) + return totals + } + + run() + for _, id := range []string{past, margin1, margin2, held} { + require.Equal(t, 1, f.split.attempts[id], id) + } + totals := run() + require.Equal(t, 3, totals.rewritten, "the held event and the two in the margin are written once more; the one past it is not") + for id, attempts := range map[string]int{past: 1, margin1: 2, margin2: 2, held: 2} { + require.Equal(t, attempts, f.split.attempts[id], id) + } + appends := f.split.appends + for range 2 { + totals = run() + require.Zero(t, totals.copied+totals.rewritten) + require.Equal(t, appends, f.split.appends) + } +} + +// With a retention of one day the details of today's events are still required, +// and those of yesterday's are written for without being required: the store +// held them until midnight. +func TestAuditHistoryAShortRetentionStillRequiresTheDetailsOfTodaysEvents(t *testing.T) { + today, yesterday := time.Date(2026, 11, 2, 3, 0, 0, 0, time.UTC), time.Date(2026, 11, 1, 3, 0, 0, 0, time.UTC) + november := month(2026, time.November) + source := &historySource{partitions: []AuditHistoryPartition{november}, rows: []AuditEntry{ + historyRow(1, "", EventDatasourceSourceSynced, yesterday), historyRow(2, "", EventDatasourceSourceSynced, today), + }} + newFixture := func(split *splitHistoryStore) *historyFixture { + return &historyFixture{source: source, store: &historyStore{}, archive: &historyArchive{}, split: split} + } + config := AuditHistoryCopyConfig{ContentRetention: 24 * time.Hour} + + // A details write that failed is made again for today's event. + f := newFixture(&splitHistoryStore{failDetails: 1}) + f.source = &historySource{partitions: []AuditHistoryPartition{november}, rows: source.rows[1:]} + _, err := f.copier(t, config).Run(context.Background(), AuditHistoryCopyOptions{}) + require.ErrorContains(t, err, "details write failed") + require.Empty(t, f.split.details) + report, err := f.copier(t, config).Run(context.Background(), AuditHistoryCopyOptions{}) + require.NoError(t, err) + require.True(t, report.Verified) + require.Equal(t, 1, totalsOf(report).rewritten) + require.Len(t, f.split.details, 1) + + // Details the store never shows fail verification for today's event alone. + f = newFixture(&splitHistoryStore{unreadable: map[string]bool{source.rows[0].ID: true, source.rows[1].ID: true}}) + report, err = f.copier(t, config).Run(context.Background(), AuditHistoryCopyOptions{}) + require.ErrorIs(t, err, ErrAuditHistoryUnverified) + totals := totalsOf(report) + require.Equal(t, 1, totals.failures) + require.Contains(t, totals.problems[0], source.rows[1].ID) + require.Contains(t, totals.problems[0], "content details are missing inside the content window") +} From 0f9fdb9e6892338ad0424cf905144fe96eaeb024 Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Wed, 7 Oct 2026 13:07:44 -0400 Subject: [PATCH 54/69] fix(audit): hold a kept event as its text, not the decode of its payload A BigQuery window counts the text of the events it keeps, but Match and Aggregator.Add decoded each event's payload and left the decoded map on the event for the rest of the window. A decode is several times the text: with a payload filter or a payload group-by over a window that filled its budget, 1500 events counted 1.7 MiB and held 6.0 MiB (3.5x) with thirty integer keys, and counted 2.0 MiB and held 14.4 MiB (7.2x) with a 200-integer array. The bound the window budget gave was a number of text bytes, not of memory. An event now holds only its text once it is kept. Detach, which a store calls on the events it keeps, drops the decode a filter left; a page drops it from the events it keeps, since the list decodes the payload again for its result; and Aggregator.Add decodes for itself and leaves none on the event. A store that streams an event through a filter and on to an aggregation (ClickHouse) still decodes it once, because Add reads the decode Match made. A BigQuery aggregation with a payload filter decodes an event twice, once to match and once to add, which is the price of counting what the window holds; answers do not change. The test keeps a window of payload-filtered and payload-grouped events, measures the heap the events hold after a collection, and holds it to 1.5 times what the window counted (0.85 to 0.9 times now). ADR 0009 states that an event is kept as its text. Co-Authored-By: Claude Sonnet 5.5 --- module/docs/adr/0009-audit-store-swap.md | 11 +- .../pkg/auditstore/auditeval/aggregate.go | 4 +- .../code/pkg/auditstore/auditeval/eval.go | 50 +++++-- .../auditeval/eval_retained_test.go | 141 ++++++++++++++++++ .../code/pkg/auditstore/auditeval/page.go | 3 + .../bigquerystore/reader_retained_test.go | 129 ++++++++++++++++ 6 files changed, 321 insertions(+), 17 deletions(-) create mode 100644 module/services/accounts/code/pkg/auditstore/auditeval/eval_retained_test.go create mode 100644 module/services/accounts/code/pkg/auditstore/bigquerystore/reader_retained_test.go diff --git a/module/docs/adr/0009-audit-store-swap.md b/module/docs/adr/0009-audit-store-swap.md index 24d4fa748..ba98fcd57 100644 --- a/module/docs/adr/0009-audit-store-swap.md +++ b/module/docs/adr/0009-audit-store-swap.md @@ -207,10 +207,13 @@ Postgres transaction. them — against a byte budget, `ReadConfig.WindowBytes` (64 MiB by default; the service exposes no setting for it). What is counted is what is kept, once per event however many copies a redelivery left in the tables: the copies of an - event share its occurrence, so no narrower window could shed them. A window - that passes the budget is read again as two halves. What outlives a window is - the answer: a page, an aggregation's buckets, an export's events. Four limits - remain: + event share its occurrence, so no narrower window could shed them. An event + is kept as its text: the decoded payload a filter or an aggregation reads is + several times that text, so it is dropped once the event is kept and decoded + again when an aggregation reads it, and the count is what the window holds. + A window that passes the budget is read again as two halves. What outlives a + window is the answer: a page, an aggregation's buckets, an export's events. + Four limits remain: - A read whose events at a single instant alone pass the budget cannot be split further and fails (`ErrReadTooDense`). - An export is the one answer as large as the history it matches. One export diff --git a/module/services/accounts/code/pkg/auditstore/auditeval/aggregate.go b/module/services/accounts/code/pkg/auditstore/auditeval/aggregate.go index caf934104..ef61d7879 100644 --- a/module/services/accounts/code/pkg/auditstore/auditeval/aggregate.go +++ b/module/services/accounts/code/pkg/auditstore/auditeval/aggregate.go @@ -210,7 +210,9 @@ func payloadKey(field string) (string, bool) { // Add counts one matching event. The caller offers each event once. func (a *Aggregator) Add(event *Event) error { - payload, err := event.payload() + // Decoded for this event alone: a store that holds a window of events + // before it adds them keeps their text, and the budget counts that. + payload, err := event.payload(false) if err != nil { return fmt.Errorf("audit read: details of event %s: %w", event.Entry.ID, err) } diff --git a/module/services/accounts/code/pkg/auditstore/auditeval/eval.go b/module/services/accounts/code/pkg/auditstore/auditeval/eval.go index c3d4afcca..451551f96 100644 --- a/module/services/accounts/code/pkg/auditstore/auditeval/eval.go +++ b/module/services/accounts/code/pkg/auditstore/auditeval/eval.go @@ -36,6 +36,10 @@ type Event struct { // kept; a store joins details to the event by it as well as by id. DetailsSHA256 string + // decoded is the details decoded, left by Match for the aggregation that + // reads the event next in a store that streams it. It is scratch for one + // visit, never part of what an event holds: Detach and Page.Offer drop it, + // and Aggregator.Add never leaves one behind (see payload). decoded map[string]any decodeErr error decodedSet bool @@ -48,7 +52,9 @@ const eventOverheadBytes = 512 // Size approximates the bytes the event holds in memory: a fixed overhead and // the text of its envelope and details. A store that buffers events counts // them by it against a budget, so its memory is a number it chose, not a -// function of how much history it reads. +// function of how much history it reads. An event holds that and no more once +// it is detached (Detach): a decoded payload is several times its text, which +// the count does not include, so a kept event never keeps one. func (e *Event) Size() int { entry := &e.Entry return eventOverheadBytes + len(entry.ID) + len(entry.OrgID) + len(entry.ActorID) + len(entry.ActorType) + @@ -56,12 +62,17 @@ func (e *Event) Size() int { len(entry.ImpersonatedBy) + len(entry.ClientID) + len(e.Details) + len(e.DetailsSHA256) } -// Detach copies the text of the event, so that an event kept does not keep -// alive what it was decoded from. A store that decodes rows out of a shared -// buffer (an Arrow record batch) hands back strings that point into it: one -// event kept for the page would hold the whole batch of thousands of rows it -// came in, and a window's worth of kept events every batch of the window. +// Detach makes the event hold its text and nothing else, so that an event kept +// does not keep alive more than Size counts. It copies the text, which a store +// that decodes rows out of a shared buffer (an Arrow record batch) hands back +// as strings that point into it: one event kept for the page would hold the +// whole batch of thousands of rows it came in, and a window's worth of kept +// events every batch of the window. And it drops the decoded payload a filter +// left on the event (Match): that is several times the text, and a window that +// kept it would hold memory its budget never counted. An aggregation that +// reads the event later decodes it again. func (e *Event) Detach() { + e.dropDecoded() entry := &e.Entry for _, text := range []*string{ &entry.ID, &entry.OrgID, &entry.ActorID, &entry.ActorType, &entry.Resource, &entry.ResourceID, @@ -72,16 +83,31 @@ func (e *Event) Detach() { entry.EventType = business.EventType(strings.Clone(string(entry.EventType))) } +// dropDecoded forgets the decode of the details. +func (e *Event) dropDecoded() { + e.decoded, e.decodeErr, e.decodedSet = nil, nil, false +} + // payload is the details decoded with json.Number, or nil when there are none. -func (e *Event) payload() (map[string]any, error) { +// A decode already made is read again, so that a store that streams an event +// through a payload filter and then an aggregation decodes it once. A decode +// that has to be made is remembered only when keep is set (by Match, whose +// event goes on to the aggregation of a streaming store or is dropped); an +// aggregation that reads events a store holds for a whole window passes false, +// so that it leaves none on them: the budget of such a store counts the text of +// the events it keeps, not a decode several times its size. +func (e *Event) payload(keep bool) (map[string]any, error) { if !e.HasDetails { return nil, nil } - if !e.decodedSet { - e.decoded, e.decodeErr = decodeDetails(e.Details) - e.decodedSet = true + if e.decodedSet { + return e.decoded, e.decodeErr + } + decoded, err := decodeDetails(e.Details) + if keep { + e.decoded, e.decodeErr, e.decodedSet = decoded, err, true } - return e.decoded, e.decodeErr + return decoded, err } // Result is the entry a read returns for the event: the envelope with its @@ -321,7 +347,7 @@ func (m *Matcher) Match(event *Event) (bool, error) { return false, nil } if m.payload != nil { - payload, err := event.payload() + payload, err := event.payload(true) if err != nil { return false, fmt.Errorf("audit read: details of event %s: %w", entry.ID, err) } diff --git a/module/services/accounts/code/pkg/auditstore/auditeval/eval_retained_test.go b/module/services/accounts/code/pkg/auditstore/auditeval/eval_retained_test.go new file mode 100644 index 000000000..a037ed7ae --- /dev/null +++ b/module/services/accounts/code/pkg/auditstore/auditeval/eval_retained_test.go @@ -0,0 +1,141 @@ +package auditeval + +import ( + "fmt" + "testing" + "time" + + "accounts/pkg/business" + + "github.com/stretchr/testify/require" +) + +// A decoded payload is several times the text it came from. A store that holds +// events for a window counts their text (Event.Size), so an event it keeps must +// hold no decode: Match may leave one for a store that streams the event on to +// an aggregation, and everything that keeps an event drops it. + +// payloadEvent is an event whose payload filter and group-by key both read its +// details, with thirty keys of small integers and an array of a hundred: text +// of about a kilobyte, a decode of several. +func payloadEvent(n int) *Event { + text := fmt.Sprintf(`{"match":true,"group":%d,"values":[`, n%3) + for k := 0; k < 100; k++ { + if k > 0 { + text += "," + } + text += fmt.Sprint((n + k) % 10) + } + text += "]" + for k := 0; k < 30; k++ { + text += fmt.Sprintf(`,"k%02d":%d`, k, k) + } + return &Event{ + Entry: business.AuditEntry{ + ID: fmt.Sprintf("00000000-0000-4000-8000-%012d", n), EventType: business.EventDocumentRead, + CreatedAt: time.Date(2026, 10, 3, 0, 0, 0, 0, time.UTC).Add(time.Duration(n) * time.Second), + }, + Details: text + "}", HasDetails: true, + } +} + +func payloadMatcher(t *testing.T) *Matcher { + t.Helper() + m, err := NewMatcher(business.AuditRead{ + Scope: business.PlatformAuditScope(), + Query: business.AuditQuery{PayloadContains: map[string]any{"match": true}}, + }) + require.NoError(t, err) + return m +} + +var groupByPayload = business.AuditAggregationSpec{ + GroupBy: []string{"payload:group"}, + Metrics: []business.AuditMetric{{Op: "sum", Field: "payload:k07"}, {Op: "count_distinct", Field: "payload:group", Alias: "groups"}}, +} + +func TestAnEventThatIsKeptHoldsNoDecodedPayload(t *testing.T) { + m := payloadMatcher(t) + + streamed := payloadEvent(1) + ok, err := m.Match(streamed) + require.NoError(t, err) + require.True(t, ok) + require.NotNil(t, streamed.decoded, "a filter leaves its decode for the aggregation that reads the event next") + + // A store that keeps the event for a window detaches it, which drops the decode. + kept := payloadEvent(2) + _, err = m.Match(kept) + require.NoError(t, err) + kept.Detach() + require.False(t, kept.decodedSet) + require.Nil(t, kept.decoded) + require.Equal(t, payloadEvent(2).Size(), kept.Size(), "what it holds is what Size counts") + + // A page drops it from every event it keeps, detached or not. + page, err := NewPage(business.AuditQuery{PageSize: 5}) + require.NoError(t, err) + paged := payloadEvent(3) + _, err = m.Match(paged) + require.NoError(t, err) + require.NotNil(t, paged.decoded) + require.True(t, page.Offer(paged)) + require.Nil(t, paged.decoded, "a page returns the payload decoded again; the decode would only be held") + + // An aggregation reads an event it does not own the life of: it leaves none. + aggregator, err := NewAggregator(groupByPayload, nil, 0) + require.NoError(t, err) + fresh := payloadEvent(4) + require.NoError(t, aggregator.Add(fresh)) + require.False(t, fresh.decodedSet, "Add decodes for itself and keeps nothing on the event") + require.Nil(t, fresh.decoded) + require.NoError(t, aggregator.Add(kept)) + require.Nil(t, kept.decoded, "a detached event is read by decoding it again") +} + +func TestAnAggregationAnswersTheSameWhateverTheEventHoldsOfItsDecode(t *testing.T) { + m := payloadMatcher(t) + answer := func(prepare func(*Event)) []business.AuditAggregateBucket { + aggregator, err := NewAggregator(groupByPayload, nil, 0) + require.NoError(t, err) + for n := 1; n <= 30; n++ { + event := payloadEvent(n) + prepare(event) + require.NoError(t, aggregator.Add(event)) + } + return aggregator.Buckets() + } + fresh := answer(func(*Event) {}) + streamed := answer(func(e *Event) { + ok, err := m.Match(e) + require.NoError(t, err) + require.True(t, ok) + }) + detached := answer(func(e *Event) { + _, err := m.Match(e) + require.NoError(t, err) + e.Detach() + }) + require.Len(t, fresh, 3) + require.Equal(t, fresh, streamed, "an event a filter has decoded") + require.Equal(t, fresh, detached, "an event a window kept, decoded again") +} + +func TestAnEventStreamedThroughAFilterAndAnAggregationIsDecodedOnce(t *testing.T) { + m := payloadMatcher(t) + aggregator, err := NewAggregator(groupByPayload, nil, 0) + require.NoError(t, err) + + allocs := func(visit func(*Event)) float64 { + event := payloadEvent(1) + return testing.AllocsPerRun(50, func() { + event.dropDecoded() + visit(event) + }) + } + match := allocs(func(e *Event) { _, _ = m.Match(e) }) + add := allocs(func(e *Event) { _ = aggregator.Add(e) }) + both := allocs(func(e *Event) { _, _ = m.Match(e); _ = aggregator.Add(e) }) + require.Greater(t, match, 100.0, "decoding is where the allocations are") + require.Less(t, both, match+add-match/2, "Add reads the decode Match made: one decode, not two (match %.0f, add %.0f, both %.0f)", match, add, both) +} diff --git a/module/services/accounts/code/pkg/auditstore/auditeval/page.go b/module/services/accounts/code/pkg/auditstore/auditeval/page.go index d108503a6..ed4643ec9 100644 --- a/module/services/accounts/code/pkg/auditstore/auditeval/page.go +++ b/module/services/accounts/code/pkg/auditstore/auditeval/page.go @@ -94,6 +94,9 @@ func (p *Page) Offer(event *Event) bool { evicted := heap.Pop(&p.kept).(*Event) delete(p.keptIDs, evicted.Entry.ID) } + // A page keeps the event as its text: the list returns the payload decoded + // again (Result), so a decode a filter left on it is of no use. + event.dropDecoded() heap.Push(&p.kept, event) // The key is its own copy: the event's id may point into a buffer the event // is about to be detached from. diff --git a/module/services/accounts/code/pkg/auditstore/bigquerystore/reader_retained_test.go b/module/services/accounts/code/pkg/auditstore/bigquerystore/reader_retained_test.go new file mode 100644 index 000000000..601d98d0f --- /dev/null +++ b/module/services/accounts/code/pkg/auditstore/bigquerystore/reader_retained_test.go @@ -0,0 +1,129 @@ +package bigquerystore + +import ( + "context" + "fmt" + "runtime" + "testing" + "time" + + "accounts/pkg/auditstore/auditeval" + "accounts/pkg/business" + + "github.com/stretchr/testify/require" +) + +// A window counts what it keeps, so what it keeps must be what it counted. An +// event held for a window is held as its text: reading its payload to match a +// filter or to aggregate it makes a decoded copy several times the size of the +// text, and a copy left on the event for the rest of the window would be +// memory the budget never counted. These tests keep a window of events that a +// payload filter or a payload group-by has read, and compare the heap the +// window holds, measured after a collection, with what it charged. + +// retainedWindowEvents is how many events a window of these tests holds. +const retainedWindowEvents = 1500 + +// retainedFactor is how far the heap a window holds may pass what it counted. +// The count is an approximation of the text (a fixed overhead per event and the +// length of its strings), and the heap lands just under it (0.85 to 0.9 times); +// a window that kept the decoded payloads of its events lands at three and a +// half to seven times it. +const retainedFactor = 1.5 + +// retainedPayloads are the shapes of payload whose decode is far larger than +// their text: many small numbers, each of which the decode boxes in an interface +// of its own. +var retainedPayloads = map[string]func(i int) map[string]any{ + "thirty integer keys": func(i int) map[string]any { + payload := map[string]any{"match": true, "group": i % 7} + for k := 0; k < 30; k++ { + payload[fmt.Sprintf("k%02d", k)] = 100000 + i*31 + k + } + return payload + }, + "an array of two hundred integers": func(i int) map[string]any { + values := make([]any, 200) + for k := range values { + values[k] = (i + k) % 100 + } + return map[string]any{"match": true, "group": i % 7, "values": values} + }, +} + +// retainedReads are the reads that decode a payload: a filter, a group-by, and +// the two together. +var retainedReads = map[string]struct { + query business.AuditQuery + spec *business.AuditAggregationSpec +}{ + "a payload filter": {query: business.AuditQuery{PayloadContains: map[string]any{"match": true}}}, + "a payload group-by": {spec: &business.AuditAggregationSpec{GroupBy: []string{"payload:group"}}}, + "a filter and a group-by": { + query: business.AuditQuery{PayloadContains: map[string]any{"match": true}}, + spec: &business.AuditAggregationSpec{GroupBy: []string{"payload:group"}, Metrics: []business.AuditMetric{{Op: "sum", Field: "payload:group"}}}, + }, +} + +// heapInUse is the live heap after two collections. +func heapInUse() int64 { + runtime.GC() + runtime.GC() + var stats runtime.MemStats + runtime.ReadMemStats(&stats) + return int64(stats.HeapAlloc) +} + +func TestAWindowHoldsInMemoryWhatItCountedNotTheDecodeOfItsPayloads(t *testing.T) { + for payloadName, payload := range retainedPayloads { + for readName, read := range retainedReads { + t.Run(payloadName+"/"+readName, func(t *testing.T) { + w := newBoundWarehouse(t) + var records []business.AuditRecord + for i := 0; i < retainedWindowEvents; i++ { + records = append(records, boundEvent(t, i+1, boundNow.Add(-time.Hour-time.Duration(i)*time.Second), payload(i))) + } + w.append(t, records...) + reader := w.reader(t, 1<<40, 0) + + audit := orgScoped(read.query) + m, err := auditeval.NewMatcher(audit) + require.NoError(t, err) + var aggregator *auditeval.Aggregator + if read.spec != nil { + aggregator, err = auditeval.NewAggregator(*read.spec, nil, 1<<40) + require.NoError(t, err) + } + + events, err := reader.loadWindow(context.Background(), m, audit.Query, window{}, true) + require.NoError(t, err) + if aggregator != nil { + for _, event := range events { + require.NoError(t, aggregator.Add(event)) + } + } + var counted int64 + for _, event := range events { + counted += int64(event.Size() + dedupeEntryBytes) + } + count := len(events) + + // What the events hold is what the heap loses when they go: the + // warehouse fake keeps the batches it served, and they stay in + // both measurements. + with := heapInUse() + runtime.KeepAlive(events) + events = nil + held := with - heapInUse() + runtime.KeepAlive(aggregator) + + require.Equal(t, retainedWindowEvents, count) + // The slack is the runtime's noise: the aggregator, whose buckets + // the events do not hold, is in both measurements. + const slack = 64 << 10 + require.LessOrEqual(t, float64(held), retainedFactor*float64(counted)+slack, + "the window counted %d bytes and held %d (%.1fx)", counted, held, float64(held)/float64(counted)) + }) + } + } +} From 669897314c1a426f7f1cb1bc9f0c5625f7800b07 Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Wed, 7 Oct 2026 13:43:01 -0400 Subject: [PATCH 55/69] fix(audit): preserve durable delivery and recover partial history copies Pin synchronous ClickHouse writes and reject transforming table engines. Reject BigQuery whole-table expiry and non-daily partitions at startup. Allow one incomplete history repair attempt per event per run after repeated details failures, retaining verification and partition removal guards. Add regression coverage and align the ADR and operator documentation. --- module/AUDIT_OPERATIONS.md | 19 ++-- module/docs/adr/0009-audit-store-swap.md | 20 +++- module/services/accounts/AGENTS.md | 5 +- .../auditstore/bigquerystore/bigquerystore.go | 10 +- .../bigquerystore/ensure_retention_test.go | 105 ++++++++++++++++++ .../clickhousestore/clickhousestore.go | 49 +++++--- .../clickhousestore/clickhousestore_test.go | 48 ++++++-- .../auditstore/clickhousestore/server_test.go | 52 +++++++++ .../code/pkg/business/audit_history.go | 24 ++-- .../code/pkg/business/audit_history_test.go | 98 ++++++++++------ 10 files changed, 345 insertions(+), 85 deletions(-) create mode 100644 module/services/accounts/code/pkg/auditstore/bigquerystore/ensure_retention_test.go diff --git a/module/AUDIT_OPERATIONS.md b/module/AUDIT_OPERATIONS.md index aefabab7b..0d36ef808 100644 --- a/module/AUDIT_OPERATIONS.md +++ b/module/AUDIT_OPERATIONS.md @@ -53,21 +53,22 @@ instant the warehouse may still hold them: the event is appended to supply them and never fails for lacking them. After that it is neither appended nor required to have them. -An event is appended again for missing details **once**. When the warehouse holds -more than one copy of an event and still no details, an earlier write did not -supply them (the warehouse refused the row, or accepted it and does not show it), -and another would add an events row and an archive object and change nothing. The -event stays a verification failure, and the report says how many copies the -warehouse holds. An event written in a run is never written again in that run, -whatever the read-back shows. +An incomplete event gets at most **one repair attempt per run**, however many +copies the warehouse already holds. Repeated temporary failures can write the +events row each time without writing its details; the number of copies does not +establish a permanent refusal. A later run can supply the missing details to all +copies once the warehouse recovers. Missing details inside the content window +continue to fail verification. An event written in a run is never written again +in that run, whatever the read-back shows. An event whose row the warehouse refuses for good (a row too large, a value its column cannot hold) fails verification **by itself**: the rest of its batch and of the run are copied, and the report names the event (the warehouse's own message is not printed). If the events row was refused, nothing of the event is stored and every run tries it again, archiving that event alone each time. If only the details -row was refused, the next run writes the event once more and later runs do not. -Either way the partitions are not dropped: the command has no way to skip an event. +row was refused, each later run attempts to supply it once more. Either way the +partitions are not dropped while verification fails: the command has no way to +skip an event. Each write archives its batch again: the archive never overwrites an object, so a rewrite adds one that holds the event a second time. That is safe because archive diff --git a/module/docs/adr/0009-audit-store-swap.md b/module/docs/adr/0009-audit-store-swap.md index ba98fcd57..8fa23b21a 100644 --- a/module/docs/adr/0009-audit-store-swap.md +++ b/module/docs/adr/0009-audit-store-swap.md @@ -280,7 +280,9 @@ Postgres transaction. warehouse. On ClickHouse, `AUDIT_EVENTS_RETENTION_DAYS` sets the table's TTL. On BigQuery the kit creates the events table with no partition expiration, and refuses at startup an events table whose partitions expire, - since its identity holds no `bigquery.tables.update` to change one; + since its identity holds no `bigquery.tables.update` to change one. Both + tables must use daily partitions and have no whole-table expiration, + including expiration inherited from the dataset when they are created; - a **details** table holding the full details of content-class types, kept for the shorter content window, `AUDIT_CONTENT_RETENTION_DAYS` — a partition expiration on BigQuery, a table TTL on ClickHouse. @@ -309,15 +311,16 @@ Postgres transaction. 6. **ClickHouse operating requirements.** - Inserts are batched: one insert per table per relay batch. A single-row insert creates a data part per insert and ends in too-many-parts errors. - `async_insert` is not used and must not be enabled for the adapter's user or - profile: the relay deletes a queue row on the insert's acknowledgement, and + Every insert explicitly sets `async_insert=0` and + `wait_for_async_insert=1`, overriding DSN and user-profile defaults: the + relay deletes a queue row on the insert's acknowledgement, and an insert acknowledged before its data is stored would let the relay delete the only copy. - A single-node deployment uses MergeTree and sends neither the quorum nor the sequential-read setting below. When `AUDIT_CLICKHOUSE_CLUSTER` is set, tables are created `ON CLUSTER` as `ReplicatedMergeTree` with ClickHouse Keeper; the cluster must exist before the adapter starts, and a table that - already exists must be a `Replicated*MergeTree` as well: a `MergeTree` table + already exists must be `ReplicatedMergeTree` as well: a `MergeTree` table lives on one node, where no quorum can make an insert durable, so startup refuses it by naming the table and its engine rather than accepting it for its columns and TTL. Every insert then @@ -330,6 +333,11 @@ Postgres transaction. so on a two-replica cluster one replica down stops appends: the insert fails at the quorum timeout, the relay retries with backoff, and the queue grows until the replica returns. + - Existing tables must use `MergeTree` or `ReplicatedMergeTree`; a declared + cluster requires the latter. Engines that sum, collapse, aggregate or + replace rows are refused on both deployment shapes: + merges must preserve every audit envelope and its details hash, including + duplicate rows left by redelivery. - The adapter's ClickHouse user holds `CREATE TABLE`, `INSERT` and `SELECT` on the database and no `ALTER`, `DELETE`, `UPDATE`, `TRUNCATE` or `DROP`, so it cannot rewrite the store of record. @@ -353,6 +361,10 @@ Postgres transaction. the deployment's events in memory at a time, outside the read budget of item 4. From then on `audit_events` receives no new rows, and Postgres holds only the queue. + Missing content details get one repair attempt per event per run, even after + several partial writes: duplicate events rows are not evidence of permanent + refusal. Verification still checks every copy and blocks removal while any + required details are missing or differ. 8. **Conformance.** Every adapter passes one shared test suite: read deduplication (after a batch is appended twice, every read returns each diff --git a/module/services/accounts/AGENTS.md b/module/services/accounts/AGENTS.md index bed96baf0..3de37da54 100644 --- a/module/services/accounts/AGENTS.md +++ b/module/services/accounts/AGENTS.md @@ -118,8 +118,9 @@ survives a restart and reaches every replica, and is only served when it is live event — and verifies them by reading them back. It reads back before it writes, and writes an event again when the store holds it without details a write supplies (`inspect`, the one rule `verify` shares), so a rerun finishes - an interrupted run, a failed details write included. That rewrite is made once - per event (a second copy in the store means it was), and an event the store + an interrupted run, repeated failed details writes included. That rewrite is + made at most once per event per run; earlier copies never prove a permanent + refusal. An event the store refuses for good fails verification alone, as the relay sets one row aside. Only with `-confirm-drop`, after a verification pass in the same run and a recount showing nothing changed, does it drop the copied partitions, named one by one through diff --git a/module/services/accounts/code/pkg/auditstore/bigquerystore/bigquerystore.go b/module/services/accounts/code/pkg/auditstore/bigquerystore/bigquerystore.go index 2dd246677..59a8de4d7 100644 --- a/module/services/accounts/code/pkg/auditstore/bigquerystore/bigquerystore.go +++ b/module/services/accounts/code/pkg/auditstore/bigquerystore/bigquerystore.go @@ -265,7 +265,9 @@ func metadataFor(spec tableSpec) *bigquery.TableMetadata { // refuses a table it finds that does not match: the writer holds no // bigquery.tables.update, so a mismatch is the deployment's to fix, and a // details table expiring at a window other than the configured one would keep -// content longer or shorter than the deployment declared. +// content longer or shorter than the deployment declared. Whole-table expiry +// is refused too, including expiry inherited from the dataset when a table is +// created: it deletes every partition regardless of the configured window. func (s *Store) Ensure(ctx context.Context) error { dataset := s.client.Dataset(s.dataset) if _, err := dataset.Metadata(ctx); err != nil { @@ -297,10 +299,16 @@ func (s *Store) Ensure(ctx context.Context) error { // conforms reports how an existing table differs from what the store writes. func conforms(spec tableSpec, existing *bigquery.TableMetadata) error { + if !existing.ExpirationTime.IsZero() { + return fmt.Errorf("table expires at %s; whole-table expiration must be disabled", existing.ExpirationTime.UTC().Format(time.RFC3339)) + } partitioning := existing.TimePartitioning if partitioning == nil || partitioning.Field != "occurred_at" { return errors.New("is not partitioned on occurred_at") } + if partitioning.Type != bigquery.DayPartitioningType { + return fmt.Errorf("partitions use %q granularity, want DAY on occurred_at", partitioning.Type) + } if partitioning.Expiration != spec.expiration { return fmt.Errorf("partitions expire after %s, the configured window is %s", partitioning.Expiration, spec.expiration) } diff --git a/module/services/accounts/code/pkg/auditstore/bigquerystore/ensure_retention_test.go b/module/services/accounts/code/pkg/auditstore/bigquerystore/ensure_retention_test.go new file mode 100644 index 000000000..d8cae5d66 --- /dev/null +++ b/module/services/accounts/code/pkg/auditstore/bigquerystore/ensure_retention_test.go @@ -0,0 +1,105 @@ +package bigquerystore + +import ( + "context" + "encoding/json" + "io" + "net/http" + "strings" + "testing" + "time" + + "cloud.google.com/go/bigquery" + "github.com/stretchr/testify/require" + bq "google.golang.org/api/bigquery/v2" + "google.golang.org/api/option" +) + +func TestEnsureRejectsUnsafeRetentionMetadata(t *testing.T) { + store := &Store{retention: 30 * 24 * time.Hour} + for _, spec := range store.tableSpecs() { + t.Run(spec.name, func(t *testing.T) { + require.NoError(t, conforms(spec, metadataFor(spec)), "daily partitions without table expiry retain the configured window") + for _, granularity := range []bigquery.TimePartitioningType{ + bigquery.HourPartitioningType, bigquery.MonthPartitioningType, bigquery.YearPartitioningType, + } { + t.Run(string(granularity), func(t *testing.T) { + meta := metadataFor(spec) + meta.TimePartitioning.Type = granularity + require.ErrorContains(t, conforms(spec, meta), "DAY") + }) + } + t.Run("whole table expiry", func(t *testing.T) { + meta := metadataFor(spec) + meta.ExpirationTime = occurredAt.Add(24 * time.Hour) + require.ErrorContains(t, conforms(spec, meta), "table expires") + }) + }) + } +} + +type retentionMetadataTransport func(*http.Request) (*http.Response, error) + +func (f retentionMetadataTransport) RoundTrip(req *http.Request) (*http.Response, error) { + return f(req) +} + +// A dataset default may add whole-table expiry to a newly created table even +// though the creation request only specifies partition expiry. Ensure must +// validate the returned table metadata, including on this first startup. +func TestEnsureChecksNewTablesForInheritedExpiration(t *testing.T) { + for _, inherited := range []bool{false, true} { + name := "no dataset expiry" + if inherited { + name = "dataset expiry inherited" + } + t.Run(name, func(t *testing.T) { + tables := map[string]*bq.Table{} + var created []string + transport := retentionMetadataTransport(func(req *http.Request) (*http.Response, error) { + code := http.StatusOK + var body any + switch { + case req.Method == http.MethodGet && strings.HasSuffix(req.URL.Path, "/datasets/audit"): + body = &bq.Dataset{DatasetReference: &bq.DatasetReference{ProjectId: "test-project", DatasetId: "audit"}} + case req.Method == http.MethodPost && strings.HasSuffix(req.URL.Path, "/tables"): + var table bq.Table + require.NoError(t, json.NewDecoder(req.Body).Decode(&table)) + if inherited { + table.ExpirationTime = occurredAt.Add(24 * time.Hour).UnixMilli() + } + created = append(created, table.TableReference.TableId) + tables[table.TableReference.TableId] = &table + body = &table + case req.Method == http.MethodGet && strings.Contains(req.URL.Path, "/tables/"): + id := req.URL.Path[strings.LastIndex(req.URL.Path, "/")+1:] + if table, ok := tables[id]; ok { + body = table + } else { + code = http.StatusNotFound + body = map[string]any{"error": map[string]any{"code": code, "message": "table not found"}} + } + default: + t.Fatalf("unexpected BigQuery request: %s %s", req.Method, req.URL.Path) + } + encoded, err := json.Marshal(body) + require.NoError(t, err) + return &http.Response{StatusCode: code, Header: http.Header{"Content-Type": []string{"application/json"}}, Body: io.NopCloser(strings.NewReader(string(encoded))), Request: req}, nil + }) + client, err := bigquery.NewClient(context.Background(), "test-project", option.WithHTTPClient(&http.Client{Transport: transport}), option.WithEndpoint("https://audit.example.test")) + require.NoError(t, err) + t.Cleanup(func() { require.NoError(t, client.Close()) }) + store, err := New(client, Config{Dataset: "audit", ContentDetailRetention: 30 * 24 * time.Hour}) + require.NoError(t, err) + err = store.Ensure(context.Background()) + if inherited { + require.ErrorContains(t, err, "table expires") + require.Equal(t, []string{EventsTable}, created, "startup stops at the first unsafe table") + } else { + require.NoError(t, err) + require.Equal(t, []string{EventsTable, DetailsTable}, created) + require.NoError(t, store.Ensure(context.Background()), "an existing valid daily table remains accepted") + } + }) + } +} diff --git a/module/services/accounts/code/pkg/auditstore/clickhousestore/clickhousestore.go b/module/services/accounts/code/pkg/auditstore/clickhousestore/clickhousestore.go index 58dc84380..2fe93eeea 100644 --- a/module/services/accounts/code/pkg/auditstore/clickhousestore/clickhousestore.go +++ b/module/services/accounts/code/pkg/auditstore/clickhousestore/clickhousestore.go @@ -114,8 +114,8 @@ type Store struct { exportMaxBytes int64 // aggregateMaxBytes is the most state an aggregation keeps before it gives up. aggregateMaxBytes int64 - // writeSettings are sent with every insert: on a cluster, the quorum that - // makes an acknowledgement durable. + // writeSettings are sent with every insert: synchronous storage acknowledgment + // and, on a cluster, the quorum that makes it durable across replicas. writeSettings clickhouse.Settings // inService, when set, is told each time an aggregation is evaluated in the // service rather than in ClickHouse; tests use it to prove which ran. @@ -179,6 +179,11 @@ func New(conn Conn, cfg Config) (*Store, error) { // An unmatched details row reads as '' — no details — not NULL. "join_use_nulls": 0, }, + // The relay removes queued rows after an acknowledged insert. A DSN or + // user profile enabling fire-and-forget async inserts must not turn that + // acknowledgment into "buffered in memory", where a crash or failed flush + // could lose the rows without a retry. The relay already batches writes. + writeSettings: clickhouse.Settings{"async_insert": 0, "wait_for_async_insert": 1}, } if cfg.Cluster != "" { // An insert is acknowledged by the replica that took it unless told to @@ -190,11 +195,9 @@ func New(conn Conn, cfg Config) (*Store, error) { // can then be read as complete against), and gives up before the // relay's own attempt does. A retried block is the same block, which a // replicated table recognizes and does not store twice. - store.writeSettings = clickhouse.Settings{ - "insert_quorum": "auto", - "insert_quorum_parallel": 0, - "insert_quorum_timeout": insertQuorumTimeoutMillis, - } + store.writeSettings["insert_quorum"] = "auto" + store.writeSettings["insert_quorum_parallel"] = 0 + store.writeSettings["insert_quorum_timeout"] = insertQuorumTimeoutMillis // A read is served only by a replica that holds every quorum insert, so // an event the relay was told is stored is never missing from a read, // and one that never reached the quorum is never in it. A replica that @@ -348,8 +351,9 @@ const unknownDatabase = 81 // and refuses a table it finds that does not match: the service's user holds // no ALTER, so a mismatch is the deployment's to fix, and a table expiring at // a window other than the configured one would keep events or content longer -// or shorter than the deployment declared. On a cluster it also refuses a table -// that is not a Replicated*MergeTree, whatever its columns and window: a table +// or shorter than the deployment declared. Every table must preserve individual +// rows: only MergeTree and ReplicatedMergeTree are accepted. On a cluster it +// also refuses a table that is not ReplicatedMergeTree: a table // on one node cannot hold a quorum insert, and the relay deletes its queue rows // on the insert's acknowledgement. func (s *Store) Ensure(ctx context.Context) error { @@ -382,6 +386,9 @@ func (s *Store) Ensure(ctx context.Context) error { return fmt.Errorf("clickhouse audit store: table %s.%s was not created", s.database, spec.name) } } + if err := rowPreserving(existing); err != nil { + return fmt.Errorf("clickhouse audit store: table %s.%s: %w", s.database, spec.name, err) + } if s.cluster != "" { if err := replicated(s.cluster, existing); err != nil { return fmt.Errorf("clickhouse audit store: table %s.%s: %w", s.database, spec.name, err) @@ -449,19 +456,31 @@ func (s *Store) describe(ctx context.Context, table string) (*tableShape, error) return shape, nil } -// replicated reports why an existing table cannot serve a cluster: its engine -// is not one of the Replicated*MergeTree engines (ReplicatedMergeTree, -// ReplicatedReplacingMergeTree, ...). A MergeTree table lives on the one node -// that holds it, so insert_quorum has no replicas to wait for and an +// rowPreserving refuses engines that replace, combine or collapse rows during +// merges. The store verifies columns and TTL, but not the key or extra columns +// those engines would need to preserve every audit event; having the expected +// schema alone therefore cannot make them safe as the store of record. +func rowPreserving(existing *tableShape) error { + switch existing.engine { + case "MergeTree", "ReplicatedMergeTree": + return nil + default: + return fmt.Errorf("engine %s is not supported for individual audit records; use MergeTree or ReplicatedMergeTree", existing.engine) + } +} + +// replicated reports why an existing row-preserving table cannot serve a +// cluster: its engine is not ReplicatedMergeTree. A MergeTree table lives on +// the one node that holds it, so insert_quorum has no replicas to wait for and an // acknowledged insert is stored once; the relay deletes its queue row on that // acknowledgement. The service's user holds no DROP or ALTER, so the // deployment replaces the table; the store creates it replicated when it is // missing. func replicated(cluster string, existing *tableShape) error { - if strings.HasPrefix(existing.engine, "Replicated") && strings.HasSuffix(existing.engine, "MergeTree") { + if existing.engine == "ReplicatedMergeTree" { return nil } - return fmt.Errorf("engine %s is not replicated, and cluster %q is configured: a cluster needs a Replicated*MergeTree table, because a quorum insert cannot be made durable on one node; replace the table with a ReplicatedMergeTree one", existing.engine, cluster) + return fmt.Errorf("engine %s is not replicated, and cluster %q is configured: a cluster needs a ReplicatedMergeTree table, because a quorum insert cannot be made durable on one node; replace the table with a ReplicatedMergeTree one", existing.engine, cluster) } // ttlClause is the TTL clause of an engine_full, up to its SETTINGS. diff --git a/module/services/accounts/code/pkg/auditstore/clickhousestore/clickhousestore_test.go b/module/services/accounts/code/pkg/auditstore/clickhousestore/clickhousestore_test.go index 5b8d2bc07..d39cb7e3b 100644 --- a/module/services/accounts/code/pkg/auditstore/clickhousestore/clickhousestore_test.go +++ b/module/services/accounts/code/pkg/auditstore/clickhousestore/clickhousestore_test.go @@ -408,7 +408,7 @@ func TestATimeBoundOutsideTheColumnRangeIsClamped(t *testing.T) { func TestAClusterInsertsWithAQuorumAndReadsSequentially(t *testing.T) { single, err := New(panicConn{}, validConfig()) require.NoError(t, err) - require.Empty(t, single.writeSettings) + require.NotContains(t, single.writeSettings, "insert_quorum") require.NotContains(t, single.readSettings, "select_sequential_consistency") cfg := validConfig() @@ -425,6 +425,10 @@ func TestAClusterInsertsWithAQuorumAndReadsSequentially(t *testing.T) { for name, value := range single.readSettings { require.Equal(t, value, clustered.readSettings[name], "the pinned reading settings are the same on a cluster: %s", name) } + for _, store := range []*Store{single, clustered} { + require.Equal(t, 0, store.writeSettings["async_insert"], "an acknowledgment must mean the rows reached storage, even under an async user profile") + require.Equal(t, 1, store.writeSettings["wait_for_async_insert"], "an insert must never acknowledge buffering alone") + } } // catalogueConn answers what Ensure asks of a server's system tables from a @@ -543,7 +547,7 @@ func same(engine string) func(string) string { return func(string) string { retu // A cluster's inserts are made durable by insert_quorum, which counts replicas; // a MergeTree table lives on one node and has none to count. The relay deletes // its queue rows on the acknowledgement, so a table found on a cluster that is -// not a Replicated*MergeTree is refused, naming the table and its engine, rather +// not ReplicatedMergeTree is refused, naming the table and its engine, rather // than accepted for having the right columns and TTL. func TestEnsureOnAClusterRequiresEveryTableToBeReplicated(t *testing.T) { clustered := validConfig() @@ -556,7 +560,7 @@ func TestEnsureOnAClusterRequiresEveryTableToBeReplicated(t *testing.T) { }{ "a cluster refuses a MergeTree events table": { cfg: clustered, engines: same("MergeTree"), - refuse: []string{"table audit.audit_events", "engine MergeTree", `cluster "audit_cluster"`, "Replicated*MergeTree"}, + refuse: []string{"table audit.audit_events", "engine MergeTree", `cluster "audit_cluster"`, "ReplicatedMergeTree"}, }, "a cluster refuses a MergeTree details table beside a replicated events table": { cfg: clustered, @@ -576,10 +580,13 @@ func TestEnsureOnAClusterRequiresEveryTableToBeReplicated(t *testing.T) { cfg: clustered, engines: same("Memory"), refuse: []string{"table audit.audit_events", "engine Memory"}, }, - "a cluster accepts ReplicatedMergeTree": {cfg: clustered, engines: same("ReplicatedMergeTree")}, - "a cluster accepts another Replicated MergeTree": {cfg: clustered, engines: same("ReplicatedReplacingMergeTree")}, - "one server accepts MergeTree": {cfg: validConfig(), engines: same("MergeTree")}, - "one server keeps accepting a replicated table": {cfg: validConfig(), engines: same("ReplicatedMergeTree")}, + "a cluster accepts ReplicatedMergeTree": {cfg: clustered, engines: same("ReplicatedMergeTree")}, + "a cluster refuses ReplicatedReplacingMergeTree": { + cfg: clustered, engines: same("ReplicatedReplacingMergeTree"), + refuse: []string{"table audit.audit_events", "engine ReplicatedReplacingMergeTree"}, + }, + "one server accepts MergeTree": {cfg: validConfig(), engines: same("MergeTree")}, + "one server keeps accepting a replicated table": {cfg: validConfig(), engines: same("ReplicatedMergeTree")}, } { t.Run(name, func(t *testing.T) { store, err := New(panicConn{}, tc.cfg) @@ -600,3 +607,30 @@ func TestEnsureOnAClusterRequiresEveryTableToBeReplicated(t *testing.T) { }) } } + +func TestEnsureRejectsTransformingEnginesForEitherTable(t *testing.T) { + for _, cluster := range []string{"", "audit_cluster"} { + for _, engine := range []string{"ReplacingMergeTree", "SummingMergeTree", "AggregatingMergeTree", "CollapsingMergeTree", "VersionedCollapsingMergeTree"} { + for _, prefix := range []string{"", "Replicated"} { + for _, table := range []string{EventsTable, DetailsTable} { + t.Run(cluster+"/"+prefix+engine+"/"+table, func(t *testing.T) { + cfg := validConfig() + cfg.Cluster = cluster + store, err := New(panicConn{}, cfg) + require.NoError(t, err) + conn := existingTables(store, func(name string) string { + if name == table { + return prefix + engine + } + return "ReplicatedMergeTree" + }) + store.conn = conn + err = store.Ensure(context.Background()) + require.ErrorContains(t, err, "table audit."+table+": engine "+prefix+engine) + require.Empty(t, conn.executed) + }) + } + } + } + } +} diff --git a/module/services/accounts/code/pkg/auditstore/clickhousestore/server_test.go b/module/services/accounts/code/pkg/auditstore/clickhousestore/server_test.go index 788bcb866..e03daafa6 100644 --- a/module/services/accounts/code/pkg/auditstore/clickhousestore/server_test.go +++ b/module/services/accounts/code/pkg/auditstore/clickhousestore/server_test.go @@ -78,6 +78,58 @@ func TestServerEnsureRefusesAMissingDatabase(t *testing.T) { require.ErrorContains(t, store.Ensure(context.Background()), "does not exist; the deployment creates it") } +func TestServerAppendOverridesFireAndForgetProfile(t *testing.T) { + _, database := serverDatabase(t) + options, err := clickhouse.ParseDSN(os.Getenv("AUDIT_CLICKHOUSE_TEST_DSN")) + require.NoError(t, err) + options.Auth.Database = database + if options.Settings == nil { + options.Settings = clickhouse.Settings{} + } + options.Settings["async_insert"] = 1 + options.Settings["wait_for_async_insert"] = 0 + options.Settings["async_insert_use_adaptive_busy_timeout"] = 0 + options.Settings["async_insert_busy_timeout_ms"] = 60_000 + conn, err := clickhouse.Open(options) + require.NoError(t, err) + t.Cleanup(func() { _ = conn.Close() }) + store := serverStore(t, conn, database) + ctx := context.Background() + require.NoError(t, store.Ensure(ctx)) + org, actor := uuid.NewString(), uuid.NewString() + at := time.Now().UTC().Truncate(time.Microsecond) + appendBatch(t, store, testDeployment, + newRecord(t, org, actor, business.EventAuthLogin, "", at, business.RetentionSecurity, map[string]any{"method": "password"}), + newRecord(t, org, actor, business.EventDocumentRead, "doc-1", at.Add(-time.Second), business.RetentionContent, map[string]any{"boundary": "b-1"}), + ) + read := business.AuditRead{Scope: business.OrganizationAuditScope(org), Query: business.AuditQuery{OrgID: org}} + listed, _, err := store.ListAuditEvents(ctx, read) + require.NoError(t, err) + require.Len(t, listed, 2, "acknowledged events must already be stored, rather than waiting in an async buffer") + require.Equal(t, map[string]any{"boundary": "b-1"}, listed[1].Payload, "the content details insert must also be synchronous") +} + +func TestServerEnsureRejectsTransformingEngines(t *testing.T) { + for _, engine := range []string{"ReplacingMergeTree", "SummingMergeTree", "AggregatingMergeTree"} { + for _, table := range []string{EventsTable, DetailsTable} { + t.Run(engine+"/"+table, func(t *testing.T) { + conn, database := serverDatabase(t) + store := serverStore(t, conn, database) + ctx := context.Background() + for _, spec := range store.tableSpecs() { + statement := store.createStatement(spec) + if spec.name == table { + statement = strings.Replace(statement, "ENGINE = MergeTree", "ENGINE = "+engine, 1) + statement = strings.Replace(statement, "ORDER BY (deployment_id, org_id, occurred_at, event_id)", "ORDER BY (deployment_id, org_id)", 1) + } + require.NoError(t, conn.Exec(ctx, statement)) + } + require.ErrorContains(t, store.Ensure(ctx), "table "+database+"."+table+": engine "+engine) + }) + } + } +} + func TestServerEnsureCreatesBothTablesOnceAndRefusesAMismatch(t *testing.T) { conn, database := serverDatabase(t) ctx := context.Background() diff --git a/module/services/accounts/code/pkg/business/audit_history.go b/module/services/accounts/code/pkg/business/audit_history.go index ec2cf875b..10effbe9a 100644 --- a/module/services/accounts/code/pkg/business/audit_history.go +++ b/module/services/accounts/code/pkg/business/audit_history.go @@ -172,10 +172,10 @@ type AuditHistoryPartitionReport struct { // Rewritten is how many events this run wrote again because the store held // them without details that a write supplies (a content-class event whose // details write was lost); the rest were already complete. An event is - // written again only while the store holds one copy of it, so a rewrite that - // does not supply the details is made once, not on every run. Events the - // store refused for good are in neither Copied nor Rewritten: they are - // problems. + // written at most once by this run, whatever the number of earlier copies: + // repeated transient details failures can leave several events rows behind. + // Events the store refused for good are in neither Copied nor Rewritten: + // they are problems. Rewritten int // Orgs counts the partition's events per organization ("" is the // platform's). @@ -699,11 +699,11 @@ func (c *AuditHistoryCopy) verify(record AuditRecord, copies []StoredAuditEvent) // operator to look at. When copies differ in what is wrong, the reason that no // write removes is the one returned. // -// A write for missing details is made once: when the store holds more than one -// copy of the event, a write has already been made and did not supply them -// (refused for good, or written and not readable), and another would add a copy -// and an archive object and conclude the same. The failure stays, with the count -// of copies as its reason. +// Earlier copies do not establish why details are missing: each can come from a +// transient failure after its events row landed. Every run gets one write pass +// for incomplete events, even after several failures. A permanent row refusal +// ends that event's write attempt in this run; missing details still prevent +// verification and removal, and a later run can repair them after recovery. func (c *AuditHistoryCopy) inspect(record AuditRecord, copies []StoredAuditEvent) (problem string, writable bool) { if len(copies) == 0 { return "missing from the store", true @@ -720,12 +720,6 @@ func (c *AuditHistoryCopy) inspect(record AuditRecord, copies []StoredAuditEvent } } } - if writable && len(copies) > 1 { - if problem != "" { - problem = fmt.Sprintf("%s; the store holds %d copies, so an earlier write did not supply them, and the event is not written again", problem, len(copies)) - } - return problem, false - } return problem, writable } diff --git a/module/services/accounts/code/pkg/business/audit_history_test.go b/module/services/accounts/code/pkg/business/audit_history_test.go index 41424b9b6..12b42590e 100644 --- a/module/services/accounts/code/pkg/business/audit_history_test.go +++ b/module/services/accounts/code/pkg/business/audit_history_test.go @@ -1077,6 +1077,35 @@ func historyInspector(t *testing.T, retention time.Duration, now time.Time) *Aud return copier } +// Repeated transient failures can leave several events rows before any details +// row lands. A healthy run must still supply the details to all those copies. +func TestAuditHistoryResumesAfterConsecutiveDetailsWriteFailures(t *testing.T) { + f := newHistoryFixture() + f.split = &splitHistoryStore{} + bad := f.source.rows[0].ID + for range 2 { + f.split.failDetails = f.split.appends + 1 + report, err := f.copier(t, AuditHistoryCopyConfig{}).Run(context.Background(), AuditHistoryCopyOptions{}) + require.ErrorContains(t, err, "details write failed") + require.False(t, report.Verified) + require.Empty(t, f.source.drops) + } + require.Equal(t, 2, f.split.copies()[bad]) + + f.split.failDetails = 0 + report, err := f.copier(t, AuditHistoryCopyConfig{}).Run(context.Background(), AuditHistoryCopyOptions{}) + require.NoError(t, err) + require.True(t, report.Verified) + require.Equal(t, int64(len(f.source.rows)), totalsOf(report).verified) + require.Equal(t, 3, f.split.copies()[bad], "the healthy repair completes every earlier events row") + require.Empty(t, f.source.drops, "repair alone never authorizes removal") + + appends := f.split.appends + _, err = f.copier(t, AuditHistoryCopyConfig{}).Run(context.Background(), AuditHistoryCopyOptions{}) + require.NoError(t, err) + require.Equal(t, appends, f.split.appends, "complete events are not written again") +} + func TestAuditHistoryInspectSaysWhatAWriteRemoves(t *testing.T) { copier := historyInspector(t, 7*24*time.Hour, historyNow) // recent is held; margin is a day the store may have expired a few hours ago @@ -1125,14 +1154,12 @@ func TestAuditHistoryInspectSaysWhatAWriteRemoves(t *testing.T) { record: inside, copies: []StoredAuditEvent{storedOf(inside, func(e *StoredAuditEvent) { otherOrg(e); hasDetails(inside)(e) }), storedOf(inside, nil)}, problem: "stored envelope differs from the row", }, - // A write for missing details is made once: a second copy of the event - // beside the first is the proof that one was made. - "every copy missing details, so a write was made": { + "several incomplete copies can still be repaired": { record: inside, copies: []StoredAuditEvent{storedOf(inside, nil), storedOf(inside, nil)}, - problem: "content details are missing inside the content window; the store holds 2 copies, so an earlier write did not supply them, and the event is not written again", + problem: "content details are missing inside the content window", writable: true, }, - "every copy missing details in the margin, so a write was made": { - record: inMargin, copies: []StoredAuditEvent{storedOf(inMargin, nil), storedOf(inMargin, nil)}, + "several incomplete copies in the margin can still be repaired": { + record: inMargin, copies: []StoredAuditEvent{storedOf(inMargin, nil), storedOf(inMargin, nil)}, writable: true, }, "two copies, the details attached": { record: inside, copies: []StoredAuditEvent{storedOf(inside, hasDetails(inside)), storedOf(inside, hasDetails(inside))}, @@ -1173,9 +1200,9 @@ func totalsOf(report AuditHistoryReport) historyTotals { // landed — as ClickHouse does when its isolation finds the row, and as BigQuery // does when it reports the row invalid (and the rest of that request stopped). // That event is a verification failure of its own. It does not stop the copy of -// the others, and it is not written again on every run: a write for missing -// details is made once, and the run that finds one already made says so. -func TestAuditHistoryAPermanentlyRefusedDetailsRowStopsNothingAndIsNotRepeated(t *testing.T) { +// the others. Each run tries the incomplete event once and keeps verification +// false while the row is refused; it never loops on that event within the run. +func TestAuditHistoryAPermanentlyRefusedDetailsRowStopsNothingAndRetriesOncePerRun(t *testing.T) { for name, stopRest := range map[string]bool{"the rest written around the refused row": false, "the rest of the request stopped": true} { t.Run(name, func(t *testing.T) { f := newHistoryFixture() @@ -1208,8 +1235,8 @@ func TestAuditHistoryAPermanentlyRefusedDetailsRowStopsNothingAndIsNotRepeated(t require.Zero(t, september.Failures) events, archived := len(f.split.events), len(f.archive.batches) - // The next run makes the one write for the details it is missing and is - // refused again; nothing else is written. + // The next run attempts the missing details once and is refused again; + // nothing else is written. _, totals = run() require.Zero(t, totals.copied+totals.rewritten) require.Equal(t, 2, f.split.attempts[bad]) @@ -1219,17 +1246,18 @@ func TestAuditHistoryAPermanentlyRefusedDetailsRowStopsNothingAndIsNotRepeated(t require.Equal(t, []string{bad}, historyRecordIDs(f.archive.batches[len(f.archive.batches)-1].Records)) require.Contains(t, totals.problems[0], "the store refused the event's row for good") - // Neither this run nor the ones after it write or archive anything. - appends := f.split.appends - for range 3 { + // Duplicate events rows do not disable future repair attempts. Each + // bounded run tries only the incomplete event, then fails verification. + for retry := range 3 { + appends := f.split.appends + objects := len(f.archive.batches) _, totals = run() require.Zero(t, totals.copied+totals.rewritten) - require.Equal(t, appends, f.split.appends, "nothing is appended") - require.Equal(t, events+1, len(f.split.events)) - require.Len(t, f.archive.batches, archived+1, "nothing is archived") - require.Equal(t, 2, f.split.attempts[bad]) - require.Contains(t, totals.problems[0], "the store holds 2 copies") - require.Contains(t, totals.problems[0], "not written again") + require.Equal(t, appends+1, f.split.appends, "one attempt per run") + require.Equal(t, events+2+retry, len(f.split.events)) + require.Len(t, f.archive.batches, objects+1, "only the incomplete event is archived") + require.Equal(t, 3+retry, f.split.attempts[bad]) + require.Contains(t, totals.problems[0], "the store refused the event's row for good") } }) } @@ -1300,10 +1328,9 @@ func (s *refusingHistoryStore) AppendAuditBatch(context.Context, AuditBatch) err return s.err } -// A write whose details the store accepts and then never shows is made once in a -// run, once more by the next, and not again: the run that finds the second copy -// says that a write was made and did not supply them. -func TestAuditHistoryDetailsWrittenButNeverReadableAreNotWrittenAgainAndAgain(t *testing.T) { +// Details the store accepts but does not yet show cause at most one repair +// attempt per run. The event remains unverified until its details are readable. +func TestAuditHistoryUnreadableDetailsAreRetriedOncePerRun(t *testing.T) { f := newHistoryFixture() bad := f.source.rows[0].ID f.split = &splitHistoryStore{unreadable: map[string]bool{bad: true}} @@ -1328,14 +1355,21 @@ func TestAuditHistoryDetailsWrittenButNeverReadableAreNotWrittenAgainAndAgain(t require.Equal(t, 2, f.split.attempts[bad]) require.Contains(t, totals.problems[0], "written by this run, but the read-back finds it incomplete") - appends, archived := f.split.appends, len(f.archive.batches) for range 2 { + appends, archived := f.split.appends, len(f.archive.batches) totals = run() - require.Zero(t, totals.copied+totals.rewritten) - require.Equal(t, appends, f.split.appends) - require.Len(t, f.archive.batches, archived) - require.Contains(t, totals.problems[0], "an earlier write did not supply them") + require.Equal(t, 1, totals.rewritten) + require.Zero(t, totals.copied) + require.Equal(t, appends+1, f.split.appends) + require.Len(t, f.archive.batches, archived+1) + require.Contains(t, totals.problems[0], "written by this run, but the read-back finds it incomplete") } + delete(f.split.unreadable, bad) + appends := f.split.appends + report, err := f.copier(t, AuditHistoryCopyConfig{}).Run(context.Background(), AuditHistoryCopyOptions{}) + require.NoError(t, err) + require.True(t, report.Verified) + require.Equal(t, appends, f.split.appends, "readable details complete every copy without another write") } // The rule for the details of a content-class event, at the instants where it @@ -1433,11 +1467,11 @@ func TestAuditHistoryContentDetailsInTheMarginAreWrittenForAndNeverFailed(t *tes for id, attempts := range map[string]int{past: 1, margin1: 2, margin2: 2, held: 2} { require.Equal(t, attempts, f.split.attempts[id], id) } - appends := f.split.appends for range 2 { totals = run() - require.Zero(t, totals.copied+totals.rewritten) - require.Equal(t, appends, f.split.appends) + require.Zero(t, totals.copied) + require.Equal(t, 3, totals.rewritten, "the held event and both margin events remain eligible for a bounded repair") + require.Equal(t, 1, f.split.attempts[past], "expired details are not rewritten") } } From 48ced8dd6c753e3a70018e451ec82fea65e1acf8 Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Wed, 7 Oct 2026 13:51:39 -0400 Subject: [PATCH 56/69] fix(telemetry): require platform state outside local runtimes Remove TELEMETRY_STATE from module defaults that deployed cells inherit. Infer an absent collector only when the SDK identifies a local runtime, and retain refusal of explicit invalid or incomplete configuration. Cover shipped defaults, both runtime modes and startup checks in both producers; update the operator docs and module breaking-change notice. --- .../skills/run-the-starter-locally/SKILL.md | 14 +-- DOGFOODING.md | 4 +- module/FEATURES.md | 11 ++- module/configurations/local/observability.env | 7 +- module/module.package.codefly.yaml | 2 +- .../accounts/code/telemetry_destination.go | 15 ++- .../code/telemetry_destination_test.go | 99 ++++++++++++++----- .../accounts/code/telemetry_metrics_test.go | 2 +- module/services/accounts/code/work.go | 14 +-- module/services/auth-gateway/code/main.go | 4 +- .../code/telemetry_destination.go | 15 ++- .../code/telemetry_destination_test.go | 90 +++++++++++++---- .../code/telemetry_metrics_test.go | 2 +- 13 files changed, 200 insertions(+), 79 deletions(-) diff --git a/.claude/skills/run-the-starter-locally/SKILL.md b/.claude/skills/run-the-starter-locally/SKILL.md index 1f96ddc95..2e9eb0259 100644 --- a/.claude/skills/run-the-starter-locally/SKILL.md +++ b/.claude/skills/run-the-starter-locally/SKILL.md @@ -18,13 +18,13 @@ codefly run service --fixture dev-admin - No TTY (CI, pipes, MCP) auto-enables `--headless`. - Docker must be running. `codefly doctor` checks prerequisites; `codefly clear` reaps stray processes and containers between runs. -- A local run has no cell collector, and the module ships none. The `observability` - group (`configurations/local/observability.env`) says so — `TELEMETRY_STATE=absent` - with a reason — and accounts and auth-gateway log the reason once, send traces to - wool's stdout tracer and export no metrics. They refuse to start, naming - `TELEMETRY_STATE`, when that group did not arrive: a composing workspace that - overrode it without the keys, or a `local-dogfood` run without its copied - `observability.env`. +- A local run has no cell collector, and the module ships none. The module's + `configurations/local/observability.env` supplies a reason but no + `TELEMETRY_STATE`: those defaults also reach deployed cells. Only a runtime the + SDK identifies as local infers `absent` when the state is missing, logs the + reason once, sends traces to wool's stdout tracer and exports no metrics. + Outside that runtime, a missing state refuses startup. The `local-dogfood` + profile must still copy its explicit state from `observability.env.example`. For a real external identity provider and the production-grade provider stack, use the `local-dogfood` environment and its configuration groups: diff --git a/DOGFOODING.md b/DOGFOODING.md index f6dc63229..5085e0f1a 100644 --- a/DOGFOODING.md +++ b/DOGFOODING.md @@ -219,7 +219,7 @@ the same Codefly-managed graph. For each of `email` (Resend), `configurations/local-dogfood/.env.example` and `.secret.env.example` to their Git-ignored names and fill them in ([LOCAL_DOGFOODING.md](./LOCAL_DOGFOODING.md#configure-providers-through-codefly)). -Keep `observability.env.example` as it is for a local run with no collector (`TELEMETRY_STATE=absent`). +Keep `observability.env.example` as it is for a `local-dogfood` run with no collector (`TELEMETRY_STATE=absent`). The plain `local` runtime infers `absent` when the state is missing; the module's local defaults do not declare it. Turnstile's published test keys drive the three abuse-protection cases: | Case | `NEXT_PUBLIC_TURNSTILE_SITE_KEY` | `TURNSTILE_SECRET_KEY` | @@ -239,7 +239,7 @@ codefly run service --env local-dogfood - [ ] Grant analytics consent, navigate through onboarding, and verify bounded browser plus durable backend events in PostHog. - [ ] Withdraw analytics consent or log out. Browser capture stops immediately and identity resets. - [ ] Trigger controlled browser and backend errors. Sentry correlates release and environment without creating performance transactions. -- [ ] With `TELEMETRY_STATE=absent`, check that accounts and auth-gateway log the absent reason once at startup, then exercise login/onboarding and observe wool's stdout trace output (metrics are not exported; logs are on stdout). Remove `TELEMETRY_STATE` and confirm both services refuse to start, naming it. +- [ ] With `TELEMETRY_STATE=absent`, check that accounts and auth-gateway log the absent reason once at startup, then exercise login/onboarding and observe wool's stdout trace output (metrics are not exported; logs are on stdout). Remove `TELEMETRY_STATE` in `local-dogfood` and confirm both services refuse to start, naming it. In the plain `local` runtime, a missing state instead starts with an absent collector. - [ ] Switch Turnstile to the **fail** keys. Registration and waitlist submission fail without database writes. - [ ] Switch Turnstile to the **replay** keys. The first deterministic verification follows Cloudflare's fixture behavior; replay rejection leaves state unchanged. diff --git a/module/FEATURES.md b/module/FEATURES.md index 15c4b824e..c38d15d93 100644 --- a/module/FEATURES.md +++ b/module/FEATURES.md @@ -655,7 +655,7 @@ Environment variables consumed by the api: | `POSTHOG_API_HOST` | Separate PostHog management/deletion origin | | `ERROR_TRACKING_MODE` | Explicit `disabled` or `sentry`; rejects partial config | | `SENTRY_DSN` | Server Sentry DSN, required in Sentry mode | -| `TELEMETRY_STATE` | `observability` group, delivered by the platform: `available` or `absent`, never empty; accounts and auth-gateway refuse to start without it | +| `TELEMETRY_STATE` | `observability` group, delivered by the platform: `available` or `absent`; missing state refuses startup except in a local runtime, where it means `absent` | | `OTEL_EXPORTER_OTLP_ENDPOINT` | `observability` group: the cell collector's OTLP/gRPC address, required when `available` and ignored when `absent`; `http://` is plaintext on the wire because the mesh supplies mTLS, `https://` is TLS | | `TELEMETRY_ABSENT_REASON` | `observability` group: why the cell has no collector, required when `absent` and ignored when `available`; logged once at startup | | `ABUSE_PROTECTION_MODE` | Explicit `disabled` or `turnstile` | @@ -672,9 +672,12 @@ then uses wool's stdout tracer for traces. The state decides: a cell's values override the module's local defaults one key at a time, so a deployed cell that says `available` with an endpoint can still carry the `absent` reason the local profile left behind. The key the state does not use is ignored, and a startup -warning names it once. Anything else — the state missing or unknown, `available` -without an endpoint, or `absent` without a reason — refuses to start, because a -group that did not arrive is not a cell without a collector. An `https://` +warning names it once. Module defaults never declare `TELEMETRY_STATE`, because +they also reach deployed cells. Only a local runtime infers `absent` from a +missing state; elsewhere missing state refuses startup, because a group that +did not arrive is not a cell without a collector. Unknown state, `available` +without an endpoint, or explicit `absent` without a reason also refuses startup, +including locally. An `https://` endpoint is refused at startup for now: wool's OTLP tracer dials plaintext only, so it would be sent in the clear and reported as TLS. Metrics leave by OTLP push alone — decided 2026-10-06, because the cell's diff --git a/module/configurations/local/observability.env b/module/configurations/local/observability.env index 4d7ddcb97..bce6d701d 100644 --- a/module/configurations/local/observability.env +++ b/module/configurations/local/observability.env @@ -1,6 +1,6 @@ # Where accounts and auth-gateway send traces and metrics, or that they send none. # The platform delivers exactly one of two answers, and a process that gets neither -# refuses to start rather than guess: +# refuses to start outside a local runtime rather than guess: # # TELEMETRY_STATE=available the cell has a collector. Also sets # OTEL_EXPORTER_OTLP_ENDPOINT to its OTLP/gRPC address, @@ -15,6 +15,7 @@ # process ignores the key its state does not use and logs that once. # # A local run has no cell, so it has no collector: traces go to wool's stdout tracer -# and metrics are not exported. Logs always go to stdout. -TELEMETRY_STATE=absent +# and metrics are not exported. Logs always go to stdout. Do not declare a state +# here: these defaults also reach deployed cells. Only a local runtime infers +# `absent` when the platform has not delivered TELEMETRY_STATE. TELEMETRY_ABSENT_REASON=A local run has no cell collector; traces go to the stdout tracer and metrics are not exported. diff --git a/module/module.package.codefly.yaml b/module/module.package.codefly.yaml index eacfad85f..2a146daf8 100644 --- a/module/module.package.codefly.yaml +++ b/module/module.package.codefly.yaml @@ -98,7 +98,7 @@ migrations: breaking-changes: - Consumers resolve the immutable package into a disposable projection instead of editing Starter-owned files. - "Audit event types are namespaced under `saas.*` (`auth.login` -> `saas.auth.login`). The wire value of `audit_events.event_type`, the `X-Webhook-Event` header, and webhook routing keys all change; migration 116 rewrites stored history and existing subscriptions in place, so a live subscriber keeps firing, but a consumer that re-creates a subscription from a hardcoded pre-namespace name, or emits one through `ModuleEmitAuditEvent`, is now rejected. No alias layer." - - "The `telemetry` service is removed: the module no longer ships an OTLP gateway, and the cell's collector takes its place. `accounts` and `auth-gateway` read where it is from the `observability` configuration group — `TELEMETRY_STATE` is `available` with `OTEL_EXPORTER_OTLP_ENDPOINT` (an `http://` endpoint is plaintext on the wire because the mesh supplies mTLS; `https://` is TLS), or `absent` with `TELEMETRY_ABSENT_REASON` — and refuse to start when the group does not say which. `OBSERVABILITY_EXPORTER` and `OTEL_EXPORTER_OTLP_HEADERS` no longer mean anything, the `/metrics` scrape endpoint of both services is gone (metrics leave by OTLP push only), and `service.name` is no longer fixed in code: `OTEL_SERVICE_NAME` or `OTEL_RESOURCE_ATTRIBUTES` sets it, else the service's own Codefly identity. A consumer that depended on the `telemetry` service, or that overrides the `observability` group, must move to the keys above." + - "The `telemetry` service is removed: the module no longer ships an OTLP gateway, and the cell's collector takes its place. `accounts` and `auth-gateway` read where it is from the `observability` configuration group — `TELEMETRY_STATE` is `available` with `OTEL_EXPORTER_OTLP_ENDPOINT` (an `http://` endpoint is plaintext on the wire because the mesh supplies mTLS; `https://` is TLS), or `absent` with `TELEMETRY_ABSENT_REASON` — and refuse to start outside a local runtime when the group does not say which. Module defaults do not declare TELEMETRY_STATE; only a local runtime infers absent when the state is missing. `OBSERVABILITY_EXPORTER` and `OTEL_EXPORTER_OTLP_HEADERS` no longer mean anything, the `/metrics` scrape endpoint of both services is gone (metrics leave by OTLP push only), and `service.name` is no longer fixed in code: `OTEL_SERVICE_NAME` or `OTEL_RESOURCE_ATTRIBUTES` sets it, else the service's own Codefly identity. A consumer that depended on the `telemetry` service, or that overrides the `observability` group, must move to the keys above." reserved-namespaces: - "@codefly/saas-" - "@codefly-dev/saas-" diff --git a/module/services/accounts/code/telemetry_destination.go b/module/services/accounts/code/telemetry_destination.go index c752d82fd..cecf41049 100644 --- a/module/services/accounts/code/telemetry_destination.go +++ b/module/services/accounts/code/telemetry_destination.go @@ -42,7 +42,8 @@ import ( // A group that did not arrive is not a cell without a backend. Reading the first // as the second would leave a deployment that converges green and exports // nothing, so a missing or unknown state, or the key the state requires being -// absent, refuses to start, naming what is missing. +// absent, refuses to start, naming what is missing. Only a local runtime may +// infer `absent` when the state is missing; module defaults never declare it. // // This file exists byte for byte in services/accounts/code and // services/auth-gateway/code: the two services are independent Go modules with @@ -124,17 +125,23 @@ func observabilityValue(key string) string { // configuredTelemetryDestination resolves the destination from the // `observability` group. -func configuredTelemetryDestination() (telemetryDestination, error) { - return resolveTelemetryDestination(observabilityValue) +func configuredTelemetryDestination(local bool) (telemetryDestination, error) { + return resolveTelemetryDestination(local, observabilityValue) } // resolveTelemetryDestination is the whole decision, over any reader of the // group's keys so a test can reach every outcome with nothing running. -func resolveTelemetryDestination(read func(key string) string) (telemetryDestination, error) { +func resolveTelemetryDestination(local bool, read func(key string) string) (telemetryDestination, error) { state := strings.TrimSpace(read(telemetryStateKey)) endpoint := strings.TrimSpace(read(telemetryEndpointKey)) reason := boundedLine(read(telemetryAbsentReasonKey), maxAbsentReasonLength) + if state == "" && local { + state = telemetryStateAbsent + if reason == "" { + reason = "A local run has no cell collector; traces go to the stdout tracer and metrics are not exported." + } + } if state == "" { return telemetryDestination{}, errors.New( "observability: " + telemetryStateKey + " is not set in the `observability` configuration group. " + diff --git a/module/services/accounts/code/telemetry_destination_test.go b/module/services/accounts/code/telemetry_destination_test.go index 4ac8a2b75..9c2010437 100644 --- a/module/services/accounts/code/telemetry_destination_test.go +++ b/module/services/accounts/code/telemetry_destination_test.go @@ -3,6 +3,7 @@ package main import ( "context" "net" + "os" "strings" "testing" "time" @@ -21,9 +22,56 @@ func group(values map[string]string) func(string) string { const collector = "http://otel-collector.otel-collector.svc.cluster.local:4317" +// Module defaults are also inherited by cells. They must not supply the state +// that only the platform can declare for a deployed workload. +func TestTelemetryDestinationShippedDefaultsCannotMaskMissingCellState(t *testing.T) { + contents, err := os.ReadFile("../../../configurations/local/observability.env") + require.NoError(t, err) + values := map[string]string{} + for _, line := range strings.Split(string(contents), "\n") { + line = strings.TrimSpace(line) + if line == "" || strings.HasPrefix(line, "#") { + continue + } + key, value, ok := strings.Cut(line, "=") + require.True(t, ok) + values[key] = value + } + _, err = resolveTelemetryDestination(false, group(values)) + require.ErrorContains(t, err, "TELEMETRY_STATE is not set") + destination, err := resolveTelemetryDestination(true, group(values)) + require.NoError(t, err) + require.False(t, destination.Available()) + require.NotEmpty(t, destination.AbsentReason) +} + +func TestTelemetryDestinationMissingStateDefaultsOnlyInLocalRuntime(t *testing.T) { + for _, values := range []map[string]string{ + {}, + {"TELEMETRY_STATE": " "}, + {"TELEMETRY_ABSENT_REASON": "local defaults inherited by a cell"}, + {"OTEL_EXPORTER_OTLP_ENDPOINT": collector}, + } { + _, err := resolveTelemetryDestination(false, group(values)) + require.ErrorContains(t, err, "TELEMETRY_STATE is not set") + destination, err := resolveTelemetryDestination(true, group(values)) + require.NoError(t, err) + require.False(t, destination.Available()) + require.NotEmpty(t, destination.AbsentReason) + } + for _, values := range []map[string]string{ + {"TELEMETRY_STATE": "unknown"}, + {"TELEMETRY_STATE": "available"}, + {"TELEMETRY_STATE": "absent"}, + } { + _, err := resolveTelemetryDestination(true, group(values)) + require.Error(t, err, "an explicit state must be valid and complete even locally") + } +} + // The first outcome: the platform delivered a collector. func TestTelemetryDestinationAvailableExportsToTheEndpoint(t *testing.T) { - destination, err := resolveTelemetryDestination(group(map[string]string{ + destination, err := resolveTelemetryDestination(false, group(map[string]string{ "TELEMETRY_STATE": "available", "OTEL_EXPORTER_OTLP_ENDPOINT": collector, })) @@ -56,7 +104,7 @@ func TestTelemetryDestinationReadsTransportFromTheScheme(t *testing.T) { "https://10.0.0.7:4317": {"https://10.0.0.7:4317", "10.0.0.7:4317", false}, "http://otel-collector.ns.svc:4317/": {"http://otel-collector.ns.svc:4317", "otel-collector.ns.svc:4317", true}, } { - destination, err := resolveTelemetryDestination(group(map[string]string{ + destination, err := resolveTelemetryDestination(false, group(map[string]string{ "TELEMETRY_STATE": "available", "OTEL_EXPORTER_OTLP_ENDPOINT": endpoint, })) @@ -70,7 +118,7 @@ func TestTelemetryDestinationReadsTransportFromTheScheme(t *testing.T) { // The second outcome: the cell has no collector, and says why. The process boots // and exports nothing. func TestTelemetryDestinationAbsentCarriesTheReason(t *testing.T) { - destination, err := resolveTelemetryDestination(group(map[string]string{ + destination, err := resolveTelemetryDestination(false, group(map[string]string{ "TELEMETRY_STATE": "absent", "TELEMETRY_ABSENT_REASON": "This cell's bucket has no ops cell, so no collector is rendered.", })) @@ -85,7 +133,7 @@ func TestTelemetryDestinationAbsentCarriesTheReason(t *testing.T) { // The reason is operator-written free text that reaches a log line. func TestTelemetryDestinationAbsentReasonIsOneBoundedLine(t *testing.T) { - destination, err := resolveTelemetryDestination(group(map[string]string{ + destination, err := resolveTelemetryDestination(false, group(map[string]string{ "TELEMETRY_STATE": "absent", "TELEMETRY_ABSENT_REASON": "no collector\n2026-01-01 FORGED entry\r\n" + strings.Repeat("x", 1000), })) @@ -170,7 +218,7 @@ func TestTelemetryDestinationRefusesEverythingElse(t *testing.T) { }, } { t.Run(name, func(t *testing.T) { - destination, err := resolveTelemetryDestination(group(tc.values)) + destination, err := resolveTelemetryDestination(false, group(tc.values)) require.Error(t, err) require.ErrorContains(t, err, tc.want) require.False(t, destination.Available(), "a refusal must never read as a usable destination") @@ -184,7 +232,7 @@ func TestTelemetryDestinationRefusesEverythingElse(t *testing.T) { // cell beside the reason the local profile left behind. That is not a // contradiction to refuse: the reason is ignored, and said so once. func TestTelemetryDestinationAvailableIgnoresALeftoverReason(t *testing.T) { - destination, err := resolveTelemetryDestination(group(map[string]string{ + destination, err := resolveTelemetryDestination(false, group(map[string]string{ "TELEMETRY_STATE": "available", "OTEL_EXPORTER_OTLP_ENDPOINT": collector, "TELEMETRY_ABSENT_REASON": "A local run has no cell collector.", @@ -210,7 +258,7 @@ func TestTelemetryDestinationAbsentIgnoresALeftoverEndpoint(t *testing.T) { "an unreadable endpoint": "http://[::1", } { t.Run(name, func(t *testing.T) { - destination, err := resolveTelemetryDestination(group(map[string]string{ + destination, err := resolveTelemetryDestination(false, group(map[string]string{ "TELEMETRY_STATE": "absent", "TELEMETRY_ABSENT_REASON": "This cell has no ops cell.", "OTEL_EXPORTER_OTLP_ENDPOINT": leftover, @@ -230,13 +278,13 @@ func TestTelemetryDestinationAbsentIgnoresALeftoverEndpoint(t *testing.T) { // A blank leftover is nothing left over. func TestTelemetryDestinationBlankLeftoversAreNotNoticed(t *testing.T) { - available, err := resolveTelemetryDestination(group(map[string]string{ + available, err := resolveTelemetryDestination(false, group(map[string]string{ "TELEMETRY_STATE": "available", "OTEL_EXPORTER_OTLP_ENDPOINT": collector, "TELEMETRY_ABSENT_REASON": " \n ", })) require.NoError(t, err) require.Empty(t, available.IgnoredNotice()) - absent, err := resolveTelemetryDestination(group(map[string]string{ + absent, err := resolveTelemetryDestination(false, group(map[string]string{ "TELEMETRY_STATE": "absent", "TELEMETRY_ABSENT_REASON": "no collector", "OTEL_EXPORTER_OTLP_ENDPOINT": " ", })) require.NoError(t, err) @@ -266,7 +314,7 @@ func TestTelemetryDestinationTheCellOverridesTheModuleDefaults(t *testing.T) { } t.Run("a cell with a collector overrides the local absent default", func(t *testing.T) { - destination, err := resolveTelemetryDestination(layered(moduleDefault, map[string]string{ + destination, err := resolveTelemetryDestination(false, layered(moduleDefault, map[string]string{ "TELEMETRY_STATE": "available", "OTEL_EXPORTER_OTLP_ENDPOINT": collector, })) @@ -278,7 +326,7 @@ func TestTelemetryDestinationTheCellOverridesTheModuleDefaults(t *testing.T) { }) t.Run("a cell with no collector, over a default that had an endpoint, is absent", func(t *testing.T) { - destination, err := resolveTelemetryDestination(layered( + destination, err := resolveTelemetryDestination(false, layered( map[string]string{"TELEMETRY_STATE": "available", "OTEL_EXPORTER_OTLP_ENDPOINT": collector}, map[string]string{"TELEMETRY_STATE": "absent", "TELEMETRY_ABSENT_REASON": "This cell has no ops cell."}, )) @@ -289,11 +337,11 @@ func TestTelemetryDestinationTheCellOverridesTheModuleDefaults(t *testing.T) { }) t.Run("a cell that overrides only the state still has to supply what it requires", func(t *testing.T) { - _, err := resolveTelemetryDestination(layered(moduleDefault, map[string]string{"TELEMETRY_STATE": "available"})) + _, err := resolveTelemetryDestination(false, layered(moduleDefault, map[string]string{"TELEMETRY_STATE": "available"})) require.ErrorContains(t, err, "OTEL_EXPORTER_OTLP_ENDPOINT is not set", "the module default's reason is not an endpoint") - _, err = resolveTelemetryDestination(layered( + _, err = resolveTelemetryDestination(false, layered( map[string]string{"TELEMETRY_STATE": "available", "OTEL_EXPORTER_OTLP_ENDPOINT": collector}, map[string]string{"TELEMETRY_STATE": "absent"}, )) @@ -304,7 +352,7 @@ func TestTelemetryDestinationTheCellOverridesTheModuleDefaults(t *testing.T) { // A refusal quotes no part of an endpoint, which may carry credentials. func TestTelemetryDestinationRefusalDoesNotEchoCredentials(t *testing.T) { - _, err := resolveTelemetryDestination(group(map[string]string{ + _, err := resolveTelemetryDestination(false, group(map[string]string{ "TELEMETRY_STATE": "available", "OTEL_EXPORTER_OTLP_ENDPOINT": "http://user:hunter2@collector.example:4317", })) @@ -322,20 +370,24 @@ func TestConfiguredTelemetryDestinationReadsTheObservabilityGroup(t *testing.T) t.Setenv(prefix+key, "") } - _, err := configuredTelemetryDestination() + localDestination, err := configuredTelemetryDestination(true) + require.NoError(t, err) + require.False(t, localDestination.Available()) + require.NotEmpty(t, localDestination.AbsentReason) + _, err = configuredTelemetryDestination(false) require.ErrorContains(t, err, "TELEMETRY_STATE is not set") // Ambient variables a shell or a pod may carry are not the group. t.Setenv("TELEMETRY_STATE", "available") t.Setenv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://ambient.example:4318") - _, err = configuredTelemetryDestination() + _, err = configuredTelemetryDestination(false) require.ErrorContains(t, err, "TELEMETRY_STATE is not set") t.Setenv("TELEMETRY_STATE", "") t.Setenv("OTEL_EXPORTER_OTLP_ENDPOINT", "") t.Setenv(prefix+"TELEMETRY_STATE", "available") t.Setenv(prefix+"OTEL_EXPORTER_OTLP_ENDPOINT", collector) - destination, err := configuredTelemetryDestination() + destination, err := configuredTelemetryDestination(false) require.NoError(t, err) require.Equal(t, "otel-collector.otel-collector.svc.cluster.local:4317", destination.Endpoint) require.True(t, destination.Insecure) @@ -343,7 +395,7 @@ func TestConfiguredTelemetryDestinationReadsTheObservabilityGroup(t *testing.T) t.Setenv(prefix+"TELEMETRY_STATE", "absent") t.Setenv(prefix+"OTEL_EXPORTER_OTLP_ENDPOINT", "") t.Setenv(prefix+"TELEMETRY_ABSENT_REASON", "a local run has no cell collector") - destination, err = configuredTelemetryDestination() + destination, err = configuredTelemetryDestination(false) require.NoError(t, err) require.False(t, destination.Available()) require.Equal(t, "a local run has no cell collector", destination.AbsentReason) @@ -358,7 +410,7 @@ func TestConfiguredTelemetryDestinationStateDecidesAcrossLayers(t *testing.T) { t.Setenv(prefix+"OTEL_EXPORTER_OTLP_ENDPOINT", collector) t.Setenv(prefix+"TELEMETRY_ABSENT_REASON", "A local run has no cell collector.") - destination, err := configuredTelemetryDestination() + destination, err := configuredTelemetryDestination(false) require.NoError(t, err) require.True(t, destination.Available()) require.Equal(t, collector, destination.URL) @@ -366,8 +418,8 @@ func TestConfiguredTelemetryDestinationStateDecidesAcrossLayers(t *testing.T) { } // Configuration that cannot work is refused before the process acquires -// anything, including in a local run: a group that did not arrive is not a cell -// without a collector anywhere. +// anything: a group that did not arrive is not a cell without a collector. +// Only a local runtime can infer an absent collector from the missing state. func TestStartupRefusesAnObservabilityGroupThatDidNotArrive(t *testing.T) { const prefix = "CODEFLY__WORKSPACE_CONFIGURATION__OBSERVABILITY__" for _, key := range []string{"TELEMETRY_STATE", "OTEL_EXPORTER_OTLP_ENDPOINT", "TELEMETRY_ABSENT_REASON"} { @@ -376,9 +428,8 @@ func TestStartupRefusesAnObservabilityGroupThatDidNotArrive(t *testing.T) { // The key-custody requirement answers first outside a local run; satisfy it // so the observability requirement is what is under test. t.Setenv("CODEFLY__WORKSPACE_CONFIGURATION__VAULT__VAULT_KEY_CUSTODY", "seed-signing-key") - for _, local := range []bool{true, false} { - require.ErrorContains(t, requireStartupConfiguration(local), "TELEMETRY_STATE is not set", "local=%v", local) - } + require.NoError(t, requireStartupConfiguration(true)) + require.ErrorContains(t, requireStartupConfiguration(false), "TELEMETRY_STATE is not set") t.Setenv(prefix+"TELEMETRY_STATE", "absent") t.Setenv(prefix+"TELEMETRY_ABSENT_REASON", "a local run has no cell collector") diff --git a/module/services/accounts/code/telemetry_metrics_test.go b/module/services/accounts/code/telemetry_metrics_test.go index 7af83c684..4ae75bfd9 100644 --- a/module/services/accounts/code/telemetry_metrics_test.go +++ b/module/services/accounts/code/telemetry_metrics_test.go @@ -116,7 +116,7 @@ func TestTelemetryMetricsExportRuntimeAndUnsampledRED(t *testing.T) { // it really receives rather than a hand-built struct. func availableAt(t *testing.T, scheme, hostPort string) telemetryDestination { t.Helper() - destination, err := resolveTelemetryDestination(group(map[string]string{ + destination, err := resolveTelemetryDestination(false, group(map[string]string{ "TELEMETRY_STATE": "available", "OTEL_EXPORTER_OTLP_ENDPOINT": scheme + "://" + hostPort, })) diff --git a/module/services/accounts/code/work.go b/module/services/accounts/code/work.go index 34faee487..7429704bb 100644 --- a/module/services/accounts/code/work.go +++ b/module/services/accounts/code/work.go @@ -86,7 +86,7 @@ func doWork(ctx context.Context) (Clean, error) { // delivers in the `observability` configuration group — or the group says the // cell has none. Either answer was already required before anything was // acquired (requireStartupConfiguration); this reads it again to act on it. - telemetryDestination, err := configuredTelemetryDestination() + telemetryDestination, err := configuredTelemetryDestination(codefly.IsLocal()) if err != nil { return nil, err } @@ -2227,17 +2227,17 @@ func requireStartupConfiguration(isLocal bool) error { if err := requireKeyCustody(isLocal); err != nil { return err } - return requireTelemetryConfiguration() + return requireTelemetryConfiguration(isLocal) } // requireTelemetryConfiguration refuses to start unless the `observability` // group says where the cell's collector is, or that the cell has none. It only // checks: doWork reads the same answer again to act on it. A group that did not -// arrive is refused here, before the database or Vault is touched, because -// reading it as "no collector" would leave a cell exporting nothing while it -// converges green. -func requireTelemetryConfiguration() error { - _, err := configuredTelemetryDestination() +// arrive is refused outside the local runtime, before the database or Vault is +// touched: reading it as "no collector" would leave a cell exporting nothing +// while it converges green. +func requireTelemetryConfiguration(isLocal bool) error { + _, err := configuredTelemetryDestination(isLocal) return err } diff --git a/module/services/auth-gateway/code/main.go b/module/services/auth-gateway/code/main.go index 77e0471c7..27b01dddd 100644 --- a/module/services/auth-gateway/code/main.go +++ b/module/services/auth-gateway/code/main.go @@ -35,8 +35,8 @@ func main() { // The cell's collector is delivered in the `observability` configuration // group, or the group says the cell has none. A group that did not arrive is - // neither, so it is refused before anything else starts. - telemetryDestination, err := configuredTelemetryDestination() + // neither, so it is refused outside a local runtime before anything else starts. + telemetryDestination, err := configuredTelemetryDestination(codefly.IsLocal()) if err != nil { panic(err) } diff --git a/module/services/auth-gateway/code/telemetry_destination.go b/module/services/auth-gateway/code/telemetry_destination.go index c752d82fd..cecf41049 100644 --- a/module/services/auth-gateway/code/telemetry_destination.go +++ b/module/services/auth-gateway/code/telemetry_destination.go @@ -42,7 +42,8 @@ import ( // A group that did not arrive is not a cell without a backend. Reading the first // as the second would leave a deployment that converges green and exports // nothing, so a missing or unknown state, or the key the state requires being -// absent, refuses to start, naming what is missing. +// absent, refuses to start, naming what is missing. Only a local runtime may +// infer `absent` when the state is missing; module defaults never declare it. // // This file exists byte for byte in services/accounts/code and // services/auth-gateway/code: the two services are independent Go modules with @@ -124,17 +125,23 @@ func observabilityValue(key string) string { // configuredTelemetryDestination resolves the destination from the // `observability` group. -func configuredTelemetryDestination() (telemetryDestination, error) { - return resolveTelemetryDestination(observabilityValue) +func configuredTelemetryDestination(local bool) (telemetryDestination, error) { + return resolveTelemetryDestination(local, observabilityValue) } // resolveTelemetryDestination is the whole decision, over any reader of the // group's keys so a test can reach every outcome with nothing running. -func resolveTelemetryDestination(read func(key string) string) (telemetryDestination, error) { +func resolveTelemetryDestination(local bool, read func(key string) string) (telemetryDestination, error) { state := strings.TrimSpace(read(telemetryStateKey)) endpoint := strings.TrimSpace(read(telemetryEndpointKey)) reason := boundedLine(read(telemetryAbsentReasonKey), maxAbsentReasonLength) + if state == "" && local { + state = telemetryStateAbsent + if reason == "" { + reason = "A local run has no cell collector; traces go to the stdout tracer and metrics are not exported." + } + } if state == "" { return telemetryDestination{}, errors.New( "observability: " + telemetryStateKey + " is not set in the `observability` configuration group. " + diff --git a/module/services/auth-gateway/code/telemetry_destination_test.go b/module/services/auth-gateway/code/telemetry_destination_test.go index fe4f32e4a..cb6409b75 100644 --- a/module/services/auth-gateway/code/telemetry_destination_test.go +++ b/module/services/auth-gateway/code/telemetry_destination_test.go @@ -3,6 +3,7 @@ package main import ( "context" "net" + "os" "strings" "testing" "time" @@ -21,9 +22,56 @@ func group(values map[string]string) func(string) string { const collector = "http://otel-collector.otel-collector.svc.cluster.local:4317" +// Module defaults are also inherited by cells. They must not supply the state +// that only the platform can declare for a deployed workload. +func TestTelemetryDestinationShippedDefaultsCannotMaskMissingCellState(t *testing.T) { + contents, err := os.ReadFile("../../../configurations/local/observability.env") + require.NoError(t, err) + values := map[string]string{} + for _, line := range strings.Split(string(contents), "\n") { + line = strings.TrimSpace(line) + if line == "" || strings.HasPrefix(line, "#") { + continue + } + key, value, ok := strings.Cut(line, "=") + require.True(t, ok) + values[key] = value + } + _, err = resolveTelemetryDestination(false, group(values)) + require.ErrorContains(t, err, "TELEMETRY_STATE is not set") + destination, err := resolveTelemetryDestination(true, group(values)) + require.NoError(t, err) + require.False(t, destination.Available()) + require.NotEmpty(t, destination.AbsentReason) +} + +func TestTelemetryDestinationMissingStateDefaultsOnlyInLocalRuntime(t *testing.T) { + for _, values := range []map[string]string{ + {}, + {"TELEMETRY_STATE": " "}, + {"TELEMETRY_ABSENT_REASON": "local defaults inherited by a cell"}, + {"OTEL_EXPORTER_OTLP_ENDPOINT": collector}, + } { + _, err := resolveTelemetryDestination(false, group(values)) + require.ErrorContains(t, err, "TELEMETRY_STATE is not set") + destination, err := resolveTelemetryDestination(true, group(values)) + require.NoError(t, err) + require.False(t, destination.Available()) + require.NotEmpty(t, destination.AbsentReason) + } + for _, values := range []map[string]string{ + {"TELEMETRY_STATE": "unknown"}, + {"TELEMETRY_STATE": "available"}, + {"TELEMETRY_STATE": "absent"}, + } { + _, err := resolveTelemetryDestination(true, group(values)) + require.Error(t, err, "an explicit state must be valid and complete even locally") + } +} + // The first outcome: the platform delivered a collector. func TestTelemetryDestinationAvailableExportsToTheEndpoint(t *testing.T) { - destination, err := resolveTelemetryDestination(group(map[string]string{ + destination, err := resolveTelemetryDestination(false, group(map[string]string{ "TELEMETRY_STATE": "available", "OTEL_EXPORTER_OTLP_ENDPOINT": collector, })) @@ -56,7 +104,7 @@ func TestTelemetryDestinationReadsTransportFromTheScheme(t *testing.T) { "https://10.0.0.7:4317": {"https://10.0.0.7:4317", "10.0.0.7:4317", false}, "http://otel-collector.ns.svc:4317/": {"http://otel-collector.ns.svc:4317", "otel-collector.ns.svc:4317", true}, } { - destination, err := resolveTelemetryDestination(group(map[string]string{ + destination, err := resolveTelemetryDestination(false, group(map[string]string{ "TELEMETRY_STATE": "available", "OTEL_EXPORTER_OTLP_ENDPOINT": endpoint, })) @@ -70,7 +118,7 @@ func TestTelemetryDestinationReadsTransportFromTheScheme(t *testing.T) { // The second outcome: the cell has no collector, and says why. The process boots // and exports nothing. func TestTelemetryDestinationAbsentCarriesTheReason(t *testing.T) { - destination, err := resolveTelemetryDestination(group(map[string]string{ + destination, err := resolveTelemetryDestination(false, group(map[string]string{ "TELEMETRY_STATE": "absent", "TELEMETRY_ABSENT_REASON": "This cell's bucket has no ops cell, so no collector is rendered.", })) @@ -85,7 +133,7 @@ func TestTelemetryDestinationAbsentCarriesTheReason(t *testing.T) { // The reason is operator-written free text that reaches a log line. func TestTelemetryDestinationAbsentReasonIsOneBoundedLine(t *testing.T) { - destination, err := resolveTelemetryDestination(group(map[string]string{ + destination, err := resolveTelemetryDestination(false, group(map[string]string{ "TELEMETRY_STATE": "absent", "TELEMETRY_ABSENT_REASON": "no collector\n2026-01-01 FORGED entry\r\n" + strings.Repeat("x", 1000), })) @@ -170,7 +218,7 @@ func TestTelemetryDestinationRefusesEverythingElse(t *testing.T) { }, } { t.Run(name, func(t *testing.T) { - destination, err := resolveTelemetryDestination(group(tc.values)) + destination, err := resolveTelemetryDestination(false, group(tc.values)) require.Error(t, err) require.ErrorContains(t, err, tc.want) require.False(t, destination.Available(), "a refusal must never read as a usable destination") @@ -184,7 +232,7 @@ func TestTelemetryDestinationRefusesEverythingElse(t *testing.T) { // cell beside the reason the local profile left behind. That is not a // contradiction to refuse: the reason is ignored, and said so once. func TestTelemetryDestinationAvailableIgnoresALeftoverReason(t *testing.T) { - destination, err := resolveTelemetryDestination(group(map[string]string{ + destination, err := resolveTelemetryDestination(false, group(map[string]string{ "TELEMETRY_STATE": "available", "OTEL_EXPORTER_OTLP_ENDPOINT": collector, "TELEMETRY_ABSENT_REASON": "A local run has no cell collector.", @@ -210,7 +258,7 @@ func TestTelemetryDestinationAbsentIgnoresALeftoverEndpoint(t *testing.T) { "an unreadable endpoint": "http://[::1", } { t.Run(name, func(t *testing.T) { - destination, err := resolveTelemetryDestination(group(map[string]string{ + destination, err := resolveTelemetryDestination(false, group(map[string]string{ "TELEMETRY_STATE": "absent", "TELEMETRY_ABSENT_REASON": "This cell has no ops cell.", "OTEL_EXPORTER_OTLP_ENDPOINT": leftover, @@ -230,13 +278,13 @@ func TestTelemetryDestinationAbsentIgnoresALeftoverEndpoint(t *testing.T) { // A blank leftover is nothing left over. func TestTelemetryDestinationBlankLeftoversAreNotNoticed(t *testing.T) { - available, err := resolveTelemetryDestination(group(map[string]string{ + available, err := resolveTelemetryDestination(false, group(map[string]string{ "TELEMETRY_STATE": "available", "OTEL_EXPORTER_OTLP_ENDPOINT": collector, "TELEMETRY_ABSENT_REASON": " \n ", })) require.NoError(t, err) require.Empty(t, available.IgnoredNotice()) - absent, err := resolveTelemetryDestination(group(map[string]string{ + absent, err := resolveTelemetryDestination(false, group(map[string]string{ "TELEMETRY_STATE": "absent", "TELEMETRY_ABSENT_REASON": "no collector", "OTEL_EXPORTER_OTLP_ENDPOINT": " ", })) require.NoError(t, err) @@ -266,7 +314,7 @@ func TestTelemetryDestinationTheCellOverridesTheModuleDefaults(t *testing.T) { } t.Run("a cell with a collector overrides the local absent default", func(t *testing.T) { - destination, err := resolveTelemetryDestination(layered(moduleDefault, map[string]string{ + destination, err := resolveTelemetryDestination(false, layered(moduleDefault, map[string]string{ "TELEMETRY_STATE": "available", "OTEL_EXPORTER_OTLP_ENDPOINT": collector, })) @@ -278,7 +326,7 @@ func TestTelemetryDestinationTheCellOverridesTheModuleDefaults(t *testing.T) { }) t.Run("a cell with no collector, over a default that had an endpoint, is absent", func(t *testing.T) { - destination, err := resolveTelemetryDestination(layered( + destination, err := resolveTelemetryDestination(false, layered( map[string]string{"TELEMETRY_STATE": "available", "OTEL_EXPORTER_OTLP_ENDPOINT": collector}, map[string]string{"TELEMETRY_STATE": "absent", "TELEMETRY_ABSENT_REASON": "This cell has no ops cell."}, )) @@ -289,11 +337,11 @@ func TestTelemetryDestinationTheCellOverridesTheModuleDefaults(t *testing.T) { }) t.Run("a cell that overrides only the state still has to supply what it requires", func(t *testing.T) { - _, err := resolveTelemetryDestination(layered(moduleDefault, map[string]string{"TELEMETRY_STATE": "available"})) + _, err := resolveTelemetryDestination(false, layered(moduleDefault, map[string]string{"TELEMETRY_STATE": "available"})) require.ErrorContains(t, err, "OTEL_EXPORTER_OTLP_ENDPOINT is not set", "the module default's reason is not an endpoint") - _, err = resolveTelemetryDestination(layered( + _, err = resolveTelemetryDestination(false, layered( map[string]string{"TELEMETRY_STATE": "available", "OTEL_EXPORTER_OTLP_ENDPOINT": collector}, map[string]string{"TELEMETRY_STATE": "absent"}, )) @@ -304,7 +352,7 @@ func TestTelemetryDestinationTheCellOverridesTheModuleDefaults(t *testing.T) { // A refusal quotes no part of an endpoint, which may carry credentials. func TestTelemetryDestinationRefusalDoesNotEchoCredentials(t *testing.T) { - _, err := resolveTelemetryDestination(group(map[string]string{ + _, err := resolveTelemetryDestination(false, group(map[string]string{ "TELEMETRY_STATE": "available", "OTEL_EXPORTER_OTLP_ENDPOINT": "http://user:hunter2@collector.example:4317", })) @@ -322,20 +370,24 @@ func TestConfiguredTelemetryDestinationReadsTheObservabilityGroup(t *testing.T) t.Setenv(prefix+key, "") } - _, err := configuredTelemetryDestination() + localDestination, err := configuredTelemetryDestination(true) + require.NoError(t, err) + require.False(t, localDestination.Available()) + require.NotEmpty(t, localDestination.AbsentReason) + _, err = configuredTelemetryDestination(false) require.ErrorContains(t, err, "TELEMETRY_STATE is not set") // Ambient variables a shell or a pod may carry are not the group. t.Setenv("TELEMETRY_STATE", "available") t.Setenv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://ambient.example:4318") - _, err = configuredTelemetryDestination() + _, err = configuredTelemetryDestination(false) require.ErrorContains(t, err, "TELEMETRY_STATE is not set") t.Setenv("TELEMETRY_STATE", "") t.Setenv("OTEL_EXPORTER_OTLP_ENDPOINT", "") t.Setenv(prefix+"TELEMETRY_STATE", "available") t.Setenv(prefix+"OTEL_EXPORTER_OTLP_ENDPOINT", collector) - destination, err := configuredTelemetryDestination() + destination, err := configuredTelemetryDestination(false) require.NoError(t, err) require.Equal(t, "otel-collector.otel-collector.svc.cluster.local:4317", destination.Endpoint) require.True(t, destination.Insecure) @@ -343,7 +395,7 @@ func TestConfiguredTelemetryDestinationReadsTheObservabilityGroup(t *testing.T) t.Setenv(prefix+"TELEMETRY_STATE", "absent") t.Setenv(prefix+"OTEL_EXPORTER_OTLP_ENDPOINT", "") t.Setenv(prefix+"TELEMETRY_ABSENT_REASON", "a local run has no cell collector") - destination, err = configuredTelemetryDestination() + destination, err = configuredTelemetryDestination(false) require.NoError(t, err) require.False(t, destination.Available()) require.Equal(t, "a local run has no cell collector", destination.AbsentReason) @@ -358,7 +410,7 @@ func TestConfiguredTelemetryDestinationStateDecidesAcrossLayers(t *testing.T) { t.Setenv(prefix+"OTEL_EXPORTER_OTLP_ENDPOINT", collector) t.Setenv(prefix+"TELEMETRY_ABSENT_REASON", "A local run has no cell collector.") - destination, err := configuredTelemetryDestination() + destination, err := configuredTelemetryDestination(false) require.NoError(t, err) require.True(t, destination.Available()) require.Equal(t, collector, destination.URL) diff --git a/module/services/auth-gateway/code/telemetry_metrics_test.go b/module/services/auth-gateway/code/telemetry_metrics_test.go index 692e9eb8c..3f66755eb 100644 --- a/module/services/auth-gateway/code/telemetry_metrics_test.go +++ b/module/services/auth-gateway/code/telemetry_metrics_test.go @@ -139,7 +139,7 @@ func authFloat64SumValue(request *collectormetricsv1.ExportMetricsServiceRequest // it really receives rather than a hand-built struct. func availableAt(t *testing.T, scheme, hostPort string) telemetryDestination { t.Helper() - destination, err := resolveTelemetryDestination(group(map[string]string{ + destination, err := resolveTelemetryDestination(false, group(map[string]string{ "TELEMETRY_STATE": "available", "OTEL_EXPORTER_OTLP_ENDPOINT": scheme + "://" + hostPort, })) From fa62f2838f6046a157c06cbdc92684cf8acbae98 Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Wed, 7 Oct 2026 14:22:12 -0400 Subject: [PATCH 57/69] fix(audit): bound history reads and validate writable schemas --- module/AUDIT_OPERATIONS.md | 21 +- module/deployment/README.md | 32 ++- module/deployment/jobs.codefly.yaml | 10 +- module/docs/adr/0009-audit-store-swap.md | 22 +- .../auditstore/bigquerystore/bigquerystore.go | 9 + .../bigquerystore/ensure_schema_test.go | 60 ++++++ .../bigquerystore/history_bound_test.go | 65 ++++++ .../pkg/auditstore/bigquerystore/reader.go | 48 ++++- .../clickhousestore/history_bound_test.go | 111 ++++++++++ .../pkg/auditstore/clickhousestore/reader.go | 63 ++++-- .../code/pkg/business/audit_history.go | 201 +++++++++++++++--- .../pkg/business/audit_history_bound_test.go | 135 ++++++++++++ .../code/pkg/business/audit_history_test.go | 6 +- .../code/pkg/infra/postgres_audit_history.go | 143 ++++++++++++- .../postgres_audit_history_bound_test.go | 23 ++ .../testdata/control_plane_call_sites.txt | 1 + module/services/accounts/service.codefly.yaml | 5 + 17 files changed, 877 insertions(+), 78 deletions(-) create mode 100644 module/services/accounts/code/pkg/auditstore/bigquerystore/ensure_schema_test.go create mode 100644 module/services/accounts/code/pkg/auditstore/bigquerystore/history_bound_test.go create mode 100644 module/services/accounts/code/pkg/auditstore/clickhousestore/history_bound_test.go create mode 100644 module/services/accounts/code/pkg/business/audit_history_bound_test.go create mode 100644 module/services/accounts/code/pkg/infra/postgres_audit_history_bound_test.go diff --git a/module/AUDIT_OPERATIONS.md b/module/AUDIT_OPERATIONS.md index 0d36ef808..b2735bc27 100644 --- a/module/AUDIT_OPERATIONS.md +++ b/module/AUDIT_OPERATIONS.md @@ -29,7 +29,7 @@ must be a **completed month in UTC**: the current month and later ones are refus as a usage error. Without it the run covers every partition, which can verify but cannot drop. `-verify-only` writes nothing. -The run is resumable. For each day-long window of each partition it reads the +The run is resumable. It starts with day-long windows of each partition and reads the source rows, reads back what the warehouse holds for that window, appends only the events whose stored copies are not complete, reads the window back again, and then verifies every source row against every stored copy of its id. Existing copies are @@ -37,6 +37,21 @@ therefore verified **after** the append, in the same pass; an earlier run's copi are not verified before new rows are written. A verification failure leaves the partitions in place. +Busy windows are retried as halves before any writes. Source records and their +verification state have a 32 MiB budget (`AuditHistoryCopyConfig.WindowBytes`); +the source streams one row at a time, counts decoded payload containers as well +as canonical details, and retains the canonical text alone. Warehouse history +reads independently bound their retained details at at most 32 MiB, deduplicate +byte-identical detail copies, and stream every event copy through verification +without retaining all duplicates. These bounds cover retained data; driver +response blocks and the current row also consume memory. `-batch-size` controls +write requests, not these byte budgets. A single timestamp whose events cannot +fit is refused, leaves the run unverified, and removes nothing. If read-back +grows beyond its budget after an append, the run stops; rerunning resumes from +the now-existing copies without repeating repairs during the failed run. +Splits run oldest first, so copy, verification and removal hash the same ordered +source stream regardless of window sizes. + An event is appended when the warehouse holds none of it, and again when it holds it without the details of a content-class event. Both adapters write an event's events row before its details row, so a run whose details write failed leaves @@ -111,9 +126,9 @@ and never `verified`. The copy and its read-back match events by `id` within a window, while the source key is `(id, created_at)`. Two source rows may in principle share an id at different -instants. When both fall in the same UTC-day window, each is compared with both +instants. When both fall in the same read window, each is compared with both stored copies, one differs from it, and verification fails closed with "stored -envelope differs from the row"; nothing is dropped. When they fall on different days +envelope differs from the row"; nothing is dropped. When they fall in different windows each copy matches its own row and verification passes, leaving two warehouse rows with one id, as the source holds. Deployments that generate ids as UUIDs do not produce either. diff --git a/module/deployment/README.md b/module/deployment/README.md index cff92c1e7..6d2872240 100644 --- a/module/deployment/README.md +++ b/module/deployment/README.md @@ -67,8 +67,18 @@ transport configuration). ## Deploy Jobs -A `deploy_jobs` topology entry is a store-writing one-shot the driver runs as a -deploy step. Unlike a service's self-serving `bootstrap_job_endpoints` Job — +A `deploy_jobs` topology entry is a required store-writing one-shot for a +job-capable promotion driver. **Automatic catalog promotion remains blocked:** +the Codefly CLI currently has no deploy-job execution or dependency migration +barrier. It must reject this bundle rather than silently omit the import. Service +environment defaults alone do not provide job execution. Driver support must +mount the catalog, use the running service's immutable image and identity, +resolve the write credential (including managed IAM), inherit the effective +service environment, and fail promotion unless each `after` dependency's +migration and the import complete. Sync-wave annotations on independently +reconciling Argo Applications do not enforce that dependency barrier. + +Unlike a service's self-serving `bootstrap_job_endpoints` Job — which reaches only the service's own endpoints — a deploy Job runs one service's image but `writes` to a dependency it declares, consuming a generated artifact. The module ships `role-catalog-import`: it runs the `accounts` image but writes @@ -77,15 +87,21 @@ migration. Each bundle `deployJobs` entry resolves the `catalog` artifact path, the target `service`/`endpoint`/`port`, the `force` flag, the ordering, and `serviceEnvironment` — the running service's environment variables the Job is given with that service's own values in the same environment; the -driver mounts the catalog, connects the target, sets those variables, runs -`command`, and fails the promotion if it exits non-zero. `role-catalog-import` +job-capable driver mounts the catalog, connects the target, sets those variables, +runs `command`, and fails the promotion if it exits non-zero. `role-catalog-import` inherits `AUDIT_SINK`: it records its audit events through the same emitter as `accounts`, so under a swap value (ADR 0009) they reach the queue the accounts relay delivers instead of `audit_events`. Unlike `accounts`, the importer refuses -an unset `AUDIT_SINK` rather than default to `postgres`, so a deployment that -relies on that default must set `AUDIT_SINK=postgres` explicitly. Re-running an unchanged catalog is an empty -no-op, so the step is idempotent. Generation rejects a deploy Job whose catalog -artifact is absent, whose write target is not a declared dependency of the +an unset `AUDIT_SINK` rather than default to `postgres`. The accounts manifest's +CLI-owned `spec.environment-defaults` declares `AUDIT_SINK: postgres`, so the +service has an explicit value to inherit after deployment projection. The +environment's `service-config` or explicit secret reference overrides that +default; a warehouse deployment's Job must inherit its resolved warehouse mode. +This requires a CLI release that projects service environment defaults and, +for automatic promotion, implements the job contract above. Re-running an +unchanged catalog is an empty no-op, so the step is idempotent. Generation rejects +a deploy Job whose catalog artifact is absent, whose write target is not a +declared dependency of the running service, that references an undeclared service, or that writes to a migration-bearing target (`bootstrap_job_endpoints`) without ordering `after` it — an import that races the migration would write against a schema that does not diff --git a/module/deployment/jobs.codefly.yaml b/module/deployment/jobs.codefly.yaml index b0e6a3da0..9abd761eb 100644 --- a/module/deployment/jobs.codefly.yaml +++ b/module/deployment/jobs.codefly.yaml @@ -18,10 +18,12 @@ deploy_jobs: # writes to. The importer reads it through the same auditsink package as the # service but, unlike the service, has no default: if the Job does not receive # the variable it exits before touching the store, rather than guess postgres - # and write its events where a warehouse deployment never reads them. A - # deployment that leaves AUDIT_SINK unset on the service (the default, - # postgres) must set AUDIT_SINK=postgres explicitly so that the step has a - # value to inherit. + # and write its events where a warehouse deployment never reads them. The + # accounts service manifest declares the raw default AUDIT_SINK=postgres; + # an explicit environment service-config value replaces it for both service + # and Job. A driver must resolve service defaults before inheriting them. + # Automatic promotion requires deploy-job execution and a migration barrier; + # a driver without that support must refuse this required step. - name: role-catalog-import service: accounts command: role-catalog-import diff --git a/module/docs/adr/0009-audit-store-swap.md b/module/docs/adr/0009-audit-store-swap.md index 8fa23b21a..15a363bbb 100644 --- a/module/docs/adr/0009-audit-store-swap.md +++ b/module/docs/adr/0009-audit-store-swap.md @@ -235,8 +235,9 @@ Postgres transaction. 16 bytes), so the service never receives what it could not keep. A time range, an actor or an event type narrows it, as does grouping by fewer dimensions. - - The history copy (item 7) reads and holds a whole window of the store at a - time outside this budget. + - The history copy (item 7) applies its own 32 MiB source budget and bounded + warehouse details joins, splitting busy windows before writes. It streams + every stored event copy through verification instead of retaining copies. The reason is the grants. The service that reads is the service that appends, so it holds `bigquery.tables.updateData`. With @@ -282,7 +283,10 @@ Postgres transaction. expiration, and refuses at startup an events table whose partitions expire, since its identity holds no `bigquery.tables.update` to change one. Both tables must use daily partitions and have no whole-table expiration, - including expiration inherited from the dataset when they are created; + including expiration inherited from the dataset when they are created. + Startup also rejects `REQUIRED` columns where the writer can supply NULL, + and extra `REQUIRED` columns without defaults. Nullable extensions and + omitted extra columns with defaults remain compatible; - a **details** table holding the full details of content-class types, kept for the shorter content window, `AUDIT_CONTENT_RETENTION_DAYS` — a partition expiration on BigQuery, a table TTL on ClickHouse. @@ -357,9 +361,15 @@ Postgres transaction. verified partitions, locks them, re-counts each against the count that was verified, and refuses on any mismatch — a partition that gained a row since verification is never dropped. `-through`, the cutoff of the copy, accepts - completed months (UTC) only. The copy and its read-back hold one UTC day of - the deployment's events in memory at a time, outside the read budget of - item 4. From then on `audit_events` receives no new rows, and Postgres holds + completed months (UTC) only. The copy starts with UTC-day windows, halves + them when source records or warehouse details would exceed their byte + budgets, and refuses an instant that cannot fit. Source records retain + canonical text alone after counting the current decoded payload; read-back + retains bounded unique details and verifies stored event copies as they + stream in. Response blocks and one current row also consume memory. Windows + are visited oldest first, so splitting never changes the source digest. + A post-append overflow ends the run safely and is resumed by the next run. + From then on `audit_events` receives no new rows, and Postgres holds only the queue. Missing content details get one repair attempt per event per run, even after several partial writes: duplicate events rows are not evidence of permanent diff --git a/module/services/accounts/code/pkg/auditstore/bigquerystore/bigquerystore.go b/module/services/accounts/code/pkg/auditstore/bigquerystore/bigquerystore.go index 59a8de4d7..8f341f353 100644 --- a/module/services/accounts/code/pkg/auditstore/bigquerystore/bigquerystore.go +++ b/module/services/accounts/code/pkg/auditstore/bigquerystore/bigquerystore.go @@ -324,6 +324,15 @@ func conforms(spec tableSpec, existing *bigquery.TableMetadata) error { if got.Type != want.Type || got.Repeated { return fmt.Errorf("column %s is %s, want %s", want.Name, got.Type, want.Type) } + if got.Required && !want.Required { + return fmt.Errorf("column %s is REQUIRED but the writer can supply NULL", want.Name) + } + delete(columns, want.Name) + } + for _, field := range existing.Schema { + if _, extra := columns[field.Name]; extra && field.Required && strings.TrimSpace(field.DefaultValueExpression) == "" { + return fmt.Errorf("extra REQUIRED column %s has no default and is not supplied by the writer", field.Name) + } } return nil } diff --git a/module/services/accounts/code/pkg/auditstore/bigquerystore/ensure_schema_test.go b/module/services/accounts/code/pkg/auditstore/bigquerystore/ensure_schema_test.go new file mode 100644 index 000000000..095823a16 --- /dev/null +++ b/module/services/accounts/code/pkg/auditstore/bigquerystore/ensure_schema_test.go @@ -0,0 +1,60 @@ +package bigquerystore + +import ( + "testing" + "time" + + "cloud.google.com/go/bigquery" + "github.com/stretchr/testify/require" +) + +func TestEnsureRejectsColumnsTheWriterCannotSupply(t *testing.T) { + store := &Store{retention: 30 * 24 * time.Hour} + for _, spec := range store.tableSpecs() { + t.Run(spec.name, func(t *testing.T) { + for _, field := range spec.schema { + if field.Required { + continue + } + t.Run(field.Name, func(t *testing.T) { + meta := metadataFor(spec) + meta.Schema = append(bigquery.Schema(nil), meta.Schema...) + for i, column := range meta.Schema { + if column.Name == field.Name { + copy := *column + copy.Required = true + meta.Schema[i] = © + break + } + } + require.ErrorContains(t, conforms(spec, meta), field.Name) + }) + } + t.Run("extra required column", func(t *testing.T) { + meta := metadataFor(spec) + meta.Schema = append(meta.Schema, &bigquery.FieldSchema{Name: "customer_tag", Type: bigquery.StringFieldType, Required: true}) + require.ErrorContains(t, conforms(spec, meta), "customer_tag") + }) + }) + } +} + +func TestEnsureAcceptsCompatibleSchemaExtensions(t *testing.T) { + store := &Store{retention: 30 * 24 * time.Hour} + for _, spec := range store.tableSpecs() { + t.Run(spec.name, func(t *testing.T) { + meta := metadataFor(spec) + meta.Schema = append(bigquery.Schema(nil), meta.Schema...) + for i, field := range meta.Schema { + copy := *field + copy.Required = false + meta.Schema[i] = © + } + meta.Schema = append(meta.Schema, + &bigquery.FieldSchema{Name: "nullable_tag", Type: bigquery.StringFieldType}, + &bigquery.FieldSchema{Name: "default_tag", Type: bigquery.StringFieldType, Required: true, DefaultValueExpression: "'audit'"}, + ) + require.NoError(t, conforms(spec, meta)) + }) + } +} diff --git a/module/services/accounts/code/pkg/auditstore/bigquerystore/history_bound_test.go b/module/services/accounts/code/pkg/auditstore/bigquerystore/history_bound_test.go new file mode 100644 index 000000000..09f2b2402 --- /dev/null +++ b/module/services/accounts/code/pkg/auditstore/bigquerystore/history_bound_test.go @@ -0,0 +1,65 @@ +package bigquerystore + +import ( + "context" + "strings" + "testing" + "time" + + "accounts/pkg/auditstore/bigqueryfake" + "accounts/pkg/business" + + "github.com/stretchr/testify/require" +) + +func TestHistoryDetailsAreBoundedBeforeRetainingTheWindow(t *testing.T) { + w := newBoundWarehouse(t) + at := boundNow.Add(-time.Hour) + w.append(t, instant(t, 8, at)...) + reader := w.reader(t, 2048, 0) + visited := 0 + err := reader.ReadStoredAuditEvents(t.Context(), at.Add(-time.Second), at.Add(time.Second), func(business.StoredAuditEvent) error { + visited++ + return nil + }) + require.ErrorIs(t, err, business.ErrAuditHistoryWindowTooLarge) + require.Zero(t, visited, "the caller can retry a smaller window before accepting any events") +} + +func TestHistoryIdenticalDetailsCopiesFitAndCorruptCopiesRemainVisible(t *testing.T) { + w := newBoundWarehouse(t) + at := boundNow.Add(-time.Hour) + record := boundEvent(t, 1, at, map[string]any{"pad": strings.Repeat("p", 500)}) + w.append(t, record) + for i := 0; i < 100; i++ { + w.insertDetails(t, record, record.Details) + } + reader := w.reader(t, 4096, 0) + read := func() business.StoredAuditEvent { + var got business.StoredAuditEvent + require.NoError(t, reader.ReadStoredAuditEvents(context.Background(), at.Add(-time.Second), at.Add(time.Second), func(event business.StoredAuditEvent) error { + got = event + return nil + })) + return got + } + require.Equal(t, record.Details, read().Details) + // The claimed hash is unchanged. Byte-identical redeliveries can be + // discarded, but a different text under that hash must remain observable. + forged := DetailRow(boundDeployment, record) + forged.Values["details"] = `{"pad":"corrupt"}` + require.NoError(t, w.fake.Insert(bigqueryfake.TablePath(boundProject, boundDataset, DetailsTable), forged.Values)) + w.insertDetails(t, record, record.Details) + err := reader.ReadStoredAuditEvents(t.Context(), at.Add(-time.Second), at.Add(time.Second), func(business.StoredAuditEvent) error { return nil }) + require.ErrorContains(t, err, "do not match their stored hash") +} + +func TestHistoryPropagatesTheCallersWindowBudgetSignal(t *testing.T) { + w := newBoundWarehouse(t) + at := boundNow.Add(-time.Hour) + w.append(t, boundEvent(t, 1, at, nil)) + err := w.reader(t, 4096, 0).ReadStoredAuditEvents(t.Context(), at.Add(-time.Second), at.Add(time.Second), func(business.StoredAuditEvent) error { + return business.ErrAuditHistoryWindowTooLarge + }) + require.ErrorIs(t, err, business.ErrAuditHistoryWindowTooLarge) +} diff --git a/module/services/accounts/code/pkg/auditstore/bigquerystore/reader.go b/module/services/accounts/code/pkg/auditstore/bigquerystore/reader.go index 25dfcc3b2..b5550bd7c 100644 --- a/module/services/accounts/code/pkg/auditstore/bigquerystore/reader.go +++ b/module/services/accounts/code/pkg/auditstore/bigquerystore/reader.go @@ -811,14 +811,18 @@ func (r *Reader) LatestSourceSyncEvents(ctx context.Context, scope business.Audi // ReadStoredAuditEvents implements business.AuditHistoryReader: one session // over each table for the window, every copy of every event of this // deployment, with the details each copy has. A content-class event's details -// come from the details table; when the copies there disagree, a copy whose -// hash does not match the event's is the one reported, so the disagreement is -// what verification sees. +// come from the details table. Only byte-identical detail copies are deduped; +// a disagreeing hash is reported, and text inconsistent with its declared hash +// fails the read. The retained +// details are bounded before they are kept; a window that exceeds the bound +// returns ErrAuditHistoryWindowTooLarge so the caller can split it. func (r *Reader) ReadStoredAuditEvents(ctx context.Context, from, to time.Time, visit func(business.StoredAuditEvent) error) error { w := window{lo: &from, hi: &to, hiExclusive: true} detailsWhere := r.scope(business.PlatformAuditScope()) w.restrict(detailsWhere) - copies := map[string][]storedDetail{} + copies := map[string]map[storedDetail]struct{}{} + b := r.newBudget() + b.limit = min(b.limit, business.DefaultAuditHistoryWindowBytes) err := r.scan(ctx, DetailsTable, detailFields, detailsWhere, func(row arrowRow) error { var detail storedDetail var eventID, deploymentID string @@ -832,7 +836,23 @@ func (r *Reader) ReadStoredAuditEvents(ctx context.Context, from, to time.Time, *into = value } if deploymentID == r.deploymentID { - copies[eventID] = append(copies[eventID], detail) + if _, identical := copies[eventID][detail]; identical { + return nil + } + if business.AuditDetailsSHA256(detail.details) != detail.sha256 { + return fmt.Errorf("bigquery audit store: history details of event %s do not match their stored hash", eventID) + } + // The nested copy set has more map overhead than an ordinary + // id/hash details join. Charge it conservatively for every copy. + if err := b.take(384 + len(eventID) + len(detail.sha256) + len(detail.details)); err != nil { + return business.ErrAuditHistoryWindowTooLarge + } + // Arrow strings refer to a decoded record batch. Clone only the + // charged strings so one small value cannot retain that whole batch. + if copies[eventID] == nil { + copies[strings.Clone(eventID)] = map[storedDetail]struct{}{} + } + copies[eventID][storedDetail{sha256: strings.Clone(detail.sha256), details: strings.Clone(detail.details)}] = struct{}{} } return nil }) @@ -859,17 +879,31 @@ func (r *Reader) ReadStoredAuditEvents(ctx context.Context, from, to time.Time, } if event.Retention != business.RetentionSecurity { event.Details, event.HasDetails = "", false - for _, detail := range copies[stored.entry.ID] { + for detail := range copies[stored.entry.ID] { event.Details, event.HasDetails = detail.details, true if detail.sha256 != stored.detailsSHA256 { break } } } - return visit(event) + return visit(cloneStoredAuditEvent(event)) }) } +// cloneStoredAuditEvent prevents a callback retaining an Arrow record batch +// through any string in its envelope or security-class details. +func cloneStoredAuditEvent(event business.StoredAuditEvent) business.StoredAuditEvent { + entry := &event.Entry + for _, text := range []*string{&event.DeploymentID, &event.DetailsSHA256, &event.Details, + &entry.ID, &entry.OrgID, &entry.ActorID, &entry.ActorType, &entry.Resource, &entry.ResourceID, + &entry.IPAddress, &entry.ImpersonatedBy, &entry.ClientID} { + *text = strings.Clone(*text) + } + entry.EventType = business.EventType(strings.Clone(string(entry.EventType))) + event.Retention = business.AuditRetentionClass(strings.Clone(string(event.Retention))) + return event +} + // scan creates one read session over table and hands every row it returns to // visit, serially. The session's streams are read in parallel; a stream broken // by a transient error is reopened at the row it reached. diff --git a/module/services/accounts/code/pkg/auditstore/clickhousestore/history_bound_test.go b/module/services/accounts/code/pkg/auditstore/clickhousestore/history_bound_test.go new file mode 100644 index 000000000..51d0f81b6 --- /dev/null +++ b/module/services/accounts/code/pkg/auditstore/clickhousestore/history_bound_test.go @@ -0,0 +1,111 @@ +package clickhousestore + +import ( + "context" + "fmt" + "reflect" + "strings" + "testing" + "time" + + "accounts/pkg/business" + + "github.com/ClickHouse/clickhouse-go/v2/lib/driver" + "github.com/stretchr/testify/require" +) + +type historyConn struct { + Conn + details, events [][]any + statements []string +} + +func (c *historyConn) Query(_ context.Context, statement string, _ ...any) (driver.Rows, error) { + c.statements = append(c.statements, statement) + rows := c.events + if strings.Contains(statement, "FROM "+DetailsTable) { + rows = c.details + } + return &historyRows{rows: rows}, nil +} + +type historyRows struct { + driver.Rows + rows [][]any + next int +} + +func (r *historyRows) Next() bool { r.next++; return r.next <= len(r.rows) } +func (r *historyRows) Scan(dest ...any) error { + row := r.rows[r.next-1] + if len(dest) != len(row) { + return fmt.Errorf("historyRows: %d destinations for %d columns", len(dest), len(row)) + } + for i, into := range dest { + reflect.ValueOf(into).Elem().Set(reflect.ValueOf(row[i])) + } + return nil +} +func (r *historyRows) Close() error { return nil } +func (r *historyRows) Err() error { return nil } + +func historyTestStore(t *testing.T, conn Conn, limit int64) *Store { + t.Helper() + cfg := validConfig() + cfg.ExportMaxBytes = limit + s, err := New(conn, cfg) + require.NoError(t, err) + return s +} + +func historyEnvelope(id, text string) []any { + return []any{"deployment-1", id, boundOrg, "actor", "user", "example.read", int64(1), "document", "doc-1", occurredAt, + "", "", false, "", string(business.RetentionContent), business.AuditDetailsSHA256(text), ""} +} + +func TestHistoryDetailsAreBoundedWithoutBuildingServerArrays(t *testing.T) { + c := &historyConn{} + for i := 0; i < 8; i++ { + text := strings.Repeat("p", 900) + c.details = append(c.details, []any{fmt.Sprint(i), business.AuditDetailsSHA256(text), text}) + } + visited := 0 + err := historyTestStore(t, c, 2048).ReadStoredAuditEvents(t.Context(), occurredAt.Add(-time.Second), occurredAt.Add(time.Second), func(business.StoredAuditEvent) error { + visited++ + return nil + }) + require.ErrorIs(t, err, business.ErrAuditHistoryWindowTooLarge) + require.Zero(t, visited) + require.Len(t, c.statements, 1) + require.NotContains(t, c.statements[0], "groupArray", "the driver must never decode an unbounded details array") +} + +func TestHistoryIdenticalDetailsCopiesFitAndCorruptCopiesRemainVisible(t *testing.T) { + text := `{"pad":"` + strings.Repeat("p", 500) + `"}` + sha := business.AuditDetailsSHA256(text) + c := &historyConn{events: [][]any{historyEnvelope("event-1", text)}} + for i := 0; i < 100; i++ { + c.details = append(c.details, []any{"event-1", sha, text}) + } + s := historyTestStore(t, c, 4096) + read := func() business.StoredAuditEvent { + var got business.StoredAuditEvent + require.NoError(t, s.ReadStoredAuditEvents(t.Context(), occurredAt.Add(-time.Second), occurredAt.Add(time.Second), func(event business.StoredAuditEvent) error { + got = event + return nil + })) + return got + } + require.Equal(t, text, read().Details) + c.details = append(c.details, []any{"event-1", sha, `{"pad":"corrupt"}`}, []any{"event-1", sha, text}) + err := s.ReadStoredAuditEvents(t.Context(), occurredAt.Add(-time.Second), occurredAt.Add(time.Second), func(business.StoredAuditEvent) error { return nil }) + require.ErrorContains(t, err, "do not match their stored hash") +} + +func TestHistoryPropagatesTheCallersWindowBudgetSignal(t *testing.T) { + c := &historyConn{events: [][]any{historyEnvelope("event-1", "{}")}} + err := historyTestStore(t, c, 4096).ReadStoredAuditEvents(t.Context(), occurredAt.Add(-time.Second), occurredAt.Add(time.Second), func(business.StoredAuditEvent) error { + return business.ErrAuditHistoryWindowTooLarge + }) + require.ErrorIs(t, err, business.ErrAuditHistoryWindowTooLarge) +} diff --git a/module/services/accounts/code/pkg/auditstore/clickhousestore/reader.go b/module/services/accounts/code/pkg/auditstore/clickhousestore/reader.go index 704585745..aa2e90e20 100644 --- a/module/services/accounts/code/pkg/auditstore/clickhousestore/reader.go +++ b/module/services/accounts/code/pkg/auditstore/clickhousestore/reader.go @@ -522,24 +522,53 @@ func (s *Store) LatestSourceSyncEvents(ctx context.Context, scope business.Audit // ReadStoredAuditEvents implements business.AuditHistoryReader: every copy of // every event of this deployment in [from, to), each with the details its // store holds. A content-class event's details come from the details table; -// when the copies there disagree, a copy whose hash does not match the -// event's is the one reported, so the disagreement is what verification sees. +// identical detail copies are deduped, while a disagreeing hash is reported and +// text inconsistent with its declared hash fails the read. Detail rows are streamed individually and charged +// before retention, never collected into an unbounded array by the server. A +// window that exceeds the bound returns ErrAuditHistoryWindowTooLarge so the +// caller can split it; envelope copies are streamed to the caller for inspection. func (s *Store) ReadStoredAuditEvents(ctx context.Context, from, to time.Time, visit func(business.StoredAuditEvent) error) error { - p := &sqlParams{} - window := func() string { + window := func(p *sqlParams) string { return "deployment_id = " + p.text(s.deploymentID) + " AND occurred_at >= " + p.instant(from) + " AND occurred_at < " + p.instant(to) } - events := window() - details := window() - statement := "SELECT e.deployment_id, e.event_id, e.org_id, e.actor_id, e.actor_type, e.event_type, e.schema_version, e.resource, " + - "e.resource_id, e.occurred_at, e.ip_address, e.impersonated_by, e.is_impersonated, e.client_id, e.retention_class, " + - "e.details_sha256, e.details, arrayMap(c -> c.1, d.copies), arrayMap(c -> c.2, d.copies) " + - "FROM (SELECT " + eventColumns + " FROM " + EventsTable + " WHERE " + events + ") AS e " + - "LEFT JOIN (SELECT event_id, groupArray((details_sha256, details)) AS copies FROM " + DetailsTable + - " WHERE " + details + " GROUP BY event_id) AS d ON e.event_id = d.event_id" + type detail struct{ sha, text string } + copies := map[string]map[detail]struct{}{} + limit := min(s.exportMaxBytes, business.DefaultAuditHistoryWindowBytes) + var held int64 + p := &sqlParams{} + statement := "SELECT event_id, details_sha256, details FROM " + DetailsTable + " WHERE " + window(p) + err := s.scan(ctx, statement, p.args, func(rows driver.Rows) (bool, error) { + var id string + var copy detail + if err := rows.Scan(&id, ©.sha, ©.text); err != nil { + return false, fmt.Errorf("clickhouse audit store: read %s: %w", DetailsTable, err) + } + if _, identical := copies[id][copy]; identical { + return true, nil + } + if business.AuditDetailsSHA256(copy.text) != copy.sha { + return false, fmt.Errorf("clickhouse audit store: history details of event %s do not match their stored hash", id) + } + // Conservatively includes the nested map and its key for each unique copy. The + // strings returned by this native driver own their decoded text. + cost := int64(384 + len(id) + len(copy.sha) + len(copy.text)) + if cost > limit-held { + return false, business.ErrAuditHistoryWindowTooLarge + } + held += cost + if copies[id] == nil { + copies[id] = map[detail]struct{}{} + } + copies[id][copy] = struct{}{} + return true, nil + }) + if err != nil { + return err + } + p = &sqlParams{} + statement = "SELECT " + eventColumns + " FROM " + EventsTable + " WHERE " + window(p) return s.scan(ctx, statement, p.args, func(rows driver.Rows) (bool, error) { - var hashes, texts []string - row, err := scanStored(rows, &hashes, &texts) + row, err := scanStored(rows) if err != nil { return false, fmt.Errorf("clickhouse audit store: read %s: %w", EventsTable, err) } @@ -556,9 +585,9 @@ func (s *Store) ReadStoredAuditEvents(ctx context.Context, from, to time.Time, v } if event.Retention != business.RetentionSecurity { event.Details, event.HasDetails = "", false - for i, text := range texts { - event.Details, event.HasDetails = text, true - if hashes[i] != row.detailsSHA256 { + for copy := range copies[row.entry.ID] { + event.Details, event.HasDetails = copy.text, true + if copy.sha != row.detailsSHA256 { break } } diff --git a/module/services/accounts/code/pkg/business/audit_history.go b/module/services/accounts/code/pkg/business/audit_history.go index 10effbe9a..11234f34b 100644 --- a/module/services/accounts/code/pkg/business/audit_history.go +++ b/module/services/accounts/code/pkg/business/audit_history.go @@ -38,6 +38,22 @@ type AuditHistoryReader interface { ReadStoredAuditEvents(ctx context.Context, from, to time.Time, visit func(StoredAuditEvent) error) error } +// ErrAuditHistoryWindowTooLarge asks the copier to retry a smaller time span. +// A reader must stop retaining rows as soon as it reaches its byte budget. +var ErrAuditHistoryWindowTooLarge = errors.New("audit history copy: window exceeds its memory budget") + +// ErrAuditHistoryTooDense refuses events sharing an instant that cannot fit a +// window. The run stays unverified and cannot remove source partitions. +var ErrAuditHistoryTooDense = errors.New("audit history copy: events at one instant exceed its memory budget") + +const DefaultAuditHistoryWindowBytes int64 = 32 << 20 + +// AuditHistoryStreamingSource avoids constructing a decoded page before the +// copier can apply its budget. Production Postgres implements this interface. +type AuditHistoryStreamingSource interface { + StreamAuditHistory(ctx context.Context, from, to time.Time, maxBytes int64, visit func(AuditEntry) error) error +} + // The one-time history copy of ADR 0009, item 7: switching a deployment from // postgres to a swap value copies the audit_events rows it already holds into // the store of record and the archive, verifies the copy by reading it back, @@ -130,6 +146,11 @@ type AuditHistoryCopyConfig struct { // BatchSize is the most events one copied batch carries (default // DefaultAuditRelayBatchSize). BatchSize int + // WindowBytes bounds source records and their verification state. A current + // decoded row is counted alongside its canonical text before it is retained. + // Zero is DefaultAuditHistoryWindowBytes. Warehouses also bound their own + // details join, and either side can ask the copy to halve a busy window. + WindowBytes int64 // Through, when set, limits the copy to the partitions whose bounds end at // or before it, and is the drop's cutoff; a confirmed drop needs it, and // refuses one that has not yet passed. Unset is every partition. @@ -229,11 +250,8 @@ var ErrAuditHistoryDropRefused = errors.New("audit history copy: drop refused") const maxAuditHistoryProblems = 20 -// auditHistoryPageSize is the rows one read of audit_events returns. -const auditHistoryPageSize = 1000 - // auditHistoryWindow is the span the copy reads, writes and verifies at a -// time: a day keeps the read-back of a busy deployment in memory. +// time initially. Busy days are retried as smaller windows. const auditHistoryWindow = 24 * time.Hour // auditHistoryDetailsMargin is how far past the instant the store is certain to @@ -267,6 +285,12 @@ func NewAuditHistoryCopy(cfg AuditHistoryCopyConfig) (*AuditHistoryCopy, error) if cfg.BatchSize < 1 || cfg.BatchSize > MaxAuditRelayBatchSize { return nil, fmt.Errorf("audit history copy: batch size must be between 1 and %d", MaxAuditRelayBatchSize) } + if cfg.WindowBytes < 0 { + return nil, errors.New("audit history copy: window byte budget cannot be negative") + } + if cfg.WindowBytes == 0 { + cfg.WindowBytes = DefaultAuditHistoryWindowBytes + } if cfg.Now == nil { cfg.Now = time.Now } @@ -452,7 +476,7 @@ func (c *AuditHistoryCopy) copyPartition(ctx context.Context, resolver *AuditEve if to.After(partition.To) { to = partition.To } - if err := c.copyWindow(ctx, resolver, from, to, opts, &report, digest); err != nil { + if err := c.copyBoundedWindow(ctx, resolver, from, to, opts, &report, digest); err != nil { return report, fmt.Errorf("audit history copy: %s [%s, %s): %w", partition.Name, from.UTC().Format(time.RFC3339), to.UTC().Format(time.RFC3339), err) } @@ -468,6 +492,37 @@ func (c *AuditHistoryCopy) copyPartition(ctx context.Context, resolver *AuditEve return report, nil } +// copyBoundedWindow visits halves oldest first, so window boundaries cannot +// change the source stream hashed into a copy, verification or drop receipt. +// Only a successful leaf adds source counts or digest bytes. Once writes have +// started an overflow fails this run; the next run can split its initial read +// without repeating a repair or permanent refusal during this run. +func (c *AuditHistoryCopy) copyBoundedWindow(ctx context.Context, resolver *AuditEventResolver, from, to time.Time, opts AuditHistoryCopyOptions, report *AuditHistoryPartitionReport, digest hash.Hash) error { + err := c.copyWindow(ctx, resolver, from, to, opts, report, digest) + var afterWrite *auditHistoryAfterWriteError + if errors.As(err, &afterWrite) { + return err + } + if !errors.Is(err, ErrAuditHistoryWindowTooLarge) { + return err + } + middle := time.UnixMicro(from.UnixMicro() + (to.UnixMicro()-from.UnixMicro())/2).UTC() + if !middle.After(from) || !middle.Before(to) { + return ErrAuditHistoryTooDense + } + if err := c.copyBoundedWindow(ctx, resolver, from, middle, opts, report, digest); err != nil { + return err + } + return c.copyBoundedWindow(ctx, resolver, middle, to, opts, report, digest) +} + +type auditHistoryAfterWriteError struct{ err error } + +func (e *auditHistoryAfterWriteError) Error() string { + return "audit history copy: read-back after append failed; rerun to resume: " + e.err.Error() +} +func (e *auditHistoryAfterWriteError) Unwrap() error { return e.err } + // copyWindow reads a window's rows, reads back what the store holds of it, // writes what is missing or incomplete, and verifies every row. // @@ -484,7 +539,7 @@ func (c *AuditHistoryCopy) copyWindow(ctx context.Context, resolver *AuditEventR if len(records) == 0 { return nil } - stored, err := c.readBack(ctx, from, to) + stored, err := c.readBack(ctx, from, to, records) if err != nil { return err } @@ -500,9 +555,9 @@ func (c *AuditHistoryCopy) copyWindow(ctx context.Context, resolver *AuditEventR var pending []AuditRecord var rewrite []bool // parallel to pending: the store already held the event for _, record := range records { - copies := stored[record.Entry.ID] - if _, writable := c.inspect(record, copies); writable { - pending, rewrite = append(pending, record), append(rewrite, len(copies) > 0) + status := stored[record.Entry.ID] + if !status.found || status.writable { + pending, rewrite = append(pending, record), append(rewrite, status.found) } } if len(pending) > 0 { @@ -525,8 +580,8 @@ func (c *AuditHistoryCopy) copyWindow(ctx context.Context, resolver *AuditEventR } } } - if stored, err = c.readBack(ctx, from, to); err != nil { - return err + if stored, err = c.readBack(ctx, from, to, records); err != nil { + return &auditHistoryAfterWriteError{err: err} } } } @@ -545,7 +600,11 @@ func (c *AuditHistoryCopy) copyWindow(ctx context.Context, resolver *AuditEventR count := report.Orgs[org] count.Postgres++ id := record.Entry.ID - if problem := c.verify(record, stored[id]); problem != "" { + problem := stored[id].problem + if !stored[id].found { + problem = "missing from the store" + } + if problem != "" { switch { case refused[id]: problem = auditHistoryRefusal + "; " + problem @@ -568,29 +627,55 @@ func (c *AuditHistoryCopy) copyWindow(ctx context.Context, resolver *AuditEventR // classifies and hashes each exactly as the relay does a live event. func (c *AuditHistoryCopy) readWindow(ctx context.Context, resolver *AuditEventResolver, from, to time.Time) ([]AuditRecord, error) { var records []AuditRecord + var held int64 + visit := func(entry AuditEntry) error { + resolved, err := resolver.Resolve(ctx, entry.EventType) + if err != nil { + return err + } + record, err := NewAuditRecord(entry, resolved.RetentionClass()) + if err != nil { + return err + } + n := AuditHistoryEntryBytes(entry) + int64(len(record.Details)+len(record.DetailsSHA256)) + 128 + if held+n > c.cfg.WindowBytes { + return ErrAuditHistoryWindowTooLarge + } + // Downstream writers use Details. Keeping the decoded map as well would + // retain several times the canonical string's memory for every event. + record.Entry.Payload = nil + held += AuditHistoryEntryBytes(record.Entry) + int64(len(record.Details)+len(record.DetailsSHA256)) + 128 + records = append(records, record) + return nil + } + if source, ok := c.cfg.Source.(AuditHistoryStreamingSource); ok { + err := c.cfg.Source.WithControlPlane(ctx, func(ctx context.Context) error { + return source.StreamAuditHistory(ctx, from, to, c.cfg.WindowBytes, visit) + }) + if err != nil { + return nil, fmt.Errorf("read audit_events: %w", err) + } + return records, nil + } + // Compatibility sources cannot apply a byte limit before returning a page. + // Ask for one row so an arbitrary decoded page cannot precede the budget. var after *AuditHistoryCursor for { var page []AuditEntry err := c.cfg.Source.WithControlPlane(ctx, func(ctx context.Context) error { var err error - page, err = c.cfg.Source.ReadAuditHistory(ctx, from, to, after, auditHistoryPageSize) + page, err = c.cfg.Source.ReadAuditHistory(ctx, from, to, after, 1) return err }) if err != nil { return nil, fmt.Errorf("read audit_events: %w", err) } for _, entry := range page { - resolved, err := resolver.Resolve(ctx, entry.EventType) - if err != nil { - return nil, err - } - record, err := NewAuditRecord(entry, resolved.RetentionClass()) - if err != nil { + if err := visit(entry); err != nil { return nil, err } - records = append(records, record) } - if len(page) < auditHistoryPageSize { + if len(page) == 0 { return records, nil } last := page[len(page)-1] @@ -598,11 +683,77 @@ func (c *AuditHistoryCopy) readWindow(ctx context.Context, resolver *AuditEventR } } -// readBack reads every copy the store holds of the window's events. -func (c *AuditHistoryCopy) readBack(ctx context.Context, from, to time.Time) (map[string][]StoredAuditEvent, error) { - stored := map[string][]StoredAuditEvent{} +// AuditHistoryEntryBytes counts an envelope and its decoded payload. The count +// includes container entries, not just JSON text, and is also used before the +// production source retains a decoded row. +func AuditHistoryEntryBytes(entry AuditEntry) int64 { + n := int64(512 + len(entry.ID) + len(entry.OrgID) + len(entry.ActorID) + len(entry.ActorType) + len(entry.EventType) + len(entry.Resource) + len(entry.ResourceID) + len(entry.IPAddress) + len(entry.ImpersonatedBy) + len(entry.ClientID) + len(entry.IdempotencyKey)) + var payloadBytes func(any) int64 + payloadBytes = func(value any) int64 { + switch v := value.(type) { + case map[string]any: + n := int64(64) + for key, value := range v { + n += 64 + int64(len(key)) + payloadBytes(value) + } + return n + case []any: + n := int64(24 + len(v)*16) + for _, value := range v { + n += payloadBytes(value) + } + return n + case string: + return 16 + int64(len(v)) + case json.Number: + return 16 + int64(len(v)) + default: + return 16 + } + } + if entry.Payload != nil { + n += payloadBytes(entry.Payload) + } + return n +} + +type auditHistoryStatus struct { + found, writable bool + problem string +} + +// readBack checks every stored copy while streaming it, retaining one status +// per source event rather than all warehouse duplicates and their payloads. +func (c *AuditHistoryCopy) readBack(ctx context.Context, from, to time.Time, records []AuditRecord) (map[string]auditHistoryStatus, error) { + stored := map[string]auditHistoryStatus{} + wanted := map[string]AuditRecord{} + for _, record := range records { + if _, duplicate := wanted[record.Entry.ID]; duplicate { + stored[record.Entry.ID] = auditHistoryStatus{found: true, problem: "stored envelope differs from the row"} + } else { + wanted[record.Entry.ID] = record + } + } err := c.cfg.ReadBack.ReadStoredAuditEvents(ctx, from, to, func(event StoredAuditEvent) error { - stored[event.Entry.ID] = append(stored[event.Entry.ID], event) + record, ok := wanted[event.Entry.ID] + if !ok { + return nil + } + status := stored[event.Entry.ID] + if status.problem != "" && !status.writable { + return nil + } + problem, writable := c.inspectCopy(record, event) + status.found = true + if problem != "" && !writable { + status.problem, status.writable = problem, false + } else if writable { + status.writable = true + if status.problem == "" { + status.problem = problem + } + } + stored[event.Entry.ID] = status return nil }) if err != nil { diff --git a/module/services/accounts/code/pkg/business/audit_history_bound_test.go b/module/services/accounts/code/pkg/business/audit_history_bound_test.go new file mode 100644 index 000000000..2e77a517f --- /dev/null +++ b/module/services/accounts/code/pkg/business/audit_history_bound_test.go @@ -0,0 +1,135 @@ +package business + +import ( + "context" + "strings" + "testing" + "time" + + "github.com/stretchr/testify/require" +) + +func TestAuditHistorySplitsBusySourceWindowsAndKeepsDigestStable(t *testing.T) { + f := newHistoryFixture() + f.source.rows = nil + day := f.source.partitions[0].From.Add(time.Hour) + for i := range 12 { + row := historyRow(i+1, "", EventAuthLogin, day.Add(time.Duration(i)*time.Hour)) + row.Payload = map[string]any{"blob": strings.Repeat("x", 1000)} + f.source.rows = append(f.source.rows, row) + } + report, err := f.copier(t, AuditHistoryCopyConfig{WindowBytes: 6000, BatchSize: 1}).Run(context.Background(), AuditHistoryCopyOptions{}) + require.NoError(t, err) + require.True(t, report.Verified) + require.Equal(t, 12, report.Partitions[0].Copied) + for _, batch := range f.archive.batches { + for _, record := range batch.Records { + require.Nil(t, record.Entry.Payload, "retained records keep only canonical details") + } + } + verify, err := f.copier(t, AuditHistoryCopyConfig{WindowBytes: 10000}).Run(context.Background(), AuditHistoryCopyOptions{VerifyOnly: true}) + require.NoError(t, err) + require.Equal(t, report.PartitionsSHA256, verify.PartitionsSHA256, "window boundaries cannot alter the approved source digest") + drop, err := f.copier(t, AuditHistoryCopyConfig{WindowBytes: 4000}).Run(context.Background(), AuditHistoryCopyOptions{ + VerifyOnly: true, ConfirmDrop: true, ExpectedPartitionsSHA256: verify.PartitionsSHA256, + }) + require.NoError(t, err) + require.Equal(t, report.PartitionsSHA256, drop.PartitionsSHA256) + require.Equal(t, int64(3), drop.Dropped) +} + +func TestAuditHistoryRejectsAnUnsplittablyDenseInstantBeforeWrites(t *testing.T) { + f := newHistoryFixture() + f.source.rows = nil + at := f.source.partitions[0].From.Add(time.Hour) + for i := range 8 { + f.source.rows = append(f.source.rows, historyRow(i+1, "", EventAuthLogin, at)) + } + report, err := f.copier(t, AuditHistoryCopyConfig{WindowBytes: 2000}).Run(context.Background(), AuditHistoryCopyOptions{}) + require.ErrorIs(t, err, ErrAuditHistoryTooDense) + require.False(t, report.Verified) + require.Empty(t, f.archive.batches) + require.Empty(t, f.source.drops) +} + +func TestAuditHistoryVerifiesEveryDuplicateWithoutRetainingCopies(t *testing.T) { + f := newHistoryFixture() + f.source.rows = f.source.rows[:1] + _, err := f.copier(t, AuditHistoryCopyConfig{}).Run(context.Background(), AuditHistoryCopyOptions{}) + require.NoError(t, err) + event := f.store.stored[0] + for range 10000 { + f.store.stored = append(f.store.stored, event) + } + report, err := f.copier(t, AuditHistoryCopyConfig{WindowBytes: 2000}).Run(context.Background(), AuditHistoryCopyOptions{VerifyOnly: true}) + require.NoError(t, err) + require.True(t, report.Verified) + f.store.stored[len(f.store.stored)-1].Details = "{\"corrupt\":true}" + report, err = f.copier(t, AuditHistoryCopyConfig{WindowBytes: 2000}).Run(context.Background(), AuditHistoryCopyOptions{VerifyOnly: true}) + require.ErrorIs(t, err, ErrAuditHistoryUnverified) + require.Contains(t, report.Partitions[0].Problems[0], "stored details do not hash") +} + +type streamingHistorySource struct { + *historySource + streams int +} + +func (s *streamingHistorySource) ReadAuditHistory(context.Context, time.Time, time.Time, *AuditHistoryCursor, int) ([]AuditEntry, error) { + panic("the streaming copy must never construct a source page") +} + +func (s *streamingHistorySource) StreamAuditHistory(ctx context.Context, from, to time.Time, _ int64, visit func(AuditEntry) error) error { + s.streams++ + rows, err := s.historySource.ReadAuditHistory(ctx, from, to, nil, len(s.rows)+1) + if err != nil { + return err + } + for _, row := range rows { + if err := visit(row); err != nil { + return err + } + } + return nil +} + +func TestAuditHistoryUsesTheStreamingSource(t *testing.T) { + f := newHistoryFixture() + c := f.copier(t, AuditHistoryCopyConfig{}) + source := &streamingHistorySource{historySource: f.source} + c.cfg.Source = source + report, err := c.Run(context.Background(), AuditHistoryCopyOptions{}) + require.NoError(t, err) + require.True(t, report.Verified) + require.Positive(t, source.streams) +} + +type historyOverflowAfterWrite struct { + *historyStore + reads int +} + +func (s *historyOverflowAfterWrite) ReadStoredAuditEvents(ctx context.Context, from, to time.Time, visit func(StoredAuditEvent) error) error { + s.reads++ + if s.reads == 2 { + return ErrAuditHistoryWindowTooLarge + } + return s.historyStore.ReadStoredAuditEvents(ctx, from, to, visit) +} + +func TestAuditHistoryPostWriteOverflowStopsWithoutRepeatingRepair(t *testing.T) { + f := newHistoryFixture() + f.source.rows = f.source.rows[:1] + c := f.copier(t, AuditHistoryCopyConfig{}) + reader := &historyOverflowAfterWrite{historyStore: f.store} + c.cfg.ReadBack = reader + report, err := c.Run(context.Background(), AuditHistoryCopyOptions{}) + require.ErrorIs(t, err, ErrAuditHistoryWindowTooLarge) + require.False(t, report.Verified) + require.Equal(t, 2, reader.reads, "no smaller read after writes in this run") + require.Equal(t, 1, f.store.appends) + require.Empty(t, f.source.drops) + report, err = c.Run(context.Background(), AuditHistoryCopyOptions{VerifyOnly: true}) + require.NoError(t, err) + require.True(t, report.Verified) +} diff --git a/module/services/accounts/code/pkg/business/audit_history_test.go b/module/services/accounts/code/pkg/business/audit_history_test.go index 12b42590e..6ddeb1a53 100644 --- a/module/services/accounts/code/pkg/business/audit_history_test.go +++ b/module/services/accounts/code/pkg/business/audit_history_test.go @@ -406,13 +406,17 @@ func TestAuditHistoryCopyCopiesVerifiesAndDropsOnlyWhenConfirmed(t *testing.T) { // Each copy carries the envelope, class and hash a live event of the row gets. resolver := NewAuditEventResolver(nil) + sourceRows := map[string]AuditEntry{} + for _, row := range f.source.rows { + sourceRows[row.ID] = row + } for _, batch := range f.archive.batches { require.Equal(t, "deployment-1", batch.DeploymentID) require.LessOrEqual(t, len(batch.Records), 4) for _, record := range batch.Records { resolved, err := resolver.Resolve(context.Background(), record.Entry.EventType) require.NoError(t, err) - want, err := NewAuditRecord(record.Entry, resolved.RetentionClass()) + want, err := NewAuditRecord(sourceRows[record.Entry.ID], resolved.RetentionClass()) require.NoError(t, err) require.Equal(t, want.DetailsSHA256, record.DetailsSHA256) require.Equal(t, want.Retention, record.Retention) diff --git a/module/services/accounts/code/pkg/infra/postgres_audit_history.go b/module/services/accounts/code/pkg/infra/postgres_audit_history.go index fb9c10f4a..f458b89b5 100644 --- a/module/services/accounts/code/pkg/infra/postgres_audit_history.go +++ b/module/services/accounts/code/pkg/infra/postgres_audit_history.go @@ -1,7 +1,9 @@ package infra import ( + "bytes" "context" + "encoding/json" "errors" "fmt" "strings" @@ -81,12 +83,33 @@ func (s *PostgresStore) ReadAuditHistory(ctx context.Context, from, to time.Time } query += fmt.Sprintf(` ORDER BY created_at, id LIMIT $%d`, len(args)+1) args = append(args, limit) - rows, err := s.getQueryExecutor(ctx).Query(ctx, query, args...) + var out []business.AuditEntry + err := s.visitAuditHistory(ctx, query, args, 0, func(entry business.AuditEntry) error { + out = append(out, entry) + return nil + }) if err != nil { return nil, err } + return out, err +} + +// StreamAuditHistory hands rows on in source digest order without retaining a +// decoded page. A busy window can stop the scan immediately and be retried as +// halves; even one row's decoded containers are bounded before allocation. +func (s *PostgresStore) StreamAuditHistory(ctx context.Context, from, to time.Time, maxBytes int64, visit func(business.AuditEntry) error) error { + if maxBytes <= 0 { + return errors.New("audit history: streaming byte budget must be positive") + } + return s.visitAuditHistory(ctx, auditHistorySelectSQL+` ORDER BY created_at, id`, []any{from, to}, maxBytes, visit) +} + +func (s *PostgresStore) visitAuditHistory(ctx context.Context, query string, args []any, maxBytes int64, visit func(business.AuditEntry) error) error { + rows, err := s.getQueryExecutor(ctx).Query(ctx, query, args...) + if err != nil { + return err + } defer rows.Close() - var out []business.AuditEntry for rows.Next() { var entry business.AuditEntry var eventType string @@ -99,16 +122,121 @@ func (s *PostgresStore) ReadAuditHistory(ctx context.Context, from, to time.Time &entry.ImpersonatedBy, &entry.IsImpersonated, &entry.ClientID, ); err != nil { - return nil, err + return err } entry.EventType = business.EventType(eventType) entry.CreatedAt = entry.CreatedAt.UTC() - if entry.Payload, err = decodeQueuedPayload(payload); err != nil { - return nil, fmt.Errorf("audit history: payload of event %s: %w", entry.ID, err) + if maxBytes > 0 { + remaining := maxBytes - business.AuditHistoryEntryBytes(entry) + entry.Payload, err = decodeBoundedHistoryPayload(payload, remaining) + } else { + entry.Payload, err = decodeQueuedPayload(payload) + } + if err != nil { + return fmt.Errorf("audit history: payload of event %s: %w", entry.ID, err) + } + if err := visit(entry); err != nil { + return err } - out = append(out, entry) } - return out, rows.Err() + return rows.Err() +} + +// decodeBoundedHistoryPayload counts the raw row and decoded container entries +// as it decodes. Checking only raw JSON length misses small JSON arrays that +// allocate many maps or interfaces; checking after Decode is too late. +func decodeBoundedHistoryPayload(raw []byte, limit int64) (map[string]any, error) { + held := int64(len(raw)) + take := func(n int64) error { + held += n + if held > limit { + return business.ErrAuditHistoryWindowTooLarge + } + return nil + } + if err := take(0); err != nil { + return nil, err + } + decoder := json.NewDecoder(bytes.NewReader(raw)) + decoder.UseNumber() + var value func() (any, error) + value = func() (any, error) { + token, err := decoder.Token() + if err != nil { + return nil, err + } + if delimiter, ok := token.(json.Delim); ok { + switch delimiter { + case '{': + if err := take(64); err != nil { + return nil, err + } + object := map[string]any{} + for decoder.More() { + key, err := decoder.Token() + if err != nil { + return nil, err + } + text, ok := key.(string) + if !ok { + return nil, errors.New("audit history: expected a JSON object key") + } + if err := take(64 + int64(len(text))); err != nil { + return nil, err + } + item, err := value() + if err != nil { + return nil, err + } + object[text] = item + } + _, err := decoder.Token() + return object, err + case '[': + if err := take(24); err != nil { + return nil, err + } + var array []any + for decoder.More() { + if err := take(16); err != nil { + return nil, err + } + item, err := value() + if err != nil { + return nil, err + } + array = append(array, item) + } + _, err := decoder.Token() + return array, err + default: + return nil, errors.New("audit history: unexpected JSON delimiter") + } + } + n := int64(16) + switch text := token.(type) { + case string: + n += int64(len(text)) + case json.Number: + n += int64(len(text)) + } + if err := take(n); err != nil { + return nil, err + } + return token, nil + } + decoded, err := value() + if err != nil { + return nil, err + } + if decoded == nil { + return nil, nil + } + payload, ok := decoded.(map[string]any) + if !ok { + return nil, errors.New("audit history: payload must be a JSON object") + } + return payload, nil } // CountAuditHistory implements business.AuditHistorySource. @@ -204,3 +332,4 @@ func (s *PostgresStore) DropVerifiedAuditPartitions(ctx context.Context, cutoff } var _ business.AuditHistorySource = (*PostgresStore)(nil) +var _ business.AuditHistoryStreamingSource = (*PostgresStore)(nil) diff --git a/module/services/accounts/code/pkg/infra/postgres_audit_history_bound_test.go b/module/services/accounts/code/pkg/infra/postgres_audit_history_bound_test.go new file mode 100644 index 000000000..8ef8da134 --- /dev/null +++ b/module/services/accounts/code/pkg/infra/postgres_audit_history_bound_test.go @@ -0,0 +1,23 @@ +package infra + +import ( + "accounts/pkg/business" + "github.com/stretchr/testify/require" + "strings" + "testing" +) + +func TestHistoryPayloadDecodeStopsAtDecodedContainerBudget(t *testing.T) { + raw := []byte(`{"items":[` + strings.Repeat(`{},`, 100) + `{}` + `]}`) + _, err := decodeBoundedHistoryPayload(raw, int64(len(raw))+256) + require.ErrorIs(t, err, business.ErrAuditHistoryWindowTooLarge) +} + +func TestHistoryPayloadBoundedDecodePreservesCanonicalNumbers(t *testing.T) { + raw := []byte(`{"large":9007199254740993,"nested":[{"value":0.0012300},null,true,"text"]}`) + expected, err := decodeQueuedPayload(raw) + require.NoError(t, err) + actual, err := decodeBoundedHistoryPayload(raw, 10000) + require.NoError(t, err) + require.Equal(t, expected, actual) +} diff --git a/module/services/accounts/code/pkg/infra/testdata/control_plane_call_sites.txt b/module/services/accounts/code/pkg/infra/testdata/control_plane_call_sites.txt index e9013bb71..2f20e7e72 100644 --- a/module/services/accounts/code/pkg/infra/testdata/control_plane_call_sites.txt +++ b/module/services/accounts/code/pkg/infra/testdata/control_plane_call_sites.txt @@ -32,6 +32,7 @@ pkg/business/audit.go: (*DurableAuditEmitter).Emit: WithControlPlane pkg/business/audit_history.go: (*AuditHistoryCopy).drop: WithControlPlane pkg/business/audit_history.go: (*AuditHistoryCopy).partitions: WithControlPlane pkg/business/audit_history.go: (*AuditHistoryCopy).readWindow: WithControlPlane +pkg/business/audit_history.go: (*AuditHistoryCopy).readWindow: WithControlPlane pkg/business/audit_history.go: (*AuditHistoryCopy).uncovered: WithControlPlane pkg/business/audit_store_read.go: (postgresAuditReader).within: WithControlPlane pkg/business/connector_credentials.go: (*Service).SourceSigningSecret: WithControlPlane diff --git a/module/services/accounts/service.codefly.yaml b/module/services/accounts/service.codefly.yaml index 60b27bbaa..4083755f0 100644 --- a/module/services/accounts/service.codefly.yaml +++ b/module/services/accounts/service.codefly.yaml @@ -55,6 +55,11 @@ endpoints: - name: grpc - name: rest spec: + # CLI-owned raw, non-secret deployment defaults. The resolved environment's + # service-config overrides this value; deploy Jobs inherit the resulting + # service value rather than guessing a sink of their own. + environment-defaults: + AUDIT_SINK: postgres connect-endpoint: true hot-reload: true # The datasource fetch reads a repository over git's smart-HTTP transport by From f4f1f9c343793d2a82eb8f45166844591e2bf0e2 Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Wed, 7 Oct 2026 14:31:36 -0400 Subject: [PATCH 58/69] fix(audit): format bounded history test imports --- .../code/pkg/infra/postgres_audit_history_bound_test.go | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/module/services/accounts/code/pkg/infra/postgres_audit_history_bound_test.go b/module/services/accounts/code/pkg/infra/postgres_audit_history_bound_test.go index 8ef8da134..cea1955ec 100644 --- a/module/services/accounts/code/pkg/infra/postgres_audit_history_bound_test.go +++ b/module/services/accounts/code/pkg/infra/postgres_audit_history_bound_test.go @@ -2,9 +2,10 @@ package infra import ( "accounts/pkg/business" - "github.com/stretchr/testify/require" "strings" "testing" + + "github.com/stretchr/testify/require" ) func TestHistoryPayloadDecodeStopsAtDecodedContainerBudget(t *testing.T) { From ee06dacbbb3c6df277069ed1e00ede6e93deac52 Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Wed, 7 Oct 2026 14:36:23 -0400 Subject: [PATCH 59/69] fix(audit): run catalog imports with deployed database credentials --- module/services/accounts/AUTHZ.md | 30 ++++ .../services/accounts/code/audit_commands.go | 3 + .../code/cmd/role-catalog-import/main.go | 141 +-------------- .../accounts/code/pkg/auditops/common.go | 13 ++ .../code/pkg/rolecatalogimport/import.go | 169 ++++++++++++++++++ .../rolecatalogimport/import_test.go} | 104 +++++++++-- 6 files changed, 311 insertions(+), 149 deletions(-) create mode 100644 module/services/accounts/code/pkg/rolecatalogimport/import.go rename module/services/accounts/code/{cmd/role-catalog-import/main_test.go => pkg/rolecatalogimport/import_test.go} (52%) diff --git a/module/services/accounts/AUTHZ.md b/module/services/accounts/AUTHZ.md index 223d544eb..7a7ad7665 100644 --- a/module/services/accounts/AUTHZ.md +++ b/module/services/accounts/AUTHZ.md @@ -198,6 +198,36 @@ the adapters' `markContentReads`): `CheckAuthorizationRevision` catches it because it re-resolves every permission before it compares revisions at all. +## Built-in role catalog import + +The accounts executable ships `role-catalog-import` as an authored subcommand: + +```sh +./app role-catalog-import -catalog /catalog/roles.json [-dry-run] [-force] +``` + +Without `-database-url` or `DATABASE_URL`, it initializes the same Codefly +configuration as accounts and opens the store's projected read-only, request and +control-plane capabilities. The existing Postgres credential provider selects +and refreshes each login's projected token (`POSTGRES_TOKEN_FILES`, or the single +`POSTGRES_TOKEN_FILE` where the deployment uses one). A deploy Job therefore runs +the immutable accounts image with its resolved configuration, identity and +credential mounts; it needs no synthesized database URL or migration credential. +The import writes built-in roles on the control-plane transaction and leaves +tenant-defined custom roles alone. + +For a standalone operator invocation, `-database-url` overrides `DATABASE_URL`. +That explicit URL keeps its supplied credential and is not replaced by a +deployment token file. Its principal must have membership in both `app_tenant` +and `app_control_plane`, as required by the shared tooling-store constructor. + +Every invocation requires the deployment's nonblank `AUDIT_SINK`, either in its +environment or explicitly as `-audit-sink`; this is checked before resolving +credentials. Warehouse modes record platform audit events in the transactional +queue for the accounts relay. `-dry-run` prints the plan without applying it; +`-force` permits removals that delete assignments or remove the whole catalog. +An unchanged catalog remains a no-op. + ## Deferred Boundary-level RLS inside module stores (an `app.current_boundaries` GUC) stays diff --git a/module/services/accounts/code/audit_commands.go b/module/services/accounts/code/audit_commands.go index 0c54094a6..4aff595d2 100644 --- a/module/services/accounts/code/audit_commands.go +++ b/module/services/accounts/code/audit_commands.go @@ -2,6 +2,7 @@ package main import ( "accounts/pkg/auditops" + "accounts/pkg/rolecatalogimport" "os" ) @@ -13,6 +14,8 @@ func init() { return } switch os.Args[1] { + case "role-catalog-import": + os.Exit(rolecatalogimport.Run(os.Args[2:], os.Stdout, os.Stderr)) case "audit-history-copy": os.Exit(auditops.RunHistory(os.Args[2:], os.Getenv, os.Stdout, os.Stderr)) case "audit-qualify": diff --git a/module/services/accounts/code/cmd/role-catalog-import/main.go b/module/services/accounts/code/cmd/role-catalog-import/main.go index 74a3d9b13..74e1bb518 100644 --- a/module/services/accounts/code/cmd/role-catalog-import/main.go +++ b/module/services/accounts/code/cmd/role-catalog-import/main.go @@ -1,146 +1,11 @@ -// Command role-catalog-import syncs an external permission catalog into the -// built-in scoped roles (roles.built_in = true, org_id IS NULL). It upserts the -// catalog's roles and diff-applies their permissions; it never touches -// org-defined custom roles and never deletes assignments unless -force. -// -// Usage: -// -// role-catalog-import -catalog roles.json -database-url "$DATABASE_URL" [-audit-sink postgres|both|bigquery|clickhouse] [-dry-run] [-force] -// -// The connection principal must be a member of app_control_plane (the same -// authority migrations run under); built-in roles cannot be written otherwise. -// See AUTHZ.md ("Built-in role catalog import") for the format and workflow. -// -// The import must run under the deployment's AUDIT_SINK, read through the same -// auditsink package the accounts service reads it with (-audit-sink, else -// $AUDIT_SINK): under a swap value (ADR 0009) the import's audit events go to -// the transactional queue, which the accounts relay delivers, rather than to -// audit_events. Unlike the service, the import has no default: a deploy step that -// did not receive the setting stops, because guessing postgres would write its -// events where a warehouse deployment never reads them. +// Command role-catalog-import also ships as an accounts executable subcommand. package main import ( - "context" - "flag" - "fmt" - "io" + "accounts/pkg/rolecatalogimport" "os" - - "accounts/pkg/auditstore/auditsink" - "accounts/pkg/business" - "accounts/pkg/infra" - "accounts/pkg/rolecatalog" ) func main() { - os.Exit(run(os.Args[1:], os.Stdout, os.Stderr)) -} - -// Writes to the process's stdout/stderr have no meaningful recovery, so these -// helpers ignore the write error while keeping the io.Writer seam that lets the -// outcome/exit mapping be tested without a database. -func line(w io.Writer, a ...any) { _, _ = fmt.Fprintln(w, a...) } -func text(w io.Writer, s string) { _, _ = fmt.Fprint(w, s) } - -// run parses flags, applies the catalog, and returns a process exit code. -// Returning (rather than calling os.Exit inside) keeps every deferred cleanup — -// notably the store's connection pool — unwinding on all paths. -func run(args []string, stdout, stderr io.Writer) int { - fs := flag.NewFlagSet("role-catalog-import", flag.ContinueOnError) - fs.SetOutput(stderr) - catalogPath := fs.String("catalog", "", "path to the JSON permission catalog (required)") - databaseURL := fs.String("database-url", os.Getenv("DATABASE_URL"), "Postgres connection URL (defaults to $DATABASE_URL)") - dryRun := fs.Bool("dry-run", false, "print the plan without applying it") - force := fs.Bool("force", false, "apply removals even when they would delete assignments or wipe the whole catalog") - auditSink := fs.String("audit-sink", "", "the deployment's AUDIT_SINK: postgres, both, bigquery or clickhouse (defaults to $AUDIT_SINK, which must then be set)") - if err := fs.Parse(args); err != nil { - return 2 - } - - if *catalogPath == "" { - line(stderr, "role-catalog-import: -catalog is required") - return 1 - } - if *databaseURL == "" { - line(stderr, "role-catalog-import: -database-url (or $DATABASE_URL) is required") - return 1 - } - - // The flag, when given, stands in for the environment variable the service - // reads; either way the value is read and checked by auditsink, and a value - // that was never set is refused rather than defaulted. - lookup := os.LookupEnv - fs.Visit(func(f *flag.Flag) { - if f.Name == "audit-sink" { - lookup = func(string) (string, bool) { return *auditSink, true } - } - }) - sinkMode, err := auditsink.RequireMode(lookup) - if err != nil { - line(stderr, "role-catalog-import:", err) - return 1 - } - - document, err := os.ReadFile(*catalogPath) - if err != nil { - line(stderr, "role-catalog-import:", err) - return 1 - } - catalog, err := rolecatalog.Parse(document) - if err != nil { - line(stderr, "role-catalog-import:", err) - return 1 - } - - ctx := context.Background() - store, err := infra.NewPostgresStoreFromURL(ctx, *databaseURL) - if err != nil { - line(stderr, "role-catalog-import:", err) - return 1 - } - defer store.Close() - - var recorderOptions []business.DurableAuditEmitterOption - if sinkMode.Swaps() { - recorderOptions = append(recorderOptions, business.WithQueuedRecords()) - } - // The import records platform events only (built-in roles have no - // organization), which neither the tee nor webhook delivery carries, so - // the recorder needs nothing from the mode but where the record goes. - recorder, err := business.NewDurableAuditEmitter(store, store, recorderOptions...) - if err != nil { - line(stderr, "role-catalog-import:", err) - return 1 - } - - result, err := store.ImportRoleCatalog(ctx, catalog, infra.ImportOptions{ - DryRun: *dryRun, Force: *force, Source: *catalogPath, Audit: recorder, - }) - if err != nil { - line(stderr, "role-catalog-import:", err) - return 1 - } - - return report(stdout, stderr, result, *dryRun) -} - -// report prints the plan and outcome and returns the exit code. Pure over its -// inputs so the outcome/exit mapping is testable without a database. -func report(stdout, stderr io.Writer, result *infra.ImportResult, dryRun bool) int { - text(stdout, result.Plan.Format()) - switch { - case dryRun: - line(stdout, "dry-run: no changes applied") - return 0 - case result.Refused: - line(stderr, "refused:", result.RefusalReason) - return 2 - case result.Applied && result.Plan.Empty(): - line(stdout, "no changes") - return 0 - default: - line(stdout, "applied") - return 0 - } + os.Exit(rolecatalogimport.Run(os.Args[1:], os.Stdout, os.Stderr)) } diff --git a/module/services/accounts/code/pkg/auditops/common.go b/module/services/accounts/code/pkg/auditops/common.go index 8dc4c7622..b50b94759 100644 --- a/module/services/accounts/code/pkg/auditops/common.go +++ b/module/services/accounts/code/pkg/auditops/common.go @@ -28,6 +28,19 @@ func commandContext(ctx context.Context) (context.Context, error) { } return provider.WithLogger(quietLog{}).Inject(ctx), nil } + +// OpenDeployedStore opens the same projected Postgres capabilities and rotating +// credentials as the accounts service. Deploy commands must initialize the +// Codefly context before resolving the store's secrets; a raw operator URL is +// deliberately handled separately by NewPostgresStoreFromURL. +func OpenDeployedStore(ctx context.Context) (*infra.PostgresStore, error) { + ctx, err := commandContext(ctx) + if err != nil { + return nil, err + } + return infra.NewPostgresStore(ctx) +} + func openStore(ctx context.Context, testURL string) (*infra.PostgresStore, error) { if testURL != "" { return infra.NewPostgresStoreFromURL(ctx, testURL) diff --git a/module/services/accounts/code/pkg/rolecatalogimport/import.go b/module/services/accounts/code/pkg/rolecatalogimport/import.go new file mode 100644 index 000000000..9069fb0de --- /dev/null +++ b/module/services/accounts/code/pkg/rolecatalogimport/import.go @@ -0,0 +1,169 @@ +// Package rolecatalogimport syncs an external permission catalog into the +// built-in scoped roles (roles.built_in = true, org_id IS NULL). It upserts the +// catalog's roles and diff-applies their permissions; it never touches +// org-defined custom roles and never deletes assignments unless -force. +// +// Usage: +// +// role-catalog-import -catalog roles.json [-database-url "$DATABASE_URL"] [-audit-sink postgres|both|bigquery|clickhouse] [-dry-run] [-force] +// +// Without an explicit URL it uses the accounts deployment's resolved Postgres +// capabilities and rotating credentials. An operator URL keeps its embedded +// credential and must have membership in app_tenant and app_control_plane; +// built-in roles cannot be written without the control-plane capability. +// See AUTHZ.md ("Built-in role catalog import") for the format and workflow. +// +// The import must run under the deployment's AUDIT_SINK, read through the same +// auditsink package the accounts service reads it with (-audit-sink, else +// $AUDIT_SINK): under a swap value (ADR 0009) the import's audit events go to +// the transactional queue, which the accounts relay delivers, rather than to +// audit_events. Unlike the service, the import has no default: a deploy step that +// did not receive the setting stops, because guessing postgres would write its +// events where a warehouse deployment never reads them. +package rolecatalogimport + +import ( + "context" + "flag" + "fmt" + "io" + "os" + "strings" + + "accounts/pkg/auditops" + "accounts/pkg/auditstore/auditsink" + "accounts/pkg/business" + "accounts/pkg/infra" + "accounts/pkg/rolecatalog" +) + +// Writes to the process's stdout/stderr have no meaningful recovery, so these +// helpers ignore the write error while keeping the io.Writer seam that lets the +// outcome/exit mapping be tested without a database. +func line(w io.Writer, a ...any) { _, _ = fmt.Fprintln(w, a...) } +func text(w io.Writer, s string) { _, _ = fmt.Fprint(w, s) } + +// Run parses flags, applies the catalog, and returns a process exit code. +// Returning (rather than calling os.Exit inside) keeps every deferred cleanup — +// notably the store's connection pool — unwinding on all paths. +func Run(args []string, stdout, stderr io.Writer) int { + return runWithOpeners(args, stdout, stderr, storeOpeners{ + deployed: auditops.OpenDeployedStore, + explicit: infra.NewPostgresStoreFromURL, + }) +} + +type storeOpeners struct { + deployed func(context.Context) (*infra.PostgresStore, error) + explicit func(context.Context, string) (*infra.PostgresStore, error) +} + +func runWithOpeners(args []string, stdout, stderr io.Writer, openers storeOpeners) int { + fs := flag.NewFlagSet("role-catalog-import", flag.ContinueOnError) + fs.SetOutput(stderr) + catalogPath := fs.String("catalog", "", "path to the JSON permission catalog (required)") + databaseURL := fs.String("database-url", os.Getenv("DATABASE_URL"), "operator Postgres URL (defaults to $DATABASE_URL; otherwise uses deployed accounts capabilities)") + dryRun := fs.Bool("dry-run", false, "print the plan without applying it") + force := fs.Bool("force", false, "apply removals even when they would delete assignments or wipe the whole catalog") + auditSink := fs.String("audit-sink", "", "the deployment's AUDIT_SINK: postgres, both, bigquery or clickhouse (defaults to $AUDIT_SINK, which must then be set)") + if err := fs.Parse(args); err != nil { + return 2 + } + + if *catalogPath == "" { + line(stderr, "role-catalog-import: -catalog is required") + return 1 + } + explicitFlag := false + fs.Visit(func(f *flag.Flag) { + if f.Name == "database-url" { + explicitFlag = true + } + }) + if explicitFlag && strings.TrimSpace(*databaseURL) == "" { + line(stderr, "role-catalog-import: an explicit -database-url must not be blank") + return 1 + } + + // The flag, when given, stands in for the environment variable the service + // reads; either way the value is read and checked by auditsink, and a value + // that was never set is refused rather than defaulted. + lookup := os.LookupEnv + fs.Visit(func(f *flag.Flag) { + if f.Name == "audit-sink" { + lookup = func(string) (string, bool) { return *auditSink, true } + } + }) + sinkMode, err := auditsink.RequireMode(lookup) + if err != nil { + line(stderr, "role-catalog-import:", err) + return 1 + } + + document, err := os.ReadFile(*catalogPath) + if err != nil { + line(stderr, "role-catalog-import:", err) + return 1 + } + catalog, err := rolecatalog.Parse(document) + if err != nil { + line(stderr, "role-catalog-import:", err) + return 1 + } + + ctx := context.Background() + var store *infra.PostgresStore + if strings.TrimSpace(*databaseURL) != "" { + store, err = openers.explicit(ctx, *databaseURL) + } else { + store, err = openers.deployed(ctx) + } + if err != nil { + line(stderr, "role-catalog-import:", err) + return 1 + } + defer store.Close() + + var recorderOptions []business.DurableAuditEmitterOption + if sinkMode.Swaps() { + recorderOptions = append(recorderOptions, business.WithQueuedRecords()) + } + // The import records platform events only (built-in roles have no + // organization), which neither the tee nor webhook delivery carries, so + // the recorder needs nothing from the mode but where the record goes. + recorder, err := business.NewDurableAuditEmitter(store, store, recorderOptions...) + if err != nil { + line(stderr, "role-catalog-import:", err) + return 1 + } + + result, err := store.ImportRoleCatalog(ctx, catalog, infra.ImportOptions{ + DryRun: *dryRun, Force: *force, Source: *catalogPath, Audit: recorder, + }) + if err != nil { + line(stderr, "role-catalog-import:", err) + return 1 + } + + return report(stdout, stderr, result, *dryRun) +} + +// report prints the plan and outcome and returns the exit code. Pure over its +// inputs so the outcome/exit mapping is testable without a database. +func report(stdout, stderr io.Writer, result *infra.ImportResult, dryRun bool) int { + text(stdout, result.Plan.Format()) + switch { + case dryRun: + line(stdout, "dry-run: no changes applied") + return 0 + case result.Refused: + line(stderr, "refused:", result.RefusalReason) + return 2 + case result.Applied && result.Plan.Empty(): + line(stdout, "no changes") + return 0 + default: + line(stdout, "applied") + return 0 + } +} diff --git a/module/services/accounts/code/cmd/role-catalog-import/main_test.go b/module/services/accounts/code/pkg/rolecatalogimport/import_test.go similarity index 52% rename from module/services/accounts/code/cmd/role-catalog-import/main_test.go rename to module/services/accounts/code/pkg/rolecatalogimport/import_test.go index 627e2028e..0bea648e2 100644 --- a/module/services/accounts/code/cmd/role-catalog-import/main_test.go +++ b/module/services/accounts/code/pkg/rolecatalogimport/import_test.go @@ -1,7 +1,9 @@ -package main +package rolecatalogimport import ( "bytes" + "context" + "errors" "os" "path/filepath" "strings" @@ -67,20 +69,100 @@ func TestReportExitCodes(t *testing.T) { func TestRunRequiresFlags(t *testing.T) { t.Setenv("DATABASE_URL", "") + unsetEnv(t, "AUDIT_SINK") var stderr bytes.Buffer - require.Equal(t, 1, run([]string{"-database-url", "postgres://x"}, &bytes.Buffer{}, &stderr)) + require.Equal(t, 1, Run([]string{"-database-url", "postgres://x"}, &bytes.Buffer{}, &stderr)) require.Contains(t, stderr.String(), "-catalog is required") stderr.Reset() - require.Equal(t, 1, run([]string{"-catalog", "roles.json"}, &bytes.Buffer{}, &stderr)) - require.Contains(t, stderr.String(), "-database-url") + require.Equal(t, 1, Run([]string{"-catalog", "roles.json"}, &bytes.Buffer{}, &stderr)) + require.Contains(t, stderr.String(), "AUDIT_SINK is not set", "without an explicit URL the deployed connection is selected after checking the sink") stderr.Reset() - require.Equal(t, 2, run([]string{"-nonexistent-flag"}, &bytes.Buffer{}, &stderr)) + require.Equal(t, 2, Run([]string{"-nonexistent-flag"}, &bytes.Buffer{}, &stderr)) require.True(t, strings.Contains(stderr.String(), "flag provided but not defined") || stderr.Len() > 0) } +func TestRunSelectsDeployedCredentialsUnlessAnExplicitURLWasSupplied(t *testing.T) { + catalog := writeCatalog(t) + t.Setenv("AUDIT_SINK", "postgres") + for _, tc := range []struct { + name, environment, flag, wantURL, mode string + }{ + {name: "deployed capabilities"}, + {name: "warehouse mode with deployed capabilities", mode: "bigquery"}, + {name: "environment operator URL", environment: "postgres://operator@example.invalid/db", wantURL: "postgres://operator@example.invalid/db"}, + {name: "flag wins over environment", environment: "postgres://ignored@example.invalid/db", flag: "postgres://selected@example.invalid/db", wantURL: "postgres://selected@example.invalid/db"}, + } { + t.Run(tc.name, func(t *testing.T) { + t.Setenv("DATABASE_URL", tc.environment) + mode := tc.mode + if mode == "" { + mode = "postgres" + } + t.Setenv("AUDIT_SINK", mode) + deployed, explicit := 0, 0 + var selectedURL string + openers := storeOpeners{ + deployed: func(context.Context) (*infra.PostgresStore, error) { + deployed++ + return nil, errors.New("stop before database connection") + }, + explicit: func(_ context.Context, url string) (*infra.PostgresStore, error) { + explicit++ + selectedURL = url + return nil, errors.New("stop before database connection") + }, + } + args := []string{"-catalog", catalog} + if tc.flag != "" { + args = append(args, "-database-url", tc.flag) + } + var stderr bytes.Buffer + require.Equal(t, 1, runWithOpeners(args, &bytes.Buffer{}, &stderr, openers)) + require.Contains(t, stderr.String(), "stop before database connection") + if tc.wantURL == "" { + require.Equal(t, 1, deployed) + require.Zero(t, explicit) + } else { + require.Zero(t, deployed) + require.Equal(t, 1, explicit) + require.Equal(t, tc.wantURL, selectedURL) + } + }) + } +} + +func TestRunRefusesABlankExplicitURLInsteadOfSwitchingToManaged(t *testing.T) { + t.Setenv("DATABASE_URL", "postgres://operator@example.invalid/db") + t.Setenv("AUDIT_SINK", "postgres") + var stderr bytes.Buffer + require.Equal(t, 1, Run([]string{"-catalog", writeCatalog(t), "-database-url", " "}, &bytes.Buffer{}, &stderr)) + require.Contains(t, stderr.String(), "an explicit -database-url must not be blank") +} + +func TestRunChecksAuditModeBeforeResolvingManagedCredentials(t *testing.T) { + t.Setenv("DATABASE_URL", "") + catalog := writeCatalog(t) + openers := storeOpeners{ + deployed: func(context.Context) (*infra.PostgresStore, error) { + t.Fatal("must not resolve credentials without an explicit audit mode") + return nil, nil + }, + explicit: func(context.Context, string) (*infra.PostgresStore, error) { + t.Fatal("must not connect without an explicit audit mode") + return nil, nil + }, + } + for _, mode := range []string{"", " ", "unsupported"} { + t.Setenv("AUDIT_SINK", mode) + var stderr bytes.Buffer + require.Equal(t, 1, runWithOpeners([]string{"-catalog", catalog}, &bytes.Buffer{}, &stderr, openers)) + require.Contains(t, stderr.String(), "AUDIT_SINK") + } +} + // unsetEnv removes name for the length of the test. func unsetEnv(t *testing.T, name string) { t.Helper() @@ -107,17 +189,17 @@ func TestRunRefusesAnAuditSinkThatIsNotSet(t *testing.T) { unsetEnv(t, "AUDIT_SINK") var stdout, stderr bytes.Buffer - require.Equal(t, 1, run([]string{"-catalog", catalog, "-database-url", database}, &stdout, &stderr)) + require.Equal(t, 1, Run([]string{"-catalog", catalog, "-database-url", database}, &stdout, &stderr)) require.Contains(t, stderr.String(), "AUDIT_SINK is not set") t.Setenv("AUDIT_SINK", " ") stderr.Reset() - require.Equal(t, 1, run([]string{"-catalog", catalog, "-database-url", database}, &stdout, &stderr)) + require.Equal(t, 1, Run([]string{"-catalog", catalog, "-database-url", database}, &stdout, &stderr)) require.Contains(t, stderr.String(), "AUDIT_SINK is not set", "a blank value is as good as none") t.Setenv("AUDIT_SINK", "kafka") stderr.Reset() - require.Equal(t, 1, run([]string{"-catalog", catalog, "-database-url", database}, &stdout, &stderr)) + require.Equal(t, 1, Run([]string{"-catalog", catalog, "-database-url", database}, &stdout, &stderr)) require.Contains(t, stderr.String(), "AUDIT_SINK must be", "a set value is read by the service's own rules") } @@ -128,15 +210,15 @@ func TestRunTakesTheAuditSinkFromTheEnvironmentOrTheFlag(t *testing.T) { t.Setenv("AUDIT_SINK", "bigquery") var stdout, stderr bytes.Buffer - require.Equal(t, 1, run([]string{"-catalog", catalog, "-database-url", database}, &stdout, &stderr)) + require.Equal(t, 1, Run([]string{"-catalog", catalog, "-database-url", database}, &stdout, &stderr)) require.NotContains(t, stderr.String(), "AUDIT_SINK", "a sink in the environment gets past the check, to the database") unsetEnv(t, "AUDIT_SINK") stderr.Reset() - require.Equal(t, 1, run([]string{"-catalog", catalog, "-database-url", database, "-audit-sink", "postgres"}, &stdout, &stderr)) + require.Equal(t, 1, Run([]string{"-catalog", catalog, "-database-url", database, "-audit-sink", "postgres"}, &stdout, &stderr)) require.NotContains(t, stderr.String(), "AUDIT_SINK", "and so does the flag") stderr.Reset() - require.Equal(t, 1, run([]string{"-catalog", catalog, "-database-url", database, "-audit-sink", ""}, &stdout, &stderr)) + require.Equal(t, 1, Run([]string{"-catalog", catalog, "-database-url", database, "-audit-sink", ""}, &stdout, &stderr)) require.Contains(t, stderr.String(), "AUDIT_SINK is not set", "a flag given empty is no sink either") } From 55699c6b35d7d77737bba8591b47d9892f74585c Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Wed, 7 Oct 2026 15:04:11 -0400 Subject: [PATCH 60/69] fix(deployment): derive consumer reach and match immutable bootstrap jobs --- gitops.go | 9 +- gitops_test.go | 3 - go.mod | 2 +- go.sum | 4 +- module/DEPLOYMENT_TOPOLOGY.md | 25 ++++++ module/deployment/README.md | 45 +++++++--- module/module.codefly.yaml | 4 - module/services/accounts/code/go.mod | 2 +- module/services/accounts/code/go.sum | 4 +- .../code/pkg/cataloggen/deployment_model.go | 63 ++++++++++---- .../pkg/cataloggen/deployment_topology.go | 22 ++--- .../cataloggen/deployment_topology_test.go | 15 ++-- .../cataloggen/endpoint_declarations_test.go | 60 ++++++++++++++ .../pkg/cataloggen/gateway_routes_test.go | 4 +- .../testdata/network-policy.golden.yaml | 2 - module/services/accounts/service.codefly.yaml | 2 - .../auth-gateway/service.codefly.yaml | 2 - module/services/frontend/service.codefly.yaml | 1 + .../services/marketing/service.codefly.yaml | 1 + module/tools/go.mod | 2 +- module/tools/go.sum | 4 +- topology_declarations_test.go | 56 +++++++++++++ topology_manifests.go | 82 +++++++++++++------ topology_ports_test.go | 2 - topology_visibility_test.go | 24 +++--- vault_binding_test.go | 5 +- 26 files changed, 329 insertions(+), 116 deletions(-) create mode 100644 module/services/accounts/code/pkg/cataloggen/endpoint_declarations_test.go create mode 100644 topology_declarations_test.go diff --git a/gitops.go b/gitops.go index cd1eff142..a2b46650f 100644 --- a/gitops.go +++ b/gitops.go @@ -252,10 +252,9 @@ type topologyModule struct { } type topologyInterface struct { - Service string `yaml:"service"` - Endpoint string `yaml:"endpoint"` - Visibility string `yaml:"visibility"` - AllowModules []string `yaml:"allow-modules,omitempty"` + Service string `yaml:"service"` + Endpoint string `yaml:"endpoint"` + Visibility string `yaml:"visibility"` } type topologyService struct { @@ -2282,7 +2281,6 @@ func bootstrapJobIngressPolicy(namespace string, labels map[string]string, servi "ingress": []any{map[string]any{ "from": []any{map[string]any{"podSelector": map[string]any{"matchLabels": map[string]string{ "codefly.dev/bootstrap-service": service, - "job-name": service, }}}}, "ports": networkPorts(ports), }}, @@ -2298,7 +2296,6 @@ func bootstrapJobEgressPolicy(namespace string, labels map[string]string, servic Spec: map[string]any{ "podSelector": map[string]any{"matchLabels": map[string]string{ "codefly.dev/bootstrap-service": service, - "job-name": service, }}, "policyTypes": []string{"Egress"}, "egress": []any{map[string]any{ diff --git a/gitops_test.go b/gitops_test.go index 4335d9580..81046bb1f 100644 --- a/gitops_test.go +++ b/gitops_test.go @@ -2390,9 +2390,6 @@ func serviceManifestFixture(name, endpoint, api, visibility string, port int, to } if visibility != "private" { manifest.WriteString(" visibility: " + visibility + "\n") - if visibility == "internal" { - manifest.WriteString(" allow-modules: [\"*\"]\n") - } } manifest.WriteString("spec:\n") manifest.WriteString(spec) diff --git a/go.mod b/go.mod index 12e08b125..59c647c80 100644 --- a/go.mod +++ b/go.mod @@ -3,7 +3,7 @@ module github.com/codefly-dev/agents/modules/saas-starter go 1.27.0 require ( - github.com/codefly-dev/core v0.13.0 + github.com/codefly-dev/core v0.14.0 gopkg.in/yaml.v3 v3.0.1 ) diff --git a/go.sum b/go.sum index d4703ab21..5af8beecf 100644 --- a/go.sum +++ b/go.sum @@ -28,8 +28,8 @@ github.com/cheggaaa/pb/v3 v3.2.1 h1:aprZbFRG+B7+ug76S8QZ6Y1PW168UHzOmbC3wa+aU6I= github.com/cheggaaa/pb/v3 v3.2.1/go.mod h1:U9hSVxoKqJrZIE3PkFG1xXXNaK/Ilzg+EF/scpXzEdw= github.com/clipperhouse/uax29/v2 v2.7.0 h1:+gs4oBZ2gPfVrKPthwbMzWZDaAFPGYK72F0NJv2v7Vk= github.com/clipperhouse/uax29/v2 v2.7.0/go.mod h1:EFJ2TJMRUaplDxHKj1qAEhCtQPW2tJSwu5BF98AuoVM= -github.com/codefly-dev/core v0.13.0 h1:FiphttBZRrKXEWoCZD7JpUB6oVuBCoqtAc06QeNdYWY= -github.com/codefly-dev/core v0.13.0/go.mod h1:rp4YZkCJEOufliWQAgK3sv8YlO7Xcva7+cN7hIlAR0I= +github.com/codefly-dev/core v0.14.0 h1:rpLFxeHSf2PrbxsRSp/NcZt8htMRM38JV2Oc8RKievU= +github.com/codefly-dev/core v0.14.0/go.mod h1:rp4YZkCJEOufliWQAgK3sv8YlO7Xcva7+cN7hIlAR0I= github.com/codefly-dev/gortk v0.2.0 h1:7bOlS5valYz2zil+fZctQNcPCYBcPj86abcw9N8h1hQ= github.com/codefly-dev/gortk v0.2.0/go.mod h1:dDWUMFgAP063OCGhTEpV7FFUj9+zTcxxO/nV80VKEwg= github.com/containerd/errdefs v1.0.0 h1:tg5yIfIlQIrxYtu9ajqY42W3lpS19XqdxRQeEwYG8PI= diff --git a/module/DEPLOYMENT_TOPOLOGY.md b/module/DEPLOYMENT_TOPOLOGY.md index 569713648..657a2ada3 100644 --- a/module/DEPLOYMENT_TOPOLOGY.md +++ b/module/DEPLOYMENT_TOPOLOGY.md @@ -49,6 +49,26 @@ module-visible without a public ingress route. ## Network-policy model +Authored endpoints and module-interface exports declare `visibility: internal` +without `allow-modules`. Core derives the modules that can reach an internal +endpoint from the consumers' declared runtime dependencies. A target never +lists its consumers. Interface visibility defaults to internal; a service +endpoint still defaults to private. The catalog retains its `MODULE` enum for +internal reach, while network and mesh policies admit only declared dependency +edges. Public frontend and marketing endpoints explicitly declare +`exposure: public`, preserving their existing public addresses. + +Promotion requires a released CLI with Core's consumer-derived allow-list +support and the audit promotion driver from +[CLI PR 919](https://github.com/codefly-dev/cli/pull/919). Merge the CLI change, +release that compatible CLI, then promote this kit cutover. The module agent, +accounts catalog generator, and module tools require stable Core 0.14 or later, +which accepts these declarations and derives consumer access. Core 0.13 refuses +internal interface exports without an authored list. The CI pin at CLI 0.1.171 +still loads the module with its older Core 0.7, but does not derive cross-module +access: an internal endpoint with no authored list denies other modules. It is +not a compatible promotion driver for this declaration model. + The topology-policy golden contains 18 `NetworkPolicy` resources: - one namespace-wide ingress/egress default deny; @@ -61,6 +81,11 @@ There is no `allow-intra-namespace` rule. Adding a service dependency or port requires changing the topology binding and reviewing both generated directions of the edge. +Store bootstrap ingress and egress select the agent's stable +`codefly.dev/bootstrap-service: store` pod label. Postgres names the immutable +migration Job with a digest of its rendered pod template, so its Kubernetes +`job-name` changes with the image or inputs and cannot be a fixed selector. + Pod selectors use the `app: ` labels emitted by the pinned Codefly agents. Services whose agent uses a different Kubernetes identity declare its Service name and app label in the topology. Endpoint ports are explicit diff --git a/module/deployment/README.md b/module/deployment/README.md index 6d2872240..819110ca4 100644 --- a/module/deployment/README.md +++ b/module/deployment/README.md @@ -68,15 +68,13 @@ transport configuration). ## Deploy Jobs A `deploy_jobs` topology entry is a required store-writing one-shot for a -job-capable promotion driver. **Automatic catalog promotion remains blocked:** -the Codefly CLI currently has no deploy-job execution or dependency migration -barrier. It must reject this bundle rather than silently omit the import. Service -environment defaults alone do not provide job execution. Driver support must -mount the catalog, use the running service's immutable image and identity, -resolve the write credential (including managed IAM), inherit the effective -service environment, and fail promotion unless each `after` dependency's -migration and the import complete. Sync-wave annotations on independently -reconciling Argo Applications do not enforce that dependency barrier. +job-capable promotion driver. The Codefly GitOps driver projects these entries +as Sync-hook Jobs inside one Application with the module's workloads. Its +resource waves order dependency readiness, migration Jobs, the import and then +the dependent workload. A failed import blocks rollout. This requires a CLI +release containing deploy-job support; a driver lacking it must refuse the +required step rather than silently omit it. Sync-wave annotations on independent +Argo Applications do not enforce that barrier. Unlike a service's self-serving `bootstrap_job_endpoints` Job — which reaches only the service's own endpoints — a deploy Job runs one service's @@ -97,8 +95,14 @@ CLI-owned `spec.environment-defaults` declares `AUDIT_SINK: postgres`, so the service has an explicit value to inherit after deployment projection. The environment's `service-config` or explicit secret reference overrides that default; a warehouse deployment's Job must inherit its resolved warehouse mode. -This requires a CLI release that projects service environment defaults and, -for automatic promotion, implements the job contract above. Re-running an +This requires a CLI release that projects service environment defaults and +the job contract above. The driver preserves resolved credentials, their +refresh containers and proxy mounts; the importer initializes Codefly runtime +configuration and opens the same scoped store capabilities as accounts. The +catalog is mounted from an immutable ConfigMap named by its content digest. +The Job retains the accounts image's entrypoint and passes +`role-catalog-import -catalog -force`. Explicit operator +`-database-url` use remains available. Re-running an unchanged catalog is an empty no-op, so the step is idempotent. Generation rejects a deploy Job whose catalog artifact is absent, whose write target is not a declared dependency of the @@ -109,6 +113,25 @@ yet exist. In an environment where the target is a managed handoff the driver owns the out-of-cluster reach, so no in-cluster reach policy is rendered — mirroring the store's own bootstrap authority. +The supported target is Kubernetes 1.33 or newer with native sidecars and an +Istio injector supporting `sidecar.istio.io/nativeSidecar`. The driver converts +ordinary proxy containers into restartable init containers and requests native +mesh injection, so their lifetime cannot hold a completed import open. Pod +policy and identity labels remain intact; a driver-owned workload-role label +excludes import pods from service endpoint selection. Argo observes Sync-hook +completion before cleanup and recreates the idempotent jobs at the next sync. +An `after` managed service without a rendered migration Job is refused. + +The aggregate Application boundary supports new installations and upgrades +already using that boundary. Existing deployments using separate unit +Applications require a governed ownership migration: preserve resources while +retiring the old owners without cascading deletion, prevent competing +reconciliation, and verify the aggregate owner has adopted resource tracking. +The CLI refuses publication across that boundary and does not yet execute or +admit the migration. Do not remove finalizers or rewrite inventory to bypass +the guard; an existing cell needs the supported migration prerequisite before +promotion with this bundle. + The bundle is transport-neutral, but the driver honours a small contract the generated reach policies assume. The Job pod runs under the **running service's** ServiceAccount (`service`, here `accounts`): the mesh already authorises that diff --git a/module/module.codefly.yaml b/module/module.codefly.yaml index 3111b5282..03d59ebcc 100644 --- a/module/module.codefly.yaml +++ b/module/module.codefly.yaml @@ -7,19 +7,15 @@ interface: - service: accounts endpoint: authority visibility: internal - allow-modules: ["*"] - service: accounts endpoint: connect visibility: internal - allow-modules: ["*"] - service: auth-gateway endpoint: grpc visibility: internal - allow-modules: ["*"] - service: auth-gateway endpoint: rest visibility: internal - allow-modules: ["*"] - service: frontend endpoint: http visibility: public diff --git a/module/services/accounts/code/go.mod b/module/services/accounts/code/go.mod index ca27b75a7..f08a92a02 100644 --- a/module/services/accounts/code/go.mod +++ b/module/services/accounts/code/go.mod @@ -16,7 +16,7 @@ require ( github.com/aws/aws-sdk-go-v2/credentials v1.20.6 github.com/aws/aws-sdk-go-v2/service/s3 v1.114.0 github.com/codefly-dev/cli/contracts/deployment v0.0.0-20261006151622-5824cf5342fc - github.com/codefly-dev/core v0.13.0 + github.com/codefly-dev/core v0.14.0 github.com/codefly-dev/interface-cache/go/cache v0.2.2-0.20260926205419-327a6e1d9aba github.com/codefly-dev/sdk-go v0.3.1 github.com/codefly-dev/sdk-go/workcontext v0.0.0-20260926212413-b91ce91eb23e diff --git a/module/services/accounts/code/go.sum b/module/services/accounts/code/go.sum index 19919c27a..36bfd99a6 100644 --- a/module/services/accounts/code/go.sum +++ b/module/services/accounts/code/go.sum @@ -136,8 +136,8 @@ github.com/cncf/xds/go v0.0.0-20260202195803-dba9d589def2 h1:aBangftG7EVZoUb69Os github.com/cncf/xds/go v0.0.0-20260202195803-dba9d589def2/go.mod h1:qwXFYgsP6T7XnJtbKlf1HP8AjxZZyzxMmc+Lq5GjlU4= github.com/codefly-dev/cli/contracts/deployment v0.0.0-20261006151622-5824cf5342fc h1:olDnVJ1A1Retn0z9A59YMwDRmDLgkV2y1lbl+HzNrj8= github.com/codefly-dev/cli/contracts/deployment v0.0.0-20261006151622-5824cf5342fc/go.mod h1:VWPYY3FH4te7SzdN2xl9odNpWidJ1Grl/Hjs1SNoEF8= -github.com/codefly-dev/core v0.13.0 h1:FiphttBZRrKXEWoCZD7JpUB6oVuBCoqtAc06QeNdYWY= -github.com/codefly-dev/core v0.13.0/go.mod h1:rp4YZkCJEOufliWQAgK3sv8YlO7Xcva7+cN7hIlAR0I= +github.com/codefly-dev/core v0.14.0 h1:rpLFxeHSf2PrbxsRSp/NcZt8htMRM38JV2Oc8RKievU= +github.com/codefly-dev/core v0.14.0/go.mod h1:rp4YZkCJEOufliWQAgK3sv8YlO7Xcva7+cN7hIlAR0I= github.com/codefly-dev/gortk v0.2.0 h1:7bOlS5valYz2zil+fZctQNcPCYBcPj86abcw9N8h1hQ= github.com/codefly-dev/gortk v0.2.0/go.mod h1:dDWUMFgAP063OCGhTEpV7FFUj9+zTcxxO/nV80VKEwg= github.com/codefly-dev/interface-cache/go/cache v0.2.2-0.20260926205419-327a6e1d9aba h1:6nFaDYsLyA5x3WiGAk6pMI3c0naqxiPeaTHMqRPvbVs= diff --git a/module/services/accounts/code/pkg/cataloggen/deployment_model.go b/module/services/accounts/code/pkg/cataloggen/deployment_model.go index f5e6c4f5f..14677bd34 100644 --- a/module/services/accounts/code/pkg/cataloggen/deployment_model.go +++ b/module/services/accounts/code/pkg/cataloggen/deployment_model.go @@ -8,6 +8,7 @@ import ( "os" "path/filepath" "sort" + "strings" "github.com/codefly-dev/core/standards" "gopkg.in/yaml.v3" @@ -145,11 +146,11 @@ func assembleDeploymentBindings(documents DeploymentDocuments) (deploymentBindin } for i := range bindings.Interface { exported := &bindings.Interface[i] - visibility, err := manifestCatalogVisibility(exported.Visibility, "", exported.AllowModules) + visibility, err := manifestCatalogVisibility(exported.Visibility, "", true) if err != nil { return deploymentBindings{}, fmt.Errorf("interface %s/%s: %w", exported.Service, exported.Endpoint, err) } - exported.Visibility, exported.AllowModules = visibility, nil + exported.Visibility = visibility } declared := make(map[string]bool, len(module.Services)) for _, reference := range module.Services { @@ -225,7 +226,7 @@ func assembleServiceBinding(moduleName, name string, document []byte) (deploymen if api == "" { api = endpoint.Name } - visibility, err := manifestCatalogVisibility(endpoint.Visibility, endpoint.Location, endpoint.AllowModules) + visibility, err := manifestCatalogVisibility(endpoint.Visibility, endpoint.Location, false) if err != nil { return deploymentServiceBinding{}, fmt.Errorf("service %q endpoint %q: %w", name, endpoint.Name, err) } @@ -270,29 +271,26 @@ func assembleServiceBinding(moduleName, name string, document []byte) (deploymen return service, nil } -// manifestCatalogVisibility projects Codefly's independent visibility/location -// axes into this catalog's categories. MODULE represents every module, so -// narrower allow-lists refuse rather than silently widening the generated policy. -func manifestCatalogVisibility(visibility, location string, allowed []string) (string, error) { +// manifestCatalogVisibility projects the authored reach into the catalog's legacy MODULE +// category. It does not grant callers: only declared dependency edges render +// network access, and the composing Core derives cross-module reach from asks. +func manifestCatalogVisibility(visibility, location string, exported bool) (string, error) { if visibility == "" { visibility = "private" + if exported { + visibility = "internal" + } } if location != "" { - if location != "external" || visibility != "private" || len(allowed) != 0 { + if location != "external" || visibility != "private" { return "", fmt.Errorf("unsupported external endpoint policy") } return "external", nil } switch visibility { case "private", "public": - if len(allowed) != 0 { - return "", fmt.Errorf("allow-modules requires internal visibility") - } return visibility, nil case "internal": - if len(allowed) != 1 || allowed[0] != "*" { - return "", fmt.Errorf("deployment catalog requires internal visibility with allow-modules [*]; narrower exports need an allow-list-aware catalog") - } return "module", nil default: return "", fmt.Errorf("unsupported authored endpoint visibility %q", visibility) @@ -352,3 +350,40 @@ func standardAllocation(name, api string, endpoints []manifestEndpoint) (uint32, } return port, true } + +func (endpoint *manifestEndpoint) UnmarshalYAML(node *yaml.Node) error { + if err := refuseAuthoredConsumers(node); err != nil { + return err + } + type plain manifestEndpoint + return node.Decode((*plain)(endpoint)) +} + +func (endpoint *deploymentInterfaceBinding) UnmarshalYAML(node *yaml.Node) error { + if err := refuseAuthoredConsumers(node); err != nil { + return err + } + type plain deploymentInterfaceBinding + return node.Decode((*plain)(endpoint)) +} + +// Judge key presence before decoding: null and alternate spellings cannot +// silently erase a forbidden grant. This matches Core's declaration boundary. +func refuseAuthoredConsumers(node *yaml.Node) error { + if node.Kind != yaml.MappingNode { + return nil + } + for i := 0; i+1 < len(node.Content); i += 2 { + key := node.Content[i].Value + var canonical strings.Builder + for _, r := range strings.ToLower(key) { + if r >= 'a' && r <= 'z' { + canonical.WriteRune(r) + } + } + if canonical.String() == "allowmodules" { + return fmt.Errorf("authors %q: endpoint access is derived from consumers' declared dependencies", key) + } + } + return nil +} diff --git a/module/services/accounts/code/pkg/cataloggen/deployment_topology.go b/module/services/accounts/code/pkg/cataloggen/deployment_topology.go index c1554e9a0..6e0a28c49 100644 --- a/module/services/accounts/code/pkg/cataloggen/deployment_topology.go +++ b/module/services/accounts/code/pkg/cataloggen/deployment_topology.go @@ -70,10 +70,9 @@ type deploymentModuleBinding struct { } type deploymentInterfaceBinding struct { - Service string `yaml:"service"` - Endpoint string `yaml:"endpoint"` - Visibility string `yaml:"visibility"` - AllowModules []string `yaml:"allow-modules,omitempty"` + Service string `yaml:"service"` + Endpoint string `yaml:"endpoint"` + Visibility string `yaml:"visibility"` } type deploymentServiceBinding struct { @@ -734,11 +733,10 @@ type manifestEndpointReference struct { } type manifestEndpoint struct { - Name string `yaml:"name"` - Visibility string `yaml:"visibility,omitempty"` - API string `yaml:"api,omitempty"` - AllowModules []string `yaml:"allow-modules,omitempty"` - Location string `yaml:"location,omitempty"` + Name string `yaml:"name"` + Visibility string `yaml:"visibility,omitempty"` + API string `yaml:"api,omitempty"` + Location string `yaml:"location,omitempty"` } func marshalGeneratedYAML(source string, value any) ([]byte, error) { @@ -1351,9 +1349,8 @@ spec: - podSelector: matchLabels: codefly.dev/bootstrap-service: %s - job-name: %s ports: -`, service, namespace, serviceApp, service, service) +`, service, namespace, serviceApp, service) writeNetworkPorts(source, ports, 8) fmt.Fprintf(source, `--- apiVersion: networking.k8s.io/v1 @@ -1365,7 +1362,6 @@ spec: podSelector: matchLabels: codefly.dev/bootstrap-service: %s - job-name: %s policyTypes: - Egress egress: @@ -1374,7 +1370,7 @@ spec: matchLabels: app: %s ports: -`, service, service, namespace, service, service, serviceApp) +`, service, service, namespace, service, serviceApp) writeNetworkPorts(source, ports, 8) } diff --git a/module/services/accounts/code/pkg/cataloggen/deployment_topology_test.go b/module/services/accounts/code/pkg/cataloggen/deployment_topology_test.go index b503b44a8..769d15ec9 100644 --- a/module/services/accounts/code/pkg/cataloggen/deployment_topology_test.go +++ b/module/services/accounts/code/pkg/cataloggen/deployment_topology_test.go @@ -150,31 +150,30 @@ func TestDeploymentTopologyIsDeterministicAndCurrent(t *testing.T) { require.NotContains(t, string(first.NetworkPolicy), "name: allow-auth-gateway-public-egress") require.NotContains(t, string(first.NetworkPolicy), "name: allow-istio-ingress-to-auth-gateway") require.Equal(t, 2, strings.Count(string(first.NetworkPolicy), "codefly.dev/bootstrap-service: store")) + require.NotContains(t, string(first.NetworkPolicy), "job-name:", "bootstrap Job names contain a content digest; policies select the stable service label") require.Contains(t, string(first.NetworkPolicy), "192.175.48.0/24") require.Contains(t, string(first.NetworkPolicy), "64:ff9b::/96") } -// MODULE is the catalog category for every composed module, not a license to -// erase a narrower authored allow-list. Legacy Codefly spellings also refuse. +// Legacy reach spellings and authored consumer lists refuse. MODULE remains +// a catalog category; declared dependencies determine which callers reach it. func TestDeploymentTopologyRefusesUnrepresentableEndpointPolicies(t *testing.T) { catalog := readFixture(t, "../../../generated/service-catalog.json") documents := readDeploymentDocuments(t) for _, policy := range []string{ "visibility: module", "visibility: internal\n allow-modules: [example]", - "visibility: internal", "visibility: public\n allow-modules: [example]", } { - changed := withModule(t, documents, "visibility: internal\n allow-modules: [\"*\"]", policy) + changed := withModule(t, documents, "visibility: internal", policy) _, err := cataloggen.BuildDeploymentArtifacts(catalog, changed) require.Error(t, err, policy) } for _, policy := range []string{ "visibility: module", "visibility: internal\n allow-modules: [example]", - "visibility: internal", } { - changed := withService(t, documents, "accounts", "visibility: internal\n allow-modules: [\"*\"]", policy) + changed := withService(t, documents, "accounts", "visibility: internal", policy) _, err := cataloggen.BuildDeploymentArtifacts(catalog, changed) require.Error(t, err, policy) } @@ -786,10 +785,10 @@ func TestDeploymentTopologyRequiresTheModuleAuthorityEndpoint(t *testing.T) { documents := readDeploymentDocuments(t) _, err := cataloggen.BuildDeploymentArtifacts(serviceCatalog, - withModule(t, documents, " - service: accounts\n endpoint: authority\n visibility: internal\n allow-modules: [\"*\"]\n", "")) + withModule(t, documents, " - service: accounts\n endpoint: authority\n visibility: internal\n", "")) require.ErrorContains(t, err, "module interface must export accounts/authority") _, err = cataloggen.BuildDeploymentArtifacts(serviceCatalog, - withService(t, documents, "accounts", " - name: authority\n api: grpc\n visibility: internal\n allow-modules: [\"*\"]\n", " - name: authority\n api: rest\n visibility: internal\n allow-modules: [\"*\"]\n")) + withService(t, documents, "accounts", " - name: authority\n api: grpc\n visibility: internal\n", " - name: authority\n api: rest\n visibility: internal\n")) require.ErrorContains(t, err, "must be a gRPC endpoint at module visibility") } diff --git a/module/services/accounts/code/pkg/cataloggen/endpoint_declarations_test.go b/module/services/accounts/code/pkg/cataloggen/endpoint_declarations_test.go new file mode 100644 index 000000000..46eb3ebb2 --- /dev/null +++ b/module/services/accounts/code/pkg/cataloggen/endpoint_declarations_test.go @@ -0,0 +1,60 @@ +package cataloggen_test + +import ( + "strings" + "testing" + + "accounts/pkg/cataloggen" + + "github.com/stretchr/testify/require" +) + +func withoutAuthoredAllowModules(documents cataloggen.DeploymentDocuments) cataloggen.DeploymentDocuments { + strip := func(document []byte) []byte { + var lines []string + for _, line := range strings.Split(string(document), "\n") { + if !strings.HasPrefix(strings.TrimSpace(line), "allow-modules:") { + lines = append(lines, line) + } + } + return []byte(strings.Join(lines, "\n")) + } + result := cataloggen.DeploymentDocuments{Module: strip(documents.Module), Services: map[string][]byte{}, Jobs: documents.Jobs} + for name, document := range documents.Services { + result.Services[name] = strip(document) + } + return result +} + +func TestDeploymentTopologyAcceptsConsumerDerivedInternalReach(t *testing.T) { + documents := withoutAuthoredAllowModules(readDeploymentDocuments(t)) + artifacts, err := cataloggen.BuildDeploymentArtifacts(readFixture(t, "../../../generated/service-catalog.json"), documents) + require.NoError(t, err) + // The reach category does not grant callers. The policies are still the + // exact dependency edges, and internal RPCs still require caller identity. + require.Equal(t, string(readFixture(t, "testdata/network-policy.golden.yaml")), string(artifacts.NetworkPolicy)) + require.Equal(t, string(readFixture(t, "testdata/mesh-policy.golden.yaml")), string(artifacts.MeshPolicy)) + // Core defaults an interface export to internal, while a service endpoint + // still defaults to private. Omitting the interface reach keeps its policy. + defaultExport := withModule(t, documents, " visibility: internal\n", "") + defaults, err := cataloggen.BuildDeploymentArtifacts(readFixture(t, "../../../generated/service-catalog.json"), defaultExport) + require.NoError(t, err) + require.Equal(t, string(artifacts.CatalogJSON), string(defaults.CatalogJSON)) +} + +func TestDeploymentTopologyRefusesAuthoredConsumersByKeyPresence(t *testing.T) { + documents := withoutAuthoredAllowModules(readDeploymentDocuments(t)) + catalog := readFixture(t, "../../../generated/service-catalog.json") + for _, key := range []string{"allow-modules", "allow_modules", "allowModules", "Allow Modules"} { + for _, value := range []string{"[\"*\"]", "[example]", "[]", "null"} { + t.Run(key+"/"+value, func(t *testing.T) { + _, err := cataloggen.BuildDeploymentArtifacts(catalog, withModule(t, documents, + " visibility: internal\n", " visibility: internal\n "+key+": "+value+"\n")) + require.ErrorContains(t, err, "derived from consumers") + _, err = cataloggen.BuildDeploymentArtifacts(catalog, withService(t, documents, "accounts", + " visibility: internal\n", " visibility: internal\n "+key+": "+value+"\n")) + require.ErrorContains(t, err, "derived from consumers") + }) + } + } +} diff --git a/module/services/accounts/code/pkg/cataloggen/gateway_routes_test.go b/module/services/accounts/code/pkg/cataloggen/gateway_routes_test.go index ea4b12c1c..ebd2ac49c 100644 --- a/module/services/accounts/code/pkg/cataloggen/gateway_routes_test.go +++ b/module/services/accounts/code/pkg/cataloggen/gateway_routes_test.go @@ -50,7 +50,7 @@ func TestGatewayRouteCatalogCompilationAndParity(t *testing.T) { require.Equal(t, "accounts", legacy.GetOwner().GetService()) require.Equal(t, "connect", legacy.GetUpstreamEndpoint()) - renamed := withService(t, topologyDocuments, "accounts", " - name: connect\n visibility: internal\n allow-modules: [\"*\"]\n", " - name: connect-api\n api: connect\n visibility: internal\n allow-modules: [\"*\"]\n") + renamed := withService(t, topologyDocuments, "accounts", " - name: connect\n visibility: internal\n", " - name: connect-api\n api: connect\n visibility: internal\n") renamed = withService(t, renamed, "auth-gateway", " - name: connect\n", " - name: connect-api\n") renamed = withModule(t, renamed, " endpoint: connect\n", " endpoint: connect-api\n") renamedRoutes, err := cataloggen.BuildGatewayRouteCatalog(serviceDocument, bindingDocument, renamed) @@ -101,7 +101,7 @@ func TestGatewayRouteValidationRejectsUnsafeDrift(t *testing.T) { wrongSchema.SchemaVersion = "saas.gateway.routes.v2" require.ErrorContains(t, cataloggen.ValidateGatewayRouteCatalog(wrongSchema), "unsupported") - badEndpoint := withService(t, topologyDocuments, "accounts", " - name: connect\n visibility: internal\n allow-modules: [\"*\"]\n", " - name: connect\n api: connect;raw\n visibility: internal\n allow-modules: [\"*\"]\n") + badEndpoint := withService(t, topologyDocuments, "accounts", " - name: connect\n visibility: internal\n", " - name: connect\n api: connect;raw\n visibility: internal\n") _, err = cataloggen.BuildGatewayRouteCatalog(serviceDocument, bindingDocument, badEndpoint) require.ErrorContains(t, err, "Codefly API") diff --git a/module/services/accounts/code/pkg/cataloggen/testdata/network-policy.golden.yaml b/module/services/accounts/code/pkg/cataloggen/testdata/network-policy.golden.yaml index da814e95a..dba58982f 100644 --- a/module/services/accounts/code/pkg/cataloggen/testdata/network-policy.golden.yaml +++ b/module/services/accounts/code/pkg/cataloggen/testdata/network-policy.golden.yaml @@ -315,7 +315,6 @@ spec: - podSelector: matchLabels: codefly.dev/bootstrap-service: store - job-name: store ports: - protocol: TCP port: 5432 @@ -329,7 +328,6 @@ spec: podSelector: matchLabels: codefly.dev/bootstrap-service: store - job-name: store policyTypes: - Egress egress: diff --git a/module/services/accounts/service.codefly.yaml b/module/services/accounts/service.codefly.yaml index ae07139fc..b62fdf90a 100644 --- a/module/services/accounts/service.codefly.yaml +++ b/module/services/accounts/service.codefly.yaml @@ -45,10 +45,8 @@ endpoints: - name: authority api: grpc visibility: internal - allow-modules: ["*"] - name: connect visibility: internal - allow-modules: ["*"] - name: grpc - name: rest spec: diff --git a/module/services/auth-gateway/service.codefly.yaml b/module/services/auth-gateway/service.codefly.yaml index ebfec5fe9..e118ecac3 100644 --- a/module/services/auth-gateway/service.codefly.yaml +++ b/module/services/auth-gateway/service.codefly.yaml @@ -35,10 +35,8 @@ workspace-configuration-dependencies: endpoints: - name: grpc visibility: internal - allow-modules: ["*"] - name: rest visibility: internal - allow-modules: ["*"] spec: hot-reload: true protocol-output-dirs: diff --git a/module/services/frontend/service.codefly.yaml b/module/services/frontend/service.codefly.yaml index 433f00361..d075eb663 100644 --- a/module/services/frontend/service.codefly.yaml +++ b/module/services/frontend/service.codefly.yaml @@ -22,6 +22,7 @@ workspace-configuration-dependencies: endpoints: - name: http visibility: public + exposure: public spec: build-args: FRONTEND_SKIN_RUNTIME: "1" diff --git a/module/services/marketing/service.codefly.yaml b/module/services/marketing/service.codefly.yaml index 99269f77f..aa80e3156 100644 --- a/module/services/marketing/service.codefly.yaml +++ b/module/services/marketing/service.codefly.yaml @@ -12,6 +12,7 @@ agent: endpoints: - name: http visibility: public + exposure: public spec: # Declare the argument at its source so regeneration preserves the opt-in # production-readiness gate; an empty default does not force it off. diff --git a/module/tools/go.mod b/module/tools/go.mod index 18f97c6b8..de491138c 100644 --- a/module/tools/go.mod +++ b/module/tools/go.mod @@ -4,7 +4,7 @@ go 1.27.0 require ( github.com/Masterminds/semver/v3 v3.5.0 - github.com/codefly-dev/core v0.13.0 + github.com/codefly-dev/core v0.14.0 github.com/yoheimuta/go-protoparser/v4 v4.14.2 golang.org/x/tools v0.49.0 google.golang.org/protobuf v1.36.12 diff --git a/module/tools/go.sum b/module/tools/go.sum index 46f722672..ff83e8a89 100644 --- a/module/tools/go.sum +++ b/module/tools/go.sum @@ -28,8 +28,8 @@ github.com/cheggaaa/pb/v3 v3.2.1 h1:aprZbFRG+B7+ug76S8QZ6Y1PW168UHzOmbC3wa+aU6I= github.com/cheggaaa/pb/v3 v3.2.1/go.mod h1:U9hSVxoKqJrZIE3PkFG1xXXNaK/Ilzg+EF/scpXzEdw= github.com/clipperhouse/uax29/v2 v2.7.0 h1:+gs4oBZ2gPfVrKPthwbMzWZDaAFPGYK72F0NJv2v7Vk= github.com/clipperhouse/uax29/v2 v2.7.0/go.mod h1:EFJ2TJMRUaplDxHKj1qAEhCtQPW2tJSwu5BF98AuoVM= -github.com/codefly-dev/core v0.13.0 h1:FiphttBZRrKXEWoCZD7JpUB6oVuBCoqtAc06QeNdYWY= -github.com/codefly-dev/core v0.13.0/go.mod h1:rp4YZkCJEOufliWQAgK3sv8YlO7Xcva7+cN7hIlAR0I= +github.com/codefly-dev/core v0.14.0 h1:rpLFxeHSf2PrbxsRSp/NcZt8htMRM38JV2Oc8RKievU= +github.com/codefly-dev/core v0.14.0/go.mod h1:rp4YZkCJEOufliWQAgK3sv8YlO7Xcva7+cN7hIlAR0I= github.com/codefly-dev/gortk v0.2.0 h1:7bOlS5valYz2zil+fZctQNcPCYBcPj86abcw9N8h1hQ= github.com/codefly-dev/gortk v0.2.0/go.mod h1:dDWUMFgAP063OCGhTEpV7FFUj9+zTcxxO/nV80VKEwg= github.com/containerd/errdefs v1.0.0 h1:tg5yIfIlQIrxYtu9ajqY42W3lpS19XqdxRQeEwYG8PI= diff --git a/topology_declarations_test.go b/topology_declarations_test.go new file mode 100644 index 000000000..34f3c4ea0 --- /dev/null +++ b/topology_declarations_test.go @@ -0,0 +1,56 @@ +package main + +import ( + "strings" + "testing" + + "gopkg.in/yaml.v3" +) + +func TestTopologyConsumerDerivedInternalReach(t *testing.T) { + for _, exported := range []bool{false, true} { + visibility, err := endpointVisibility("internal", "", exported) + if err != nil || visibility != "module" { + t.Fatalf("internal exported=%v: %q, %v", exported, visibility, err) + } + } + visibility, err := endpointVisibility("", "", true) + if err != nil || visibility != "module" { + t.Fatalf("default interface reach: %q, %v", visibility, err) + } + visibility, err = endpointVisibility("", "", false) + if err != nil || visibility != "private" { + t.Fatalf("default service reach: %q, %v", visibility, err) + } +} + +func TestTopologyRefusesAuthoredConsumersByKeyPresence(t *testing.T) { + for _, key := range []string{"allow-modules", "allow_modules", "allowModules", "Allow Modules"} { + for _, value := range []string{"[\"*\"]", "[example]", "[]", "null"} { + document := []byte("visibility: internal\n" + key + ": " + value + "\n") + for _, endpoint := range []any{&topologyManifestEndpoint{}, &topologyInterface{}} { + err := yaml.Unmarshal(document, endpoint) + if err == nil || !strings.Contains(err.Error(), "derived from consumers") { + t.Fatalf("%T %s=%s: %v", endpoint, key, value, err) + } + } + } + } +} + +func TestBootstrapPoliciesSelectImmutableJobByServiceLabel(t *testing.T) { + // Postgres appends a digest to the immutable Job name. Kubernetes copies + // that name into job-name; the agent's service label remains stable. + for _, policy := range []kubeObject{ + bootstrapJobIngressPolicy("example", nil, "store", "store", []uint32{5432}), + bootstrapJobEgressPolicy("example", nil, "store", "store", []uint32{5432}), + } { + document, err := yaml.Marshal(policy) + if err != nil { + t.Fatal(err) + } + if strings.Contains(string(document), "job-name:") || !strings.Contains(string(document), "codefly.dev/bootstrap-service: store") { + t.Fatalf("policy cannot select a content-addressed bootstrap Job:\n%s", document) + } + } +} diff --git a/topology_manifests.go b/topology_manifests.go index e0f91c7d5..fbe9da781 100644 --- a/topology_manifests.go +++ b/topology_manifests.go @@ -9,6 +9,7 @@ import ( "os" "path/filepath" "sort" + "strings" basev0 "github.com/codefly-dev/core/generated/go/codefly/base/v0" "github.com/codefly-dev/core/network" @@ -54,14 +55,8 @@ type serviceTopologyManifest struct { } `yaml:"service-dependencies"` WorkspaceConfigurationDependencies []string `yaml:"workspace-configuration-dependencies"` SecretServiceConfigurations []topologySecretServiceConfiguration `yaml:"secret-service-configurations"` - Endpoints []struct { - Name string `yaml:"name"` - Visibility string `yaml:"visibility"` - API string `yaml:"api"` - AllowModules []string `yaml:"allow-modules"` - Location string `yaml:"location"` - } `yaml:"endpoints"` - Spec map[string]any `yaml:"spec"` + Endpoints []topologyManifestEndpoint `yaml:"endpoints"` + Spec map[string]any `yaml:"spec"` } // deploymentSpecManifest is the spec.deployment block of a service manifest, @@ -108,11 +103,11 @@ func assembleDeploymentTopology(moduleDir, moduleName string, services []service } for i := range topology.Interface { exported := &topology.Interface[i] - visibility, err := endpointVisibility(exported.Visibility, "", exported.AllowModules) + visibility, err := endpointVisibility(exported.Visibility, "", true) if err != nil { return deploymentTopology{}, fmt.Errorf("interface %s/%s: %w", exported.Service, exported.Endpoint, err) } - exported.Visibility, exported.AllowModules = visibility, nil + exported.Visibility = visibility } ordered := append([]serviceDefinition(nil), services...) sort.Slice(ordered, func(i, j int) bool { return ordered[i].name < ordered[j].name }) @@ -171,7 +166,7 @@ func loadServiceTopology(moduleName string, service serviceDefinition) (topology // declares none and takes the Service port. coreEndpoints := make([]*basev0.Endpoint, 0, len(manifest.Endpoints)) for _, endpoint := range manifest.Endpoints { - visibility, err := endpointVisibility(endpoint.Visibility, endpoint.Location, endpoint.AllowModules) + visibility, err := endpointVisibility(endpoint.Visibility, endpoint.Location, false) if err != nil { return topologyService{}, fmt.Errorf("service %q endpoint %q: %w", service.name, endpoint.Name, err) } @@ -186,7 +181,7 @@ func loadServiceTopology(moduleName string, service serviceDefinition) (topology seen := make(map[string]bool, len(manifest.Endpoints)) for _, endpoint := range manifest.Endpoints { api := endpointAPI(endpoint.Name, endpoint.API) - visibility, err := endpointVisibility(endpoint.Visibility, endpoint.Location, endpoint.AllowModules) + visibility, err := endpointVisibility(endpoint.Visibility, endpoint.Location, false) if err != nil { return topologyService{}, err } @@ -231,29 +226,26 @@ func endpointAPI(name, api string) string { return api } -// endpointVisibility projects the authored Codefly axes into the host's -// deployment catalog categories. Its MODULE category means all composed -// modules, so a narrower allow-list cannot be represented and must refuse. -func endpointVisibility(visibility, location string, allowed []string) (string, error) { +// endpointVisibility projects the authored reach into the catalog's legacy MODULE +// category. It does not grant callers: only declared dependency edges render +// network access, and the composing Core derives cross-module reach from asks. +func endpointVisibility(visibility, location string, exported bool) (string, error) { if visibility == "" { visibility = "private" + if exported { + visibility = "internal" + } } if location != "" { - if location != "external" || visibility != "private" || len(allowed) != 0 { + if location != "external" || visibility != "private" { return "", fmt.Errorf("unsupported external endpoint policy") } return "external", nil } switch visibility { case "private", "public": - if len(allowed) != 0 { - return "", fmt.Errorf("allow-modules requires internal visibility") - } return visibility, nil case "internal": - if len(allowed) != 1 || allowed[0] != "*" { - return "", fmt.Errorf("deployment catalog requires internal visibility with allow-modules [*]; narrower exports need an allow-list-aware catalog") - } return "module", nil default: return "", fmt.Errorf("unsupported authored endpoint visibility %q", visibility) @@ -302,3 +294,47 @@ func loadDeployJobs(moduleDir string) ([]topologyDeployJob, error) { } return manifest.DeployJobs, nil } + +type topologyManifestEndpoint struct { + Name string `yaml:"name"` + Visibility string `yaml:"visibility"` + API string `yaml:"api"` + Location string `yaml:"location"` +} + +func (endpoint *topologyManifestEndpoint) UnmarshalYAML(node *yaml.Node) error { + if err := refuseAuthoredConsumers(node); err != nil { + return err + } + type plain topologyManifestEndpoint + return node.Decode((*plain)(endpoint)) +} + +func (endpoint *topologyInterface) UnmarshalYAML(node *yaml.Node) error { + if err := refuseAuthoredConsumers(node); err != nil { + return err + } + type plain topologyInterface + return node.Decode((*plain)(endpoint)) +} + +// Judge key presence before decoding: null and alternate spellings cannot +// silently erase a forbidden grant. This matches Core's declaration boundary. +func refuseAuthoredConsumers(node *yaml.Node) error { + if node.Kind != yaml.MappingNode { + return nil + } + for i := 0; i+1 < len(node.Content); i += 2 { + key := node.Content[i].Value + var canonical strings.Builder + for _, r := range strings.ToLower(key) { + if r >= 'a' && r <= 'z' { + canonical.WriteRune(r) + } + } + if canonical.String() == "allowmodules" { + return fmt.Errorf("authors %q: endpoint access is derived from consumers' declared dependencies", key) + } + } + return nil +} diff --git a/topology_ports_test.go b/topology_ports_test.go index 63d3ca478..dc3aad2b6 100644 --- a/topology_ports_test.go +++ b/topology_ports_test.go @@ -23,10 +23,8 @@ endpoints: - name: authority api: grpc visibility: internal - allow-modules: ["*"] - name: connect visibility: internal - allow-modules: ["*"] - name: grpc - name: rest spec: diff --git a/topology_visibility_test.go b/topology_visibility_test.go index 6c8c88e83..4e8b0bb7d 100644 --- a/topology_visibility_test.go +++ b/topology_visibility_test.go @@ -5,22 +5,20 @@ import "testing" func TestEndpointCatalogProjectionPreservesOrRefusesPolicy(t *testing.T) { for _, tc := range []struct { visibility, location string - allowed []string + exported bool want string }{ - {"internal", "", []string{"*"}, "module"}, - {"private", "external", nil, "external"}, - {"public", "", nil, "public"}, - {"", "", nil, "private"}, - {"internal", "", []string{"example"}, ""}, - {"internal", "", nil, ""}, - {"public", "", []string{"example"}, ""}, - {"module", "", nil, ""}, - {"external", "", nil, ""}, - {"private", "elsewhere", nil, ""}, - {"internal", "external", []string{"example"}, ""}, + {"internal", "", false, "module"}, + {"private", "external", false, "external"}, + {"public", "", false, "public"}, + {"", "", false, "private"}, + {"internal", "", true, "module"}, + {"module", "", false, ""}, + {"external", "", false, ""}, + {"private", "elsewhere", false, ""}, + {"internal", "external", false, ""}, } { - got, err := endpointVisibility(tc.visibility, tc.location, tc.allowed) + got, err := endpointVisibility(tc.visibility, tc.location, tc.exported) if tc.want == "" { if err == nil { t.Errorf("accepted unsupported policy %#v as %q", tc, got) diff --git a/vault_binding_test.go b/vault_binding_test.go index a3992f9ce..6098250d1 100644 --- a/vault_binding_test.go +++ b/vault_binding_test.go @@ -64,11 +64,12 @@ func TestVaultEndpointIsPrivateAndResolvesOnlyWithinItsModule(t *testing.T) { Endpoints: []*resources.EndpointReference{{Name: "http"}}, }} mappings := []*basev0.NetworkMapping{{Endpoint: declaration}} - resolved, err := resources.ResolveDependencyNetworkMappings("saas-starter", dependencies, mappings) + composition := &resources.Workspace{Name: "example-workspace", Modules: []*resources.ModuleReference{{Name: "saas-starter"}, {Name: "example"}}} + resolved, err := resources.ResolveDependencyNetworkMappings(composition, "saas-starter", dependencies, mappings) if err != nil || len(resolved) != 1 { t.Fatalf("same-module vault dependency: resolved %d mappings, error %v", len(resolved), err) } - if _, err := resources.ResolveDependencyNetworkMappings("example", dependencies, mappings); err == nil { + if _, err := resources.ResolveDependencyNetworkMappings(composition, "example", dependencies, mappings); err == nil { t.Fatal("vault/http was handed to a different module") } } From 2a8170cc65c3d29b7792f3fce90295fcbccd4b2b Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Wed, 7 Oct 2026 15:14:32 -0400 Subject: [PATCH 61/69] fix(deployment): align SDK runtime loaders with endpoint declarations --- module/DEPLOYMENT_TOPOLOGY.md | 5 +++-- module/services/auth-gateway/code/go.mod | 2 +- module/services/auth-gateway/code/go.sum | 4 ++-- module/services/store/code/go.mod | 2 +- module/services/store/code/go.sum | 4 ++-- 5 files changed, 9 insertions(+), 8 deletions(-) diff --git a/module/DEPLOYMENT_TOPOLOGY.md b/module/DEPLOYMENT_TOPOLOGY.md index 657a2ada3..b00832104 100644 --- a/module/DEPLOYMENT_TOPOLOGY.md +++ b/module/DEPLOYMENT_TOPOLOGY.md @@ -62,8 +62,9 @@ Promotion requires a released CLI with Core's consumer-derived allow-list support and the audit promotion driver from [CLI PR 919](https://github.com/codefly-dev/cli/pull/919). Merge the CLI change, release that compatible CLI, then promote this kit cutover. The module agent, -accounts catalog generator, and module tools require stable Core 0.14 or later, -which accepts these declarations and derives consumer access. Core 0.13 refuses +accounts, auth-gateway, store migrator, and module tools require stable Core 0.14 +or later to load these declarations and derive consumer access. The runtime SDK +also loads the module when resolving local endpoints or secrets. Core 0.13 refuses internal interface exports without an authored list. The CI pin at CLI 0.1.171 still loads the module with its older Core 0.7, but does not derive cross-module access: an internal endpoint with no authored list denies other modules. It is diff --git a/module/services/auth-gateway/code/go.mod b/module/services/auth-gateway/code/go.mod index 25805db5f..d2fd257aa 100644 --- a/module/services/auth-gateway/code/go.mod +++ b/module/services/auth-gateway/code/go.mod @@ -4,7 +4,7 @@ go 1.27.0 require ( buf.build/gen/go/bufbuild/protovalidate/protocolbuffers/go v1.36.12-20260825204119-511051f7f437.2 - github.com/codefly-dev/core v0.13.0 + github.com/codefly-dev/core v0.14.0 github.com/codefly-dev/sdk-go v0.3.1 github.com/codefly-dev/sdk-go/workcontext v0.0.0-20260926212413-b91ce91eb23e github.com/envoyproxy/go-control-plane/envoy v1.39.0 diff --git a/module/services/auth-gateway/code/go.sum b/module/services/auth-gateway/code/go.sum index 2e176b4a0..9bd434b03 100644 --- a/module/services/auth-gateway/code/go.sum +++ b/module/services/auth-gateway/code/go.sum @@ -42,8 +42,8 @@ github.com/cloudflare/circl v1.6.3 h1:9GPOhQGF9MCYUeXyMYlqTR6a5gTrgR/fBLXvUgtVcg github.com/cloudflare/circl v1.6.3/go.mod h1:2eXP6Qfat4O/Yhh8BznvKnJ+uzEoTQ6jVKJRn81BiS4= github.com/cncf/xds/go v0.0.0-20260202195803-dba9d589def2 h1:aBangftG7EVZoUb69Os8IaYg++6uMOdKK83QtkkvJik= github.com/cncf/xds/go v0.0.0-20260202195803-dba9d589def2/go.mod h1:qwXFYgsP6T7XnJtbKlf1HP8AjxZZyzxMmc+Lq5GjlU4= -github.com/codefly-dev/core v0.13.0 h1:FiphttBZRrKXEWoCZD7JpUB6oVuBCoqtAc06QeNdYWY= -github.com/codefly-dev/core v0.13.0/go.mod h1:rp4YZkCJEOufliWQAgK3sv8YlO7Xcva7+cN7hIlAR0I= +github.com/codefly-dev/core v0.14.0 h1:rpLFxeHSf2PrbxsRSp/NcZt8htMRM38JV2Oc8RKievU= +github.com/codefly-dev/core v0.14.0/go.mod h1:rp4YZkCJEOufliWQAgK3sv8YlO7Xcva7+cN7hIlAR0I= github.com/codefly-dev/gortk v0.2.0 h1:7bOlS5valYz2zil+fZctQNcPCYBcPj86abcw9N8h1hQ= github.com/codefly-dev/gortk v0.2.0/go.mod h1:dDWUMFgAP063OCGhTEpV7FFUj9+zTcxxO/nV80VKEwg= github.com/codefly-dev/sdk-go v0.3.1 h1:ljKwrfHIiESV+R/MXO3a7XTdKwFNaY6vdw0+8YAo9g8= diff --git a/module/services/store/code/go.mod b/module/services/store/code/go.mod index 05f10868b..2281d8f11 100644 --- a/module/services/store/code/go.mod +++ b/module/services/store/code/go.mod @@ -16,7 +16,7 @@ require ( github.com/Masterminds/semver v1.5.0 // indirect github.com/Masterminds/semver/v3 v3.5.0 // indirect github.com/antlr4-go/antlr/v4 v4.13.1 // indirect - github.com/codefly-dev/core v0.13.0 // indirect + github.com/codefly-dev/core v0.14.0 // indirect github.com/go-openapi/analysis v1.0.0 // indirect github.com/go-openapi/errors v0.22.8 // indirect github.com/go-openapi/jsonpointer v1.0.1 // indirect diff --git a/module/services/store/code/go.sum b/module/services/store/code/go.sum index adfc358a3..f4fbfc4fd 100644 --- a/module/services/store/code/go.sum +++ b/module/services/store/code/go.sum @@ -30,8 +30,8 @@ github.com/cheggaaa/pb/v3 v3.2.1 h1:aprZbFRG+B7+ug76S8QZ6Y1PW168UHzOmbC3wa+aU6I= github.com/cheggaaa/pb/v3 v3.2.1/go.mod h1:U9hSVxoKqJrZIE3PkFG1xXXNaK/Ilzg+EF/scpXzEdw= github.com/clipperhouse/uax29/v2 v2.7.0 h1:+gs4oBZ2gPfVrKPthwbMzWZDaAFPGYK72F0NJv2v7Vk= github.com/clipperhouse/uax29/v2 v2.7.0/go.mod h1:EFJ2TJMRUaplDxHKj1qAEhCtQPW2tJSwu5BF98AuoVM= -github.com/codefly-dev/core v0.13.0 h1:FiphttBZRrKXEWoCZD7JpUB6oVuBCoqtAc06QeNdYWY= -github.com/codefly-dev/core v0.13.0/go.mod h1:rp4YZkCJEOufliWQAgK3sv8YlO7Xcva7+cN7hIlAR0I= +github.com/codefly-dev/core v0.14.0 h1:rpLFxeHSf2PrbxsRSp/NcZt8htMRM38JV2Oc8RKievU= +github.com/codefly-dev/core v0.14.0/go.mod h1:rp4YZkCJEOufliWQAgK3sv8YlO7Xcva7+cN7hIlAR0I= github.com/codefly-dev/gortk v0.2.0 h1:7bOlS5valYz2zil+fZctQNcPCYBcPj86abcw9N8h1hQ= github.com/codefly-dev/gortk v0.2.0/go.mod h1:dDWUMFgAP063OCGhTEpV7FFUj9+zTcxxO/nV80VKEwg= github.com/codefly-dev/sdk-go v0.3.1 h1:ljKwrfHIiESV+R/MXO3a7XTdKwFNaY6vdw0+8YAo9g8= From 85247e2e8d37cdb72bd733cce657f56d02f52893 Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Wed, 7 Oct 2026 15:38:54 -0400 Subject: [PATCH 62/69] fix(audit): point rollback tests at renumbered migrations --- .../code/pkg/infra/postgres_audit_quarantine_test.go | 2 +- .../code/pkg/infra/postgres_audit_rollback_test.go | 10 +++++----- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/module/services/accounts/code/pkg/infra/postgres_audit_quarantine_test.go b/module/services/accounts/code/pkg/infra/postgres_audit_quarantine_test.go index 2dbadcfb0..3ba61c69c 100644 --- a/module/services/accounts/code/pkg/infra/postgres_audit_quarantine_test.go +++ b/module/services/accounts/code/pkg/infra/postgres_audit_quarantine_test.go @@ -15,7 +15,7 @@ import ( "accounts/pkg/infra/storetx" ) -// The audit relay's quarantine (migration 23): who may touch it, and that moving +// The audit relay's quarantine (migration 33): who may touch it, and that moving // a row into it is one transaction. func enqueueN(t *testing.T, orgID string, n int) []business.AuditEntry { diff --git a/module/services/accounts/code/pkg/infra/postgres_audit_rollback_test.go b/module/services/accounts/code/pkg/infra/postgres_audit_rollback_test.go index b2cab2dbf..3fdbbf74e 100644 --- a/module/services/accounts/code/pkg/infra/postgres_audit_rollback_test.go +++ b/module/services/accounts/code/pkg/infra/postgres_audit_rollback_test.go @@ -16,7 +16,7 @@ import ( "accounts/pkg/business" ) -// The queue's and the quarantine's down migrations (20 and 23) drop the only +// The queue's and the quarantine's down migrations (30 and 33) drop the only // copy of whatever rows they hold, so they refuse while any row is there. // migrationOwner runs fn on a connection with the authority migrations run @@ -61,7 +61,7 @@ func asRole(t *testing.T, ctx context.Context, tx pgx.Tx, role, sql string, args // Rolling the queue's migration back drops the only copy of every event the // relay has not delivered, so it must refuse while any row is queued. func TestRollingBackTheQueueRefusesWhileEventsAreQueued(t *testing.T) { - down := readMigration(t, "20_audit_event_queue.down.sql") + down := readMigration(t, "30_audit_event_queue.down.sql") orgID := seedOrg(t, seedUser(t)) migrationOwner(t, func(ctx context.Context, tx pgx.Tx) { @@ -88,7 +88,7 @@ func TestRollingBackTheQueueRefusesWhileEventsAreQueued(t *testing.T) { } func TestRollingBackTheQuarantineRefusesWhileRowsAreSetAside(t *testing.T) { - down := readMigration(t, "23_audit_event_quarantine.down.sql") + down := readMigration(t, "33_audit_event_quarantine.down.sql") migrationOwner(t, func(ctx context.Context, tx pgx.Tx) { countQuarantine := func() int { @@ -154,7 +154,7 @@ func TestRollingBackWaitsForAnInFlightWriterBeforeItCounts(t *testing.T) { }{ { name: "queue", - down: "20_audit_event_queue.down.sql", + down: "30_audit_event_queue.down.sql", table: "public.audit_event_queue", writerRole: "app_control_plane", insert: `INSERT INTO public.audit_event_queue (id, event_type, schema_version, actor_type, resource, created_at) @@ -163,7 +163,7 @@ func TestRollingBackWaitsForAnInFlightWriterBeforeItCounts(t *testing.T) { }, { name: "quarantine", - down: "23_audit_event_quarantine.down.sql", + down: "33_audit_event_quarantine.down.sql", table: "public.audit_event_quarantine", writerRole: "app_job_worker", insert: `INSERT INTO public.audit_event_quarantine (seq, xact_id, id, event_type, schema_version, actor_type, resource, created_at, enqueued_at, error) From 0800ededa38246c2ce80f3a7296d7df9a090f3f9 Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Wed, 7 Oct 2026 16:32:21 -0400 Subject: [PATCH 63/69] fix(telemetry): ship neither the telemetry state nor its reason in the module profile (Refs #1022) Every deployed cell inherits the module's local `observability` profile, and it shipped TELEMETRY_ABSENT_REASON. A healthy `available` cell therefore logged a leftover-key warning on every boot, and a cell that sent `absent` without a reason passed validation with the reason written for a laptop. The profile now declares neither TELEMETRY_STATE nor TELEMETRY_ABSENT_REASON. Both services supply the `absent` state and its reason in code (localAbsentReason in the two mirrored telemetry_destination.go copies), and only when the Codefly runtime is `local`, so an explicit `absent` that does not say why is refused everywhere, a local run included. The one assignment the file keeps is an empty OTEL_EXPORTER_OTLP_ENDPOINT: the base-sync guard requires every group a service declares to assign at least one key, and an empty endpoint is no endpoint. The tests that read the shipped file now pin that it carries no state, no reason and no endpoint, that a missing state is refused outside a local run and gets the code's reason inside one, and that a cell's own answer layered over the profile neither warns nor passes without a reason. local-dogfood is not the `local` runtime. codefly.IsLocal matches the environment named `local` and no other, and the pinned sdk-go (v0.3.1) exports no accessor for a process that runs from a workspace: Environment, IsLocal and Workspace are all it has, and Workspace resolves inside a deployed pod as well. So a local-dogfood run with `TELEMETRY_STATE=absent` registers no tracer and exports no traces or metrics, although LOCAL_DOGFOODING.md and observability.env.example promised wool's stdout tracer. They now say what happens, as do DOGFOODING.md, the run-the-starter-locally skill, FEATURES.md and the package notes, and the example file's reason text no longer claims a stdout tracer. Co-Authored-By: Claude Opus 5.5 --- .../skills/run-the-starter-locally/SKILL.md | 14 ++-- DOGFOODING.md | 4 +- LOCAL_DOGFOODING.md | 8 ++- .../local-dogfood/observability.env.example | 9 +-- module/FEATURES.md | 27 ++++---- module/configurations/local/observability.env | 30 +++++--- module/module.package.codefly.yaml | 2 +- .../accounts/code/telemetry_destination.go | 23 ++++--- .../code/telemetry_destination_test.go | 68 +++++++++++++++---- .../code/telemetry_destination.go | 23 ++++--- .../code/telemetry_destination_test.go | 68 +++++++++++++++---- 11 files changed, 193 insertions(+), 83 deletions(-) diff --git a/.claude/skills/run-the-starter-locally/SKILL.md b/.claude/skills/run-the-starter-locally/SKILL.md index 2e9eb0259..c6e6a9a11 100644 --- a/.claude/skills/run-the-starter-locally/SKILL.md +++ b/.claude/skills/run-the-starter-locally/SKILL.md @@ -19,12 +19,14 @@ codefly run service --fixture dev-admin - Docker must be running. `codefly doctor` checks prerequisites; `codefly clear` reaps stray processes and containers between runs. - A local run has no cell collector, and the module ships none. The module's - `configurations/local/observability.env` supplies a reason but no - `TELEMETRY_STATE`: those defaults also reach deployed cells. Only a runtime the - SDK identifies as local infers `absent` when the state is missing, logs the - reason once, sends traces to wool's stdout tracer and exports no metrics. - Outside that runtime, a missing state refuses startup. The `local-dogfood` - profile must still copy its explicit state from `observability.env.example`. + `configurations/local/observability.env` declares neither `TELEMETRY_STATE` + nor `TELEMETRY_ABSENT_REASON`: those defaults also reach deployed cells. Only + the runtime the SDK identifies as `local` infers `absent` when the state is + missing, supplies the reason in code, logs it once, sends traces to wool's + stdout tracer and exports no metrics. Outside that runtime, a missing state + refuses startup. The `local-dogfood` profile is not that runtime: it must copy + its explicit state from `observability.env.example`, and with no collector it + exports no traces or metrics at all. For a real external identity provider and the production-grade provider stack, use the `local-dogfood` environment and its configuration groups: diff --git a/DOGFOODING.md b/DOGFOODING.md index 5085e0f1a..b4240ec6d 100644 --- a/DOGFOODING.md +++ b/DOGFOODING.md @@ -219,7 +219,7 @@ the same Codefly-managed graph. For each of `email` (Resend), `configurations/local-dogfood/.env.example` and `.secret.env.example` to their Git-ignored names and fill them in ([LOCAL_DOGFOODING.md](./LOCAL_DOGFOODING.md#configure-providers-through-codefly)). -Keep `observability.env.example` as it is for a `local-dogfood` run with no collector (`TELEMETRY_STATE=absent`). The plain `local` runtime infers `absent` when the state is missing; the module's local defaults do not declare it. +Keep `observability.env.example` as it is for a `local-dogfood` run with no collector (`TELEMETRY_STATE=absent`). The plain `local` runtime infers `absent` when the state is missing, and supplies the reason itself; the module's local defaults declare neither. `local-dogfood` is not that runtime, so a run with no collector exports no traces or metrics. Turnstile's published test keys drive the three abuse-protection cases: | Case | `NEXT_PUBLIC_TURNSTILE_SITE_KEY` | `TURNSTILE_SECRET_KEY` | @@ -239,7 +239,7 @@ codefly run service --env local-dogfood - [ ] Grant analytics consent, navigate through onboarding, and verify bounded browser plus durable backend events in PostHog. - [ ] Withdraw analytics consent or log out. Browser capture stops immediately and identity resets. - [ ] Trigger controlled browser and backend errors. Sentry correlates release and environment without creating performance transactions. -- [ ] With `TELEMETRY_STATE=absent`, check that accounts and auth-gateway log the absent reason once at startup, then exercise login/onboarding and observe wool's stdout trace output (metrics are not exported; logs are on stdout). Remove `TELEMETRY_STATE` in `local-dogfood` and confirm both services refuse to start, naming it. In the plain `local` runtime, a missing state instead starts with an absent collector. +- [ ] With `TELEMETRY_STATE=absent`, check that accounts and auth-gateway log the absent reason once at startup, then exercise login/onboarding; no traces or metrics are exported in `local-dogfood` (wool's stdout tracer is for the plain `local` runtime), so observe the logs on stdout. To see traces, point `OTEL_EXPORTER_OTLP_ENDPOINT` at a collector this machine can reach and set `TELEMETRY_STATE=available`. Remove `TELEMETRY_STATE` in `local-dogfood` and confirm both services refuse to start, naming it. In the plain `local` runtime, a missing state instead starts with an absent collector. - [ ] Switch Turnstile to the **fail** keys. Registration and waitlist submission fail without database writes. - [ ] Switch Turnstile to the **replay** keys. The first deterministic verification follows Cloudflare's fixture behavior; replay rejection leaves state unchanged. diff --git a/LOCAL_DOGFOODING.md b/LOCAL_DOGFOODING.md index a08cab495..17ab22dd9 100644 --- a/LOCAL_DOGFOODING.md +++ b/LOCAL_DOGFOODING.md @@ -62,9 +62,11 @@ committed default to fall back on. Codefly skips a declared group whose files ar absent rather than failing, and both services refuse to start without an explicit `TELEMETRY_STATE` — a group that did not arrive is not a cell without a collector, and they will not guess. Copy `observability.env.example` as it is to run with -none: `TELEMETRY_STATE=absent` and its reason, traces on wool's stdout tracer, -metrics not exported. To export to a collector this machine can reach, set -`TELEMETRY_STATE=available` and `OTEL_EXPORTER_OTLP_ENDPOINT` as the example says. +none: `TELEMETRY_STATE=absent` and its reason. Traces and metrics are then not +exported at all (logs still go to stdout): `local-dogfood` is not the `local` +runtime, and wool's stdout tracer belongs to that runtime alone. To export to a +collector this machine can reach, set `TELEMETRY_STATE=available` and +`OTEL_EXPORTER_OTLP_ENDPOINT` as the example says. `email` is required because this profile is not the `local` environment, and accounts refuses to start there without an explicit `EMAIL_PROVIDER`: `resend` diff --git a/configurations/local-dogfood/observability.env.example b/configurations/local-dogfood/observability.env.example index 84f833283..2276db415 100644 --- a/configurations/local-dogfood/observability.env.example +++ b/configurations/local-dogfood/observability.env.example @@ -1,11 +1,12 @@ # Copy to observability.env and fill it in. This module owns no collector: the # cell's collector is where traces and metrics go, and the platform says where it is. # -# A dogfood run on a laptop is not a cell, so by default it has none. The default -# below is what a plain local run uses: wool's stdout tracer stands in for traces -# and metrics are not exported. +# A dogfood run on a laptop is not a cell, so by default it has none. Neither +# traces nor metrics are exported then: this profile is not the plain `local` +# runtime, and wool's stdout tracer belongs to that runtime alone. Logs still go +# to stdout. TELEMETRY_STATE=absent -TELEMETRY_ABSENT_REASON=A local dogfood run has no cell collector; traces go to the stdout tracer and metrics are not exported. +TELEMETRY_ABSENT_REASON=A local dogfood run has no cell collector; traces and metrics are not exported. # To export to a collector this machine can reach (for example a cell's, through a # port-forward), replace the two lines above with: diff --git a/module/FEATURES.md b/module/FEATURES.md index 6fc66037b..a3980a6d2 100644 --- a/module/FEATURES.md +++ b/module/FEATURES.md @@ -665,7 +665,7 @@ Environment variables consumed by the api: | `AUDIT_RELAY_BATCH_SIZE`, `AUDIT_RELAY_MAX_WAIT` | Optional under a swap value: events per delivery (default 500, at most 5000) and how long a partial batch waits (default `5s`, at most `60s`: a longer wait would reach the five-minute relay-lag alert on a healthy relay, so a larger value is refused) | | `ERROR_TRACKING_MODE` | Explicit `disabled` or `sentry`; rejects partial config | | `SENTRY_DSN` | Server Sentry DSN, required in Sentry mode | -| `TELEMETRY_STATE` | `observability` group, delivered by the platform: `available` or `absent`; missing state refuses startup except in a local runtime, where it means `absent` | +| `TELEMETRY_STATE` | `observability` group, delivered by the platform: `available` or `absent`; missing state refuses startup except in the `local` runtime, where it means `absent` | | `OTEL_EXPORTER_OTLP_ENDPOINT` | `observability` group: the cell collector's OTLP/gRPC address, required when `available` and ignored when `absent`; `http://` is plaintext on the wire because the mesh supplies mTLS, `https://` is TLS | | `TELEMETRY_ABSENT_REASON` | `observability` group: why the cell has no collector, required when `absent` and ignored when `available`; logged once at startup | | `ABUSE_PROTECTION_MODE` | Explicit `disabled` or `turnstile` | @@ -677,17 +677,20 @@ When `TELEMETRY_STATE` is `available`, the accounts service and the auth-gateway export traces and unsampled request and Go runtime metrics over OTLP/gRPC to the cell's collector at `OTEL_EXPORTER_OTLP_ENDPOINT`. The auth-gateway covers both its HTTP gateway and gRPC ext_authz authorization service. When it is `absent`, they -export nothing over OTLP, log `TELEMETRY_ABSENT_REASON` once and boot; a local run -then uses wool's stdout tracer for traces. The state decides: a cell's values -override the module's local defaults one key at a time, so a deployed cell that -says `available` with an endpoint can still carry the `absent` reason the local -profile left behind. The key the state does not use is ignored, and a startup -warning names it once. Module defaults never declare `TELEMETRY_STATE`, because -they also reach deployed cells. Only a local runtime infers `absent` from a -missing state; elsewhere missing state refuses startup, because a group that -did not arrive is not a cell without a collector. Unknown state, `available` -without an endpoint, or explicit `absent` without a reason also refuses startup, -including locally. An `https://` +export nothing over OTLP, log `TELEMETRY_ABSENT_REASON` once and boot. A local +runtime, meaning the Codefly environment named `local` and no other, then uses +wool's stdout tracer for traces; `local-dogfood` is not that runtime, so it exports +no traces either. The state decides: the group is layered, so a key the state does +not use can still arrive from a lower layer, and is ignored, with a startup warning +that names it once. The module's own profile declares neither `TELEMETRY_STATE` +nor `TELEMETRY_ABSENT_REASON`, because module defaults also reach deployed cells: a +healthy `available` cell would warn on every boot about a reason it never set, +and a cell that sent `absent` without a reason would pass with one written for a +laptop. Only a local runtime infers `absent` from a missing state, and it +supplies that state and its reason in code; elsewhere missing state refuses +startup, because a group that did not arrive is not a cell without a collector. +Unknown state, `available` without an endpoint, or explicit `absent` without a +reason also refuses startup, including locally. An `https://` endpoint is refused at startup for now: wool's OTLP tracer dials plaintext only, so it would be sent in the clear and reported as TLS. Metrics leave by OTLP push alone — decided 2026-10-06, because the cell's diff --git a/module/configurations/local/observability.env b/module/configurations/local/observability.env index bce6d701d..be06056cb 100644 --- a/module/configurations/local/observability.env +++ b/module/configurations/local/observability.env @@ -8,14 +8,26 @@ # (http:// is plaintext because the mesh supplies mTLS; # https:// is TLS). # TELEMETRY_STATE=absent the cell has none. Also sets TELEMETRY_ABSENT_REASON to -# say why. +# say why; an absent cell that does not is refused. # -# The state decides. A cell's values override these defaults one key at a time, so a -# cell that sets `available` and an endpoint still carries the reason below; the -# process ignores the key its state does not use and logs that once. +# The state decides: the key the state does not use is ignored, and the process logs +# that once. # -# A local run has no cell, so it has no collector: traces go to wool's stdout tracer -# and metrics are not exported. Logs always go to stdout. Do not declare a state -# here: these defaults also reach deployed cells. Only a local runtime infers -# `absent` when the platform has not delivered TELEMETRY_STATE. -TELEMETRY_ABSENT_REASON=A local run has no cell collector; traces go to the stdout tracer and metrics are not exported. +# This profile declares neither the state nor the reason, on purpose. Module defaults +# also reach deployed cells, so either shipped here would be inherited by a cell that +# never set it: a healthy `available` cell would warn about a leftover reason on +# every boot, and a cell that sent `absent` without a reason would pass with a +# reason written for a laptop. +# +# A local run has no cell, so it has no collector, and says so itself: the services +# supply the `absent` state and its reason in code, only when the Codefly runtime is +# `local`. Traces then go to wool's stdout tracer and metrics are not exported. Logs +# always go to stdout. Any other environment, `local-dogfood` included, must deliver +# the state itself (see LOCAL_DOGFOODING.md). +# +# The one assignment below is empty, and is here only so the group exists: a +# composing workspace inherits a group from the key it assigns, and a group with no +# key is met at runtime-init as "no configuration found for observability". An empty +# endpoint is no endpoint, so it neither masks a cell's answer nor triggers the +# leftover-key warning. +OTEL_EXPORTER_OTLP_ENDPOINT= diff --git a/module/module.package.codefly.yaml b/module/module.package.codefly.yaml index b075132a4..80c035253 100644 --- a/module/module.package.codefly.yaml +++ b/module/module.package.codefly.yaml @@ -100,7 +100,7 @@ migrations: breaking-changes: - Consumers resolve the immutable package into a disposable projection instead of editing Starter-owned files. - "Audit event types are namespaced under `saas.*` (`auth.login` -> `saas.auth.login`). The wire value of `audit_events.event_type`, the `X-Webhook-Event` header, and webhook routing keys all change; migration 116 rewrites stored history and existing subscriptions in place, so a live subscriber keeps firing, but a consumer that re-creates a subscription from a hardcoded pre-namespace name, or emits one through `ModuleEmitAuditEvent`, is now rejected. No alias layer." - - "The `telemetry` service is removed: the module no longer ships an OTLP gateway, and the cell's collector takes its place. `accounts` and `auth-gateway` read where it is from the `observability` configuration group — `TELEMETRY_STATE` is `available` with `OTEL_EXPORTER_OTLP_ENDPOINT` (an `http://` endpoint is plaintext on the wire because the mesh supplies mTLS; `https://` is TLS), or `absent` with `TELEMETRY_ABSENT_REASON` — and refuse to start outside a local runtime when the group does not say which. Module defaults do not declare TELEMETRY_STATE; only a local runtime infers absent when the state is missing. `OBSERVABILITY_EXPORTER` and `OTEL_EXPORTER_OTLP_HEADERS` no longer mean anything, the `/metrics` scrape endpoint of both services is gone (metrics leave by OTLP push only), and `service.name` is no longer fixed in code: `OTEL_SERVICE_NAME` or `OTEL_RESOURCE_ATTRIBUTES` sets it, else the service's own Codefly identity. A consumer that depended on the `telemetry` service, or that overrides the `observability` group, must move to the keys above." + - "The `telemetry` service is removed: the module no longer ships an OTLP gateway, and the cell's collector takes its place. `accounts` and `auth-gateway` read where it is from the `observability` configuration group — `TELEMETRY_STATE` is `available` with `OTEL_EXPORTER_OTLP_ENDPOINT` (an `http://` endpoint is plaintext on the wire because the mesh supplies mTLS; `https://` is TLS), or `absent` with `TELEMETRY_ABSENT_REASON` — and refuse to start outside a local runtime when the group does not say which. Module defaults declare neither TELEMETRY_STATE nor TELEMETRY_ABSENT_REASON, because they also reach deployed cells; only the `local` runtime infers absent when the state is missing, and it supplies the state and its reason in code. `OBSERVABILITY_EXPORTER` and `OTEL_EXPORTER_OTLP_HEADERS` no longer mean anything, the `/metrics` scrape endpoint of both services is gone (metrics leave by OTLP push only), and `service.name` is no longer fixed in code: `OTEL_SERVICE_NAME` or `OTEL_RESOURCE_ATTRIBUTES` sets it, else the service's own Codefly identity. A consumer that depended on the `telemetry` service, or that overrides the `observability` group, must move to the keys above." reserved-namespaces: - "@codefly/saas-" - "@codefly-dev/saas-" diff --git a/module/services/accounts/code/telemetry_destination.go b/module/services/accounts/code/telemetry_destination.go index cecf41049..650372193 100644 --- a/module/services/accounts/code/telemetry_destination.go +++ b/module/services/accounts/code/telemetry_destination.go @@ -32,18 +32,19 @@ import ( // collector: the cell does, and a workload that cannot find it does not // substitute one. // -// The state decides. The group is layered: the cell's values override the -// module's local defaults one key at a time, so a deployed workload sees the -// cell's `available` and endpoint beside the `absent` reason the module's local -// profile left behind. The key that does not belong to the state is therefore -// ignored, not refused, and reported once through IgnoredNotice. What each state -// requires is unchanged: `available` needs an endpoint, `absent` needs a reason. +// The state decides. The group is layered, so a key the state does not use can +// still arrive from a lower layer: it is ignored, not refused, and reported once +// through IgnoredNotice. What each state requires is unchanged: `available` needs +// an endpoint, `absent` needs a reason. The module's own profile ships neither +// the state nor the reason, so a cell that delivers `available` carries no +// leftover and a cell that delivers `absent` without a reason is refused. // // A group that did not arrive is not a cell without a backend. Reading the first // as the second would leave a deployment that converges green and exports // nothing, so a missing or unknown state, or the key the state requires being // absent, refuses to start, naming what is missing. Only a local runtime may -// infer `absent` when the state is missing; module defaults never declare it. +// infer `absent` when the state is missing, and it supplies the reason itself +// (localAbsentReason): module defaults declare neither. // // This file exists byte for byte in services/accounts/code and // services/auth-gateway/code: the two services are independent Go modules with @@ -62,6 +63,12 @@ const ( // a log line: an unbounded value from configuration would otherwise let // whoever writes the group forge log entries around it. maxAbsentReasonLength = 300 + + // localAbsentReason is the reason a local run gives for having no collector. + // It lives in code, and is used only when the process is local, because the + // module's configuration profile is inherited by deployed cells: a reason + // shipped there would reach a cell that sent `absent` without saying why. + localAbsentReason = "A local run has no cell collector; traces go to the stdout tracer and metrics are not exported." ) // telemetryDestination is the resolved answer. Exactly one of URL and @@ -139,7 +146,7 @@ func resolveTelemetryDestination(local bool, read func(key string) string) (tele if state == "" && local { state = telemetryStateAbsent if reason == "" { - reason = "A local run has no cell collector; traces go to the stdout tracer and metrics are not exported." + reason = localAbsentReason } } if state == "" { diff --git a/module/services/accounts/code/telemetry_destination_test.go b/module/services/accounts/code/telemetry_destination_test.go index 9c2010437..4372fbe6b 100644 --- a/module/services/accounts/code/telemetry_destination_test.go +++ b/module/services/accounts/code/telemetry_destination_test.go @@ -22,9 +22,10 @@ func group(values map[string]string) func(string) string { const collector = "http://otel-collector.otel-collector.svc.cluster.local:4317" -// Module defaults are also inherited by cells. They must not supply the state -// that only the platform can declare for a deployed workload. -func TestTelemetryDestinationShippedDefaultsCannotMaskMissingCellState(t *testing.T) { +// shippedObservabilityProfile is what the module's own `observability` profile +// contributes to the group, read from the file it ships. +func shippedObservabilityProfile(t *testing.T) map[string]string { + t.Helper() contents, err := os.ReadFile("../../../configurations/local/observability.env") require.NoError(t, err) values := map[string]string{} @@ -37,12 +38,49 @@ func TestTelemetryDestinationShippedDefaultsCannotMaskMissingCellState(t *testin require.True(t, ok) values[key] = value } - _, err = resolveTelemetryDestination(false, group(values)) + return values +} + +// Module defaults are also inherited by cells, so the profile ships neither the +// state nor the reason: a deployed workload must get both from the platform, and +// a local run gets them from code (localAbsentReason), only when it is local. +func TestTelemetryDestinationShippedProfileDeclaresNothingACellCouldInherit(t *testing.T) { + shipped := shippedObservabilityProfile(t) + require.NotContains(t, shipped, "TELEMETRY_STATE", "every deployed cell inherits the profile") + require.NotContains(t, shipped, "TELEMETRY_ABSENT_REASON", "every deployed cell inherits the profile") + require.Empty(t, shipped["OTEL_EXPORTER_OTLP_ENDPOINT"], "the profile keeps the group in existence and names no collector") + + _, err := resolveTelemetryDestination(false, group(shipped)) require.ErrorContains(t, err, "TELEMETRY_STATE is not set") - destination, err := resolveTelemetryDestination(true, group(values)) + destination, err := resolveTelemetryDestination(true, group(shipped)) require.NoError(t, err) require.False(t, destination.Available()) - require.NotEmpty(t, destination.AbsentReason) + require.Equal(t, localAbsentReason, destination.AbsentReason, "a local run supplies its own reason in code") + require.Empty(t, destination.IgnoredNotice()) +} + +// Layered over the shipped profile, a healthy cell leaves nothing to warn about, +// and a cell that says `absent` without a reason is refused rather than passed +// with the local-run reason. +func TestTelemetryDestinationCellAnswersAreNotMaskedByTheShippedProfile(t *testing.T) { + shipped := shippedObservabilityProfile(t) + + available, err := resolveTelemetryDestination(false, layered(shipped, map[string]string{ + "TELEMETRY_STATE": "available", + "OTEL_EXPORTER_OTLP_ENDPOINT": collector, + })) + require.NoError(t, err) + require.True(t, available.Available()) + require.Empty(t, available.Ignored, "an available cell must not log a leftover-key warning on every boot") + require.Empty(t, available.IgnoredNotice()) + + _, err = resolveTelemetryDestination(false, layered(shipped, map[string]string{"TELEMETRY_STATE": "absent"})) + require.ErrorContains(t, err, "TELEMETRY_ABSENT_REASON is not set") + + // The same answer from a local run is refused too: the local reason fills in + // a missing state, never an explicit `absent` that did not say why. + _, err = resolveTelemetryDestination(true, layered(shipped, map[string]string{"TELEMETRY_STATE": "absent"})) + require.ErrorContains(t, err, "TELEMETRY_ABSENT_REASON is not set") } func TestTelemetryDestinationMissingStateDefaultsOnlyInLocalRuntime(t *testing.T) { @@ -227,9 +265,9 @@ func TestTelemetryDestinationRefusesEverythingElse(t *testing.T) { } } -// The state decides. The cell's values override the module's local defaults one -// key at a time, so a deployed producer sees `available` and an endpoint from the -// cell beside the reason the local profile left behind. That is not a +// The state decides. The group is layered, so a deployed producer can still see +// `available` and an endpoint from the cell beside a reason a lower layer (a +// consuming workspace's own profile, say) left behind. That is not a // contradiction to refuse: the reason is ignored, and said so once. func TestTelemetryDestinationAvailableIgnoresALeftoverReason(t *testing.T) { destination, err := resolveTelemetryDestination(false, group(map[string]string{ @@ -304,16 +342,16 @@ func layered(layers ...map[string]string) func(string) string { return group(merged) } -// The case this exists for: the module's local profile defaults to `absent` with -// a reason, and a deployed cell overrides the state and the endpoint but not the +// The case this exists for: a lower layer of the group says `absent` with a +// reason, and a deployed cell overrides the state and the endpoint but not the // reason. The cell's answer wins, whichever way it points. -func TestTelemetryDestinationTheCellOverridesTheModuleDefaults(t *testing.T) { +func TestTelemetryDestinationTheCellOverridesALowerLayer(t *testing.T) { moduleDefault := map[string]string{ "TELEMETRY_STATE": "absent", "TELEMETRY_ABSENT_REASON": "A local run has no cell collector.", } - t.Run("a cell with a collector overrides the local absent default", func(t *testing.T) { + t.Run("a cell with a collector overrides a lower layer's absent default", func(t *testing.T) { destination, err := resolveTelemetryDestination(false, layered(moduleDefault, map[string]string{ "TELEMETRY_STATE": "available", "OTEL_EXPORTER_OTLP_ENDPOINT": collector, @@ -402,8 +440,8 @@ func TestConfiguredTelemetryDestinationReadsTheObservabilityGroup(t *testing.T) } // Through the SDK's lookup, with the keys exactly as a deployed cell delivers -// them after the layers merge: the cell's state and endpoint, and the reason the -// module's local profile left behind. +// them after the layers merge: the cell's state and endpoint, and a reason a +// lower layer left behind. func TestConfiguredTelemetryDestinationStateDecidesAcrossLayers(t *testing.T) { const prefix = "CODEFLY__WORKSPACE_CONFIGURATION__OBSERVABILITY__" t.Setenv(prefix+"TELEMETRY_STATE", "available") diff --git a/module/services/auth-gateway/code/telemetry_destination.go b/module/services/auth-gateway/code/telemetry_destination.go index cecf41049..650372193 100644 --- a/module/services/auth-gateway/code/telemetry_destination.go +++ b/module/services/auth-gateway/code/telemetry_destination.go @@ -32,18 +32,19 @@ import ( // collector: the cell does, and a workload that cannot find it does not // substitute one. // -// The state decides. The group is layered: the cell's values override the -// module's local defaults one key at a time, so a deployed workload sees the -// cell's `available` and endpoint beside the `absent` reason the module's local -// profile left behind. The key that does not belong to the state is therefore -// ignored, not refused, and reported once through IgnoredNotice. What each state -// requires is unchanged: `available` needs an endpoint, `absent` needs a reason. +// The state decides. The group is layered, so a key the state does not use can +// still arrive from a lower layer: it is ignored, not refused, and reported once +// through IgnoredNotice. What each state requires is unchanged: `available` needs +// an endpoint, `absent` needs a reason. The module's own profile ships neither +// the state nor the reason, so a cell that delivers `available` carries no +// leftover and a cell that delivers `absent` without a reason is refused. // // A group that did not arrive is not a cell without a backend. Reading the first // as the second would leave a deployment that converges green and exports // nothing, so a missing or unknown state, or the key the state requires being // absent, refuses to start, naming what is missing. Only a local runtime may -// infer `absent` when the state is missing; module defaults never declare it. +// infer `absent` when the state is missing, and it supplies the reason itself +// (localAbsentReason): module defaults declare neither. // // This file exists byte for byte in services/accounts/code and // services/auth-gateway/code: the two services are independent Go modules with @@ -62,6 +63,12 @@ const ( // a log line: an unbounded value from configuration would otherwise let // whoever writes the group forge log entries around it. maxAbsentReasonLength = 300 + + // localAbsentReason is the reason a local run gives for having no collector. + // It lives in code, and is used only when the process is local, because the + // module's configuration profile is inherited by deployed cells: a reason + // shipped there would reach a cell that sent `absent` without saying why. + localAbsentReason = "A local run has no cell collector; traces go to the stdout tracer and metrics are not exported." ) // telemetryDestination is the resolved answer. Exactly one of URL and @@ -139,7 +146,7 @@ func resolveTelemetryDestination(local bool, read func(key string) string) (tele if state == "" && local { state = telemetryStateAbsent if reason == "" { - reason = "A local run has no cell collector; traces go to the stdout tracer and metrics are not exported." + reason = localAbsentReason } } if state == "" { diff --git a/module/services/auth-gateway/code/telemetry_destination_test.go b/module/services/auth-gateway/code/telemetry_destination_test.go index cb6409b75..63cb2077e 100644 --- a/module/services/auth-gateway/code/telemetry_destination_test.go +++ b/module/services/auth-gateway/code/telemetry_destination_test.go @@ -22,9 +22,10 @@ func group(values map[string]string) func(string) string { const collector = "http://otel-collector.otel-collector.svc.cluster.local:4317" -// Module defaults are also inherited by cells. They must not supply the state -// that only the platform can declare for a deployed workload. -func TestTelemetryDestinationShippedDefaultsCannotMaskMissingCellState(t *testing.T) { +// shippedObservabilityProfile is what the module's own `observability` profile +// contributes to the group, read from the file it ships. +func shippedObservabilityProfile(t *testing.T) map[string]string { + t.Helper() contents, err := os.ReadFile("../../../configurations/local/observability.env") require.NoError(t, err) values := map[string]string{} @@ -37,12 +38,49 @@ func TestTelemetryDestinationShippedDefaultsCannotMaskMissingCellState(t *testin require.True(t, ok) values[key] = value } - _, err = resolveTelemetryDestination(false, group(values)) + return values +} + +// Module defaults are also inherited by cells, so the profile ships neither the +// state nor the reason: a deployed workload must get both from the platform, and +// a local run gets them from code (localAbsentReason), only when it is local. +func TestTelemetryDestinationShippedProfileDeclaresNothingACellCouldInherit(t *testing.T) { + shipped := shippedObservabilityProfile(t) + require.NotContains(t, shipped, "TELEMETRY_STATE", "every deployed cell inherits the profile") + require.NotContains(t, shipped, "TELEMETRY_ABSENT_REASON", "every deployed cell inherits the profile") + require.Empty(t, shipped["OTEL_EXPORTER_OTLP_ENDPOINT"], "the profile keeps the group in existence and names no collector") + + _, err := resolveTelemetryDestination(false, group(shipped)) require.ErrorContains(t, err, "TELEMETRY_STATE is not set") - destination, err := resolveTelemetryDestination(true, group(values)) + destination, err := resolveTelemetryDestination(true, group(shipped)) require.NoError(t, err) require.False(t, destination.Available()) - require.NotEmpty(t, destination.AbsentReason) + require.Equal(t, localAbsentReason, destination.AbsentReason, "a local run supplies its own reason in code") + require.Empty(t, destination.IgnoredNotice()) +} + +// Layered over the shipped profile, a healthy cell leaves nothing to warn about, +// and a cell that says `absent` without a reason is refused rather than passed +// with the local-run reason. +func TestTelemetryDestinationCellAnswersAreNotMaskedByTheShippedProfile(t *testing.T) { + shipped := shippedObservabilityProfile(t) + + available, err := resolveTelemetryDestination(false, layered(shipped, map[string]string{ + "TELEMETRY_STATE": "available", + "OTEL_EXPORTER_OTLP_ENDPOINT": collector, + })) + require.NoError(t, err) + require.True(t, available.Available()) + require.Empty(t, available.Ignored, "an available cell must not log a leftover-key warning on every boot") + require.Empty(t, available.IgnoredNotice()) + + _, err = resolveTelemetryDestination(false, layered(shipped, map[string]string{"TELEMETRY_STATE": "absent"})) + require.ErrorContains(t, err, "TELEMETRY_ABSENT_REASON is not set") + + // The same answer from a local run is refused too: the local reason fills in + // a missing state, never an explicit `absent` that did not say why. + _, err = resolveTelemetryDestination(true, layered(shipped, map[string]string{"TELEMETRY_STATE": "absent"})) + require.ErrorContains(t, err, "TELEMETRY_ABSENT_REASON is not set") } func TestTelemetryDestinationMissingStateDefaultsOnlyInLocalRuntime(t *testing.T) { @@ -227,9 +265,9 @@ func TestTelemetryDestinationRefusesEverythingElse(t *testing.T) { } } -// The state decides. The cell's values override the module's local defaults one -// key at a time, so a deployed producer sees `available` and an endpoint from the -// cell beside the reason the local profile left behind. That is not a +// The state decides. The group is layered, so a deployed producer can still see +// `available` and an endpoint from the cell beside a reason a lower layer (a +// consuming workspace's own profile, say) left behind. That is not a // contradiction to refuse: the reason is ignored, and said so once. func TestTelemetryDestinationAvailableIgnoresALeftoverReason(t *testing.T) { destination, err := resolveTelemetryDestination(false, group(map[string]string{ @@ -304,16 +342,16 @@ func layered(layers ...map[string]string) func(string) string { return group(merged) } -// The case this exists for: the module's local profile defaults to `absent` with -// a reason, and a deployed cell overrides the state and the endpoint but not the +// The case this exists for: a lower layer of the group says `absent` with a +// reason, and a deployed cell overrides the state and the endpoint but not the // reason. The cell's answer wins, whichever way it points. -func TestTelemetryDestinationTheCellOverridesTheModuleDefaults(t *testing.T) { +func TestTelemetryDestinationTheCellOverridesALowerLayer(t *testing.T) { moduleDefault := map[string]string{ "TELEMETRY_STATE": "absent", "TELEMETRY_ABSENT_REASON": "A local run has no cell collector.", } - t.Run("a cell with a collector overrides the local absent default", func(t *testing.T) { + t.Run("a cell with a collector overrides a lower layer's absent default", func(t *testing.T) { destination, err := resolveTelemetryDestination(false, layered(moduleDefault, map[string]string{ "TELEMETRY_STATE": "available", "OTEL_EXPORTER_OTLP_ENDPOINT": collector, @@ -402,8 +440,8 @@ func TestConfiguredTelemetryDestinationReadsTheObservabilityGroup(t *testing.T) } // Through the SDK's lookup, with the keys exactly as a deployed cell delivers -// them after the layers merge: the cell's state and endpoint, and the reason the -// module's local profile left behind. +// them after the layers merge: the cell's state and endpoint, and a reason a +// lower layer left behind. func TestConfiguredTelemetryDestinationStateDecidesAcrossLayers(t *testing.T) { const prefix = "CODEFLY__WORKSPACE_CONFIGURATION__OBSERVABILITY__" t.Setenv(prefix+"TELEMETRY_STATE", "available") From db384ee7750c472d964194ed5dffb88fdaeed092 Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Wed, 7 Oct 2026 16:32:24 -0400 Subject: [PATCH 64/69] fix(audit): a store refusing every row is a store failure, and a refused details row leaves no events row (Refs #996) A warehouse that refuses every row of a write for one cause (a table changed outside the kit: BigQuery lists every row as `invalid`, ClickHouse refuses every block) was read as that many bad rows. The relay set each one aside, so the whole stream moved into the quarantine and reads showed gaps nobody was told of. business.PermanentRowRejection now carries a Reason: what the store says is wrong, in words that are the same for every row refused for the same cause and that name no row. business.RefusedForOneReason reads a refusal as a whole. When one write of two or more rows is refused in every row for one and the same non-empty Reason, appendRows sets nothing aside: the rows stay queued, the delivery fails with the reason, and the relay logs it, backs off and retries while the queue's depth and age grow until the relay-lag alert fires. A refusal that singles out some of the rows of a write is unchanged, and so is a row refused alone: one row cannot be told from a store that refuses everything, and treating it as the store's would wedge a relay whose batch holds one row behind a poison row. A rejection with no Reason is never alike to another, so a store that does not state one keeps its behaviour. The relay only sees a whole write, but a batch holds events and details rows in two tables, and a change to the details table refuses the content-class rows alone. So both adapters apply the same reading to each insert of their own (a BigQuery request, a ClickHouse insert into one table) and return it as a plain failure instead of naming rows. They state the Reason: BigQuery the row errors it listed, without the row's position; ClickHouse the exception code, or the driver's own words for a value it cannot encode. A value over the size a column holds is always the row's own and states none. Both adapters also wrote the events row before the details row. A content-class event keeps its details nowhere but the details table, so a refused details row left an events row with an empty payload in the table every read starts from, and the quarantine reason said the warehouse held nothing of the event. The details now go first, and an event gets its events row only once its details are accepted: BigQuery writes no events row in a call whose details were not all accepted, ClickHouse writes the events rows of the events whose details were not refused (withoutRefused). A details row left by an events row that is then refused is read by nothing, since every read starts from the events table. The relay's contract in ADR 0009, the accounts AGENTS.md, the relay runbook and FEATURES.md describe both rules. The live-server ClickHouse test, which this environment cannot run, now expects the details of events its events table refuses to be written first. Co-Authored-By: Claude Opus 5.5 --- module/FEATURES.md | 2 +- module/MEASUREMENT_RUNBOOKS.md | 22 +- module/docs/adr/0009-audit-store-swap.md | 12 +- module/services/accounts/AGENTS.md | 8 +- .../auditstore/bigquerystore/bigquerystore.go | 62 ++++- .../bigquerystore/bigquerystore_test.go | 18 +- .../bigquerystore/rejection_test.go | 163 ++++++++++++ .../clickhousestore/clickhousestore.go | 57 ++-- .../clickhousestore/clickhousestore_test.go | 3 +- .../auditstore/clickhousestore/rejection.go | 27 ++ .../clickhousestore/rejection_server_test.go | 5 +- .../clickhousestore/rejection_test.go | 191 +++++++++++++- .../accounts/code/pkg/business/audit_relay.go | 43 ++- .../audit_relay_refused_alike_test.go | 244 ++++++++++++++++++ .../code/pkg/business/audit_row_rejection.go | 43 +++ 15 files changed, 857 insertions(+), 43 deletions(-) create mode 100644 module/services/accounts/code/pkg/business/audit_relay_refused_alike_test.go diff --git a/module/FEATURES.md b/module/FEATURES.md index a3980a6d2..050797f81 100644 --- a/module/FEATURES.md +++ b/module/FEATURES.md @@ -398,7 +398,7 @@ see `JOBS.md` for the exact boundary and sequencing. | Export (JSON/CSV) | ✅ | `audit_export.go`; under a warehouse store of record one export holds at most 32 MiB of event text, past which it fails with `ErrAuditExportTooLarge` (`ResourceExhausted`): narrow it by actor or event type | | Impersonation tracking | ✅ | Records both real actor + viewed-as user | | Replay / event sourcing| ❌ | Audit log is read-only history; not used to reconstruct state | -| Warehouse store of record | 🟡 | `AUDIT_SINK=bigquery` or `AUDIT_SINK=clickhouse` (ADR 0009): each record commits to a Postgres queue in the mutation's transaction, then a relay delivers it to the warehouse and a locked GCS or S3 archive. Reads are scoped to the organization and deployment and deduplicated by event id; BigQuery uses the Storage Read API without query jobs, and ClickHouse uses SQL with service evaluation where needed. A BigQuery read holds a bounded amount in memory whatever the history: a window that needs more than its byte budget is read again as halves. What an aggregation keeps across the windows (its buckets, percentile samples and distinct values) is counted too, and one that keeps more than 32 MiB fails with `ErrAuditAggregateTooLarge` (`ResourceExhausted`), never with a partial answer: narrow it by time range, actor or event type, or group by fewer dimensions. A row the warehouse itself reports it refuses for good (never one that merely failed while the warehouse was flaky) is set aside, whole, in `audit_event_quarantine` instead of holding every organization's events behind it (`saas.audit_queue.quarantined`; replaying quarantined rows is not built yet). The shared parity suite compares both with Postgres. `audit-history-copy` verifies a switching deployment's copied history before a confirmed monthly-partition drop. No Azure Blob archive writer exists yet. | +| Warehouse store of record | 🟡 | `AUDIT_SINK=bigquery` or `AUDIT_SINK=clickhouse` (ADR 0009): each record commits to a Postgres queue in the mutation's transaction, then a relay delivers it to the warehouse and a locked GCS or S3 archive. Reads are scoped to the organization and deployment and deduplicated by event id; BigQuery uses the Storage Read API without query jobs, and ClickHouse uses SQL with service evaluation where needed. A BigQuery read holds a bounded amount in memory whatever the history: a window that needs more than its byte budget is read again as halves. What an aggregation keeps across the windows (its buckets, percentile samples and distinct values) is counted too, and one that keeps more than 32 MiB fails with `ErrAuditAggregateTooLarge` (`ResourceExhausted`), never with a partial answer: narrow it by time range, actor or event type, or group by fewer dimensions. A row the warehouse itself reports it refuses for good (never one that merely failed while the warehouse was flaky) is set aside, whole, in `audit_event_quarantine` instead of holding every organization's events behind it (`saas.audit_queue.quarantined`; replaying quarantined rows is not built yet). A write the warehouse refuses in every row for one and the same reason (a table changed outside the kit) is not a set of bad rows: nothing is set aside, the rows stay queued and the delivery is retried, so the queue's depth and age raise the relay-lag alert instead of the quarantine silently taking the stream. The shared parity suite compares both with Postgres. `audit-history-copy` verifies a switching deployment's copied history before a confirmed monthly-partition drop. No Azure Blob archive writer exists yet. | ### Admin panel diff --git a/module/MEASUREMENT_RUNBOOKS.md b/module/MEASUREMENT_RUNBOOKS.md index 1b7608cef..5483f8aa0 100644 --- a/module/MEASUREMENT_RUNBOOKS.md +++ b/module/MEASUREMENT_RUNBOOKS.md @@ -70,6 +70,21 @@ recognize. A flapping or unreachable warehouse grows `saas.audit_queue.depth` an `oldest_age`, never the quarantine. The relay draws no conclusion from what else the warehouse accepted or from how often a row failed. +One thing it does read from a refusal as a whole. A write the warehouse refuses +in **every** row, for one and the same reason, is not a set of bad rows: a table +was changed under the relay (a column added, dropped or retyped outside the kit) +and refuses whatever it is sent. Setting those rows aside would move the whole +stream into the quarantine and leave reads with gaps nobody was told of. So the +relay sets nothing aside: the rows stay queued, the delivery fails with the +reason in its error (`audit relay delivery failed; the rows stay queued and are +retried`, and an error-level `audit store refused every row of a write for one +reason`), and the relay retries with backoff while `depth` and `oldest_age` grow +until `audit_relay_lag` fires. Read the reason, put the table back, and the rows +are delivered as they are. The adapters apply the same reading to each insert of +their own (a BigQuery request, a ClickHouse insert into one table), so a change +to the details table alone is caught too. It takes two rows: one row refused +alone cannot be told from a warehouse that refuses everything, and is a row. + What the adapters report as a permanent refusal: - **BigQuery**: a row that `insertAll` lists with the reason `invalid` (a value @@ -87,7 +102,12 @@ What the adapters report as a permanent refusal: retried and never searched. When a write names refused rows, the relay sets exactly those aside and writes the -rest again, so a batch of one is judged as a batch of five thousand. The archive +rest again, so a batch of one is judged as a batch of five thousand. Both adapters +write a batch's details before its events, and write an event's events row only +once its details are accepted: a content-class event keeps its details nowhere +but the details table, so an events row beside a refused details row would be an +event with an empty payload in the table every read starts from. A refused event +therefore has no events row in the warehouse. The archive is written once, whole, before the warehouse is tried, so a row the warehouse refused is already in the archive. A row whose own details cannot be serialized is the exception: the archive's line for an event carries the hash of its canonical diff --git a/module/docs/adr/0009-audit-store-swap.md b/module/docs/adr/0009-audit-store-swap.md index e5abbaa99..0752b089a 100644 --- a/module/docs/adr/0009-audit-store-swap.md +++ b/module/docs/adr/0009-audit-store-swap.md @@ -140,8 +140,16 @@ Postgres transaction. failure — a transport or server error, a throttle, a quota, a timeout, a failed quorum, an error the store does not recognize — is retried with backoff and sets nothing aside, and the relay infers nothing from how often a - row failed or from what else the store accepted. After a write that names - refused rows, the relay sets those aside and writes the rest again; + row failed. After a write that names refused rows, the relay sets those aside + and writes the rest again. One refusal is read as a whole and not taken at + its word: a write of two or more rows refused in every row for one and the + same reason (`PermanentRowRejection.Reason`) is a table the store cannot + take, not a set of bad rows, so nothing is set aside, the rows stay queued + and the delivery is retried with backoff, and the queue's depth and age raise + the relay-lag alert. The adapters read each insert of their own the same way, + and write a batch's details before its events, giving an event an events row + only once its details are accepted, so that a refused event is not left in + the events table with an empty payload; - a set-aside row does not stall the queue. It moves, whole, in the same transaction that deletes the delivered rows, to an `audit_event_quarantine` table beside the queue, and is counted in `saas.audit_queue.quarantined`. diff --git a/module/services/accounts/AGENTS.md b/module/services/accounts/AGENTS.md index e88afdd38..571b90309 100644 --- a/module/services/accounts/AGENTS.md +++ b/module/services/accounts/AGENTS.md @@ -857,7 +857,13 @@ reconnected with a PAT or through the App, all reach. cannot be serialized; every other failure — transport, 5xx, 429, quota, timeout, quorum or replica, or an error the adapter cannot classify — is retried and sets nothing aside, and no decision is drawn from what else the - warehouse accepted. The reason starts with its class + warehouse accepted. The one exception reads the write as a whole: a write of + two or more rows refused in every row for one and the same + `PermanentRowRejection.Reason` (`business.RefusedForOneReason`) is a store that + cannot take them, not rows at fault, so nothing is set aside and the relay + retries; the adapters apply the same rule to each of their inserts, and write + a batch's details before its events so a refused event leaves no events row. + The reason starts with its class (`warehouse_rejected_archived` or `unserializable_not_archived`) and is bounded valid text. The move from queue to quarantine is one statement, and nothing in the kit ever deletes a queue or quarantine row (the down migrations refuse while either holds rows). (3) The queue is observed diff --git a/module/services/accounts/code/pkg/auditstore/bigquerystore/bigquerystore.go b/module/services/accounts/code/pkg/auditstore/bigquerystore/bigquerystore.go index 8f341f353..f84c85fd6 100644 --- a/module/services/accounts/code/pkg/auditstore/bigquerystore/bigquerystore.go +++ b/module/services/accounts/code/pkg/auditstore/bigquerystore/bigquerystore.go @@ -351,9 +351,19 @@ func BatchRows(batch business.AuditBatch) (events, details []Row) { } // AppendAuditBatch implements business.AuditStoreWriter: the batch's rows -// (BatchRows) streamed to the events and details tables, each table in as few +// (BatchRows) streamed to the details and events tables, each table in as few // requests as hold at most maxRowsPerInsert rows and maxRequestBytes bytes. // +// The details go first and the events only once every details row is accepted, +// because a content-class event keeps its details nowhere but the details table: +// an events row written beside a details row BigQuery then refuses would be an +// event with an empty payload, in a table every read starts from, for an event +// the relay is about to set aside. A details row whose events row then fails is +// harmless instead — no read starts from the details table — and the relay +// delivers the batch again. So when the details are not all accepted, no events +// row is written in this call: the events of the rows that were accepted follow +// when the relay sends the batch again without the rows refused. +// // What a failure says about the batch is what the relay acts on, so the error // is classified here, where BigQuery's answer is known: // @@ -366,6 +376,11 @@ func BatchRows(batch business.AuditBatch) (events, details []Row) { // is wrong for its column, and sending it again gives the same answer. The // batch's other rows, which BigQuery reports as "stopped", are not refused; // the relay sends them again without the refused rows. +// - A request every row of which BigQuery refuses as invalid for one and the +// same reason names no row: a table changed under the writer refuses every +// row alike, and the rows are as deliverable as they were. It is returned as +// a failure of the request, which the relay retries, so that the whole stream +// is not set aside for a cause that is not in it (business.RefusedForOneReason). // - Every other failure is returned as it came — a request that failed (any // HTTP error, a throttle, a quota, a timeout, a dead backend), a row // reported with any other reason, an error of a kind not known here — and @@ -377,12 +392,12 @@ func (s *Store) AppendAuditBatch(ctx context.Context, batch business.AuditBatch) if refused := append(eventsRefused, detailsRefused...); len(refused) > 0 { return errors.Join(refused...) } - if err := put(ctx, s.events, events); err != nil { - return fmt.Errorf("bigquery audit store: append to %s: %w", EventsTable, err) - } if err := put(ctx, s.details, details); err != nil { return fmt.Errorf("bigquery audit store: append to %s: %w", DetailsTable, err) } + if err := put(ctx, s.events, events); err != nil { + return fmt.Errorf("bigquery audit store: append to %s: %w", EventsTable, err) + } return nil } @@ -435,7 +450,7 @@ func planRequests(table string, rows []Row) (requests [][]bigquery.ValueSaver, r func put(ctx context.Context, inserter rowInserter, requests [][]bigquery.ValueSaver) error { for _, request := range requests { if err := inserter.Put(ctx, request); err != nil { - return classifyPutError(err) + return classifyPutError(err, request) } } return nil @@ -448,13 +463,17 @@ func put(ctx context.Context, inserter rowInserter, requests [][]bigquery.ValueS // "quotaExceeded" — say nothing against the row. const reasonInvalid = "invalid" -// classifyPutError turns what Put returned into what the relay needs. A -// PutMultiError lists every row BigQuery refused with the reasons it gave; each -// row it gave the reason "invalid" is a *business.PermanentRowRejection, and the -// error also carries the original, so the rows that were merely stopped, and any -// that failed for another reason, are described and retried. Every other error is -// the request's, not a row's, and is returned as it came. -func classifyPutError(err error) error { +// classifyPutError turns what Put returned for request into what the relay +// needs. A PutMultiError lists every row BigQuery refused with the reasons it +// gave; each row it gave the reason "invalid" is a *business.PermanentRowRejection, +// and the error also carries the original, so the rows that were merely stopped, +// and any that failed for another reason, are described and retried. Every other +// error is the request's, not a row's, and is returned as it came. +// +// When every row of a request of two or more is refused as invalid for one and +// the same reason, that is the table's, not the rows': none of them is named, so +// the relay retries the batch instead of setting every row aside. +func classifyPutError(err error, request []bigquery.ValueSaver) error { var multi bigquery.PutMultiError if !errors.As(err, &multi) || len(multi) == 0 { return err @@ -467,12 +486,29 @@ func classifyPutError(err error) error { } refused = append(refused, &business.PermanentRowRejection{ EventID: row.InsertID, - Cause: fmt.Errorf("bigquery audit store: the row at index %d was refused as invalid: %v", row.RowIndex, row.Errors), + Cause: fmt.Errorf("bigquery audit store: the row was refused as invalid: %v", row.Errors), + // What BigQuery said of the row, which for a table that changed is the + // same for every row: the row's place in the request is not part of it. + Reason: fmt.Sprint(row.Errors), }) } + if reason, whole := business.RefusedForOneReason(errors.Join(refused...), insertIDs(request)); whole { + return fmt.Errorf("bigquery audit store: every one of the %d rows of a request was refused as invalid for one reason, which is the table's and not the rows': %s: %w", len(request), reason, failures[0]) + } return errors.Join(append(refused, failures...)...) } +// insertIDs is the insert id of each row of a request, which is its event id. +func insertIDs(request []bigquery.ValueSaver) []string { + ids := make([]string, 0, len(request)) + for _, saver := range request { + if row, ok := saver.(Row); ok { + ids = append(ids, row.InsertID) + } + } + return ids +} + // rowIsInvalid reports whether BigQuery gave the row the reason "invalid". func rowIsInvalid(row bigquery.RowInsertionError) bool { for _, rowErr := range row.Errors { diff --git a/module/services/accounts/code/pkg/auditstore/bigquerystore/bigquerystore_test.go b/module/services/accounts/code/pkg/auditstore/bigquerystore/bigquerystore_test.go index 76153aa7b..d73e4b106 100644 --- a/module/services/accounts/code/pkg/auditstore/bigquerystore/bigquerystore_test.go +++ b/module/services/accounts/code/pkg/auditstore/bigquerystore/bigquerystore_test.go @@ -224,7 +224,7 @@ func TestAppendAuditBatchRefusesARowThatCannotFitARequestAndSendsNothing(t *test func TestAppendAuditBatchFailsWholeAndNamesTheRejectedRow(t *testing.T) { rejected := bigquery.PutMultiError{{InsertID: "event-7", RowIndex: 7, Errors: bigquery.MultiError{errors.New("no such field: extra")}}} - events, details := &fakeInserter{fail: rejected}, &fakeInserter{} + events, details := &fakeInserter{}, &fakeInserter{fail: rejected} store := &Store{events: events, details: details} err := store.AppendAuditBatch(context.Background(), business.AuditBatch{ @@ -233,7 +233,21 @@ func TestAppendAuditBatchFailsWholeAndNamesTheRejectedRow(t *testing.T) { }) require.ErrorContains(t, err, "row 7, insert id event-7") require.ErrorContains(t, err, "no such field: extra") - require.Empty(t, details.requests, "a failed events append stops the batch; the relay retries it whole") + require.Empty(t, events.requests, "a failed details append stops the batch before any events row is written; the relay retries it whole") +} + +func TestAppendAuditBatchFailsWholeWhenTheEventsAppendFails(t *testing.T) { + events, details := &fakeInserter{fail: errors.New("backendError")}, &fakeInserter{} + store := &Store{events: events, details: details} + content := record(t, business.RetentionContent, "") + + err := store.AppendAuditBatch(context.Background(), business.AuditBatch{ + DeploymentID: "deployment-1", Records: []business.AuditRecord{content}, + }) + + require.ErrorContains(t, err, "backendError") + require.ErrorContains(t, err, EventsTable) + require.Equal(t, []string{content.Entry.ID}, details.ids(), "the details were accepted first; with no events row they are read by nothing, and the relay sends the batch again") } func TestTablesArePartitionedByDayOnOccurredAtAndClusteredByOrganization(t *testing.T) { diff --git a/module/services/accounts/code/pkg/auditstore/bigquerystore/rejection_test.go b/module/services/accounts/code/pkg/auditstore/bigquerystore/rejection_test.go index a8dc4c66e..c1023af80 100644 --- a/module/services/accounts/code/pkg/auditstore/bigquerystore/rejection_test.go +++ b/module/services/accounts/code/pkg/auditstore/bigquerystore/rejection_test.go @@ -4,6 +4,7 @@ import ( "context" "errors" "fmt" + "slices" "testing" "accounts/pkg/business" @@ -177,6 +178,7 @@ func TestARowRefusedInTheDetailsTableIsARefusalOfItsEvent(t *testing.T) { require.Equal(t, []string{ids[2]}, refusedIDs(err)) require.ErrorContains(t, err, DetailsTable) + require.Empty(t, events.requests, "no events row is written when a details row is refused: it would be an event with an empty payload") } func TestEveryRowThatCannotFitARequestIsARefusalAndNothingIsSent(t *testing.T) { @@ -195,3 +197,164 @@ func TestEveryRowThatCannotFitARequestIsARefusalAndNothingIsSent(t *testing.T) { require.Empty(t, events.requests) require.Empty(t, details.requests) } + +// orderedInserter is a scriptedInserter that records, in a log shared with the +// other table's, which table each request went to. +type orderedInserter struct { + scriptedInserter + table string + log *[]string +} + +func (o *orderedInserter) Put(ctx context.Context, src any) error { + *o.log = append(*o.log, o.table) + return o.scriptedInserter.Put(ctx, src) +} + +func (s *scriptedInserter) ids() []string { + var ids []string + for _, request := range s.requests { + for _, row := range request { + ids = append(ids, row.InsertID) + } + } + return ids +} + +func TestTheDetailsAreWrittenBeforeTheEventsTheyBelongTo(t *testing.T) { + var log []string + events := &orderedInserter{table: EventsTable, log: &log} + details := &orderedInserter{table: DetailsTable, log: &log} + store := &Store{events: events, details: details} + records, ids := batchOf(t, 3, business.RetentionContent) + security := record(t, business.RetentionSecurity, "") + + require.NoError(t, store.AppendAuditBatch(context.Background(), business.AuditBatch{ + DeploymentID: "deployment-1", Records: append(records, security), + })) + + require.Equal(t, []string{DetailsTable, EventsTable}, log, "a content event's events row is written only once its details are accepted") + require.Equal(t, ids, details.ids()) + require.Equal(t, append(slices.Clone(ids), security.Entry.ID), events.ids(), "and every event, the security one included, then has its events row") +} + +// A details row BigQuery refuses leaves no events row of its event: the events +// table is where every read starts, and an event there with no details is an +// event with an empty payload whose quarantine reason says the warehouse holds +// nothing of it. +func TestARefusedDetailsRowLeavesNoEventsRowOfItsEvent(t *testing.T) { + records, ids := batchOf(t, 4, business.RetentionContent) + security := record(t, business.RetentionSecurity, "") + details := &scriptedInserter{answer: func(request []Row) error { + reasons := map[int]string{} + for i := range request { + reasons[i] = "stopped" + } + reasons[1] = "invalid" + return insertErrors(request, reasons) + }} + events := &scriptedInserter{} + store := &Store{events: events, details: details} + + err := store.AppendAuditBatch(context.Background(), business.AuditBatch{ + DeploymentID: "deployment-1", Records: append(records, security), + }) + + require.Equal(t, []string{ids[1]}, refusedIDs(err)) + require.Empty(t, events.requests, "nothing of the batch reaches the events table until its details are all accepted") + + // The relay sends the rest again without the refused row, and every events row + // of it is then written, security-class ones included. + details.answer = nil + remaining := append(slices.Clone(records[:1]), records[2:]...) + require.NoError(t, store.AppendAuditBatch(context.Background(), business.AuditBatch{ + DeploymentID: "deployment-1", Records: append(remaining, security), + })) + require.NotContains(t, events.ids(), ids[1], "the refused event never has an events row") + require.Len(t, events.ids(), 4) +} + +// A table changed under the writer refuses every row alike. That is not a set of +// bad rows, and naming each of them would have the relay set the whole stream +// aside. +func TestARequestEveryRowOfWhichIsRefusedForOneReasonNamesNoRow(t *testing.T) { + for _, table := range []string{"details", "events"} { + records, _ := batchOf(t, 6, business.RetentionContent) + refuse := &scriptedInserter{answer: func(request []Row) error { + reasons := map[int]string{} + for i := range request { + reasons[i] = "invalid" + } + return insertErrors(request, reasons) + }} + events, details := &scriptedInserter{}, &scriptedInserter{} + if table == "events" { + events = refuse + } else { + details = refuse + } + store := &Store{events: events, details: details} + + err := store.AppendAuditBatch(context.Background(), business.AuditBatch{DeploymentID: "deployment-1", Records: records}) + + require.Error(t, err, table) + require.Empty(t, refusedIDs(err), "%s: no row is named, so the relay retries the batch and sets nothing aside", table) + require.ErrorContains(t, err, "every one of the 6 rows of a request was refused as invalid for one reason", table) + require.ErrorContains(t, err, "cannot parse invalid", "%s: the reason BigQuery gave is kept", table) + } +} + +func TestARequestRefusedRowByRowForDifferentReasonsNamesEachRow(t *testing.T) { + records, ids := batchOf(t, 3, business.RetentionContent) + details := &scriptedInserter{answer: func(request []Row) error { + var multi bigquery.PutMultiError + for i, row := range request { + multi = append(multi, bigquery.RowInsertionError{ + InsertID: row.InsertID, RowIndex: i, + Errors: bigquery.MultiError{&bigquery.Error{Reason: "invalid", Location: "details", Message: "invalid value " + row.InsertID}}, + }) + } + return multi + }} + store := &Store{events: &scriptedInserter{}, details: details} + + err := store.AppendAuditBatch(context.Background(), business.AuditBatch{DeploymentID: "deployment-1", Records: records}) + + require.Equal(t, ids, refusedIDs(err), "each row has its own reason, so each row is the one at fault") +} + +func TestOneRowRefusedAloneIsARefusalOfThatRowWhateverTheReason(t *testing.T) { + records, ids := batchOf(t, 1, business.RetentionContent) + details := &scriptedInserter{answer: func(request []Row) error { + return insertErrors(request, map[int]string{0: "invalid"}) + }} + store := &Store{events: &scriptedInserter{}, details: details} + + err := store.AppendAuditBatch(context.Background(), business.AuditBatch{DeploymentID: "deployment-1", Records: records}) + + require.Equal(t, ids, refusedIDs(err), "a request of one row cannot be told from a table that refuses everything") +} + +// The reason is what BigQuery said of the row and nothing of where the row sat in +// the request, so that two rows refused for one cause have one reason. +func TestTheReasonOfARefusedRowIsTheSameForRowsRefusedForOneCause(t *testing.T) { + records, _ := batchOf(t, 5, business.RetentionContent) + details := &scriptedInserter{answer: func(request []Row) error { + reasons := map[int]string{} + for i := range request { + reasons[i] = "stopped" + } + reasons[1], reasons[3] = "invalid", "invalid" + return insertErrors(request, reasons) + }} + store := &Store{events: &scriptedInserter{}, details: details} + + err := store.AppendAuditBatch(context.Background(), business.AuditBatch{DeploymentID: "deployment-1", Records: records}) + + rejections := business.PermanentRowRejections(err) + require.Len(t, rejections, 2) + require.NotEmpty(t, rejections[0].Reason) + require.Equal(t, rejections[0].Reason, rejections[1].Reason) + require.NotContains(t, rejections[0].Reason, rejections[0].EventID) + require.NotContains(t, rejections[0].Reason, "index") +} diff --git a/module/services/accounts/code/pkg/auditstore/clickhousestore/clickhousestore.go b/module/services/accounts/code/pkg/auditstore/clickhousestore/clickhousestore.go index 2fe93eeea..7e244d9ef 100644 --- a/module/services/accounts/code/pkg/auditstore/clickhousestore/clickhousestore.go +++ b/module/services/accounts/code/pkg/auditstore/clickhousestore/clickhousestore.go @@ -565,29 +565,39 @@ func DetailRow(deploymentID string, record business.AuditRecord) []any { } // AppendAuditBatch implements business.AuditStoreWriter: the batch's rows -// (BatchRows) as one insert into the events table and, when the batch holds -// content-class events, one into the details table. Each insert is one block, -// written atomically; a failure between the two leaves the events without -// their details until the relay delivers the batch again, and every read -// returns each event once by event id. +// (BatchRows) as one insert into the details table, when the batch holds +// content-class events, and then one into the events table. Each insert is one +// block, written atomically. +// +// The details go first, and an event whose details row is refused is given no +// events row, because a content-class event keeps its details nowhere but the +// details table: an events row written beside a refused details row would be an +// event with an empty payload, in the table every read starts from, for an event +// the relay is about to set aside. A details row whose events row then fails is +// harmless instead — no read starts from the details table — and the relay +// delivers the batch again. A failure between the two inserts leaves details +// without events until then, and every read returns each event once by event id. // // What a failure says about the batch is what the relay acts on, so it is // classified here (rejection.go): a row ClickHouse refuses for its own content is // returned as a business.PermanentRowRejection naming its event, found by -// splitting the refused block down to single rows; every other failure is -// returned as it came and the relay retries the batch whole. The rows around a -// refused one are written while it is found, so the relay's retry of them is a -// redelivery, which every read discards. +// splitting the refused block down to single rows; an insert every row of which +// is refused for one and the same reason names no row and is returned as a +// failure, because that is a table changed under the writer and not a set of bad +// rows (business.RefusedForOneReason); every other failure is returned as it came +// and the relay retries the batch whole. The rows around a refused one are +// written while it is found, so the relay's retry of them is a redelivery, which +// every read discards. func (s *Store) AppendAuditBatch(ctx context.Context, batch business.AuditBatch) error { events, details := BatchRows(batch) - refused, err := s.insert(ctx, EventsTable, EventsColumns(), events) + refused, err := s.insert(ctx, DetailsTable, DetailsColumns(), details) if err != nil { return failure(refused, err) } - // An event whose events row was refused is not given a details row: the relay - // sets it aside, and its details would outlive the event they belong to. - details = withoutRefused(details, refused) - more, err := s.insert(ctx, DetailsTable, DetailsColumns(), details) + // An event whose details row was refused is not given an events row: the relay + // sets it aside, and its events row would be an event with no payload. + events = withoutRefused(events, refused) + more, err := s.insert(ctx, EventsTable, EventsColumns(), events) return failure(append(refused, more...), err) } @@ -619,7 +629,8 @@ func withoutRefused(rows [][]any, refused []error) [][]any { // insert writes rows to table. It returns the rows ClickHouse refused for their // own content, as business.PermanentRowRejection, and the failure that stopped // it when the rest could not be written; the rows it did not refuse are -// written when it returns no failure. +// written when it returns no failure. When every row is refused for one and the +// same reason it returns that as a failure and names no row. func (s *Store) insert(ctx context.Context, table string, columns []Column, rows [][]any) ([]error, error) { if len(rows) == 0 { return nil, nil @@ -627,7 +638,21 @@ func (s *Store) insert(ctx context.Context, table string, columns []Column, rows if len(s.writeSettings) > 0 { ctx = clickhouse.Context(ctx, clickhouse.WithSettings(s.writeSettings)) } - return s.isolate(ctx, table, columns, rows) + refused, err := s.isolate(ctx, table, columns, rows) + if err != nil { + return refused, err + } + // Every row refused for one reason is the table's refusal, not the rows': a + // column changed under the writer refuses them all alike. None is named, so the + // relay retries instead of setting the whole stream aside. + ids := make([]string, len(rows)) + for i, row := range rows { + ids[i] = fmt.Sprint(row[0]) + } + if reason, whole := business.RefusedForOneReason(errors.Join(refused...), ids); whole { + return nil, fmt.Errorf("clickhouse audit store: %s refused every one of its %d rows for one reason, which is the table's and not the rows': %s", table, len(rows), reason) + } + return refused, nil } // isolate sends rows as one block. When the block is refused it finds the rows diff --git a/module/services/accounts/code/pkg/auditstore/clickhousestore/clickhousestore_test.go b/module/services/accounts/code/pkg/auditstore/clickhousestore/clickhousestore_test.go index d39cb7e3b..810456749 100644 --- a/module/services/accounts/code/pkg/auditstore/clickhousestore/clickhousestore_test.go +++ b/module/services/accounts/code/pkg/auditstore/clickhousestore/clickhousestore_test.go @@ -249,7 +249,8 @@ func TestAppendAuditBatchIsOneInsertPerTable(t *testing.T) { require.NoError(t, store.AppendAuditBatch(context.Background(), batch)) require.Len(t, conn.batches, 2, "one insert per table for the whole relay batch, never one per row") - events, details := conn.batches[0], conn.batches[1] + details, events := conn.batches[0], conn.batches[1] + require.Contains(t, details.query, "audit_event_details", "the details insert goes first: a content event's events row follows only once its details are accepted") require.Equal(t, "INSERT INTO audit_events (event_id, deployment_id, org_id, actor_id, actor_type, event_type, schema_version, "+ "resource, resource_id, occurred_at, ip_address, impersonated_by, is_impersonated, client_id, retention_class, details_sha256, details)", events.query) require.Equal(t, "INSERT INTO audit_event_details (event_id, deployment_id, org_id, event_type, occurred_at, details_sha256, details)", details.query) diff --git a/module/services/accounts/code/pkg/auditstore/clickhousestore/rejection.go b/module/services/accounts/code/pkg/auditstore/clickhousestore/rejection.go index 1dcccd855..2055c43b8 100644 --- a/module/services/accounts/code/pkg/auditstore/clickhousestore/rejection.go +++ b/module/services/accounts/code/pkg/auditstore/clickhousestore/rejection.go @@ -77,5 +77,32 @@ func refusal(table string, row []any, cause error) error { return &business.PermanentRowRejection{ EventID: fmt.Sprint(row[0]), Cause: fmt.Errorf("clickhouse audit store: %s refused the row: %w", table, cause), + Reason: refusalReason(table, cause), } } + +// refusalReason is what was wrong with a refused row in words that do not depend +// on which row it was, so that one cause refusing every row of an insert reads as +// one reason: the server's exception code, or the driver's own words for the +// value it cannot encode, never the event or the value. Empty when the refusal +// is of a kind a table's change does not explain — a value over the size a +// column holds is the row's own — which keeps such a row judged as a row. +func refusalReason(table string, cause error) string { + var exception *clickhouse.Exception + if errors.As(cause, &exception) { + name, defect := rowDefectCodes[exception.Code] + if !defect || exception.Code == codeTooLargeString { + return "" + } + return fmt.Sprintf("%s: %s (code %d)", table, name, exception.Code) + } + var block *proto.BlockError + if errors.As(cause, &block) && block.Op == "AppendRow" { + return fmt.Sprintf("%s: %s", table, block.Error()) + } + return "" +} + +// codeTooLargeString is TOO_LARGE_STRING_SIZE, the one row-defect code that is +// about the size of one value and never about the shape of the table. +const codeTooLargeString = 131 diff --git a/module/services/accounts/code/pkg/auditstore/clickhousestore/rejection_server_test.go b/module/services/accounts/code/pkg/auditstore/clickhousestore/rejection_server_test.go index 9c53f7c2b..a8db74c54 100644 --- a/module/services/accounts/code/pkg/auditstore/clickhousestore/rejection_server_test.go +++ b/module/services/accounts/code/pkg/auditstore/clickhousestore/rejection_server_test.go @@ -42,7 +42,10 @@ func TestServerIsolatesARowTheTableRefusesAndNothingElse(t *testing.T) { require.NoError(t, store.queryRow(ctx, "SELECT count() FROM "+EventsTable, nil, &events)) require.NoError(t, store.queryRow(ctx, "SELECT count() FROM "+DetailsTable, nil, &details)) require.EqualValues(t, 5, events, "the rows around the refused ones are written") - require.EqualValues(t, 5, details) + // The details go first, so those of the two events the events table then refused + // are already there: read by nothing, since no read starts from the details + // table, and held by the archive and the quarantine as well. + require.EqualValues(t, 7, details) // A failure that is not about a row: a table that is gone refuses nothing. require.NoError(t, conn.Exec(ctx, "DROP TABLE "+DetailsTable)) diff --git a/module/services/accounts/code/pkg/auditstore/clickhousestore/rejection_test.go b/module/services/accounts/code/pkg/auditstore/clickhousestore/rejection_test.go index e3ef4315d..6e7fcf6c4 100644 --- a/module/services/accounts/code/pkg/auditstore/clickhousestore/rejection_test.go +++ b/module/services/accounts/code/pkg/auditstore/clickhousestore/rejection_test.go @@ -31,6 +31,8 @@ type blameConn struct { // the server was sent. stored map[string][]string sends int + // order is the table of each block the server was sent, in order. + order []string } func (c *blameConn) PrepareBatch(_ context.Context, query string, _ ...driver.PrepareBatchOption) (driver.Batch, error) { @@ -68,6 +70,7 @@ func (b *blameBatch) Append(values ...any) error { func (b *blameBatch) Send() error { b.conn.sends++ + b.conn.order = append(b.conn.order, b.table) for _, row := range b.rows { if b.conn.answer == nil { continue @@ -150,10 +153,11 @@ func TestSeveralRowsRefusedForTheirContentAreAllFound(t *testing.T) { err = store.AppendAuditBatch(context.Background(), business.AuditBatch{DeploymentID: "deployment-1", Records: records}) require.ElementsMatch(t, bad, refusedIDs(err)) - require.Len(t, conn.ids(EventsTable), 16) require.Len(t, conn.ids(DetailsTable), 16, "the details of the rows that were not refused are written") + require.Len(t, conn.ids(EventsTable), 16, "and so are their events") for _, id := range bad { - require.NotContains(t, conn.ids(DetailsTable), id, "an event refused in the events table is not given details") + require.NotContains(t, conn.ids(DetailsTable), id, "a refused row is not written") + require.NotContains(t, conn.ids(EventsTable), id, "and its event is given no events row") } } @@ -289,3 +293,186 @@ func (c *encodingConn) PrepareBatch(ctx context.Context, query string, opts ...d inner.(*blameBatch).unencodable = c.unencodable return inner, nil } + +func TestTheDetailsAreWrittenBeforeTheEventsTheyBelongTo(t *testing.T) { + records, _ := recordsOf(t, 5, business.RetentionContent) + conn := &blameConn{} + store, err := New(conn, validConfig()) + require.NoError(t, err) + + require.NoError(t, store.AppendAuditBatch(context.Background(), business.AuditBatch{DeploymentID: "deployment-1", Records: records})) + + require.Equal(t, []string{DetailsTable, EventsTable}, conn.order, "a content event's events row is written only once its details are accepted") +} + +// A details row ClickHouse refuses leaves no events row of its event: the events +// table is where every read starts, and an event there with no details is an +// event with an empty payload whose quarantine reason says the warehouse holds +// nothing of it. +func TestARefusedDetailsRowLeavesNoEventsRowOfItsEvent(t *testing.T) { + records, ids := recordsOf(t, 8, business.RetentionContent) + security := record(t, business.RetentionSecurity, "") + conn := &blameConn{answer: func(table, id string) error { + if table == DetailsTable && id == ids[3] { + return exception(41) + } + return nil + }} + store, err := New(conn, validConfig()) + require.NoError(t, err) + + err = store.AppendAuditBatch(context.Background(), business.AuditBatch{DeploymentID: "deployment-1", Records: append(records, security)}) + + require.Equal(t, []string{ids[3]}, refusedIDs(err)) + require.NotContains(t, conn.ids(EventsTable), ids[3], "the refused event has no events row") + require.NotContains(t, conn.ids(DetailsTable), ids[3]) + want := append(slices.Delete(slices.Clone(ids), 3, 4), security.Entry.ID) + sort.Strings(want) + require.Equal(t, want, conn.ids(EventsTable), "every other event, the security one included, has its events row") +} + +// A details insert that fails for a reason that says nothing of a row stops the +// call before any events row is written. +func TestAFailedDetailsInsertWritesNoEventsRow(t *testing.T) { + records, _ := recordsOf(t, 4, business.RetentionContent) + conn := &blameConn{answer: func(table, _ string) error { + if table == DetailsTable { + return exception(285) + } + return nil + }} + store, err := New(conn, validConfig()) + require.NoError(t, err) + + err = store.AppendAuditBatch(context.Background(), business.AuditBatch{DeploymentID: "deployment-1", Records: records}) + + require.Error(t, err) + require.Empty(t, refusedIDs(err)) + require.Empty(t, conn.ids(EventsTable)) +} + +// A table changed under the writer refuses every row alike. That is not a set of +// bad rows, and naming each of them would have the relay set the whole stream +// aside. +func TestAnInsertEveryRowOfWhichIsRefusedForOneReasonNamesNoRow(t *testing.T) { + for _, tc := range []struct { + name string + table string + class business.AuditRetentionClass + refuse error + }{ + {"details, parse defect", DetailsTable, business.RetentionContent, exception(41)}, + {"details, type mismatch", DetailsTable, business.RetentionContent, exception(53)}, + {"events, parse defect", EventsTable, business.RetentionSecurity, exception(41)}, + {"events, constraint", EventsTable, business.RetentionSecurity, exception(469)}, + } { + records, _ := recordsOf(t, 6, tc.class) + conn := &blameConn{answer: func(table, _ string) error { + if table == tc.table { + return tc.refuse + } + return nil + }} + store, err := New(conn, validConfig()) + require.NoError(t, err) + + err = store.AppendAuditBatch(context.Background(), business.AuditBatch{DeploymentID: "deployment-1", Records: records}) + + require.Error(t, err, tc.name) + require.Empty(t, refusedIDs(err), "%s: no row is named, so the relay retries the batch and sets nothing aside", tc.name) + require.ErrorContains(t, err, "refused every one of its 6 rows for one reason", tc.name) + require.ErrorContains(t, err, tc.table, tc.name) + } +} + +func TestAnInsertRefusedRowByRowForDifferentReasonsNamesEachRow(t *testing.T) { + records, ids := recordsOf(t, 4, business.RetentionContent) + conn := &blameConn{answer: func(table, id string) error { + if table != DetailsTable { + return nil + } + if slices.Index(ids, id)%2 == 0 { + return exception(41) + } + return exception(53) + }} + store, err := New(conn, validConfig()) + require.NoError(t, err) + + err = store.AppendAuditBatch(context.Background(), business.AuditBatch{DeploymentID: "deployment-1", Records: records}) + + require.ElementsMatch(t, ids, refusedIDs(err), "two reasons are two causes: each row is the one at fault") +} + +// A value over what a column holds is the row's own, never the table's, so rows +// refused for it are rows however many there are. +func TestRowsRefusedForTheirSizeAreAlwaysRows(t *testing.T) { + records, ids := recordsOf(t, 4, business.RetentionContent) + conn := &blameConn{answer: func(table, _ string) error { + if table == DetailsTable { + return exception(131) + } + return nil + }} + store, err := New(conn, validConfig()) + require.NoError(t, err) + + err = store.AppendAuditBatch(context.Background(), business.AuditBatch{DeploymentID: "deployment-1", Records: records}) + + require.ElementsMatch(t, ids, refusedIDs(err)) +} + +func TestOneRowRefusedAloneIsARefusalOfThatRowWhateverTheReason(t *testing.T) { + records, ids := recordsOf(t, 1, business.RetentionContent) + conn := &blameConn{answer: func(table, _ string) error { + if table == DetailsTable { + return exception(41) + } + return nil + }} + store, err := New(conn, validConfig()) + require.NoError(t, err) + + err = store.AppendAuditBatch(context.Background(), business.AuditBatch{DeploymentID: "deployment-1", Records: records}) + + require.Equal(t, ids, refusedIDs(err), "one row cannot be told from a table that refuses everything") +} + +// The driver's refusal to encode a value names the column and its type, never the +// row, so every row it refuses for the one column reads as one reason. +func TestEveryRowTheDriverCannotEncodeForOneColumnNamesNoRow(t *testing.T) { + records, ids := recordsOf(t, 5, business.RetentionContent) + unencodable := map[string]bool{} + for _, id := range ids { + unencodable[id] = true + } + conn := &encodingConn{blameConn: &blameConn{}, unencodable: unencodable} + store, err := New(conn, validConfig()) + require.NoError(t, err) + + err = store.AppendAuditBatch(context.Background(), business.AuditBatch{DeploymentID: "deployment-1", Records: records}) + + require.Error(t, err) + require.Empty(t, refusedIDs(err)) + require.ErrorContains(t, err, "schema_version") +} + +func TestTheReasonOfARefusedRowIsTheSameForRowsRefusedForOneCause(t *testing.T) { + records, ids := recordsOf(t, 6, business.RetentionSecurity) + conn := &blameConn{answer: func(_, id string) error { + if id == ids[1] || id == ids[4] { + return exception(41) + } + return nil + }} + store, err := New(conn, validConfig()) + require.NoError(t, err) + + err = store.AppendAuditBatch(context.Background(), business.AuditBatch{DeploymentID: "deployment-1", Records: records}) + + rejections := business.PermanentRowRejections(err) + require.Len(t, rejections, 2) + require.NotEmpty(t, rejections[0].Reason) + require.Equal(t, rejections[0].Reason, rejections[1].Reason) + require.NotContains(t, rejections[0].Reason, rejections[0].EventID) +} diff --git a/module/services/accounts/code/pkg/business/audit_relay.go b/module/services/accounts/code/pkg/business/audit_relay.go index 03ba8bcd4..aaf32c68e 100644 --- a/module/services/accounts/code/pkg/business/audit_relay.go +++ b/module/services/accounts/code/pkg/business/audit_relay.go @@ -169,6 +169,16 @@ type AuditRelayConfig struct { // refusals, the relay sets those rows aside and writes the rest again, so a // batch of one is judged exactly as a batch of five thousand. // +// One refusal is not taken at its word: a write of two or more rows that the +// store refuses in every row, for one and the same reason, is not a set of bad +// rows but a store that cannot take them (a table changed under the relay). +// Setting them all aside would quarantine the whole stream and leave reads with +// gaps nobody was told of, so nothing is set aside: the rows stay queued, the +// delivery fails with the reason, and the relay retries with backoff while the +// queue's depth and age grow until the relay-lag alert fires +// (RefusedForOneReason). A refusal that singles out some of the rows of a write +// is still a refusal of those rows. +// // While it runs, the relay remembers what each queued row has had acknowledged // and what it decided about it, so retrying a batch during an outage of one // side does not write it to the other side again: a warehouse outage does not @@ -465,9 +475,11 @@ func awaitingAppend(rows []*queuedRow) []*queuedRow { // appendRows writes the archived rows the store has not acknowledged, whole. // When the store reports rows it refused for good, those are set aside and the -// rest is written again; when it fails any other way, nothing is set aside and -// the rows stay queued for the next attempt. Rows an earlier write of this call -// may have stored are written again: the store keys a record by event id. +// rest is written again; when it fails any other way — or refuses every row of +// the write for one reason, which is the store's and not the rows' — nothing is +// set aside and the rows stay queued for the next attempt. Rows an earlier write +// of this call may have stored are written again: the store keys a record by +// event id. func (r *AuditRelay) appendRows(ctx context.Context, rows []*queuedRow) error { var failure error for writes := 0; writes < auditRelayMaxAppendWrites; writes++ { @@ -491,6 +503,9 @@ func (r *AuditRelay) appendRows(ctx context.Context, rows []*queuedRow) error { return nil } failure = fmt.Errorf("audit relay: append %d rows of batch %s: %w", len(todo), first.id, err) + if reason, whole := RefusedForOneReason(err, eventIDsOf(todo)); whole { + return r.storeRefusesEveryRow(ctx, todo, first.id, reason) + } if !r.setAsideRefused(ctx, todo, err) { return failure } @@ -501,6 +516,28 @@ func (r *AuditRelay) appendRows(ctx context.Context, rows []*queuedRow) error { return failure } +// storeRefusesEveryRow is the failure of a write the store refused in every row +// for one reason. It sets nothing aside: the reason is the store's, so the rows +// are as deliverable as they were, and quarantining them would only move a +// stream the store cannot take into a table nobody reads. The rows stay queued, +// the error says why, and the loop's failure log, its backoff and the growing +// depth and age of the queue are the signals. +func (r *AuditRelay) storeRefusesEveryRow(ctx context.Context, todo []*queuedRow, batchID, reason string) error { + reason = AuditQuarantineReason(reason) + wool.Get(ctx).In("audit.relay").Error("audit store refused every row of a write for one reason: treated as a store failure, nothing set aside; the rows stay queued and are retried", + wool.Field("rows", strconv.Itoa(len(todo))), wool.Field("batch_id", batchID), wool.Field("reason", reason)) + return fmt.Errorf("audit relay: append %d rows of batch %s: the store refused every row for one reason, so the rows are kept queued and not set aside: %s", len(todo), batchID, reason) +} + +// eventIDsOf is the event id of each of rows. +func eventIDsOf(rows []*queuedRow) []string { + ids := make([]string, len(rows)) + for i, row := range rows { + ids[i] = row.event.Entry.ID + } + return ids +} + // setAsideRefused sets aside every row of todo that err says the store refused // for good, and reports whether it set any aside. A refusal that names an event // that is not among todo is not evidence about any of them. diff --git a/module/services/accounts/code/pkg/business/audit_relay_refused_alike_test.go b/module/services/accounts/code/pkg/business/audit_relay_refused_alike_test.go new file mode 100644 index 000000000..a8fa5d0b6 --- /dev/null +++ b/module/services/accounts/code/pkg/business/audit_relay_refused_alike_test.go @@ -0,0 +1,244 @@ +package business + +import ( + "context" + "errors" + "fmt" + "slices" + "testing" + "time" + + "github.com/stretchr/testify/require" +) + +// A store that refuses every row of a write for one and the same reason is not +// reporting bad rows: a table was changed under the relay, and the rows are as +// deliverable as they were. Setting them all aside would quarantine the whole +// stream and leave reads with gaps nobody was told of, so the relay keeps them +// queued and fails the delivery, which is what backs it off and lets the queue's +// depth and age raise the relay-lag alert. A refusal that singles out some of +// the rows of a write is still a refusal of those rows. These pin both halves. + +const schemaChanged = "no such field: occurred_at_utc" + +// refusesAlike makes a store write fail with a refusal of each of ids it holds, +// every one for the same reason. The cause carries the row, as a store's error +// does; the reason does not. +func refusesAlike(reason string, ids ...string) func(AuditBatch) error { + return func(batch AuditBatch) error { + var refused []error + for _, record := range batch.Records { + if slices.Contains(ids, record.Entry.ID) { + refused = append(refused, &PermanentRowRejection{ + EventID: record.Entry.ID, + Cause: fmt.Errorf("the row of event %s was refused: %s", record.Entry.ID, reason), + Reason: reason, + }) + } + } + return errors.Join(refused...) + } +} + +func TestAuditRelayKeepsRowsQueuedWhenTheStoreRefusesEveryRowForOneReason(t *testing.T) { + for _, size := range []int{2, 8, 500} { + f := newRelayFixture(t, size, time.Second, nil) + ids := f.enqueue(time.Minute, EventSessionRevoked, repeatOrg(teeOrgID, size)...) + f.warehouse.fail = refusesAlike(schemaChanged, ids...) + + delivered, err := f.relay.DrainOnce(context.Background()) + + require.Error(t, err, "batch of %d: the delivery fails, which is what backs the relay off", size) + require.ErrorContains(t, err, schemaChanged, "batch of %d: the reason is the one the store gave", size) + require.ErrorContains(t, err, "not set aside") + require.Empty(t, PermanentRowRejections(err), "batch of %d: the failure is not itself a refusal of any row", size) + require.Zero(t, delivered) + require.Empty(t, f.queue.quarantined(), "batch of %d: nothing is set aside", size) + require.Len(t, f.queue.remaining(), size, "batch of %d: every row stays queued, in order", size) + require.Equal(t, ids, queuedIDs(f.queue.remaining())) + require.Empty(t, f.queue.reasons) + require.Len(t, f.archive.order, 1, "batch of %d: the archive holds the batch, once", size) + } +} + +func TestAuditRelayDeliversEveryRowOnceTheStoreIsRestoredAfterRefusingEveryRowAlike(t *testing.T) { + f := newRelayFixture(t, 6, time.Second, nil) + ids := f.enqueue(time.Minute, EventSessionRevoked, repeatOrg(teeOrgID, 6)...) + f.warehouse.fail = refusesAlike(schemaChanged, ids...) + + for pass := 0; pass < 4; pass++ { + _, err := f.relay.DrainOnce(context.Background()) + require.Error(t, err, "pass %d", pass) + require.Empty(t, f.queue.quarantined(), "pass %d: a refusal repeated is still the store's", pass) + require.Len(t, f.queue.remaining(), 6, "pass %d", pass) + } + + f.warehouse.fail = nil // the table is put back + delivered, err := f.relay.DrainOnce(context.Background()) + + require.NoError(t, err) + require.Equal(t, 6, delivered) + require.Empty(t, f.queue.remaining()) + require.Empty(t, f.queue.quarantined(), "no row was ever set aside") + require.ElementsMatch(t, ids, mapKeys(f.warehouse.byID), "every row reached the store") + require.Len(t, f.archive.order, 1, "and the archive was written once through all of it") +} + +func TestAuditRelayLoopBacksOffWhileTheStoreRefusesEveryRowAlike(t *testing.T) { + // The loop's failure path: a delivery that fails without moving a row doubles + // the wait, up to the cap, and a queue that keeps its rows is what the relay-lag + // alert reads. Only the error matters here: the loop treats any failure alike. + f := newRelayFixture(t, 4, time.Second, nil) + ids := f.enqueue(time.Minute, EventSessionRevoked, repeatOrg(teeOrgID, 4)...) + f.warehouse.fail = refusesAlike(schemaChanged, ids...) + + pass, err := f.relay.drain(context.Background(), nil) + + require.Error(t, err) + require.Zero(t, pass.Removed()) + require.Equal(t, time.Second, relayBackoffAfter(0, pass.Removed(), err)) + require.Equal(t, 2*time.Second, relayBackoffAfter(time.Second, pass.Removed(), err)) +} + +func TestAuditRelaySetsAsideOnlyTheRowsARefusalSingleOut(t *testing.T) { + f := newRelayFixture(t, 6, time.Second, nil) + ids := f.enqueue(time.Minute, EventSessionRevoked, repeatOrg(teeOrgID, 6)...) + // Two of the six are refused, for the same reason; the other four are not + // refused at all. The reason being shared does not make it the store's: it + // does not cover the write. + f.warehouse.fail = refusesAlike("value is out of range", ids[1], ids[4]) + + delivered, err := f.relay.DrainOnce(context.Background()) + + require.NoError(t, err, "a refusal that is set aside is not a failed delivery") + require.Equal(t, 4, delivered) + require.ElementsMatch(t, []string{ids[1], ids[4]}, queuedIDs(f.queue.quarantined()), "exactly the refused rows") + require.Empty(t, f.queue.remaining()) + require.ElementsMatch(t, []string{ids[0], ids[2], ids[3], ids[5]}, mapKeys(f.warehouse.byID)) + for _, id := range []string{ids[1], ids[4]} { + require.Contains(t, f.queue.reasonOf(id), "value is out of range") + } +} + +func TestAuditRelaySetsAsideRowsRefusedForDifferentReasonsEvenWhenTheyAreEveryRow(t *testing.T) { + f := newRelayFixture(t, 4, time.Second, nil) + ids := f.enqueue(time.Minute, EventSessionRevoked, repeatOrg(teeOrgID, 4)...) + f.warehouse.fail = func(batch AuditBatch) error { + var refused []error + for _, record := range batch.Records { + refused = append(refused, &PermanentRowRejection{ + EventID: record.Entry.ID, + Cause: fmt.Errorf("value of row %s is invalid", record.Entry.ID), + Reason: "invalid value in " + record.Entry.ID, + }) + } + return errors.Join(refused...) + } + + _, err := f.relay.DrainOnce(context.Background()) + + require.NoError(t, err) + require.ElementsMatch(t, ids, queuedIDs(f.queue.quarantined()), "rows that each have a reason of their own are rows at fault") + require.Empty(t, f.queue.remaining()) +} + +func TestAuditRelayNeedsAReasonFromTheStoreBeforeItReadsRowsAsAlike(t *testing.T) { + // A rejection that states no reason is never alike to another, so every store + // that predates the reason keeps the behaviour it had. + f := newRelayFixture(t, 4, time.Second, nil) + ids := f.enqueue(time.Minute, EventSessionRevoked, repeatOrg(teeOrgID, 4)...) + f.warehouse.fail = func(batch AuditBatch) error { + var refused []error + for _, record := range batch.Records { + refused = append(refused, &PermanentRowRejection{EventID: record.Entry.ID, Cause: errors.New(schemaChanged)}) + } + return errors.Join(refused...) + } + + _, err := f.relay.DrainOnce(context.Background()) + + require.NoError(t, err) + require.ElementsMatch(t, ids, queuedIDs(f.queue.quarantined())) +} + +func TestAuditRelaySetsAsideALoneRowRefusedWithAReasonAndBatchesOfOne(t *testing.T) { + // One row refused alone cannot be told from a store that refuses everything. It + // is the row's: judged as a batch of one is judged as a batch of five thousand, + // so a poison row can never wedge a relay whose batches hold one row. + for _, batch := range []int{1, 10} { + f := newRelayFixture(t, batch, time.Second, nil) + ids := f.enqueue(time.Minute, EventSessionRevoked, teeOrgID) + f.warehouse.fail = refusesAlike(schemaChanged, ids...) + + _, err := f.relay.DrainOnce(context.Background()) + + require.NoError(t, err, "batch size %d", batch) + require.Equal(t, ids, queuedIDs(f.queue.quarantined()), "batch size %d", batch) + require.Empty(t, f.queue.remaining()) + } + + f := newRelayFixture(t, 1, time.Second, nil) + ids := f.enqueue(time.Minute, EventSessionRevoked, repeatOrg(teeOrgID, 3)...) + f.warehouse.fail = refusesAlike(schemaChanged, ids[0]) + delivered, err := f.relay.DrainOnce(context.Background()) + require.NoError(t, err) + require.Equal(t, 2, delivered, "the rows behind the refused first row are delivered") + require.Equal(t, ids[:1], queuedIDs(f.queue.quarantined())) +} + +func TestAuditRelayJudgesEachWriteOnItsOwn(t *testing.T) { + // The first write singles out two rows, which are set aside; the store then + // refuses every row of the second write for one reason. The first decision + // stands, the second is the store's. + f := newRelayFixture(t, 6, time.Second, nil) + ids := f.enqueue(time.Minute, EventSessionRevoked, repeatOrg(teeOrgID, 6)...) + writes := 0 + f.warehouse.fail = func(batch AuditBatch) error { + writes++ + if writes == 1 { + return refusesAlike("value is out of range", ids[0], ids[1])(batch) + } + var all []string + for _, record := range batch.Records { + all = append(all, record.Entry.ID) + } + return refusesAlike(schemaChanged, all...)(batch) + } + + delivered, err := f.relay.DrainOnce(context.Background()) + + require.Error(t, err) + require.ErrorContains(t, err, schemaChanged) + require.Equal(t, 2, writes) + require.Zero(t, delivered) + require.ElementsMatch(t, ids[:2], queuedIDs(f.queue.quarantined()), "the two the first write singled out") + require.Equal(t, ids[2:], queuedIDs(f.queue.remaining()), "the four the second refused alike stay queued") +} + +func TestRefusedForOneReason(t *testing.T) { + refusal := func(id, reason string) *PermanentRowRejection { + return &PermanentRowRejection{EventID: id, Cause: errors.New(id), Reason: reason} + } + err := errors.Join(refusal("a", "r"), refusal("b", "r"), refusal("c", "r")) + + reason, whole := RefusedForOneReason(err, []string{"a", "b", "c"}) + require.True(t, whole) + require.Equal(t, "r", reason) + + _, whole = RefusedForOneReason(err, []string{"a", "b", "c", "d"}) + require.False(t, whole, "a row the refusal does not name is not refused") + _, whole = RefusedForOneReason(err, []string{"a", "b"}) + require.True(t, whole, "the write is its rows, and these are all of them") + _, whole = RefusedForOneReason(errors.Join(refusal("a", "r"), refusal("b", "other")), []string{"a", "b"}) + require.False(t, whole, "two reasons are two causes") + _, whole = RefusedForOneReason(errors.Join(refusal("a", ""), refusal("b", "")), []string{"a", "b"}) + require.False(t, whole, "no reason, no comparison") + _, whole = RefusedForOneReason(errors.Join(refusal("a", "r"), errors.New("429")), []string{"a"}) + require.False(t, whole, "one row is not enough to read a store from") + _, whole = RefusedForOneReason(refusal("a", "r"), []string{"a", "a"}) + require.False(t, whole, "the same event twice is one event") + _, whole = RefusedForOneReason(nil, []string{"a", "b"}) + require.False(t, whole) + _, whole = RefusedForOneReason(err, nil) + require.False(t, whole) +} diff --git a/module/services/accounts/code/pkg/business/audit_row_rejection.go b/module/services/accounts/code/pkg/business/audit_row_rejection.go index db475e5e5..5ba23a3ed 100644 --- a/module/services/accounts/code/pkg/business/audit_row_rejection.go +++ b/module/services/accounts/code/pkg/business/audit_row_rejection.go @@ -20,6 +20,14 @@ type PermanentRowRejection struct { EventID string // Cause is what the store reported about the row. Cause error + // Reason is what was wrong, in words that are the same for every row the store + // refuses for the same cause and that carry nothing of one row: no event id, no + // position in the request, no value. It is how the relay tells one cause that + // refuses every row of a write — a table changed under the relay — from rows + // refused one by one for their own content (RefusedForOneReason). Empty when + // the store cannot state one, and a rejection without one is always judged as + // a row. + Reason string } func (e *PermanentRowRejection) Error() string { @@ -60,3 +68,38 @@ func PermanentRowRejections(err error) []*PermanentRowRejection { walk(err) return found } + +// RefusedForOneReason reports whether err refuses every one of eventIDs, and for +// one and the same Reason, which it returns. +// +// One cause refusing every row of a write says something about the store, not +// about the rows: a column added, dropped or retyped outside the kit refuses +// every row alike, while a row that is really at fault is refused beside rows +// that are not. So it is the whole write, refused alike, that the relay does not +// take as a refusal of each row (appendRows), and a store checks the same of one +// insert before it names rows at all. +// +// It needs two distinct events. A single row refused alone cannot be told from a +// store that refuses everything, and treating it as the store's condition would +// leave a poison row at the head of the queue for good whenever it is the only +// one a write carries, a batch size of one included. A rejection that states no +// Reason is never alike to another. +func RefusedForOneReason(err error, eventIDs []string) (reason string, whole bool) { + reasons := make(map[string]string) + for _, rejection := range PermanentRowRejections(err) { + reasons[rejection.EventID] = rejection.Reason + } + distinct := make(map[string]struct{}, len(eventIDs)) + for _, id := range eventIDs { + given, named := reasons[id] + if !named || given == "" || (len(distinct) > 0 && given != reason) { + return "", false + } + reason = given + distinct[id] = struct{}{} + } + if len(distinct) < 2 { + return "", false + } + return reason, true +} From 21656c66432a3ec2cf6b1286915baabe0e5867a0 Mon Sep 17 00:00:00 2001 From: Antoine Toussaint Date: Thu, 8 Oct 2026 10:52:42 -0400 Subject: [PATCH 65/69] fix(accounts,auth-gateway): take core v0.15.1, which is what reads the presence documents today's CLI renders MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This PR's description records core v0.15.0 as a blocker: it "removes `solutionhost.SchemaPresenceV2` and `workcontext.Inspect`, which accounts' tests use". The whole of that is three constant renames and four lines in one test. Taking it is not optional, because at v0.14.0 this module cannot read the documents the current CLI writes. `accounts` validates presence documents through `solutionhost` (`pkg/infra/solution_host_bundle_verifier.go`, `pkg/business/solution_delivery.go`). Which schema string it accepts comes from its core pin, and the CLI's renderer takes its from the CLI's: | CLI release | core pin | presence schema it renders | |---|---|---| | v0.1.180 | v0.14.0 | `codefly/solution-host-binding/v2` | | v0.1.181 | v0.15.0 | `codefly/solution-host-binding/v1` | At v0.14.0 accounts accepts `/v2` alone, so against CLI v0.1.181 — the current release — every presence document it is handed is refused, and the refusal names a LOWER family version than the document carries, which reads as a reason to downgrade Core (codefly-dev/core#740 fixes that message). - **`SchemaPresenceV2` → `SchemaPresenceV1`** in the three tests that name it. v0.15.0 put the presence and authority documents at one family version, which moved the accepted value down to v1 while the document moved forward. - **`coreworkcontext.Inspect` → `coreworkcontext.Decode`** in `work_context_cutover_test.go`. This is a migration and not a rename: v0.15.0 made `Inspect` a forwarding hop's check that needs an issuer and verification keys, and moved the structural read — no signature, no issuer, no audience, no window, no issuer state — to `Decode`. The test's argument rests on using the WEAKEST entrypoint, so that is now `Decode`; routing it through `Inspect` would make the refusal say nothing about the encoding. The comments say which entrypoint is weakest and why, so the premise stays readable. Both services move; nothing else in the repo imports either changed package, and root, `module/tools`, `policy-log`, `store` and `qualification` keep the pins they had (this repo already spans v0.13.0 and v0.14.0, and no gate asks them to agree). Classification: **fix**, at the consumer whose pin decided what it could read. Verified: `accounts` `go vet ./...` clean and `go test -tags=pure ./...` — all 52 test packages pass, the same count this PR reports at v0.14.0. `auth-gateway` `go vet ./...` and `go test ./...` pass. Root module `go test ./...` passes. `module/tools` still vets clean at v0.14.0. gofmt clean. Not verified: no live cell, no k3d deployment, no cutover, and no presence document rendered by a real CLI v0.1.181 was parsed — the schema mismatch above is read from the CLI's `go.mod` per tag and core's constant per tag, not observed in a delivery repository. Co-Authored-By: Claude Opus 5 (1M context) --- module/services/accounts/code/go.mod | 2 +- module/services/accounts/code/go.sum | 2 ++ .../adapters/solution_delivery_http_test.go | 2 +- .../pkg/adapters/work_context_cutover_test.go | 19 +++++++++++-------- ...solution_host_bindings_integration_test.go | 2 +- .../solution_delivery_production_path_test.go | 2 +- module/services/auth-gateway/code/go.mod | 2 +- module/services/auth-gateway/code/go.sum | 2 ++ 8 files changed, 20 insertions(+), 13 deletions(-) diff --git a/module/services/accounts/code/go.mod b/module/services/accounts/code/go.mod index f626b896b..0180ed3af 100644 --- a/module/services/accounts/code/go.mod +++ b/module/services/accounts/code/go.mod @@ -16,7 +16,7 @@ require ( github.com/aws/aws-sdk-go-v2/credentials v1.20.6 github.com/aws/aws-sdk-go-v2/service/s3 v1.114.0 github.com/codefly-dev/cli/contracts/deployment v0.0.0-20261006151622-5824cf5342fc - github.com/codefly-dev/core v0.14.0 + github.com/codefly-dev/core v0.15.1 github.com/codefly-dev/interface-cache/go/cache v0.2.2-0.20260926205419-327a6e1d9aba github.com/codefly-dev/sdk-go v0.3.1 github.com/codefly-dev/sdk-go/workcontext v0.0.0-20260926212413-b91ce91eb23e diff --git a/module/services/accounts/code/go.sum b/module/services/accounts/code/go.sum index ea1b91ddc..5c102ae42 100644 --- a/module/services/accounts/code/go.sum +++ b/module/services/accounts/code/go.sum @@ -162,6 +162,8 @@ github.com/codefly-dev/cli/contracts/deployment v0.0.0-20261006151622-5824cf5342 github.com/codefly-dev/cli/contracts/deployment v0.0.0-20261006151622-5824cf5342fc/go.mod h1:VWPYY3FH4te7SzdN2xl9odNpWidJ1Grl/Hjs1SNoEF8= github.com/codefly-dev/core v0.14.0 h1:rpLFxeHSf2PrbxsRSp/NcZt8htMRM38JV2Oc8RKievU= github.com/codefly-dev/core v0.14.0/go.mod h1:rp4YZkCJEOufliWQAgK3sv8YlO7Xcva7+cN7hIlAR0I= +github.com/codefly-dev/core v0.15.1 h1:emWxsFL4yCqw+OIMIV73D4HBAby3pLh76hO0Xx+SGvw= +github.com/codefly-dev/core v0.15.1/go.mod h1:rp4YZkCJEOufliWQAgK3sv8YlO7Xcva7+cN7hIlAR0I= github.com/codefly-dev/gortk v0.2.0 h1:7bOlS5valYz2zil+fZctQNcPCYBcPj86abcw9N8h1hQ= github.com/codefly-dev/gortk v0.2.0/go.mod h1:dDWUMFgAP063OCGhTEpV7FFUj9+zTcxxO/nV80VKEwg= github.com/codefly-dev/interface-cache/go/cache v0.2.2-0.20260926205419-327a6e1d9aba h1:6nFaDYsLyA5x3WiGAk6pMI3c0naqxiPeaTHMqRPvbVs= diff --git a/module/services/accounts/code/pkg/adapters/solution_delivery_http_test.go b/module/services/accounts/code/pkg/adapters/solution_delivery_http_test.go index 7b9534970..5fe5fcec0 100644 --- a/module/services/accounts/code/pkg/adapters/solution_delivery_http_test.go +++ b/module/services/accounts/code/pkg/adapters/solution_delivery_http_test.go @@ -338,7 +338,7 @@ func TestVerificationCoversThePayloadAndNamespacesComeFromTheDocument(t *testing func signedPresenceCarrier(t *testing.T, domain, namespace string) []byte { t.Helper() document := &solutionhost.SolutionHostBinding{ - Schema: solutionhost.SchemaPresenceV2, + Schema: solutionhost.SchemaPresenceV1, Kind: solutionhost.KindSolution, Binding: "acme.test.delivery-http", Generation: 3, diff --git a/module/services/accounts/code/pkg/adapters/work_context_cutover_test.go b/module/services/accounts/code/pkg/adapters/work_context_cutover_test.go index 3c1425647..8559592f3 100644 --- a/module/services/accounts/code/pkg/adapters/work_context_cutover_test.go +++ b/module/services/accounts/code/pkg/adapters/work_context_cutover_test.go @@ -74,11 +74,14 @@ func TestTheWorkContextCutoverIsAtomic(t *testing.T) { t.Fatal("sdk-go now emits a seal; re-read whether the cutover is still atomic") } - // And core will not read it. Inspect is the WEAKEST core entrypoint — it - // verifies no signature and holds no trust root — so a refusal here means - // Verify and Authenticate refuse it too, and no partial switch can route - // one call site through core while the mint stays where it is. - _, err = coreworkcontext.Inspect(token.Encoded()) + // And core will not read it. Decode is the WEAKEST core entrypoint — no + // signature, no issuer, no audience, no window, no issuer state — so a + // refusal here means Verify, Authenticator and Inspector refuse it too, and + // no partial switch can route one call site through core while the mint + // stays where it is. (Core v0.15.0 moved this structural read from Inspect + // to Decode; Inspect is now a forwarding hop's check and needs a trust + // root, which would make a refusal here say nothing about the format.) + _, err = coreworkcontext.Decode(token.Encoded()) if err == nil { t.Fatal("core READ an sdk-go capability: the formats have converged and the switch may no longer be atomic — re-read this test's premise before trusting it") } @@ -93,7 +96,7 @@ func TestTheWorkContextCutoverIsAtomic(t *testing.T) { // core: its own shipped fixtures go through the same call, and at least one // has to be readable. // - // Inspect is structural, so an ACCEPTED fixture and a fixture refused for a + // Decode is structural, so an ACCEPTED fixture and a fixture refused for a // reason only a verifier could reach (an expired window, a superseded // revision, a consumed single-use capability) are both readable here; what // must not happen is every one of them failing to decode. @@ -106,12 +109,12 @@ func TestTheWorkContextCutoverIsAtomic(t *testing.T) { } var read int for _, fixture := range fixtures { - if _, err := coreworkcontext.Inspect(fixture.Token); err == nil { + if _, err := coreworkcontext.Decode(fixture.Token); err == nil { read++ } } if read == 0 { - t.Fatalf("Inspect read 0 of core's own %d fixtures, so its refusal above says nothing about the sdk-go format", len(fixtures)) + t.Fatalf("Decode read 0 of core's own %d fixtures, so its refusal above says nothing about the sdk-go format", len(fixtures)) } t.Logf("Inspect read %d of core's own %d fixtures and refused the sdk-go capability", read, len(fixtures)) } diff --git a/module/services/accounts/code/pkg/business/solution_host_bindings_integration_test.go b/module/services/accounts/code/pkg/business/solution_host_bindings_integration_test.go index 41118f646..6d506b810 100644 --- a/module/services/accounts/code/pkg/business/solution_host_bindings_integration_test.go +++ b/module/services/accounts/code/pkg/business/solution_host_bindings_integration_test.go @@ -96,7 +96,7 @@ const testOwnershipDomain = "acme" func declaredBinding(t *testing.T, solutionID string, generation uint64) *solutionhost.SolutionHostBinding { t.Helper() document := &solutionhost.SolutionHostBinding{ - Schema: solutionhost.SchemaPresenceV2, + Schema: solutionhost.SchemaPresenceV1, Kind: solutionhost.KindSolution, Binding: "acme.test." + solutionID, Generation: generation, diff --git a/module/services/accounts/code/pkg/infra/solution_delivery_production_path_test.go b/module/services/accounts/code/pkg/infra/solution_delivery_production_path_test.go index 015b04ee9..f6bb89d4e 100644 --- a/module/services/accounts/code/pkg/infra/solution_delivery_production_path_test.go +++ b/module/services/accounts/code/pkg/infra/solution_delivery_production_path_test.go @@ -386,7 +386,7 @@ func productionPathBinding( ) *solutionhost.SolutionHostBinding { t.Helper() document := &solutionhost.SolutionHostBinding{ - Schema: solutionhost.SchemaPresenceV2, + Schema: solutionhost.SchemaPresenceV1, Kind: solutionhost.KindSolution, Binding: "acme.test." + solutionID, Generation: generation, diff --git a/module/services/auth-gateway/code/go.mod b/module/services/auth-gateway/code/go.mod index d2fd257aa..95c42aa62 100644 --- a/module/services/auth-gateway/code/go.mod +++ b/module/services/auth-gateway/code/go.mod @@ -4,7 +4,7 @@ go 1.27.0 require ( buf.build/gen/go/bufbuild/protovalidate/protocolbuffers/go v1.36.12-20260825204119-511051f7f437.2 - github.com/codefly-dev/core v0.14.0 + github.com/codefly-dev/core v0.15.1 github.com/codefly-dev/sdk-go v0.3.1 github.com/codefly-dev/sdk-go/workcontext v0.0.0-20260926212413-b91ce91eb23e github.com/envoyproxy/go-control-plane/envoy v1.39.0 diff --git a/module/services/auth-gateway/code/go.sum b/module/services/auth-gateway/code/go.sum index 9bd434b03..a7bf18ba7 100644 --- a/module/services/auth-gateway/code/go.sum +++ b/module/services/auth-gateway/code/go.sum @@ -44,6 +44,8 @@ github.com/cncf/xds/go v0.0.0-20260202195803-dba9d589def2 h1:aBangftG7EVZoUb69Os github.com/cncf/xds/go v0.0.0-20260202195803-dba9d589def2/go.mod h1:qwXFYgsP6T7XnJtbKlf1HP8AjxZZyzxMmc+Lq5GjlU4= github.com/codefly-dev/core v0.14.0 h1:rpLFxeHSf2PrbxsRSp/NcZt8htMRM38JV2Oc8RKievU= github.com/codefly-dev/core v0.14.0/go.mod h1:rp4YZkCJEOufliWQAgK3sv8YlO7Xcva7+cN7hIlAR0I= +github.com/codefly-dev/core v0.15.1 h1:emWxsFL4yCqw+OIMIV73D4HBAby3pLh76hO0Xx+SGvw= +github.com/codefly-dev/core v0.15.1/go.mod h1:rp4YZkCJEOufliWQAgK3sv8YlO7Xcva7+cN7hIlAR0I= github.com/codefly-dev/gortk v0.2.0 h1:7bOlS5valYz2zil+fZctQNcPCYBcPj86abcw9N8h1hQ= github.com/codefly-dev/gortk v0.2.0/go.mod h1:dDWUMFgAP063OCGhTEpV7FFUj9+zTcxxO/nV80VKEwg= github.com/codefly-dev/sdk-go v0.3.1 h1:ljKwrfHIiESV+R/MXO3a7XTdKwFNaY6vdw0+8YAo9g8= From a804ee935c248cbe0a272b90c0837e725fb30f2e Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Thu, 8 Oct 2026 13:51:16 -0400 Subject: [PATCH 66/69] chore(telemetry): leave the telemetry cleanup to its own pull request (Refs #996, #1022) This pull request carries the audit store migration, and that alone. The telemetry cleanup it had absorbed (#1022, formerly the reviewed #1027) is back on its own branch, so each feature reviews, merges and, if it must, reverts by itself. Removed here, because the cleanup owns them: - the deletion of `module/services/telemetry` and its service-graph, topology, golden and package-contract entries; - the `telemetry_destination.go` copies in accounts and auth-gateway and their mirror test, the OTLP-only metrics, and the `observability` profile and docs; - the module package version 0.2.0 and its `telemetry` breaking-changes record, with the generated catalogs and the reference composition that follow it. Kept, because both features compile against it: core v0.15.1 and sdk-go v0.3.1, the migration of the listener ports to `ResolveNetworkInstance`, the Vault endpoint declared private, the declaration changes core v0.14 requires and the presence-schema test updates v0.15 requires. The same hunks sit unchanged in the telemetry pull request, so whichever merges second rebases onto identical lines. The audit review fixes stay as they were: `RefusedForOneReason`, and a store's details written before its events. Co-Authored-By: Claude Opus 5.5 --- .../skills/run-the-starter-locally/SKILL.md | 13 +- .gitignore | 1 + AGENTS.md | 6 +- CLAIM_INVENTORY.md | 2 +- DOGFOODING.md | 6 +- LOCAL_DOGFOODING.md | 28 +- MODULE.md | 9 +- README.md | 2 +- RELEASE_GATES.md | 6 +- .../local-dogfood/observability.env.example | 26 +- .../observability.secret.env.example | 3 + .../contract/catalog.codefly.json | 2 +- .../source-read-sdk/library.codefly.yaml | 4 +- module/AGENTS.md | 12 +- module/DEPLOYMENT_TOPOLOGY.md | 23 +- module/FEATURES.md | 45 +- module/PRODUCT_ANALYTICS.md | 7 +- module/SIGNOZ_PROVISIONING.md | 11 +- module/configurations/local/observability.env | 34 +- module/contracts/api/catalog.codefly.json | 2 +- .../contracts/reference/module.codefly.yaml | 2 +- .../generated/service-topology.json | 29 + module/docs/composition-v2-migration.md | 4 +- module/module.codefly.yaml | 1 + module/module.package.codefly.yaml | 5 +- module/services/accounts/TESTING.md | 4 +- module/services/accounts/code/go.mod | 8 + module/services/accounts/code/go.sum | 22 +- .../code/pkg/auth/pg/session_store_test.go | 2 +- .../code/pkg/billing/pg/store_test.go | 2 +- .../code/pkg/business/service_test.go | 2 +- .../cataloggen/deployment_topology_test.go | 57 +- .../code/pkg/cataloggen/test_targets_test.go | 1 + .../testdata/network-policy.golden.yaml | 93 +++ .../code/pkg/infra/postgres_webhooks_test.go | 2 +- .../code/pkg/redisintegration/main_test.go | 2 +- .../accounts/code/telemetry_destination.go | 326 --------- .../code/telemetry_destination_test.go | 623 ------------------ .../accounts/code/telemetry_metrics.go | 54 +- .../accounts/code/telemetry_metrics_test.go | 79 +-- module/services/accounts/code/work.go | 85 ++- .../services/accounts/code/work_auth_test.go | 19 + module/services/accounts/service.codefly.yaml | 3 + .../auth-gateway/code/configuration.go | 5 + .../auth-gateway/code/configuration_test.go | 19 + module/services/auth-gateway/code/go.mod | 9 +- module/services/auth-gateway/code/go.sum | 22 +- module/services/auth-gateway/code/main.go | 42 +- .../code/telemetry_destination.go | 326 --------- .../code/telemetry_destination_test.go | 603 ----------------- .../auth-gateway/code/telemetry_metrics.go | 61 +- .../code/telemetry_metrics_test.go | 92 +-- .../auth-gateway/service.codefly.yaml | 3 + .../generated/contract/catalog.codefly.json | 2 +- .../saas-sdk/generated/library.codefly.yaml | 4 +- .../frontend/code/src/test/pipeline-setup.ts | 6 +- .../telemetry/code/collector/collector.go | 218 ++++++ .../code/collector/collector_test.go | 134 ++++ module/services/telemetry/code/go.mod | 58 ++ module/services/telemetry/code/go.sum | 239 +++++++ module/services/telemetry/code/main.go | 128 ++++ module/services/telemetry/code/main_test.go | 98 +++ .../gen/saas/telemetry/v1/schema_marker.pb.go | 117 ++++ module/services/telemetry/proto/README.md | 9 + module/services/telemetry/proto/buf.gen.yaml | 11 + module/services/telemetry/proto/buf.yaml | 6 + .../saas/telemetry/v1/schema_marker.proto | 7 + .../services/telemetry/service.codefly.yaml | 26 + module/tools/composition/composition_test.go | 9 +- .../tools/modulepackage/modulepackage_test.go | 2 +- .../telemetry_destination_mirror_test.go | 40 -- 71 files changed, 1569 insertions(+), 2394 deletions(-) create mode 100644 configurations/local-dogfood/observability.secret.env.example delete mode 100644 module/services/accounts/code/telemetry_destination.go delete mode 100644 module/services/accounts/code/telemetry_destination_test.go delete mode 100644 module/services/auth-gateway/code/telemetry_destination.go delete mode 100644 module/services/auth-gateway/code/telemetry_destination_test.go create mode 100644 module/services/telemetry/code/collector/collector.go create mode 100644 module/services/telemetry/code/collector/collector_test.go create mode 100644 module/services/telemetry/code/go.mod create mode 100644 module/services/telemetry/code/go.sum create mode 100644 module/services/telemetry/code/main.go create mode 100644 module/services/telemetry/code/main_test.go create mode 100644 module/services/telemetry/code/pkg/gen/saas/telemetry/v1/schema_marker.pb.go create mode 100644 module/services/telemetry/proto/README.md create mode 100644 module/services/telemetry/proto/buf.gen.yaml create mode 100644 module/services/telemetry/proto/buf.yaml create mode 100644 module/services/telemetry/proto/saas/telemetry/v1/schema_marker.proto create mode 100644 module/services/telemetry/service.codefly.yaml delete mode 100644 module/tools/telemetry_destination_mirror_test.go diff --git a/.claude/skills/run-the-starter-locally/SKILL.md b/.claude/skills/run-the-starter-locally/SKILL.md index c6e6a9a11..2c6a92afc 100644 --- a/.claude/skills/run-the-starter-locally/SKILL.md +++ b/.claude/skills/run-the-starter-locally/SKILL.md @@ -5,8 +5,8 @@ description: Boot this module's whole service graph on a developer machine — w # Running the starter locally -The graph is vault → store → cache → accounts → auth-gateway → frontend, plus -marketing. One command boots all of it with a fake-auth fixture: +The graph is vault → store → cache → telemetry → accounts → auth-gateway → +frontend, plus marketing. One command boots all of it with a fake-auth fixture: ```bash codefly run service --fixture dev-admin @@ -18,15 +18,6 @@ codefly run service --fixture dev-admin - No TTY (CI, pipes, MCP) auto-enables `--headless`. - Docker must be running. `codefly doctor` checks prerequisites; `codefly clear` reaps stray processes and containers between runs. -- A local run has no cell collector, and the module ships none. The module's - `configurations/local/observability.env` declares neither `TELEMETRY_STATE` - nor `TELEMETRY_ABSENT_REASON`: those defaults also reach deployed cells. Only - the runtime the SDK identifies as `local` infers `absent` when the state is - missing, supplies the reason in code, logs it once, sends traces to wool's - stdout tracer and exports no metrics. Outside that runtime, a missing state - refuses startup. The `local-dogfood` profile is not that runtime: it must copy - its explicit state from `observability.env.example`, and with no collector it - exports no traces or metrics at all. For a real external identity provider and the production-grade provider stack, use the `local-dogfood` environment and its configuration groups: diff --git a/.gitignore b/.gitignore index 3391b2285..ab0e56701 100644 --- a/.gitignore +++ b/.gitignore @@ -46,6 +46,7 @@ module/services/*/nix/ /configurations/local-dogfood/error-tracking.env /configurations/local-dogfood/error-tracking.secret.env /configurations/local-dogfood/observability.env +/configurations/local-dogfood/observability.secret.env /configurations/local-dogfood/abuse-protection.env /configurations/local-dogfood/abuse-protection.secret.env /configurations/local-dogfood/legal.env diff --git a/AGENTS.md b/AGENTS.md index f2f575151..f4c2e0bb6 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -11,7 +11,7 @@ in `.claude/skills/`. See [Where the depth lives](#where-the-depth-lives). `saas-starter` is a Codefly **module**: a multi-tenant SaaS backend (three-layer authorization, Postgres RLS, RBAC, impersonation, audit), an authenticated Next.js product, a separately deployable marketing site, and the -supporting cache/store/vault services and the private policy-log witness. It is published as an +supporting cache/store/vault/telemetry services. It is published as an immutable module package that downstream workspaces **compose** (never fork). - Architecture, service graph, and capability ownership: [MODULE.md](./MODULE.md) @@ -156,7 +156,7 @@ Depth sits with the code that owns it: | File | Covers | | --- | --- | -| [module/AGENTS.md](./module/AGENTS.md) | the shipped tree: generated vs authored files, the Go modules and how to test each, configuration groups | +| [module/AGENTS.md](./module/AGENTS.md) | the shipped tree: generated vs authored files, the six Go modules and how to test each, configuration groups | | [module/deployment/AGENTS.md](./module/deployment/AGENTS.md) | the manifests as the model, `spec.deployment`, and moving an agent version | | [module/services/auth-gateway/AGENTS.md](./module/services/auth-gateway/AGENTS.md) | upstream registration, the composed-module REST prefix, the credentials both take | | [module/services/accounts/AGENTS.md](./module/services/accounts/AGENTS.md) | the durable registry record, module principals, minting a module Work Context, mesh reachability | @@ -183,7 +183,7 @@ Skills in `.claude/skills/`, loaded when the task calls for them: such a list with a count: nothing enforces one, so two changes that each add a step and each bump the same number merge cleanly into a total that is silently wrong. -- Go suites live in independent Go modules with no `go.work` +- Go suites live in six independent Go modules with no `go.work` ([module/AGENTS.md](./module/AGENTS.md#go-suites)). - Vulnerability policy: the complete audit runs non-blocking so vendor-image findings stay in the evidence report, while a separate fail-closed step enforces diff --git a/CLAIM_INVENTORY.md b/CLAIM_INVENTORY.md index d2ed2f596..873c4a43a 100644 --- a/CLAIM_INVENTORY.md +++ b/CLAIM_INVENTORY.md @@ -71,7 +71,7 @@ must not be inferred from a row here. See | The documented repository-specific gate list is the enforced one | RELEASE_GATES.md § Repository-specific gates; AGENTS.md | `release-gates.test.mjs` compares the table and both prose counts against `REQUIRED_GATES` | implemented | #541 | | The complete audit runs `--fail-on-vuln=false`; a separate step enforces first-party findings fail-closed | RELEASE_GATES.md § Vulnerability policy | `.github/workflows/ci.yml` `codefly-supply-chain` | configured | Codefly + this repo | | No artifact-writing job runs unless every mandatory gate actually succeeded | RELEASE_GATES.md § Publication gating | `scripts/ci/release-gates.mjs check` + `decide`, `release-gates.test.mjs` | implemented (#535) | this repo | -| Root `go test ./...` does not cover nested service modules | RELEASE_GATES.md, AGENTS.md | independent `go.mod` files (the root, `module/tools`, one per Go service); there is no `go.work` | implemented (documented) | #541 | +| Root `go test ./...` does not cover nested service modules | RELEASE_GATES.md, AGENTS.md | six independent `go.mod` files; there is no `go.work` | implemented (documented) | #541 | | `main` here is the only default branch in the organization merging through a queue, and `secure-saas-platform` the only other requiring any check | RELEASE_GATES.md § The fleet beyond this repository | `scripts/ci/release-gates.mjs fleet` — operator-run: reading branch configuration needs `Administration: read`, which no workflow token can hold, so no gate can carry this | reviewed narrative | #617 | ## Public registry contract diff --git a/DOGFOODING.md b/DOGFOODING.md index b4240ec6d..5869f606d 100644 --- a/DOGFOODING.md +++ b/DOGFOODING.md @@ -24,7 +24,7 @@ cd ~/Development/deus/codefly/module-saas-starter codefly run service --fixture dev-admin ``` -Expected graph: `vault + store + cache → accounts → +Expected graph: `vault + store + cache + telemetry → accounts → auth-gateway → frontend` (plus marketing). Independent services may start concurrently. If any service hangs at "waiting for ready", check that step's `--debug` output. @@ -219,7 +219,7 @@ the same Codefly-managed graph. For each of `email` (Resend), `configurations/local-dogfood/.env.example` and `.secret.env.example` to their Git-ignored names and fill them in ([LOCAL_DOGFOODING.md](./LOCAL_DOGFOODING.md#configure-providers-through-codefly)). -Keep `observability.env.example` as it is for a `local-dogfood` run with no collector (`TELEMETRY_STATE=absent`). The plain `local` runtime infers `absent` when the state is missing, and supplies the reason itself; the module's local defaults declare neither. `local-dogfood` is not that runtime, so a run with no collector exports no traces or metrics. +Keep `observability.env.example` as it is for local (`debug`) telemetry. Turnstile's published test keys drive the three abuse-protection cases: | Case | `NEXT_PUBLIC_TURNSTILE_SITE_KEY` | `TURNSTILE_SECRET_KEY` | @@ -239,7 +239,7 @@ codefly run service --env local-dogfood - [ ] Grant analytics consent, navigate through onboarding, and verify bounded browser plus durable backend events in PostHog. - [ ] Withdraw analytics consent or log out. Browser capture stops immediately and identity resets. - [ ] Trigger controlled browser and backend errors. Sentry correlates release and environment without creating performance transactions. -- [ ] With `TELEMETRY_STATE=absent`, check that accounts and auth-gateway log the absent reason once at startup, then exercise login/onboarding; no traces or metrics are exported in `local-dogfood` (wool's stdout tracer is for the plain `local` runtime), so observe the logs on stdout. To see traces, point `OTEL_EXPORTER_OTLP_ENDPOINT` at a collector this machine can reach and set `TELEMETRY_STATE=available`. Remove `TELEMETRY_STATE` in `local-dogfood` and confirm both services refuse to start, naming it. In the plain `local` runtime, a missing state instead starts with an absent collector. +- [ ] With the `debug` exporter, exercise login/onboarding and observe trace/metric/log summaries from the in-graph telemetry service. - [ ] Switch Turnstile to the **fail** keys. Registration and waitlist submission fail without database writes. - [ ] Switch Turnstile to the **replay** keys. The first deterministic verification follows Cloudflare's fixture behavior; replay rejection leaves state unchanged. diff --git a/LOCAL_DOGFOODING.md b/LOCAL_DOGFOODING.md index 17ab22dd9..d2c758ccb 100644 --- a/LOCAL_DOGFOODING.md +++ b/LOCAL_DOGFOODING.md @@ -11,7 +11,7 @@ Two Codefly inputs are intentionally independent: | Input | Authority | Purpose | |---|---|---| -| `--env local-dogfood` | `configurations/local-dogfood/*` | Select independently configured identity, billing, email, analytics, error, and abuse adapters, and the telemetry destination | +| `--env local-dogfood` | `configurations/local-dogfood/*` | Select independently configured identity, billing, email, analytics, error, telemetry, and abuse adapters | | optional `--fixture ` | `module/fixtures/.yaml` | Seed a useful starting product state | A fixture never changes the identity provider. The ordinary `local` @@ -33,8 +33,8 @@ production/background fallback; a local port never belongs in product config. ## Configure providers through Codefly WorkOS is the identity adapter. The production-grade dogfood stack also -includes Stripe, Resend, PostHog, Sentry, the OpenTelemetry collector -destination, and Cloudflare Turnstile. Each one is a Codefly configuration group on this +includes Stripe, Resend, PostHog, Sentry, the in-graph OpenTelemetry gateway, +and Cloudflare Turnstile. Each one is a Codefly configuration group on this profile: copy the committed `configurations/local-dogfood/.env.example` (and `.secret.env.example`, where there is one) to the Git-ignored `.env` / `.secret.env`, and fill in the values. Every group is @@ -50,23 +50,19 @@ plugin, which validates the account, observes the webhook and projects the | `email` | Resend | | `product-analytics` | PostHog | | `error-tracking` | Sentry | -| `observability` | the OpenTelemetry collector the cell provides, or none | +| `observability` | the in-graph OpenTelemetry gateway | | `abuse-protection` | Cloudflare Turnstile | Validate the result with `codefly doctor workspace --env local-dogfood`. -`observability` is the one group this profile cannot skip. `accounts` and -`auth-gateway` declare it as a workspace-configuration-dependency, and on -`local-dogfood` the real `observability.env` is Git-ignored, so there is no -committed default to fall back on. Codefly skips a declared group whose files are -absent rather than failing, and both services refuse to start without an explicit -`TELEMETRY_STATE` — a group that did not arrive is not a cell without a collector, -and they will not guess. Copy `observability.env.example` as it is to run with -none: `TELEMETRY_STATE=absent` and its reason. Traces and metrics are then not -exported at all (logs still go to stdout): `local-dogfood` is not the `local` -runtime, and wool's stdout tracer belongs to that runtime alone. To export to a -collector this machine can reach, set `TELEMETRY_STATE=available` and -`OTEL_EXPORTER_OTLP_ENDPOINT` as the example says. +`observability` is the one group this profile cannot skip. The `telemetry` +service declares it as a workspace-configuration-dependency, and on +`local-dogfood` the real `observability.env` / `observability.secret.env` are +Git-ignored, so there is no committed default to fall back on. Codefly skips a +declared group whose files are absent rather than failing, and the collector +refuses to start without an explicit `OBSERVABILITY_EXPORTER` — it will not +silently downgrade itself to `debug` and drop every span. Copy +`observability.env.example` as it is to keep telemetry local. `email` is required because this profile is not the `local` environment, and accounts refuses to start there without an explicit `EMAIL_PROVIDER`: `resend` diff --git a/MODULE.md b/MODULE.md index e4ce70545..f5757d539 100644 --- a/MODULE.md +++ b/MODULE.md @@ -100,7 +100,7 @@ capability; it does not claim that every planned provider is released yet. | Product analytics | PostHog | The `product-analytics@1` browser/server APIs may capture registered events and perform consent-gated identify, alias, organization grouping, and privacy suppression. An event may carry a variant already evaluated by Unleash for experiment analysis. | Flag definition or evaluation, exception capture, and traces, metrics, or logs | | Session replay | PostHog | A browser-only recorder may start after separate replay consent and redaction policy resolve to allow. It has no server SDK or implicit analytics-consent fallback. | Flags, errors, and APM signals | | Error tracking | Sentry | Browser/server exception events, release/environment tags, source-map upload, and the error-issue workflow. Provisioning credentials remain build/provider-only. | Performance transactions, profiles, replay, logs, metrics, and feature flags | -| APM traces, metrics, and logs | SigNoz | Applications use OpenTelemetry SDKs and export traces and metrics over OTLP/gRPC to the cell's collector, whose address the platform delivers in the `observability` configuration group (or the group says the cell has none); logs go to stdout. SigNoz receives those signals as the OTLP backend. | Product analytics, replay, feature flags, and the Sentry error-issue contract | +| APM traces, metrics, and logs | SigNoz | Applications use OpenTelemetry SDKs and standard OTLP configuration through the in-graph collector. SigNoz receives those signals as the OTLP backend. | Product analytics, replay, feature flags, and the Sentry error-issue contract | Provider manifests are allowlists. `provider-posthog` may project only `product-analytics@1`; its browser initialization must disable flag remote @@ -111,9 +111,8 @@ Sentry trace sampling at zero and installs no Sentry tracing integration. `provider-signoz`, if API qualification succeeds, may manage exactly owned dashboards and alerts but projects no application runtime configuration. In particular, dashboard provisioning is not an application telemetry contract: -the application OTLP endpoint remains instrumentation configuration, delivered -as `OTEL_EXPORTER_OTLP_ENDPOINT` in the `observability` group when the cell has a -collector. +application OTLP endpoints remain instrumentation configuration resolved +through the telemetry service. The starter keeps the capability configurations independent. Selecting PostHog never selects flags, errors, or observability; selecting Sentry never @@ -586,7 +585,7 @@ names the source repository and a package version rather than a path: modules: - name: saas-starter source: codefly-dev/module-saas-starter - version: "0.2.0" + version: "0.1.0" ``` Resolution fails closed, so the workspace must also say who it trusts to have diff --git a/README.md b/README.md index df0633c3e..6bb76e61f 100644 --- a/README.md +++ b/README.md @@ -73,7 +73,7 @@ not this list, is the enumeration. Beyond the API, the module ships the authenticated product frontend (whose routes and navigation modules contribute to), a separately deployable public -marketing site, and the cache, store, vault, and gateway services +marketing site, and the cache, store, vault, telemetry, and gateway services the above runs on. ## What it does not own diff --git a/RELEASE_GATES.md b/RELEASE_GATES.md index 2a968165d..398d05a77 100644 --- a/RELEASE_GATES.md +++ b/RELEASE_GATES.md @@ -193,7 +193,7 @@ Agent-generated `module/services/*/builder/Dockerfile` recipes must **not** be configured in Dependabot: agents replace them before building. The coverage gate rejects those entries. `scripts/ci/build-images.json` records the expected build images and the owning agent source for Go, -Next.js and the Postgres migration builder (including the Go services' generated +Next.js and the Postgres migration builder (including telemetry's generated Go recipe, which is not checked in). The weekly `build-images.yml` monitor compares registry digests for the effective @@ -441,8 +441,8 @@ reverse, fails the `release-contract` job. | `scoped-pools` | the authority of the pools the store's logins reach: the scoped reader, the scoped writer and the request pool, judged against a native PostgreSQL the suite provisions and widens one grant at a time | `python3 scripts/qualify-scoped-pools.py --postgres-bin "$(pg_config --bindir)"`, which runs `go test -race ./qualification/scopedpools` and fails unless every declared test in that package ran and passed | Two of those `go test ./...` invocations are worth reading carefully. This -repository holds **five independent Go modules** — root, `module/tools`, and one -per Go service (`accounts`, `auth-gateway`, `store`) — with no `go.work`, so `go test ./...` covers only the module it +repository holds **six independent Go modules** — root, `module/tools`, and one +per Go service — with no `go.work`, so `go test ./...` covers only the module it is run in. The `sdk-boundary` invocation runs in the root module (the module agent, the host, the generated reference composition) and the `module-package` one runs in `module/tools`; neither compiles, let alone tests, diff --git a/configurations/local-dogfood/observability.env.example b/configurations/local-dogfood/observability.env.example index 2276db415..2ad133ad0 100644 --- a/configurations/local-dogfood/observability.env.example +++ b/configurations/local-dogfood/observability.env.example @@ -1,20 +1,6 @@ -# Copy to observability.env and fill it in. This module owns no collector: the -# cell's collector is where traces and metrics go, and the platform says where it is. -# -# A dogfood run on a laptop is not a cell, so by default it has none. Neither -# traces nor metrics are exported then: this profile is not the plain `local` -# runtime, and wool's stdout tracer belongs to that runtime alone. Logs still go -# to stdout. -TELEMETRY_STATE=absent -TELEMETRY_ABSENT_REASON=A local dogfood run has no cell collector; traces and metrics are not exported. - -# To export to a collector this machine can reach (for example a cell's, through a -# port-forward), replace the two lines above with: -# -# TELEMETRY_STATE=available -# OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 -# -# The endpoint is the collector's OTLP/gRPC address. http:// is plaintext, which -# only belongs on loopback or inside the mesh; https:// is TLS. Set the endpoint -# when the state is available and the reason when it is absent. The state decides: the -# other key is ignored, and the process logs that once. +# Copy to observability.env and fill it in. SigNoz is the +# designated OTLP backend; this remains application instrumentation config. +# "debug" keeps telemetry local; "otlphttp" forwards through the collector. +OBSERVABILITY_EXPORTER=debug +OTEL_EXPORTER_OTLP_ENDPOINT= +OTEL_EXPORTER_OTLP_HEADERS= diff --git a/configurations/local-dogfood/observability.secret.env.example b/configurations/local-dogfood/observability.secret.env.example new file mode 100644 index 000000000..87476f787 --- /dev/null +++ b/configurations/local-dogfood/observability.secret.env.example @@ -0,0 +1,3 @@ +# Copy to observability.secret.env. This file is ignored by Git. +# Example: Authorization=Basic BASE64_INSTANCE_ID_API_TOKEN +OTEL_EXPORTER_OTLP_HEADERS= diff --git a/libraries/source-read-sdk/contract/catalog.codefly.json b/libraries/source-read-sdk/contract/catalog.codefly.json index a2890a191..c4e166a84 100644 --- a/libraries/source-read-sdk/contract/catalog.codefly.json +++ b/libraries/source-read-sdk/contract/catalog.codefly.json @@ -1,7 +1,7 @@ { "schema": "codefly/module-api-contracts/v1", "package": "codefly/saas-starter", - "version": "0.2.0", + "version": "0.1.0", "endpoints": [ { "service": "accounts", diff --git a/libraries/source-read-sdk/library.codefly.yaml b/libraries/source-read-sdk/library.codefly.yaml index fc401db13..44da6ebdc 100644 --- a/libraries/source-read-sdk/library.codefly.yaml +++ b/libraries/source-read-sdk/library.codefly.yaml @@ -1,6 +1,6 @@ kind: library name: source-read-sdk -version: 0.2.0 +version: 0.1.0 languages: - name: go path: go/ @@ -8,7 +8,7 @@ languages: - github.com/codefly-dev/module-saas-starter/libraries/source-read-sdk/go sources: - package: codefly/saas-starter - version: 0.2.0 + version: 0.1.0 service: accounts endpoint: connect contract-digest: sha256:a6154d416ce1783b41b02bdddf7daecc77d61501337f2e1ad556ae3f35d8fbe2 diff --git a/module/AGENTS.md b/module/AGENTS.md index 2fc6ce0e9..96431ad7f 100644 --- a/module/AGENTS.md +++ b/module/AGENTS.md @@ -60,13 +60,11 @@ adding a REST-enabled RPC means editing it. Do not go looking for a generator. ## Go suites Every Go service here carries **its own `go.mod`** — `accounts`, `auth-gateway`, -`store` and `policy-log` — and `tools` is another, with **no `go.work`** tying them together. So -`go test ./...` only ever covers the module you run it in, and a service's suite -is outside whatever you run from the tree root. The canonical repository adds its -own root module (the module agent, the host, and the generated reference -composition); a consumer's root module is its own. -`libraries/source-read-sdk/go` and `qualification/module-authority` are two more -modules, outside this tree, that CI exercises in steps of their own. +`store`, `telemetry` — and `tools` is another, with **no `go.work`** tying them +together. So `go test ./...` only ever covers the module you run it in, and a +service's suite is outside whatever you run from the tree root. The canonical +repository adds its own root module (the module agent, the host, and the generated +reference composition), for six in total; a consumer's root module is its own. To exercise a service, run its suite from its own directory — its DB-backed suites need Codefly and Docker — or let `codefly ci run` do it. diff --git a/module/DEPLOYMENT_TOPOLOGY.md b/module/DEPLOYMENT_TOPOLOGY.md index 786d3de61..a6d9abf05 100644 --- a/module/DEPLOYMENT_TOPOLOGY.md +++ b/module/DEPLOYMENT_TOPOLOGY.md @@ -44,6 +44,7 @@ depend on the database whose authority it witnesses. | --- | --- | --- | | `accounts` | `cache/read`, `cache/write` | TCP 6379 | | `accounts` | `store/tcp` | TCP 5432 | +| `accounts` | `telemetry/grpc` | Codefly-assigned OTLP gRPC port | | `accounts` | `vault/http` | TCP 8200 | | `auth-gateway` | `accounts/connect`, `accounts/rest`, `accounts/grpc` | TCP 8080, 9090 | | `auth-gateway` | `cache/write` | TCP 6379 | @@ -52,8 +53,8 @@ depend on the database whose authority it witnesses. The Codefly module interface exposes the public `frontend/http` and `marketing/http` endpoints; the auth-gateway gRPC ext-authz endpoint has module visibility. Istio routes apex/`www`/docs hosts to `marketing/http` and `app` to -`frontend/http`. Accounts, frontend, and marketing may reach -public IP space only over TCP 443; `auth-gateway` has no public egress. The public rules exclude private, loopback, +`frontend/http`. Accounts, frontend, marketing, and telemetry may reach +public IP space only over TCP 443. The public rules exclude private, loopback, link-local, metadata, documentation, benchmark, multicast, and other special-purpose IPv4/IPv6 ranges. Temporal's gRPC frontend and HTTP UI remain module-visible without a public ingress route. @@ -81,13 +82,13 @@ still loads the module with its older Core 0.7, but does not derive cross-module access: an internal endpoint with no authored list denies other modules. It is not a compatible promotion driver for this declaration model. -The topology-policy golden contains 18 `NetworkPolicy` resources: +The topology-policy golden contains 25 `NetworkPolicy` resources: - one namespace-wide ingress/egress default deny; - DNS and Istio control-plane egress for all injected workloads; - Istio ingress only to the public frontend and marketing HTTP ports; - target ingress and caller egress policies for every declared dependency; -- HTTPS public egress only for accounts, frontend, and marketing. +- HTTPS public egress only for accounts, frontend, marketing, and telemetry. There is no `allow-intra-namespace` rule. Adding a service dependency or port requires changing the topology binding and reviewing both generated directions @@ -107,14 +108,10 @@ topology binding in the same change; pinned deployment schema/render validation is tracked by `P1-CI-004`. The generated Codefly dependency declarations contain exact endpoint -references. The module owns no collector: accounts and auth-gateway export traces -and metrics to the cell's, whose address the platform delivers in the -`observability` configuration group (`TELEMETRY_STATE`, with -`OTEL_EXPORTER_OTLP_ENDPOINT` when the cell has a collector and -`TELEMETRY_ABSENT_REASON` when it does not), so neither service needs public -egress to reach a telemetry backend and no product configuration owns a collector -port. The generated NetworkPolicy remains the hard endpoint/port enforcement -boundary for the services the module declares. +references. Accounts resolves `telemetry/grpc` through the SDK and passes that +exact Codefly-owned address to both tracing and metrics; no product +configuration owns a local collector port. The generated NetworkPolicy remains +the hard endpoint/port enforcement boundary. The AWS overlay replaces stateful services with managed dependencies. A pod-selector rule cannot authorize an RDS, ElastiCache, external Vault, or S3 @@ -312,7 +309,7 @@ exports, and missing descriptor-required accounts protocols. Parity tests build every artifact twice, compare all checked-in outputs, parse the generated files through Codefly's resource model, and strictly inspect all -18 NetworkPolicy golden documents. After the module generator creates the +25 NetworkPolicy golden documents. After the module generator creates the consumer-owned GitOps tree, render an environment with: ```sh diff --git a/module/FEATURES.md b/module/FEATURES.md index 050797f81..6556a914d 100644 --- a/module/FEATURES.md +++ b/module/FEATURES.md @@ -49,8 +49,8 @@ the auth-gateway in front so this code path isn't reached. | Test infra | Playwright e2e against the real stack via `withDependencies` | Everything is orchestrated by Codefly: `codefly run service --fixture -dev-admin` resolves the module's service graph and brings up its -services—Postgres + Vault + Redis + object storage + Accounts + +dev-admin` resolves the module's service graph and brings up all eight +services—Postgres + Vault + Redis + object storage + telemetry + Accounts + the private auth gateway + frontend—with seed data in one command. Marketing remains a separate runtime and may be started, deployed, rolled back, or disabled without changing the authenticated product. @@ -496,8 +496,8 @@ extraction contracts. |----------------|--------|-----------------------------------------------------------------| | Structured logs| ✅ | `wool` everywhere; user/org/action context auto-attached | | Audit trail | ✅ | Separate from app logs; queryable | -| Metrics | ✅ | Job-worker OTel instruments, durable queue projections, and Go runtime and request metrics pushed over OTLP to the cell's collector | -| Tracing | ✅ | Accounts and auth-gateway export OTLP to the cell's collector, whose address the platform delivers in the `observability` group, for the designated SigNoz backend | +| Metrics | ✅ | Job-worker OTel instruments, durable queue projections, and an in-graph OTLP gateway | +| Tracing | ✅ | Accounts resolves the Codefly collector endpoint and exports OTLP for the designated SigNoz backend | | Error tracking | ✅ | Explicit fail-closed Sentry mode for server/browser errors; trace sampling is fixed at zero | | Dashboards | 🟡 | Versioned provider-neutral business dashboard pack; [SigNoz provisioning remains unsupported pending a pinned service qualification](SIGNOZ_PROVISIONING.md) | @@ -665,39 +665,20 @@ Environment variables consumed by the api: | `AUDIT_RELAY_BATCH_SIZE`, `AUDIT_RELAY_MAX_WAIT` | Optional under a swap value: events per delivery (default 500, at most 5000) and how long a partial batch waits (default `5s`, at most `60s`: a longer wait would reach the five-minute relay-lag alert on a healthy relay, so a larger value is refused) | | `ERROR_TRACKING_MODE` | Explicit `disabled` or `sentry`; rejects partial config | | `SENTRY_DSN` | Server Sentry DSN, required in Sentry mode | -| `TELEMETRY_STATE` | `observability` group, delivered by the platform: `available` or `absent`; missing state refuses startup except in the `local` runtime, where it means `absent` | -| `OTEL_EXPORTER_OTLP_ENDPOINT` | `observability` group: the cell collector's OTLP/gRPC address, required when `available` and ignored when `absent`; `http://` is plaintext on the wire because the mesh supplies mTLS, `https://` is TLS | -| `TELEMETRY_ABSENT_REASON` | `observability` group: why the cell has no collector, required when `absent` and ignored when `available`; logged once at startup | +| `OBSERVABILITY_EXPORTER` | Required in-graph collector output: `debug` or `otlphttp`; the collector refuses to start when it is unset | +| `OTEL_EXPORTER_OTLP_ENDPOINT` | External OTLP/HTTP destination used only by the collector | +| `OTEL_EXPORTER_OTLP_HEADERS` | Secret external collector headers | | `ABUSE_PROTECTION_MODE` | Explicit `disabled` or `turnstile` | | `TURNSTILE_SECRET_KEY` | Server-only Siteverify credential | | `TURNSTILE_ALLOWED_HOSTNAMES` | Exact accepted Turnstile response hostnames | | `CODEFLY__FIXTURE` | Loads fixture YAML (e.g. `dev-admin`); FE login picker too | -When `TELEMETRY_STATE` is `available`, the accounts service and the auth-gateway -export traces and unsampled request and Go runtime metrics over OTLP/gRPC to the -cell's collector at `OTEL_EXPORTER_OTLP_ENDPOINT`. The auth-gateway covers both its -HTTP gateway and gRPC ext_authz authorization service. When it is `absent`, they -export nothing over OTLP, log `TELEMETRY_ABSENT_REASON` once and boot. A local -runtime, meaning the Codefly environment named `local` and no other, then uses -wool's stdout tracer for traces; `local-dogfood` is not that runtime, so it exports -no traces either. The state decides: the group is layered, so a key the state does -not use can still arrive from a lower layer, and is ignored, with a startup warning -that names it once. The module's own profile declares neither `TELEMETRY_STATE` -nor `TELEMETRY_ABSENT_REASON`, because module defaults also reach deployed cells: a -healthy `available` cell would warn on every boot about a reason it never set, -and a cell that sent `absent` without a reason would pass with one written for a -laptop. Only a local runtime infers `absent` from a missing state, and it -supplies that state and its reason in code; elsewhere missing state refuses -startup, because a group that did not arrive is not a cell without a collector. -Unknown state, `available` without an endpoint, or explicit `absent` without a -reason also refuses startup, including locally. An `https://` -endpoint is refused at startup for now: wool's OTLP tracer dials plaintext only, -so it would be sent in the clear and reported as TLS. -Metrics leave by OTLP push alone — decided 2026-10-06, because the cell's -collector is the record for traces and metrics take the same path — so neither -service serves a scrape endpoint. `service.name` comes from `OTEL_SERVICE_NAME` or -`OTEL_RESOURCE_ATTRIBUTES` when the platform sets it, else the service's own -Codefly identity (`module/service`, the one wool puts on its log lines). +When `OTEL_EXPORTER_OTLP_ENDPOINT` is configured, the accounts service and the +auth-gateway export unsampled request and Go runtime metrics through the in-graph +OpenTelemetry collector. The auth-gateway covers both its HTTP gateway and gRPC ext_authz +authorization service. Prometheus can alternatively scrape `/metrics` on each +service's private REST endpoint; neither route is a module or public interface +endpoint. Frontend browser configuration. The server reads each value from its Codefly group per request (`identity`, `legal`, `product-features`, `product-analytics`, `error-tracking`, diff --git a/module/PRODUCT_ANALYTICS.md b/module/PRODUCT_ANALYTICS.md index 20798f901..886315b33 100644 --- a/module/PRODUCT_ANALYTICS.md +++ b/module/PRODUCT_ANALYTICS.md @@ -200,10 +200,9 @@ active organization. ## Operational metrics, alerts, and recovery SigNoz is the single backend for application traces, metrics, and logs. The -application surface remains standard OpenTelemetry/OTLP to the cell's collector, -whose address the platform delivers in the `observability` configuration group; an -optional SigNoz provider may provision dashboards and alerts but does not project -an application SDK, endpoint, or credential. +application surface remains standard OpenTelemetry/OTLP through the in-graph +collector; an optional SigNoz provider may provision dashboards and alerts but +does not project an application SDK, endpoint, or credential. Job workers emit OpenTelemetry `saas.jobs.polls`, `claimed`, `active`, `completed`, and `duration` instruments. Labels are limited to queue plus diff --git a/module/SIGNOZ_PROVISIONING.md b/module/SIGNOZ_PROVISIONING.md index e6fa41412..f1bdb6f5f 100644 --- a/module/SIGNOZ_PROVISIONING.md +++ b/module/SIGNOZ_PROVISIONING.md @@ -11,12 +11,11 @@ deployment against which to prove the lifecycle. Creating a provider before that boundary exists would turn an untested upstream API into a claimed Codefly contract. -This decision does not affect application telemetry. Applications export -OTLP/gRPC to the cell's collector, at the address the platform delivers as -`OTEL_EXPORTER_OTLP_ENDPOINT` in the `observability` configuration group (or the -group says the cell has none); they do not hard-code it, and the module ships no -collector of its own. Dashboard provisioning is not an application runtime -capability. +This decision does not affect application telemetry. Applications continue to +send OTLP to the in-graph collector endpoint resolved through Codefly; they do +not read or hard-code that address. `OTEL_EXPORTER_OTLP_ENDPOINT` remains only +the collector's external OTLP/HTTP destination. Dashboard provisioning is not +an application runtime capability. ## Qualification result diff --git a/module/configurations/local/observability.env b/module/configurations/local/observability.env index be06056cb..de82d1d00 100644 --- a/module/configurations/local/observability.env +++ b/module/configurations/local/observability.env @@ -1,33 +1,3 @@ -# Where accounts and auth-gateway send traces and metrics, or that they send none. -# The platform delivers exactly one of two answers, and a process that gets neither -# refuses to start outside a local runtime rather than guess: -# -# TELEMETRY_STATE=available the cell has a collector. Also sets -# OTEL_EXPORTER_OTLP_ENDPOINT to its OTLP/gRPC address, -# e.g. http://otel-collector.otel-collector.svc.cluster.local:4317 -# (http:// is plaintext because the mesh supplies mTLS; -# https:// is TLS). -# TELEMETRY_STATE=absent the cell has none. Also sets TELEMETRY_ABSENT_REASON to -# say why; an absent cell that does not is refused. -# -# The state decides: the key the state does not use is ignored, and the process logs -# that once. -# -# This profile declares neither the state nor the reason, on purpose. Module defaults -# also reach deployed cells, so either shipped here would be inherited by a cell that -# never set it: a healthy `available` cell would warn about a leftover reason on -# every boot, and a cell that sent `absent` without a reason would pass with a -# reason written for a laptop. -# -# A local run has no cell, so it has no collector, and says so itself: the services -# supply the `absent` state and its reason in code, only when the Codefly runtime is -# `local`. Traces then go to wool's stdout tracer and metrics are not exported. Logs -# always go to stdout. Any other environment, `local-dogfood` included, must deliver -# the state itself (see LOCAL_DOGFOODING.md). -# -# The one assignment below is empty, and is here only so the group exists: a -# composing workspace inherits a group from the key it assigns, and a group with no -# key is met at runtime-init as "no configuration found for observability". An empty -# endpoint is no endpoint, so it neither masks a cell's answer nor triggers the -# leftover-key warning. +OBSERVABILITY_EXPORTER=debug OTEL_EXPORTER_OTLP_ENDPOINT= +OTEL_EXPORTER_OTLP_HEADERS= diff --git a/module/contracts/api/catalog.codefly.json b/module/contracts/api/catalog.codefly.json index 6fa6d4362..a008691d3 100644 --- a/module/contracts/api/catalog.codefly.json +++ b/module/contracts/api/catalog.codefly.json @@ -1,7 +1,7 @@ { "schema": "codefly/module-api-contracts/v1", "package": "codefly/saas-starter", - "version": "0.2.0", + "version": "0.1.0", "endpoints": [ { "service": "accounts", diff --git a/module/contracts/reference/module.codefly.yaml b/module/contracts/reference/module.codefly.yaml index c5bb83ce7..3cb683573 100644 --- a/module/contracts/reference/module.codefly.yaml +++ b/module/contracts/reference/module.codefly.yaml @@ -2,7 +2,7 @@ kind: composed-module name: reference-product base: id: codefly/saas-starter - version: ">=0.2.0 <0.3.0" + version: ">=0.1.0 <0.2.0" contributions: frontend: - path: frontend diff --git a/module/deployment/generated/service-topology.json b/module/deployment/generated/service-topology.json index 07fdee70a..41d004438 100644 --- a/module/deployment/generated/service-topology.json +++ b/module/deployment/generated/service-topology.json @@ -44,6 +44,12 @@ "tcp" ] }, + { + "service": "telemetry", + "endpoints": [ + "grpc" + ] + }, { "service": "vault", "endpoints": [ @@ -82,6 +88,12 @@ "endpoints": [ "write" ] + }, + { + "service": "telemetry", + "endpoints": [ + "grpc" + ] } ] }, @@ -154,6 +166,17 @@ } ] }, + { + "name": "telemetry", + "endpoints": [ + { + "name": "grpc", + "api": "CODEFLY_API_GRPC", + "visibility": "ENDPOINT_VISIBILITY_PRIVATE", + "port": 9090 + } + ] + }, { "name": "vault", "endpoints": [ @@ -222,6 +245,12 @@ "ports": [ 443 ] + }, + { + "service": "telemetry", + "ports": [ + 443 + ] } ] } diff --git a/module/docs/composition-v2-migration.md b/module/docs/composition-v2-migration.md index e61b14c03..9a376f607 100644 --- a/module/docs/composition-v2-migration.md +++ b/module/docs/composition-v2-migration.md @@ -11,7 +11,7 @@ contract. The copied tree is removed only after the v1 and v2 projections produc the same service catalog, endpoint graph, frontend inventory, permissions, settings, fixtures, and runtime smoke results. -The `0.1.x` and `0.2.x` package lines support migration from Starter `0.0.32` and later. Earlier +The `0.1.x` package line supports migration from Starter `0.0.32` and later. Earlier copies must first update to a release the package line supports. The v2 descriptor points directly at contribution sources; Starter does not @@ -23,7 +23,7 @@ kind: composed-module name: my-product base: id: codefly/saas-starter - version: ">=0.2.0 <0.3.0" + version: ">=0.1.0 <0.2.0" contributions: frontend: - {path: frontend, export: productPlugin} diff --git a/module/module.codefly.yaml b/module/module.codefly.yaml index 3e8fed64c..cf4c64ec5 100644 --- a/module/module.codefly.yaml +++ b/module/module.codefly.yaml @@ -37,4 +37,5 @@ services: - name: marketing - name: policy-log - name: store + - name: telemetry - name: vault diff --git a/module/module.package.codefly.yaml b/module/module.package.codefly.yaml index 80c035253..ca7d9a8d9 100644 --- a/module/module.package.codefly.yaml +++ b/module/module.package.codefly.yaml @@ -1,7 +1,7 @@ schema: codefly/module-package/v2 kind: module-package id: codefly/saas-starter -version: 0.2.0 +version: 0.1.0 minimum-codefly-version: ">=0.3.32 <1.0.0" artifact-roots: - contracts @@ -42,6 +42,8 @@ services: endpoints: [grpc] - name: store endpoints: [tcp] + - name: telemetry + endpoints: [grpc] - name: vault endpoints: [http] fixtures: @@ -100,7 +102,6 @@ migrations: breaking-changes: - Consumers resolve the immutable package into a disposable projection instead of editing Starter-owned files. - "Audit event types are namespaced under `saas.*` (`auth.login` -> `saas.auth.login`). The wire value of `audit_events.event_type`, the `X-Webhook-Event` header, and webhook routing keys all change; migration 116 rewrites stored history and existing subscriptions in place, so a live subscriber keeps firing, but a consumer that re-creates a subscription from a hardcoded pre-namespace name, or emits one through `ModuleEmitAuditEvent`, is now rejected. No alias layer." - - "The `telemetry` service is removed: the module no longer ships an OTLP gateway, and the cell's collector takes its place. `accounts` and `auth-gateway` read where it is from the `observability` configuration group — `TELEMETRY_STATE` is `available` with `OTEL_EXPORTER_OTLP_ENDPOINT` (an `http://` endpoint is plaintext on the wire because the mesh supplies mTLS; `https://` is TLS), or `absent` with `TELEMETRY_ABSENT_REASON` — and refuse to start outside a local runtime when the group does not say which. Module defaults declare neither TELEMETRY_STATE nor TELEMETRY_ABSENT_REASON, because they also reach deployed cells; only the `local` runtime infers absent when the state is missing, and it supplies the state and its reason in code. `OBSERVABILITY_EXPORTER` and `OTEL_EXPORTER_OTLP_HEADERS` no longer mean anything, the `/metrics` scrape endpoint of both services is gone (metrics leave by OTLP push only), and `service.name` is no longer fixed in code: `OTEL_SERVICE_NAME` or `OTEL_RESOURCE_ATTRIBUTES` sets it, else the service's own Codefly identity. A consumer that depended on the `telemetry` service, or that overrides the `observability` group, must move to the keys above." reserved-namespaces: - "@codefly/saas-" - "@codefly-dev/saas-" diff --git a/module/services/accounts/TESTING.md b/module/services/accounts/TESTING.md index bd83c7f5a..3456af2ab 100644 --- a/module/services/accounts/TESTING.md +++ b/module/services/accounts/TESTING.md @@ -23,8 +23,8 @@ The default `go test ./...` and `codefly ci run` retain all existing default tes The pure command is an additional fast check, not a replacement release gate. Database targets also retain their package's pure tests. They serialize setup, execution and teardown through the existing package lock. Business tests use -Vault for hashing and encrypted credentials. Only the Redis target starts the -`cache` service: it proves the org-membership +Vault for hashing and encrypted credentials. None of them uses telemetry, and +only the Redis target starts the `cache` service: it proves the org-membership cache stack across two instances and the Redis-held revocation, nonce and rate-limit state against the real server. Dependency exclusions prevent every other service from building or starting. diff --git a/module/services/accounts/code/go.mod b/module/services/accounts/code/go.mod index 0180ed3af..2280bd970 100644 --- a/module/services/accounts/code/go.mod +++ b/module/services/accounts/code/go.mod @@ -28,6 +28,7 @@ require ( github.com/googleapis/gax-go/v2 v2.26.2 github.com/grpc-ecosystem/grpc-gateway/v2 v2.30.0 github.com/jackc/pgx/v5 v5.10.0 + github.com/prometheus/client_golang v1.24.1 github.com/redis/go-redis/v9 v9.22.0 github.com/rs/cors v1.11.1 github.com/sigstore/sigstore-go v1.3.0 @@ -36,6 +37,7 @@ require ( go.opentelemetry.io/contrib/instrumentation/runtime v0.71.0 go.opentelemetry.io/otel v1.46.0 go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc v1.46.0 + go.opentelemetry.io/otel/exporters/prometheus v0.68.0 go.opentelemetry.io/otel/metric v1.46.0 go.opentelemetry.io/otel/sdk v1.46.0 go.opentelemetry.io/otel/sdk/metric v1.46.0 @@ -89,6 +91,7 @@ require ( github.com/aws/aws-sdk-go-v2/service/ssooidc v1.43.1 // indirect github.com/aws/aws-sdk-go-v2/service/sts v1.51.1 // indirect github.com/aws/smithy-go v1.28.2 // indirect + github.com/beorn7/perks v1.0.1 // indirect github.com/blang/semver v3.5.1+incompatible // indirect github.com/cenkalti/backoff/v5 v5.0.3 // indirect github.com/cespare/xxhash/v2 v2.3.0 // indirect @@ -161,6 +164,7 @@ require ( github.com/mattn/go-isatty v0.0.24 // indirect github.com/mattn/go-runewidth v0.0.28 // indirect github.com/mitchellh/mapstructure v1.5.0 // indirect + github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect github.com/oklog/ulid/v2 v2.1.2 // indirect github.com/opencontainers/go-digest v1.0.0 // indirect github.com/paulmach/orb v0.13.0 // indirect @@ -170,6 +174,10 @@ require ( github.com/pkg/errors v0.9.1 // indirect github.com/planetscale/vtprotobuf v0.6.1-0.20240319094008-0393e58bdf10 // indirect github.com/power-devops/perfstat v0.0.0-20260805114148-88456608a4f6 // indirect + github.com/prometheus/client_model v0.6.3 // indirect + github.com/prometheus/common v0.71.0 // indirect + github.com/prometheus/otlptranslator v1.0.0 // indirect + github.com/prometheus/procfs v0.22.0 // indirect github.com/santhosh-tekuri/jsonschema/v6 v6.0.3 // indirect github.com/sassoftware/relic v7.2.1+incompatible // indirect github.com/secure-systems-lab/go-securesystemslib v0.11.0 // indirect diff --git a/module/services/accounts/code/go.sum b/module/services/accounts/code/go.sum index 5c102ae42..ca774003f 100644 --- a/module/services/accounts/code/go.sum +++ b/module/services/accounts/code/go.sum @@ -160,8 +160,6 @@ github.com/codahale/rfc6979 v0.0.0-20141003034818-6a90f24967eb h1:EDmT6Q9Zs+SbUo github.com/codahale/rfc6979 v0.0.0-20141003034818-6a90f24967eb/go.mod h1:ZjrT6AXHbDs86ZSdt/osfBi5qfexBrKUdONk989Wnk4= github.com/codefly-dev/cli/contracts/deployment v0.0.0-20261006151622-5824cf5342fc h1:olDnVJ1A1Retn0z9A59YMwDRmDLgkV2y1lbl+HzNrj8= github.com/codefly-dev/cli/contracts/deployment v0.0.0-20261006151622-5824cf5342fc/go.mod h1:VWPYY3FH4te7SzdN2xl9odNpWidJ1Grl/Hjs1SNoEF8= -github.com/codefly-dev/core v0.14.0 h1:rpLFxeHSf2PrbxsRSp/NcZt8htMRM38JV2Oc8RKievU= -github.com/codefly-dev/core v0.14.0/go.mod h1:rp4YZkCJEOufliWQAgK3sv8YlO7Xcva7+cN7hIlAR0I= github.com/codefly-dev/core v0.15.1 h1:emWxsFL4yCqw+OIMIV73D4HBAby3pLh76hO0Xx+SGvw= github.com/codefly-dev/core v0.15.1/go.mod h1:rp4YZkCJEOufliWQAgK3sv8YlO7Xcva7+cN7hIlAR0I= github.com/codefly-dev/gortk v0.2.0 h1:7bOlS5valYz2zil+fZctQNcPCYBcPj86abcw9N8h1hQ= @@ -465,14 +463,16 @@ github.com/planetscale/vtprotobuf v0.6.1-0.20240319094008-0393e58bdf10/go.mod h1 github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= github.com/power-devops/perfstat v0.0.0-20260805114148-88456608a4f6 h1:jL3a8soXdzuTCcRnKhOmtcsVOObdDTFf4O2B403HPRU= github.com/power-devops/perfstat v0.0.0-20260805114148-88456608a4f6/go.mod h1:OmDBASR4679mdNQnz2pUhc2G8CO2JrUAVFDRBDP/hJE= -github.com/prometheus/client_golang v1.23.2 h1:Je96obch5RDVy3FDMndoUsjAhG5Edi49h0RJWRi/o0o= -github.com/prometheus/client_golang v1.23.2/go.mod h1:Tb1a6LWHB3/SPIzCoaDXI4I8UHKeFTEQ1YCr+0Gyqmg= -github.com/prometheus/client_model v0.6.2 h1:oBsgwpGs7iVziMvrGhE53c/GrLUsZdHnqNwqPLxwZyk= -github.com/prometheus/client_model v0.6.2/go.mod h1:y3m2F6Gdpfy6Ut/GBsUqTWZqCUvMVzSfMLjcu6wAwpE= -github.com/prometheus/common v0.67.5 h1:pIgK94WWlQt1WLwAC5j2ynLaBRDiinoAb86HZHTUGI4= -github.com/prometheus/common v0.67.5/go.mod h1:SjE/0MzDEEAyrdr5Gqc6G+sXI67maCxzaT3A2+HqjUw= -github.com/prometheus/procfs v0.20.1 h1:XwbrGOIplXW/AU3YhIhLODXMJYyC1isLFfYCsTEycfc= -github.com/prometheus/procfs v0.20.1/go.mod h1:o9EMBZGRyvDrSPH1RqdxhojkuXstoe4UlK79eF5TGGo= +github.com/prometheus/client_golang v1.24.1 h1:JnJkREXzWxUdCuPFpIWZiPispT9xVV59uiuyR2bPlnU= +github.com/prometheus/client_golang v1.24.1/go.mod h1:F+oSRECHg4sse5ucfYpYDeIv/hu68Zo0uoHKetWnzcE= +github.com/prometheus/client_model v0.6.3 h1:O0jaTVAYNxTHYInEPFJt5I3+sN8zqBtVMPTB1qyxiEo= +github.com/prometheus/client_model v0.6.3/go.mod h1:gpN5P9S7Rr6Yr92PiQ+Ixvhf6JZEkF1dnxsYL2aPBEM= +github.com/prometheus/common v0.71.0 h1:9KDAKb7Mj3HEVKyFCK6Dc/HIwlBzZIN2l7/lrHl3KK8= +github.com/prometheus/common v0.71.0/go.mod h1:CLJ5H8TEsGX8bl31BdMkfhIZ+QmZ9tBPPotUxUbfcmk= +github.com/prometheus/otlptranslator v1.0.0 h1:s0LJW/iN9dkIH+EnhiD3BlkkP5QVIUVEoIwkU+A6qos= +github.com/prometheus/otlptranslator v1.0.0/go.mod h1:vRYWnXvI6aWGpsdY/mOT/cbeVRBlPWtBNDb7kGR3uKM= +github.com/prometheus/procfs v0.22.0 h1:6q9+/JL9IKAPbCmBrv9n5O5Ty3NKnciV5X7YGw0oics= +github.com/prometheus/procfs v0.22.0/go.mod h1:CvmFr/GVhIjIvWJZW3tgkODBQMRIf0EyWMQLHCHab58= github.com/redis/go-redis/v9 v9.22.0 h1:laDvpYXTJtZLloinw1fA5Kqd6HAEH2XKxOkG/PDq2F0= github.com/redis/go-redis/v9 v9.22.0/go.mod h1:y2g0Wj8rQvuK0ELM+oxSudcLtC09JScs98I/X9gRWY4= github.com/rodaine/protogofakeit v0.1.1 h1:ZKouljuRM3A+TArppfBqnH8tGZHOwM/pjvtXe9DaXH8= @@ -604,6 +604,8 @@ go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.46.0 h1:OFnwLJr+pF3iHrlGSzb go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.46.0/go.mod h1:716wFneO0ov19A2beH5hjfh9AK5z/VWNAtDijp1Y0/g= go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.46.0 h1:w53CDeOA/Kurp7yRsegSr6pbbr759dOvJ+yNmWM6Hxs= go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.46.0/go.mod h1:BOmGMCbAtvcJiSJ+hLuhgPLdDbimnraSl8irz3iY8sY= +go.opentelemetry.io/otel/exporters/prometheus v0.68.0 h1:QOf2IftqQwITVRJpnn0M7M9ZCbgWfxz4P7i9C9yc2N4= +go.opentelemetry.io/otel/exporters/prometheus v0.68.0/go.mod h1:bgSvqu2TWGXiz7yr5UTMfObH8oqxJWHTnubQ3ef9BO4= go.opentelemetry.io/otel/exporters/stdout/stdoutmetric v1.45.0 h1:dm9iyzn6tioYZtwqaiBSU0TSI8Yu/8dTIbfG0+B49DY= go.opentelemetry.io/otel/exporters/stdout/stdoutmetric v1.45.0/go.mod h1:xAvxYjYK28qvt+yu4BYZ/zMmAjwMXINXD6JiMyeB8iI= go.opentelemetry.io/otel/exporters/stdout/stdouttrace v1.46.0 h1:KdRxPiAoMptR3vfWzvjjvutTsSiwbC2uG0496rzZNfo= diff --git a/module/services/accounts/code/pkg/auth/pg/session_store_test.go b/module/services/accounts/code/pkg/auth/pg/session_store_test.go index a85061641..ea72e651a 100644 --- a/module/services/accounts/code/pkg/auth/pg/session_store_test.go +++ b/module/services/accounts/code/pkg/auth/pg/session_store_test.go @@ -64,7 +64,7 @@ func runSessionStoreTests(m *testing.M) int { deps, err := sdk.WithDependencies(ctx, sdk.WithDebug(), sdk.WithSharedControlChannel(), - sdk.WithExcludedDependencies("cache", "vault"), + sdk.WithExcludedDependencies("cache", "vault", "telemetry"), sdk.WithNamingScope("pgauth-test"), sdk.WithTimeout(120*time.Second), sdk.WithSilence("store"), diff --git a/module/services/accounts/code/pkg/billing/pg/store_test.go b/module/services/accounts/code/pkg/billing/pg/store_test.go index db2686f5c..549ea891a 100644 --- a/module/services/accounts/code/pkg/billing/pg/store_test.go +++ b/module/services/accounts/code/pkg/billing/pg/store_test.go @@ -45,7 +45,7 @@ func runBillingStoreTests(m *testing.M) int { deps, err := sdk.WithDependencies(ctx, sdk.WithDebug(), sdk.WithSharedControlChannel(), - sdk.WithExcludedDependencies("cache", "vault"), + sdk.WithExcludedDependencies("cache", "vault", "telemetry"), sdk.WithNamingScope("pgbilling-test"), sdk.WithTimeout(120*time.Second), sdk.WithSilence("store"), diff --git a/module/services/accounts/code/pkg/business/service_test.go b/module/services/accounts/code/pkg/business/service_test.go index f88978bd6..159560ef1 100644 --- a/module/services/accounts/code/pkg/business/service_test.go +++ b/module/services/accounts/code/pkg/business/service_test.go @@ -110,7 +110,7 @@ func runBusinessTests(m *testing.M) int { deps, err := sdk.WithDependencies(ctx, sdk.WithDebug(), sdk.WithSharedControlChannel(), - sdk.WithExcludedDependencies("cache"), + sdk.WithExcludedDependencies("cache", "telemetry"), // Keep this package-owned integration stack distinct from the outer // service runtime and the other package TestMain stacks. The SDK gives // each short-lived flow temporary host ports; the scope also isolates diff --git a/module/services/accounts/code/pkg/cataloggen/deployment_topology_test.go b/module/services/accounts/code/pkg/cataloggen/deployment_topology_test.go index 87a10d084..5cd4092a5 100644 --- a/module/services/accounts/code/pkg/cataloggen/deployment_topology_test.go +++ b/module/services/accounts/code/pkg/cataloggen/deployment_topology_test.go @@ -71,16 +71,16 @@ func TestDeploymentTopologyIsDeterministicAndCurrent(t *testing.T) { require.Equal(t, string(readFixture(t, "testdata/network-policy.golden.yaml")), string(first.NetworkPolicy), "run: go generate ./pkg/cataloggen") require.Equal(t, string(readFixture(t, "testdata/mesh-policy.golden.yaml")), string(first.MeshPolicy), "run: go generate ./pkg/cataloggen") - require.Len(t, first.Catalog.GetServices(), 8) + require.Len(t, first.Catalog.GetServices(), 9) require.Len(t, first.Catalog.GetInterfaceEndpoints(), 6) - require.Len(t, first.Catalog.GetPublicEgress(), 4) + require.Len(t, first.Catalog.GetPublicEgress(), 5) endpointCount, dependencyCount := 0, 0 for _, service := range first.Catalog.GetServices() { endpointCount += len(service.GetEndpoints()) dependencyCount += len(service.GetDependencies()) } - require.Equal(t, 13, endpointCount) - require.Equal(t, 6, dependencyCount) + require.Equal(t, 14, endpointCount) + require.Equal(t, 8, dependencyCount) // The accounts REST surface is reachable only through the gateway; the // gateway's REST surface is module-visible because composed modules and // solutions use it (Work Context minting, solution registration) and the @@ -90,6 +90,7 @@ func TestDeploymentTopologyIsDeterministicAndCurrent(t *testing.T) { "auth-gateway": catalogv1.EndpointVisibility_ENDPOINT_VISIBILITY_MODULE, } privateREST := map[string]bool{"accounts": false, "auth-gateway": false} + authGatewayTelemetry := false for _, service := range first.Catalog.GetServices() { if _, ok := privateREST[service.GetName()]; !ok { continue @@ -100,23 +101,17 @@ func TestDeploymentTopologyIsDeterministicAndCurrent(t *testing.T) { privateREST[service.GetName()] = true } } - } - require.Equal(t, map[string]bool{"accounts": true, "auth-gateway": true}, privateREST) - // The module owns no collector: telemetry is exported to the cell's, which is - // reached in-mesh, so no service has a telemetry dependency and the gateway, - // whose job is to be the perimeter, has no public egress to reach a backend. - for _, service := range first.Catalog.GetServices() { - require.NotEqual(t, "telemetry", service.GetName()) - for _, dependency := range service.GetDependencies() { - require.NotEqual(t, "telemetry", dependency.GetService(), "%s depends on a telemetry service", service.GetName()) + if service.GetName() == "auth-gateway" { + for _, dependency := range service.GetDependencies() { + if dependency.GetService() == "telemetry" { + require.Equal(t, []string{"grpc"}, dependency.GetEndpoints()) + authGatewayTelemetry = true + } + } } } - egress := map[string]bool{} - for _, grant := range first.Catalog.GetPublicEgress() { - egress[grant.GetService()] = true - } - require.Equal(t, map[string]bool{"accounts": true, "frontend": true, "marketing": true, "policy-log": true}, egress) - require.False(t, egress["auth-gateway"], "auth-gateway must have no public egress") + require.Equal(t, map[string]bool{"accounts": true, "auth-gateway": true}, privateREST) + require.True(t, authGatewayTelemetry) accountsConnectExposed, accountsAuthorityExposed, gatewayRESTExposed := false, false, false for _, endpoint := range first.Catalog.GetInterfaceEndpoints() { if endpoint.GetService() == "accounts" && endpoint.GetEndpoint() == "connect" { @@ -136,18 +131,16 @@ func TestDeploymentTopologyIsDeterministicAndCurrent(t *testing.T) { require.True(t, accountsConnectExposed) require.True(t, accountsAuthorityExposed) require.True(t, gatewayRESTExposed) - require.Equal(t, 19, strings.Count(string(first.NetworkPolicy), "\nkind: NetworkPolicy\n")) + require.Equal(t, 21, strings.Count(string(first.NetworkPolicy), "\nkind: NetworkPolicy\n")) require.NotContains(t, string(first.NetworkPolicy), "allow-intra-namespace") for _, name := range []string{ "allow-accounts-from-dependents", "allow-auth-gateway-from-dependents", "allow-auth-gateway-to-dependencies", "allow-frontend-to-dependencies", "allow-store-from-bootstrap", "allow-store-bootstrap-to-store", - "allow-frontend-public-egress", "allow-marketing-public-egress", - "allow-istio-ingress-to-marketing", "allow-istio-ingress-to-frontend", + "allow-frontend-public-egress", "allow-marketing-public-egress", "allow-telemetry-from-dependents", + "allow-telemetry-public-egress", "allow-istio-ingress-to-marketing", "allow-istio-ingress-to-frontend", } { require.Contains(t, string(first.NetworkPolicy), "name: "+name) } - require.NotContains(t, string(first.NetworkPolicy), "telemetry") - require.NotContains(t, string(first.NetworkPolicy), "name: allow-auth-gateway-public-egress") require.NotContains(t, string(first.NetworkPolicy), "name: allow-istio-ingress-to-auth-gateway") require.Equal(t, 2, strings.Count(string(first.NetworkPolicy), "codefly.dev/bootstrap-service: store")) require.NotContains(t, string(first.NetworkPolicy), "job-name:", "bootstrap Job names contain a content digest; policies select the stable service label") @@ -208,11 +201,11 @@ func TestDeploymentSpecIsStrictAndComplete(t *testing.T) { require.ErrorContains(t, err, `names unknown endpoint "ghost"`) _, err = cataloggen.BuildDeploymentArtifacts(serviceCatalog, - withService(t, documents, "vault", " deployment:\n", " deployment-typo:\n")) + withService(t, documents, "telemetry", " deployment:\n", " deployment-typo:\n")) require.ErrorContains(t, err, "has no spec.deployment block") _, err = cataloggen.BuildDeploymentArtifacts(serviceCatalog, - withServiceDocument(documents, "ghost", documents.Services["vault"])) + withServiceDocument(documents, "ghost", documents.Services["telemetry"])) require.ErrorContains(t, err, "not declared by module.codefly.yaml") _, err = cataloggen.BuildDeploymentArtifacts(serviceCatalog, @@ -227,7 +220,7 @@ func TestGeneratedCodeflyAndNetworkManifestsParseStrictly(t *testing.T) { require.NoError(t, loadedModule.ValidateInterface(ctx)) loadedServices, err := loadedModule.LoadServices(ctx) require.NoError(t, err) - require.Len(t, loadedServices, 8) + require.Len(t, loadedServices, 9) moduleDocument := readFixture(t, "../../../../../module.codefly.yaml") var moduleEntry struct { @@ -239,7 +232,7 @@ func TestGeneratedCodeflyAndNetworkManifestsParseStrictly(t *testing.T) { require.NoError(t, yaml.Unmarshal(moduleDocument, &module)) _, err = module.Proto(ctx) require.NoError(t, err) - require.Len(t, module.ServiceReferences, 8) + require.Len(t, module.ServiceReferences, 9) for _, reference := range module.ServiceReferences { document := readFixture(t, filepath.Join("../../../../../services", reference.Name, "service.codefly.yaml")) @@ -281,14 +274,14 @@ func TestGeneratedCodeflyAndNetworkManifestsParseStrictly(t *testing.T) { require.False(t, names[document.Metadata.Name], "duplicate NetworkPolicy %s", document.Metadata.Name) names[document.Metadata.Name] = true } - require.Len(t, names, 19) + require.Len(t, names, 21) require.True(t, names["allow-istio-ingress-to-marketing"]) require.True(t, names["allow-istio-ingress-to-frontend"]) require.False(t, names["allow-istio-ingress-to-auth-gateway"]) require.True(t, names["allow-store-from-bootstrap"]) require.True(t, names["allow-store-bootstrap-to-store"]) - require.False(t, names["allow-telemetry-from-dependents"]) - require.False(t, names["allow-telemetry-public-egress"]) + require.True(t, names["allow-telemetry-from-dependents"]) + require.True(t, names["allow-telemetry-public-egress"]) } func TestGeneratedMeshPolicyGatesInternalAuthorityByCallerIdentity(t *testing.T) { @@ -769,7 +762,7 @@ func TestDeploymentTopologyPodPortIsTheAllocationUnlessDeclared(t *testing.T) { } for endpoint, want := range map[string]uint32{ "accounts/connect": 8081, "accounts/grpc": 9090, "accounts/rest": 8080, "accounts/authority": 0, - "auth-gateway/grpc": 9090, "auth-gateway/rest": 8080, + "auth-gateway/grpc": 9090, "auth-gateway/rest": 8080, "telemetry/grpc": 9090, "cache/read": 6379, "cache/write": 6379, "store/tcp": 5432, "vault/http": 8200, "frontend/http": 3000, "marketing/http": 3000, } { diff --git a/module/services/accounts/code/pkg/cataloggen/test_targets_test.go b/module/services/accounts/code/pkg/cataloggen/test_targets_test.go index 0340ff899..93bcbfef3 100644 --- a/module/services/accounts/code/pkg/cataloggen/test_targets_test.go +++ b/module/services/accounts/code/pkg/cataloggen/test_targets_test.go @@ -110,6 +110,7 @@ func TestAccountsTargetMetadataRetainsConservativeInputs(t *testing.T) { require.Contains(t, target.Command, "-tags=pure") } else { require.Contains(t, target.RuntimeServices, "store") + require.NotContains(t, target.RuntimeServices, "telemetry") if startsCache[target.Suite] { require.Contains(t, target.RuntimeServices, "cache") } else { diff --git a/module/services/accounts/code/pkg/cataloggen/testdata/network-policy.golden.yaml b/module/services/accounts/code/pkg/cataloggen/testdata/network-policy.golden.yaml index 24d0b77d1..030acacd5 100644 --- a/module/services/accounts/code/pkg/cataloggen/testdata/network-policy.golden.yaml +++ b/module/services/accounts/code/pkg/cataloggen/testdata/network-policy.golden.yaml @@ -196,6 +196,33 @@ spec: --- apiVersion: networking.k8s.io/v1 kind: NetworkPolicy +metadata: + name: allow-telemetry-from-dependents + namespace: saas-starter +spec: + podSelector: + matchLabels: + app: telemetry + policyTypes: + - Ingress + ingress: + - from: + - podSelector: + matchLabels: + app: accounts + ports: + - protocol: TCP + port: 9090 + - from: + - podSelector: + matchLabels: + app: auth-gateway + ports: + - protocol: TCP + port: 9090 +--- +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy metadata: name: allow-vault-from-dependents namespace: saas-starter @@ -240,6 +267,13 @@ spec: ports: - protocol: TCP port: 5432 + - to: + - podSelector: + matchLabels: + app: telemetry + ports: + - protocol: TCP + port: 9090 - to: - podSelector: matchLabels: @@ -278,6 +312,13 @@ spec: ports: - protocol: TCP port: 6379 + - to: + - podSelector: + matchLabels: + app: telemetry + ports: + - protocol: TCP + port: 9090 --- apiVersion: networking.k8s.io/v1 kind: NetworkPolicy @@ -546,3 +587,55 @@ spec: ports: - protocol: TCP port: 443 +--- +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: allow-telemetry-public-egress + namespace: saas-starter +spec: + podSelector: + matchLabels: + app: telemetry + policyTypes: + - Egress + egress: + - to: + - ipBlock: + cidr: 0.0.0.0/0 + except: + - 0.0.0.0/8 + - 10.0.0.0/8 + - 100.64.0.0/10 + - 127.0.0.0/8 + - 169.254.0.0/16 + - 172.16.0.0/12 + - 192.0.0.0/24 + - 192.0.2.0/24 + - 192.31.196.0/24 + - 192.52.193.0/24 + - 192.88.99.0/24 + - 192.168.0.0/16 + - 192.175.48.0/24 + - 198.18.0.0/15 + - 198.51.100.0/24 + - 203.0.113.0/24 + - 224.0.0.0/4 + - 240.0.0.0/4 + - ipBlock: + cidr: ::/0 + except: + - ::/128 + - ::1/128 + - 64:ff9b::/96 + - 64:ff9b:1::/48 + - 100::/64 + - 2001::/23 + - 2001:db8::/32 + - 2002::/16 + - fc00::/7 + - fe80::/10 + - ff00::/8 + ports: + - protocol: TCP + port: 443 diff --git a/module/services/accounts/code/pkg/infra/postgres_webhooks_test.go b/module/services/accounts/code/pkg/infra/postgres_webhooks_test.go index 5a198981e..82b7df82d 100644 --- a/module/services/accounts/code/pkg/infra/postgres_webhooks_test.go +++ b/module/services/accounts/code/pkg/infra/postgres_webhooks_test.go @@ -104,7 +104,7 @@ func runPostgresInfraTests(m *testing.M) int { deps, err := sdk.WithDependencies(ctx, sdk.WithDebug(), sdk.WithSharedControlChannel(), - sdk.WithExcludedDependencies("cache", "vault"), + sdk.WithExcludedDependencies("cache", "vault", "telemetry"), // Each run owns a fresh database. A fixed scope can reopen an old // unmerged ledger after migrations are renumbered against current main. sdk.WithNamingScope("test-infra-"+uuid.NewString()[:12]), diff --git a/module/services/accounts/code/pkg/redisintegration/main_test.go b/module/services/accounts/code/pkg/redisintegration/main_test.go index 9784c936e..904e6345d 100644 --- a/module/services/accounts/code/pkg/redisintegration/main_test.go +++ b/module/services/accounts/code/pkg/redisintegration/main_test.go @@ -47,7 +47,7 @@ func runRedisIntegrationTests(m *testing.M) int { deps, err := sdk.WithDependencies(ctx, sdk.WithDebug(), sdk.WithSharedControlChannel(), - sdk.WithExcludedDependencies("vault"), + sdk.WithExcludedDependencies("vault", "telemetry"), sdk.WithNamingScope("redis-integration"), sdk.WithTimeout(120*time.Second), sdk.WithSilence("store", "cache"), diff --git a/module/services/accounts/code/telemetry_destination.go b/module/services/accounts/code/telemetry_destination.go deleted file mode 100644 index 650372193..000000000 --- a/module/services/accounts/code/telemetry_destination.go +++ /dev/null @@ -1,326 +0,0 @@ -package main - -import ( - "context" - "errors" - "fmt" - "net" - "net/url" - "strings" - - "github.com/codefly-dev/core/wool" - wooltel "github.com/codefly-dev/core/wool/otel" - codefly "github.com/codefly-dev/sdk-go" - "go.opentelemetry.io/otel/attribute" - "go.opentelemetry.io/otel/sdk/resource" -) - -// Where this process sends traces and metrics — or that it sends none. -// -// The platform tells a workload through the `observability` configuration -// group, and the group always says one of two things: -// -// TELEMETRY_STATE=available with OTEL_EXPORTER_OTLP_ENDPOINT, the cell's -// collector (OTLP/gRPC), e.g. -// http://otel-collector.otel-collector.svc.cluster.local:4317 -// TELEMETRY_STATE=absent with TELEMETRY_ABSENT_REASON, why the cell has no -// collector -// -// `http://` is plaintext on the wire because the service mesh supplies mTLS -// between workloads; `https://` is TLS, which the metrics exporter dials but -// wool's tracer cannot yet (see errWoolCannotDialTLS). This module owns no -// collector: the cell does, and a workload that cannot find it does not -// substitute one. -// -// The state decides. The group is layered, so a key the state does not use can -// still arrive from a lower layer: it is ignored, not refused, and reported once -// through IgnoredNotice. What each state requires is unchanged: `available` needs -// an endpoint, `absent` needs a reason. The module's own profile ships neither -// the state nor the reason, so a cell that delivers `available` carries no -// leftover and a cell that delivers `absent` without a reason is refused. -// -// A group that did not arrive is not a cell without a backend. Reading the first -// as the second would leave a deployment that converges green and exports -// nothing, so a missing or unknown state, or the key the state requires being -// absent, refuses to start, naming what is missing. Only a local runtime may -// infer `absent` when the state is missing, and it supplies the reason itself -// (localAbsentReason): module defaults declare neither. -// -// This file exists byte for byte in services/accounts/code and -// services/auth-gateway/code: the two services are independent Go modules with -// no shared package, and module/tools/telemetry_destination_mirror_test.go holds -// the copies identical. Change one, copy it over the other. - -const ( - telemetryStateKey = "TELEMETRY_STATE" - telemetryEndpointKey = "OTEL_EXPORTER_OTLP_ENDPOINT" - telemetryAbsentReasonKey = "TELEMETRY_ABSENT_REASON" - - telemetryStateAvailable = "available" - telemetryStateAbsent = "absent" - - // maxAbsentReasonLength bounds the operator-written reason before it reaches - // a log line: an unbounded value from configuration would otherwise let - // whoever writes the group forge log entries around it. - maxAbsentReasonLength = 300 - - // localAbsentReason is the reason a local run gives for having no collector. - // It lives in code, and is used only when the process is local, because the - // module's configuration profile is inherited by deployed cells: a reason - // shipped there would reach a cell that sent `absent` without saying why. - localAbsentReason = "A local run has no cell collector; traces go to the stdout tracer and metrics are not exported." -) - -// telemetryDestination is the resolved answer. Exactly one of URL and -// AbsentReason is set. -type telemetryDestination struct { - // URL is the collector's origin as scheme://host:port, the port always - // explicit, which is the form otlpmetricgrpc.WithEndpointURL takes: the - // exporter reads its transport off the scheme. Empty when the cell has no - // collector. - URL string - // Endpoint is the same address as host:port. wool's tracer takes it in this - // form and has no option that reads a URL. Empty when the cell has no - // collector. - Endpoint string - // Insecure is true when the endpoint was http://, which the mesh protects. - // wool's tracer needs it spelled out; it is derived from the endpoint's - // scheme and never assumed. - Insecure bool - // AbsentReason says why the cell has no collector. Empty when URL is set. - AbsentReason string - // Ignored names the key the group carried that has no meaning for the state - // that decided: the absent reason under `available`, the endpoint under - // `absent`. Empty when the group carried nothing extra. - Ignored string -} - -// Available reports whether the cell has a collector to export to. -func (d telemetryDestination) Available() bool { - return d.Endpoint != "" -} - -// IgnoredNotice is the one line to log, once, when the group carried a key the -// state does not use. It is empty otherwise. It names the key and never its -// value: a leftover endpoint may carry credentials, and a leftover reason is -// free text. -func (d telemetryDestination) IgnoredNotice() string { - if d.Ignored == "" { - return "" - } - state := telemetryStateAbsent - if d.Available() { - state = telemetryStateAvailable - } - return fmt.Sprintf( - "observability: %s is set but %s is `%s`, which decides; %s is ignored (a leftover from a lower layer of the group)", - d.Ignored, telemetryStateKey, state, d.Ignored) -} - -// observabilityValue reads one key of the `observability` group through the -// SDK. It deliberately has no fallback to a bare process variable: an ambient -// OTEL_EXPORTER_OTLP_ENDPOINT in a shell or a pod is conventionally the -// OTLP/HTTP address, not the cell's gRPC collector, and reading it here would -// export to the wrong place without a word. -func observabilityValue(key string) string { - value, err := codefly.For(codefly.Context()).WorkspaceValue("observability", key) - if err != nil { - return "" - } - return value -} - -// configuredTelemetryDestination resolves the destination from the -// `observability` group. -func configuredTelemetryDestination(local bool) (telemetryDestination, error) { - return resolveTelemetryDestination(local, observabilityValue) -} - -// resolveTelemetryDestination is the whole decision, over any reader of the -// group's keys so a test can reach every outcome with nothing running. -func resolveTelemetryDestination(local bool, read func(key string) string) (telemetryDestination, error) { - state := strings.TrimSpace(read(telemetryStateKey)) - endpoint := strings.TrimSpace(read(telemetryEndpointKey)) - reason := boundedLine(read(telemetryAbsentReasonKey), maxAbsentReasonLength) - - if state == "" && local { - state = telemetryStateAbsent - if reason == "" { - reason = localAbsentReason - } - } - if state == "" { - return telemetryDestination{}, errors.New( - "observability: " + telemetryStateKey + " is not set in the `observability` configuration group. " + - "The platform delivers `" + telemetryStateAvailable + "` (with " + telemetryEndpointKey + ") or `" + - telemetryStateAbsent + "` (with " + telemetryAbsentReasonKey + "); a group that did not arrive is " + - "not a cell without a collector, so this process does not start without it") - } - if state != telemetryStateAvailable && state != telemetryStateAbsent { - return telemetryDestination{}, fmt.Errorf( - "observability: %s is %q in the `observability` configuration group; it must be `%s` or `%s`", - telemetryStateKey, state, telemetryStateAvailable, telemetryStateAbsent) - } - - // The state decides: the key that belongs to the other state is a leftover - // of a lower layer and is neither read nor refused. The endpoint under - // `absent` is not even parsed, so a stale one cannot stop a cell that says - // it has no collector. - if state == telemetryStateAbsent { - if reason == "" { - return telemetryDestination{}, fmt.Errorf( - "observability: %s is `%s` but %s is not set in the `observability` configuration group; "+ - "an absent collector must say why", - telemetryStateKey, telemetryStateAbsent, telemetryAbsentReasonKey) - } - destination := telemetryDestination{AbsentReason: reason} - if endpoint != "" { - destination.Ignored = telemetryEndpointKey - } - return destination, nil - } - - if endpoint == "" { - return telemetryDestination{}, fmt.Errorf( - "observability: %s is `%s` but %s is not set in the `observability` configuration group", - telemetryStateKey, telemetryStateAvailable, telemetryEndpointKey) - } - origin, hostPort, insecure, err := parseCollectorEndpoint(endpoint) - if err != nil { - return telemetryDestination{}, fmt.Errorf("observability: %s: %w", telemetryEndpointKey, err) - } - destination := telemetryDestination{URL: origin, Endpoint: hostPort, Insecure: insecure} - if reason != "" { - destination.Ignored = telemetryAbsentReasonKey - } - return destination, nil -} - -// parseCollectorEndpoint turns the delivered URL into the two forms the -// exporters take — the origin with an explicit port, and host:port — and reads -// the transport off its scheme. The URL is never echoed back: a malformed one -// may carry credentials. -func parseCollectorEndpoint(raw string) (origin, hostPort string, insecure bool, err error) { - parsed, parseErr := url.Parse(raw) - if parseErr != nil { - return "", "", false, errors.New("is not a URL; expected http://host:port or https://host:port") - } - switch parsed.Scheme { - case "http": - insecure = true - case "https": - default: - return "", "", false, fmt.Errorf("must start with http:// or https://, got scheme %q", parsed.Scheme) - } - if parsed.Hostname() == "" { - return "", "", false, errors.New("has no host") - } - if parsed.User != nil { - return "", "", false, errors.New("must not carry credentials") - } - // OTLP/gRPC addresses a collector, not a path: a path or a query is the - // OTLP/HTTP form (/v1/traces), which this exporter does not speak. - if (parsed.Path != "" && parsed.Path != "/") || parsed.RawQuery != "" || parsed.Fragment != "" { - return "", "", false, errors.New("must be the collector's origin only, with no path, query or fragment") - } - // The port is made explicit because gRPC would otherwise dial 443 whatever - // the scheme, and a plaintext http:// collector with no port is not on 443. - port := parsed.Port() - if port == "" { - port = "80" - if !insecure { - port = "443" - } - } - hostPort = net.JoinHostPort(parsed.Hostname(), port) - return parsed.Scheme + "://" + hostPort, hostPort, insecure, nil -} - -// boundedLine collapses a free-text value to one bounded line. -func boundedLine(value string, limit int) string { - line := strings.Join(strings.Fields(value), " ") - if runes := []rune(line); len(runes) > limit { - return string(runes[:limit]) + "…" - } - return line -} - -// telemetryResource is the one resource both signals carry. service.name is -// whatever the environment says — OTEL_SERVICE_NAME, or service.name in -// OTEL_RESOURCE_ATTRIBUTES — and otherwise the service identity the Codefly SDK -// holds. No name is typed here: a literal would override the environment, so the -// platform's identity could never reach the backend. -func telemetryResource(ctx context.Context, identity string) (*resource.Resource, error) { - options := []resource.Option{resource.WithService()} - if identity != "" { - options = append(options, resource.WithAttributes(attribute.String("service.name", identity))) - } - // Last, so what the environment declares wins over the identity above. - options = append(options, resource.WithFromEnv()) - return resource.New(ctx, options...) -} - -// currentTelemetryResource is telemetryResource for this process. -func currentTelemetryResource(ctx context.Context) (*resource.Resource, error) { - return telemetryResource(ctx, codeflyServiceName(codefly.Context())) -} - -// codeflyServiceName is the service identity the Codefly SDK holds: the -// `module/service` identity of the provider it built at Init, which wool stamps on -// every log line from this process. It is empty when the SDK has not been -// initialised. The SDK exposes no accessor of its own, and the environment -// carrier it reads this from is not product code's to read, so this asks the -// provider. -func codeflyServiceName(ctx context.Context) string { - provider, ok := ctx.Value(wool.ProviderKey).(*wool.Provider) - if !ok || provider == nil { - return "" - } - source := provider.Get(ctx).Source() - if source == nil { - return "" - } - // An identity with no module reads `/service`; with neither, `/`. - return strings.Trim(strings.TrimSpace(source.Unique), "/") -} - -// telemetryServiceName is the service.name the resource resolved to. -func telemetryServiceName(res *resource.Resource) string { - if value, ok := res.Set().Value(attribute.Key("service.name")); ok { - return value.AsString() - } - return "" -} - -// errWoolCannotDialTLS names the one transport the tracer cannot serve. -var errWoolCannotDialTLS = errors.New( - "the collector endpoint is https://, but wool's OTLP tracer cannot dial TLS: wool/otel.Enable " + - "always sets an insecure transport and offers no option to turn it off. Deliver an http:// " + - "endpoint (in-mesh, where the mesh supplies mTLS) until wool can dial TLS") - -// enableTracing registers wool's tracer for the destination: OTLP/gRPC to the -// cell's collector when there is one; wool's own stdout tracer when there is -// none and this is a local run; nothing otherwise. A nil provider means nothing -// was registered. -func enableTracing(ctx context.Context, destination telemetryDestination, local bool) (*wooltel.Provider, error) { - res, err := currentTelemetryResource(ctx) - if err != nil { - return nil, fmt.Errorf("resolve telemetry resource: %w", err) - } - options := []wooltel.Option{wooltel.WithServiceName(telemetryServiceName(res))} - switch { - case destination.Available(): - // wool dials plaintext unless told otherwise and cannot be told - // otherwise, so an https:// endpoint is refused rather than dialed in - // the clear and reported as TLS. - if !destination.Insecure { - return nil, errWoolCannotDialTLS - } - options = append(options, wooltel.WithEndpoint(destination.Endpoint), wooltel.WithInsecure()) - case local: - options = append(options, wooltel.WithStdout()) - default: - return nil, nil - } - return wooltel.Enable(options...) -} diff --git a/module/services/accounts/code/telemetry_destination_test.go b/module/services/accounts/code/telemetry_destination_test.go deleted file mode 100644 index e06350de2..000000000 --- a/module/services/accounts/code/telemetry_destination_test.go +++ /dev/null @@ -1,623 +0,0 @@ -package main - -import ( - "context" - "net" - "os" - "strings" - "testing" - "time" - - "github.com/codefly-dev/core/wool" - "github.com/stretchr/testify/require" - "go.opentelemetry.io/otel" - collectortracev1 "go.opentelemetry.io/proto/otlp/collector/trace/v1" - "google.golang.org/grpc" -) - -// group is a reader over a fixed set of `observability` keys. -func group(values map[string]string) func(string) string { - return func(key string) string { return values[key] } -} - -const collector = "http://otel-collector.otel-collector.svc.cluster.local:4317" - -// shippedObservabilityProfile is what the module's own `observability` profile -// contributes to the group, read from the file it ships. -func shippedObservabilityProfile(t *testing.T) map[string]string { - t.Helper() - contents, err := os.ReadFile("../../../configurations/local/observability.env") - require.NoError(t, err) - values := map[string]string{} - for _, line := range strings.Split(string(contents), "\n") { - line = strings.TrimSpace(line) - if line == "" || strings.HasPrefix(line, "#") { - continue - } - key, value, ok := strings.Cut(line, "=") - require.True(t, ok) - values[key] = value - } - return values -} - -// Module defaults are also inherited by cells, so the profile ships neither the -// state nor the reason: a deployed workload must get both from the platform, and -// a local run gets them from code (localAbsentReason), only when it is local. -func TestTelemetryDestinationShippedProfileDeclaresNothingACellCouldInherit(t *testing.T) { - shipped := shippedObservabilityProfile(t) - require.NotContains(t, shipped, "TELEMETRY_STATE", "every deployed cell inherits the profile") - require.NotContains(t, shipped, "TELEMETRY_ABSENT_REASON", "every deployed cell inherits the profile") - require.Empty(t, shipped["OTEL_EXPORTER_OTLP_ENDPOINT"], "the profile keeps the group in existence and names no collector") - - _, err := resolveTelemetryDestination(false, group(shipped)) - require.ErrorContains(t, err, "TELEMETRY_STATE is not set") - destination, err := resolveTelemetryDestination(true, group(shipped)) - require.NoError(t, err) - require.False(t, destination.Available()) - require.Equal(t, localAbsentReason, destination.AbsentReason, "a local run supplies its own reason in code") - require.Empty(t, destination.IgnoredNotice()) -} - -// Layered over the shipped profile, a healthy cell leaves nothing to warn about, -// and a cell that says `absent` without a reason is refused rather than passed -// with the local-run reason. -func TestTelemetryDestinationCellAnswersAreNotMaskedByTheShippedProfile(t *testing.T) { - shipped := shippedObservabilityProfile(t) - - available, err := resolveTelemetryDestination(false, layered(shipped, map[string]string{ - "TELEMETRY_STATE": "available", - "OTEL_EXPORTER_OTLP_ENDPOINT": collector, - })) - require.NoError(t, err) - require.True(t, available.Available()) - require.Empty(t, available.Ignored, "an available cell must not log a leftover-key warning on every boot") - require.Empty(t, available.IgnoredNotice()) - - _, err = resolveTelemetryDestination(false, layered(shipped, map[string]string{"TELEMETRY_STATE": "absent"})) - require.ErrorContains(t, err, "TELEMETRY_ABSENT_REASON is not set") - - // The same answer from a local run is refused too: the local reason fills in - // a missing state, never an explicit `absent` that did not say why. - _, err = resolveTelemetryDestination(true, layered(shipped, map[string]string{"TELEMETRY_STATE": "absent"})) - require.ErrorContains(t, err, "TELEMETRY_ABSENT_REASON is not set") -} - -func TestTelemetryDestinationMissingStateDefaultsOnlyInLocalRuntime(t *testing.T) { - for _, values := range []map[string]string{ - {}, - {"TELEMETRY_STATE": " "}, - {"TELEMETRY_ABSENT_REASON": "local defaults inherited by a cell"}, - {"OTEL_EXPORTER_OTLP_ENDPOINT": collector}, - } { - _, err := resolveTelemetryDestination(false, group(values)) - require.ErrorContains(t, err, "TELEMETRY_STATE is not set") - destination, err := resolveTelemetryDestination(true, group(values)) - require.NoError(t, err) - require.False(t, destination.Available()) - require.NotEmpty(t, destination.AbsentReason) - } - for _, values := range []map[string]string{ - {"TELEMETRY_STATE": "unknown"}, - {"TELEMETRY_STATE": "available"}, - {"TELEMETRY_STATE": "absent"}, - } { - _, err := resolveTelemetryDestination(true, group(values)) - require.Error(t, err, "an explicit state must be valid and complete even locally") - } -} - -// The first outcome: the platform delivered a collector. -func TestTelemetryDestinationAvailableExportsToTheEndpoint(t *testing.T) { - destination, err := resolveTelemetryDestination(false, group(map[string]string{ - "TELEMETRY_STATE": "available", - "OTEL_EXPORTER_OTLP_ENDPOINT": collector, - })) - require.NoError(t, err) - require.True(t, destination.Available()) - require.Equal(t, collector, destination.URL, "the metrics exporter reads its transport off this URL") - require.Equal(t, "otel-collector.otel-collector.svc.cluster.local:4317", destination.Endpoint) - require.True(t, destination.Insecure, "http:// is plaintext on the wire; the mesh supplies mTLS") - require.Empty(t, destination.AbsentReason) - require.Empty(t, destination.Ignored) - require.Empty(t, destination.IgnoredNotice()) -} - -// The transport comes from the endpoint's scheme, never from a constant, and -// the URL handed to the metrics exporter always carries an explicit port: gRPC -// would otherwise dial 443 whatever the scheme. -func TestTelemetryDestinationReadsTransportFromTheScheme(t *testing.T) { - for endpoint, want := range map[string]struct { - url string - hostPort string - insecure bool - }{ - "http://collector.example:4317": {"http://collector.example:4317", "collector.example:4317", true}, - "https://collector.example:4317": {"https://collector.example:4317", "collector.example:4317", false}, - "https://collector.example": {"https://collector.example:443", "collector.example:443", false}, - "http://collector.example": {"http://collector.example:80", "collector.example:80", true}, - "http://collector.example:4317/": {"http://collector.example:4317", "collector.example:4317", true}, - " http://collector.example:4317 ": {"http://collector.example:4317", "collector.example:4317", true}, - "http://[2001:db8::1]:4317": {"http://[2001:db8::1]:4317", "[2001:db8::1]:4317", true}, - "https://10.0.0.7:4317": {"https://10.0.0.7:4317", "10.0.0.7:4317", false}, - "http://otel-collector.ns.svc:4317/": {"http://otel-collector.ns.svc:4317", "otel-collector.ns.svc:4317", true}, - } { - destination, err := resolveTelemetryDestination(false, group(map[string]string{ - "TELEMETRY_STATE": "available", - "OTEL_EXPORTER_OTLP_ENDPOINT": endpoint, - })) - require.NoError(t, err, endpoint) - require.Equal(t, want.url, destination.URL, endpoint) - require.Equal(t, want.hostPort, destination.Endpoint, endpoint) - require.Equal(t, want.insecure, destination.Insecure, endpoint) - } -} - -// The second outcome: the cell has no collector, and says why. The process boots -// and exports nothing. -func TestTelemetryDestinationAbsentCarriesTheReason(t *testing.T) { - destination, err := resolveTelemetryDestination(false, group(map[string]string{ - "TELEMETRY_STATE": "absent", - "TELEMETRY_ABSENT_REASON": "This cell's bucket has no ops cell, so no collector is rendered.", - })) - require.NoError(t, err) - require.False(t, destination.Available()) - require.Empty(t, destination.URL) - require.Empty(t, destination.Endpoint) - require.Equal(t, "This cell's bucket has no ops cell, so no collector is rendered.", destination.AbsentReason) - require.Empty(t, destination.Ignored) - require.Empty(t, destination.IgnoredNotice()) -} - -// The reason is operator-written free text that reaches a log line. -func TestTelemetryDestinationAbsentReasonIsOneBoundedLine(t *testing.T) { - destination, err := resolveTelemetryDestination(false, group(map[string]string{ - "TELEMETRY_STATE": "absent", - "TELEMETRY_ABSENT_REASON": "no collector\n2026-01-01 FORGED entry\r\n" + strings.Repeat("x", 1000), - })) - require.NoError(t, err) - require.NotContains(t, destination.AbsentReason, "\n") - require.NotContains(t, destination.AbsentReason, "\r") - require.LessOrEqual(t, len([]rune(destination.AbsentReason)), maxAbsentReasonLength+1) -} - -// The third outcome: anything else refuses to start, naming what is wrong. -// Configuration that did not arrive is never read as "no collector". A key the -// state does not use is not a refusal — it is ignored, below — but a key the -// state requires, missing, always is. -func TestTelemetryDestinationRefusesEverythingElse(t *testing.T) { - for name, tc := range map[string]struct { - values map[string]string - want string - }{ - "state missing": { - values: map[string]string{}, - want: "TELEMETRY_STATE is not set", - }, - "state missing though an ambient endpoint is present": { - values: map[string]string{"OTEL_EXPORTER_OTLP_ENDPOINT": collector}, - want: "TELEMETRY_STATE is not set", - }, - "state missing though a reason is present": { - values: map[string]string{"TELEMETRY_ABSENT_REASON": "no collector"}, - want: "TELEMETRY_STATE is not set", - }, - "state blank": { - values: map[string]string{"TELEMETRY_STATE": " "}, - want: "TELEMETRY_STATE is not set", - }, - "state unknown": { - values: map[string]string{"TELEMETRY_STATE": "disabled", "TELEMETRY_ABSENT_REASON": "no collector"}, - want: `TELEMETRY_STATE is "disabled"`, - }, - "state is not matched case-insensitively": { - values: map[string]string{"TELEMETRY_STATE": "Available", "OTEL_EXPORTER_OTLP_ENDPOINT": collector}, - want: `TELEMETRY_STATE is "Available"`, - }, - "available without an endpoint": { - values: map[string]string{"TELEMETRY_STATE": "available"}, - want: "OTEL_EXPORTER_OTLP_ENDPOINT is not set", - }, - "available with only a reason": { - values: map[string]string{"TELEMETRY_STATE": "available", "TELEMETRY_ABSENT_REASON": "no collector"}, - want: "OTEL_EXPORTER_OTLP_ENDPOINT is not set", - }, - "absent without a reason": { - values: map[string]string{"TELEMETRY_STATE": "absent"}, - want: "TELEMETRY_ABSENT_REASON is not set", - }, - "absent with a blank reason": { - values: map[string]string{"TELEMETRY_STATE": "absent", "TELEMETRY_ABSENT_REASON": " \n "}, - want: "TELEMETRY_ABSENT_REASON is not set", - }, - "absent with only an endpoint": { - values: map[string]string{"TELEMETRY_STATE": "absent", "OTEL_EXPORTER_OTLP_ENDPOINT": collector}, - want: "TELEMETRY_ABSENT_REASON is not set", - }, - "endpoint without a scheme": { - values: map[string]string{"TELEMETRY_STATE": "available", "OTEL_EXPORTER_OTLP_ENDPOINT": "otel-collector.ns.svc:4317"}, - want: "must start with http:// or https://", - }, - "endpoint with another scheme": { - values: map[string]string{"TELEMETRY_STATE": "available", "OTEL_EXPORTER_OTLP_ENDPOINT": "grpc://otel-collector.ns.svc:4317"}, - want: `got scheme "grpc"`, - }, - "endpoint without a host": { - values: map[string]string{"TELEMETRY_STATE": "available", "OTEL_EXPORTER_OTLP_ENDPOINT": "http://:4317"}, - want: "has no host", - }, - "endpoint naming an OTLP/HTTP path": { - values: map[string]string{"TELEMETRY_STATE": "available", "OTEL_EXPORTER_OTLP_ENDPOINT": "http://collector.example:4318/v1/traces"}, - want: "origin only", - }, - "endpoint with a query": { - values: map[string]string{"TELEMETRY_STATE": "available", "OTEL_EXPORTER_OTLP_ENDPOINT": "http://collector.example:4317?x=1"}, - want: "origin only", - }, - } { - t.Run(name, func(t *testing.T) { - destination, err := resolveTelemetryDestination(false, group(tc.values)) - require.Error(t, err) - require.ErrorContains(t, err, tc.want) - require.False(t, destination.Available(), "a refusal must never read as a usable destination") - require.Empty(t, destination.AbsentReason, "a refusal must never read as an absent collector") - }) - } -} - -// The state decides. The group is layered, so a deployed producer can still see -// `available` and an endpoint from the cell beside a reason a lower layer (a -// consuming workspace's own profile, say) left behind. That is not a -// contradiction to refuse: the reason is ignored, and said so once. -func TestTelemetryDestinationAvailableIgnoresALeftoverReason(t *testing.T) { - destination, err := resolveTelemetryDestination(false, group(map[string]string{ - "TELEMETRY_STATE": "available", - "OTEL_EXPORTER_OTLP_ENDPOINT": collector, - "TELEMETRY_ABSENT_REASON": "A local run has no cell collector.", - })) - require.NoError(t, err) - require.True(t, destination.Available()) - require.Equal(t, collector, destination.URL) - require.Empty(t, destination.AbsentReason, "a leftover reason must not turn an available collector into an absent one") - require.Equal(t, "TELEMETRY_ABSENT_REASON", destination.Ignored) - notice := destination.IgnoredNotice() - require.Contains(t, notice, "TELEMETRY_ABSENT_REASON is set but TELEMETRY_STATE is `available`") - require.NotContains(t, notice, "A local run", "the notice names the key, never its free-text value") -} - -// ... and the other way round: a cell that says it has no collector is not -// refused for an endpoint a lower layer left behind, and that endpoint is not -// even parsed. -func TestTelemetryDestinationAbsentIgnoresALeftoverEndpoint(t *testing.T) { - for name, leftover := range map[string]string{ - "a usable endpoint": collector, - "a malformed endpoint": "otel-collector.ns.svc:4317", - "an endpoint with a user": "http://user:hunter2@collector.example:4317", - "an unreadable endpoint": "http://[::1", - } { - t.Run(name, func(t *testing.T) { - destination, err := resolveTelemetryDestination(false, group(map[string]string{ - "TELEMETRY_STATE": "absent", - "TELEMETRY_ABSENT_REASON": "This cell has no ops cell.", - "OTEL_EXPORTER_OTLP_ENDPOINT": leftover, - })) - require.NoError(t, err) - require.False(t, destination.Available(), "a leftover endpoint must not turn an absent collector into an available one") - require.Empty(t, destination.URL) - require.Empty(t, destination.Endpoint) - require.Equal(t, "This cell has no ops cell.", destination.AbsentReason) - require.Equal(t, "OTEL_EXPORTER_OTLP_ENDPOINT", destination.Ignored) - notice := destination.IgnoredNotice() - require.Contains(t, notice, "OTEL_EXPORTER_OTLP_ENDPOINT is set but TELEMETRY_STATE is `absent`") - require.NotContains(t, notice, "hunter2", "the notice names the key, never its value") - }) - } -} - -// A blank leftover is nothing left over. -func TestTelemetryDestinationBlankLeftoversAreNotNoticed(t *testing.T) { - available, err := resolveTelemetryDestination(false, group(map[string]string{ - "TELEMETRY_STATE": "available", "OTEL_EXPORTER_OTLP_ENDPOINT": collector, "TELEMETRY_ABSENT_REASON": " \n ", - })) - require.NoError(t, err) - require.Empty(t, available.IgnoredNotice()) - - absent, err := resolveTelemetryDestination(false, group(map[string]string{ - "TELEMETRY_STATE": "absent", "TELEMETRY_ABSENT_REASON": "no collector", "OTEL_EXPORTER_OTLP_ENDPOINT": " ", - })) - require.NoError(t, err) - require.Empty(t, absent.IgnoredNotice()) -} - -// layered is what the platform hands a workload: each layer overrides the one -// below it key by key, so a key a higher layer does not set survives from the -// lower one. -func layered(layers ...map[string]string) func(string) string { - merged := map[string]string{} - for _, layer := range layers { - for key, value := range layer { - merged[key] = value - } - } - return group(merged) -} - -// The case this exists for: a lower layer of the group says `absent` with a -// reason, and a deployed cell overrides the state and the endpoint but not the -// reason. The cell's answer wins, whichever way it points. -func TestTelemetryDestinationTheCellOverridesALowerLayer(t *testing.T) { - moduleDefault := map[string]string{ - "TELEMETRY_STATE": "absent", - "TELEMETRY_ABSENT_REASON": "A local run has no cell collector.", - } - - t.Run("a cell with a collector overrides a lower layer's absent default", func(t *testing.T) { - destination, err := resolveTelemetryDestination(false, layered(moduleDefault, map[string]string{ - "TELEMETRY_STATE": "available", - "OTEL_EXPORTER_OTLP_ENDPOINT": collector, - })) - require.NoError(t, err) - require.True(t, destination.Available()) - require.Equal(t, collector, destination.URL) - require.Empty(t, destination.AbsentReason) - require.Equal(t, "TELEMETRY_ABSENT_REASON", destination.Ignored) - }) - - t.Run("a cell with no collector, over a default that had an endpoint, is absent", func(t *testing.T) { - destination, err := resolveTelemetryDestination(false, layered( - map[string]string{"TELEMETRY_STATE": "available", "OTEL_EXPORTER_OTLP_ENDPOINT": collector}, - map[string]string{"TELEMETRY_STATE": "absent", "TELEMETRY_ABSENT_REASON": "This cell has no ops cell."}, - )) - require.NoError(t, err) - require.False(t, destination.Available()) - require.Equal(t, "This cell has no ops cell.", destination.AbsentReason) - require.Equal(t, "OTEL_EXPORTER_OTLP_ENDPOINT", destination.Ignored) - }) - - t.Run("a cell that overrides only the state still has to supply what it requires", func(t *testing.T) { - _, err := resolveTelemetryDestination(false, layered(moduleDefault, map[string]string{"TELEMETRY_STATE": "available"})) - require.ErrorContains(t, err, "OTEL_EXPORTER_OTLP_ENDPOINT is not set", - "the module default's reason is not an endpoint") - - _, err = resolveTelemetryDestination(false, layered( - map[string]string{"TELEMETRY_STATE": "available", "OTEL_EXPORTER_OTLP_ENDPOINT": collector}, - map[string]string{"TELEMETRY_STATE": "absent"}, - )) - require.ErrorContains(t, err, "TELEMETRY_ABSENT_REASON is not set", - "the module default's endpoint is not a reason") - }) -} - -// A refusal quotes no part of an endpoint, which may carry credentials. -func TestTelemetryDestinationRefusalDoesNotEchoCredentials(t *testing.T) { - _, err := resolveTelemetryDestination(false, group(map[string]string{ - "TELEMETRY_STATE": "available", - "OTEL_EXPORTER_OTLP_ENDPOINT": "http://user:hunter2@collector.example:4317", - })) - require.ErrorContains(t, err, "must not carry credentials") - require.NotContains(t, err.Error(), "hunter2") -} - -// The group is read through the SDK's workspace-configuration lookup, by the -// names the platform delivers them under. A bare process variable of the same -// name is not the cell's answer. -func TestConfiguredTelemetryDestinationReadsTheObservabilityGroup(t *testing.T) { - const prefix = "CODEFLY__WORKSPACE_CONFIGURATION__OBSERVABILITY__" - for _, key := range []string{"TELEMETRY_STATE", "OTEL_EXPORTER_OTLP_ENDPOINT", "TELEMETRY_ABSENT_REASON"} { - t.Setenv(key, "") - t.Setenv(prefix+key, "") - } - - localDestination, err := configuredTelemetryDestination(true) - require.NoError(t, err) - require.False(t, localDestination.Available()) - require.NotEmpty(t, localDestination.AbsentReason) - _, err = configuredTelemetryDestination(false) - require.ErrorContains(t, err, "TELEMETRY_STATE is not set") - - // Ambient variables a shell or a pod may carry are not the group. - t.Setenv("TELEMETRY_STATE", "available") - t.Setenv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://ambient.example:4318") - _, err = configuredTelemetryDestination(false) - require.ErrorContains(t, err, "TELEMETRY_STATE is not set") - t.Setenv("TELEMETRY_STATE", "") - t.Setenv("OTEL_EXPORTER_OTLP_ENDPOINT", "") - - t.Setenv(prefix+"TELEMETRY_STATE", "available") - t.Setenv(prefix+"OTEL_EXPORTER_OTLP_ENDPOINT", collector) - destination, err := configuredTelemetryDestination(false) - require.NoError(t, err) - require.Equal(t, "otel-collector.otel-collector.svc.cluster.local:4317", destination.Endpoint) - require.True(t, destination.Insecure) - - t.Setenv(prefix+"TELEMETRY_STATE", "absent") - t.Setenv(prefix+"OTEL_EXPORTER_OTLP_ENDPOINT", "") - t.Setenv(prefix+"TELEMETRY_ABSENT_REASON", "a local run has no cell collector") - destination, err = configuredTelemetryDestination(false) - require.NoError(t, err) - require.False(t, destination.Available()) - require.Equal(t, "a local run has no cell collector", destination.AbsentReason) -} - -// Through the SDK's lookup, with the keys exactly as a deployed cell delivers -// them after the layers merge: the cell's state and endpoint, and a reason a -// lower layer left behind. -func TestConfiguredTelemetryDestinationStateDecidesAcrossLayers(t *testing.T) { - const prefix = "CODEFLY__WORKSPACE_CONFIGURATION__OBSERVABILITY__" - t.Setenv(prefix+"TELEMETRY_STATE", "available") - t.Setenv(prefix+"OTEL_EXPORTER_OTLP_ENDPOINT", collector) - t.Setenv(prefix+"TELEMETRY_ABSENT_REASON", "A local run has no cell collector.") - - destination, err := configuredTelemetryDestination(false) - require.NoError(t, err) - require.True(t, destination.Available()) - require.Equal(t, collector, destination.URL) - require.NotEmpty(t, destination.IgnoredNotice()) -} - -// Configuration that cannot work is refused before the process acquires -// anything: a group that did not arrive is not a cell without a collector. -// Only a local runtime can infer an absent collector from the missing state. -func TestStartupRefusesAnObservabilityGroupThatDidNotArrive(t *testing.T) { - const prefix = "CODEFLY__WORKSPACE_CONFIGURATION__OBSERVABILITY__" - for _, key := range []string{"TELEMETRY_STATE", "OTEL_EXPORTER_OTLP_ENDPOINT", "TELEMETRY_ABSENT_REASON"} { - t.Setenv(prefix+key, "") - } - // The key-custody requirement answers first outside a local run; satisfy it - // so the observability requirement is what is under test. - t.Setenv("CODEFLY__WORKSPACE_CONFIGURATION__VAULT__VAULT_KEY_CUSTODY", "seed-signing-key") - require.NoError(t, requireStartupConfiguration(t.Context(), true)) - require.ErrorContains(t, requireStartupConfiguration(t.Context(), false), "TELEMETRY_STATE is not set") - - t.Setenv(prefix+"TELEMETRY_STATE", "absent") - t.Setenv(prefix+"TELEMETRY_ABSENT_REASON", "a local run has no cell collector") - require.NoError(t, requireStartupConfiguration(t.Context(), true)) - require.NoError(t, requireStartupConfiguration(t.Context(), false)) -} - -// service.name is never a literal typed here: the environment wins, then the -// service's own Codefly identity. -func TestTelemetryResourceServiceName(t *testing.T) { - resolve := func(t *testing.T, identity string) string { - t.Helper() - res, err := telemetryResource(t.Context(), identity) - require.NoError(t, err) - return telemetryServiceName(res) - } - - t.Run("falls back to the Codefly service identity", func(t *testing.T) { - t.Setenv("OTEL_SERVICE_NAME", "") - t.Setenv("OTEL_RESOURCE_ATTRIBUTES", "") - require.Equal(t, "saas-starter/accounts", resolve(t, "saas-starter/accounts")) - }) - t.Run("OTEL_SERVICE_NAME wins over the identity", func(t *testing.T) { - t.Setenv("OTEL_SERVICE_NAME", "named-by-the-platform") - t.Setenv("OTEL_RESOURCE_ATTRIBUTES", "") - require.Equal(t, "named-by-the-platform", resolve(t, "saas-starter/accounts")) - }) - t.Run("OTEL_RESOURCE_ATTRIBUTES can set it", func(t *testing.T) { - t.Setenv("OTEL_SERVICE_NAME", "") - t.Setenv("OTEL_RESOURCE_ATTRIBUTES", "service.name=attributed-by-the-platform,deployment.environment=test") - require.Equal(t, "attributed-by-the-platform", resolve(t, "saas-starter/accounts")) - }) - t.Run("without either, the SDK default rather than a literal", func(t *testing.T) { - t.Setenv("OTEL_SERVICE_NAME", "") - t.Setenv("OTEL_RESOURCE_ATTRIBUTES", "") - require.True(t, strings.HasPrefix(resolve(t, ""), "unknown_service"), resolve(t, "")) - }) -} - -// The identity is the one the SDK built its provider with, the same one wool -// stamps on this process's log lines — read from the provider, not from the -// runtime's environment carrier. -func TestCodeflyServiceNameIsTheSDKProviderIdentity(t *testing.T) { - carrying := func(unique string) context.Context { - return wool.New(t.Context(), &wool.Resource{Kind: "service", Unique: unique}).Inject(t.Context()) - } - require.Equal(t, "saas-starter/accounts", codeflyServiceName(carrying("saas-starter/accounts"))) - require.Equal(t, "accounts", codeflyServiceName(carrying("/accounts")), "an identity with no module") - require.Empty(t, codeflyServiceName(carrying("/")), "an identity with neither") - require.Empty(t, codeflyServiceName(t.Context()), "no provider: the SDK was not initialised") -} - -// wool's tracer speaks plaintext only. An https:// endpoint is refused rather -// than dialed in the clear and reported as TLS. -func TestEnableTracingRefusesATransportWoolCannotDial(t *testing.T) { - provider, err := enableTracing(t.Context(), - telemetryDestination{Endpoint: "collector.example:4317", Insecure: false}, false) - require.ErrorIs(t, err, errWoolCannotDialTLS) - require.Nil(t, provider) -} - -func TestEnableTracingRegistersOnlyWhatTheDestinationCalls(t *testing.T) { - restore := func(t *testing.T) { - t.Helper() - previous := otel.GetTracerProvider() - t.Cleanup(func() { - wool.RegisterTelemetry(nil) - otel.SetTracerProvider(previous) - }) - } - absent := telemetryDestination{AbsentReason: "a local run has no cell collector"} - - t.Run("absent outside a local run registers no tracer", func(t *testing.T) { - restore(t) - provider, err := enableTracing(t.Context(), absent, false) - require.NoError(t, err) - require.Nil(t, provider) - require.False(t, wool.TelemetryEnabled()) - }) - t.Run("absent in a local run uses wool's stdout tracer", func(t *testing.T) { - restore(t) - provider, err := enableTracing(t.Context(), absent, true) - require.NoError(t, err) - require.NotNil(t, provider) - require.True(t, wool.TelemetryEnabled()) - shutdown(t, provider) - }) - t.Run("available over http exports spans to the collector, named by the environment", func(t *testing.T) { - restore(t) - t.Setenv("OTEL_SERVICE_NAME", "named-by-the-platform") - t.Setenv("OTEL_RESOURCE_ATTRIBUTES", "") - endpoint, capture := startTraceCapture(t) - provider, err := enableTracing(t.Context(), telemetryDestination{Endpoint: endpoint, Insecure: true}, false) - require.NoError(t, err) - require.NotNil(t, provider) - require.True(t, wool.TelemetryEnabled()) - - _, span := otel.Tracer("test").Start(t.Context(), "operation") - span.End() - shutdown(t, provider) // flushes the batch - - select { - case request := <-capture.requests: - var name string - for _, resourceSpans := range request.GetResourceSpans() { - for _, attribute := range resourceSpans.GetResource().GetAttributes() { - if attribute.GetKey() == "service.name" { - name = attribute.GetValue().GetStringValue() - } - } - } - require.Equal(t, "named-by-the-platform", name) - case <-time.After(5 * time.Second): - t.Fatal("timed out waiting for OTLP spans") - } - }) -} - -type traceCapture struct { - collectortracev1.UnimplementedTraceServiceServer - requests chan *collectortracev1.ExportTraceServiceRequest -} - -func (c *traceCapture) Export( - _ context.Context, - request *collectortracev1.ExportTraceServiceRequest, -) (*collectortracev1.ExportTraceServiceResponse, error) { - c.requests <- request - return &collectortracev1.ExportTraceServiceResponse{}, nil -} - -// startTraceCapture serves an in-process OTLP/gRPC trace receiver on loopback. -func startTraceCapture(t *testing.T) (string, *traceCapture) { - t.Helper() - listener, err := net.Listen("tcp", "127.0.0.1:0") - require.NoError(t, err) - capture := &traceCapture{requests: make(chan *collectortracev1.ExportTraceServiceRequest, 4)} - server := grpc.NewServer() - collectortracev1.RegisterTraceServiceServer(server, capture) - go func() { _ = server.Serve(listener) }() - t.Cleanup(func() { - server.Stop() - _ = listener.Close() - }) - return listener.Addr().String(), capture -} - -func shutdown(t *testing.T, provider interface{ Shutdown(context.Context) error }) { - t.Helper() - ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) - defer cancel() - require.NoError(t, provider.Shutdown(ctx)) -} diff --git a/module/services/accounts/code/telemetry_metrics.go b/module/services/accounts/code/telemetry_metrics.go index b7ce54122..b088b2346 100644 --- a/module/services/accounts/code/telemetry_metrics.go +++ b/module/services/accounts/code/telemetry_metrics.go @@ -2,46 +2,59 @@ package main import ( "context" - "errors" + "net/http" runtimemetrics "runtime/metrics" "time" + "github.com/prometheus/client_golang/prometheus" + "github.com/prometheus/client_golang/prometheus/promhttp" otelruntime "go.opentelemetry.io/contrib/instrumentation/runtime" "go.opentelemetry.io/otel" + "go.opentelemetry.io/otel/attribute" "go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc" + otelprometheus "go.opentelemetry.io/otel/exporters/prometheus" otelmetric "go.opentelemetry.io/otel/metric" "go.opentelemetry.io/otel/sdk/metric" + "go.opentelemetry.io/otel/sdk/resource" ) -// otelMetrics owns the process's MeterProvider. Metrics leave by one path only: -// an OTLP push to the cell's collector, which is the record for traces and -// metrics alike. There is no scrape endpoint to mount, expose or exempt. type otelMetrics struct { provider *metric.MeterProvider + handler http.Handler } -func enableOTELMetrics(ctx context.Context, destination telemetryDestination) (*otelMetrics, error) { - if destination.URL == "" { - // WithEndpointURL("") would fall back to the exporter's own default, - // localhost:4317: a process that exports to nowhere and reports success. - return nil, errors.New("telemetry: no collector URL to export metrics to") +func enableOTELMetrics(ctx context.Context, serviceName, endpoint string) (*otelMetrics, error) { + exporter, err := otlpmetricgrpc.New( + ctx, + otlpmetricgrpc.WithEndpoint(endpoint), + otlpmetricgrpc.WithInsecure(), + ) + if err != nil { + return nil, err } - // The resource comes before the exporter, so a failure here leaves nothing - // to shut down: an exporter holds a client connection that only Shutdown - // releases. - res, err := currentTelemetryResource(ctx) + registry := prometheus.NewRegistry() + scrapeExporter, err := otelprometheus.New( + otelprometheus.WithRegisterer(registry), + otelprometheus.WithResourceAsConstantLabels( + attribute.NewAllowKeysFilter("service.name"), + ), + ) if err != nil { return nil, err } - // The exporter reads the transport off the URL's scheme itself: http:// is - // plaintext on the wire, because the mesh supplies mTLS, and https:// is TLS. - exporter, err := otlpmetricgrpc.New(ctx, otlpmetricgrpc.WithEndpointURL(destination.URL)) + res, err := resource.New( + ctx, + resource.WithService(), + resource.WithFromEnv(), + resource.WithAttributes(attribute.String("service.name", serviceName)), + ) if err != nil { return nil, err } provider := metric.NewMeterProvider( metric.WithResource(res), metric.WithReader(metric.NewPeriodicReader(exporter, metric.WithInterval(30*time.Second))), + metric.WithReader(scrapeExporter), ) if err := otelruntime.Start(otelruntime.WithMeterProvider(provider)); err != nil { _ = provider.Shutdown(ctx) @@ -52,7 +65,10 @@ func enableOTELMetrics(ctx context.Context, destination telemetryDestination) (* return nil, err } otel.SetMeterProvider(provider) - return &otelMetrics{provider: provider}, nil + return &otelMetrics{ + provider: provider, + handler: promhttp.HandlerFor(registry, promhttp.HandlerOpts{}), + }, nil } func startGCActivityMetrics(provider *metric.MeterProvider) error { @@ -86,6 +102,10 @@ func startGCActivityMetrics(provider *metric.MeterProvider) error { return err } +func (m *otelMetrics) Handler() http.Handler { + return m.handler +} + func (m *otelMetrics) Shutdown(ctx context.Context) error { return m.provider.Shutdown(ctx) } diff --git a/module/services/accounts/code/telemetry_metrics_test.go b/module/services/accounts/code/telemetry_metrics_test.go index 4ae75bfd9..0429dcd90 100644 --- a/module/services/accounts/code/telemetry_metrics_test.go +++ b/module/services/accounts/code/telemetry_metrics_test.go @@ -60,7 +60,6 @@ func (versionConnectHandler) Version( func TestTelemetryMetricsExportRuntimeAndUnsampledRED(t *testing.T) { endpoint, capture := startMetricCapture(t) t.Setenv("OTEL_RESOURCE_ATTRIBUTES", "service.instance.id=test-instance") - t.Setenv("OTEL_SERVICE_NAME", "named-by-the-platform") previousMeterProvider := otel.GetMeterProvider() previousTracerProvider := otel.GetTracerProvider() @@ -71,7 +70,7 @@ func TestTelemetryMetricsExportRuntimeAndUnsampledRED(t *testing.T) { ) otel.SetTracerProvider(tracerProvider) - metrics, err := enableOTELMetrics(t.Context(), availableAt(t, "http", endpoint)) + metrics, err := enableOTELMetrics(t.Context(), "test-service", endpoint) require.NoError(t, err) require.NotNil(t, metrics) t.Cleanup(func() { @@ -90,7 +89,6 @@ func TestTelemetryMetricsExportRuntimeAndUnsampledRED(t *testing.T) { require.NoError(t, metrics.provider.ForceFlush(t.Context())) request := receiveMetrics(t, capture.requests) require.Equal(t, "test-instance", resourceAttribute(request, "service.instance.id")) - require.Equal(t, "named-by-the-platform", resourceAttribute(request, "service.name")) metricNames := exportedMetricNames(request) require.Contains(t, metricNames, "go.goroutine.count") require.Contains(t, metricNames, "go.memory.allocated") @@ -109,65 +107,24 @@ func TestTelemetryMetricsExportRuntimeAndUnsampledRED(t *testing.T) { require.Equal(t, uint64(1), connectCounts["permission_denied"]) require.Empty(t, spanExporter.GetSpans()) -} - -// availableAt resolves the destination the way the process does, from the -// endpoint the platform would deliver, so these tests hand the exporter the URL -// it really receives rather than a hand-built struct. -func availableAt(t *testing.T, scheme, hostPort string) telemetryDestination { - t.Helper() - destination, err := resolveTelemetryDestination(false, group(map[string]string{ - "TELEMETRY_STATE": "available", - "OTEL_EXPORTER_OTLP_ENDPOINT": scheme + "://" + hostPort, - })) - require.NoError(t, err) - return destination -} -// The exporter reads its transport off the URL's scheme, so an https:// URL is -// TLS and cannot talk to a plaintext receiver. A constant WithInsecure() would -// have let it. -func TestEnableOTELMetricsDerivesTransportFromTheURLScheme(t *testing.T) { - endpoint, capture := startMetricCapture(t) - previousMeterProvider := otel.GetMeterProvider() - metrics, err := enableOTELMetrics(t.Context(), availableAt(t, "https", endpoint)) - require.NoError(t, err) - require.NotNil(t, metrics) - t.Cleanup(func() { - shutdownCtx, cancel := context.WithTimeout(context.Background(), time.Second) - defer cancel() - _ = metrics.Shutdown(shutdownCtx) // the flush it attempts is the failure under test - otel.SetMeterProvider(previousMeterProvider) - }) - - flushCtx, cancel := context.WithTimeout(t.Context(), time.Second) - defer cancel() - require.Error(t, metrics.provider.ForceFlush(flushCtx), "a TLS client cannot export to a plaintext receiver") - select { - case <-capture.requests: - t.Fatal("an https:// destination reached a plaintext receiver") - default: - } -} - -// A destination with no collector URL is refused. WithEndpointURL("") would -// otherwise fall back to the exporter's own default, localhost:4317. -func TestEnableOTELMetricsRefusesADestinationWithoutAURL(t *testing.T) { - metrics, err := enableOTELMetrics(t.Context(), telemetryDestination{AbsentReason: "no collector"}) - require.ErrorContains(t, err, "no collector URL") - require.Nil(t, metrics) -} - -// A resource that cannot be built fails the whole call, and the call builds it -// before it creates the exporter: there is no exporter, and so no client -// connection, left running behind the failure. -func TestEnableOTELMetricsFailsWhenTheResourceCannotBeBuilt(t *testing.T) { - t.Setenv("OTEL_SERVICE_NAME", "") - t.Setenv("OTEL_RESOURCE_ATTRIBUTES", "an-attribute-with-no-value") - endpoint, _ := startMetricCapture(t) - metrics, err := enableOTELMetrics(t.Context(), availableAt(t, "http", endpoint)) - require.Error(t, err) - require.Nil(t, metrics) + recorder := httptest.NewRecorder() + metrics.Handler().ServeHTTP( + recorder, + httptest.NewRequest(http.MethodGet, "http://accounts.internal/metrics", nil), + ) + require.Equal(t, http.StatusOK, recorder.Code) + require.Contains(t, recorder.Header().Get("Content-Type"), "text/plain") + body := recorder.Body.String() + require.Contains(t, body, "go_goroutine_count") + require.Contains(t, body, "go_memory_allocated") + require.Contains(t, body, "go_memory_gc_goal") + require.Contains(t, body, "go_gc_cycle_count") + require.Contains(t, body, "go_gc_pause_cpu_time") + require.Contains(t, body, "rpc_server_call_duration_seconds_count") + require.Contains(t, body, `rpc_method="grpc.health.v1.Health/Check"`) + require.Contains(t, body, `rpc_response_status_code="OK"`) + require.Contains(t, body, `service_name="test-service"`) } func startMetricCapture(t *testing.T) (string, *metricCapture) { diff --git a/module/services/accounts/code/work.go b/module/services/accounts/code/work.go index 5d85af456..ad542223e 100644 --- a/module/services/accounts/code/work.go +++ b/module/services/accounts/code/work.go @@ -38,6 +38,7 @@ import ( "time" "github.com/codefly-dev/core/wool" + wooltel "github.com/codefly-dev/core/wool/otel" codefly "github.com/codefly-dev/sdk-go" "github.com/google/uuid" "go.opentelemetry.io/otel" @@ -82,22 +83,33 @@ func doWork(ctx context.Context) (Clean, error) { return nil, err } - // Traces and metrics go to the cell's collector, whose address the platform - // delivers in the `observability` configuration group — or the group says the - // cell has none. Either answer was already required before anything was - // acquired (requireStartupConfiguration); this reads it again to act on it. - telemetryDestination, err := configuredTelemetryDestination(codefly.IsLocal()) - if err != nil { - return nil, err - } - // A local run with no cell collector gets wool's own stdout tracer instead. - otelProvider, err := enableTracing(ctx, telemetryDestination, codefly.IsLocal()) - if err != nil { - return nil, fmt.Errorf("configure OTEL tracing: %w", err) - } + // When external observability is configured, OpenTelemetry always targets + // the in-graph collector. Codefly owns its host and port; the accounts + // process never reads or hardcodes a collector address. + var otelProvider *wooltel.Provider var otelMetricProvider *otelMetrics - if telemetryDestination.Available() { - metricProvider, oerr := enableOTELMetrics(ctx, telemetryDestination) + if observabilityEnabled() { + collectorNetwork, err := codefly.For(ctx). + Service("telemetry"). + Endpoint("grpc"). + API("grpc"). + ResolveNetworkInstance() + if err != nil { + return nil, fmt.Errorf("resolve telemetry collector through Codefly: %w", err) + } + p, oerr := wooltel.Enable( + wooltel.WithServiceName("saas-starter-api"), + wooltel.WithEndpoint(collectorNetwork.Host), + wooltel.WithInsecure(), + ) + if oerr != nil { + return nil, fmt.Errorf("configure OTEL tracing: %w", oerr) + } + otelProvider = p + w.Info("OTEL enabled", + wool.Field("endpoint", collectorNetwork.Host), + wool.Field("service.name", "saas-starter-api")) + metricProvider, oerr := enableOTELMetrics(ctx, "saas-starter-api", collectorNetwork.Host) if oerr != nil { shutdownCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second) _ = otelProvider.Shutdown(shutdownCtx) @@ -105,17 +117,8 @@ func doWork(ctx context.Context) (Clean, error) { return nil, fmt.Errorf("configure OTEL metrics: %w", oerr) } otelMetricProvider = metricProvider - w.Info("OTEL enabled", - wool.Field("endpoint", telemetryDestination.Endpoint), - wool.Field("insecure", telemetryDestination.Insecure)) - } else { - w.Info("OTEL export disabled: the cell has no collector", - wool.Field("reason", telemetryDestination.AbsentReason)) - } - // The state decided; a key it does not use is a leftover, reported here once. - // requireStartupConfiguration reads the group too, but only to refuse. - if notice := telemetryDestination.IgnoredNotice(); notice != "" { - w.Warn(notice) + // codefly:gateway-route-exempt the OTEL scrape endpoint; the collector reaches it over the mesh and it must never be public + adapters.RegisterHTTPRoute("/metrics", otelMetricProvider.Handler()) } store, err := infra.NewPostgresStore(ctx) @@ -168,10 +171,10 @@ func doWork(ctx context.Context) (Clean, error) { } jobStore := infra.NewPostgresJobStore(jobWorkerPool) // Durable job-operations metrics are enabled only when OTEL metrics are, i.e. - // otelMetricProvider != nil (the cell has a collector). Without one the global - // meter is a no-op, so building the monitor would poll the job store every - // interval to feed instruments nothing can read; keep it nil and let the - // Start/Shutdown nil-checks below skip it entirely. + // otelMetricProvider != nil (observabilityEnabled()). With observability off + // the global meter is a no-op, so building the monitor would poll the job + // store every interval to feed instruments nothing can read; keep it nil and + // let the Start/Shutdown nil-checks below skip it entirely. jobOperationsMonitor, err := newDurableJobMetricsMonitor(otelMetricProvider != nil, jobStore) if err != nil { return nil, fmt.Errorf("configure durable job metrics: %w", err) @@ -1986,9 +1989,13 @@ func workspaceEnv(configuration, key string) string { return os.Getenv(key) } +func observabilityEnabled() bool { + return strings.TrimSpace(workspaceEnv("observability", "OTEL_EXPORTER_OTLP_ENDPOINT")) != "" +} + // newDurableJobMetricsMonitor builds the durable job-operations metrics monitor // only when OTEL metrics are enabled (metricsEnabled mirrors a non-nil -// otelMetricProvider, i.e. the cell has a collector). When metrics are disabled the +// otelMetricProvider, i.e. observabilityEnabled()). When metrics are disabled the // global meter is a no-op, so a monitor would poll the job store every interval // only to record into instruments nothing can read; returning a nil monitor lets // the caller's Start/Shutdown nil-checks skip that background work entirely. @@ -2554,21 +2561,7 @@ func requireStartupConfiguration(ctx context.Context, isLocal bool) error { if err != nil { return err } - if err := requireKeyCustody(selection.SigningBackend, isLocal); err != nil { - return err - } - return requireTelemetryConfiguration(isLocal) -} - -// requireTelemetryConfiguration refuses to start unless the `observability` -// group says where the cell's collector is, or that the cell has none. It only -// checks: doWork reads the same answer again to act on it. A group that did not -// arrive is refused outside the local runtime, before the database or Vault is -// touched: reading it as "no collector" would leave a cell exporting nothing -// while it converges green. -func requireTelemetryConfiguration(isLocal bool) error { - _, err := configuredTelemetryDestination(isLocal) - return err + return requireKeyCustody(selection.SigningBackend, isLocal) } // requireKeyCustody refuses to start outside the local environment without the diff --git a/module/services/accounts/code/work_auth_test.go b/module/services/accounts/code/work_auth_test.go index d520c8868..9fd49b998 100644 --- a/module/services/accounts/code/work_auth_test.go +++ b/module/services/accounts/code/work_auth_test.go @@ -95,6 +95,25 @@ func TestWorkspaceEnvPreservesExactCodeflyConfigurationName(t *testing.T) { require.Equal(t, "legacy-normalized", workspaceEnv("internal-auth", key)) } +func TestObservabilityEnabledRequiresExternalOTLPEndpoint(t *testing.T) { + const key = "OTEL_EXPORTER_OTLP_ENDPOINT" + configurationKey := "CODEFLY__WORKSPACE_CONFIGURATION__OBSERVABILITY__" + key + secretKey := "CODEFLY__WORKSPACE_SECRET_CONFIGURATION__OBSERVABILITY__" + key + t.Setenv(key, "") + t.Setenv(configurationKey, "") + t.Setenv(secretKey, "") + require.False(t, observabilityEnabled()) + + t.Setenv(key, " ") + require.False(t, observabilityEnabled()) + t.Setenv(key, "https://otel.example") + require.True(t, observabilityEnabled()) + + t.Setenv(key, "") + t.Setenv(configurationKey, "https://workspace-otel.example") + require.True(t, observabilityEnabled()) +} + func TestConfiguredMFAStepUpMaxAge(t *testing.T) { t.Setenv("CODEFLY__WORKSPACE_CONFIGURATION__SECURITY__MFA_STEP_UP_MAX_AGE", "") t.Setenv("CODEFLY__WORKSPACE_SECRET_CONFIGURATION__SECURITY__MFA_STEP_UP_MAX_AGE", "") diff --git a/module/services/accounts/service.codefly.yaml b/module/services/accounts/service.codefly.yaml index 82bae09a1..85d63f3ae 100644 --- a/module/services/accounts/service.codefly.yaml +++ b/module/services/accounts/service.codefly.yaml @@ -18,6 +18,9 @@ service-dependencies: - name: store endpoints: - name: tcp + - name: telemetry + endpoints: + - name: grpc - name: vault endpoints: - name: http diff --git a/module/services/auth-gateway/code/configuration.go b/module/services/auth-gateway/code/configuration.go index b557942fe..45b601c36 100644 --- a/module/services/auth-gateway/code/configuration.go +++ b/module/services/auth-gateway/code/configuration.go @@ -2,10 +2,15 @@ package main import ( "os" + "strings" codefly "github.com/codefly-dev/sdk-go" ) +func observabilityEnabled() bool { + return strings.TrimSpace(workspaceEnv("observability", "OTEL_EXPORTER_OTLP_ENDPOINT")) != "" +} + // workspaceEnv resolves Codefly workspace configuration and secret values // before falling back to a raw environment variable. Services must use this // boundary for declared workspace-configuration-dependencies; Codefly keeps diff --git a/module/services/auth-gateway/code/configuration_test.go b/module/services/auth-gateway/code/configuration_test.go index 513f79245..8c66bd328 100644 --- a/module/services/auth-gateway/code/configuration_test.go +++ b/module/services/auth-gateway/code/configuration_test.go @@ -27,3 +27,22 @@ func TestWorkspaceEnvResolvesConfigurationSecretAndFallback(t *testing.T) { t.Setenv(configurationKey, "configuration") require.Equal(t, "configuration", workspaceEnv("internal-auth", key)) } + +func TestObservabilityEnabledRequiresExternalOTLPEndpoint(t *testing.T) { + const key = "OTEL_EXPORTER_OTLP_ENDPOINT" + configurationKey := "CODEFLY__WORKSPACE_CONFIGURATION__OBSERVABILITY__" + key + secretKey := "CODEFLY__WORKSPACE_SECRET_CONFIGURATION__OBSERVABILITY__" + key + t.Setenv(key, "") + t.Setenv(configurationKey, "") + t.Setenv(secretKey, "") + require.False(t, observabilityEnabled()) + + t.Setenv(key, " ") + require.False(t, observabilityEnabled()) + t.Setenv(key, "https://otel.example") + require.True(t, observabilityEnabled()) + + t.Setenv(key, "") + t.Setenv(configurationKey, "https://workspace-otel.example") + require.True(t, observabilityEnabled()) +} diff --git a/module/services/auth-gateway/code/go.mod b/module/services/auth-gateway/code/go.mod index 95c42aa62..15fc066ca 100644 --- a/module/services/auth-gateway/code/go.mod +++ b/module/services/auth-gateway/code/go.mod @@ -11,12 +11,14 @@ require ( github.com/golang-jwt/jwt/v5 v5.3.1 github.com/google/uuid v1.6.0 github.com/grpc-ecosystem/grpc-gateway/v2 v2.30.0 + github.com/prometheus/client_golang v1.24.1 github.com/redis/go-redis/v9 v9.22.0 github.com/stretchr/testify v1.12.1 go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.71.0 go.opentelemetry.io/contrib/instrumentation/runtime v0.71.0 go.opentelemetry.io/otel v1.46.0 go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc v1.46.0 + go.opentelemetry.io/otel/exporters/prometheus v0.68.0 go.opentelemetry.io/otel/metric v1.46.0 go.opentelemetry.io/otel/sdk v1.46.0 go.opentelemetry.io/otel/sdk/metric v1.46.0 @@ -36,6 +38,7 @@ require ( github.com/Masterminds/semver/v3 v3.5.0 // indirect github.com/VividCortex/ewma v1.2.0 // indirect github.com/antlr4-go/antlr/v4 v4.13.1 // indirect + github.com/beorn7/perks v1.0.1 // indirect github.com/blang/semver v3.5.1+incompatible // indirect github.com/cenkalti/backoff/v5 v5.0.3 // indirect github.com/cespare/xxhash/v2 v2.3.0 // indirect @@ -75,10 +78,15 @@ require ( github.com/mattn/go-isatty v0.0.24 // indirect github.com/mattn/go-runewidth v0.0.27 // indirect github.com/mitchellh/mapstructure v1.5.0 // indirect + github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect github.com/oklog/ulid/v2 v2.1.2 // indirect github.com/pkg/errors v0.9.1 // indirect github.com/planetscale/vtprotobuf v0.6.1-0.20240319094008-0393e58bdf10 // indirect github.com/power-devops/perfstat v0.0.0-20240221224432-82ca36839d55 // indirect + github.com/prometheus/client_model v0.6.3 // indirect + github.com/prometheus/common v0.71.0 // indirect + github.com/prometheus/otlptranslator v1.0.0 // indirect + github.com/prometheus/procfs v0.22.0 // indirect github.com/santhosh-tekuri/jsonschema/v6 v6.0.3 // indirect github.com/shirou/gopsutil/v3 v3.24.5 // indirect github.com/shoenig/go-m1cpu v0.2.1 // indirect @@ -93,7 +101,6 @@ require ( go.opentelemetry.io/otel/exporters/stdout/stdouttrace v1.46.0 // indirect go.opentelemetry.io/otel/trace v1.46.0 // indirect go.uber.org/atomic v1.11.0 // indirect - go.yaml.in/yaml/v2 v2.4.4 // indirect go.yaml.in/yaml/v3 v3.0.5 // indirect golang.org/x/exp v0.0.0-20260824195058-e88cd73687aa // indirect golang.org/x/net v0.59.0 // indirect diff --git a/module/services/auth-gateway/code/go.sum b/module/services/auth-gateway/code/go.sum index a7bf18ba7..ce04c63b7 100644 --- a/module/services/auth-gateway/code/go.sum +++ b/module/services/auth-gateway/code/go.sum @@ -20,6 +20,8 @@ github.com/VividCortex/ewma v1.2.0 h1:f58SaIzcDXrSy3kWaHNvuJgJ3Nmz59Zji6XoJR/q1o github.com/VividCortex/ewma v1.2.0/go.mod h1:nz4BbCtbLyFDeC9SUHbtcT5644juEuWfUAUnGx7j5l4= github.com/antlr4-go/antlr/v4 v4.13.1 h1:SqQKkuVZ+zWkMMNkjy5FZe5mr5WURWnlpmOuzYWrPrQ= github.com/antlr4-go/antlr/v4 v4.13.1/go.mod h1:GKmUxMtwp6ZgGwZSva4eWPC5mS6vUAmOABFgjdkM7Nw= +github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM= +github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw= github.com/blang/semver v3.5.1+incompatible h1:cQNTCjp13qL8KC3Nbxr/y2Bqb63oX6wdnnjpJbkM4JQ= github.com/blang/semver v3.5.1+incompatible/go.mod h1:kRBLl5iJ+tD4TcOOxsy/0fnwebNt5EWlYSAyrTnjyyk= github.com/blang/semver/v4 v4.0.0 h1:1PFHFE6yCCTv8C1TeyNNarDzntLi7wMI5i/pzqYIsAM= @@ -42,8 +44,6 @@ github.com/cloudflare/circl v1.6.3 h1:9GPOhQGF9MCYUeXyMYlqTR6a5gTrgR/fBLXvUgtVcg github.com/cloudflare/circl v1.6.3/go.mod h1:2eXP6Qfat4O/Yhh8BznvKnJ+uzEoTQ6jVKJRn81BiS4= github.com/cncf/xds/go v0.0.0-20260202195803-dba9d589def2 h1:aBangftG7EVZoUb69Os8IaYg++6uMOdKK83QtkkvJik= github.com/cncf/xds/go v0.0.0-20260202195803-dba9d589def2/go.mod h1:qwXFYgsP6T7XnJtbKlf1HP8AjxZZyzxMmc+Lq5GjlU4= -github.com/codefly-dev/core v0.14.0 h1:rpLFxeHSf2PrbxsRSp/NcZt8htMRM38JV2Oc8RKievU= -github.com/codefly-dev/core v0.14.0/go.mod h1:rp4YZkCJEOufliWQAgK3sv8YlO7Xcva7+cN7hIlAR0I= github.com/codefly-dev/core v0.15.1 h1:emWxsFL4yCqw+OIMIV73D4HBAby3pLh76hO0Xx+SGvw= github.com/codefly-dev/core v0.15.1/go.mod h1:rp4YZkCJEOufliWQAgK3sv8YlO7Xcva7+cN7hIlAR0I= github.com/codefly-dev/gortk v0.2.0 h1:7bOlS5valYz2zil+fZctQNcPCYBcPj86abcw9N8h1hQ= @@ -175,12 +175,16 @@ github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99 h1:BQSFePA1RWJOl github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99/go.mod h1:1lJo3i6rXxKeerYnT8Nvf0QmHCRC1n8sfWVwXF2Frvo= github.com/kevinburke/ssh_config v1.6.0 h1:J1FBfmuVosPHf5GRdltRLhPJtJpTlMdKTBjRgTaQBFY= github.com/kevinburke/ssh_config v1.6.0/go.mod h1:q2RIzfka+BXARoNexmF9gkxEX7DmvbW9P4hIVx2Kg4M= +github.com/klauspost/compress v1.19.1 h1:VsB4HPswih7mmZ8WleSFQ75c/Ui1M4trX5oAsJnhSlk= +github.com/klauspost/compress v1.19.1/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ= github.com/klauspost/cpuid/v2 v2.3.0 h1:S4CRMLnYUhGeDFDqkGriYKdfoFlDnMtqTiI/sFzhA9Y= github.com/klauspost/cpuid/v2 v2.3.0/go.mod h1:hqwkgyIinND0mEev00jJYCxPNVRVXFQeu1XKlok6oO0= github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE= github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk= github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY= github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE= +github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc= +github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw= github.com/lufia/plan9stats v0.0.0-20260330125221-c963978e514e h1:Q6MvJtQK/iRcRtzAscm/zF23XxJlbECiGPyRicsX+Ak= github.com/lufia/plan9stats v0.0.0-20260330125221-c963978e514e/go.mod h1:autxFIvghDt3jPTLoqZ9OZ7s9qTGNAWmYCjVFWPX/zg= github.com/mattn/go-colorable v0.1.15 h1:+u9SLTRGnXv73cEsnsmoZBom+dMU88B2M0aDcWy0/jY= @@ -197,6 +201,8 @@ github.com/moby/patternmatcher v0.6.1 h1:qlhtafmr6kgMIJjKJMDmMWq7WLkKIo23hsrpR3x github.com/moby/patternmatcher v0.6.1/go.mod h1:hDPoyOpDY7OrrMDLaYoY3hf52gNCR/YOUYxkhApJIxc= github.com/monochromegane/go-gitignore v0.0.0-20200626010858-205db1a8cc00 h1:n6/2gBQ3RWajuToeY6ZtZTIKv2v7ThUy5KKusIT0yc0= github.com/monochromegane/go-gitignore v0.0.0-20200626010858-205db1a8cc00/go.mod h1:Pm3mSP3c5uWn86xMLZ5Sa7JB9GsEZySvHYXCTK4E9q4= +github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA= +github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ= github.com/oklog/ulid/v2 v2.1.2 h1:IEclFb9JNvzYA6MW2SCxbLzcHTVsfqm3PrqGQJH5zec= github.com/oklog/ulid/v2 v2.1.2/go.mod h1:rcEKHmBBKfef9DhnvX7y1HZBYxjXb0cP5ExxNsTT1QQ= github.com/opencontainers/go-digest v1.0.0 h1:apOUWs51W5PlhuyGyz9FCeeBIOUDA/6nW8Oi/yOhh5U= @@ -212,6 +218,16 @@ github.com/planetscale/vtprotobuf v0.6.1-0.20240319094008-0393e58bdf10 h1:GFCKgm github.com/planetscale/vtprotobuf v0.6.1-0.20240319094008-0393e58bdf10/go.mod h1:t/avpk3KcrXxUnYOhZhMXJlSEyie6gQbtLq5NM3loB8= github.com/power-devops/perfstat v0.0.0-20240221224432-82ca36839d55 h1:o4JXh1EVt9k/+g42oCprj/FisM4qX9L3sZB3upGN2ZU= github.com/power-devops/perfstat v0.0.0-20240221224432-82ca36839d55/go.mod h1:OmDBASR4679mdNQnz2pUhc2G8CO2JrUAVFDRBDP/hJE= +github.com/prometheus/client_golang v1.24.1 h1:JnJkREXzWxUdCuPFpIWZiPispT9xVV59uiuyR2bPlnU= +github.com/prometheus/client_golang v1.24.1/go.mod h1:F+oSRECHg4sse5ucfYpYDeIv/hu68Zo0uoHKetWnzcE= +github.com/prometheus/client_model v0.6.3 h1:O0jaTVAYNxTHYInEPFJt5I3+sN8zqBtVMPTB1qyxiEo= +github.com/prometheus/client_model v0.6.3/go.mod h1:gpN5P9S7Rr6Yr92PiQ+Ixvhf6JZEkF1dnxsYL2aPBEM= +github.com/prometheus/common v0.71.0 h1:9KDAKb7Mj3HEVKyFCK6Dc/HIwlBzZIN2l7/lrHl3KK8= +github.com/prometheus/common v0.71.0/go.mod h1:CLJ5H8TEsGX8bl31BdMkfhIZ+QmZ9tBPPotUxUbfcmk= +github.com/prometheus/otlptranslator v1.0.0 h1:s0LJW/iN9dkIH+EnhiD3BlkkP5QVIUVEoIwkU+A6qos= +github.com/prometheus/otlptranslator v1.0.0/go.mod h1:vRYWnXvI6aWGpsdY/mOT/cbeVRBlPWtBNDb7kGR3uKM= +github.com/prometheus/procfs v0.22.0 h1:6q9+/JL9IKAPbCmBrv9n5O5Ty3NKnciV5X7YGw0oics= +github.com/prometheus/procfs v0.22.0/go.mod h1:CvmFr/GVhIjIvWJZW3tgkODBQMRIf0EyWMQLHCHab58= github.com/redis/go-redis/v9 v9.22.0 h1:laDvpYXTJtZLloinw1fA5Kqd6HAEH2XKxOkG/PDq2F0= github.com/redis/go-redis/v9 v9.22.0/go.mod h1:y2g0Wj8rQvuK0ELM+oxSudcLtC09JScs98I/X9gRWY4= github.com/rodaine/protogofakeit v0.1.1 h1:ZKouljuRM3A+TArppfBqnH8tGZHOwM/pjvtXe9DaXH8= @@ -262,6 +278,8 @@ go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.46.0 h1:OFnwLJr+pF3iHrlGSzb go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.46.0/go.mod h1:716wFneO0ov19A2beH5hjfh9AK5z/VWNAtDijp1Y0/g= go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.46.0 h1:w53CDeOA/Kurp7yRsegSr6pbbr759dOvJ+yNmWM6Hxs= go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.46.0/go.mod h1:BOmGMCbAtvcJiSJ+hLuhgPLdDbimnraSl8irz3iY8sY= +go.opentelemetry.io/otel/exporters/prometheus v0.68.0 h1:QOf2IftqQwITVRJpnn0M7M9ZCbgWfxz4P7i9C9yc2N4= +go.opentelemetry.io/otel/exporters/prometheus v0.68.0/go.mod h1:bgSvqu2TWGXiz7yr5UTMfObH8oqxJWHTnubQ3ef9BO4= go.opentelemetry.io/otel/exporters/stdout/stdouttrace v1.46.0 h1:KdRxPiAoMptR3vfWzvjjvutTsSiwbC2uG0496rzZNfo= go.opentelemetry.io/otel/exporters/stdout/stdouttrace v1.46.0/go.mod h1:K/qSA+3G7Eovxi4K09wzrAgkWRnosS0DAOZeEpve7sM= go.opentelemetry.io/otel/metric v1.46.0 h1:yBnkXvgV7AXFILZc5K6IZe/CBFF3OS7BJ8ov6/lj0K8= diff --git a/module/services/auth-gateway/code/main.go b/module/services/auth-gateway/code/main.go index 284d99ae7..d8a05c252 100644 --- a/module/services/auth-gateway/code/main.go +++ b/module/services/auth-gateway/code/main.go @@ -33,15 +33,6 @@ func main() { panic(fmt.Sprintf("codefly init failed: %v", err)) } - // The cell's collector is delivered in the `observability` configuration - // group, or the group says the cell has none. A group that did not arrive is - // neither, so it is refused outside a local runtime before anything else starts. - telemetryDestination, err := configuredTelemetryDestination(codefly.IsLocal()) - if err != nil { - panic(err) - } - - // The SDK resolves an endpoint from the carrier the runtime injected or, in a // local run, from the workspace's endpoint map, and never invents a port. grpcNet, grpcNetErr := codefly.For(ctx).API(standards.GRPC).ResolveNetworkInstance() if grpcNetErr != nil || grpcNet == nil { @@ -54,27 +45,32 @@ func main() { panic(fmt.Sprintf("Codefly REST endpoint is unavailable: %v", httpNetErr)) } httpPort = httpNet.Port - // A local run with no cell collector gets wool's own stdout tracer instead. - otelProvider, err := enableTracing(ctx, telemetryDestination, codefly.IsLocal()) - if err != nil { - panic(fmt.Sprintf("configure OTEL tracing: %v", err)) - } + var otelProvider *wooltel.Provider var otelMetricProvider *otelMetrics - if telemetryDestination.Available() { - otelMetricProvider, err = enableOTELMetrics(ctx, telemetryDestination) + if observabilityEnabled() { + collectorNetwork, err := codefly.For(ctx). + Service("telemetry"). + Endpoint("grpc"). + API("grpc"). + ResolveNetworkInstance() + if err != nil { + panic(fmt.Sprintf("resolve telemetry collector through Codefly: %v", err)) + } + otelProvider, err = wooltel.Enable( + wooltel.WithServiceName("saas-starter-auth-gateway"), + wooltel.WithEndpoint(collectorNetwork.Host), + wooltel.WithInsecure(), + ) + if err != nil { + panic(fmt.Sprintf("configure OTEL tracing: %v", err)) + } + otelMetricProvider, err = enableOTELMetrics(ctx, "saas-starter-auth-gateway", collectorNetwork.Host) if err != nil { shutdownCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second) _ = otelProvider.Shutdown(shutdownCtx) cancel() panic(fmt.Sprintf("configure OTEL metrics: %v", err)) } - log.Printf("OTEL enabled: endpoint=%s insecure=%t", telemetryDestination.Endpoint, telemetryDestination.Insecure) - } else { - log.Printf("OTEL export disabled: the cell has no collector: %s", telemetryDestination.AbsentReason) - } - // The state decided; a key it does not use is a leftover, reported here once. - if notice := telemetryDestination.IgnoredNotice(); notice != "" { - log.Print(notice) } defer func() { shutdownCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second) diff --git a/module/services/auth-gateway/code/telemetry_destination.go b/module/services/auth-gateway/code/telemetry_destination.go deleted file mode 100644 index 650372193..000000000 --- a/module/services/auth-gateway/code/telemetry_destination.go +++ /dev/null @@ -1,326 +0,0 @@ -package main - -import ( - "context" - "errors" - "fmt" - "net" - "net/url" - "strings" - - "github.com/codefly-dev/core/wool" - wooltel "github.com/codefly-dev/core/wool/otel" - codefly "github.com/codefly-dev/sdk-go" - "go.opentelemetry.io/otel/attribute" - "go.opentelemetry.io/otel/sdk/resource" -) - -// Where this process sends traces and metrics — or that it sends none. -// -// The platform tells a workload through the `observability` configuration -// group, and the group always says one of two things: -// -// TELEMETRY_STATE=available with OTEL_EXPORTER_OTLP_ENDPOINT, the cell's -// collector (OTLP/gRPC), e.g. -// http://otel-collector.otel-collector.svc.cluster.local:4317 -// TELEMETRY_STATE=absent with TELEMETRY_ABSENT_REASON, why the cell has no -// collector -// -// `http://` is plaintext on the wire because the service mesh supplies mTLS -// between workloads; `https://` is TLS, which the metrics exporter dials but -// wool's tracer cannot yet (see errWoolCannotDialTLS). This module owns no -// collector: the cell does, and a workload that cannot find it does not -// substitute one. -// -// The state decides. The group is layered, so a key the state does not use can -// still arrive from a lower layer: it is ignored, not refused, and reported once -// through IgnoredNotice. What each state requires is unchanged: `available` needs -// an endpoint, `absent` needs a reason. The module's own profile ships neither -// the state nor the reason, so a cell that delivers `available` carries no -// leftover and a cell that delivers `absent` without a reason is refused. -// -// A group that did not arrive is not a cell without a backend. Reading the first -// as the second would leave a deployment that converges green and exports -// nothing, so a missing or unknown state, or the key the state requires being -// absent, refuses to start, naming what is missing. Only a local runtime may -// infer `absent` when the state is missing, and it supplies the reason itself -// (localAbsentReason): module defaults declare neither. -// -// This file exists byte for byte in services/accounts/code and -// services/auth-gateway/code: the two services are independent Go modules with -// no shared package, and module/tools/telemetry_destination_mirror_test.go holds -// the copies identical. Change one, copy it over the other. - -const ( - telemetryStateKey = "TELEMETRY_STATE" - telemetryEndpointKey = "OTEL_EXPORTER_OTLP_ENDPOINT" - telemetryAbsentReasonKey = "TELEMETRY_ABSENT_REASON" - - telemetryStateAvailable = "available" - telemetryStateAbsent = "absent" - - // maxAbsentReasonLength bounds the operator-written reason before it reaches - // a log line: an unbounded value from configuration would otherwise let - // whoever writes the group forge log entries around it. - maxAbsentReasonLength = 300 - - // localAbsentReason is the reason a local run gives for having no collector. - // It lives in code, and is used only when the process is local, because the - // module's configuration profile is inherited by deployed cells: a reason - // shipped there would reach a cell that sent `absent` without saying why. - localAbsentReason = "A local run has no cell collector; traces go to the stdout tracer and metrics are not exported." -) - -// telemetryDestination is the resolved answer. Exactly one of URL and -// AbsentReason is set. -type telemetryDestination struct { - // URL is the collector's origin as scheme://host:port, the port always - // explicit, which is the form otlpmetricgrpc.WithEndpointURL takes: the - // exporter reads its transport off the scheme. Empty when the cell has no - // collector. - URL string - // Endpoint is the same address as host:port. wool's tracer takes it in this - // form and has no option that reads a URL. Empty when the cell has no - // collector. - Endpoint string - // Insecure is true when the endpoint was http://, which the mesh protects. - // wool's tracer needs it spelled out; it is derived from the endpoint's - // scheme and never assumed. - Insecure bool - // AbsentReason says why the cell has no collector. Empty when URL is set. - AbsentReason string - // Ignored names the key the group carried that has no meaning for the state - // that decided: the absent reason under `available`, the endpoint under - // `absent`. Empty when the group carried nothing extra. - Ignored string -} - -// Available reports whether the cell has a collector to export to. -func (d telemetryDestination) Available() bool { - return d.Endpoint != "" -} - -// IgnoredNotice is the one line to log, once, when the group carried a key the -// state does not use. It is empty otherwise. It names the key and never its -// value: a leftover endpoint may carry credentials, and a leftover reason is -// free text. -func (d telemetryDestination) IgnoredNotice() string { - if d.Ignored == "" { - return "" - } - state := telemetryStateAbsent - if d.Available() { - state = telemetryStateAvailable - } - return fmt.Sprintf( - "observability: %s is set but %s is `%s`, which decides; %s is ignored (a leftover from a lower layer of the group)", - d.Ignored, telemetryStateKey, state, d.Ignored) -} - -// observabilityValue reads one key of the `observability` group through the -// SDK. It deliberately has no fallback to a bare process variable: an ambient -// OTEL_EXPORTER_OTLP_ENDPOINT in a shell or a pod is conventionally the -// OTLP/HTTP address, not the cell's gRPC collector, and reading it here would -// export to the wrong place without a word. -func observabilityValue(key string) string { - value, err := codefly.For(codefly.Context()).WorkspaceValue("observability", key) - if err != nil { - return "" - } - return value -} - -// configuredTelemetryDestination resolves the destination from the -// `observability` group. -func configuredTelemetryDestination(local bool) (telemetryDestination, error) { - return resolveTelemetryDestination(local, observabilityValue) -} - -// resolveTelemetryDestination is the whole decision, over any reader of the -// group's keys so a test can reach every outcome with nothing running. -func resolveTelemetryDestination(local bool, read func(key string) string) (telemetryDestination, error) { - state := strings.TrimSpace(read(telemetryStateKey)) - endpoint := strings.TrimSpace(read(telemetryEndpointKey)) - reason := boundedLine(read(telemetryAbsentReasonKey), maxAbsentReasonLength) - - if state == "" && local { - state = telemetryStateAbsent - if reason == "" { - reason = localAbsentReason - } - } - if state == "" { - return telemetryDestination{}, errors.New( - "observability: " + telemetryStateKey + " is not set in the `observability` configuration group. " + - "The platform delivers `" + telemetryStateAvailable + "` (with " + telemetryEndpointKey + ") or `" + - telemetryStateAbsent + "` (with " + telemetryAbsentReasonKey + "); a group that did not arrive is " + - "not a cell without a collector, so this process does not start without it") - } - if state != telemetryStateAvailable && state != telemetryStateAbsent { - return telemetryDestination{}, fmt.Errorf( - "observability: %s is %q in the `observability` configuration group; it must be `%s` or `%s`", - telemetryStateKey, state, telemetryStateAvailable, telemetryStateAbsent) - } - - // The state decides: the key that belongs to the other state is a leftover - // of a lower layer and is neither read nor refused. The endpoint under - // `absent` is not even parsed, so a stale one cannot stop a cell that says - // it has no collector. - if state == telemetryStateAbsent { - if reason == "" { - return telemetryDestination{}, fmt.Errorf( - "observability: %s is `%s` but %s is not set in the `observability` configuration group; "+ - "an absent collector must say why", - telemetryStateKey, telemetryStateAbsent, telemetryAbsentReasonKey) - } - destination := telemetryDestination{AbsentReason: reason} - if endpoint != "" { - destination.Ignored = telemetryEndpointKey - } - return destination, nil - } - - if endpoint == "" { - return telemetryDestination{}, fmt.Errorf( - "observability: %s is `%s` but %s is not set in the `observability` configuration group", - telemetryStateKey, telemetryStateAvailable, telemetryEndpointKey) - } - origin, hostPort, insecure, err := parseCollectorEndpoint(endpoint) - if err != nil { - return telemetryDestination{}, fmt.Errorf("observability: %s: %w", telemetryEndpointKey, err) - } - destination := telemetryDestination{URL: origin, Endpoint: hostPort, Insecure: insecure} - if reason != "" { - destination.Ignored = telemetryAbsentReasonKey - } - return destination, nil -} - -// parseCollectorEndpoint turns the delivered URL into the two forms the -// exporters take — the origin with an explicit port, and host:port — and reads -// the transport off its scheme. The URL is never echoed back: a malformed one -// may carry credentials. -func parseCollectorEndpoint(raw string) (origin, hostPort string, insecure bool, err error) { - parsed, parseErr := url.Parse(raw) - if parseErr != nil { - return "", "", false, errors.New("is not a URL; expected http://host:port or https://host:port") - } - switch parsed.Scheme { - case "http": - insecure = true - case "https": - default: - return "", "", false, fmt.Errorf("must start with http:// or https://, got scheme %q", parsed.Scheme) - } - if parsed.Hostname() == "" { - return "", "", false, errors.New("has no host") - } - if parsed.User != nil { - return "", "", false, errors.New("must not carry credentials") - } - // OTLP/gRPC addresses a collector, not a path: a path or a query is the - // OTLP/HTTP form (/v1/traces), which this exporter does not speak. - if (parsed.Path != "" && parsed.Path != "/") || parsed.RawQuery != "" || parsed.Fragment != "" { - return "", "", false, errors.New("must be the collector's origin only, with no path, query or fragment") - } - // The port is made explicit because gRPC would otherwise dial 443 whatever - // the scheme, and a plaintext http:// collector with no port is not on 443. - port := parsed.Port() - if port == "" { - port = "80" - if !insecure { - port = "443" - } - } - hostPort = net.JoinHostPort(parsed.Hostname(), port) - return parsed.Scheme + "://" + hostPort, hostPort, insecure, nil -} - -// boundedLine collapses a free-text value to one bounded line. -func boundedLine(value string, limit int) string { - line := strings.Join(strings.Fields(value), " ") - if runes := []rune(line); len(runes) > limit { - return string(runes[:limit]) + "…" - } - return line -} - -// telemetryResource is the one resource both signals carry. service.name is -// whatever the environment says — OTEL_SERVICE_NAME, or service.name in -// OTEL_RESOURCE_ATTRIBUTES — and otherwise the service identity the Codefly SDK -// holds. No name is typed here: a literal would override the environment, so the -// platform's identity could never reach the backend. -func telemetryResource(ctx context.Context, identity string) (*resource.Resource, error) { - options := []resource.Option{resource.WithService()} - if identity != "" { - options = append(options, resource.WithAttributes(attribute.String("service.name", identity))) - } - // Last, so what the environment declares wins over the identity above. - options = append(options, resource.WithFromEnv()) - return resource.New(ctx, options...) -} - -// currentTelemetryResource is telemetryResource for this process. -func currentTelemetryResource(ctx context.Context) (*resource.Resource, error) { - return telemetryResource(ctx, codeflyServiceName(codefly.Context())) -} - -// codeflyServiceName is the service identity the Codefly SDK holds: the -// `module/service` identity of the provider it built at Init, which wool stamps on -// every log line from this process. It is empty when the SDK has not been -// initialised. The SDK exposes no accessor of its own, and the environment -// carrier it reads this from is not product code's to read, so this asks the -// provider. -func codeflyServiceName(ctx context.Context) string { - provider, ok := ctx.Value(wool.ProviderKey).(*wool.Provider) - if !ok || provider == nil { - return "" - } - source := provider.Get(ctx).Source() - if source == nil { - return "" - } - // An identity with no module reads `/service`; with neither, `/`. - return strings.Trim(strings.TrimSpace(source.Unique), "/") -} - -// telemetryServiceName is the service.name the resource resolved to. -func telemetryServiceName(res *resource.Resource) string { - if value, ok := res.Set().Value(attribute.Key("service.name")); ok { - return value.AsString() - } - return "" -} - -// errWoolCannotDialTLS names the one transport the tracer cannot serve. -var errWoolCannotDialTLS = errors.New( - "the collector endpoint is https://, but wool's OTLP tracer cannot dial TLS: wool/otel.Enable " + - "always sets an insecure transport and offers no option to turn it off. Deliver an http:// " + - "endpoint (in-mesh, where the mesh supplies mTLS) until wool can dial TLS") - -// enableTracing registers wool's tracer for the destination: OTLP/gRPC to the -// cell's collector when there is one; wool's own stdout tracer when there is -// none and this is a local run; nothing otherwise. A nil provider means nothing -// was registered. -func enableTracing(ctx context.Context, destination telemetryDestination, local bool) (*wooltel.Provider, error) { - res, err := currentTelemetryResource(ctx) - if err != nil { - return nil, fmt.Errorf("resolve telemetry resource: %w", err) - } - options := []wooltel.Option{wooltel.WithServiceName(telemetryServiceName(res))} - switch { - case destination.Available(): - // wool dials plaintext unless told otherwise and cannot be told - // otherwise, so an https:// endpoint is refused rather than dialed in - // the clear and reported as TLS. - if !destination.Insecure { - return nil, errWoolCannotDialTLS - } - options = append(options, wooltel.WithEndpoint(destination.Endpoint), wooltel.WithInsecure()) - case local: - options = append(options, wooltel.WithStdout()) - default: - return nil, nil - } - return wooltel.Enable(options...) -} diff --git a/module/services/auth-gateway/code/telemetry_destination_test.go b/module/services/auth-gateway/code/telemetry_destination_test.go deleted file mode 100644 index 63cb2077e..000000000 --- a/module/services/auth-gateway/code/telemetry_destination_test.go +++ /dev/null @@ -1,603 +0,0 @@ -package main - -import ( - "context" - "net" - "os" - "strings" - "testing" - "time" - - "github.com/codefly-dev/core/wool" - "github.com/stretchr/testify/require" - "go.opentelemetry.io/otel" - collectortracev1 "go.opentelemetry.io/proto/otlp/collector/trace/v1" - "google.golang.org/grpc" -) - -// group is a reader over a fixed set of `observability` keys. -func group(values map[string]string) func(string) string { - return func(key string) string { return values[key] } -} - -const collector = "http://otel-collector.otel-collector.svc.cluster.local:4317" - -// shippedObservabilityProfile is what the module's own `observability` profile -// contributes to the group, read from the file it ships. -func shippedObservabilityProfile(t *testing.T) map[string]string { - t.Helper() - contents, err := os.ReadFile("../../../configurations/local/observability.env") - require.NoError(t, err) - values := map[string]string{} - for _, line := range strings.Split(string(contents), "\n") { - line = strings.TrimSpace(line) - if line == "" || strings.HasPrefix(line, "#") { - continue - } - key, value, ok := strings.Cut(line, "=") - require.True(t, ok) - values[key] = value - } - return values -} - -// Module defaults are also inherited by cells, so the profile ships neither the -// state nor the reason: a deployed workload must get both from the platform, and -// a local run gets them from code (localAbsentReason), only when it is local. -func TestTelemetryDestinationShippedProfileDeclaresNothingACellCouldInherit(t *testing.T) { - shipped := shippedObservabilityProfile(t) - require.NotContains(t, shipped, "TELEMETRY_STATE", "every deployed cell inherits the profile") - require.NotContains(t, shipped, "TELEMETRY_ABSENT_REASON", "every deployed cell inherits the profile") - require.Empty(t, shipped["OTEL_EXPORTER_OTLP_ENDPOINT"], "the profile keeps the group in existence and names no collector") - - _, err := resolveTelemetryDestination(false, group(shipped)) - require.ErrorContains(t, err, "TELEMETRY_STATE is not set") - destination, err := resolveTelemetryDestination(true, group(shipped)) - require.NoError(t, err) - require.False(t, destination.Available()) - require.Equal(t, localAbsentReason, destination.AbsentReason, "a local run supplies its own reason in code") - require.Empty(t, destination.IgnoredNotice()) -} - -// Layered over the shipped profile, a healthy cell leaves nothing to warn about, -// and a cell that says `absent` without a reason is refused rather than passed -// with the local-run reason. -func TestTelemetryDestinationCellAnswersAreNotMaskedByTheShippedProfile(t *testing.T) { - shipped := shippedObservabilityProfile(t) - - available, err := resolveTelemetryDestination(false, layered(shipped, map[string]string{ - "TELEMETRY_STATE": "available", - "OTEL_EXPORTER_OTLP_ENDPOINT": collector, - })) - require.NoError(t, err) - require.True(t, available.Available()) - require.Empty(t, available.Ignored, "an available cell must not log a leftover-key warning on every boot") - require.Empty(t, available.IgnoredNotice()) - - _, err = resolveTelemetryDestination(false, layered(shipped, map[string]string{"TELEMETRY_STATE": "absent"})) - require.ErrorContains(t, err, "TELEMETRY_ABSENT_REASON is not set") - - // The same answer from a local run is refused too: the local reason fills in - // a missing state, never an explicit `absent` that did not say why. - _, err = resolveTelemetryDestination(true, layered(shipped, map[string]string{"TELEMETRY_STATE": "absent"})) - require.ErrorContains(t, err, "TELEMETRY_ABSENT_REASON is not set") -} - -func TestTelemetryDestinationMissingStateDefaultsOnlyInLocalRuntime(t *testing.T) { - for _, values := range []map[string]string{ - {}, - {"TELEMETRY_STATE": " "}, - {"TELEMETRY_ABSENT_REASON": "local defaults inherited by a cell"}, - {"OTEL_EXPORTER_OTLP_ENDPOINT": collector}, - } { - _, err := resolveTelemetryDestination(false, group(values)) - require.ErrorContains(t, err, "TELEMETRY_STATE is not set") - destination, err := resolveTelemetryDestination(true, group(values)) - require.NoError(t, err) - require.False(t, destination.Available()) - require.NotEmpty(t, destination.AbsentReason) - } - for _, values := range []map[string]string{ - {"TELEMETRY_STATE": "unknown"}, - {"TELEMETRY_STATE": "available"}, - {"TELEMETRY_STATE": "absent"}, - } { - _, err := resolveTelemetryDestination(true, group(values)) - require.Error(t, err, "an explicit state must be valid and complete even locally") - } -} - -// The first outcome: the platform delivered a collector. -func TestTelemetryDestinationAvailableExportsToTheEndpoint(t *testing.T) { - destination, err := resolveTelemetryDestination(false, group(map[string]string{ - "TELEMETRY_STATE": "available", - "OTEL_EXPORTER_OTLP_ENDPOINT": collector, - })) - require.NoError(t, err) - require.True(t, destination.Available()) - require.Equal(t, collector, destination.URL, "the metrics exporter reads its transport off this URL") - require.Equal(t, "otel-collector.otel-collector.svc.cluster.local:4317", destination.Endpoint) - require.True(t, destination.Insecure, "http:// is plaintext on the wire; the mesh supplies mTLS") - require.Empty(t, destination.AbsentReason) - require.Empty(t, destination.Ignored) - require.Empty(t, destination.IgnoredNotice()) -} - -// The transport comes from the endpoint's scheme, never from a constant, and -// the URL handed to the metrics exporter always carries an explicit port: gRPC -// would otherwise dial 443 whatever the scheme. -func TestTelemetryDestinationReadsTransportFromTheScheme(t *testing.T) { - for endpoint, want := range map[string]struct { - url string - hostPort string - insecure bool - }{ - "http://collector.example:4317": {"http://collector.example:4317", "collector.example:4317", true}, - "https://collector.example:4317": {"https://collector.example:4317", "collector.example:4317", false}, - "https://collector.example": {"https://collector.example:443", "collector.example:443", false}, - "http://collector.example": {"http://collector.example:80", "collector.example:80", true}, - "http://collector.example:4317/": {"http://collector.example:4317", "collector.example:4317", true}, - " http://collector.example:4317 ": {"http://collector.example:4317", "collector.example:4317", true}, - "http://[2001:db8::1]:4317": {"http://[2001:db8::1]:4317", "[2001:db8::1]:4317", true}, - "https://10.0.0.7:4317": {"https://10.0.0.7:4317", "10.0.0.7:4317", false}, - "http://otel-collector.ns.svc:4317/": {"http://otel-collector.ns.svc:4317", "otel-collector.ns.svc:4317", true}, - } { - destination, err := resolveTelemetryDestination(false, group(map[string]string{ - "TELEMETRY_STATE": "available", - "OTEL_EXPORTER_OTLP_ENDPOINT": endpoint, - })) - require.NoError(t, err, endpoint) - require.Equal(t, want.url, destination.URL, endpoint) - require.Equal(t, want.hostPort, destination.Endpoint, endpoint) - require.Equal(t, want.insecure, destination.Insecure, endpoint) - } -} - -// The second outcome: the cell has no collector, and says why. The process boots -// and exports nothing. -func TestTelemetryDestinationAbsentCarriesTheReason(t *testing.T) { - destination, err := resolveTelemetryDestination(false, group(map[string]string{ - "TELEMETRY_STATE": "absent", - "TELEMETRY_ABSENT_REASON": "This cell's bucket has no ops cell, so no collector is rendered.", - })) - require.NoError(t, err) - require.False(t, destination.Available()) - require.Empty(t, destination.URL) - require.Empty(t, destination.Endpoint) - require.Equal(t, "This cell's bucket has no ops cell, so no collector is rendered.", destination.AbsentReason) - require.Empty(t, destination.Ignored) - require.Empty(t, destination.IgnoredNotice()) -} - -// The reason is operator-written free text that reaches a log line. -func TestTelemetryDestinationAbsentReasonIsOneBoundedLine(t *testing.T) { - destination, err := resolveTelemetryDestination(false, group(map[string]string{ - "TELEMETRY_STATE": "absent", - "TELEMETRY_ABSENT_REASON": "no collector\n2026-01-01 FORGED entry\r\n" + strings.Repeat("x", 1000), - })) - require.NoError(t, err) - require.NotContains(t, destination.AbsentReason, "\n") - require.NotContains(t, destination.AbsentReason, "\r") - require.LessOrEqual(t, len([]rune(destination.AbsentReason)), maxAbsentReasonLength+1) -} - -// The third outcome: anything else refuses to start, naming what is wrong. -// Configuration that did not arrive is never read as "no collector". A key the -// state does not use is not a refusal — it is ignored, below — but a key the -// state requires, missing, always is. -func TestTelemetryDestinationRefusesEverythingElse(t *testing.T) { - for name, tc := range map[string]struct { - values map[string]string - want string - }{ - "state missing": { - values: map[string]string{}, - want: "TELEMETRY_STATE is not set", - }, - "state missing though an ambient endpoint is present": { - values: map[string]string{"OTEL_EXPORTER_OTLP_ENDPOINT": collector}, - want: "TELEMETRY_STATE is not set", - }, - "state missing though a reason is present": { - values: map[string]string{"TELEMETRY_ABSENT_REASON": "no collector"}, - want: "TELEMETRY_STATE is not set", - }, - "state blank": { - values: map[string]string{"TELEMETRY_STATE": " "}, - want: "TELEMETRY_STATE is not set", - }, - "state unknown": { - values: map[string]string{"TELEMETRY_STATE": "disabled", "TELEMETRY_ABSENT_REASON": "no collector"}, - want: `TELEMETRY_STATE is "disabled"`, - }, - "state is not matched case-insensitively": { - values: map[string]string{"TELEMETRY_STATE": "Available", "OTEL_EXPORTER_OTLP_ENDPOINT": collector}, - want: `TELEMETRY_STATE is "Available"`, - }, - "available without an endpoint": { - values: map[string]string{"TELEMETRY_STATE": "available"}, - want: "OTEL_EXPORTER_OTLP_ENDPOINT is not set", - }, - "available with only a reason": { - values: map[string]string{"TELEMETRY_STATE": "available", "TELEMETRY_ABSENT_REASON": "no collector"}, - want: "OTEL_EXPORTER_OTLP_ENDPOINT is not set", - }, - "absent without a reason": { - values: map[string]string{"TELEMETRY_STATE": "absent"}, - want: "TELEMETRY_ABSENT_REASON is not set", - }, - "absent with a blank reason": { - values: map[string]string{"TELEMETRY_STATE": "absent", "TELEMETRY_ABSENT_REASON": " \n "}, - want: "TELEMETRY_ABSENT_REASON is not set", - }, - "absent with only an endpoint": { - values: map[string]string{"TELEMETRY_STATE": "absent", "OTEL_EXPORTER_OTLP_ENDPOINT": collector}, - want: "TELEMETRY_ABSENT_REASON is not set", - }, - "endpoint without a scheme": { - values: map[string]string{"TELEMETRY_STATE": "available", "OTEL_EXPORTER_OTLP_ENDPOINT": "otel-collector.ns.svc:4317"}, - want: "must start with http:// or https://", - }, - "endpoint with another scheme": { - values: map[string]string{"TELEMETRY_STATE": "available", "OTEL_EXPORTER_OTLP_ENDPOINT": "grpc://otel-collector.ns.svc:4317"}, - want: `got scheme "grpc"`, - }, - "endpoint without a host": { - values: map[string]string{"TELEMETRY_STATE": "available", "OTEL_EXPORTER_OTLP_ENDPOINT": "http://:4317"}, - want: "has no host", - }, - "endpoint naming an OTLP/HTTP path": { - values: map[string]string{"TELEMETRY_STATE": "available", "OTEL_EXPORTER_OTLP_ENDPOINT": "http://collector.example:4318/v1/traces"}, - want: "origin only", - }, - "endpoint with a query": { - values: map[string]string{"TELEMETRY_STATE": "available", "OTEL_EXPORTER_OTLP_ENDPOINT": "http://collector.example:4317?x=1"}, - want: "origin only", - }, - } { - t.Run(name, func(t *testing.T) { - destination, err := resolveTelemetryDestination(false, group(tc.values)) - require.Error(t, err) - require.ErrorContains(t, err, tc.want) - require.False(t, destination.Available(), "a refusal must never read as a usable destination") - require.Empty(t, destination.AbsentReason, "a refusal must never read as an absent collector") - }) - } -} - -// The state decides. The group is layered, so a deployed producer can still see -// `available` and an endpoint from the cell beside a reason a lower layer (a -// consuming workspace's own profile, say) left behind. That is not a -// contradiction to refuse: the reason is ignored, and said so once. -func TestTelemetryDestinationAvailableIgnoresALeftoverReason(t *testing.T) { - destination, err := resolveTelemetryDestination(false, group(map[string]string{ - "TELEMETRY_STATE": "available", - "OTEL_EXPORTER_OTLP_ENDPOINT": collector, - "TELEMETRY_ABSENT_REASON": "A local run has no cell collector.", - })) - require.NoError(t, err) - require.True(t, destination.Available()) - require.Equal(t, collector, destination.URL) - require.Empty(t, destination.AbsentReason, "a leftover reason must not turn an available collector into an absent one") - require.Equal(t, "TELEMETRY_ABSENT_REASON", destination.Ignored) - notice := destination.IgnoredNotice() - require.Contains(t, notice, "TELEMETRY_ABSENT_REASON is set but TELEMETRY_STATE is `available`") - require.NotContains(t, notice, "A local run", "the notice names the key, never its free-text value") -} - -// ... and the other way round: a cell that says it has no collector is not -// refused for an endpoint a lower layer left behind, and that endpoint is not -// even parsed. -func TestTelemetryDestinationAbsentIgnoresALeftoverEndpoint(t *testing.T) { - for name, leftover := range map[string]string{ - "a usable endpoint": collector, - "a malformed endpoint": "otel-collector.ns.svc:4317", - "an endpoint with a user": "http://user:hunter2@collector.example:4317", - "an unreadable endpoint": "http://[::1", - } { - t.Run(name, func(t *testing.T) { - destination, err := resolveTelemetryDestination(false, group(map[string]string{ - "TELEMETRY_STATE": "absent", - "TELEMETRY_ABSENT_REASON": "This cell has no ops cell.", - "OTEL_EXPORTER_OTLP_ENDPOINT": leftover, - })) - require.NoError(t, err) - require.False(t, destination.Available(), "a leftover endpoint must not turn an absent collector into an available one") - require.Empty(t, destination.URL) - require.Empty(t, destination.Endpoint) - require.Equal(t, "This cell has no ops cell.", destination.AbsentReason) - require.Equal(t, "OTEL_EXPORTER_OTLP_ENDPOINT", destination.Ignored) - notice := destination.IgnoredNotice() - require.Contains(t, notice, "OTEL_EXPORTER_OTLP_ENDPOINT is set but TELEMETRY_STATE is `absent`") - require.NotContains(t, notice, "hunter2", "the notice names the key, never its value") - }) - } -} - -// A blank leftover is nothing left over. -func TestTelemetryDestinationBlankLeftoversAreNotNoticed(t *testing.T) { - available, err := resolveTelemetryDestination(false, group(map[string]string{ - "TELEMETRY_STATE": "available", "OTEL_EXPORTER_OTLP_ENDPOINT": collector, "TELEMETRY_ABSENT_REASON": " \n ", - })) - require.NoError(t, err) - require.Empty(t, available.IgnoredNotice()) - - absent, err := resolveTelemetryDestination(false, group(map[string]string{ - "TELEMETRY_STATE": "absent", "TELEMETRY_ABSENT_REASON": "no collector", "OTEL_EXPORTER_OTLP_ENDPOINT": " ", - })) - require.NoError(t, err) - require.Empty(t, absent.IgnoredNotice()) -} - -// layered is what the platform hands a workload: each layer overrides the one -// below it key by key, so a key a higher layer does not set survives from the -// lower one. -func layered(layers ...map[string]string) func(string) string { - merged := map[string]string{} - for _, layer := range layers { - for key, value := range layer { - merged[key] = value - } - } - return group(merged) -} - -// The case this exists for: a lower layer of the group says `absent` with a -// reason, and a deployed cell overrides the state and the endpoint but not the -// reason. The cell's answer wins, whichever way it points. -func TestTelemetryDestinationTheCellOverridesALowerLayer(t *testing.T) { - moduleDefault := map[string]string{ - "TELEMETRY_STATE": "absent", - "TELEMETRY_ABSENT_REASON": "A local run has no cell collector.", - } - - t.Run("a cell with a collector overrides a lower layer's absent default", func(t *testing.T) { - destination, err := resolveTelemetryDestination(false, layered(moduleDefault, map[string]string{ - "TELEMETRY_STATE": "available", - "OTEL_EXPORTER_OTLP_ENDPOINT": collector, - })) - require.NoError(t, err) - require.True(t, destination.Available()) - require.Equal(t, collector, destination.URL) - require.Empty(t, destination.AbsentReason) - require.Equal(t, "TELEMETRY_ABSENT_REASON", destination.Ignored) - }) - - t.Run("a cell with no collector, over a default that had an endpoint, is absent", func(t *testing.T) { - destination, err := resolveTelemetryDestination(false, layered( - map[string]string{"TELEMETRY_STATE": "available", "OTEL_EXPORTER_OTLP_ENDPOINT": collector}, - map[string]string{"TELEMETRY_STATE": "absent", "TELEMETRY_ABSENT_REASON": "This cell has no ops cell."}, - )) - require.NoError(t, err) - require.False(t, destination.Available()) - require.Equal(t, "This cell has no ops cell.", destination.AbsentReason) - require.Equal(t, "OTEL_EXPORTER_OTLP_ENDPOINT", destination.Ignored) - }) - - t.Run("a cell that overrides only the state still has to supply what it requires", func(t *testing.T) { - _, err := resolveTelemetryDestination(false, layered(moduleDefault, map[string]string{"TELEMETRY_STATE": "available"})) - require.ErrorContains(t, err, "OTEL_EXPORTER_OTLP_ENDPOINT is not set", - "the module default's reason is not an endpoint") - - _, err = resolveTelemetryDestination(false, layered( - map[string]string{"TELEMETRY_STATE": "available", "OTEL_EXPORTER_OTLP_ENDPOINT": collector}, - map[string]string{"TELEMETRY_STATE": "absent"}, - )) - require.ErrorContains(t, err, "TELEMETRY_ABSENT_REASON is not set", - "the module default's endpoint is not a reason") - }) -} - -// A refusal quotes no part of an endpoint, which may carry credentials. -func TestTelemetryDestinationRefusalDoesNotEchoCredentials(t *testing.T) { - _, err := resolveTelemetryDestination(false, group(map[string]string{ - "TELEMETRY_STATE": "available", - "OTEL_EXPORTER_OTLP_ENDPOINT": "http://user:hunter2@collector.example:4317", - })) - require.ErrorContains(t, err, "must not carry credentials") - require.NotContains(t, err.Error(), "hunter2") -} - -// The group is read through the SDK's workspace-configuration lookup, by the -// names the platform delivers them under. A bare process variable of the same -// name is not the cell's answer. -func TestConfiguredTelemetryDestinationReadsTheObservabilityGroup(t *testing.T) { - const prefix = "CODEFLY__WORKSPACE_CONFIGURATION__OBSERVABILITY__" - for _, key := range []string{"TELEMETRY_STATE", "OTEL_EXPORTER_OTLP_ENDPOINT", "TELEMETRY_ABSENT_REASON"} { - t.Setenv(key, "") - t.Setenv(prefix+key, "") - } - - localDestination, err := configuredTelemetryDestination(true) - require.NoError(t, err) - require.False(t, localDestination.Available()) - require.NotEmpty(t, localDestination.AbsentReason) - _, err = configuredTelemetryDestination(false) - require.ErrorContains(t, err, "TELEMETRY_STATE is not set") - - // Ambient variables a shell or a pod may carry are not the group. - t.Setenv("TELEMETRY_STATE", "available") - t.Setenv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://ambient.example:4318") - _, err = configuredTelemetryDestination(false) - require.ErrorContains(t, err, "TELEMETRY_STATE is not set") - t.Setenv("TELEMETRY_STATE", "") - t.Setenv("OTEL_EXPORTER_OTLP_ENDPOINT", "") - - t.Setenv(prefix+"TELEMETRY_STATE", "available") - t.Setenv(prefix+"OTEL_EXPORTER_OTLP_ENDPOINT", collector) - destination, err := configuredTelemetryDestination(false) - require.NoError(t, err) - require.Equal(t, "otel-collector.otel-collector.svc.cluster.local:4317", destination.Endpoint) - require.True(t, destination.Insecure) - - t.Setenv(prefix+"TELEMETRY_STATE", "absent") - t.Setenv(prefix+"OTEL_EXPORTER_OTLP_ENDPOINT", "") - t.Setenv(prefix+"TELEMETRY_ABSENT_REASON", "a local run has no cell collector") - destination, err = configuredTelemetryDestination(false) - require.NoError(t, err) - require.False(t, destination.Available()) - require.Equal(t, "a local run has no cell collector", destination.AbsentReason) -} - -// Through the SDK's lookup, with the keys exactly as a deployed cell delivers -// them after the layers merge: the cell's state and endpoint, and a reason a -// lower layer left behind. -func TestConfiguredTelemetryDestinationStateDecidesAcrossLayers(t *testing.T) { - const prefix = "CODEFLY__WORKSPACE_CONFIGURATION__OBSERVABILITY__" - t.Setenv(prefix+"TELEMETRY_STATE", "available") - t.Setenv(prefix+"OTEL_EXPORTER_OTLP_ENDPOINT", collector) - t.Setenv(prefix+"TELEMETRY_ABSENT_REASON", "A local run has no cell collector.") - - destination, err := configuredTelemetryDestination(false) - require.NoError(t, err) - require.True(t, destination.Available()) - require.Equal(t, collector, destination.URL) - require.NotEmpty(t, destination.IgnoredNotice()) -} - -// service.name is never a literal typed here: the environment wins, then the -// service's own Codefly identity. -func TestTelemetryResourceServiceName(t *testing.T) { - resolve := func(t *testing.T, identity string) string { - t.Helper() - res, err := telemetryResource(t.Context(), identity) - require.NoError(t, err) - return telemetryServiceName(res) - } - - t.Run("falls back to the Codefly service identity", func(t *testing.T) { - t.Setenv("OTEL_SERVICE_NAME", "") - t.Setenv("OTEL_RESOURCE_ATTRIBUTES", "") - require.Equal(t, "saas-starter/accounts", resolve(t, "saas-starter/accounts")) - }) - t.Run("OTEL_SERVICE_NAME wins over the identity", func(t *testing.T) { - t.Setenv("OTEL_SERVICE_NAME", "named-by-the-platform") - t.Setenv("OTEL_RESOURCE_ATTRIBUTES", "") - require.Equal(t, "named-by-the-platform", resolve(t, "saas-starter/accounts")) - }) - t.Run("OTEL_RESOURCE_ATTRIBUTES can set it", func(t *testing.T) { - t.Setenv("OTEL_SERVICE_NAME", "") - t.Setenv("OTEL_RESOURCE_ATTRIBUTES", "service.name=attributed-by-the-platform,deployment.environment=test") - require.Equal(t, "attributed-by-the-platform", resolve(t, "saas-starter/accounts")) - }) - t.Run("without either, the SDK default rather than a literal", func(t *testing.T) { - t.Setenv("OTEL_SERVICE_NAME", "") - t.Setenv("OTEL_RESOURCE_ATTRIBUTES", "") - require.True(t, strings.HasPrefix(resolve(t, ""), "unknown_service"), resolve(t, "")) - }) -} - -// The identity is the one the SDK built its provider with, the same one wool -// stamps on this process's log lines — read from the provider, not from the -// runtime's environment carrier. -func TestCodeflyServiceNameIsTheSDKProviderIdentity(t *testing.T) { - carrying := func(unique string) context.Context { - return wool.New(t.Context(), &wool.Resource{Kind: "service", Unique: unique}).Inject(t.Context()) - } - require.Equal(t, "saas-starter/accounts", codeflyServiceName(carrying("saas-starter/accounts"))) - require.Equal(t, "accounts", codeflyServiceName(carrying("/accounts")), "an identity with no module") - require.Empty(t, codeflyServiceName(carrying("/")), "an identity with neither") - require.Empty(t, codeflyServiceName(t.Context()), "no provider: the SDK was not initialised") -} - -// wool's tracer speaks plaintext only. An https:// endpoint is refused rather -// than dialed in the clear and reported as TLS. -func TestEnableTracingRefusesATransportWoolCannotDial(t *testing.T) { - provider, err := enableTracing(t.Context(), - telemetryDestination{Endpoint: "collector.example:4317", Insecure: false}, false) - require.ErrorIs(t, err, errWoolCannotDialTLS) - require.Nil(t, provider) -} - -func TestEnableTracingRegistersOnlyWhatTheDestinationCalls(t *testing.T) { - restore := func(t *testing.T) { - t.Helper() - previous := otel.GetTracerProvider() - t.Cleanup(func() { - wool.RegisterTelemetry(nil) - otel.SetTracerProvider(previous) - }) - } - absent := telemetryDestination{AbsentReason: "a local run has no cell collector"} - - t.Run("absent outside a local run registers no tracer", func(t *testing.T) { - restore(t) - provider, err := enableTracing(t.Context(), absent, false) - require.NoError(t, err) - require.Nil(t, provider) - require.False(t, wool.TelemetryEnabled()) - }) - t.Run("absent in a local run uses wool's stdout tracer", func(t *testing.T) { - restore(t) - provider, err := enableTracing(t.Context(), absent, true) - require.NoError(t, err) - require.NotNil(t, provider) - require.True(t, wool.TelemetryEnabled()) - shutdown(t, provider) - }) - t.Run("available over http exports spans to the collector, named by the environment", func(t *testing.T) { - restore(t) - t.Setenv("OTEL_SERVICE_NAME", "named-by-the-platform") - t.Setenv("OTEL_RESOURCE_ATTRIBUTES", "") - endpoint, capture := startTraceCapture(t) - provider, err := enableTracing(t.Context(), telemetryDestination{Endpoint: endpoint, Insecure: true}, false) - require.NoError(t, err) - require.NotNil(t, provider) - require.True(t, wool.TelemetryEnabled()) - - _, span := otel.Tracer("test").Start(t.Context(), "operation") - span.End() - shutdown(t, provider) // flushes the batch - - select { - case request := <-capture.requests: - var name string - for _, resourceSpans := range request.GetResourceSpans() { - for _, attribute := range resourceSpans.GetResource().GetAttributes() { - if attribute.GetKey() == "service.name" { - name = attribute.GetValue().GetStringValue() - } - } - } - require.Equal(t, "named-by-the-platform", name) - case <-time.After(5 * time.Second): - t.Fatal("timed out waiting for OTLP spans") - } - }) -} - -type traceCapture struct { - collectortracev1.UnimplementedTraceServiceServer - requests chan *collectortracev1.ExportTraceServiceRequest -} - -func (c *traceCapture) Export( - _ context.Context, - request *collectortracev1.ExportTraceServiceRequest, -) (*collectortracev1.ExportTraceServiceResponse, error) { - c.requests <- request - return &collectortracev1.ExportTraceServiceResponse{}, nil -} - -// startTraceCapture serves an in-process OTLP/gRPC trace receiver on loopback. -func startTraceCapture(t *testing.T) (string, *traceCapture) { - t.Helper() - listener, err := net.Listen("tcp", "127.0.0.1:0") - require.NoError(t, err) - capture := &traceCapture{requests: make(chan *collectortracev1.ExportTraceServiceRequest, 4)} - server := grpc.NewServer() - collectortracev1.RegisterTraceServiceServer(server, capture) - go func() { _ = server.Serve(listener) }() - t.Cleanup(func() { - server.Stop() - _ = listener.Close() - }) - return listener.Addr().String(), capture -} - -func shutdown(t *testing.T, provider interface{ Shutdown(context.Context) error }) { - t.Helper() - ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) - defer cancel() - require.NoError(t, provider.Shutdown(ctx)) -} diff --git a/module/services/auth-gateway/code/telemetry_metrics.go b/module/services/auth-gateway/code/telemetry_metrics.go index 114e1e107..ad2e7f8e0 100644 --- a/module/services/auth-gateway/code/telemetry_metrics.go +++ b/module/services/auth-gateway/code/telemetry_metrics.go @@ -2,48 +2,60 @@ package main import ( "context" - "errors" "net/http" runtimemetrics "runtime/metrics" "time" + "github.com/prometheus/client_golang/prometheus" + "github.com/prometheus/client_golang/prometheus/promhttp" "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp" otelruntime "go.opentelemetry.io/contrib/instrumentation/runtime" "go.opentelemetry.io/otel" + "go.opentelemetry.io/otel/attribute" "go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc" + otelprometheus "go.opentelemetry.io/otel/exporters/prometheus" otelmetric "go.opentelemetry.io/otel/metric" "go.opentelemetry.io/otel/sdk/metric" + "go.opentelemetry.io/otel/sdk/resource" ) -// otelMetrics owns the process's MeterProvider. Metrics leave by one path only: -// an OTLP push to the cell's collector, which is the record for traces and -// metrics alike. There is no scrape endpoint to mount, expose or exempt. type otelMetrics struct { provider *metric.MeterProvider + handler http.Handler } -func enableOTELMetrics(ctx context.Context, destination telemetryDestination) (*otelMetrics, error) { - if destination.URL == "" { - // WithEndpointURL("") would fall back to the exporter's own default, - // localhost:4317: a process that exports to nowhere and reports success. - return nil, errors.New("telemetry: no collector URL to export metrics to") +func enableOTELMetrics(ctx context.Context, serviceName, endpoint string) (*otelMetrics, error) { + exporter, err := otlpmetricgrpc.New( + ctx, + otlpmetricgrpc.WithEndpoint(endpoint), + otlpmetricgrpc.WithInsecure(), + ) + if err != nil { + return nil, err } - // The resource comes before the exporter, so a failure here leaves nothing - // to shut down: an exporter holds a client connection that only Shutdown - // releases. - res, err := currentTelemetryResource(ctx) + registry := prometheus.NewRegistry() + scrapeExporter, err := otelprometheus.New( + otelprometheus.WithRegisterer(registry), + otelprometheus.WithResourceAsConstantLabels( + attribute.NewAllowKeysFilter("service.name"), + ), + ) if err != nil { return nil, err } - // The exporter reads the transport off the URL's scheme itself: http:// is - // plaintext on the wire, because the mesh supplies mTLS, and https:// is TLS. - exporter, err := otlpmetricgrpc.New(ctx, otlpmetricgrpc.WithEndpointURL(destination.URL)) + res, err := resource.New( + ctx, + resource.WithService(), + resource.WithFromEnv(), + resource.WithAttributes(attribute.String("service.name", serviceName)), + ) if err != nil { return nil, err } provider := metric.NewMeterProvider( metric.WithResource(res), metric.WithReader(metric.NewPeriodicReader(exporter, metric.WithInterval(30*time.Second))), + metric.WithReader(scrapeExporter), ) if err := otelruntime.Start(otelruntime.WithMeterProvider(provider)); err != nil { _ = provider.Shutdown(ctx) @@ -54,7 +66,10 @@ func enableOTELMetrics(ctx context.Context, destination telemetryDestination) (* return nil, err } otel.SetMeterProvider(provider) - return &otelMetrics{provider: provider}, nil + return &otelMetrics{ + provider: provider, + handler: promhttp.HandlerFor(registry, promhttp.HandlerOpts{}), + }, nil } func startGCActivityMetrics(provider *metric.MeterProvider) error { @@ -88,14 +103,18 @@ func startGCActivityMetrics(provider *metric.MeterProvider) error { return err } -// newGatewayHTTPHandler instruments the gateway's HTTP listener when metrics are -// on. It adds no route of its own: the listener serves the gateway's routes and -// nothing else, with or without telemetry. func newGatewayHTTPHandler(gateway http.Handler, metrics *otelMetrics) http.Handler { if metrics == nil { return gateway } - return otelhttp.NewHandler(gateway, "auth-gateway.gateway") + mux := http.NewServeMux() + mux.Handle("GET /metrics", metrics.Handler()) + mux.Handle("/", otelhttp.NewHandler(gateway, "auth-gateway.gateway")) + return mux +} + +func (m *otelMetrics) Handler() http.Handler { + return m.handler } func (m *otelMetrics) Shutdown(ctx context.Context) error { diff --git a/module/services/auth-gateway/code/telemetry_metrics_test.go b/module/services/auth-gateway/code/telemetry_metrics_test.go index 3f66755eb..6e461dae4 100644 --- a/module/services/auth-gateway/code/telemetry_metrics_test.go +++ b/module/services/auth-gateway/code/telemetry_metrics_test.go @@ -39,9 +39,8 @@ func (c *authMetricCapture) Export( func TestAuthGatewayTelemetryExportsGatewayAndGRPCRED(t *testing.T) { endpoint, capture := startAuthMetricCapture(t) t.Setenv("OTEL_RESOURCE_ATTRIBUTES", "service.instance.id=auth-test-instance") - t.Setenv("OTEL_SERVICE_NAME", "named-by-the-platform") previousMeterProvider := otel.GetMeterProvider() - metrics, err := enableOTELMetrics(t.Context(), availableAt(t, "http", endpoint)) + metrics, err := enableOTELMetrics(t.Context(), "test-auth-gateway", endpoint) require.NoError(t, err) require.NotNil(t, metrics) t.Cleanup(func() { @@ -58,10 +57,19 @@ func TestAuthGatewayTelemetryExportsGatewayAndGRPCRED(t *testing.T) { require.Equal(t, http.StatusNotFound, getStatus(t, httpServer.URL+"/missing")) runtime.GC() + response, err := http.Get(httpServer.URL + "/metrics") + require.NoError(t, err) + t.Cleanup(func() { _ = response.Body.Close() }) + require.Equal(t, http.StatusOK, response.StatusCode) + prometheusBody, err := io.ReadAll(response.Body) + require.NoError(t, err) + require.Contains(t, string(prometheusBody), "http_server_request_duration_seconds_count") + require.Contains(t, string(prometheusBody), `http_route="/health"`) + require.Contains(t, string(prometheusBody), `service_name="test-auth-gateway"`) + require.NoError(t, metrics.provider.ForceFlush(t.Context())) request := receiveAuthMetrics(t, capture.requests) require.Equal(t, "auth-test-instance", authResourceAttribute(request, "service.instance.id")) - require.Equal(t, "named-by-the-platform", authResourceAttribute(request, "service.name")) metricNames := authExportedMetricNames(request) require.Contains(t, metricNames, "go.goroutine.count") require.Contains(t, metricNames, "go.gc.cycle.count") @@ -80,26 +88,19 @@ func TestAuthGatewayTelemetryExportsGatewayAndGRPCRED(t *testing.T) { require.Equal(t, uint64(2), httpRoutes["/health"]) } -// Metrics leave by OTLP push alone: the gateway's listener serves its own routes -// and no scrape endpoint, whether or not telemetry is on. -func TestGatewayHandlerHasNoMetricsRoute(t *testing.T) { +func TestGatewayHandlerHasNoMetricsRouteWhenTelemetryIsDisabled(t *testing.T) { matcher := NewRouteMatcher([]*RouteEntry{{ Service: "self", Method: http.MethodGet, Path: "/health", }}, nil) gateway := NewGateway(nil, matcher, nil, nil, newFakeSolutionRegistry(), newFakeClientRegistry()) + request := httptest.NewRequest(http.MethodGet, "/metrics", nil) + recorder := httptest.NewRecorder() - for name, metrics := range map[string]*otelMetrics{ - "telemetry disabled": nil, - "telemetry enabled": {}, - } { - t.Run(name, func(t *testing.T) { - recorder := httptest.NewRecorder() - newGatewayHTTPHandler(gateway, metrics).ServeHTTP(recorder, httptest.NewRequest(http.MethodGet, "/metrics", nil)) - require.Equal(t, http.StatusNotFound, recorder.Code) - }) - } + newGatewayHTTPHandler(gateway, nil).ServeHTTP(recorder, request) + + require.Equal(t, http.StatusNotFound, recorder.Code) } func authInt64SumValue(request *collectormetricsv1.ExportMetricsServiceRequest, name string) int64 { @@ -134,65 +135,6 @@ func authFloat64SumValue(request *collectormetricsv1.ExportMetricsServiceRequest return value } -// availableAt resolves the destination the way the process does, from the -// endpoint the platform would deliver, so these tests hand the exporter the URL -// it really receives rather than a hand-built struct. -func availableAt(t *testing.T, scheme, hostPort string) telemetryDestination { - t.Helper() - destination, err := resolveTelemetryDestination(false, group(map[string]string{ - "TELEMETRY_STATE": "available", - "OTEL_EXPORTER_OTLP_ENDPOINT": scheme + "://" + hostPort, - })) - require.NoError(t, err) - return destination -} - -// The exporter reads its transport off the URL's scheme, so an https:// URL is -// TLS and cannot talk to a plaintext receiver. A constant WithInsecure() would -// have let it. -func TestEnableOTELMetricsDerivesTransportFromTheURLScheme(t *testing.T) { - endpoint, capture := startAuthMetricCapture(t) - previousMeterProvider := otel.GetMeterProvider() - metrics, err := enableOTELMetrics(t.Context(), availableAt(t, "https", endpoint)) - require.NoError(t, err) - require.NotNil(t, metrics) - t.Cleanup(func() { - shutdownCtx, cancel := context.WithTimeout(context.Background(), time.Second) - defer cancel() - _ = metrics.Shutdown(shutdownCtx) // the flush it attempts is the failure under test - otel.SetMeterProvider(previousMeterProvider) - }) - - flushCtx, cancel := context.WithTimeout(t.Context(), time.Second) - defer cancel() - require.Error(t, metrics.provider.ForceFlush(flushCtx), "a TLS client cannot export to a plaintext receiver") - select { - case <-capture.requests: - t.Fatal("an https:// destination reached a plaintext receiver") - default: - } -} - -// A destination with no collector URL is refused. WithEndpointURL("") would -// otherwise fall back to the exporter's own default, localhost:4317. -func TestEnableOTELMetricsRefusesADestinationWithoutAURL(t *testing.T) { - metrics, err := enableOTELMetrics(t.Context(), telemetryDestination{AbsentReason: "no collector"}) - require.ErrorContains(t, err, "no collector URL") - require.Nil(t, metrics) -} - -// A resource that cannot be built fails the whole call, and the call builds it -// before it creates the exporter: there is no exporter, and so no client -// connection, left running behind the failure. -func TestEnableOTELMetricsFailsWhenTheResourceCannotBeBuilt(t *testing.T) { - t.Setenv("OTEL_SERVICE_NAME", "") - t.Setenv("OTEL_RESOURCE_ATTRIBUTES", "an-attribute-with-no-value") - endpoint, _ := startAuthMetricCapture(t) - metrics, err := enableOTELMetrics(t.Context(), availableAt(t, "http", endpoint)) - require.Error(t, err) - require.Nil(t, metrics) -} - func startAuthMetricCapture(t *testing.T) (string, *authMetricCapture) { t.Helper() listener, err := net.Listen("tcp", "127.0.0.1:0") diff --git a/module/services/auth-gateway/service.codefly.yaml b/module/services/auth-gateway/service.codefly.yaml index e118ecac3..ae4e05e2c 100644 --- a/module/services/auth-gateway/service.codefly.yaml +++ b/module/services/auth-gateway/service.codefly.yaml @@ -20,6 +20,9 @@ service-dependencies: - name: cache endpoints: - name: write + - name: telemetry + endpoints: + - name: grpc workspace-configuration-dependencies: # APP_BASE_URL: this host's own public origin. The gateway publishes it in # the RFC 9728 document for a solution's MCP endpoint and requires a token's diff --git a/module/services/frontend/code/packages/saas-sdk/generated/contract/catalog.codefly.json b/module/services/frontend/code/packages/saas-sdk/generated/contract/catalog.codefly.json index 78363e5ae..03ccde58c 100644 --- a/module/services/frontend/code/packages/saas-sdk/generated/contract/catalog.codefly.json +++ b/module/services/frontend/code/packages/saas-sdk/generated/contract/catalog.codefly.json @@ -1,7 +1,7 @@ { "schema": "codefly/module-api-contracts/v1", "package": "codefly/saas-starter", - "version": "0.2.0", + "version": "0.1.0", "endpoints": [ { "service": "accounts", diff --git a/module/services/frontend/code/packages/saas-sdk/generated/library.codefly.yaml b/module/services/frontend/code/packages/saas-sdk/generated/library.codefly.yaml index 4169168c5..68ddfecbd 100644 --- a/module/services/frontend/code/packages/saas-sdk/generated/library.codefly.yaml +++ b/module/services/frontend/code/packages/saas-sdk/generated/library.codefly.yaml @@ -1,6 +1,6 @@ kind: library name: saas-sdk -version: 0.2.0 +version: 0.1.0 languages: - name: typescript path: typescript/ @@ -8,7 +8,7 @@ languages: - '@codefly-dev/saas-sdk' sources: - package: codefly/saas-starter - version: 0.2.0 + version: 0.1.0 module: saas-starter service: accounts endpoint: connect diff --git a/module/services/frontend/code/src/test/pipeline-setup.ts b/module/services/frontend/code/src/test/pipeline-setup.ts index 85d2f0d0f..08746fad9 100644 --- a/module/services/frontend/code/src/test/pipeline-setup.ts +++ b/module/services/frontend/code/src/test/pipeline-setup.ts @@ -15,8 +15,8 @@ // workers, so starting the graph per file would start one graph per file. // // withDependencies passes --exclude-root: it brings up the frontend's -// dependencies (auth-gateway, Accounts, Postgres, Vault, Redis, storage) -// without the Next server the frontend itself would run. The +// dependencies (auth-gateway, Accounts, Postgres, Vault, Redis, storage, +// telemetry) without the Next server the frontend itself would run. The // pipeline tier addresses the gateway directly, so the root service is dead // weight — and leaving it out means these tests can run alongside a // `codefly run service` dev stack instead of fighting it for the Next dev lock. @@ -47,7 +47,7 @@ export default async function setup(): Promise<(() => Promise) | void> { readyService: "auth-gateway", scope, fixture: "dev-admin", - silents: ["store", "cache"], + silents: ["store", "cache", "telemetry"], echo: process.env.CODEFLY_TEST_ECHO === "1", }); } catch (error) { diff --git a/module/services/telemetry/code/collector/collector.go b/module/services/telemetry/code/collector/collector.go new file mode 100644 index 000000000..ed9196499 --- /dev/null +++ b/module/services/telemetry/code/collector/collector.go @@ -0,0 +1,218 @@ +package collector + +import ( + "bytes" + "context" + "errors" + "fmt" + "io" + "log/slog" + "net/http" + "net/url" + "strings" + "time" + + collectorlogsv1 "go.opentelemetry.io/proto/otlp/collector/logs/v1" + collectormetricsv1 "go.opentelemetry.io/proto/otlp/collector/metrics/v1" + collectortracev1 "go.opentelemetry.io/proto/otlp/collector/trace/v1" + "google.golang.org/protobuf/proto" +) + +type Config struct { + Exporter string + Endpoint string + Headers string + HTTPClient *http.Client + Logger *slog.Logger +} + +type Collector struct { + exporter string + endpoint string + headers http.Header + client *http.Client + logger *slog.Logger +} + +type TraceService struct { + collectortracev1.UnimplementedTraceServiceServer + collector *Collector +} + +type MetricsService struct { + collectormetricsv1.UnimplementedMetricsServiceServer + collector *Collector +} + +type LogsService struct { + collectorlogsv1.UnimplementedLogsServiceServer + collector *Collector +} + +func (c *Collector) TraceService() *TraceService { return &TraceService{collector: c} } + +func (c *Collector) MetricsService() *MetricsService { + return &MetricsService{collector: c} +} + +func (c *Collector) LogsService() *LogsService { return &LogsService{collector: c} } + +func New(config Config) (*Collector, error) { + exporter := strings.ToLower(strings.TrimSpace(config.Exporter)) + // Fail closed on an empty exporter instead of defaulting to "debug". + // An empty value never meant "the operator chose debug" — every committed + // observability.env sets this key explicitly, and so does + // configurations/local-dogfood/observability.env.example — it meant the configuration never reached the process. + // Defaulting it to debug turned that into SILENT trace loss: the collector + // logged "received OTLP signal" and dropped every span while its own + // ConfigMap said otlphttp. Refusing to start surfaces the same fault in one + // line of pod logs. + if exporter == "" { + return nil, errors.New( + "telemetry: OBSERVABILITY_EXPORTER is required and must be debug or otlphttp; " + + "the observability workspace configuration did not reach this process " + + "(on the local-dogfood profile, copy configurations/local-dogfood/observability.env.example to observability.env)") + } + switch exporter { + case "debug": + if strings.TrimSpace(config.Endpoint) != "" || strings.TrimSpace(config.Headers) != "" { + return nil, errors.New("telemetry: debug exporter cannot have an external endpoint or headers") + } + case "otlphttp": + endpoint, err := url.Parse(strings.TrimRight(strings.TrimSpace(config.Endpoint), "/")) + if err != nil || endpoint.Scheme == "" || endpoint.Host == "" { + return nil, errors.New("telemetry: OTLP/HTTP endpoint must be absolute") + } + // Plaintext is allowed only to loopback, which never leaves the pod and so + // is governed by no network policy. Everything else must be HTTPS, because + // that is the only egress this module actually grants telemetry: + // services/telemetry/service.codefly.yaml declares spec.deployment.public-egress-ports + // [443], which renders allow-telemetry-public-egress — TCP 443 to public IP + // space with 10/8, 172.16/12, 192.168/16 and fc00::/7 excepted — on top of a + // namespace-wide default-deny that has no allow-intra-namespace rule. + // + // A cluster-internal plaintext address (a *.svc name on 4318) is therefore + // denied at the network layer no matter what this check says; accepting it + // here would only move the failure from a startup error to a 10s export + // timeout per batch. Reaching an in-cluster collector is a topology change — + // a declared dependency edge with its regenerated NetworkPolicy — not a + // transport exemption in application code. + local := endpoint.Hostname() == "localhost" || endpoint.Hostname() == "127.0.0.1" + if endpoint.Scheme != "https" && (endpoint.Scheme != "http" || !local) { + return nil, errors.New("telemetry: OTLP/HTTP endpoint must use HTTPS") + } + config.Endpoint = endpoint.String() + default: + return nil, errors.New("telemetry: OBSERVABILITY_EXPORTER must be debug or otlphttp") + } + headers, err := parseHeaders(config.Headers) + if err != nil { + return nil, err + } + client := config.HTTPClient + if client == nil { + client = &http.Client{Timeout: 10 * time.Second} + } + logger := config.Logger + if logger == nil { + logger = slog.Default() + } + return &Collector{ + exporter: exporter, + endpoint: strings.TrimRight(config.Endpoint, "/"), + headers: headers, + client: client, + logger: logger, + }, nil +} + +func (s *TraceService) Export( + ctx context.Context, + request *collectortracev1.ExportTraceServiceRequest, +) (*collectortracev1.ExportTraceServiceResponse, error) { + if err := s.collector.deliver(ctx, "traces", request, len(request.GetResourceSpans())); err != nil { + return nil, err + } + return &collectortracev1.ExportTraceServiceResponse{}, nil +} + +func (s *MetricsService) Export( + ctx context.Context, + request *collectormetricsv1.ExportMetricsServiceRequest, +) (*collectormetricsv1.ExportMetricsServiceResponse, error) { + if err := s.collector.deliver(ctx, "metrics", request, len(request.GetResourceMetrics())); err != nil { + return nil, err + } + return &collectormetricsv1.ExportMetricsServiceResponse{}, nil +} + +func (s *LogsService) Export( + ctx context.Context, + request *collectorlogsv1.ExportLogsServiceRequest, +) (*collectorlogsv1.ExportLogsServiceResponse, error) { + if err := s.collector.deliver(ctx, "logs", request, len(request.GetResourceLogs())); err != nil { + return nil, err + } + return &collectorlogsv1.ExportLogsServiceResponse{}, nil +} + +func (c *Collector) deliver(ctx context.Context, signal string, message proto.Message, resources int) error { + if c.exporter == "debug" { + c.logger.Info("received OTLP signal", "signal", signal, "resources", resources) + return nil + } + body, err := proto.Marshal(message) + if err != nil { + return fmt.Errorf("telemetry: encode %s: %w", signal, err) + } + request, err := http.NewRequestWithContext( + ctx, + http.MethodPost, + c.endpoint+"/v1/"+signal, + bytes.NewReader(body), + ) + if err != nil { + return fmt.Errorf("telemetry: create %s export: %w", signal, err) + } + for name, values := range c.headers { + for _, value := range values { + request.Header.Add(name, value) + } + } + request.Header.Set("Content-Type", "application/x-protobuf") + response, err := c.client.Do(request) + if err != nil { + return fmt.Errorf("telemetry: export %s: %w", signal, err) + } + defer func() { _ = response.Body.Close() }() + _, _ = io.Copy(io.Discard, io.LimitReader(response.Body, 64*1024)) + if response.StatusCode < http.StatusOK || response.StatusCode >= http.StatusMultipleChoices { + return fmt.Errorf("telemetry: export %s returned HTTP %d", signal, response.StatusCode) + } + return nil +} + +func parseHeaders(raw string) (http.Header, error) { + headers := make(http.Header) + raw = strings.TrimSpace(raw) + if raw == "" { + return headers, nil + } + values, err := url.ParseQuery(strings.ReplaceAll(raw, ",", "&")) + if err != nil { + return nil, errors.New("telemetry: OTLP headers must use URL-encoded key=value pairs") + } + for name, entries := range values { + name = strings.TrimSpace(name) + if name == "" { + return nil, errors.New("telemetry: OTLP header name is required") + } + for _, value := range entries { + if strings.ContainsAny(value, "\r\n") { + return nil, errors.New("telemetry: OTLP header value contains a newline") + } + headers.Add(name, value) + } + } + return headers, nil +} diff --git a/module/services/telemetry/code/collector/collector_test.go b/module/services/telemetry/code/collector/collector_test.go new file mode 100644 index 000000000..dc121fbb4 --- /dev/null +++ b/module/services/telemetry/code/collector/collector_test.go @@ -0,0 +1,134 @@ +package collector_test + +import ( + "context" + "io" + "net" + "net/http" + "net/http/httptest" + "sync" + "testing" + + "telemetry/collector" + + collectortracev1 "go.opentelemetry.io/proto/otlp/collector/trace/v1" + tracev1 "go.opentelemetry.io/proto/otlp/trace/v1" + "google.golang.org/grpc" + "google.golang.org/grpc/credentials/insecure" + "google.golang.org/grpc/test/bufconn" + "google.golang.org/protobuf/proto" + + "github.com/stretchr/testify/require" +) + +func TestCollectorReceivesGRPCAndForwardsOTLPHTTP(t *testing.T) { + var mu sync.Mutex + var forwarded collectortracev1.ExportTraceServiceRequest + upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + require.Equal(t, "/v1/traces", r.URL.Path) + require.Equal(t, "Bearer test", r.Header.Get("Authorization")) + body, err := io.ReadAll(r.Body) + require.NoError(t, err) + mu.Lock() + defer mu.Unlock() + require.NoError(t, proto.Unmarshal(body, &forwarded)) + w.WriteHeader(http.StatusOK) + })) + defer upstream.Close() + + sink, err := collector.New(collector.Config{ + Exporter: "otlphttp", + Endpoint: upstream.URL, + Headers: "Authorization=Bearer+test", + }) + require.NoError(t, err) + listener := bufconn.Listen(1024 * 1024) + server := grpc.NewServer() + collectortracev1.RegisterTraceServiceServer(server, sink.TraceService()) + go func() { _ = server.Serve(listener) }() + t.Cleanup(server.Stop) + + connection, err := grpc.NewClient( + "passthrough:///bufconn", + grpc.WithTransportCredentials(insecure.NewCredentials()), + grpc.WithContextDialer(func(context.Context, string) (net.Conn, error) { + return listener.Dial() + }), + ) + require.NoError(t, err) + defer func() { _ = connection.Close() }() + client := collectortracev1.NewTraceServiceClient(connection) + _, err = client.Export(t.Context(), &collectortracev1.ExportTraceServiceRequest{ + ResourceSpans: []*tracev1.ResourceSpans{{}}, + }) + require.NoError(t, err) + mu.Lock() + defer mu.Unlock() + require.Len(t, forwarded.GetResourceSpans(), 1) +} + +func TestCollectorConfigurationFailsClosed(t *testing.T) { + _, err := collector.New(collector.Config{ + Exporter: "debug", Endpoint: "https://collector.example", + }) + require.ErrorContains(t, err, "cannot have") + _, err = collector.New(collector.Config{ + Exporter: "otlphttp", Endpoint: "http://collector.example", + }) + require.ErrorContains(t, err, "HTTPS") +} + +// TestCollectorRequiresExplicitExporter pins the fail-closed behaviour: an empty +// OBSERVABILITY_EXPORTER used to fall back to "debug", which turned a broken +// configuration carrier into silent trace loss instead of a startup failure. +func TestCollectorRequiresExplicitExporter(t *testing.T) { + _, err := collector.New(collector.Config{}) + require.ErrorContains(t, err, "OBSERVABILITY_EXPORTER is required") + + _, err = collector.New(collector.Config{Exporter: " "}) + require.ErrorContains(t, err, "OBSERVABILITY_EXPORTER is required") +} + +// TestCollectorEndpointTransportPolicy pins which OTLP/HTTP destinations may be +// reached in plaintext: loopback only, because loopback never leaves the pod. +// +// The cluster-internal cases are the regression. Exempting *.svc / +// *.svc.cluster.local from the HTTPS requirement looks safe — those names +// resolve only inside the cluster — but this module grants telemetry exactly one +// egress path: public_egress_ports [443] in +// services/telemetry/service.codefly.yaml (spec.deployment), rendered as +// allow-telemetry-public-egress (TCP 443 to public IP space, private ranges +// excepted) over a namespace-wide default-deny with no allow-intra-namespace +// rule. A ClusterIP on 4318 is denied there no matter what this package accepts, +// so accepting it only converts a startup error into a 10s export timeout per +// batch. Reaching an in-cluster collector is a topology change, not a transport +// exemption in application code. +func TestCollectorEndpointTransportPolicy(t *testing.T) { + for _, testCase := range []struct { + name string + endpoint string + accepted bool + }{ + {"loopback name", "http://localhost:4318", true}, + {"loopback address", "http://127.0.0.1:4318", true}, + {"external TLS", "https://example.com", true}, + {"in-cluster collector FQDN is not reachable and not exempt", + "http://otel-collector.otel-collector.svc.cluster.local:4318", false}, + {"in-cluster collector short form", "http://otel-collector.otel-collector.svc:4318", false}, + {"external plaintext", "http://example.com:4318", false}, + {"external plaintext subdomain", "http://otel.internal.ops.example.com:4318", false}, + {"host merely containing svc", "http://svc.example.com:4318", false}, + } { + t.Run(testCase.name, func(t *testing.T) { + _, err := collector.New(collector.Config{ + Exporter: "otlphttp", + Endpoint: testCase.endpoint, + }) + if testCase.accepted { + require.NoError(t, err) + return + } + require.ErrorContains(t, err, "HTTPS") + }) + } +} diff --git a/module/services/telemetry/code/go.mod b/module/services/telemetry/code/go.mod new file mode 100644 index 000000000..0ef711b29 --- /dev/null +++ b/module/services/telemetry/code/go.mod @@ -0,0 +1,58 @@ +module telemetry + +go 1.27.0 + +require ( + github.com/codefly-dev/sdk-go v0.2.1-0.20260926212413-b91ce91eb23e + github.com/stretchr/testify v1.12.1 + go.opentelemetry.io/proto/otlp v1.11.0 + google.golang.org/grpc v1.83.2 + google.golang.org/protobuf v1.36.12 +) + +require ( + buf.build/gen/go/bufbuild/protovalidate/protocolbuffers/go v1.36.12-20260825204119-511051f7f437.2 // indirect + buf.build/go/protovalidate v1.4.0 // indirect + cel.dev/cel-go v0.32.0 // indirect + cel.dev/expr v0.25.3 // indirect + github.com/Masterminds/semver v1.5.0 // indirect + github.com/Masterminds/semver/v3 v3.5.0 // indirect + github.com/antlr4-go/antlr/v4 v4.13.1 // indirect + github.com/codefly-dev/core v0.9.0 // indirect + github.com/go-openapi/analysis v1.0.0 // indirect + github.com/go-openapi/errors v0.22.8 // indirect + github.com/go-openapi/jsonpointer v1.0.1 // indirect + github.com/go-openapi/jsonreference v1.0.2 // indirect + github.com/go-openapi/loads v0.25.3 // indirect + github.com/go-openapi/spec v1.0.1 // indirect + github.com/go-openapi/strfmt v0.27.2 // indirect + github.com/go-openapi/swag/conv v0.29.1 // indirect + github.com/go-openapi/swag/jsonutils v0.29.1 // indirect + github.com/go-openapi/swag/loading v0.29.1 // indirect + github.com/go-openapi/swag/mangling v0.29.1 // indirect + github.com/go-openapi/swag/pools v0.29.1 // indirect + github.com/go-openapi/swag/stringutils v0.29.1 // indirect + github.com/go-openapi/swag/typeutils v0.29.1 // indirect + github.com/go-openapi/swag/yamlutils v0.29.1 // indirect + github.com/go-viper/mapstructure/v2 v2.5.0 // indirect + github.com/gofrs/flock v0.13.1 // indirect + github.com/google/go-github/v89 v89.0.0 // indirect + github.com/google/go-querystring v1.2.0 // indirect + github.com/google/uuid v1.6.0 // indirect + github.com/grpc-ecosystem/grpc-gateway/v2 v2.30.0 // indirect + github.com/hashicorp/errwrap v1.1.0 // indirect + github.com/hashicorp/go-multierror v1.1.1 // indirect + github.com/mitchellh/mapstructure v1.5.0 // indirect + github.com/oklog/ulid/v2 v2.1.2 // indirect + github.com/pkg/errors v0.9.1 // indirect + github.com/yoheimuta/go-protoparser/v4 v4.14.2 // indirect + go.yaml.in/yaml/v3 v3.0.5 // indirect + golang.org/x/exp v0.0.0-20260824195058-e88cd73687aa // indirect + golang.org/x/net v0.59.0 // indirect + golang.org/x/sys v0.48.0 // indirect + golang.org/x/term v0.46.0 // indirect + golang.org/x/text v0.42.0 // indirect + google.golang.org/genproto/googleapis/api v0.0.0-20260831171406-18b4a7587f8a // indirect + google.golang.org/genproto/googleapis/rpc v0.0.0-20260831171406-18b4a7587f8a // indirect + gopkg.in/yaml.v3 v3.0.1 // indirect +) diff --git a/module/services/telemetry/code/go.sum b/module/services/telemetry/code/go.sum new file mode 100644 index 000000000..c77cc861b --- /dev/null +++ b/module/services/telemetry/code/go.sum @@ -0,0 +1,239 @@ +buf.build/gen/go/bufbuild/protovalidate/protocolbuffers/go v1.36.12-20260825204119-511051f7f437.2 h1:NbnlmV26O7oZ1iM5tsCI+GEx+3ZSdrhvnKQ/eWSrLiY= +buf.build/gen/go/bufbuild/protovalidate/protocolbuffers/go v1.36.12-20260825204119-511051f7f437.2/go.mod h1:TCt1lluMFnctISJXvkIQ4x3ABrPuUKCWKyjKdkJNBpw= +buf.build/go/protovalidate v1.4.0 h1:UjLrYbt5VX7+TMOs2+pG5FhZhIG1mSfK4EIopbb4LcM= +buf.build/go/protovalidate v1.4.0/go.mod h1:8vJfzNT6NIG2qm3uFsJDXMlRmG+bQJzbcIn1Aa0vPGs= +cel.dev/cel-go v0.32.0 h1:irvpFKr5EuGPyxeME03ERh0rii1TX+BDAnB9eL3IvNk= +cel.dev/cel-go v0.32.0/go.mod h1:DnVip7tpJSsgZymwfT+m1tnEVy3ivAjSMXPx12YrMkU= +cel.dev/expr v0.25.3 h1:A2jO8jwOugrrovveCWfj0KEZOfqiLgAcwjpHPhzIGw0= +cel.dev/expr v0.25.3/go.mod h1:hrXvqGP6G6gyx8UAHSHJ5RGk//1Oj5nXQ2NI02Nrsg4= +github.com/Masterminds/semver v1.5.0 h1:H65muMkzWKEuNDnfl9d70GUjFniHKHRbFPGBuZ3QEww= +github.com/Masterminds/semver v1.5.0/go.mod h1:MB6lktGJrhw8PrUyiEoblNEGEQ+RzHPF078ddwwvV3Y= +github.com/Masterminds/semver/v3 v3.5.0 h1:kQceYJfbupGfZOKZQg0kou0DgAKhzDg2NZPAwZ/2OOE= +github.com/Masterminds/semver/v3 v3.5.0/go.mod h1:4V+yj/TJE1HU9XfppCwVMZq3I84lprf4nC11bSS5beM= +github.com/Microsoft/go-winio v0.6.2 h1:F2VQgta7ecxGYO8k3ZZz3RS8fVIXVxONVUPlNERoyfY= +github.com/Microsoft/go-winio v0.6.2/go.mod h1:yd8OoFMLzJbo9gZq8j5qaps8bJ9aShtEA8Ipt1oGCvU= +github.com/VividCortex/ewma v1.2.0 h1:f58SaIzcDXrSy3kWaHNvuJgJ3Nmz59Zji6XoJR/q1ow= +github.com/VividCortex/ewma v1.2.0/go.mod h1:nz4BbCtbLyFDeC9SUHbtcT5644juEuWfUAUnGx7j5l4= +github.com/antlr4-go/antlr/v4 v4.13.1 h1:SqQKkuVZ+zWkMMNkjy5FZe5mr5WURWnlpmOuzYWrPrQ= +github.com/antlr4-go/antlr/v4 v4.13.1/go.mod h1:GKmUxMtwp6ZgGwZSva4eWPC5mS6vUAmOABFgjdkM7Nw= +github.com/blang/semver v3.5.1+incompatible h1:cQNTCjp13qL8KC3Nbxr/y2Bqb63oX6wdnnjpJbkM4JQ= +github.com/blang/semver v3.5.1+incompatible/go.mod h1:kRBLl5iJ+tD4TcOOxsy/0fnwebNt5EWlYSAyrTnjyyk= +github.com/blang/semver/v4 v4.0.0 h1:1PFHFE6yCCTv8C1TeyNNarDzntLi7wMI5i/pzqYIsAM= +github.com/blang/semver/v4 v4.0.0/go.mod h1:IbckMUScFkM3pff0VJDNKRiT6TG/YpiHIM2yvyW5YoQ= +github.com/brianvoe/gofakeit/v6 v6.28.0 h1:Xib46XXuQfmlLS2EXRuJpqcw8St6qSZz75OUo0tgAW4= +github.com/brianvoe/gofakeit/v6 v6.28.0/go.mod h1:Xj58BMSnFqcn/fAQeSK+/PLtC5kSb7FJIq4JyGa8vEs= +github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs= +github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= +github.com/cheggaaa/pb/v3 v3.2.1 h1:aprZbFRG+B7+ug76S8QZ6Y1PW168UHzOmbC3wa+aU6I= +github.com/cheggaaa/pb/v3 v3.2.1/go.mod h1:U9hSVxoKqJrZIE3PkFG1xXXNaK/Ilzg+EF/scpXzEdw= +github.com/clipperhouse/uax29/v2 v2.7.0 h1:+gs4oBZ2gPfVrKPthwbMzWZDaAFPGYK72F0NJv2v7Vk= +github.com/clipperhouse/uax29/v2 v2.7.0/go.mod h1:EFJ2TJMRUaplDxHKj1qAEhCtQPW2tJSwu5BF98AuoVM= +github.com/codefly-dev/core v0.7.2-0.20261003173032-735244e805bf h1:fvJJFtvYVDgXMI1BU1s//hqTAWCCouyzuO7A3PD4drA= +github.com/codefly-dev/core v0.7.2-0.20261003173032-735244e805bf/go.mod h1:2wt5DdL+OnJbIQT0w4cc7dS2sAGVK3QjnwW43r5cQQg= +github.com/codefly-dev/core v0.7.2-0.20261003222502-9d5f3fb7d5c3 h1:PSDuh+/pGQPLSPJ7DNVnE5dVypL7TkLvsYu1xbM4W6U= +github.com/codefly-dev/core v0.7.2-0.20261003222502-9d5f3fb7d5c3/go.mod h1:2wt5DdL+OnJbIQT0w4cc7dS2sAGVK3QjnwW43r5cQQg= +github.com/codefly-dev/core v0.9.0 h1:fZ+jnP33uQ4JqYBrABjOvdqF/A6K/hLcQumykjHWrgY= +github.com/codefly-dev/core v0.9.0/go.mod h1:2wt5DdL+OnJbIQT0w4cc7dS2sAGVK3QjnwW43r5cQQg= +github.com/codefly-dev/gortk v0.2.0 h1:7bOlS5valYz2zil+fZctQNcPCYBcPj86abcw9N8h1hQ= +github.com/codefly-dev/gortk v0.2.0/go.mod h1:dDWUMFgAP063OCGhTEpV7FFUj9+zTcxxO/nV80VKEwg= +github.com/codefly-dev/sdk-go v0.2.1-0.20260926212413-b91ce91eb23e h1:V8YSUvAm0QQgHY9V0QmHiHuvf+beHnI10D6cJLWB2eI= +github.com/codefly-dev/sdk-go v0.2.1-0.20260926212413-b91ce91eb23e/go.mod h1:X0JAk/gnr6ZcjuPsM+5jvisqY4iyTC1SeVac93iWX8A= +github.com/containerd/errdefs v1.0.0 h1:tg5yIfIlQIrxYtu9ajqY42W3lpS19XqdxRQeEwYG8PI= +github.com/containerd/errdefs v1.0.0/go.mod h1:+YBYIdtsnF4Iw6nWZhJcqGSg/dwvV7tyJ/kCkyJ2k+M= +github.com/containerd/errdefs/pkg v0.3.0 h1:9IKJ06FvyNlexW690DXuQNx2KA2cUJXx151Xdx3ZPPE= +github.com/containerd/errdefs/pkg v0.3.0/go.mod h1:NJw6s9HwNuRhnjJhM7pylWwMyAkmCQvQ4GpJHEqRLVk= +github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1VwoXQT9A3Wy9MM3WgvqSxFWenqJduM= +github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/distribution/reference v0.6.0 h1:0IXCQ5g4/QMHHkarYzh5l+u8T3t73zM5QvfrDyIgxBk= +github.com/distribution/reference v0.6.0/go.mod h1:BbU0aIcezP1/5jX/8MP0YiH4SdvB5Y4f/wlDRiLyi3E= +github.com/docker/docker v28.5.2+incompatible h1:DBX0Y0zAjZbSrm1uzOkdr1onVghKaftjlSWt4AFexzM= +github.com/docker/docker v28.5.2+incompatible/go.mod h1:eEKB0N0r5NX/I1kEveEz05bcu8tLC/8azJZsviup8Sk= +github.com/docker/go-connections v0.8.1 h1:JibmG5hULs5qXSr/cp/w3Pw5fZuStt4MOHMUExb29/M= +github.com/docker/go-connections v0.8.1/go.mod h1:no1qkHdjq7kLMGUXYAduOhYPSJxxvgWBh7ogVvptn3Q= +github.com/docker/go-units v0.5.0 h1:69rxXcBk27SvSaaxTtLh/8llcHD8vYHT7WSdRZ/jvr4= +github.com/docker/go-units v0.5.0/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDDbaIK4Dk= +github.com/fatih/color v1.19.0 h1:Zp3PiM21/9Ld6FzSKyL5c/BULoe/ONr9KlbYVOfG8+w= +github.com/fatih/color v1.19.0/go.mod h1:zNk67I0ZUT1bEGsSGyCZYZNrHuTkJJB+r6Q9VuMi0LE= +github.com/felixge/httpsnoop v1.1.0 h1:3YtUj32ZZkqZtt3sZZsClsymw/QDuVfpNhoA31zeORc= +github.com/felixge/httpsnoop v1.1.0/go.mod h1:Zqxgdd+1Rkcz8euOqdr7lqgCRJztwr5hp9vDSi5UZCE= +github.com/fsnotify/fsnotify v1.10.1 h1:b0/UzAf9yR5rhf3RPm9gf3ehBPpf0oZKIjtpKrx59Ho= +github.com/fsnotify/fsnotify v1.10.1/go.mod h1:TLheqan6HD6GBK6PrDWyDPBaEV8LspOxvPSjC+bVfgo= +github.com/go-errors/errors v1.4.2 h1:J6MZopCL4uSllY1OfXM374weqZFFItUbrImctkmUxIA= +github.com/go-errors/errors v1.4.2/go.mod h1:sIVyrIiJhuEF+Pj9Ebtd6P/rEYROXFi3BopGUQ5a5Og= +github.com/go-logr/logr v1.4.4 h1:tG4xh9yMsRCAiodLVTxyrkzSZ9+o0L1Kg/+cPVcbP/8= +github.com/go-logr/logr v1.4.4/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY= +github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag= +github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE= +github.com/go-ole/go-ole v1.3.0 h1:Dt6ye7+vXGIKZ7Xtk4s6/xVdGDQynvom7xCFEdWr6uE= +github.com/go-ole/go-ole v1.3.0/go.mod h1:5LS6F96DhAwUc7C+1HLexzMXY1xGRSryjyPPKW6zv78= +github.com/go-openapi/analysis v1.0.0 h1:sNvbAGCJqUTqIAodr9IVqJMmuZas3YS9ms1dGK9yiJ4= +github.com/go-openapi/analysis v1.0.0/go.mod h1:NhYjJ57fnE+bcE7UwrJyMkhWA3Dfz7TdiBfTVAnos4Y= +github.com/go-openapi/errors v0.22.8 h1:oP7sW7TWc3wFFjrzzj0nI83H2qMBkNjNfSd+XRejk/I= +github.com/go-openapi/errors v0.22.8/go.mod h1:BuUoHcYrU6E7V9gfj1I5wLQqgtIHnup/alXZ8KdgQ0w= +github.com/go-openapi/jsonpointer v1.0.1 h1:2KxywRmNwJkT/FMBa3iRNHEaAxSJvjqoufQZy3au1Mg= +github.com/go-openapi/jsonpointer v1.0.1/go.mod h1:wI7ZYsFmbIi9nBXOZqgDaS/bqOchRGZjqxFli7FBYxY= +github.com/go-openapi/jsonreference v1.0.2 h1:oS4et8FOf3p3UQxEo4Xt0esijmBUM+F259Xl72OSZsc= +github.com/go-openapi/jsonreference v1.0.2/go.mod h1:TbUNSOo+fcorZjFaNoiSDSoaNnnZtqJtLGR1PuvE/Cs= +github.com/go-openapi/loads v0.25.3 h1:V+jKy/thXWdLJUuYC8sZX2dICyAa8M3DokF70jj34P0= +github.com/go-openapi/loads v0.25.3/go.mod h1:LgLyCSOLBL2Qnj0Ps1oo2YH8jZoEKsMeVZCc+ALIxFo= +github.com/go-openapi/spec v1.0.1 h1:lj2vdGpNDcVgwRc6qXdw6qt/KQpCtSa9tnUH6vpDPDk= +github.com/go-openapi/spec v1.0.1/go.mod h1:M//GWQGtDUAjnP37gE6fInLgaczB+FatoipV3H1fYw8= +github.com/go-openapi/strfmt v0.27.2 h1:SG32SlbwNy92s0KJiVxt2joJeFdqIYHvwrA0OU6HqzQ= +github.com/go-openapi/strfmt v0.27.2/go.mod h1:M4CKsMO0Fb8qR10+1Ra75wCKNNquy+Vj+4LWZrhTo2E= +github.com/go-openapi/swag v0.28.0 h1:xkgbOSKj6DZziNpyqRRAOt3GJGtgjgsd2RoyT30VWuw= +github.com/go-openapi/swag v0.28.0/go.mod h1:4qYnT3Cqr1p1VknOdPo70evN4rgQnAg6jwApHyxSGIg= +github.com/go-openapi/swag/cmdutils v0.28.0 h1:7TOeNtkYru1SG8Y34tDh9WBbLsMqGnptuxWiHREPZ4Q= +github.com/go-openapi/swag/cmdutils v0.28.0/go.mod h1:Sm1MVFMkF6guJJ+pQqHnQA3N0j9qALV3NxzDSv6bETM= +github.com/go-openapi/swag/conv v0.29.1 h1:AC4Eh/5c/eUDOUCzzsRC9ghmFgOSBHeRMGIngY0ZUGA= +github.com/go-openapi/swag/conv v0.29.1/go.mod h1:S1X7/ZrBEZOC0Wc8AGxjbcGS92l3WEjA7aPtpl+RaqM= +github.com/go-openapi/swag/fileutils v0.28.0 h1:Z04XWQD7R8Eq+7GnOrjovBxPPmZzsS4gt2H2GPGIViU= +github.com/go-openapi/swag/fileutils v0.28.0/go.mod h1:VvJFZLTZS0AI854gEQz5tk7dBESdLjiNUMSZ/th2ry8= +github.com/go-openapi/swag/jsonutils v0.29.1 h1:AFCxs0eQZ24/QyfhVHM2t49rMz7Vv3XCsZQI6yrNy+c= +github.com/go-openapi/swag/jsonutils v0.29.1/go.mod h1:u3+sCfJpttDpcmS5kpm0yxL6GK0eWgODsx8Yw8fcqNM= +github.com/go-openapi/swag/jsonutils/fixtures_test v0.29.1 h1:BiiXE31Bx9SfpsMmOQj5KYpUhTZBpLVriVhJDuLuY2o= +github.com/go-openapi/swag/jsonutils/fixtures_test v0.29.1/go.mod h1:julgTUKZ9/D0j6O7GKajmRs+812FWxQg/mMpGunWSjg= +github.com/go-openapi/swag/loading v0.29.1 h1:FCv5fG8UhTdDJa2R7w+5O9Ekpcbw7tt0nFWvmDKGBjc= +github.com/go-openapi/swag/loading v0.29.1/go.mod h1:N0ESuem4p2oedKal8EJhciqnJ9Q9Wmt83L1CRB3Fouw= +github.com/go-openapi/swag/mangling v0.29.1 h1:lHALtvYCdxVnRl4GrHmFPwfBTZYIObqdGNSKyu/8D6I= +github.com/go-openapi/swag/mangling v0.29.1/go.mod h1:SAop9pB7PUjQ/CGCNf/JmCKTRK+GDO+RqE9UHqC/N6s= +github.com/go-openapi/swag/netutils v0.28.0 h1:YXN6TALEi2pzts8/8GNm6T61HTAZsieukGZidap989k= +github.com/go-openapi/swag/netutils v0.28.0/go.mod h1:J+WYyFMLtvtCGqa6jLv+YNUmIKI3ZRQRrvfNDMoQoEQ= +github.com/go-openapi/swag/pools v0.29.1 h1:NRogYxdEW9SjRM4mkAOji9iefO4MRXq3p/ZJcoQbUKg= +github.com/go-openapi/swag/pools v0.29.1/go.mod h1:leDcaghjkRAhCuCRv9NfJU5f0mjoU3cT/XZObhMk3pc= +github.com/go-openapi/swag/stringutils v0.29.1 h1:1ykunK7iJQk1uOO7+oUH1ukbsK85fFCOiCFMOVSY+F0= +github.com/go-openapi/swag/stringutils v0.29.1/go.mod h1:7fSqZ+z8Qc0tOfAAK0jVa5qFGrnIlRi6n7NeGGrr1vc= +github.com/go-openapi/swag/typeutils v0.29.1 h1:Nzv9nhnlLCRBPQqfOX+7lB6Guju370or8StT+lIOf6M= +github.com/go-openapi/swag/typeutils v0.29.1/go.mod h1:hxpgDZJVBkBsi/d3MIUosafoFdE5exaQRmVp0zwu3YE= +github.com/go-openapi/swag/yamlutils v0.29.1 h1:69w3tsBajm7MR/fejLy7HD/3J68Ys1SeeZMEzZ3w2sk= +github.com/go-openapi/swag/yamlutils v0.29.1/go.mod h1:rgsp3vT/QdWzKwn43CigDwjOGIenPyTZMKnxEM8jZOA= +github.com/go-openapi/testify/enable/yaml/v2 v2.7.0 h1:wPW6YRgx3+SID1yUy/Xwa17L8kFEaEKod2VRbJDZNUs= +github.com/go-openapi/testify/enable/yaml/v2 v2.7.0/go.mod h1:mI1M88etYbc3PhgHsWQK2kwvNwW5aGFqMPbmib+SGIs= +github.com/go-openapi/testify/v2 v2.7.0 h1:bycOreEj6wfBvijg3YFogZ/sFjTCDmQnwSodSzHa3X8= +github.com/go-openapi/testify/v2 v2.7.0/go.mod h1:SgsVHtfooshd0tublTtJ50FPKhujf47YRqauXXOUxfw= +github.com/go-viper/mapstructure/v2 v2.5.0 h1:vM5IJoUAy3d7zRSVtIwQgBj7BiWtMPfmPEgAXnvj1Ro= +github.com/go-viper/mapstructure/v2 v2.5.0/go.mod h1:oJDH3BJKyqBA2TXFhDsKDGDTlndYOZ6rGS0BRZIxGhM= +github.com/gofrs/flock v0.13.1 h1:jjREztyBeSKBZYAC+mgc1laB+xsgy4kYMf3FbKF2UBo= +github.com/gofrs/flock v0.13.1/go.mod h1:sf4BFiHwnvgxa25DlQoDqXQnwRMEOwqxRq37P6MzzmE= +github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek= +github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps= +github.com/google/gnostic-models v0.6.9 h1:MU/8wDLif2qCXZmzncUQ/BOfxWfthHi63KqpoNbWqVw= +github.com/google/gnostic-models v0.6.9/go.mod h1:CiWsm0s6BSQd1hRn8/QmxqB6BesYcbSZxsz9b0KuDBw= +github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY= +github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8= +github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU= +github.com/google/go-github/v89 v89.0.0 h1:35bEK5XoEcF3PZrlVbl9XN63f5BcJRA/UGkxeC9xPg0= +github.com/google/go-github/v89 v89.0.0/go.mod h1:QLcbU0ipeAqQuR5KSg8c2lql4Qk1EwJ2dWz/0rP4Nho= +github.com/google/go-querystring v1.2.0 h1:yhqkPbu2/OH+V9BfpCVPZkNmUXhb2gBxJArfhIxNtP0= +github.com/google/go-querystring v1.2.0/go.mod h1:8IFJqpSRITyJ8QhQ13bmbeMBDfmeEJZD5A0egEOmkqU= +github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0= +github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= +github.com/grpc-ecosystem/grpc-gateway/v2 v2.30.0 h1:/Tnpcb2E0Pz/tN9s3bfEY2Q8ePCEX9iuS+cneUwncnw= +github.com/grpc-ecosystem/grpc-gateway/v2 v2.30.0/go.mod h1:zOBXOsUaBSjKgmH4OGzV1esUpR3oUSCPYVd2cUBjKYY= +github.com/hashicorp/errwrap v1.0.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4= +github.com/hashicorp/errwrap v1.1.0 h1:OxrOeh75EUXMY8TBjag2fzXGZ40LB6IKw45YeGUDY2I= +github.com/hashicorp/errwrap v1.1.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4= +github.com/hashicorp/go-multierror v1.1.1 h1:H5DkEtf6CXdFp0N0Em5UCwQpXMWke8IA0+lD48awMYo= +github.com/hashicorp/go-multierror v1.1.1/go.mod h1:iw975J/qwKPdAO1clOe2L8331t/9/fmwbPZ6JB6eMoM= +github.com/lufia/plan9stats v0.0.0-20260330125221-c963978e514e h1:Q6MvJtQK/iRcRtzAscm/zF23XxJlbECiGPyRicsX+Ak= +github.com/lufia/plan9stats v0.0.0-20260330125221-c963978e514e/go.mod h1:autxFIvghDt3jPTLoqZ9OZ7s9qTGNAWmYCjVFWPX/zg= +github.com/mattn/go-colorable v0.1.15 h1:+u9SLTRGnXv73cEsnsmoZBom+dMU88B2M0aDcWy0/jY= +github.com/mattn/go-colorable v0.1.15/go.mod h1:6LmQG8QLFO4G5z1gPvYEzlUgJ2wF+stgPZH1UqBm1s8= +github.com/mattn/go-isatty v0.0.24 h1:tGZZoVgT/KiqK1c8ocVLeDS8BSWMRd47J3Lbz7vsReI= +github.com/mattn/go-isatty v0.0.24/go.mod h1:nMCL3Zebbrt45jsMDgnfIwz6ydEQApk5oEI3HqDio6A= +github.com/mattn/go-runewidth v0.0.27 h1:Feg/Oou5zI/wnpgDF6omIU0OokC9GxLC/WRknhVlIR0= +github.com/mattn/go-runewidth v0.0.27/go.mod h1:3qAiGCV4Koz/yuveO58qUefmUTRm8r0IGEXZ9jeHp/8= +github.com/mitchellh/mapstructure v1.5.0 h1:jeMsZIYE/09sWLaz43PL7Gy6RuMjD2eJVyuac5Z2hdY= +github.com/mitchellh/mapstructure v1.5.0/go.mod h1:bFUtVrKA4DC2yAKiSyO/QUcy7e+RRV2QTWOzhPopBRo= +github.com/moby/docker-image-spec v1.3.1 h1:jMKff3w6PgbfSa69GfNg+zN/XLhfXJGnEx3Nl2EsFP0= +github.com/moby/docker-image-spec v1.3.1/go.mod h1:eKmb5VW8vQEh/BAr2yvVNvuiJuY6UIocYsFu/DxxRpo= +github.com/moby/patternmatcher v0.6.1 h1:qlhtafmr6kgMIJjKJMDmMWq7WLkKIo23hsrpR3x084U= +github.com/moby/patternmatcher v0.6.1/go.mod h1:hDPoyOpDY7OrrMDLaYoY3hf52gNCR/YOUYxkhApJIxc= +github.com/monochromegane/go-gitignore v0.0.0-20200626010858-205db1a8cc00 h1:n6/2gBQ3RWajuToeY6ZtZTIKv2v7ThUy5KKusIT0yc0= +github.com/monochromegane/go-gitignore v0.0.0-20200626010858-205db1a8cc00/go.mod h1:Pm3mSP3c5uWn86xMLZ5Sa7JB9GsEZySvHYXCTK4E9q4= +github.com/oklog/ulid/v2 v2.1.2 h1:IEclFb9JNvzYA6MW2SCxbLzcHTVsfqm3PrqGQJH5zec= +github.com/oklog/ulid/v2 v2.1.2/go.mod h1:rcEKHmBBKfef9DhnvX7y1HZBYxjXb0cP5ExxNsTT1QQ= +github.com/opencontainers/go-digest v1.0.0 h1:apOUWs51W5PlhuyGyz9FCeeBIOUDA/6nW8Oi/yOhh5U= +github.com/opencontainers/go-digest v1.0.0/go.mod h1:0JzlMkj0TRzQZfJkVvzbP0HBR3IKzErnv2BNG4W4MAM= +github.com/opencontainers/image-spec v1.1.1 h1:y0fUlFfIZhPF1W537XOLg0/fcx6zcHCJwooC2xJA040= +github.com/opencontainers/image-spec v1.1.1/go.mod h1:qpqAh3Dmcf36wStyyWU+kCeDgrGnAve2nCC8+7h8Q0M= +github.com/pborman/getopt v0.0.0-20170112200414-7148bc3a4c30/go.mod h1:85jBQOZwpVEaDAr341tbn15RS4fCAsIst0qp7i8ex1o= +github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4= +github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= +github.com/power-devops/perfstat v0.0.0-20240221224432-82ca36839d55 h1:o4JXh1EVt9k/+g42oCprj/FisM4qX9L3sZB3upGN2ZU= +github.com/power-devops/perfstat v0.0.0-20240221224432-82ca36839d55/go.mod h1:OmDBASR4679mdNQnz2pUhc2G8CO2JrUAVFDRBDP/hJE= +github.com/rodaine/protogofakeit v0.1.1 h1:ZKouljuRM3A+TArppfBqnH8tGZHOwM/pjvtXe9DaXH8= +github.com/rodaine/protogofakeit v0.1.1/go.mod h1:pXn/AstBYMaSfc1/RqH3N82pBuxtWgejz1AlYpY1mI0= +github.com/santhosh-tekuri/jsonschema/v6 v6.0.3 h1:1EYB5IzjZawrrnELUi78f9fPu57HuXjmddZPjrls/28= +github.com/santhosh-tekuri/jsonschema/v6 v6.0.3/go.mod h1:JXeL+ps8p7/KNMjDQk3TCwPpBy0wYklyWTfbkIzdIFU= +github.com/shirou/gopsutil/v3 v3.24.5 h1:i0t8kL+kQTvpAYToeuiVk3TgDeKOFioZO3Ztz/iZ9pI= +github.com/shirou/gopsutil/v3 v3.24.5/go.mod h1:bsoOS1aStSs9ErQ1WWfxllSeS1K5D+U30r2NfcubMVk= +github.com/shoenig/go-m1cpu v0.2.1 h1:yqRB4fvOge2+FyRXFkXqsyMoqPazv14Yyy+iyccT2E4= +github.com/shoenig/go-m1cpu v0.2.1/go.mod h1:KkDOw6m3ZJQAPHbrzkZki4hnx+pDRR1Lo+ldA56wD5w= +github.com/stretchr/testify v1.12.1 h1:EuwCh5fleGS7H32xRwO3wRGT7DxrDhLAT6FF8MpWDWE= +github.com/stretchr/testify v1.12.1/go.mod h1:MDEgiDPPsNp5cuIrHPPCyornHKgEVbtFUmoNlxoYthg= +github.com/tklauser/go-sysconf v0.4.0 h1:7H0uAN+7RkwWRaxhYXDLqa5V3LPrJeV8wmD9dRUgPQU= +github.com/tklauser/go-sysconf v0.4.0/go.mod h1:8mTNWyog7H+MpKijp4VmKJAd2bbYQ2zuUwkYRbUArPI= +github.com/tklauser/numcpus v0.12.0 h1:NR85qdvHA9pFse3x3weVZ0r0ST8R6l5RHbZrlRaqob4= +github.com/tklauser/numcpus v0.12.0/go.mod h1:ABHeXzJnr/qqwguhClkZKT1/8VABcYrsyUiUGobwWJg= +github.com/xlab/treeprint v1.2.0 h1:HzHnuAF1plUN2zGlAFHbSQP2qJ0ZAD3XF5XD7OesXRQ= +github.com/xlab/treeprint v1.2.0/go.mod h1:gj5Gd3gPdKtR1ikdDK6fnFLdmIS0X30kTTuNd/WEJu0= +github.com/yoheimuta/go-protoparser/v4 v4.14.2 h1:/P/LlX1CF9NaTWEltGcIZVvNlPbhABuAnBtAWpb3+74= +github.com/yoheimuta/go-protoparser/v4 v4.14.2/go.mod h1:AHNNnSWnb0UoL4QgHPiOAg2BniQceFscPI5X/BZNHl8= +github.com/yusufpapurcu/wmi v1.2.4 h1:zFUKzehAFReQwLys1b/iSMl+JQGSCSjtVqQn9bBrPo0= +github.com/yusufpapurcu/wmi v1.2.4/go.mod h1:SBZ9tNy3G9/m5Oi98Zks0QjeHVDvuK0qfxQmPyzfmi0= +go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ64= +go.opentelemetry.io/auto/sdk v1.2.1/go.mod h1:KRTj+aOaElaLi+wW1kO/DZRXwkF4C5xPbEe3ZiIhN7Y= +go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.71.0 h1:B2h3uqicet1CT2N5TOFhS+Gq++9i0/CLmaxvhmhtP5s= +go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.71.0/go.mod h1:dylvB+ZiiwMvsDij9O84Uy7SijLgHMX4mbkncds+4Sw= +go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.70.0 h1:LMuyCAyfalSjDyjdC65nK6N0zoTT63+E/u95X0JovZI= +go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.70.0/go.mod h1:085m8qbm4hgc8rZWGDEa4vmyyo2c3nPxUslYUKUIU04= +go.opentelemetry.io/otel v1.46.0 h1:FHt5/CDyVxi/8IM1CH7VE/rRgq3kLHa2mSTVMO8AWyc= +go.opentelemetry.io/otel v1.46.0/go.mod h1:Gj3SEScelsNC45tp4nSxRYlS+f5iez7W8XPMCt905kE= +go.opentelemetry.io/otel/metric v1.46.0 h1:yBnkXvgV7AXFILZc5K6IZe/CBFF3OS7BJ8ov6/lj0K8= +go.opentelemetry.io/otel/metric v1.46.0/go.mod h1:iPmdWqifKUdzziPkvvzIJXITl56fQx2mGM/DHLB3/2o= +go.opentelemetry.io/otel/sdk v1.46.0 h1:h5CNQQjEbuQXY/JfZtgt3i7HVFV3aHPO2OAwO2eTYPI= +go.opentelemetry.io/otel/sdk v1.46.0/go.mod h1:GAERFXFt5SYCEB+YiKUbMBeza6UaDH7GmGOZEfh2gSM= +go.opentelemetry.io/otel/sdk/metric v1.44.0 h1:3LlKgI+VjbVsjNRFZJZAJ30WjXC5VkNRks6si09iEfI= +go.opentelemetry.io/otel/sdk/metric v1.44.0/go.mod h1:5B5pMARnXxKhltooO4xUuCBorl65a4EpnTalObqOigA= +go.opentelemetry.io/otel/trace v1.46.0 h1:OULy7ccdJnZtJ0UDYFOIGaCmiWzJ8Vi2G/Rsu60qs1c= +go.opentelemetry.io/otel/trace v1.46.0/go.mod h1:J7GAXweO77XSFkB/rmAqk9D6ihszhFjLU+d9WuUxDLI= +go.opentelemetry.io/proto/otlp v1.11.0 h1:5rrYs0Ykyj50sdU/JU0x8etU+LubXWb+gED6TbEdMIk= +go.opentelemetry.io/proto/otlp v1.11.0/go.mod h1:SmVizdCOAm3XBtG1g1NnOdhW6jtddT72hLMhv8VwA8E= +go.yaml.in/yaml/v2 v2.4.2 h1:DzmwEr2rDGHl7lsFgAHxmNz/1NlQ7xLIrlN2h5d1eGI= +go.yaml.in/yaml/v2 v2.4.2/go.mod h1:081UH+NErpNdqlCXm3TtEran0rJZGxAYx9hb/ELlsPU= +go.yaml.in/yaml/v3 v3.0.5 h1:N6y/pJk8buWs9NY5ERU2HSMfm+IuD/OtfdAnq6kESPw= +go.yaml.in/yaml/v3 v3.0.5/go.mod h1:HVTZu1O7/Vkt2N+BFy8Zza+lnLsABggaTM2ZpNIGuKg= +golang.org/x/exp v0.0.0-20260824195058-e88cd73687aa h1:QSyA8ishJCyT21kER9KwNt0b7BM3iRK4x9QXhjN5Fdk= +golang.org/x/exp v0.0.0-20260824195058-e88cd73687aa/go.mod h1:zeBbvyFKDaLwa7CH/zI8KXt7gTl14SF7sO08Pl5jBCM= +golang.org/x/net v0.59.0 h1:5zfYln+w5XCxwrnMMJPufRgNoXEaGxl0wo5GqPXyues= +golang.org/x/net v0.59.0/go.mod h1:2DA/G1UfVbCpQPeWTmMPGY7Cs2PkBkwu743bVX5PIVg= +golang.org/x/sys v0.48.0 h1:bbX/i/6MgT9BVLM9RT1thmxL04yeTAhbEz4SyadbXoo= +golang.org/x/sys v0.48.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og= +golang.org/x/term v0.46.0 h1:3+OXuTbaKDgwk8jTi3aSLHRlmWqHEUDUtxnbFigO4YE= +golang.org/x/term v0.46.0/go.mod h1:+K02xbkittuwc0Am4abfA3Fc+XRGXkvBXNO88NCXPoc= +golang.org/x/text v0.42.0 h1:JbOZXgfeCPU9gacVtYliJqOhD+zhrEqK4LfdpmlUZqI= +golang.org/x/text v0.42.0/go.mod h1:ojzP1Z+2QtioaF8DTtO8K5q7JWVVYwZKenzujK0Zd0E= +gonum.org/v1/gonum v0.17.0 h1:VbpOemQlsSMrYmn7T2OUvQ4dqxQXU+ouZFQsZOx50z4= +gonum.org/v1/gonum v0.17.0/go.mod h1:El3tOrEuMpv2UdMrbNlKEh9vd86bmQ6vqIcDwxEOc1E= +google.golang.org/genproto/googleapis/api v0.0.0-20260831171406-18b4a7587f8a h1:i3TAXhpKc7TUP1VAPiBBrv45kamjoizCC3rOC0cAbOs= +google.golang.org/genproto/googleapis/api v0.0.0-20260831171406-18b4a7587f8a/go.mod h1:CvYJHpbzPlT0fb/PsgtAamdwru/GVxUsomFdXTpOTI8= +google.golang.org/genproto/googleapis/rpc v0.0.0-20260831171406-18b4a7587f8a h1:3Dnd1cDaZlB68lziofO+bJXpjOy8UfRv8Unt+yH8tQ4= +google.golang.org/genproto/googleapis/rpc v0.0.0-20260831171406-18b4a7587f8a/go.mod h1:DjtHYE8FKJLivXcBEjGwndXfIC23G0VpXiXKqG179uA= +google.golang.org/grpc v1.83.2 h1:EManeRomTObA0BU7I8vXgg/78uE5MJ9M8B39EX2WscU= +google.golang.org/grpc v1.83.2/go.mod h1:YPI1hK3kDked6iHvgX3tR0y+nX/qpMFKhPgFsokw1S8= +google.golang.org/protobuf v1.36.12 h1:pJOKDDOyeXErUroCihFAd5LQuwXBSpVnKGrj5o/fwxc= +google.golang.org/protobuf v1.36.12/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco= +gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM= +gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/evanphx/json-patch.v4 v4.13.0 h1:czT3CmqEaQ1aanPc5SdlgQrrEIb8w/wwCvWWnfEbYzo= +gopkg.in/evanphx/json-patch.v4 v4.13.0/go.mod h1:p8EYWUEYMpynmqDbY58zCKCFZw8pRWMG4EsWvDvM72M= +gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= +gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= +k8s.io/kube-openapi v0.0.0-20241212222426-2c72e554b1e7 h1:hcha5B1kVACrLujCKLbr8XWMxCxzQx42DY8QKYJrDLg= +k8s.io/kube-openapi v0.0.0-20241212222426-2c72e554b1e7/go.mod h1:GewRfANuJ70iYzvn+i4lezLDAFzvjxZYK1gn1lWcfas= +sigs.k8s.io/kustomize/api v0.21.1 h1:lzqbzvz2CSvsjIUZUBNFKtIMsEw7hVLJp0JeSIVmuJs= +sigs.k8s.io/kustomize/api v0.21.1/go.mod h1:f3wkKByTrgpgltLgySCntrYoq5d3q7aaxveSagwTlwI= +sigs.k8s.io/kustomize/kyaml v0.21.1 h1:IVlbmhC076nf6foyL6Taw4BkrLuEsXUXNpsE+ScX7fI= +sigs.k8s.io/kustomize/kyaml v0.21.1/go.mod h1:hmxADesM3yUN2vbA5z1/YTBnzLJ1dajdqpQonwBL1FQ= +sigs.k8s.io/yaml v1.5.0 h1:M10b2U7aEUY6hRtU870n2VTPgR5RZiL/I6Lcc2F4NUQ= +sigs.k8s.io/yaml v1.5.0/go.mod h1:wZs27Rbxoai4C0f8/9urLZtZtF3avA3gKvGyPdDqTO4= diff --git a/module/services/telemetry/code/main.go b/module/services/telemetry/code/main.go new file mode 100644 index 000000000..f4d0ee527 --- /dev/null +++ b/module/services/telemetry/code/main.go @@ -0,0 +1,128 @@ +package main + +import ( + "context" + "fmt" + "log" + "net" + "os" + "os/signal" + "strconv" + "syscall" + + "telemetry/collector" + + codefly "github.com/codefly-dev/sdk-go" + collectorlogsv1 "go.opentelemetry.io/proto/otlp/collector/logs/v1" + collectormetricsv1 "go.opentelemetry.io/proto/otlp/collector/metrics/v1" + collectortracev1 "go.opentelemetry.io/proto/otlp/collector/trace/v1" + "google.golang.org/grpc" +) + +// observabilityConfiguration is the workspace configuration group that carries +// the collector's upstream-exporter settings. Values arrive from either half of +// the group — configurations//observability.env or its .secret.env +// sibling, which is where OTEL_EXPORTER_OTLP_HEADERS lives — because +// WorkspaceValue consults the public namespace first and then the secret one. +const observabilityConfiguration = "observability" + +// exporterKeys are the three settings that make up one exporter configuration. +// They are resolved together, never individually; see resolveExporterConfig. +var exporterKeys = struct{ exporter, endpoint, headers string }{ + exporter: "OBSERVABILITY_EXPORTER", + endpoint: "OTEL_EXPORTER_OTLP_ENDPOINT", + headers: "OTEL_EXPORTER_OTLP_HEADERS", +} + +// workspaceExporterConfig reads all three settings from the observability +// workspace configuration group, reporting whether the group delivered any of +// them. +// +// Reading the group is load-bearing, not cosmetic. telemetry declares +// `observability` as a workspace-configuration-dependency, and a declared group +// reaches a deployed pod ONLY under prefixed names — +// CODEFLY__WORKSPACE_CONFIGURATION__OBSERVABILITY__, via envFrom a +// ConfigMap — while codefly.Init builds an in-process map without ever mutating +// the process environment. A bare os.Getenv("OBSERVABILITY_EXPORTER") therefore +// resolved to "" wherever the configuration arrives that way; collector.New +// defaulted that empty value to "debug", and the collector logged and dropped +// every span it received while its own ConfigMap said otlphttp. +func workspaceExporterConfig(ctx context.Context) (collector.Config, bool) { + value := func(key string) string { + resolved, err := codefly.For(ctx).WorkspaceValue(observabilityConfiguration, key) + if err != nil { + return "" + } + return resolved + } + config := collector.Config{ + Exporter: value(exporterKeys.exporter), + Endpoint: value(exporterKeys.endpoint), + Headers: value(exporterKeys.headers), + } + return config, config.Exporter != "" || config.Endpoint != "" || config.Headers != "" +} + +// resolveExporterConfig picks ONE authority for the whole exporter +// configuration: the workspace group if it delivered anything at all, otherwise +// the bare process environment, so a local or non-Codefly run still works. +// +// The choice is deliberately made for the tuple, not per key. collector.New +// cross-validates these three values (debug rejects an endpoint; otlphttp +// requires one), so mixing sources produces configurations no operator wrote. +// Resolving key by key meant a developer who exported OBSERVABILITY_EXPORTER +// and OTEL_EXPORTER_OTLP_ENDPOINT in their shell could no +// longer start the collector: the committed local group supplied +// OBSERVABILITY_EXPORTER=debug while its empty endpoint fell through to the +// shell's, and debug+endpoint is refused. The same split let an ambient +// OTEL_EXPORTER_OTLP_HEADERS — the credential, and a name the OpenTelemetry +// spec makes ubiquitous — attach itself to a workspace-configured endpoint it +// was never issued for, whenever the observability.secret.env half was not +// provisioned. One authority for all three makes both states unrepresentable. +func resolveExporterConfig(ctx context.Context) collector.Config { + if config, delivered := workspaceExporterConfig(ctx); delivered { + return config + } + return collector.Config{ + Exporter: os.Getenv(exporterKeys.exporter), + Endpoint: os.Getenv(exporterKeys.endpoint), + Headers: os.Getenv(exporterKeys.headers), + } +} + +func main() { + ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM) + defer stop() + provider, err := codefly.Init(ctx) + if err != nil { + log.Fatal(err) + } + ctx = provider.Inject(ctx) + defer codefly.CatchPanic(ctx) + + port := codefly.For(ctx).WithDefaultNetwork().API("grpc").NetworkInstance().Port + if port == 0 { + log.Fatal("telemetry: Codefly did not inject the collector gRPC port") + } + sink, err := collector.New(resolveExporterConfig(ctx)) + if err != nil { + log.Fatal(err) + } + listener, err := net.Listen("tcp", net.JoinHostPort("", strconv.Itoa(int(port)))) + if err != nil { + log.Fatal(err) + } + server := grpc.NewServer() + collectortracev1.RegisterTraceServiceServer(server, sink.TraceService()) + collectormetricsv1.RegisterMetricsServiceServer(server, sink.MetricsService()) + collectorlogsv1.RegisterLogsServiceServer(server, sink.LogsService()) + go func() { + if err := server.Serve(listener); err != nil { + log.Printf("telemetry: serve: %v", err) + stop() + } + }() + fmt.Printf("OpenTelemetry collector listening on Codefly gRPC endpoint %d\n", port) + <-ctx.Done() + server.GracefulStop() +} diff --git a/module/services/telemetry/code/main_test.go b/module/services/telemetry/code/main_test.go new file mode 100644 index 000000000..d7002508f --- /dev/null +++ b/module/services/telemetry/code/main_test.go @@ -0,0 +1,98 @@ +package main + +import ( + "context" + "testing" + + "telemetry/collector" + + codefly "github.com/codefly-dev/sdk-go" + "github.com/stretchr/testify/require" +) + +// clearWorkspaceObservability removes every carrier the observability group can +// arrive under, so a test that means "no workspace configuration" actually gets +// it. Without this a test passes only because a neighbouring test happened to +// restore these first, and fails wherever they are genuinely exported — which is +// any process started by `codefly run`. +func clearWorkspaceObservability(t *testing.T) { + t.Helper() + for _, prefix := range []string{ + "CODEFLY__WORKSPACE_CONFIGURATION__OBSERVABILITY__", + "CODEFLY__WORKSPACE_SECRET_CONFIGURATION__OBSERVABILITY__", + } { + for _, key := range []string{ + "OBSERVABILITY_EXPORTER", + "OTEL_EXPORTER_OTLP_ENDPOINT", + "OTEL_EXPORTER_OTLP_HEADERS", + } { + t.Setenv(prefix+key, "") + } + } +} + +// TestResolveExporterConfigPrefersWorkspaceValue is the regression test for the +// boundary that broke in deployment: workspace configuration reaches the pod +// under prefixed names only, so reading the bare name returned "" and the +// collector silently downgraded itself to the debug exporter. +func TestResolveExporterConfigPrefersWorkspaceValue(t *testing.T) { + clearWorkspaceObservability(t) + t.Setenv("CODEFLY__WORKSPACE_CONFIGURATION__OBSERVABILITY__OBSERVABILITY_EXPORTER", "otlphttp") + t.Setenv("CODEFLY__WORKSPACE_CONFIGURATION__OBSERVABILITY__OTEL_EXPORTER_OTLP_ENDPOINT", + "https://collector.example") + t.Setenv("CODEFLY__WORKSPACE_SECRET_CONFIGURATION__OBSERVABILITY__OTEL_EXPORTER_OTLP_HEADERS", + "Authorization=Bearer+workspace") + // The bare names are what main.go used to read; they must lose to the + // prefixed workspace values above — including the header, which is the value + // that carries the credential. + t.Setenv("OBSERVABILITY_EXPORTER", "debug") + t.Setenv("OTEL_EXPORTER_OTLP_ENDPOINT", "https://wrong-backend.example") + t.Setenv("OTEL_EXPORTER_OTLP_HEADERS", "Authorization=Bearer+ambient") + require.NoError(t, codefly.LoadEnvironmentVariables()) + + config := resolveExporterConfig(context.Background()) + require.Equal(t, "otlphttp", config.Exporter) + require.Equal(t, "https://collector.example", config.Endpoint) + require.Equal(t, "Authorization=Bearer+workspace", config.Headers) +} + +// TestResolveExporterConfigTakesWholeTupleFromOneAuthority is the regression +// test for per-key resolution. The workspace group selects debug and says +// nothing about an endpoint or headers; the process environment carries both. +// Resolving key by key produced debug+endpoint — a combination collector.New +// refuses — so a developer who had exported these in their shell could no +// longer start the collector at all. It also +// stopped an ambient OTEL_EXPORTER_OTLP_HEADERS credential from being attached +// to an endpoint it was never issued for. +func TestResolveExporterConfigTakesWholeTupleFromOneAuthority(t *testing.T) { + clearWorkspaceObservability(t) + t.Setenv("CODEFLY__WORKSPACE_CONFIGURATION__OBSERVABILITY__OBSERVABILITY_EXPORTER", "debug") + t.Setenv("OTEL_EXPORTER_OTLP_ENDPOINT", "https://shell-backend.example") + t.Setenv("OTEL_EXPORTER_OTLP_HEADERS", "Authorization=Bearer+ambient") + require.NoError(t, codefly.LoadEnvironmentVariables()) + + config := resolveExporterConfig(context.Background()) + require.Equal(t, "debug", config.Exporter) + require.Empty(t, config.Endpoint, "endpoint must not be mixed in from the process environment") + require.Empty(t, config.Headers, "an ambient credential must not reach a workspace-selected exporter") + + // The consequence the mixing produced: the collector refused to start. + _, err := collector.New(config) + require.NoError(t, err) +} + +// TestResolveExporterConfigFallsBackToProcessEnvironment keeps a local or +// non-Codefly run working: with no workspace configuration injected, the bare +// process variables are still honoured — as a complete tuple. +func TestResolveExporterConfigFallsBackToProcessEnvironment(t *testing.T) { + clearWorkspaceObservability(t) + t.Setenv("OBSERVABILITY_EXPORTER", "otlphttp") + t.Setenv("OTEL_EXPORTER_OTLP_ENDPOINT", "https://collector.example") + t.Setenv("OTEL_EXPORTER_OTLP_HEADERS", "Authorization=Bearer+ambient") + require.NoError(t, codefly.LoadEnvironmentVariables()) + + config := resolveExporterConfig(context.Background()) + require.Equal(t, "otlphttp", config.Exporter) + require.Equal(t, "https://collector.example", config.Endpoint) + require.Equal(t, "Authorization=Bearer+ambient", config.Headers) +} diff --git a/module/services/telemetry/code/pkg/gen/saas/telemetry/v1/schema_marker.pb.go b/module/services/telemetry/code/pkg/gen/saas/telemetry/v1/schema_marker.pb.go new file mode 100644 index 000000000..6821690f7 --- /dev/null +++ b/module/services/telemetry/code/pkg/gen/saas/telemetry/v1/schema_marker.pb.go @@ -0,0 +1,117 @@ +// Code generated by protoc-gen-go. DO NOT EDIT. +// versions: +// protoc-gen-go v1.36.11 +// protoc (unknown) +// source: saas/telemetry/v1/schema_marker.proto + +package telemetryv1 + +import ( + reflect "reflect" + sync "sync" + unsafe "unsafe" + + protoreflect "google.golang.org/protobuf/reflect/protoreflect" + protoimpl "google.golang.org/protobuf/runtime/protoimpl" +) + +const ( + // Verify that this generated code is sufficiently up-to-date. + _ = protoimpl.EnforceVersion(20 - protoimpl.MinVersion) + // Verify that runtime/protoimpl is sufficiently up-to-date. + _ = protoimpl.EnforceVersion(protoimpl.MaxVersion - 20) +) + +// SchemaMarker keeps the Codefly protobuf sync input valid without duplicating +// the standard OpenTelemetry Protocol service definitions. +type SchemaMarker struct { + state protoimpl.MessageState `protogen:"open.v1"` + unknownFields protoimpl.UnknownFields + sizeCache protoimpl.SizeCache +} + +func (x *SchemaMarker) Reset() { + *x = SchemaMarker{} + mi := &file_saas_telemetry_v1_schema_marker_proto_msgTypes[0] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) +} + +func (x *SchemaMarker) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*SchemaMarker) ProtoMessage() {} + +func (x *SchemaMarker) ProtoReflect() protoreflect.Message { + mi := &file_saas_telemetry_v1_schema_marker_proto_msgTypes[0] + if x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use SchemaMarker.ProtoReflect.Descriptor instead. +func (*SchemaMarker) Descriptor() ([]byte, []int) { + return file_saas_telemetry_v1_schema_marker_proto_rawDescGZIP(), []int{0} +} + +var File_saas_telemetry_v1_schema_marker_proto protoreflect.FileDescriptor + +const file_saas_telemetry_v1_schema_marker_proto_rawDesc = "" + + "\n" + + "%saas/telemetry/v1/schema_marker.proto\x12\x11saas.telemetry.v1\"\x0e\n" + + "\fSchemaMarkerB\xc1\x01\n" + + "\x15com.saas.telemetry.v1B\x11SchemaMarkerProtoP\x01Z/telemetry/pkg/gen/saas/telemetry/v1;telemetryv1\xa2\x02\x03STX\xaa\x02\x11Saas.Telemetry.V1\xca\x02\x11Saas\\Telemetry\\V1\xe2\x02\x1dSaas\\Telemetry\\V1\\GPBMetadata\xea\x02\x13Saas::Telemetry::V1b\x06proto3" + +var ( + file_saas_telemetry_v1_schema_marker_proto_rawDescOnce sync.Once + file_saas_telemetry_v1_schema_marker_proto_rawDescData []byte +) + +func file_saas_telemetry_v1_schema_marker_proto_rawDescGZIP() []byte { + file_saas_telemetry_v1_schema_marker_proto_rawDescOnce.Do(func() { + file_saas_telemetry_v1_schema_marker_proto_rawDescData = protoimpl.X.CompressGZIP(unsafe.Slice(unsafe.StringData(file_saas_telemetry_v1_schema_marker_proto_rawDesc), len(file_saas_telemetry_v1_schema_marker_proto_rawDesc))) + }) + return file_saas_telemetry_v1_schema_marker_proto_rawDescData +} + +var file_saas_telemetry_v1_schema_marker_proto_msgTypes = make([]protoimpl.MessageInfo, 1) +var file_saas_telemetry_v1_schema_marker_proto_goTypes = []any{ + (*SchemaMarker)(nil), // 0: saas.telemetry.v1.SchemaMarker +} +var file_saas_telemetry_v1_schema_marker_proto_depIdxs = []int32{ + 0, // [0:0] is the sub-list for method output_type + 0, // [0:0] is the sub-list for method input_type + 0, // [0:0] is the sub-list for extension type_name + 0, // [0:0] is the sub-list for extension extendee + 0, // [0:0] is the sub-list for field type_name +} + +func init() { file_saas_telemetry_v1_schema_marker_proto_init() } +func file_saas_telemetry_v1_schema_marker_proto_init() { + if File_saas_telemetry_v1_schema_marker_proto != nil { + return + } + type x struct{} + out := protoimpl.TypeBuilder{ + File: protoimpl.DescBuilder{ + GoPackagePath: reflect.TypeOf(x{}).PkgPath(), + RawDescriptor: unsafe.Slice(unsafe.StringData(file_saas_telemetry_v1_schema_marker_proto_rawDesc), len(file_saas_telemetry_v1_schema_marker_proto_rawDesc)), + NumEnums: 0, + NumMessages: 1, + NumExtensions: 0, + NumServices: 0, + }, + GoTypes: file_saas_telemetry_v1_schema_marker_proto_goTypes, + DependencyIndexes: file_saas_telemetry_v1_schema_marker_proto_depIdxs, + MessageInfos: file_saas_telemetry_v1_schema_marker_proto_msgTypes, + }.Build() + File_saas_telemetry_v1_schema_marker_proto = out.File + file_saas_telemetry_v1_schema_marker_proto_goTypes = nil + file_saas_telemetry_v1_schema_marker_proto_depIdxs = nil +} diff --git a/module/services/telemetry/proto/README.md b/module/services/telemetry/proto/README.md new file mode 100644 index 000000000..e70d7c6dc --- /dev/null +++ b/module/services/telemetry/proto/README.md @@ -0,0 +1,9 @@ +# Telemetry protobuf input + +The telemetry gateway implements the standard OpenTelemetry Protocol services +from `go.opentelemetry.io/proto/otlp`; it does not duplicate those upstream +service definitions. + +The local `SchemaMarker` is deliberately code-generation-free. It keeps the +Codefly Go gRPC agent's required protobuf sync input valid for a service that +depends exclusively on upstream wire schemas. diff --git a/module/services/telemetry/proto/buf.gen.yaml b/module/services/telemetry/proto/buf.gen.yaml new file mode 100644 index 000000000..63afd5b44 --- /dev/null +++ b/module/services/telemetry/proto/buf.gen.yaml @@ -0,0 +1,11 @@ +version: v2 +clean: true +managed: + enabled: true + override: + - file_option: go_package_prefix + value: telemetry/pkg/gen +plugins: + - local: protoc-gen-go + out: ../code/pkg/gen + opt: paths=source_relative diff --git a/module/services/telemetry/proto/buf.yaml b/module/services/telemetry/proto/buf.yaml new file mode 100644 index 000000000..cb3b14b9f --- /dev/null +++ b/module/services/telemetry/proto/buf.yaml @@ -0,0 +1,6 @@ +version: v2 +modules: + - path: . +lint: + use: + - STANDARD diff --git a/module/services/telemetry/proto/saas/telemetry/v1/schema_marker.proto b/module/services/telemetry/proto/saas/telemetry/v1/schema_marker.proto new file mode 100644 index 000000000..ad48f2919 --- /dev/null +++ b/module/services/telemetry/proto/saas/telemetry/v1/schema_marker.proto @@ -0,0 +1,7 @@ +syntax = "proto3"; + +package saas.telemetry.v1; + +// SchemaMarker keeps the Codefly protobuf sync input valid without duplicating +// the standard OpenTelemetry Protocol service definitions. +message SchemaMarker {} diff --git a/module/services/telemetry/service.codefly.yaml b/module/services/telemetry/service.codefly.yaml new file mode 100644 index 000000000..b4e307607 --- /dev/null +++ b/module/services/telemetry/service.codefly.yaml @@ -0,0 +1,26 @@ +name: telemetry +version: 0.0.0 +description: | + In-graph OTLP gateway. Accounts resolves this endpoint exclusively + through the Codefly SDK; the collector either logs locally or forwards + trace, metric, and log protobufs to a configured OTLP/HTTP destination. +agent: + kind: codefly:service + name: go-grpc + version: 0.1.50 + publisher: codefly.dev +workspace-configuration-dependencies: + - observability +endpoints: + - name: grpc +spec: + hot-reload: true + protocol-output-dirs: + - code/pkg/gen + deployment: + # No endpoint-ports: go-grpc binds every endpoint on the port Codefly + # allocates for it, so each pod port is its deployed port, which the + # render takes from Codefly. endpoint-ports is only for a port the + # process binds at a fixed number the allocation does not give it. + public-egress-ports: + - 443 diff --git a/module/tools/composition/composition_test.go b/module/tools/composition/composition_test.go index 7f99bf294..cf7e16a9f 100644 --- a/module/tools/composition/composition_test.go +++ b/module/tools/composition/composition_test.go @@ -329,18 +329,11 @@ permissions: func writeCoreInput(t *testing.T, fixture *coreCompositionFixture) { t.Helper() - // The input names the package release it was made for, which the generator - // holds to the package it runs inside: the version is the manifest's, not a - // literal that every release would have to chase. - manifest, err := corecomposition.LoadPackageManifest(findModuleRoot(t)) - if err != nil { - t.Fatalf("load package manifest: %v", err) - } input := corecomposition.CompositionInput{ Schema: "codefly/composition-input/v2", Module: fixture.descriptor.Name, Package: "codefly/saas-starter", - Version: manifest.Version, + Version: "0.1.0", ConsumerRoot: fixture.consumerRoot, Projection: fixture.projection, Contracts: map[string]string{ diff --git a/module/tools/modulepackage/modulepackage_test.go b/module/tools/modulepackage/modulepackage_test.go index 6ffb21567..a76274bb0 100644 --- a/module/tools/modulepackage/modulepackage_test.go +++ b/module/tools/modulepackage/modulepackage_test.go @@ -23,7 +23,7 @@ func TestStarterManifestIsTheCoreV2Contract(t *testing.T) { if err != nil { t.Fatal(err) } - if manifest.ID != "codefly/saas-starter" || manifest.Version != "0.2.0" { + if manifest.ID != "codefly/saas-starter" || manifest.Version != "0.1.0" { t.Fatalf("unexpected package identity: %s@%s", manifest.ID, manifest.Version) } // THE SERVICE SET IS DERIVED, NOT COUNTED. This pinned the number 8, and a diff --git a/module/tools/telemetry_destination_mirror_test.go b/module/tools/telemetry_destination_mirror_test.go deleted file mode 100644 index e7310335e..000000000 --- a/module/tools/telemetry_destination_mirror_test.go +++ /dev/null @@ -1,40 +0,0 @@ -package tools - -import ( - "os" - "path/filepath" - "testing" -) - -// telemetryDestinationCopies are source files that exist in both Go services, -// keyed by the copy and valued by the file it must match. -// -// accounts and auth-gateway both resolve where the cell's collector is from the -// `observability` configuration group: the same three keys, the same three -// outcomes, the same refusal to start on anything else. They are separate Go -// modules with no go.work, so neither can import the other, and a shared module -// would be a third thing to version for one small decision. The copy is the -// price; this test keeps the price from being paid twice, because two producers -// that read the group differently would let one service export while the other -// refuses to start on the same configuration. -var telemetryDestinationCopies = map[string]string{ - "services/auth-gateway/code/telemetry_destination.go": "services/accounts/code/telemetry_destination.go", -} - -func TestTelemetryDestinationCopiesMatchTheirSource(t *testing.T) { - moduleDir := findModuleDir(t) - for copyPath, sourcePath := range telemetryDestinationCopies { - copyBody, err := os.ReadFile(filepath.Join(moduleDir, filepath.FromSlash(copyPath))) - if err != nil { - t.Fatalf("read telemetry destination copy %s: %v", copyPath, err) - } - sourceBody, err := os.ReadFile(filepath.Join(moduleDir, filepath.FromSlash(sourcePath))) - if err != nil { - t.Fatalf("read telemetry destination source %s: %v", sourcePath, err) - } - if string(copyBody) != string(sourceBody) { - t.Errorf("%s has diverged from %s; change the source and copy it over the other so both services resolve the collector the same way", - copyPath, sourcePath) - } - } -} From ded6edaf1113a5bc635a8146b395275b42987cc3 Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Thu, 8 Oct 2026 13:51:40 -0400 Subject: [PATCH 67/69] fix(audit): renumber the audit migrations after main took migration 30 (Refs #996) Main merged per-organization envelope keys with its own migration 30 (`org_key_bindings`), so the audit queue's 30 collided with it and the reference gate refused the branch ("new version must exceed reference frontier 30"). The four audit migrations move up by one, 31 through 34, keeping their order and their contents; nothing has shipped, so nothing has applied them under the old numbers. The rollback and quarantine tests, the ADR, the authority inventory, the runbook and the accounts guide name the migrations by number, and now name the new ones. Co-Authored-By: Claude Opus 5.5 --- module/DATABASE_AUTHORITY.md | 2 +- module/MEASUREMENT_RUNBOOKS.md | 4 ++-- module/docs/adr/0009-audit-store-swap.md | 10 +++++----- module/services/accounts/AGENTS.md | 4 ++-- .../code/pkg/infra/postgres_audit_quarantine_test.go | 2 +- .../code/pkg/infra/postgres_audit_rollback_test.go | 8 ++++---- ...nt_queue.down.sql => 31_audit_event_queue.down.sql} | 0 ..._event_queue.up.sql => 31_audit_event_queue.up.sql} | 0 ...ql => 32_audit_event_type_retention_class.down.sql} | 0 ....sql => 32_audit_event_type_retention_class.up.sql} | 0 ...n.sql => 33_audit_verified_partition_drop.down.sql} | 2 +- ....up.sql => 33_audit_verified_partition_drop.up.sql} | 0 ...ine.down.sql => 34_audit_event_quarantine.down.sql} | 0 ...rantine.up.sql => 34_audit_event_quarantine.up.sql} | 0 14 files changed, 16 insertions(+), 16 deletions(-) rename module/services/store/migrations/{30_audit_event_queue.down.sql => 31_audit_event_queue.down.sql} (100%) rename module/services/store/migrations/{30_audit_event_queue.up.sql => 31_audit_event_queue.up.sql} (100%) rename module/services/store/migrations/{31_audit_event_type_retention_class.down.sql => 32_audit_event_type_retention_class.down.sql} (100%) rename module/services/store/migrations/{31_audit_event_type_retention_class.up.sql => 32_audit_event_type_retention_class.up.sql} (100%) rename module/services/store/migrations/{32_audit_verified_partition_drop.down.sql => 33_audit_verified_partition_drop.down.sql} (72%) rename module/services/store/migrations/{32_audit_verified_partition_drop.up.sql => 33_audit_verified_partition_drop.up.sql} (100%) rename module/services/store/migrations/{33_audit_event_quarantine.down.sql => 34_audit_event_quarantine.down.sql} (100%) rename module/services/store/migrations/{33_audit_event_quarantine.up.sql => 34_audit_event_quarantine.up.sql} (100%) diff --git a/module/DATABASE_AUTHORITY.md b/module/DATABASE_AUTHORITY.md index 1470bf912..12862cbf9 100644 --- a/module/DATABASE_AUTHORITY.md +++ b/module/DATABASE_AUTHORITY.md @@ -602,7 +602,7 @@ four hours are September's events, unverified. It also drops without looking: a row committed into a partition after the copy's last recount, before the `DROP`, was destroyed with the table. -Migration `32_audit_verified_partition_drop` adds +Migration `33_audit_verified_partition_drop` adds `audit_events_drop_verified_partitions(cutoff, names, expected_rows)`, which takes the explicit list of verified partition names with the row count each held when verified, and in one transaction: diff --git a/module/MEASUREMENT_RUNBOOKS.md b/module/MEASUREMENT_RUNBOOKS.md index 5483f8aa0..c3d70f6f8 100644 --- a/module/MEASUREMENT_RUNBOOKS.md +++ b/module/MEASUREMENT_RUNBOOKS.md @@ -142,8 +142,8 @@ an error with the number of queued events, and the depth and age series keep reporting them. To roll back for good, first restore the warehouse sink and let the relay drain the queue to zero. -Migration 20's down refuses (`audit_event_queue still holds N queued audit -events`) while any row is queued, and migration 33's down refuses while any row +Migration 31's down refuses (`audit_event_queue still holds N queued audit +events`) while any row is queued, and migration 34's down refuses while any row is quarantined: rolling the schema back would drop the only copy of those events. Do not work around the refusal by deleting rows. diff --git a/module/docs/adr/0009-audit-store-swap.md b/module/docs/adr/0009-audit-store-swap.md index 0752b089a..4b97c700a 100644 --- a/module/docs/adr/0009-audit-store-swap.md +++ b/module/docs/adr/0009-audit-store-swap.md @@ -365,7 +365,7 @@ Postgres transaction. per-row hashes. Once verified, the copied monthly partitions are removed with `DROP TABLE`, as `RunRetention` drops expired ones, so no row is deleted and the append-only triggers stay as they are. The removal is one - database function (migration 32) that takes an explicit list of the + database function (migration 33) that takes an explicit list of the verified partitions, locks them, re-counts each against the count that was verified, and refuses on any mismatch — a partition that gained a row since verification is never dropped. `-through`, the cutoff of the copy, accepts @@ -472,13 +472,13 @@ The sink shape itself: as well, through its own verified-partition function. Under a swap value, expiry belongs to the warehouse where it sets one — the details table on both, the events table on ClickHouse only — and to the archive's lock. -- `store/migrations` — migration 30 adds the queue table, under the same tenant +- `store/migrations` — migration 31 adds the queue table, under the same tenant row-level security as `audit_events` (`audit_events_tenant`, in `1_baseline.up.sql`), with insert granted to the writers and select and delete to the relay; its down migration refuses to drop a non-empty queue, since the - rows in it are audit records no warehouse holds yet. Migration 21 adds the - retention class to `audit_event_types`. Migration 22 adds the - verified-partition removal function of item 7. Migration 23 adds the + rows in it are audit records no warehouse holds yet. Migration 32 adds the + retention class to `audit_event_types`. Migration 33 adds the + verified-partition removal function of item 7. Migration 34 adds the `audit_event_quarantine` table of item 2, which only the relay's role can read or insert into; its down migration refuses to drop a non-empty quarantine. `audit_events` itself is untouched: its `audit_events_no_delete` and diff --git a/module/services/accounts/AGENTS.md b/module/services/accounts/AGENTS.md index 571b90309..e24412fc1 100644 --- a/module/services/accounts/AGENTS.md +++ b/module/services/accounts/AGENTS.md @@ -843,7 +843,7 @@ reconnected with a PAT or through the App, all reach. `pkg/business/audit_store_parity_test.go` holds every store to one fixture (ClickHouse joins when `AUDIT_CLICKHOUSE_TEST_DSN` names a server). - **Audit relay and queue** (ADR 0009, `pkg/business/audit_relay.go`, - `pkg/infra/postgres_audit_queue.go`, migrations 30 and 33). Under a swap value + `pkg/infra/postgres_audit_queue.go`, migrations 31 and 34). Under a swap value `EmitTx` writes a short-lived `audit_event_queue` row on the caller's transaction; the relay drains it to the archive and then the warehouse and deletes the row only after both acknowledged. It is at-least-once, so every @@ -886,7 +886,7 @@ reconnected with a PAT or through the App, all reach. refuses for good fails verification alone, as the relay sets one row aside. Only with `-confirm-drop`, after a verification pass in the same run and a recount showing nothing changed, does it drop the copied partitions, named one by one through - `audit_events_drop_verified_partitions` (migration 32), which recounts each + `audit_events_drop_verified_partitions` (migration 33), which recounts each under a lock and drops exactly those; retention's `audit_events_drop_partitions_before` is not used. It never deletes a row. Its exit statuses (2 usage, 3 unverified, 4 drop refused, 5 nothing to verify) diff --git a/module/services/accounts/code/pkg/infra/postgres_audit_quarantine_test.go b/module/services/accounts/code/pkg/infra/postgres_audit_quarantine_test.go index 3ba61c69c..9118f2212 100644 --- a/module/services/accounts/code/pkg/infra/postgres_audit_quarantine_test.go +++ b/module/services/accounts/code/pkg/infra/postgres_audit_quarantine_test.go @@ -15,7 +15,7 @@ import ( "accounts/pkg/infra/storetx" ) -// The audit relay's quarantine (migration 33): who may touch it, and that moving +// The audit relay's quarantine (migration 34): who may touch it, and that moving // a row into it is one transaction. func enqueueN(t *testing.T, orgID string, n int) []business.AuditEntry { diff --git a/module/services/accounts/code/pkg/infra/postgres_audit_rollback_test.go b/module/services/accounts/code/pkg/infra/postgres_audit_rollback_test.go index 3fdbbf74e..ae60c993c 100644 --- a/module/services/accounts/code/pkg/infra/postgres_audit_rollback_test.go +++ b/module/services/accounts/code/pkg/infra/postgres_audit_rollback_test.go @@ -61,7 +61,7 @@ func asRole(t *testing.T, ctx context.Context, tx pgx.Tx, role, sql string, args // Rolling the queue's migration back drops the only copy of every event the // relay has not delivered, so it must refuse while any row is queued. func TestRollingBackTheQueueRefusesWhileEventsAreQueued(t *testing.T) { - down := readMigration(t, "30_audit_event_queue.down.sql") + down := readMigration(t, "31_audit_event_queue.down.sql") orgID := seedOrg(t, seedUser(t)) migrationOwner(t, func(ctx context.Context, tx pgx.Tx) { @@ -88,7 +88,7 @@ func TestRollingBackTheQueueRefusesWhileEventsAreQueued(t *testing.T) { } func TestRollingBackTheQuarantineRefusesWhileRowsAreSetAside(t *testing.T) { - down := readMigration(t, "33_audit_event_quarantine.down.sql") + down := readMigration(t, "34_audit_event_quarantine.down.sql") migrationOwner(t, func(ctx context.Context, tx pgx.Tx) { countQuarantine := func() int { @@ -154,7 +154,7 @@ func TestRollingBackWaitsForAnInFlightWriterBeforeItCounts(t *testing.T) { }{ { name: "queue", - down: "30_audit_event_queue.down.sql", + down: "31_audit_event_queue.down.sql", table: "public.audit_event_queue", writerRole: "app_control_plane", insert: `INSERT INTO public.audit_event_queue (id, event_type, schema_version, actor_type, resource, created_at) @@ -163,7 +163,7 @@ func TestRollingBackWaitsForAnInFlightWriterBeforeItCounts(t *testing.T) { }, { name: "quarantine", - down: "33_audit_event_quarantine.down.sql", + down: "34_audit_event_quarantine.down.sql", table: "public.audit_event_quarantine", writerRole: "app_job_worker", insert: `INSERT INTO public.audit_event_quarantine (seq, xact_id, id, event_type, schema_version, actor_type, resource, created_at, enqueued_at, error) diff --git a/module/services/store/migrations/30_audit_event_queue.down.sql b/module/services/store/migrations/31_audit_event_queue.down.sql similarity index 100% rename from module/services/store/migrations/30_audit_event_queue.down.sql rename to module/services/store/migrations/31_audit_event_queue.down.sql diff --git a/module/services/store/migrations/30_audit_event_queue.up.sql b/module/services/store/migrations/31_audit_event_queue.up.sql similarity index 100% rename from module/services/store/migrations/30_audit_event_queue.up.sql rename to module/services/store/migrations/31_audit_event_queue.up.sql diff --git a/module/services/store/migrations/31_audit_event_type_retention_class.down.sql b/module/services/store/migrations/32_audit_event_type_retention_class.down.sql similarity index 100% rename from module/services/store/migrations/31_audit_event_type_retention_class.down.sql rename to module/services/store/migrations/32_audit_event_type_retention_class.down.sql diff --git a/module/services/store/migrations/31_audit_event_type_retention_class.up.sql b/module/services/store/migrations/32_audit_event_type_retention_class.up.sql similarity index 100% rename from module/services/store/migrations/31_audit_event_type_retention_class.up.sql rename to module/services/store/migrations/32_audit_event_type_retention_class.up.sql diff --git a/module/services/store/migrations/32_audit_verified_partition_drop.down.sql b/module/services/store/migrations/33_audit_verified_partition_drop.down.sql similarity index 72% rename from module/services/store/migrations/32_audit_verified_partition_drop.down.sql rename to module/services/store/migrations/33_audit_verified_partition_drop.down.sql index 0eb8c503b..e38d33976 100644 --- a/module/services/store/migrations/32_audit_verified_partition_drop.down.sql +++ b/module/services/store/migrations/33_audit_verified_partition_drop.down.sql @@ -1,3 +1,3 @@ --- Reverts migration 32. The history copy would then have to retire partitions +-- Reverts migration 33. The history copy would then have to retire partitions -- through audit_events_drop_partitions_before again. DROP FUNCTION public.audit_events_drop_verified_partitions(timestamp with time zone, text[], bigint[]); diff --git a/module/services/store/migrations/32_audit_verified_partition_drop.up.sql b/module/services/store/migrations/33_audit_verified_partition_drop.up.sql similarity index 100% rename from module/services/store/migrations/32_audit_verified_partition_drop.up.sql rename to module/services/store/migrations/33_audit_verified_partition_drop.up.sql diff --git a/module/services/store/migrations/33_audit_event_quarantine.down.sql b/module/services/store/migrations/34_audit_event_quarantine.down.sql similarity index 100% rename from module/services/store/migrations/33_audit_event_quarantine.down.sql rename to module/services/store/migrations/34_audit_event_quarantine.down.sql diff --git a/module/services/store/migrations/33_audit_event_quarantine.up.sql b/module/services/store/migrations/34_audit_event_quarantine.up.sql similarity index 100% rename from module/services/store/migrations/33_audit_event_quarantine.up.sql rename to module/services/store/migrations/34_audit_event_quarantine.up.sql From 981e0d3ee25dab90397021c2f7c95047029e2066 Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Thu, 8 Oct 2026 13:56:10 -0400 Subject: [PATCH 68/69] test(accounts): declare the subject module's queues, namespaces and scope ceilings in the authority fixtures Core v0.15 requires every authority document to state the subject module's three declarations, `[]` when it owns none, so that a renderer which dropped one cannot read as "owns nothing". The two delivery fixtures that build an `AuthorityDocument` by hand left them nil, and eight database-backed tests in `pkg/infra` failed validation with "declares no queues" once the module took the bump. The fixtures now write all three as empty lists. A withdrawn generation states them empty too, which is what the document rule asks of it. This hunk is shared: it rides with the core bump, and the other feature's pull request carries the same lines, so whichever merges second rebases onto them. Co-Authored-By: Claude Opus 5.5 --- .../accounts/code/pkg/infra/approved_build_reconcile_test.go | 5 +++++ .../code/pkg/infra/solution_delivery_activation_test.go | 5 +++++ 2 files changed, 10 insertions(+) diff --git a/module/services/accounts/code/pkg/infra/approved_build_reconcile_test.go b/module/services/accounts/code/pkg/infra/approved_build_reconcile_test.go index 6c9e2a3cb..c90341e9c 100644 --- a/module/services/accounts/code/pkg/infra/approved_build_reconcile_test.go +++ b/module/services/accounts/code/pkg/infra/approved_build_reconcile_test.go @@ -395,6 +395,11 @@ func (s *approvedBuildScenario) authority( Host: solutionhost.HostTarget{Coordinate: productionPathCoordinate, Component: "saas-host"}, OwnershipDomain: productionPathDomain, EnvelopeRevision: solutionhost.FixtureEnvelopeRevision, + // Core requires the subject module's three declarations on every + // generation, [] when it has none: a nil list reads as one a renderer dropped. + Queues: []string{}, + Namespaces: []string{}, + ScopeCeilings: []solutionhost.ScopeCeiling{}, } if removed { // A withdrawn generation names no principal, approves no build and is diff --git a/module/services/accounts/code/pkg/infra/solution_delivery_activation_test.go b/module/services/accounts/code/pkg/infra/solution_delivery_activation_test.go index 69028779f..59604c05d 100644 --- a/module/services/accounts/code/pkg/infra/solution_delivery_activation_test.go +++ b/module/services/accounts/code/pkg/infra/solution_delivery_activation_test.go @@ -225,6 +225,11 @@ func productionPathAuthority( Host: solutionhost.HostTarget{Coordinate: productionPathCoordinate, Component: "saas-host"}, OwnershipDomain: productionPathDomain, EnvelopeRevision: solutionhost.FixtureEnvelopeRevision, + // Core requires the subject module's three declarations on every + // generation, [] when it has none: a nil list reads as one a renderer dropped. + Queues: []string{}, + Namespaces: []string{}, + ScopeCeilings: []solutionhost.ScopeCeiling{}, } if !removed { document.ApprovedBuild = build From fa441b6237f2e8af4e5172be01a130f7390b284f Mon Sep 17 00:00:00 2001 From: crisfiorini-oss <301773486+crisfiorini-oss@users.noreply.github.com> Date: Thu, 8 Oct 2026 13:56:10 -0400 Subject: [PATCH 69/69] docs(deployment): keep the audit promotion driver apart from the shared Core 0.14 paragraph (Refs #996) The paragraph on promoting a consumer-derived reach cutover named the audit promotion driver beside Core's allow-list support. Only the audit store's deploy Job needs that driver; the Core requirement is shared with the other feature's pull request. The shared sentences now stand alone and the driver has its own paragraph, so the two pull requests no longer edit the same lines. Co-Authored-By: Claude Opus 5.5 --- module/DEPLOYMENT_TOPOLOGY.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/module/DEPLOYMENT_TOPOLOGY.md b/module/DEPLOYMENT_TOPOLOGY.md index a6d9abf05..b6d814d87 100644 --- a/module/DEPLOYMENT_TOPOLOGY.md +++ b/module/DEPLOYMENT_TOPOLOGY.md @@ -71,9 +71,7 @@ edges. Public frontend and marketing endpoints explicitly declare `exposure: public`, preserving their existing public addresses. Promotion requires a released CLI with Core's consumer-derived allow-list -support and the audit promotion driver from -[CLI PR 919](https://github.com/codefly-dev/cli/pull/919). Merge the CLI change, -release that compatible CLI, then promote this kit cutover. The module agent, +support. The module agent, accounts, auth-gateway, store migrator, and module tools require stable Core 0.14 or later to load these declarations and derive consumer access. The runtime SDK also loads the module when resolving local endpoints or secrets. Core 0.13 refuses @@ -82,6 +80,10 @@ still loads the module with its older Core 0.7, but does not derive cross-module access: an internal endpoint with no authored list denies other modules. It is not a compatible promotion driver for this declaration model. +The audit store's deploy Job also needs the audit promotion driver from +[CLI PR 919](https://github.com/codefly-dev/cli/pull/919). Merge the CLI change, +release that compatible CLI, then promote this kit cutover. + The topology-policy golden contains 25 `NetworkPolicy` resources: - one namespace-wide ingress/egress default deny;