Skip to content

feat(accounts): add audit storage interface and journal delivery - #1106

Open
crisfiorini-oss wants to merge 4 commits into
codefly-dev:mainfrom
crisfiorini-oss:feat/audit-storage-switch
Open

crisfiorini-oss wants to merge 4 commits into
codefly-dev:mainfrom
crisfiorini-oss:feat/audit-storage-switch

Conversation

@crisfiorini-oss

@crisfiorini-oss crisfiorini-oss commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Make audit storage a startup choice: postgres preserves current behavior; warehouse records full audit entries in the existing Postgres journal for delivery through AuditStorage. This PR supplies the storage interface, journal read/delivery foundation, and receipt-proven journal retention.

Warehouse startup remains refused until the Warehouse AuditStorage and admin page can use a released service-warehouse Scan API. This PR does not claim the complete warehouse feature.

Behavior

  • ReadJournal requires a live cross-tenant module grant, held through response construction. Positions are (xact_id,seq), only finished transactions are returned, and pages respect cumulative JSON/protobuf limits. Oversized audit records are refused before commit.
  • DeliverAuditEvents and lookup accept the same {from,to} body. Derive the range receipt identity internally; preserve the caller's separate transport effect identity. Replay/lookup verify the request digest; missing lookup receipts return NOT_FOUND.
  • Delivery requires an idempotent saas-audit operation context and live cross-tenant installation grant with saas.audit-delivery invoke/read authority, rechecked after the receipt hold. The Runnable uses audit_delivery, three attempts and a 60-second attempt timeout. Delivery emits no audit of itself.
  • Machinery marking checks module identity and binding kind/ID. Shared Authority task/child-session issuance records the authenticated module and the binding the core authorized; those current audit types use the same marker. Ordinary person issues remain unmarked. Connect preserves the module credential and refuses ambiguous duplicate credentials.
  • Successful new delivery receipts include from/to. Warehouse-only RunRetention deletes internal audit journal rows only when older than 30 days and covered by a verified committed delivery receipt. Verify method, effect identity, request digest and valid range; preserve legacy/malformed/mismatched/unproven coverage. Merge proven ranges without filling holes. Expose old_unproven counts in retention logs.
  • Existing source-receipt cleanup excludes audit delivery, preserving the full range proof. Postgres-mode journal behavior remains unchanged; other domain-event types are outside this cleanup.
  • Unshipped migrations move to 61 (user-scoped journal publication) and 62 (xact_id), preserving current main through 60. Migration 63 grants the maintenance role audit-delivery receipt SELECT and audit-journal DELETE, with a restrictive audit-type/720-hour age policy. Other event types and young audit rows remain undeletable through that role.
  • Migration 61 widens the existing tenant-less publish exception: both envelope and record must name the current user, and the record must name no organization. This publish-guard change needs explicit review.
  • AUDIT_SINK=both remains refused with warehouse. Warehouse storage, percentile refusal and its operator page remain with the released Scan dependency.

Fix or hack

Fixes at the owners of audit storage, journal publication, authorization, transport and receipt retention. Reuse the existing journal and SDK receipts; add no ledger, outbox, host-to-runtime call or delivery identity.

Build or reuse

Component Searched and verified Decision
Storage and journal This repository, core/sdk-go, closed #1000 interfaces/parity suite, existing domain_events and transaction visibility gate Extend the owner and preserve existing Postgres queries.
Shared Authority footprint Existing Issue handler, transactional RecordAuthorityIssue and journal marker; current shared task/child-session audit events Preserve authenticated module/binding metadata and extend the same caller-bound marker. No new queue, event type or issuer.
Delivery/retention proof Existing RunRetention, SDK codefly_effect_receipts, marked receipt guard and source-receipt cleanup Extend committed response and existing maintenance transaction; legacy/unproven receipts retain rows. No second receipt store.
Authority and transport Existing live module verification, Connect adapters, generic SERVICE same-body recovery contract Extend existing checks/adapters and derive the range receipt identity internally.
Shared interface compatibility Existing host adapters, canonical installation schema and Authority #41, which already implements member-held installation reads and conformance Reuse core HeldInstallations through thin gRPC/Connect adapters and authenticated method policy; no new role filter or query. Adopt the actual shared development revision bfe4876a8b96 and regenerate complete bindings.
Generated bindings and compiler Existing pinned proto, full service Sync, business/adapters/catalog, contract/Runnable/frontend SDK owners; released service-go-grpc 0.1.58 and its verified builder/runtime images Regenerate complete outputs from combined source. Reuse the released Go 1.27.3 compiler/runtime fix from service-go#150 and service-go-grpc#189/#190; no generated-file selection, hand edits or invented pin.

Compiler and generated output

Preserve current main's released service-go-grpc 0.1.58, Go 1.27.3 service/CI floors and patched builder/runtime images. The rebase preserves sdk-go 0.4.1 and current main's OAuth API-source UI, module-package metadata and patched frontend dependencies. Required owner-generated outputs remain committed; dedicated generated directories are marked while authored proto, templates, configuration and locks remain visible.

Validation

Rebased onto upstream main df044ce68; head c1d3b1be0510377f6e909866cd2c82d6a8418c61. Preserve current main's shared Authority implementation and published migrations through 60. Only the unmerged audit migrations move to 61–63; replay assertions follow those versions.

  • Exact upstream main and feature proto generation PASS with released CLI 0.1.215 / proto companion 0.0.18. Complete business, adapters, catalogs, contracts, Runnables and both generated libraries are regenerated by their owners. The canonical generated-library clean-tree drift gate PASSes on this head. No generated output was hand-resolved or selected from a parent.
  • Real disposable-PostgreSQL publisher and xid forward/down replay tests PASS (21.340s), with MIGRATION_CONSENT_REPLAY=1; these are real replay runs, not the default skipped invocation. Migration pairing and reference checks PASS.
  • The owner-generated control-plane golden PASSes (3.386s) under the documented pure build tag. Naming, migration pairing and whitespace checks PASS. Scoped audit business, adapter and authorization tests PASS (0.302s, 2.794s and 0.303s); full suites run in GitHub CI.
  • CLI #1186 supplies the owner fix for malformed generated-library metadata. Its exact development build was used only for recovery, followed by complete generation with released 0.1.215 and clean-tree reproduction. No development generator pin is committed. The authored TypeScript SDK version is 0.3.41 and npm-owned root lock regeneration PASSes; main's app dependency remains selected by main.
  • Shared interface compatibility reuses core ListHeld through thin gRPC/Connect mappings and authenticated policy. The previously failing real member-listing case PASSes (51.238s). Both catalog and real-interceptor impersonation inventories now explicitly retain ListHeld and count 69 protected methods; the parity assertions remain intact.
  • Shared issuance/machinery regression PASSes (0.458s), all 11 audit registry tests PASS (0.196s), and the real producer workload-issuance conformance case PASSes (44.250s, including dependencies). Existing transactional audit recording carries verified module/binding metadata; no new event type or receipt store.
  • All runnable GitHub checks PASS on this exact head, required or not, in full CI, naming and boundaries. Only conditional publication jobs skip. Full service tests, lint, compile, build, sync drift, authorization coverage, shared Authority conformance and release aggregate are green.
  • No release, production image publication or deployment is claimed.

Remaining / not run

  • Warehouse AuditStorage and admin page await the service-warehouse Scan release; warehouse startup stays refused.
  • Runtime's module-identity credential route is not composed: generic Authority Issue requires a binding and the host reserves module-capabilities. That producer decision belongs to the consuming integration, not an invented host authentication shortcut.
  • No composed delivery, live installation mint or real warehouse round trip.
  • SDK Generate all GitOps environments atomically (#39) #40 must regenerate from the host release tag before SDK publication; consuming modules then adopt the released journal contract. No unreleased runtime pin is introduced.

Related: codefly-dev/saas-sdk-go#40.

@crisfiorini-oss crisfiorini-oss changed the title feat(accounts): audit storage switch — Postgres or Warehouse, delivered through the journal feat(accounts): add audit storage interface and journal delivery Oct 13, 2026
@crisfiorini-oss
crisfiorini-oss force-pushed the feat/audit-storage-switch branch from a48c8cb to 74d7bb0 Compare October 13, 2026 14:48
@crisfiorini-oss
crisfiorini-oss marked this pull request as ready for review October 13, 2026 15:18
@crisfiorini-oss
crisfiorini-oss force-pushed the feat/audit-storage-switch branch 3 times, most recently from 4e19aad to b640a5f Compare October 15, 2026 15:57
@crisfiorini-oss
crisfiorini-oss force-pushed the feat/audit-storage-switch branch 5 times, most recently from 757fa74 to 61484a8 Compare October 26, 2026 18:45
@crisfiorini-oss
crisfiorini-oss marked this pull request as draft October 27, 2026 16:11
@crisfiorini-oss
crisfiorini-oss force-pushed the feat/audit-storage-switch branch from 80e238c to 21c1213 Compare October 27, 2026 16:11
@crisfiorini-oss
crisfiorini-oss force-pushed the feat/audit-storage-switch branch from 21c1213 to 13cf753 Compare October 27, 2026 16:24
@crisfiorini-oss
crisfiorini-oss marked this pull request as ready for review October 27, 2026 18:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant