Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
77 commits
Select commit Hold shift + click to select a range
a1b81d7
docs(adr): make the audit store swappable to a warehouse (ADR 0009)
crisfiorini-oss Oct 2, 2026
eb41724
docs(adr): settle the audit queue, history removal and archive write …
crisfiorini-oss Oct 2, 2026
0c1953c
docs(adr): settle duplicate archive objects in ADR 0009
crisfiorini-oss Oct 2, 2026
623ba87
feat(audit): write the audit store swap to BigQuery through a transac…
crisfiorini-oss Oct 2, 2026
e19aaa3
feat(audit): take the deployment's archive URL, let declared types st…
crisfiorini-oss Oct 2, 2026
510b7ae
docs(adr): settle warehouse duplicates, declared retention classes an…
crisfiorini-oss Oct 2, 2026
fe18a78
feat(audit): read the BigQuery store of record through the Storage Re…
crisfiorini-oss Oct 4, 2026
21feb10
docs(adr): settle how ADR 0009 reads BigQuery
crisfiorini-oss Oct 4, 2026
58ceb09
refactor(audit): share the sink settings, read back a store's window,…
crisfiorini-oss Oct 4, 2026
1226c8f
feat(audit): copy a switching deployment's audit history into the sto…
crisfiorini-oss Oct 4, 2026
9a2dc17
feat(audit): expose queue lag and alert when delivery stalls
crisfiorini-oss Oct 4, 2026
b84e1a5
docs(audit): add the relay lag response runbook
crisfiorini-oss Oct 4, 2026
6952a57
feat(audit): add the ClickHouse store of record and an S3 archive writer
crisfiorini-oss Oct 4, 2026
25fc80d
docs(adr): record ClickHouse audit settings
crisfiorini-oss Oct 4, 2026
9978e2d
Merge remote-tracking branch 'origin/main' into feat/audit-store-bigq…
crisfiorini-oss Oct 5, 2026
f4ad4f9
feat(audit): ship bounded qualification and content-bound history rec…
crisfiorini-oss Oct 5, 2026
59bdacf
style(audit): apply required Go import formatting
crisfiorini-oss Oct 5, 2026
ac94469
docs(adr): correct ADR 0009 settings, retention and relay ordering
crisfiorini-oss Oct 5, 2026
62b4ba4
fix(audit): set aside the row a warehouse refuses instead of stalling…
crisfiorini-oss Oct 5, 2026
2b5f1b5
fix(audit): watch the audit queue under every sink and refuse a rollb…
crisfiorini-oss Oct 5, 2026
266ed09
fix(audit): make the role catalog import refuse an unset AUDIT_SINK a…
crisfiorini-oss Oct 5, 2026
302b2c5
fix(audit): declare the payload keys the role catalog import records …
crisfiorini-oss Oct 5, 2026
897ffa3
fix(audit): retire only the partitions a history copy verified, atomi…
crisfiorini-oss Oct 5, 2026
f6ee3e2
fix(audit): give audit-history-copy honest exit codes, a completed-mo…
crisfiorini-oss Oct 5, 2026
64b3cbd
fix(audit): make postgres_no_new_rows check the interval it reports
crisfiorini-oss Oct 5, 2026
8c517c4
docs(audit): say what the history copy and the qualification checks a…
crisfiorini-oss Oct 5, 2026
df10979
fix(audit): read an organization id in its canonical form
crisfiorini-oss Oct 5, 2026
de07a78
fix(audit): clamp a ClickHouse time bound to the column's range
crisfiorini-oss Oct 5, 2026
0d37078
fix(audit): make a ClickHouse cluster insert wait for a quorum
crisfiorini-oss Oct 5, 2026
7062bff
fix(audit): reset a BigQuery stream's retry budget when it makes prog…
crisfiorini-oss Oct 5, 2026
0edbabb
fix(audit): anchor BigQuery read windows at the present, not at a far…
crisfiorini-oss Oct 5, 2026
fd670eb
fix(audit): split BigQuery appends by request size, not only row count
crisfiorini-oss Oct 5, 2026
fff787c
fix(audit): bound the memory of a warehouse read by a budget, not the…
crisfiorini-oss Oct 5, 2026
e24ab16
fix(audit): register the quarantine relation in the authority inventory
crisfiorini-oss Oct 5, 2026
5db2d00
docs(audit): make the docs say what the audit relay, importer and exp…
crisfiorini-oss Oct 5, 2026
6665255
fix(audit): let each store name the rows it refuses for good
crisfiorini-oss Oct 6, 2026
f323ab9
fix(audit): quarantine only the rows a store reports it refused for good
crisfiorini-oss Oct 6, 2026
c20b198
docs(adr): match ADR 0009 to the relay, grants and read bounds as built
crisfiorini-oss Oct 6, 2026
7806c04
docs(adr): renumber the audit migrations ADR 0009 cites to 19 through 22
crisfiorini-oss Oct 6, 2026
b65e4d9
Merge origin/main into feat/audit-store-bigquery
crisfiorini-oss Oct 6, 2026
15b6915
Merge the ADR 0009 branch into the audit store branch
crisfiorini-oss Oct 6, 2026
5196a05
chore(frontend): bump the published kit version for the regenerated S…
crisfiorini-oss Oct 6, 2026
d0b7e4d
feat(accounts,auth-gateway): resolve the cell's collector from the ob…
crisfiorini-oss Oct 6, 2026
7e0ee42
feat(module)!: remove the telemetry service; the cell's collector rep…
crisfiorini-oss Oct 6, 2026
0d37dc7
fix(audit): lock the audit queue and quarantine before a rollback cou…
crisfiorini-oss Oct 6, 2026
da9d88a
fix(audit): refuse an existing ClickHouse table that is not replicate…
crisfiorini-oss Oct 6, 2026
2cafb08
fix(audit): bound the state an aggregation keeps across read windows
crisfiorini-oss Oct 6, 2026
3308622
Merge origin/main into feat/audit-store-bigquery
crisfiorini-oss Oct 7, 2026
faa8477
Merge remote-tracking branch 'origin/main' into feat/telemetry-servic…
crisfiorini-oss Oct 7, 2026
152fd50
fix(accounts,auth-gateway): the observability state decides; build th…
crisfiorini-oss Oct 7, 2026
2991e7b
refactor(accounts,auth-gateway): resolve the listener ports without a…
crisfiorini-oss Oct 7, 2026
2720aa8
chore(deps): bump codefly core to v0.13.0 and sdk-go to v0.3.1 (Refs …
crisfiorini-oss Oct 7, 2026
badfb6c
fix(accounts,auth-gateway): restore the generated telemetry client st…
crisfiorini-oss Oct 7, 2026
feafbdd
fix(audit): history copy rewrites events whose details a failed write…
crisfiorini-oss Oct 7, 2026
7de6534
fix(audit): charge a BigQuery read once per event, not per delivered …
crisfiorini-oss Oct 7, 2026
270df7a
fix(audit): bound ClickHouse percentile samples before they reach the…
crisfiorini-oss Oct 7, 2026
3532058
fix(vault): declare its endpoint private within the module
crisfiorini-oss Oct 7, 2026
23eb519
fix(audit): history copy sets aside an event the store refuses and ke…
crisfiorini-oss Oct 7, 2026
0f9fdb9
fix(audit): hold a kept event as its text, not the decode of its payload
crisfiorini-oss Oct 7, 2026
6698973
fix(audit): preserve durable delivery and recover partial history copies
crisfiorini-oss Oct 7, 2026
48ced8d
fix(telemetry): require platform state outside local runtimes
crisfiorini-oss Oct 7, 2026
fa62f28
fix(audit): bound history reads and validate writable schemas
crisfiorini-oss Oct 7, 2026
f4f1f9c
fix(audit): format bounded history test imports
crisfiorini-oss Oct 7, 2026
ee06dac
fix(audit): run catalog imports with deployed database credentials
crisfiorini-oss Oct 7, 2026
20b8e01
feat(audit,telemetry)!: combine warehouse migration and cell collecto…
crisfiorini-oss Oct 7, 2026
55699c6
fix(deployment): derive consumer reach and match immutable bootstrap …
crisfiorini-oss Oct 7, 2026
2a8170c
fix(deployment): align SDK runtime loaders with endpoint declarations
crisfiorini-oss Oct 7, 2026
a257a09
Merge main and preserve warehouse migration and cell telemetry delivery
crisfiorini-oss Oct 7, 2026
85247e2
fix(audit): point rollback tests at renumbered migrations
crisfiorini-oss Oct 7, 2026
0800ede
fix(telemetry): ship neither the telemetry state nor its reason in th…
crisfiorini-oss Oct 7, 2026
db384ee
fix(audit): a store refusing every row is a store failure, and a refu…
crisfiorini-oss Oct 7, 2026
21656c6
fix(accounts,auth-gateway): take core v0.15.1, which is what reads th…
antoinetoussaint-byte Oct 8, 2026
8c95567
Merge main into the audit store branch
crisfiorini-oss Oct 8, 2026
a804ee9
chore(telemetry): leave the telemetry cleanup to its own pull request…
crisfiorini-oss Oct 8, 2026
ded6eda
fix(audit): renumber the audit migrations after main took migration 3…
crisfiorini-oss Oct 8, 2026
981e0d3
test(accounts): declare the subject module's queues, namespaces and s…
crisfiorini-oss Oct 8, 2026
fa441b6
docs(deployment): keep the audit promotion driver apart from the shar…
crisfiorini-oss Oct 8, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions AUTHZ.md
Original file line number Diff line number Diff line change
Expand Up @@ -908,6 +908,18 @@ go run ./cmd/role-catalog-import -catalog roles.json -database-url "$DATABASE_UR
go run ./cmd/role-catalog-import -catalog roles.json -database-url "$DATABASE_URL"
```

The importer records one audit event per changed role through the same emitter
the service uses, so it must run under the deployment's `AUDIT_SINK`
(`-audit-sink`, defaulting to `$AUDIT_SINK`): under a swap value (ADR 0009) its
events go to the transactional queue the accounts relay delivers, not to
`audit_events`. Unlike the service, the importer has no default: an unset (or
blank) `AUDIT_SINK` makes it exit before touching the store, because guessing
`postgres` would write its events where a warehouse deployment never reads
them. A deployment that relies on the service's `postgres` default must
therefore set `AUDIT_SINK=postgres` explicitly. The deploy step below declares
it (`service_environment: [AUDIT_SINK]`), so the driver gives the Job the
accounts service's own value, which is why the service must have one.

Domain core is `pkg/rolecatalog` (parse + diff + deterministic plan, no DB);
`pkg/infra` snapshots current state and applies the plan in one transaction.

Expand Down
34 changes: 27 additions & 7 deletions gitops.go
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,11 @@ type bundleDeployJob struct {
Force bool `json:"force,omitempty"`
Writes bundleDeployTarget `json:"writes"`
After []string `json:"after,omitempty"`
// ServiceEnvironment names environment variables the driver sets on the Job
// to the values the running service has in the same environment, so the
// step behaves as the service would — the import, for one, records its audit
// events wherever the service's AUDIT_SINK sends them.
ServiceEnvironment []string `json:"serviceEnvironment,omitempty"`
}

type bundleDeployTarget struct {
Expand Down Expand Up @@ -247,10 +252,9 @@ type topologyModule struct {
}

type topologyInterface struct {
Service string `yaml:"service"`
Endpoint string `yaml:"endpoint"`
Visibility string `yaml:"visibility"`
AllowModules []string `yaml:"allow-modules,omitempty"`
Service string `yaml:"service"`
Endpoint string `yaml:"endpoint"`
Visibility string `yaml:"visibility"`
}

type topologyService struct {
Expand Down Expand Up @@ -324,6 +328,9 @@ type topologyDeployJob struct {
Force bool `yaml:"force,omitempty"`
Writes topologyDeployJobTarget `yaml:"writes"`
After []string `yaml:"after,omitempty"`
// ServiceEnvironment names the running service's environment variables the
// Job receives with the service's own values (bundleDeployJob).
ServiceEnvironment []string `yaml:"service_environment,omitempty"`
}

type topologyDeployJobTarget struct {
Expand Down Expand Up @@ -460,7 +467,8 @@ func planDeployJobs(topology deploymentTopology, plan environmentPlan) []bundleD
Endpoint: job.Writes.Endpoint,
Port: endpoint.Port,
},
After: append([]string(nil), job.After...),
After: append([]string(nil), job.After...),
ServiceEnvironment: append([]string(nil), job.ServiceEnvironment...),
})
}
if len(jobs) == 0 {
Expand Down Expand Up @@ -1907,10 +1915,24 @@ func validateDeployJobs(moduleDir string, topology deploymentTopology, services
return fmt.Errorf("deployment topology deploy Job %q runs after undeclared service %q", job.Name, after)
}
}
inherited := make(map[string]struct{}, len(job.ServiceEnvironment))
for _, name := range job.ServiceEnvironment {
if !environmentVariableName.MatchString(name) {
return fmt.Errorf("deployment topology deploy Job %q service_environment entry %q is not an environment variable name", job.Name, name)
}
if _, repeated := inherited[name]; repeated {
return fmt.Errorf("deployment topology deploy Job %q service_environment repeats %q", job.Name, name)
}
inherited[name] = struct{}{}
}
}
return nil
}

// environmentVariableName is the shape of a name a deploy Job may inherit from
// its running service: upper-case letters, digits and underscores.
var environmentVariableName = regexp.MustCompile(`^[A-Z][A-Z0-9_]*$`)

func serviceDependsOn(service topologyService, dependency, endpoint string) bool {
for _, declared := range service.Dependencies {
if declared.Service == dependency && slices.Contains(declared.Endpoints, endpoint) {
Expand Down Expand Up @@ -2259,7 +2281,6 @@ func bootstrapJobIngressPolicy(namespace string, labels map[string]string, servi
"ingress": []any{map[string]any{
"from": []any{map[string]any{"podSelector": map[string]any{"matchLabels": map[string]string{
"codefly.dev/bootstrap-service": service,
"job-name": service,
}}}},
"ports": networkPorts(ports),
}},
Expand All @@ -2275,7 +2296,6 @@ func bootstrapJobEgressPolicy(namespace string, labels map[string]string, servic
Spec: map[string]any{
"podSelector": map[string]any{"matchLabels": map[string]string{
"codefly.dev/bootstrap-service": service,
"job-name": service,
}},
"policyTypes": []string{"Egress"},
"egress": []any{map[string]any{
Expand Down
30 changes: 26 additions & 4 deletions gitops_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -597,6 +597,26 @@ func TestGenerateBundleRejectsHostileContracts(t *testing.T) {
},
want: "references undeclared service",
},
{
name: "deploy job inheriting a malformed environment name",
mutate: func(t *testing.T, moduleDir string, _ *workspaceManifest) {
t.Helper()
writeCatalogArtifact(t, moduleDir)
appendDeployJob(t, moduleDir, "store", "http")
replaceInDeployJobs(t, moduleDir, " - AUDIT_SINK\n", " - audit-sink\n")
},
want: "is not an environment variable name",
},
{
name: "deploy job inheriting an environment name twice",
mutate: func(t *testing.T, moduleDir string, _ *workspaceManifest) {
t.Helper()
writeCatalogArtifact(t, moduleDir)
appendDeployJob(t, moduleDir, "store", "http")
replaceInDeployJobs(t, moduleDir, " - AUDIT_SINK\n", " - AUDIT_SINK\n - AUDIT_SINK\n")
},
want: "service_environment repeats",
},
{
name: "deploy job to a migration-bearing target without ordering",
mutate: func(t *testing.T, moduleDir string, _ *workspaceManifest) {
Expand Down Expand Up @@ -809,6 +829,9 @@ func TestDeployJobWiresCatalogImport(t *testing.T) {
if !slices.Equal(job.After, []string{"store"}) {
t.Errorf("%s deploy job ordering = %v", env.Name, job.After)
}
if !slices.Equal(job.ServiceEnvironment, []string{"AUDIT_SINK"}) {
t.Errorf("%s deploy job service environment = %v, want the running service's AUDIT_SINK", env.Name, job.ServiceEnvironment)
}
}

overlays := filepath.Join(moduleDir, filepath.FromSlash(bundleRelativeDir), "overlays")
Expand Down Expand Up @@ -2348,7 +2371,9 @@ func appendDeployJob(t *testing.T, moduleDir, writesService, writesEndpoint stri
" service: "+writesService+"\n"+
" endpoint: "+writesEndpoint+"\n"+
" after:\n"+
" - store\n")
" - store\n"+
" service_environment:\n"+
" - AUDIT_SINK\n")
}

// serviceManifestFixture renders an authored service.codefly.yaml: one endpoint,
Expand All @@ -2365,9 +2390,6 @@ func serviceManifestFixture(name, endpoint, api, visibility string, port int, to
}
if visibility != "private" {
manifest.WriteString(" visibility: " + visibility + "\n")
if visibility == "internal" {
manifest.WriteString(" allow-modules: [\"*\"]\n")
}
}
manifest.WriteString("spec:\n")
manifest.WriteString(spec)
Expand Down
2 changes: 1 addition & 1 deletion go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ module github.com/codefly-dev/agents/modules/saas-starter
go 1.27.0

require (
github.com/codefly-dev/core v0.5.11-0.20260926231217-d481a2d49683
github.com/codefly-dev/core v0.14.0
gopkg.in/yaml.v3 v3.0.1
)

Expand Down
4 changes: 2 additions & 2 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -28,8 +28,8 @@ github.com/cheggaaa/pb/v3 v3.2.1 h1:aprZbFRG+B7+ug76S8QZ6Y1PW168UHzOmbC3wa+aU6I=
github.com/cheggaaa/pb/v3 v3.2.1/go.mod h1:U9hSVxoKqJrZIE3PkFG1xXXNaK/Ilzg+EF/scpXzEdw=
github.com/clipperhouse/uax29/v2 v2.7.0 h1:+gs4oBZ2gPfVrKPthwbMzWZDaAFPGYK72F0NJv2v7Vk=
github.com/clipperhouse/uax29/v2 v2.7.0/go.mod h1:EFJ2TJMRUaplDxHKj1qAEhCtQPW2tJSwu5BF98AuoVM=
github.com/codefly-dev/core v0.5.11-0.20260926231217-d481a2d49683 h1:/vzymsu5B9NWwdhWRtlU+68VcVEVDmeZNChRCXAUwlE=
github.com/codefly-dev/core v0.5.11-0.20260926231217-d481a2d49683/go.mod h1:SVF+t040X+9pAXAqBnNE/JkXwmw9peVuKnHKFSOzyzk=
github.com/codefly-dev/core v0.14.0 h1:rpLFxeHSf2PrbxsRSp/NcZt8htMRM38JV2Oc8RKievU=
github.com/codefly-dev/core v0.14.0/go.mod h1:rp4YZkCJEOufliWQAgK3sv8YlO7Xcva7+cN7hIlAR0I=
github.com/codefly-dev/gortk v0.2.0 h1:7bOlS5valYz2zil+fZctQNcPCYBcPj86abcw9N8h1hQ=
github.com/codefly-dev/gortk v0.2.0/go.mod h1:dDWUMFgAP063OCGhTEpV7FFUj9+zTcxxO/nV80VKEwg=
github.com/containerd/errdefs v1.0.0 h1:tg5yIfIlQIrxYtu9ajqY42W3lpS19XqdxRQeEwYG8PI=
Expand Down
6 changes: 5 additions & 1 deletion libraries/source-read-sdk/contract/catalog.codefly.json
Original file line number Diff line number Diff line change
Expand Up @@ -10,13 +10,15 @@
"kind": "protobuf",
"package": "saas.accounts.v1",
"path": "contract.binpb",
"digest": "sha256:b8ef1ba87e5d9b96c067194fa0fb07892c2ba937c623e54a8c5290cc6609cab8",
"digest": "sha256:a6154d416ce1783b41b02bdddf7daecc77d61501337f2e1ad556ae3f35d8fbe2",
"services": [
{
"name": "ModuleCapabilitiesService",
"fullName": "saas.accounts.v1.ModuleCapabilitiesService",
"procedures": [
"/saas.accounts.v1.ModuleCapabilitiesService/AckJob",
"/saas.accounts.v1.ModuleCapabilitiesService/ApproveExecutableArtifact",
"/saas.accounts.v1.ModuleCapabilitiesService/AuthorizeExecutableArtifact",
"/saas.accounts.v1.ModuleCapabilitiesService/CancelApproval",
"/saas.accounts.v1.ModuleCapabilitiesService/CheckWorkContextRecordAccess",
"/saas.accounts.v1.ModuleCapabilitiesService/ClaimJobs",
Expand All @@ -28,6 +30,7 @@
"/saas.accounts.v1.ModuleCapabilitiesService/FetchDatasourceBlob",
"/saas.accounts.v1.ModuleCapabilitiesService/FetchDatasourceFiles",
"/saas.accounts.v1.ModuleCapabilitiesService/GetApproval",
"/saas.accounts.v1.ModuleCapabilitiesService/GetCurrentInstallation",
"/saas.accounts.v1.ModuleCapabilitiesService/HeartbeatJob",
"/saas.accounts.v1.ModuleCapabilitiesService/ListReadableSourceCollections",
"/saas.accounts.v1.ModuleCapabilitiesService/ListSubjectVisibility",
Expand All @@ -44,6 +47,7 @@
"/saas.accounts.v1.ModuleCapabilitiesService/PublishEvent",
"/saas.accounts.v1.ModuleCapabilitiesService/ReplayEvents",
"/saas.accounts.v1.ModuleCapabilitiesService/RequestApproval",
"/saas.accounts.v1.ModuleCapabilitiesService/RevokeExecutableArtifact",
"/saas.accounts.v1.ModuleCapabilitiesService/Subscribe",
"/saas.accounts.v1.ModuleCapabilitiesService/Unsubscribe"
]
Expand Down
Binary file modified libraries/source-read-sdk/contract/contract.binpb
Binary file not shown.
32 changes: 32 additions & 0 deletions libraries/source-read-sdk/go/accounts_facade.pb.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading