diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 87ecf60..7b1a967 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -65,7 +65,7 @@ jobs: - uses: docker/setup-qemu-action@c7c53464625b32c7a7e944ae62b3e17d2b600130 - name: Run unit tests - run: go test ./... -skip '^(TestCreateToRunDocker|TestDirtyMigrationFailsClosed|TestLifecycleContractDocker|TestMigrationLockBudgetExpiresWhilePeerHoldsTheLock|TestMigrationStatementBudgetExpiresAndPreservesTheLedger)$' -timeout 20m + run: go test ./... -skip '^(TestCreateToRunDocker|TestDirtyMigrationFailsClosed|TestLifecycleContractDocker|TestLocalServerLifecycleDocker|TestMigrationLockBudgetExpiresWhilePeerHoldsTheLock|TestMigrationStatementBudgetExpiresAndPreservesTheLedger)$' -timeout 20m - name: Build runtime image run: docker build --build-arg SOURCE_DATE_EPOCH=0 --tag service-postgres:test . @@ -90,6 +90,11 @@ jobs: SERVICE_POSTGRES_TEST_IMAGE: service-postgres:test run: go test . -run '^(TestMigrationLockBudgetExpiresWhilePeerHoldsTheLock|TestMigrationStatementBudgetExpiresAndPreservesTheLedger)$' -timeout 20m + - name: Shared local server regression + env: + SERVICE_POSTGRES_TEST_IMAGE: service-postgres:test + run: go test . -race -run '^TestLocalServerLifecycleDocker$' -count=1 -timeout 10m + - name: Smoke test runtime image shell: bash run: | diff --git a/AGENTS.md b/AGENTS.md index 2fd815c..6977b08 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -72,18 +72,18 @@ Derived from `.github/workflows/ci.yml` and the reusable go build ./... go vet ./... go mod tidy -diff # CI fails on any diff -go test ./... -skip '^(TestCreateToRunDocker|TestDirtyMigrationFailsClosed|TestLifecycleContractDocker|TestMigrationLockBudgetExpiresWhilePeerHoldsTheLock|TestMigrationStatementBudgetExpiresAndPreservesTheLedger)$' -timeout 20m +go test ./... -skip '^(TestCreateToRunDocker|TestDirtyMigrationFailsClosed|TestLifecycleContractDocker|TestLocalServerLifecycleDocker|TestMigrationLockBudgetExpiresWhilePeerHoldsTheLock|TestMigrationStatementBudgetExpiresAndPreservesTheLedger)$' -timeout 20m ``` That skip list is CI's unit step, and it is the inventory of what needs a container. Everything else is template rendering, lock parsing, workflow -assertions and unit tests. The three migration regressions run after it, +assertions and unit tests. The shared-server and three migration regressions run after it, against a locally built image: ```bash docker build --build-arg SOURCE_DATE_EPOCH=0 --tag service-postgres:test . SERVICE_POSTGRES_TEST_IMAGE=service-postgres:test go test . \ - -run '^(TestDirtyMigrationFailsClosed|TestMigrationLockBudgetExpiresWhilePeerHoldsTheLock|TestMigrationStatementBudgetExpiresAndPreservesTheLedger)$' -timeout 20m + -run '^(TestLocalServerLifecycleDocker|TestDirtyMigrationFailsClosed|TestMigrationLockBudgetExpiresWhilePeerHoldsTheLock|TestMigrationStatementBudgetExpiresAndPreservesTheLedger)$' -timeout 20m ``` `TestLifecycleContractDocker` runs later still, against the *published* image. diff --git a/docs/local-shared-server.md b/docs/local-shared-server.md new file mode 100644 index 0000000..462ad6d --- /dev/null +++ b/docs/local-shared-server.md @@ -0,0 +1,127 @@ +# Share a local Postgres server + +A local composition can run one Postgres server with a separate logical database +for each database service. This is an explicit dependency, using Codefly's existing +network/configuration handoff and lifecycle ordering. It works with an owned Nix +or Docker server. The default remains one owned server per service. + +## Compose it + +On the service that owns the server, set: + +```yaml +spec: + database-name: local_control + shared-local-server: true +``` + +On each database service using that server, declare a normal managed service +dependency (not `kind: external`) and set `local-server` to its module/service: + +```yaml +service-dependencies: + - name: postgres + module: storage +spec: + database-name: accounts + local-server: storage/postgres +``` + +These are excerpts of services scaffolded with `codefly add service --agent=postgres`. +Use the containing module's actual dependency identity. The server and borrowers +must use an agent containing this feature. A missing opt-in, dependency mapping or +exported configuration fails before provisioning; there is no default address. + +Give each database service a **distinct `POSTGRES_USER` migration-owner name** +through its Codefly configuration. Keep its own generated owner/reader/writer +passwords. Names must fit PostgreSQL's 63-byte identifier limit. Application +principals continue to be derived from the database name, and declared delegated +roles must be unique across databases. A collision is refused before changing +another database's credentials. This matters because PostgreSQL roles are global +to a server, even though tables and migrations belong to individual databases. + +Run the consuming services together with `codefly run service`; their shared +server dependency starts once. Endpoints and credentials come from the dependency's +accepted Init response. Each logical service republishes its own endpoint identity +and reader/writer configuration at that server's addresses, so application clients +continue using their existing service dependencies. No application DSN changes. +Independent naming scopes resolve separate owning servers and retained state. +There is no process-global or machine-global pool. + +The server owner exports the secret `postgres.local-server-connection` only after +`shared-local-server: true` is selected. A borrower does not export it: sharing +chains are refused. This is a local administrative capability, available to the +database runtime, not a credential applications should consume. As with existing +local owned clusters, migration owners are trusted superusers. Reader/writer +logins are not; CONNECT is explicitly granted per database and revoked from PUBLIC +on both the sharing server's database and each borrowed database. Do not use an +owner connection in an application. This feature is not an isolation boundary +between mutually untrusted migration authors. + +The Postgres major must match the agent's managed runtime image. Borrowers cannot +set `docker-image`, `keep-running`, or external-identity authentication: those are +server-owner decisions. Required extensions still go through normal schema +preparation and fail if unavailable. Hosted deployment still uses the existing +`external-instances` binding; these local settings do not replace that binding or +change deployed manifests. + +## Lifecycle and recovery + +The existing Codefly graph owns server ordering and supervision. A logical service +runs its own migrations and runtime-grant reconciliation before reporting ready. +Concurrent initializations serialize cluster-wide role allocation through an +advisory lock in the server owner's database. Durable role reservations in +`codefly_local_server.roles` survive interrupted/dirty bootstrap and prevent +another database from taking the same role name. Do not delete this schema to +resolve a collision: choose distinct declared identities instead. + +Stopping or destroying a borrower stops its watchers and releases its local +lifecycle state; it never stops the dependency's server or drops data. The owning +service stops the physical server when its composition stops. Restarting uses the +same retained server directory and per-database migration ledgers. Directly +stopping the server owner while applications still use it stops their shared +server, just as stopping any other required dependency does. + +## Move existing local data deliberately + +Changing `local-server` does **not** import an existing service-owned cluster. It +would create an empty logical database on the shared server. Keep the old cluster +and backup until the composed application has been verified against the restored +state; none of this agent's Stop/Destroy operations delete data. + +1. Quiesce application writes and make a consistent `pg_dump` custom-format + backup of each source database, using Codefly-resolved owner connections. + Record row counts, migration versions and representative reference identities. +2. Prove each backup restores into an isolated test composition. Resolve its + connections through Codefly as well. Bootstrap the target service with its + distinct owner and normal migrations, then restore the application's data + using a reviewed restore list. Do not restore global roles from the source + server, replace the target migration ledger blindly, or use `--clean` on an + existing target. A schema/version mismatch must be resolved before cutover. +3. Repeat the restore into the intended fresh shared composition while writes + remain stopped. Verify counts, relationships, application queries, permissions + and migration ledgers; retain the original cluster for rollback. +4. Switch the application composition only after verification. Rollback means + stopping writes and returning to the original dependency configuration; writes + accepted after cutover require reconciliation before such a rollback. + +The PR's lifecycle test proves persistence on a new shared composition. It does +not claim to have migrated any user's existing database or validated an arbitrary +application backup. + +## Qualification + +`go test . -run '^TestLocalServer' -count=1` includes actual Nix/Docker Postgres startup, +two simultaneous independently migrated database services, exported mappings, +reader access to own data, denied cross-database connections, role collision +refusal, borrower Stop/Destroy, and data surviving a whole-server restart. +Unavailable local runtimes are explicit skips. CI runs the Docker lifecycle +with a required locally built image; a missing daemon then fails. The pure unit +cases cover missing/ambiguous dependency handoff, mismatched addresses, missing +opt-in and incompatible settings. Run with +`-race` for concurrent-bootstrap qualification. + +A standalone `codefly endpoint` invocation that cannot attach to the running flow +may fall back to the borrower's proposed port. For shared databases, consume the +accepted mappings/runtime configurations from the live composition; a deterministic +pre-run endpoint is not evidence of the actual shared listener. diff --git a/go.mod b/go.mod index db12e80..a3caf5f 100644 --- a/go.mod +++ b/go.mod @@ -11,6 +11,7 @@ require ( github.com/lib/pq v1.12.3 github.com/stretchr/testify v1.12.1 google.golang.org/grpc v1.84.0 + google.golang.org/protobuf v1.36.12 gopkg.in/yaml.v3 v3.0.1 ) @@ -120,7 +121,6 @@ require ( golang.org/x/text v0.42.0 // indirect google.golang.org/genproto/googleapis/api v0.0.0-20260928230214-8a89bd6388cc // indirect google.golang.org/genproto/googleapis/rpc v0.0.0-20260928230214-8a89bd6388cc // indirect - google.golang.org/protobuf v1.36.12 // indirect gopkg.in/evanphx/json-patch.v4 v4.13.0 // indirect k8s.io/kube-openapi v0.0.0-20260721132016-d427ff9ee9ad // indirect sigs.k8s.io/kustomize/api v0.21.1 // indirect diff --git a/local_server.go b/local_server.go new file mode 100644 index 0000000..9adf192 --- /dev/null +++ b/local_server.go @@ -0,0 +1,271 @@ +package main + +import ( + "context" + "database/sql" + "fmt" + "net/url" + "strings" + "time" + + basev0 "github.com/codefly-dev/core/generated/go/codefly/base/v0" + runtimev0 "github.com/codefly-dev/core/generated/go/codefly/services/runtime/v0" + "github.com/codefly-dev/core/resources" + "github.com/lib/pq" + "google.golang.org/protobuf/proto" +) + +const localServerConnectionKey = "local-server-connection" + +// resolveLocalServer consumes only the mappings and configuration Codefly +// supplied for a declared dependency. In particular, there is no DSN setting, +// fixed port, discovery file or second process supervisor here. +func (s *Runtime) resolveLocalServer(ctx context.Context, req *runtimev0.InitRequest) (*basev0.NetworkMapping, string, error) { + if len(strings.Split(s.LocalServer, "/")) != 2 || s.LocalServer == s.Unique() { + return nil, "", fmt.Errorf("local-server must name another declared module/service dependency") + } + if s.externalIdentity() || s.KeepRunning || s.Image != "" { + return nil, "", fmt.Errorf("local-server cannot be combined with external-identity, keep-running or docker-image: the dependency owns server authentication and execution") + } + if s.DatabaseName == "" || len(s.DatabaseName) > 63 || s.DatabaseName == "postgres" || strings.HasPrefix(s.DatabaseName, "template") { + return nil, "", fmt.Errorf("local-server requires a non-system database name of at most 63 bytes") + } + var mapping *basev0.NetworkMapping + for _, candidate := range req.DependenciesNetworkMappings { + if candidate.GetEndpoint() != nil && resources.ServiceUniqueFromEndpoint(candidate.Endpoint) == s.LocalServer { + if mapping != nil { + return nil, "", fmt.Errorf("local-server dependency must expose exactly one database endpoint") + } + mapping = candidate + } + } + if mapping == nil { + return nil, "", fmt.Errorf("local-server %q has no declared dependency mapping", s.LocalServer) + } + native := resources.FilterNetworkInstance(ctx, mapping.Instances, resources.NewNativeNetworkAccess()) + if native == nil { + return nil, "", fmt.Errorf("local-server has no native mapping for bootstrap") + } + var connection string + for _, configuration := range resources.FilterConfigurations(req.DependenciesConfigurations, resources.NewRuntimeContextNative()) { + if configuration.Origin != s.LocalServer { + continue + } + if connection != "" { + return nil, "", fmt.Errorf("local-server has ambiguous native configuration") + } + value, err := resources.GetConfigurationValue(ctx, configuration, "postgres", localServerConnectionKey) + if err != nil { + return nil, "", err + } + connection = value + } + parsed, err := url.Parse(connection) + if err != nil || parsed == nil || parsed.Scheme != "postgresql" || parsed.User == nil || parsed.Host != native.Address || parsed.Path == "" { + return nil, "", fmt.Errorf("local-server requires its owner's exported connection matching the resolved native mapping; borrowing from another logical database is not supported") + } + if parsed.User.Username() == s.postgresUser || strings.TrimPrefix(parsed.Path, "/") == s.DatabaseName { + return nil, "", fmt.Errorf("local-server requires a distinct database and POSTGRES_USER; the server owner's identity cannot be reused") + } + return mapping, connection, nil +} + +func (s *Runtime) initOnLocalServer(ctx context.Context, req *runtimev0.InitRequest) (*runtimev0.InitResponse, error) { + s.released = true // Failed Init must never permit Start against a partial schema. + if err := s.LoadConfiguration(ctx, req.Configuration); err != nil { + return s.Runtime.InitError(err) + } + mapping, bootstrap, err := s.resolveLocalServer(ctx, req) + if err != nil { + return s.Runtime.InitError(err) + } + // Keep this service's logical endpoint identity; accept the dependency's + // concrete mappings for every consumer network (native and container). + own := proto.Clone(mapping).(*basev0.NetworkMapping) + own.Endpoint = proto.Clone(s.TcpEndpoint).(*basev0.Endpoint) + s.NetworkMappings = []*basev0.NetworkMapping{own} + s.Runtime.RuntimeConfigurations = nil + for _, instance := range own.Instances { + configuration, err := s.CreateConnectionConfiguration(ctx, req.Configuration, instance, false) + if err != nil { + return s.Runtime.InitError(err) + } + s.Runtime.RuntimeConfigurations = append(s.Runtime.RuntimeConfigurations, configuration) + } + native := resources.FilterNetworkInstance(ctx, own.Instances, resources.NewNativeNetworkAccess()) + owner, err := s.createOwnerConnectionString(ctx, req.Configuration, native.Address, false) + if err != nil { + return s.Runtime.InitError(err) + } + s.connection, err = withConnectTimeout(owner, s.Timeouts.connect()) + if err != nil { + return s.Runtime.InitError(err) + } + bootstrap, err = withConnectTimeout(bootstrap, s.Timeouts.connect()) + if err != nil { + return s.Runtime.InitError(err) + } + if err = s.provisionLocalDatabase(ctx, bootstrap); err != nil { + return s.Runtime.InitError(err) + } + s.released = false + return s.Runtime.InitResponse() +} + +// The server owner remains the only lifecycle owner. A cluster-wide lock in +// that owner's database serializes role claims across concurrent borrowers; +// migrations retain their existing per-database advisory lock as well. +func (s *Runtime) provisionLocalDatabase(ctx context.Context, bootstrap string) error { + db, err := sql.Open("postgres", bootstrap) + if err != nil { + return fmt.Errorf("open local server bootstrap: %w", err) + } + defer db.Close() + conn, err := db.Conn(ctx) + if err != nil { + return fmt.Errorf("connect local server bootstrap: %w", err) + } + defer conn.Close() + lockCtx, cancelLock := context.WithTimeout(ctx, s.Timeouts.migrationLock()) + _, err = conn.ExecContext(lockCtx, `SELECT pg_advisory_lock(193, 1)`) + cancelLock() + if err != nil { + return fmt.Errorf("local-server role allocation lock: %w", err) + } + defer func() { + releaseCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + // This pool belongs only to this operation; closing it also releases the + // session lock if an expired context prevented explicit unlock. + _, _ = conn.ExecContext(releaseCtx, `SELECT pg_advisory_unlock(193, 1)`) + }() + var version int + if err = conn.QueryRowContext(ctx, `SELECT current_setting('server_version_num')::integer / 10000`).Scan(&version); err != nil { + return err + } + var expected int + if _, err = fmt.Sscanf(image.Tag, "postgres-%d", &expected); err != nil { + return err + } + if version != expected { + return fmt.Errorf("local-server PostgreSQL major %d differs from agent's required major %d", version, expected) + } + roles, err := s.localDatabaseRoles() + if err != nil { + return err + } + marker := "codefly-local-database:" + s.Unique() + ":" + s.DatabaseName + // Durable claims survive interruption between CREATE DATABASE, migrations + // and grants. A failed/dirty migration does not surrender its role names. + if _, err = conn.ExecContext(ctx, `CREATE SCHEMA IF NOT EXISTS codefly_local_server; + CREATE TABLE IF NOT EXISTS codefly_local_server.roles (name text PRIMARY KEY, owner text NOT NULL); + REVOKE ALL ON SCHEMA codefly_local_server FROM PUBLIC; + REVOKE ALL ON codefly_local_server.roles FROM PUBLIC`); err != nil { + return err + } + tx, err := conn.BeginTx(ctx, nil) + if err != nil { + return err + } + defer tx.Rollback() + for _, role := range roles { + var claim string + err = tx.QueryRowContext(ctx, `SELECT owner FROM codefly_local_server.roles WHERE name=$1`, role).Scan(&claim) + if err != nil && err != sql.ErrNoRows { + return err + } + if err == nil && claim != marker { + return fmt.Errorf("local-server role %q is already claimed by another database", role) + } + if err == sql.ErrNoRows { + var exists bool + if err = tx.QueryRowContext(ctx, `SELECT EXISTS(SELECT FROM pg_roles WHERE rolname=$1)`, role).Scan(&exists); err != nil { + return err + } + if exists { + return fmt.Errorf("local-server role %q already exists outside this database's claim", role) + } + if _, err = tx.ExecContext(ctx, `INSERT INTO codefly_local_server.roles(name, owner) VALUES ($1, $2)`, role, marker); err != nil { + return err + } + } + } + if err = tx.Commit(); err != nil { + return err + } + // Migration owners are privileged local bootstrap identities, as on the + // existing owned-server path. Application consumers still receive only the + // distinct managed reader/writer logins. Never use owner-connection in apps. + var ownerExists bool + if err = conn.QueryRowContext(ctx, `SELECT EXISTS(SELECT FROM pg_roles WHERE rolname=$1)`, s.postgresUser).Scan(&ownerExists); err != nil { + return err + } + if !ownerExists { + if _, err = conn.ExecContext(ctx, `CREATE ROLE `+pq.QuoteIdentifier(s.postgresUser)+` LOGIN SUPERUSER`); err != nil { + return err + } + } + if _, err = conn.ExecContext(ctx, `ALTER ROLE `+pq.QuoteIdentifier(s.postgresUser)+` PASSWORD `+pq.QuoteLiteral(s.postgresPassword)); err != nil { + return err + } + var owner string + err = conn.QueryRowContext(ctx, `SELECT pg_get_userbyid(datdba) FROM pg_database WHERE datname=$1`, s.DatabaseName).Scan(&owner) + if err == sql.ErrNoRows { + _, err = conn.ExecContext(ctx, `CREATE DATABASE `+pq.QuoteIdentifier(s.DatabaseName)+` OWNER `+pq.QuoteIdentifier(s.postgresUser)) + } else if err == nil && owner != s.postgresUser { + return fmt.Errorf("local-server database %q belongs to another owner", s.DatabaseName) + } + if err != nil { + return err + } + if _, err = conn.ExecContext(ctx, `REVOKE ALL ON DATABASE `+pq.QuoteIdentifier(s.DatabaseName)+` FROM PUBLIC`); err != nil { + return err + } + return s.migrateOnInit(ctx) +} + +func (s *Runtime) localDatabaseRoles() ([]string, error) { + reader, writer, err := s.runtimeAccess() + if err != nil { + return nil, err + } + roles := []string{s.postgresUser, reader, writer} + delegated, err := normalizedRuntimeReadWriteRoles(s.RuntimeReadWriteRoles, reader, writer) + if err != nil { + return nil, err + } + roles = append(roles, delegated...) + logins, err := s.runtimeLogins() + if err != nil { + return nil, err + } + for _, login := range logins { + roles = append(roles, login.role) + roles = append(roles, login.readWriteRoles...) + } + seen := map[string]bool{} + result := []string{} + for _, role := range roles { + if role == "" || len(role) > 63 || strings.HasPrefix(role, "pg_") { + return nil, fmt.Errorf("local-server role names must be non-system identifiers of at most 63 bytes") + } + if !seen[role] { + seen[role] = true + result = append(result, role) + } + } + return result, nil +} + +// Once an owned server is explicitly shared, its database must also require +// explicit CONNECT grants, just like the borrowed databases. Existing managed +// runtime grants are retained; no database or table content is deleted. +func (s *Runtime) restrictLocalDatabase(ctx context.Context) error { + db, err := sql.Open("postgres", s.connection) + if err != nil { + return err + } + defer db.Close() + _, err = db.ExecContext(ctx, `REVOKE ALL ON DATABASE `+pq.QuoteIdentifier(s.DatabaseName)+` FROM PUBLIC`) + return err +} diff --git a/local_server_test.go b/local_server_test.go new file mode 100644 index 0000000..277fcf3 --- /dev/null +++ b/local_server_test.go @@ -0,0 +1,274 @@ +package main + +import ( + "context" + "database/sql" + "github.com/codefly-dev/core/runners/dockerrun" + "net/url" + "os" + "strings" + "sync" + "testing" + "time" + + basev0 "github.com/codefly-dev/core/generated/go/codefly/base/v0" + runtimev0 "github.com/codefly-dev/core/generated/go/codefly/services/runtime/v0" + "github.com/codefly-dev/core/resources" + runners "github.com/codefly-dev/core/runners/base" + "github.com/codefly-dev/core/shared" + "github.com/stretchr/testify/require" + "google.golang.org/protobuf/proto" +) + +func localServerRequest(t *testing.T) (*Runtime, *runtimev0.InitRequest) { + t.Helper() + s := &Runtime{Service: newTestPostgresService()} + s.LocalServer = "test/server" + s.DatabaseName = "borrowed" + s.postgresUser = "borrowed_owner" + endpoint := &basev0.Endpoint{Module: "test", Service: "server", Name: "tcp", Api: "tcp"} + instance := resources.NewNetworkInstance("localhost", 12345) + instance.Access = resources.NewNativeNetworkAccess() + provider := newTestPostgresService() + provider.DatabaseName = "server" + provider.SharedLocalServer = true + c, err := provider.CreateConnectionConfiguration(context.Background(), testPostgresConfiguration("server_owner", "owner", "reader", "writer"), instance, false) + require.NoError(t, err) + c.Origin = "test/server" + return s, &runtimev0.InitRequest{DependenciesNetworkMappings: []*basev0.NetworkMapping{{Endpoint: endpoint, Instances: []*basev0.NetworkInstance{instance}}}, DependenciesConfigurations: []*basev0.Configuration{c}} +} + +func TestLocalServerRequiresResolvedOptInDependency(t *testing.T) { + cases := map[string]func(*Runtime, *runtimev0.InitRequest){ + "missing mapping": func(s *Runtime, r *runtimev0.InitRequest) { r.DependenciesNetworkMappings = nil }, + "missing configuration": func(s *Runtime, r *runtimev0.InitRequest) { r.DependenciesConfigurations = nil }, + "wrong origin": func(s *Runtime, r *runtimev0.InitRequest) { r.DependenciesConfigurations[0].Origin = "other/server" }, + "missing opt in": func(s *Runtime, r *runtimev0.InitRequest) { + r.DependenciesConfigurations[0].Infos[0].ConfigurationValues = nil + }, + "wrong address": func(s *Runtime, r *runtimev0.InitRequest) { + r.DependenciesNetworkMappings[0].Instances[0].Address = "other:12345" + }, + "no native mapping": func(s *Runtime, r *runtimev0.InitRequest) { + r.DependenciesNetworkMappings[0].Instances[0].Access = resources.NewContainerNetworkAccess() + }, + "duplicate mapping": func(s *Runtime, r *runtimev0.InitRequest) { + r.DependenciesNetworkMappings = append(r.DependenciesNetworkMappings, r.DependenciesNetworkMappings[0]) + }, + "duplicate configuration": func(s *Runtime, r *runtimev0.InitRequest) { + r.DependenciesConfigurations = append(r.DependenciesConfigurations, r.DependenciesConfigurations[0]) + }, + "same owner": func(s *Runtime, r *runtimev0.InitRequest) { s.postgresUser = "server_owner" }, + "same database": func(s *Runtime, r *runtimev0.InitRequest) { s.DatabaseName = "server" }, + "system database": func(s *Runtime, r *runtimev0.InitRequest) { s.DatabaseName = "postgres" }, + "truncated database": func(s *Runtime, r *runtimev0.InitRequest) { s.DatabaseName = strings.Repeat("a", 64) }, + "custom image": func(s *Runtime, r *runtimev0.InitRequest) { s.Image = "different:18" }, + "identity authentication": func(s *Runtime, r *runtimev0.InitRequest) { s.AuthMode = authModeExternalIdentity }, + "keep running": func(s *Runtime, r *runtimev0.InitRequest) { s.KeepRunning = true }, + "self": func(s *Runtime, r *runtimev0.InitRequest) { s.LocalServer = s.Unique() }, + } + for name, change := range cases { + t.Run(name, func(t *testing.T) { + s, r := localServerRequest(t) + change(s, r) + _, _, err := s.resolveLocalServer(context.Background(), r) + require.Error(t, err) + }) + } + s, r := localServerRequest(t) + _, connection, err := s.resolveLocalServer(context.Background(), r) + require.NoError(t, err) + require.NotEmpty(t, connection) +} + +func TestLocalServerConfigurationNeverForwardsBootstrap(t *testing.T) { + s := newTestPostgresService() + s.DatabaseName = "borrower" + s.LocalServer = "test/server" + instance := resources.NewNetworkInstance("localhost", 12345) + instance.Access = resources.NewNativeNetworkAccess() + c, err := s.CreateConnectionConfiguration(context.Background(), testPostgresConfiguration("owner", "owner-password", "reader-password", "writer-password"), instance, false) + require.NoError(t, err) + require.Empty(t, configurationValue(t, c, localServerConnectionKey)) + require.NotEmpty(t, configurationValue(t, c, readOnlyConnectionKey)) +} + +// Uses the actual generated lifecycle request/response types and exported +// dependency configurations, with one real server and two independently +// migrated logical services. No fixed database address or credential lookup. +func TestLocalServerLifecycleNix(t *testing.T) { + if !runners.CheckNixInstalled() || !runners.IsNixSupported() { + t.Skip("nix not installed/supported") + } + assertLocalServerLifecycle(t, resources.NewRuntimeContextNix()) +} + +func TestLocalServerLifecycleDocker(t *testing.T) { + _, err := dockerrun.NewDockerHeadlessEnvironment(context.Background(), testRuntimeImage(t), uniqueDockerName("shared-server-prerequisite")) + if err != nil { + if os.Getenv(testRuntimeImageEnv) == "" && dockerDaemonUnavailable(err) { + t.Skip("docker daemon unavailable") + } + t.Fatal(err) + } + assertLocalServerLifecycle(t, resources.NewRuntimeContextContainer()) +} + +func assertLocalServerLifecycle(t *testing.T, runtimeContext *basev0.RuntimeContext) { + ctx, cancel := context.WithTimeout(context.Background(), 4*time.Minute) + defer cancel() + serverFixture := newPostgresFixture(t, ctx, runtimeContext) + load := func(f *postgresFixture, localServer string) *Runtime { + s := NewRuntime() + _, err := s.Load(ctx, &runtimev0.LoadRequest{Identity: f.identity, Environment: shared.Must(f.environment.Proto()), DisableCatch: true}) + require.NoError(t, err) + s.DatabaseName = f.serviceName + s.LocalServer = localServer + s.SharedLocalServer = localServer == "" + if localServer == "" && runtimeContext.Kind == resources.RuntimeContextContainer { + s.Image = testRuntimeImage(t).String() + } + return s + } + request := func(f *postgresFixture, provider *runtimev0.InitResponse) *runtimev0.InitRequest { + r := &runtimev0.InitRequest{RuntimeContext: f.runtimeContext, Configuration: f.configuration, ProposedNetworkMappings: f.networkMappings} + if provider != nil { + r.DependenciesNetworkMappings = provider.NetworkMappings + r.DependenciesConfigurations = provider.RuntimeConfigurations + } + return r + } + initOK := func(s *Runtime, r *runtimev0.InitRequest) *runtimev0.InitResponse { + result, err := s.Init(ctx, r) + require.NoError(t, err) + require.Equal(t, runtimev0.InitStatus_READY, result.GetStatus().GetState(), result.GetStatus().GetMessage()) + return result + } + startOK := func(s *Runtime) { + r, err := s.Start(ctx, &runtimev0.StartRequest{}) + require.NoError(t, err) + require.Equal(t, runtimev0.StartStatus_STARTED, r.GetStatus().GetState(), r.GetStatus().GetMessage()) + } + server := load(serverFixture, "") + provider := initOK(server, request(serverFixture, nil)) + startOK(server) + t.Cleanup(func() { _, _ = server.Destroy(context.Background(), &runtimev0.DestroyRequest{}) }) + fixtures := []*postgresFixture{newPostgresFixture(t, ctx, runtimeContext), newPostgresFixture(t, ctx, runtimeContext)} + borrowers := make([]*Runtime, 2) + responses := make([]*runtimev0.InitResponse, 2) + errs := make([]error, 2) + for i, f := range fixtures { + f.configuration.Infos[0].ConfigurationValues[0].Value = f.serviceName + "_owner" + borrowers[i] = load(f, server.Unique()) + } + var wg sync.WaitGroup + for i := range borrowers { + wg.Add(1) + go func(i int) { + defer wg.Done() + responses[i], errs[i] = borrowers[i].Init(ctx, request(fixtures[i], provider)) + }(i) + } + wg.Wait() + const sentinel = "00000000-0000-0000-0000-000000000193" + for i, s := range borrowers { + require.NoError(t, errs[i]) + require.Equal(t, runtimev0.InitStatus_READY, responses[i].GetStatus().GetState(), responses[i].GetStatus().GetMessage()) + startOK(s) + require.Nil(t, s.nixRuntime) + require.Nil(t, s.runnerEnvironment) + native := resources.FilterNetworkInstance(ctx, responses[i].NetworkMappings[0].Instances, resources.NewNativeNetworkAccess()) + require.Equal(t, serverFixture.address, native.Address) + require.Equal(t, fixtures[i].identity.Name, responses[i].NetworkMappings[0].Endpoint.Service) + writeSentinel(t, ctx, s.connection, fixtures[i].serviceName, sentinel) + } + reader := func(i int) string { + configs := resources.FilterConfigurations(responses[i].RuntimeConfigurations, resources.NewRuntimeContextNative()) + require.Len(t, configs, 1) + return configurationValue(t, configs[0], readOnlyConnectionKey) + } + for i := range borrowers { + require.True(t, hasSentinel(t, ctx, reader(i), fixtures[i].serviceName, sentinel)) + for _, target := range []string{borrowers[1-i].DatabaseName, server.DatabaseName} { + u, err := url.Parse(reader(i)) + require.NoError(t, err) + u.Path = "/" + target + db, err := sql.Open("postgres", u.String()) + require.NoError(t, err) + err = db.PingContext(ctx) + db.Close() + require.ErrorContains(t, err, "permission denied for database") + } + } + + // Respect the existing lock budget/cancellation rather than waiting forever + // behind a peer. Releasing that peer permits an unchanged retry. + bootstrap := configurationValue(t, resources.FilterConfigurations(provider.RuntimeConfigurations, resources.NewRuntimeContextNative())[0], localServerConnectionKey) + blockerDB, err := sql.Open("postgres", bootstrap) + require.NoError(t, err) + defer blockerDB.Close() + blocker, err := blockerDB.Conn(ctx) + require.NoError(t, err) + defer blocker.Close() + _, err = blocker.ExecContext(ctx, `SELECT pg_advisory_lock(193, 1)`) + require.NoError(t, err) + retry := load(fixtures[0], server.Unique()) + short, cancelShort := context.WithTimeout(ctx, 100*time.Millisecond) + blocked, err := retry.Init(short, request(fixtures[0], provider)) + cancelShort() + require.NoError(t, err) + require.Equal(t, runtimev0.InitStatus_ERROR, blocked.GetStatus().GetState()) + require.Contains(t, blocked.GetStatus().GetMessage(), "role allocation lock") + _, err = blocker.ExecContext(ctx, `SELECT pg_advisory_unlock(193, 1)`) + require.NoError(t, err) + require.NoError(t, blocker.Close()) + require.NoError(t, blockerDB.Close()) + initOK(retry, request(fixtures[0], provider)) + _, err = retry.Destroy(ctx, &runtimev0.DestroyRequest{}) + require.NoError(t, err) + // A role collision fails before changing the peer's credentials or data. + conflicting := load(fixtures[1], server.Unique()) + conflictRequest := request(fixtures[1], provider) + conflictRequest.Configuration = proto.Clone(conflictRequest.Configuration).(*basev0.Configuration) + conflictRequest.Configuration.Infos[0].ConfigurationValues[0].Value = borrowers[0].postgresUser + refused, err := conflicting.Init(ctx, conflictRequest) + require.NoError(t, err) + require.Equal(t, runtimev0.InitStatus_ERROR, refused.GetStatus().GetState()) + require.Contains(t, refused.GetStatus().GetMessage(), "already claimed") + require.True(t, hasSentinel(t, ctx, reader(0), fixtures[0].serviceName, sentinel)) + for _, s := range borrowers { + _, err := s.Stop(ctx, &runtimev0.StopRequest{}) + require.NoError(t, err) + _, err = s.Destroy(ctx, &runtimev0.DestroyRequest{}) + require.NoError(t, err) + requireListener(t, serverFixture.address) + } + // The server owner alone releases the process; a later whole composition + // start sees both databases and their migration ledgers unchanged. + _, err = server.Stop(ctx, &runtimev0.StopRequest{}) + require.NoError(t, err) + requireNoListener(t, serverFixture.address) + server = load(serverFixture, "") + provider = initOK(server, request(serverFixture, nil)) + startOK(server) + for _, f := range fixtures { + s := load(f, server.Unique()) + initOK(s, request(f, provider)) + startOK(s) + require.True(t, hasSentinel(t, ctx, s.connection, f.serviceName, sentinel)) + _, err = s.Destroy(ctx, &runtimev0.DestroyRequest{}) + require.NoError(t, err) + } +} + +func TestLocalServerClaimsNormalizedRoleNames(t *testing.T) { + s := &Runtime{Service: newTestPostgresService()} + s.DatabaseName = "accounts" + s.postgresUser = "accounts_owner" + s.RuntimeReadWriteRoles = []string{" accounts_writer ", "accounts_writer"} + roles, err := s.localDatabaseRoles() + require.NoError(t, err) + require.Contains(t, roles, "accounts_writer") + require.NotContains(t, roles, " accounts_writer ") + require.Len(t, roles, 4) +} diff --git a/main.go b/main.go index 285a45f..17e4a0a 100644 --- a/main.go +++ b/main.go @@ -36,6 +36,13 @@ type Settings struct { DatabaseName string `yaml:"database-name"` HotReload bool `yaml:"hot-reload"` + // LocalServer is a declared Postgres dependency (module/service) whose + // server hosts this logical database during local runs. Deployment bindings + // remain governed by ExternalInstances. Empty retains the owned server. + LocalServer string `yaml:"local-server"` + // SharedLocalServer opts the owned local server into dependency sharing. + SharedLocalServer bool `yaml:"shared-local-server"` + // ExternalInstances binds named deployment environments to Postgres servers // provisioned outside this service. DatabaseName and RuntimeReadWriteRoles // attest that each environment targets this service's logical database @@ -512,6 +519,9 @@ func (s *Service) CreateConnectionConfiguration(ctx context.Context, conf *basev {Key: readOnlyConnectionKey, Value: readOnlyConnection, Secret: true}, {Key: readWriteConnectionKey, Value: readWriteConnection, Secret: true}, } + if s.SharedLocalServer && s.LocalServer == "" { + values = append(values, &basev0.ConfigurationValue{Key: localServerConnectionKey, Value: ownerConnection, Secret: true}) + } logins, err := s.runtimeLogins() if err != nil { return nil, err diff --git a/runtime.go b/runtime.go index b8efc1a..70db20a 100644 --- a/runtime.go +++ b/runtime.go @@ -169,6 +169,12 @@ func (s *Runtime) Init(ctx context.Context, req *runtimev0.InitRequest) (*runtim w := s.Wool.In("runtime::init") s.NetworkMappings = req.ProposedNetworkMappings + if s.SharedLocalServer && s.LocalServer != "" { + return s.Runtime.InitError(fmt.Errorf("shared-local-server and local-server are mutually exclusive")) + } + if s.LocalServer != "" { + return s.initOnLocalServer(ctx, req) + } configuration := req.GetConfiguration() @@ -355,7 +361,13 @@ func (s *Runtime) migrateOnInit(ctx context.Context) error { if err := s.applySchema(ctx, prerequisites); err != nil { return err } - return s.ensureRuntimeAccessLocked(ctx) + if err := s.ensureRuntimeAccessLocked(ctx); err != nil { + return err + } + if s.SharedLocalServer { + return s.restrictLocalDatabase(ctx) + } + return nil } // applySchema brings the database up to the resolved prerequisites: extensions @@ -677,6 +689,10 @@ func (s *Runtime) Stop(ctx context.Context, req *runtimev0.StopRequest) (*runtim // watcher is running. Under lifecycleMu because it writes s.watcherCancel, // which Destroy also clears. s.StopWatcher() + if s.LocalServer != "" { + s.released = true + return s.stopResponse("released logical database; dependency owns the server and retained data") + } // keep-running asks for a server the NEXT invocation reattaches to, and only // the container backend can reattach: GetContainer adopts an existing @@ -733,6 +749,10 @@ func (s *Runtime) Destroy(ctx context.Context, req *runtimev0.DestroyRequest) (* // A leaked watcher would keep firing EventHandler at a database that is // being removed. Idempotent when Stop already ran. s.StopWatcher() + if s.LocalServer != "" { + s.released = true + return s.destroyResponse("released logical database; dependency owns the server and retained data") + } if s.nixRuntime != nil { if err := s.nixRuntime.Stop(ctx); err != nil { diff --git a/templates/agent/README.md.tmpl b/templates/agent/README.md.tmpl index efd8d73..823fff1 100644 --- a/templates/agent/README.md.tmpl +++ b/templates/agent/README.md.tmpl @@ -22,7 +22,7 @@ The bootstrap job applies migrations and reconciles runtime roles and grants on ## Externally provisioned deployments -Add an entry to `external-instances` when a deployment environment uses a managed Postgres server instead of a server owned by this service. Bindings are keyed by environment so staging and production cannot silently share an address. The deployment emits the bootstrap Job but no Service, StatefulSet, or persistent volume. Local Docker and Nix runs still provision their own Postgres server. +Add an entry to `external-instances` when a deployment environment uses a managed Postgres server instead of a server owned by this service. Bindings are keyed by environment so staging and production cannot silently share an address. The deployment emits the bootstrap Job but no Service, StatefulSet, or persistent volume. Local Docker and Nix runs provision their own Postgres server by default. For local sharing, opt the owning service in with `shared-local-server: true`; a database service with a managed dependency on it can set `local-server: /`. Each borrower keeps a distinct database and migration-owner name, its migrations and runtime logins. It consumes Codefly-resolved configuration and never stops the owning server. See the agent repository’s `docs/local-shared-server.md` for composition, isolation and data migration instructions. ```yaml database-name: accounts diff --git a/workflow_test.go b/workflow_test.go index 2284d41..c484f5f 100644 --- a/workflow_test.go +++ b/workflow_test.go @@ -335,3 +335,15 @@ func TestReleaseStepPublishesBothLockedTagsPastALaggingRead(t *testing.T) { ) require.Len(t, calls, 5, "one create, then each tag read past its 404") } + +func TestCIWorkflowEnforcesSharedServerRegression(t *testing.T) { + job := readWorkflow(t, ".github/workflows/ci.yml").Jobs["image"] + buildIndex, _ := findWorkflowStepAt(t, job, "Build runtime image") + regressionIndex, step := findWorkflowStepAt(t, job, "Shared local server regression") + require.Less(t, buildIndex, regressionIndex) + require.Empty(t, step.If) + require.Equal(t, "service-postgres:test", step.Env["SERVICE_POSTGRES_TEST_IMAGE"]) + require.Contains(t, step.Run, "-race") + require.Contains(t, step.Run, "TestLocalServerLifecycleDocker") + require.Contains(t, findWorkflowStep(t, job, "Run unit tests").Run, "TestLocalServerLifecycleDocker") +}