[antoinetoussaint-byte]

Why

On the obin-blue cell, the tracing evidence ClickHouse password reached Cloud Logging: an old migrate CLI printed its DSN. That was fixed upstream in service-clickhouse v0.0.25. The password still has to be rotated, and codefly had no way to do it. The plan kept every stored value (pkg/deploysecrets/plan.go: "a value is never rotated here"). The consumer's …__CLICKHOUSE__CONNECTION URL has no producer template, so the plan only reported it as require. That left it typed by hand, with nothing to keep it in step with the password.

Design

codefly deploy secrets --rotate <module>/<service>/<configuration>:<KEY> (for blue: tracing/evidence/clickhouse:CLICKHOUSE_PASSWORD) (also <configuration>:<KEY> for a workspace configuration, or the CODEFLY__…_SECRET_CONFIGURATION__… key itself; the flag can be repeated). The selector is spelled like the service-secrets.generate entry that declares the key.

• Narrow: only a key that service-secrets.generate declares can be rotated. Each of these is refused with its reason:
  • a supplied (require) key
  • a configured key (change the file instead)
  • an ed25519-pkcs8 signing key (its public half is registered elsewhere)
  • an identifier key (a new user name in the store renames nobody on the server)
  • a key no rendered ExternalSecret reads
  • a holder or derived value outside --module
  • a configuration read through more than one store (a plan sees one store)
• One value everywhere: the new value is minted the way the generator mints it, and written to every remote key that holds the key. The holder-agreement check runs first, so a store that already disagrees with itself is refused before anything is rotated.
• No silent overwrite: before each write, the key's newest version (now reported by Describe, --sort-by=~createTime) is compared with the one the plan read, and a key that moved is refused by name. Secret Manager has no conditional versions add, so this narrows the race; it does not close it.
• Writes: each write is a new store version (gcloud secrets versions add --format=value(name)), so nothing is destroyed. Store.Write now returns the version. The result prints Wrote <key> as version <n> (<properties>), with no values.
• Dry run: --dry-run shows rotate rows with property names and sources only. --metadata-only shows the rotated key's rows. Derived rows need the store to be read, and a metadata-only plan can never be applied.
• Idempotence: after a successful rotation, a plain deploy secrets re-run writes nothing. Running --rotate again rotates again; that is documented.

What is derived, and how

I reused the template seam: a producer that declares its connection as a ConfigurationValue.template gets it assembled by External Secrets in the cluster. Nothing is stored in that case, so a rotated primitive is picked up on the next sync and there is nothing to rewrite.

For a connection stored by hand (blue's case), the derivation is read from the store, not guessed from a key name:

• During a rotation, every stored property in the plan is parsed.
• If the property is a URL whose decoded userinfo password is the old value, the password segment of its userinfo is replaced, encoded the way net/url encodes userinfo. Every other byte is kept as stored; the URL is not re-serialised.
  • The result is re-parsed: user, host, path, query and fragment must be unchanged, and the password must equal the new value.
  • The rewritten URL must not hold the old value in any form.
• Any other stored value that holds the old value, raw or percent-encoded (raw, QueryEscape, PathEscape, userinfo escaping), is refused, naming remote-key#property (key). Rotating would otherwise leave it stale.
• Not seen: a copy in another store, or anywhere no rendered ExternalSecret reads. The docs say so.

Only rotation does this scan, and only on stored values in memory. A refusal never quotes a value.

Mid-apply failure: GCP Secret Manager has no transaction across secrets. A rotation that fails part-way writes the previous document of every key it already wrote back as a new version, in reverse order; a key it created gets {}.

• The restore runs on its own context (WithoutCancel plus a 2-minute timeout).
• The failing key is restored too when its write landed anyway, which is detected by the store holding exactly the document this rotation sent.
• A key that moved since this rotation wrote it is never overwritten.
• So no half-rotated pair is left, and the same command can be run again (the test shows it converges). If a restore fails too, the error names the keys that may still hold the rotated document and tells the operator to re-add their previous versions; a plain re-run would not converge from there.
• Plans with no rotation keep their existing write behaviour, plus the version check.

The store is not the server's credential. Whether a restart makes the server accept the new value depends on the agent:

• The ClickHouse agent's entrypoint rewrites users.d from CLICKHOUSE_USER/CLICKHOUSE_PASSWORD on every start. I checked this in service-clickhouse main.go at fe09e97.
• A Postgres-style image applies the password only when it first creates the data directory, so there the operator must change it on the server.

No agent declaration says which behaviour an agent has; I checked core's protos. So this is documented, not enforced.

Operator follow-up (documented; the CLI does none of it): force-sync the affected ExternalSecrets, make the server accept the new credential, then restart its consumers. Undo is to re-add each key's previous version as current (a gcloud … access | … add --data-file=- pipe, printing nothing).

Classification

Fix, at the place that owns the behaviour. pkg/deploysecrets is the only writer of the environment's secret store. Rotation is a new resolution in the same plan; it reuses the generator index, alias groups, scope test and the Store seam rather than adding a second mechanism. I deliberately did not teach the CLI any producer-specific connection format (ClickHouse, Postgres). The one generic rule is "a URL whose password equals the old value", and everything else is refused.

Tests (fake store, pkg/deploysecrets/rotate_test.go)

• TestRotateReplacesAGeneratedKeyAndTheConnectionDerivedFromIt (blue's shape):
  • the password rotates and the stored URL is parsed: its password equals the new stored password, and user, host, path and query are unchanged (compared without printing values)
  • the result lists key, properties and versions
  • the plan's report holds neither the old nor the new value
  • a plain re-run writes nothing
• TestRotateRefusesAKeyCodeflyDoesNotProduce: supplied, configured, signing and identifier keys are each refused with their reason, and nothing is written.
• TestRotateRefusesADerivedValueItCannotKeepConsistent: a value that embeds the password other than as a URL password is refused by name, and the refusal carries no value. A derived value outside --module is refused.
• TestRotateFindsAPercentEncodedPassword: a base64 password with / and +, stored with userinfo escaping and with query escaping, is found and rewritten consistently, and found again by a second rotation. This test fails if the old raw pre-filter is put back.
• TestRotateRestoresAFailedWriteThatLanded and TestRotateRefusesAKeyThatMovedSinceThePlan.
• TestRotateMetadataOnlyShowsRowsWithoutReading: shows rotate rows with no reads, no values, and no way to apply.
• TestRotateFailingPartWayLeavesNothingHalfRotated: the consumer write fails, the evidence document is restored as version 2, the pair is consistent, and a retry converges.
• TestRotationKeyResolvesTheGeneratorSpelling, TestCheckRotationStoresRefusesAConfigurationSpanningStores, and cmd/deploy TestRotationKeysResolveAgainstTheRender.
• TestGoogleSecretManagerWritesThroughStdin now also checks the version is parsed from versions add.
• I checked that the derived-rewrite and failure tests fail when the rewrite and the restore are disabled.

go test ./pkg/deploysecrets/... ./cmd/... passes and make lint reports 0 issues.

Docs

• docs/commands.md: a rotate row in the action table, a new "Rotating a generated key" section with the operator follow-up, and the "never rotated" rule now notes the --rotate exception.
• docs/deployment-secret-safety.md
• docs/development-loop.md
• docs/cli-reference.md, regenerated
• the command's --help

Not verified

• Nothing was run against GCP Secret Manager or the blue cell. I did not run these gcloud behaviours here: versions add --format=value(name) printing the version resource name, and versions list --sort-by=~createTime --limit=1 returning the newest enabled version.
• I did not read blue's real rendered ExternalSecrets or stored documents. The tests model their shape from the description: evidence document with PASSWORD/USER, consumer document with the CONNECTION URL. Rotating a USER key is refused if it is identifier-format, and also refused if the URL embeds it as the username.
• I did not check whether CLICKHOUSE_PASSWORD is declared in blue's service-secrets.generate. If it is not, the rotation is refused, and the declaration has to be added first.
• The force-sync and restart steps are documented, not automated.

🤖 Generated with Claude Code