Repository navigation
feat(cluster-secret-store): GCP Secret Manager provider - #55
Merged
Merged
Conversation
DigitalOcean has no managed secret store. Vault is the platform's intended one and arrives at wave 6, but cert-manager needs its DNS-01 token at wave 2 — and with no store to read from, the cert-manager-config chart falls to its other branch and renders that token in cleartext onto an Application spec, readable by anyone with Argo CD access. That is the circle this breaks. `provider: gcpsm` names the secret BACKEND, not the cloud the cluster runs on — the same way `aws` here means Secrets Manager and `azure` means Key Vault. A DigitalOcean cluster reading from Google sets `gcpsm`, and that case is why the value is not a cloud name. AUTHENTICATION `workloadIdentityFederation`, not `workloadIdentity`. The latter is GKE-only: it reads cluster metadata from the GCE metadata server, which a node that is not Google's does not have. Federation is what ESO documents for "AKS, EKS, self-hosted clusters, and OpenShift". No key is placed in the cluster — ESO exchanges a bound ServiceAccount token for a short-lived Google one. `gcpServiceAccountEmail` impersonates a Google service account after federating, so reads are authorised by THAT account's bindings — typically roles/secretmanager.secretAccessor scoped to a single secret — rather than on the federated principal. One stable member to write IAM against, and the grant survives the cluster being rebuilt. Optional: left empty, the federated principal must hold the permission itself. THE TWO AUDIENCES ARE DIFFERENT VALUES `gcpsm.audience` is the STS audience, `//iam.googleapis.com/...`. `gcpsm.serviceAccount.audiences` is the `aud` the projected Kubernetes token carries, which Google requires as `https://iam.googleapis.com/...` when the pool provider sets no allowedAudiences. Both are configurable because swapping them fails at the exchange, not at render, and the resulting error names neither field. The template comment and values.yaml both say so. Also documented, because it is not configurable here and refuses the exchange on its own: the pool provider's attributeCondition must admit the ServiceAccount's `sub`. That is Terraform's side and it is evaluated before IAM, so a workload missing from it is rejected while every binding it needs looks correct. VERIFIED BY RENDERING, not by reading default (no values) → azure, byte-identical to before provider=aws → unchanged provider=gcpsm → the store above, single and multi-store missing projectID → fails with the field name missing audience → fails with the field name no gcpServiceAccountEmail → field omitted, not rendered as "" vault.enabled=true → app-secret-store still renders alongside helm lint passes.
`remoteRef.key` was the flat name `cloudflare-api-token`, which made the AWS path silently wrong. platform-root has been passing kvSecrets.cloudflareApiToken with the full Secrets Manager path since v0.62.0; this template ignored it and looked up a name that only resolves against Azure Key Vault, where names are flat. Nothing errored. The ExternalSecret simply never resolved, which reads as a cert-manager problem rather than a lookup pointed at the wrong key. Same shape and same default cert-manager-config took in v0.42.11, so Azure deployments render byte-identically. default → "cloudflare-api-token" with kvSecrets → "/estabilis/prd/platform-cloudflare-api-token"
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
DigitalOcean has no managed secret store. Vault is the platform's intended one
and arrives at wave 6, but cert-manager needs its DNS-01 token at wave
2 — and with no store to read from,
cert-manager-configfalls to its otherbranch and renders that token in cleartext onto an Application spec,
readable by anyone with Argo CD access.
That circle is why this exists: ExternalSecret needs a ClusterSecretStore → on
DigitalOcean the only one this chart could build was Vault-backed → Vault is
four waves later.
provider: gcpsmproviderhere names the secret backend, not the cloud the cluster runson — the same way
awsmeans Secrets Manager andazuremeans Key Vault. ADigitalOcean cluster reading from Google sets
gcpsm; that case is why thevalue is not a cloud name.
Authentication
workloadIdentityFederation, notworkloadIdentity. The latter isGKE-only: it reads cluster metadata from the GCE metadata server, which a node
that is not Google's does not have. Federation is what ESO documents for "AKS,
EKS, self-hosted clusters, and OpenShift". No key is placed in the cluster — a
bound ServiceAccount token is exchanged for a short-lived Google one.
gcpServiceAccountEmailimpersonates a Google service account afterfederating, so reads are authorised by that account's bindings (typically
roles/secretmanager.secretAccessorscoped to a single secret) rather than onthe federated principal. One stable member to write IAM against, and the grant
survives the cluster being rebuilt. Optional — left empty, the federated
principal must hold the permission itself.
The two audiences are different values
This is the trap, so both are configurable and both are documented in the
template and in
values.yaml:gcpsm.audience//iam.googleapis.com/…gcpsm.serviceAccount.audienceshttps://iam.googleapis.com/…audof the projected Kubernetes tokenSwapping them fails at the token exchange, not at render, and the resulting
error names neither field.
Also documented because it is not configurable here and refuses the exchange on
its own: the pool provider's
attributeConditionmust admit theServiceAccount's
sub. That is Terraform's side, evaluated before IAM — aworkload missing from it is rejected while every binding it needs looks
correct.
Verified by rendering, not by reading
azure, identical to beforeprovider=awsprovider=gcpsmgcpsmmissingprojectIDgcpsmmissingaudiencegcpServiceAccountEmail""vault.enabled=trueapp-secret-storestill renders alongsidehelm lintpasses.The
azuredefault is untouched — existing deployments render exactly asbefore.
Not in this PR
platform-rootstill has to passprovider=gcpsmand thegcpsm.*parameters on the DigitalOcean branch. That is a change to
estabilis-platform, and it lands separately.