Skip to content

feat: run the standalone aggregated API server as its own ServiceAccount - #147

Merged
ThomasK33 merged 1 commit into
mainfrom
feat/apiserver-standalone-sa
Sep 25, 2026
Merged

ThomasK33 merged 1 commit into
mainfrom
feat/apiserver-standalone-sa

Conversation

@ThomasK33

@ThomasK33 ThomasK33 commented Sep 25, 2026 •

Copy link
Copy Markdown
Member

Refs #143. Step 3 of 3.

A standalone aggregated API server (--app=aggregated-apiserver) now runs as its own ServiceAccount, coder-k8s-apiserver, with only the access that mode uses. Before this PR it ran as coder-k8s, whose manager-role allows every verb on Secrets in every namespace. --app=all and the controller-only dist/install.yaml are unchanged; dist/install.yaml is byte-identical.

What changed

  • config/apiserver-standalone/ (new; outside config/rbac/, so the controller bundle does not pick it up):
    • ServiceAccount coder-k8s-apiserver.
    • A Role with only get and update on coder-k8s-apiserver-tls (resourceNames). There is no create, list or watch: namespace-wide create would let this identity mint tokens for other ServiceAccounts in the namespace.
    • The empty placeholder Secret, which the server fills (feat: fill an empty placeholder serving-CA Secret instead of creating it #146).
    • Bindings for delegated authentication and authorization (system:auth-delegator, and extension-apiserver-authentication-reader in kube-system), plus a binding to the existing APIService caBundle ClusterRole.
  • Docs (deploy-aggregated-apiserver.md, troubleshooting.md, architecture.md):
    • Option B uses the new ServiceAccount and gives the apply order. A section explains how to move an existing standalone server off coder-k8s; the existing Secret and CA are reused.
    • The "Who can read the CA key" table is updated.
    • "Replace the CA" and corrupt-Secret recovery now cover standalone mode: clear data, or delete the Secret and re-apply the placeholder.
    • Troubleshooting names the standalone bindings and the Forbidden-create path.
  • Error text: the corrupt-Secret error now also says to re-apply the placeholder when the identity may not create Secrets.

Tests

standalone_rbac_envtest_test.go runs against a real kube-apiserver (envtest v1.35) and impersonates coder-k8s-apiserver. Only the shipped manifests are applied, with no manager-role. It uses the production TLS and delegated-auth paths and checks that:

  • the server fills the placeholder, renews it, and adopts the other writer's CA after a real 409 conflict;
  • it patches its APIService, creates a TokenReview, and serves an authorized request through delegated auth (a SubjectAccessReview);
  • 10 other actions return 403: other Secrets, list, watch, cluster-wide list, create, delete, patch, ConfigMaps, and Deployments;
  • changing the Secret's type returns 422 (field is immutable), and an update of an absent Secret returns 404. Startup then fails with a Forbidden create that names the placeholder.

Mutation checks caught 14 of 15 mutations: dropping resourceNames, removing update, adding delete/patch, adding separate create/list/watch rules, wrong subjects in each binding, an extra cluster-wide Secret reader, wrong or non-empty placeholders, and the old error text. The one not caught adds create or list/watch inside the resourceNames rule, which does not widen access.

Kind proof (Kubernetes v1.32, image from this tree)

  • Upgrade from coder-k8s: the CA fingerprint and data digest were unchanged through a client-side apply of the placeholder, a server-side apply, and the ServiceAccount switch.
  • 5 minutes as coder-k8s-apiserver: 0 "forbidden" log lines, 0 restarts, and proxied discovery works. The can-i matrix passed 17 of 17.
  • CA replacement: both ways (clearing data, and delete plus re-apply) produced a new CA with a matching caBundle, and discovery worked.
  • Pod before placeholder: the pod exits with a Forbidden create that names the placeholder, then becomes Ready once the placeholder is applied.

Upgrade from coder-k8s keeps the CA

can-i matrix for coder-k8s-apiserver

CA replacement in standalone mode

Pod started before the placeholder recovers

upgrade.webm

Validation

On the exact tree 8aa09c98, these pass: make verify-vendor, build, test (including the new envtest), lint, test-integration, make manifests (tree stays clean), mkdocs build --strict, markdownlint and cspell.


Generated with xum • Model: anthropic:claude-opus-5-5 • Thinking: medium

Add config/apiserver-standalone/: the coder-k8s-apiserver
ServiceAccount, a Role with get and update on coder-k8s-apiserver-tls
only, the delegated authentication and authorization bindings, the
APIService caBundle binding, and the empty placeholder Secret the
server fills. No create on Secrets: namespace-wide create would let
this identity mint tokens for other ServiceAccounts in the namespace.
Standalone mode needs nothing from manager-role; dist/install.yaml is
unchanged.

An envtest runs the server's clients as that ServiceAccount against a
real kube-apiserver with only these manifests: it fills, renews and
adopts after a real conflict, patches the APIService, serves an
authorized request through delegation, and gets 403 for every other
Secret access, 422 for a type change and 404 for an update of the
absent Secret.

The docs move Option B to the new identity, describe the apply order
and the upgrade from coder-k8s, and split CA replacement and corrupt
Secret recovery by mode. The corrupt-Secret error now says to re-apply
the placeholder when the identity may not create Secrets.

Refs #143

Change-Id: I3c24ee1c1ed3d55427d76106a3226ebf2d404af8
Signed-off-by: Thomas Kosiewski <tk@coder.com>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 25, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-25T14:12:53.499758Z 79cea75 PR opened
🔒 Security Review ✅ Completed 2026-09-25T14:13:33.558825Z 79cea75 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@ThomasK33

Copy link
Copy Markdown
Member Author

@codex security review

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: 79cea7590a

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@ThomasK33
ThomasK33 added this pull request to the merge queue Sep 25, 2026
Merged via the queue into main with commit 8aadab6 Sep 25, 2026
13 checks passed
@ThomasK33
ThomasK33 deleted the feat/apiserver-standalone-sa branch September 25, 2026 14:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant