feat: verify the aggregated API server certificate through the APIService caBundle

[ThomasK33]

Refs #137. Step 2 of 3.

The aggregated API server now puts its serving CA into the APIService caBundle and turns insecureSkipTLSVerify off, so kube-apiserver verifies the server's certificate. The certificate itself comes from #140.

What changed

• New apiservicetrust controller in the aggregated API server process. It runs only when the managed serving certificate is in use, so never with --app=controller and never outside a Pod.
  • It watches only v1alpha1.aggregation.coder.com, using a metadata.name field selector, and reacts to CA changes. It does not poll.
  • It sends one merge patch, {caBundle, insecureSkipTLSVerify:false}, only when something differs, with FieldManager coder-k8s-apiservice-cabundle.
  • The CA comes from the live Secret after the startup checks pass, so every replica writes the same bytes and invalid bytes are never written.
  • Failures never affect serving or readiness. A missing permission is logged at most every 5 minutes, and retries back off to at most 60 s.
• Opt-out: coder.com/manage-ca-bundle=false on the APIService, for clusters where cert-manager or GitOps owns caBundle.
• RBAC: config/rbac/apiservice-cabundle-role.yaml grants get/list/watch/patch on that one APIService, limited by resourceNames. It is left out of dist/install.yaml (the controller-only bundle), which is byte-identical.
• Manifest: deploy/apiserver-apiservice.yaml no longer sets insecureSkipTLSVerify: true.
• Docs: how kube-apiserver trusts the server, upgrade order, rollback, the opt-out, and restarting every replica after a CA rotation.
• CI: both checks send a real request through kube-apiserver.
  • Positive: caBundle equals the Secret CA, and kubectl get --raw /apis/aggregation.coder.com/v1alpha1 succeeds.
  • Negative: revoke the sync's binding, set a wrong CA, and require a 503 plus kube-apiserver's x509 "unknown authority" log line. Then restore the binding and require recovery.

Finding: Available does not show that verification works

With a wrong caBundle, the APIService stayed Available=True ("all checks passed"), while proxied requests failed with 503 and kube-apiserver logged an x509 error. So neither check relies on Available.

Risks

• A CA rotation (delete the Secret, restart every replica) causes about 10 s of 503s on proxied requests while old pods drain. Renewing the serving certificate causes none.
• If the new image rolls out before its RBAC, it logs the missing permission (rate-limited) and the APIService keeps working without verification until the RBAC is applied.
• Applying the new manifest before the new image gives 503s until the image runs. The docs give the order to follow.
• The sync would fight any other tool that sets caBundle; the opt-out annotation covers that.

Validation

• make verify-vendor, build, lint, test, test-integration, docs-check, markdownlint, cspell, actionlint, and the installer freshness check all pass.
• 12 of 12 mutation checks were killed, including RBAC without resourceNames, no field selector, ignoring the opt-out, writing unvalidated bytes, and a missing FieldManager.
• An envtest proves the resourceNames list/watch works with the field selector, and that drift is repaired.
• Kind v1.32 dogfood covered all three upgrade orders, rollback, the opt-out, and the CI steps copied from ci.yaml (the x509 line appeared in 4 of 4 runs). A 2-replica CA rotation produced exactly one patch and no flapping.

optout-ci.mp4

Next

Step 3 adds an x509 troubleshooting entry and more rotation docs; #137 will be closed after it lands. Follow-ups: a namespaced Role for the serving-CA Secret, and bring-your-own Secret support.

---

Generated with xum • Model: anthropic:claude-opus-5-5 • Thinking: medium

[ThomasK33]

@codex security review

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-25T11:24:26.614996Z	17a93c2	PR opened
🔒 Security Review	✅ Completed 2026-09-25T11:26:58.349423Z	17a93c2	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: 17a93c2121

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

• You comment "@codex security review"
• A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.