docs: troubleshoot x509 503s and document CA replacement

.cspell.json

@@ -10,6 +10,9 @@
     "apiserverapp",
     "apiserver",
     "apiservice",
+    "apiservices",
+    "kubeadm",
+    "noout",
     "apisvc",
     "clusterrole",
     "clusterrolebinding",

docs/how-to/deploy-aggregated-apiserver.md

@@ -118,13 +118,46 @@ In a cluster, the aggregated API server serves a certificate signed by its own C
 
 - The server creates the Secret on first start and reuses it afterwards. With several replicas, they all use the same Secret.
 - The serving certificate is valid for 1 year. The server checks it at startup and every 12 hours, and renews it with the same CA when less than a third of its lifetime is left. The new certificate is served without a restart.
-- The CA is valid for 10 years. To replace it, delete the Secret and restart **every** replica (`kubectl -n coder-system rollout restart deployment/coder-k8s`). The first new pod generates a CA and updates the APIService `caBundle`; until the old pods are gone, requests routed to them fail certificate verification, so expect `503 ServiceUnavailable` for several seconds (about 10 seconds with two replicas in testing). Other clients that trusted the old CA must then trust the new one.
+- The CA is valid for 10 years. To replace it earlier (for example after the Secret was exposed), see [Replace the CA](#replace-the-ca).
 - If the Secret exists but is unusable (a missing key, unparsable PEM, a key that does not match its certificate, a serving certificate not signed by the CA, or an expired CA), the server does not start and the log names the field. Fix the Secret or delete it.
 - Outside a cluster (for example `go run`), the server serves a self-signed certificate for `localhost` instead.
 
 !!! warning "The Secret holds the CA private key"
     Anyone who can read Secrets in the server's namespace can issue certificates that the aggregated API server's CA vouches for. Restrict Secret read access in `coder-system` accordingly.
 
+### Replace the CA
+
+Replacing the CA takes a restart of every replica, and requests through kube-apiserver fail with `503 ServiceUnavailable` for several seconds while it happens (about 10 seconds with two replicas in testing). Plan it as a short maintenance window.
+
+1. Note the current CA fingerprint, so you can tell the new one apart:
+
+    ```bash
+    kubectl -n coder-system get secret coder-k8s-apiserver-tls -o jsonpath='{.data.ca\.crt}' |
+      base64 -d | openssl x509 -noout -fingerprint -sha256
+    ```
+
+2. Delete the Secret and restart every replica. Do both: a running replica keeps serving the old certificate until it restarts.
+
+    ```bash
+    kubectl -n coder-system delete secret coder-k8s-apiserver-tls
+    kubectl -n coder-system rollout restart deployment/coder-k8s
+    kubectl -n coder-system rollout status deployment/coder-k8s
+    ```
+
+    The first new replica generates a new CA and sets the APIService `caBundle` to it. Requests that kube-apiserver sends to an old replica fail verification until that replica is gone; that is the `503` window.
+
+3. Check the result. The fingerprint differs from step 1, the APIService `caBundle` equals the Secret's `ca.crt` (the two commands print the same value), and a request through kube-apiserver succeeds:
+
+    ```bash
+    kubectl -n coder-system get secret coder-k8s-apiserver-tls -o jsonpath='{.data.ca\.crt}' |
+      base64 -d | openssl x509 -noout -fingerprint -sha256
+    kubectl -n coder-system get secret coder-k8s-apiserver-tls -o jsonpath='{.data.ca\.crt}'; echo
+    kubectl get apiservice v1alpha1.aggregation.coder.com -o jsonpath='{.spec.caBundle}'; echo
+    kubectl get --raw /apis/aggregation.coder.com/v1alpha1
+    ```
+
+If the APIService is [opted out](#opt-out), the server does not update the `caBundle`: set it to the new `ca.crt` yourself, or requests keep failing with `503`. Clients outside kube-apiserver that pinned the old CA must be given the new one. If requests still fail after the rollout, see [Proxied requests fail with 503 and an x509 error](troubleshooting.md#proxied-requests-fail-with-503-and-an-x509-error).
+
 ## How kube-apiserver trusts the server
 
 kube-apiserver verifies the aggregated API server's certificate against the APIService `spec.caBundle`. The aggregated API server keeps that field set to the CA in `coder-k8s-apiserver-tls`, and keeps `insecureSkipTLSVerify` off:
@@ -136,6 +169,8 @@ kube-apiserver verifies the aggregated API server's certificate against the APIS
 - On a fresh install, requests through kube-apiserver fail with `503 ServiceUnavailable` for a few seconds, until the first patch. `Available=True` alone does not show that verification works, because kube-apiserver's availability check does not verify the certificate. Check a real request instead: `kubectl get --raw /apis/aggregation.coder.com/v1alpha1`.
 - Running `kubectl apply -f deploy/apiserver-apiservice.yaml` again keeps the injected `caBundle`. Replacing or re-creating the APIService clears it; the aggregated API server sets it again within seconds.
 
+If requests through kube-apiserver fail with `503` while the APIService reports `Available=True`, see [Proxied requests fail with 503 and an x509 error](troubleshooting.md#proxied-requests-fail-with-503-and-an-x509-error).
+
 ### Upgrade from a version that used `insecureSkipTLSVerify`
 
 Apply the changes in this order:

docs/how-to/troubleshooting.md

@@ -64,6 +64,67 @@ kubectl get apiservice v1alpha1.aggregation.coder.com -o yaml
 
 Do not install CRDs for the same resources (`coderworkspaces.aggregation.coder.com`, `codertemplates.aggregation.coder.com`); they conflict with the aggregated API.
 
+## Proxied requests fail with 503 and an x509 error
+
+Symptoms: `kubectl get` on `codertemplates` or `coderworkspaces`, or `kubectl get --raw /apis/aggregation.coder.com/v1alpha1`, fails with `Error from server (ServiceUnavailable): the server is currently unable to handle the request`. The kube-apiserver log shows `error trying to reach service: tls: failed to verify certificate: x509: certificate signed by unknown authority` for `v1alpha1.aggregation.coder.com`.
+
+kube-apiserver cannot verify the aggregated API server's certificate against the APIService `caBundle`. The APIService can still report `Available=True`: kube-apiserver's availability check does not verify the certificate, so it does not rule this out. For how the `caBundle` is managed, see [How kube-apiserver trusts the server](deploy-aggregated-apiserver.md#how-kube-apiserver-trusts-the-server).
+
+Confirm the error. On kubeadm-based clusters (including Kind), kube-apiserver runs as a Pod; on managed clusters, look in your provider's control-plane logs instead.
+
+```bash
+kubectl get --raw /apis/aggregation.coder.com/v1alpha1
+kubectl -n kube-system logs -l component=kube-apiserver --since=10m --tail=-1 |
+  grep v1alpha1.aggregation.coder.com | grep x509
+```
+
+Then compare the APIService `caBundle` with the CA the server uses. The two commands must print the same value:
+
+```bash
+kubectl -n coder-system get secret coder-k8s-apiserver-tls -o jsonpath='{.data.ca\.crt}'; echo
+kubectl get apiservice v1alpha1.aggregation.coder.com -o jsonpath='{.spec.caBundle}'; echo
+```
+
+If they differ, find the cause:
+
+- **The APIService is opted out** and its `caBundle` is stale or wrong. `kubectl get apiservice v1alpha1.aggregation.coder.com -o yaml` shows `coder.com/manage-ca-bundle: "false"` under `metadata.annotations`. Either set the `caBundle` to the Secret's `ca.crt` yourself, or hand the field back to coder-k8s:
+
+    ```bash
+    kubectl annotate apiservice v1alpha1.aggregation.coder.com coder.com/manage-ca-bundle-
+    ```
+
+- **The server may not update the APIService.** The server log names the missing permission:
+
+    ```bash
+    kubectl -n coder-system logs deploy/coder-k8s | grep 'missing permission'
+    kubectl auth can-i patch apiservices.apiregistration.k8s.io/v1alpha1.aggregation.coder.com \
+      --as=system:serviceaccount:coder-system:coder-k8s
+    ```
+
+    Apply the RBAC (it must name the ServiceAccount the pod runs as); the server retries within about a minute:
+
+    ```bash
+    kubectl apply -f config/rbac/apiservice-cabundle-role.yaml
+    ```
+
+- **Another tool writes the `caBundle`,** for example cert-manager's CA injector (a `cert-manager.io/inject-ca-from` annotation) or a GitOps tool that sets `caBundle` from Git. The value keeps changing back, and the server logs `Set the APIService caBundle` again each time. Look for the other writer in the managed fields; coder-k8s writes as `coder-k8s-apiservice-cabundle`:
+
+    ```bash
+    kubectl get apiservice v1alpha1.aggregation.coder.com -o yaml --show-managed-fields
+    kubectl -n coder-system logs deploy/coder-k8s | grep -c 'Set the APIService caBundle'
+    ```
+
+    Keep one owner. Remove the other tool's `caBundle` injection, or [opt out](deploy-aggregated-apiserver.md#opt-out) and make that tool inject the `ca.crt` from `coder-k8s-apiserver-tls` (the certificate the server actually serves).
+
+If they match, you are most likely inside a **CA rotation window**: old replicas still serve a certificate from the previous CA. Wait for the rollout to finish and restart any replica that still runs from before the Secret changed:
+
+```bash
+kubectl -n coder-system rollout status deployment/coder-k8s
+kubectl -n coder-system rollout restart deployment/coder-k8s
+```
+
+See [Replace the CA](deploy-aggregated-apiserver.md#replace-the-ca) for the full procedure and the expected window.
+
 ## The pod exits with `configure delegated authentication`
 
 The aggregated API server checks every caller with the Kubernetes API and refuses to start without it.
@@ -91,6 +152,8 @@ A read or create error instead of a field name means the ServiceAccount cannot g
 
 ## Aggregated reads return `ServiceUnavailable`
 
+This section covers `ServiceUnavailable` errors that come from the aggregated API server itself. Their messages are specific, for example that no eligible `CoderControlPlane` was found, or that standalone mode is missing configuration. kube-apiserver instead returns the generic `the server is currently unable to handle the request`; for that error, see [Proxied requests fail with 503 and an x509 error](#proxied-requests-fail-with-503-and-an-x509-error).
+
 - **`all` mode:** no eligible `CoderControlPlane` exists yet, or its operator access is not ready.
 - **Standalone mode (`--app=aggregated-apiserver`):** set all three flags: `--coder-url`, `--coder-session-token`, and `--coder-namespace`.