feat(supervisor): recover shared instances with hash-based HA

.agents/skills/helm-dev-environment/SKILL.md

@@ -68,8 +68,9 @@ gatewayConfig:
     supervisor_topology: gateway
 ```
 
-The gateway chart creates a single-replica shared supervisor Deployment,
-Service, NetworkPolicy, and an empty `<release>-supervisor-token` Secret in
+The gateway chart creates a shared supervisor Deployment (`supervisor.replicas`,
+default one), Service, NetworkPolicy, and an empty
+`<release>-supervisor-token` Secret in
 the release namespace, regardless of shared/managed/operator workspace mode.
 The gateway mints a supervisor token from its sandbox-JWT signing key, writes
 it into that Secret (the gateway Role may `get`/`patch` only that Secret), and
@@ -96,6 +97,88 @@ OPENSHELL_E2E_KUBE_IMAGE_PULL_SECRET=e2e-regcred \
   mise run e2e:kubernetes:shared-supervisor
 ```
 
+### Shared supervisor high availability
+
+The shared supervisor is always a pool of `supervisor.replicas` members
+(default 1). With the gateway topology the chart renders the gateway's
+`[openshell.gateway.supervisor_pool]` tuning table (only
+`recovery_timeout_secs` from `supervisor.recoveryTimeoutSeconds`, default 60;
+the driver's `shared_supervisor` table stays unchanged), a
+RollingUpdate Deployment (`maxSurge: 1`, `maxUnavailable: 0`) with preferred
+hostname anti-affinity and a 60-second grace period, the pool identity env vars
+(`OPENSHELL_SUPERVISOR_POOL_MEMBER_ID`, `_MEMBER_UID`, `_ADVERTISE_IP` from the
+Pod name, UID, and IP), and a `maxUnavailable: 1` PodDisruptionBudget.
+`ci/values-supervisor-ha.yaml` selects two replicas and a 60-second recovery
+timeout. The topology needs gateway TLS: the e2e harness keeps it on with
+`ci/values-tls-port-forward.yaml` and `OPENSHELL_E2E_KUBE_TLS=1`, which reaches
+the gateway over mTLS through its port-forward. `mise run
+e2e:kubernetes:shared-supervisor-ha` sets both, creates a task-scoped k3d
+cluster, kills the supervisor Pod that owns a sandbox, and checks that the
+sandbox keeps its Pod, generation, main process, and in-memory state.
+
+The rendered gateway configuration and supervisor Pod template do not
+depend on `supervisor.replicas`; `deploy/helm/test-supervisor-pool-scaling.sh`
+(part of `mise run helm:test`) renders a pool at 1, 2, and 3 replicas and
+fails if anything but the replica count differs. `mise run
+e2e:kubernetes:shared-supervisor-scaling` installs a one-member pool
+(`ci/values-supervisor-ha-scaling.yaml`), then the test runs `helm upgrade
+--reuse-values --set supervisor.replicas=2` and back to 1 against the chart
+the harness installed (`OPENSHELL_E2E_KUBE_CHART`), probing exec and
+policy-mediated networking throughout. It checks that scaling up moves
+nothing, that the member removed by scaling down hands its sandboxes to the
+other (`cause=leaving`), that no workload, gateway Pod, or remaining member
+restarts, and that a new sandbox runs after each step. It upgrades its
+release, so it runs on a release of its own; the two pool tests also share a
+nextest test group that never runs them at once. Against an existing cluster
+whose kubeconfig the shell environment may override, set
+`OPENSHELL_E2E_KUBECONFIG` (passed as `--kubeconfig` to kubectl and helm).
+
+On k3d, deploy the pool with the Skaffold profile, which layers the overlay and
+turns gateway TLS back on over `ci/values-skaffold.yaml`:
+
+```bash
+cd deploy/helm/openshell
+KUBECONFIG=../../../kubeconfig skaffold run -p shared-supervisor-high-availability
+cd ../../..
+```
+
+The plaintext registration from `helm:skaffold:run` does not apply. Keep
+`mise run helm:k3s:forward` running and set up the client certificate as in
+"With mTLS enabled" under
+[Connecting through the forwarding task](#connecting-through-the-forwarding-task).
+Both members share the single k3d node, so exercise Pod failures:
+
+```bash
+export KUBECONFIG=kubeconfig OPENSHELL_GATEWAY_ENDPOINT=https://localhost:8090
+openshell sandbox create --detach --name ha-probe -- sh -c 'echo $$ > /sandbox/main.pid; exec sleep infinity'
+openshell sandbox exec --name ha-probe --no-tty -- cat /sandbox/main.pid
+openshell sandbox get ha-probe  # note the sandbox ID
+# Supervisor logs name the instance by its route, not its sandbox ID.
+kubectl get sandboxes.agents.x-k8s.io -A -l openshell.ai/sandbox-name=ha-probe \
+  -o jsonpath='{.items[0].metadata.annotations.openshell\.ai/sandbox-runtime-supervisor-session}'
+# The member whose latest admission, removal, or release line names the route owns it.
+kubectl -n openshell logs -l app.kubernetes.io/name=openshell-supervisor --prefix --tail=-1 \
+  | grep -E '(admitted assigned|removing withdrawn|relinquishing a) sandbox instance.* <route>$'
+# Force-delete the owner, the closest local stand-in for a crash.
+kubectl -n openshell delete pod <owner-pod> --grace-period=0 --force
+kubectl -n openshell logs -l app.kubernetes.io/name=openshell,app.kubernetes.io/instance=openshell \
+  -c openshell-gateway --prefix --tail=200 | grep 'taking over a shared sandbox instance'
+kubectl -n openshell logs -l openshell.ai/sandbox-id=<sandbox-id>,openshell.ai/boundary-role=workload \
+  --tail=200 | grep -E 'sandbox-supervisor-|Supervisor takeover confirmed'
+openshell sandbox exec --name ha-probe --no-tty -- cat /sandbox/main.pid  # same PID
+```
+
+A plain `kubectl delete pod` exercises the graceful leave instead: the member
+announces it is leaving and the gateway hands its instances over at once. A
+member that stops without announcing it, as on node loss, is declared dead
+only after the 15-second membership TTL.
+`kubectl -n openshell rollout restart deployment/openshell-supervisor` replaces
+every member one at a time. The PID must survive each case, while an open
+`sandbox connect` or `exec` stream breaks and must reconnect. A
+`sandbox-supervisor-recovery-expired` finding in the workload log means no
+replacement confirmed within the recovery timeout, so the runtime terminated
+the workload.
+
 ### 2. Deploy OpenShell
 
 **Iterative dev** (rebuilds on file changes, recommended during active development):
@@ -498,6 +581,8 @@ for dependencies still declared in `Chart.yaml`.
 | `deploy/helm/openshell/ci/values-cert-manager.yaml` | cert-manager PKI overlay (opt-in; disables pkiInitJob) |
 | `deploy/helm/openshell/ci/values-gateway.yaml` | Envoy Gateway GRPCRoute + Gateway overlay |
 | `deploy/helm/openshell/ci/values-high-availability.yaml` | HA test overlay (`replicaCount: 2` with external PostgreSQL Secret) |
+| `deploy/helm/openshell/ci/values-supervisor.yaml`, `ci/values-supervisor-ha.yaml`, `ci/values-supervisor-ha-scaling.yaml` | Shared supervisor pool overlays: one member, two members (Skaffold profile `shared-supervisor-high-availability`), or one member to scale to two and back |
+| `deploy/helm/openshell/ci/values-tls-port-forward.yaml` | Keeps gateway TLS on over the Skaffold values for e2e topologies that need it (`OPENSHELL_E2E_KUBE_TLS=1`) |
 | `deploy/helm/openshell/ci/values-autoscaling.yaml` | Render-only overlay for the optional gateway HorizontalPodAutoscaler (helm lint and helm-unittest) |
 | `deploy/helm/openshell/ci/values-keycloak.yaml` | Keycloak OIDC overlay |
 | `deploy/helm/openshell/ci/values-spire.yaml` | SPIFFE/SPIRE provider token grant overlay |

.config/nextest.toml

@@ -26,11 +26,26 @@ slow-timeout = { period = "60s", terminate-after = 5 }
 # names because binary() fails in workspaces without the HA test binary.
 [test-groups]
 kubernetes-ha = { max-threads = 1 }
+kubernetes-shared-supervisor-pool = { max-threads = 1 }
 
 [[profile.e2e-kubernetes.overrides]]
 filter = "test(/gateway_(scale_and_rollout|pod_rolls)$/)"
 test-group = "kubernetes-ha"
 
+# Both change the shared supervisor pool of the release they run against: one
+# kills a member, the other scales the pool. Never run them at once. Test
+# names, not binary(), for the reason above.
+[[profile.e2e-kubernetes.overrides]]
+filter = "test(/^(losing_a_supervisor_pod_moves_its_sandboxes_without_restarting_them|scaling_a_supervisor_pool_up_and_down_keeps_its_sandboxes_running)$/)"
+test-group = "kubernetes-shared-supervisor-pool"
+
+# Scaling the pool runs two bounded Helm upgrades (up to 6 minutes each), waits
+# for each member to join or leave, and creates sandboxes until one lands on
+# the new member, so it gets 20 minutes instead of five.
+[[profile.e2e-kubernetes.overrides]]
+filter = "test(/^scaling_a_supervisor_pool_up_and_down_keeps_its_sandboxes_running$/)"
+slow-timeout = { period = "60s", terminate-after = 20 }
+
 # Relative to the profile store dir (`e2e/rust/target/nextest/e2e-kubernetes/`).
 [profile.e2e-kubernetes.junit]
 path = "../../../../../results/e2e-kubernetes.xml"

TESTING.md

@@ -578,6 +578,9 @@ Available task variants:
 | `e2e:kubernetes:workspace-operator` | Operator workspace mode (pre-provisioned namespaces) |
 | `e2e:kubernetes:v1alpha1` | Agent Sandbox v1alpha1 compatibility |
 | `e2e:kubernetes:external-driver` | External Kubernetes driver sidecar |
+| `e2e:kubernetes:shared-supervisor` | One-member shared supervisor pool for the gateway's sandboxes (gateway TLS through the port-forward) |
+| `e2e:kubernetes:shared-supervisor-ha` | Two-replica shared supervisor pool: kills the member that owns a sandbox and checks the sandbox is adopted, not restarted |
+| `e2e:kubernetes:shared-supervisor-scaling` | One-member shared supervisor pool: scales it to two members and back with `helm upgrade`, checking sandboxes stay running and the removed member hands its sandboxes off |
 
 Kubernetes e2e environment variables:
 
@@ -586,6 +589,8 @@ Kubernetes e2e environment variables:
 | `OPENSHELL_E2E_KUBE_CONTEXT` | kubectl context for an existing cluster (skips k3d creation) |
 | `OPENSHELL_E2E_KUBE_TEST` | Scope to a single test (e.g. `smoke`) |
 | `OPENSHELL_E2E_KUBE_EXTRA_VALUES` | Colon-separated additional Helm values files |
+| `OPENSHELL_E2E_KUBE_CHART`, `OPENSHELL_E2E_KUBECONFIG` | Set by the harness for tests that upgrade the release, such as the supervisor pool scaling test: the chart it installed, and the kubeconfig passed as `--kubeconfig` to kubectl and helm. Set them yourself when running such a test directly against an existing release |
+| `OPENSHELL_E2E_KUBE_TLS` | `1` reaches the gateway over mTLS through the port-forward; pair it with an overlay that keeps gateway TLS on, such as `ci/values-tls-port-forward.yaml` |
 | `OPENSHELL_E2E_KUBERNETES_FEATURES` | Cargo feature flags (default: `e2e,e2e-host-gateway,e2e-kubernetes`) |
 | `IMAGE_TAG` | Gateway/supervisor image tag (default: `latest` for existing clusters) |
 | `OPENSHELL_REGISTRY` | Image registry prefix (default: `ghcr.io/nvidia/openshell`) |