From 960173e2e805afa6f53e8d0e328a979d920f88ab Mon Sep 17 00:00:00 2001 From: Robert Shelton Date: Wed, 4 Nov 2026 09:08:23 -0800 Subject: [PATCH] Iris docs: suite operations page and M6 preflight and availability details Add operate/iris/self-managed/operations.md (upgrade, rollback, metadata Redis rotation and outage, namespace restore, service removal, uninstall) from the redis-iris M6 runbooks, and update the suite preflight and availability sections. Draft until redis-iris 1.0.0 is cut. Co-Authored-By: Claude Opus 5.5 --- content/operate/iris/self-managed/_index.md | 43 ++- .../operate/iris/self-managed/operations.md | 270 ++++++++++++++++++ 2 files changed, 306 insertions(+), 7 deletions(-) create mode 100644 content/operate/iris/self-managed/operations.md diff --git a/content/operate/iris/self-managed/_index.md b/content/operate/iris/self-managed/_index.md index 642272629e..8216a799c3 100644 --- a/content/operate/iris/self-managed/_index.md +++ b/content/operate/iris/self-managed/_index.md @@ -153,7 +153,18 @@ Use admin token mode for evaluation, or as the break-glass path when your IdP is ## 4. Run the preflight -The chart includes a preflight that checks the Kubernetes version, baseline sizing, the Metrics API, the console's IngressClass, and that the workload identity issuer is reachable. Run it with the [`kubectl preflight`](https://troubleshoot.sh/docs/#installation) plugin, v0.124.0 or later: +The chart includes a preflight that checks the cluster before an install. It reads no Secret value, prints none, and creates no permissions. + +| Check | Result | +| --- | --- | +| Kubernetes version | Fails below 1.28. | +| Baseline sizing: 3 worker nodes, with 12 vCPU and 24 GiB allocatable in total | Warns below each figure. Control-plane nodes don't count. | +| Metrics API | Warns when it's missing, because the HorizontalPodAutoscalers can't scale. | +| Ingress class, with `ui.ingress.enabled` | Fails when no IngressClass named `global.iris.ingress.className` exists. | +| Credential keys, with `global.iris.secrets.generate: false` | Fails for each enabled service whose `-dp` or `` key is missing or empty in the supplied Secrets. | +| Workload identity, for Agent Memory, and for Agent Playbooks with extraction | Fails when `global.iris.workloadIdentity.issuer` isn't the cluster's ServiceAccount issuer, or when its JSON Web Key Set (JWKS) isn't reachable without a credential. | + +Run it with the [`kubectl preflight`](https://troubleshoot.sh/docs/#installation) plugin, v0.124.0 or later. Older versions only warn on the ingress class check. The namespace must exist: ```bash helm template iris redis-ai/redis-iris --version 1.0.0 -n -f iris-values.yaml \ @@ -161,6 +172,14 @@ helm template iris redis-ai/redis-iris --version 1.0.0 -n -f ir | yq '.stringData["preflight.yaml"]' | kubectl preflight - ``` +After the install, run it again from the Secret the chart creates: + +```bash +kubectl preflight secret//iris-preflight +``` + +On Amazon EKS, set both `global.iris.workloadIdentity` fields from the cluster's OIDC issuer: `issuer` from `aws eks describe-cluster --query cluster.identity.oidc.issuer`, and `jwksURI` as `/keys`. + On clusters that refuse anonymous access to the ServiceAccount issuer, such as kubeadm clusters, the workload identity check fails and prints the binding to create. The chart creates no cluster-scoped permissions: ```bash @@ -207,17 +226,27 @@ By default, the chart generates every shared credential once, as Secrets it keep kubectl -n get secret iris-memory-admin-token -o jsonpath='{.data.token}' | base64 -d ``` -To rotate a credential, change its Secret in place. A rollback never restores an old value. To start over after `helm uninstall`, delete the Secrets yourself: - -```bash -kubectl -n delete secret -l app.kubernetes.io/component=credentials,app.kubernetes.io/name=redis-iris -``` +To rotate a credential, change its Secret in place. A rollback never restores an old value. `helm uninstall` keeps these Secrets, so a reinstall reuses them. To start over, delete them as described in [Uninstall](/content/operate/iris/self-managed/operations.md#uninstall). To supply your own credentials, for example for an offline render with Argo CD or Flux, set `global.iris.renderMode: gitops` and `global.iris.secrets.generate: false`. Then name two Secrets: `global.iris.secrets.dataplaneServiceCredentials.existingSecret`, with a `-dp` key for each installed service, and `global.iris.secrets.controlplaneInternalTokens.existingSecret`, with a `` key for each. ## Availability -The chart runs two replicas of the Identity Service, the console, and every data plane, and one of each control plane. Data planes roll one surge pod at a time and remove an old pod only when the new one is ready, so leave room for one extra pod. Every Deployment with more than one replica gets a PodDisruptionBudget with `maxUnavailable: 1`. The rollout strategy and the PodDisruptionBudgets are fixed. Replica counts are yours to size. +Rolling upgrades and node drains keep every data plane serving. + +| Setting | Value | Can you change it? | +| --- | --- | --- | +| Data-plane rollout | `RollingUpdate`, with `maxUnavailable: 0` and `maxSurge: 1`, also under a HorizontalPodAutoscaler | No. Another strategy fails the render and names the field. | +| PodDisruptionBudgets | `maxUnavailable: 1`, for each Deployment that keeps more than one pod | No. | +| Replicas | Two for the Identity Service, the console gateway and sign-in service, and the Agent Playbooks, Lens, Context Gateway, and Intent Router data planes. The Agent Memory server and worker autoscale from 2 to 10 and from 2 to 5. One for each control plane and for the Agent Playbooks and Lens workers. | Yes. Size them for your load. | + +A rollout starts one new pod, and removes an old one only when the new one is ready, so leave room for one extra pod in your requests and quota. Without that room, the rollout waits, with every old pod still serving. A stopping pod keeps accepting requests for 5 seconds after it receives `SIGTERM`, while it leaves its Services' endpoints. It then finishes the requests in flight within the rest of its 30-second grace period. + +A Deployment with one replica gets no PodDisruptionBudget, so a node drain evicts its pod. Scale it to two to keep it serving through drains. + +## Operate the suite + +To upgrade, roll back, rotate the metadata Redis credential, restore a lost namespace, remove a service, or uninstall, see [Operate the Redis Iris suite](/content/operate/iris/self-managed/operations.md). ## Move from a preview or standalone install diff --git a/content/operate/iris/self-managed/operations.md b/content/operate/iris/self-managed/operations.md new file mode 100644 index 0000000000..8b1b8dd213 --- /dev/null +++ b/content/operate/iris/self-managed/operations.md @@ -0,0 +1,270 @@ +--- +title: Operate the Redis Iris suite +description: Upgrade and roll back the redis-iris release, rotate the metadata Redis credential, recover a lost namespace, remove a service, and uninstall. +linkTitle: Operations +alwaysopen: false +categories: +- docs +- operate +- iris +weight: 10 +hideListLinks: true +--- + +These procedures apply to the whole `redis-iris` release. For one service's own operations, see its guide. The examples use release `iris` in namespace ``. + +The suite keeps no state in the cluster beyond Secrets. Data, agent keys, and queued jobs live in your Redis databases, so most procedures here come down to keeping the Secrets and the databases consistent. + +## Upgrade + +1. Read the release notes of the new chart version. +1. Run the [preflight](/content/operate/iris/self-managed/_index.md#4-run-the-preflight) with the new version and your values. A new version can run more pods. +1. Upgrade the release with the same values: + + ```bash + helm upgrade iris redis-ai/redis-iris \ + --version \ + --namespace \ + -f iris-values.yaml \ + --atomic + ``` + +1. Verify the release: + + ```bash + helm test iris --namespace --logs + ``` + +Helm upgrades every component at the same time. Upgrade within one minor version, or from the last patch of the previous minor version to the new minor version. Don't skip a minor version. + +With `--atomic`, Helm returns to the previous revision by itself when the upgrade doesn't become ready. A value that the new version refuses fails the render, names the field, and applies nothing. + +### Recover an interrupted upgrade + +An upgrade whose client was interrupted leaves the release in `pending-upgrade`, and Helm refuses every later upgrade. Find the last `deployed` revision, then roll back to it: + +```bash +helm history iris --namespace +helm rollback iris --namespace --wait +``` + +## Roll back + +```bash +helm rollback iris --namespace --wait +``` + +A rollback restores the workloads and the values of that revision. It never restores a credential: the generated Secrets aren't part of the release, so a rotated or revoked value stays rotated or revoked. It also doesn't change the keys and grants in the metadata Redis. The same version rules as for an upgrade apply. + +Keep the chart of each version you install, so you can roll back without the repository: + +```bash +helm pull redis-ai/redis-iris --version +``` + +## Rotate the metadata Redis credential + +The metadata Redis holds the Identity Service's keys, every service's metadata, the Agent Memory job queue, and the console's session store. The components read its URLs, credentials included, only at startup, so a new credential reaches them through restarts in a fixed order. Nothing is unavailable during the rotation, because both users work until you delete the old one. + +These values name the Secrets that hold the metadata Redis connection. Update every one that your values set: + +| Component | Values field | What it holds | +| --- | --- | --- | +| Identity Service | `identityService.metadata.existingSecret` | `metadata.urls` | +| Agent Memory | `memory.secrets.secretName` | The `metadata`, `databases`, and `background_jobs` URLs | +| Agent Memory queue monitor, with `memory.controlplane.queueMonitor.enabled` | `memory.controlplane.queueMonitor.redis.existingSecret` | `REDIS_URL`, the `background_jobs` Redis | +| Agent Playbooks | `playbook.storage.existingSecret` | `storage.metadata`, and `storage.content` | +| Agent Playbooks worker, with extraction | `playbook.worker.existingSecret` | `worker.redis`, the extraction queue | +| Lens | `lens.storage.existingSecret` | `storage.metadata` in both planes | +| Context Gateway | `contextGateway.metadataDB.existingSecret` | `metadata_db` | +| Intent Router | `intent.storage.existingSecret` | `storage.metadata` for the control plane, and `runtime.redis` for the data plane | +| Iris console, with IdP sign-in | `ui.sso.redis.existingSecret` | The session store URLs | + +1. Create a second database user with the same permissions as the current one. On Redis Software, add the user to the database's role. On Redis Open Source: + + ```bash + redis-cli --user --pass ACL SETUSER on '>' '~*' '&*' '+@all' + redis-cli --user --pass ACL SAVE + ``` + +1. Write the new user and password into every Secret in the table, in place, with `kubectl apply` or your secret manager. +1. Restart the components in this order, and wait for each group to finish before the next. This function skips a Deployment you didn't install, such as a disabled service, the Agent Playbooks worker without extraction, `iris-ui-auth` without IdP sign-in, or the Agent Memory queue monitor: + + ```bash + restart() { local d present=() + for d in "$@"; do kubectl -n get deployment/$d >/dev/null 2>&1 && present+=("$d"); done + for d in "${present[@]}"; do kubectl -n rollout restart deployment/$d; done + for d in "${present[@]}"; do kubectl -n rollout status deployment/$d --timeout=5m; done; } + restart iris-identity-service + restart redis-agent-memory-controlplane redis-agent-playbook-controlplane redis-lens-controlplane \ + redis-context-gateway-controlplane redis-intent-router-controlplane + restart redis-agent-memory redis-agent-memory-worker redis-agent-memory-asynqmon redis-agent-playbook-dataplane \ + redis-agent-playbook-worker redis-lens-dataplane redis-lens-worker redis-context-gateway-dataplane \ + redis-intent-router-dataplane + restart iris-ui iris-ui-auth + ``` + + Jobs queued before the restart stay queued, and run once the workers are back. + +1. Verify the release: + + ```bash + helm test iris --namespace --logs + ``` + +1. Keep the old user. Each Agent Memory store keeps the database credential it was created with, so deleting the old user breaks every store created before the rotation with `Database Authentication Failed`. + + + +## Planned outage of the metadata Redis + +For maintenance that stops the metadata Redis, such as an upgrade without replication or a host move, the suite reports the outage and recovers by itself. Nothing needs a restart. + +1. Plan the window. Queued Agent Memory and Agent Playbooks jobs stay in the database when it persists, with an append-only file or replication, and run after it returns. New writes fail during the window. +1. Stop the database. The Identity Service, the Agent Memory control plane, both Agent Playbooks planes and its worker, Intent Router, and the Lens planes turn unready, so their Services stop routing to them. Kubernetes doesn't restart the pods. The Agent Memory data plane, both Context Gateway planes, and the Lens worker stay ready, and return errors instead. +1. Start the database again. Readiness recovers without intervention, usually within a minute: + + ```bash + kubectl -n wait pod -l app.kubernetes.io/instance=iris \ + --field-selector=status.phase=Running --for=condition=Ready --timeout=5m + ``` + +1. Verify the release with `helm test iris --namespace --logs`. Queued jobs complete on their own. + + + +## Restore a lost namespace + +If the release namespace is lost, for example deleted or on a rebuilt cluster, and the databases survive, restore its Secrets and reinstall the same chart with the same values. The suite comes back with its data, and existing agent keys keep working. + +### Back up ahead of time + +Export every Secret in the release namespace, and keep the export with your values file and the installed chart archive: + +```bash +kubectl -n get secrets -o json \ + | jq '{apiVersion: "v1", kind: "List", items: [.items[] + | select(.type != "helm.sh/release.v1" and .type != "kubernetes.io/service-account-token") + | del(.metadata.uid, .metadata.resourceVersion, .metadata.creationTimestamp, .metadata.managedFields, + .metadata.ownerReferences, .metadata.namespace)]}' > iris-secrets.json +``` + +The export holds credentials in clear text, base64 encoded, so store it as you store database passwords. It covers the Secrets you supplied and the ones the chart generated. Refresh it after every credential rotation. + +### Restore + +Restore into a namespace with the same name. Workload identity trusts the ServiceAccounts of that namespace, and the services reach the Identity Service by a name that includes it. A different namespace needs a new install, not a restore. + +1. Recreate the namespace: + + ```bash + kubectl create namespace + ``` + +1. Apply the export: + + ```bash + kubectl -n apply -f iris-secrets.json + ``` + +1. Install the same chart version with the same values. The chart finds its generated credentials and reuses them: + + ```bash + helm install iris ./redis-iris-.tgz \ + --namespace \ + -f iris-values.yaml \ + --wait --timeout 10m + ``` + +1. Run `helm test iris --namespace --logs`, then read a known record with an existing agent key. + +If the generated credential Secrets are lost too, the databases still hold the data, but the services and the Identity Service no longer share their runtime credentials and internal tokens. Install again with the same values: the chart generates new credentials, and agent keys keep working, because the Identity Service keeps them in its database. + +## Remove a service permanently + +Turning a service off with `global.iris.products..enabled: false` keeps its credentials and its data, so turning it on again continues where it stopped. To remove a service for good, so that a new install of it starts empty with new credentials: + +1. Set `global.iris.products..enabled: false`, and run `helm upgrade`. +1. Delete its generated Secrets, where `` is `memory`, `playbook`, `lens`, `context-gateway`, or `intent`: + + ```bash + kubectl -n delete secret \ + iris--dataplane-credential iris--internal-token iris--admin-token + ``` + +1. Delete its data by key pattern, on every Redis database that its Secrets name. + + | Service | Key patterns, with the default prefixes | Where | + | --- | --- | --- | + | Agent Memory | `iris:memory:*`, `{iris:memory:*` | The metadata database and the job idempotency URL | + | | `memory:*`, `memory-write:*`, `{memory-write:*` | Its store databases, from `databases` | + | | `asynq:{ram:*` | Its job queue | + | Agent Playbooks | `iris:playbook:*` | The metadata and content databases | + | | `asynq:{playbook-extraction:*`, `playbook-queue-admission:{playbook-extraction:*` | The worker's queue | + | Lens | `iris:lens:*` | The metadata database | + | | `lens:*` | The streams, content, and index databases | + | Context Gateway | `iris:context-gateway:*` | The metadata database | + | | `context-gateway/*` | The credential storage Redis, with `credentialStorage.provider: redis` | + | Intent Router | `iris:intent:*` | The metadata database, the mailbox, and the Router databases | + + A database can hold other services' data too. Never delete `iris:*` or `asynq:*`: they include the Identity Service's keys and the other services' queues. Flush or delete a whole database only after you check that no other component's Secret names it. + + Delete the keys one per command, with the same URL that the Secret holds. A clustered database refuses a multi-key delete across slots with `CROSSSLOT`, which would leave data behind: + + ```bash + R=(-u 'rediss://:@:/') + redis-cli "${R[@]}" --scan --pattern '' | xargs -r -n 1 redis-cli "${R[@]}" unlink + ``` + + - On a Redis Open Source cluster, `--scan` sees only the node it connects to, so run it against each primary. + - For a queue, also remove its names from the shared queue set. List them with `redis-cli "${R[@]}" smembers asynq:queues`, then run `redis-cli "${R[@]}" srem asynq:queues ` for each `ram:*` (Agent Memory) or `playbook-extraction:*` (Agent Playbooks) name. + - Deleting keys leaves search indexes in place. List them with `redis-cli "${R[@]}" FT._LIST`, then run `redis-cli "${R[@]}" FT.DROPINDEX ` for each index that starts with one of the service's prefixes. + +1. For Context Gateway with `credentialStorage.provider` set to `vault` or `aws_secrets_manager`, also delete its upstream credentials there. They're named `/tenant//gateway//credentials/`, where `` is `credentialStorage.basePath`: + + - **Vault:** every secret under `/` in the KV v2 mount, `credentialStorage.vault.mount`. List them with `vault kv list -mount= /tenant/` and the levels below it, and delete each one with `vault kv metadata delete -mount= `. + - **AWS Secrets Manager:** every secret whose name starts with `/`, in `credentialStorage.aws.region`: + + ```bash + aws secretsmanager list-secrets --region --filters Key=name,Values=/ \ + --query 'SecretList[].Name' --output text | tr '\t' '\n' \ + | xargs -r -n 1 aws secretsmanager delete-secret --region --force-delete-without-recovery --secret-id + ``` + +The agent keys of a removed service stay in the Identity Service. [Revoke them](/content/operate/iris/identity-service/_index.md) if you don't need them. + +## Uninstall + +`helm uninstall` removes the release's workloads, Services, Ingresses, and configuration. It keeps the credential Secrets the chart generated, so that a reinstall keeps every service and agent key working, and the pods and permissions of the last `helm test`. It never touches your own Secrets or your databases. + +1. Uninstall the release: + + ```bash + helm uninstall iris --namespace --wait --timeout 5m + ``` + + Wait until only the `Completed` `helm test` pods are left: + + ```bash + kubectl -n get pods -l app.kubernetes.io/instance=iris + ``` + +1. List what's left: + + ```bash + kubectl -n get secrets -l app.kubernetes.io/instance=iris,app.kubernetes.io/component=credentials + kubectl -n get pods,serviceaccounts,roles,rolebindings,configmaps -l app.kubernetes.io/instance=iris + ``` + + To reinstall later with the same credentials, stop here. + +1. To remove everything, delete it: + + ```bash + kubectl -n delete secrets -l app.kubernetes.io/instance=iris,app.kubernetes.io/component=credentials + kubectl -n delete pods,serviceaccounts,roles,rolebindings,configmaps -l app.kubernetes.io/instance=iris + ``` + + A new install then generates new admin tokens, a new control token, and new service credentials. Agent keys live in the Identity Service's database, not in these Secrets. + +1. The databases keep the suite's data. Delete it with your database tooling when you decommission the install. The Secrets you supplied, such as the database URLs, the license, and the IdP client, are yours to delete too.