diff --git a/.github/workflows/ansible.yml b/.github/workflows/ansible.yml index 30ab94e..7ec69e5 100644 --- a/.github/workflows/ansible.yml +++ b/.github/workflows/ansible.yml @@ -70,6 +70,7 @@ jobs: go run sigs.k8s.io/kustomize/kustomize/v5@v5.7.1 build infrastructure/security >/dev/null go run sigs.k8s.io/kustomize/kustomize/v5@v5.7.1 build infrastructure/storage >/dev/null go run sigs.k8s.io/kustomize/kustomize/v5@v5.7.1 build infrastructure/traefik >/dev/null + go run sigs.k8s.io/kustomize/kustomize/v5@v5.7.1 build infrastructure/sso >/dev/null go run sigs.k8s.io/kustomize/kustomize/v5@v5.7.1 build infrastructure/defguard >/dev/null go run sigs.k8s.io/kustomize/kustomize/v5@v5.7.1 build infrastructure/wg-portal >/dev/null for app in apps/*/; do diff --git a/README.md b/README.md index b75b924..0ee37dc 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ documented in [ansible/README.md](ansible/README.md). ``` ansible/ host provisioning and verification clusters/k3s/ Flux entry point and reconciliation order -infrastructure/ storage, Traefik, External Secrets, cloudflared, image automation, Flux UI +infrastructure/ storage, Traefik, SSO gateway, External Secrets, cloudflared, image automation, Flux UI apps// one folder per application namespace tests/ manifest checks run in CI bot-prod/ bot-rooms/ mariadb/ legacy AKS folders, disabled @@ -23,11 +23,16 @@ bot-prod/ bot-rooms/ mariadb/ legacy AKS folders, disabled denies all inbound traffic except WireGuard on UDP 48726, which reaches the WireGuard Portal Pod through a hostPort; the tunnel cannot carry it. Auth uses `AUTH_CLIENT_IP_HEADER=cf-connecting-ip` for per-address rate limits. - Cloudflare supplies that header; Traefik's default handling of `X-Forwarded-For` - does not preserve the visitor address without trusted proxy configuration. - Keep Auth behind the tunnel and ensure Cloudflare does not remove the header. + Cloudflare supplies that header. The `web` entrypoint trusts `X-Forwarded-*` + from the pod network, where cloudflared runs, but Auth keeps reading + `cf-connecting-ip`. Keep Auth behind the tunnel and ensure Cloudflare does + not remove the header. Cluster-internal IdP calls carry no visitor address and share Auth's bounded token budget of 60 requests per minute. +- **SSO gateway:** apps without their own OIDC login sit behind oauth2-proxy + ([`infrastructure/sso`](infrastructure/sso)), which Traefik calls through a + `forwardAuth` Middleware. People sign in once at `auth.polinetwork.org`, and each + app's Middleware requires one permission (`?allowed_groups=`). - **Secrets:** `ExternalSecret`s sync from Azure Key Vault (`kv-pn-apps`, `kv-pn-infra`). No secrets live in Git. - **Images:** PoliNetwork apps run `ghcr.io/polinetworkorg/:latest`. The @@ -46,6 +51,28 @@ bot-prod/ bot-rooms/ mariadb/ legacy AKS folders, disabled - **New app:** add `apps//` and `clusters/k3s/apps/.yaml`, list it in `clusters/k3s/apps/kustomization.yaml`, and allow the namespace in `infrastructure/secret-stores/` if it needs secrets. +- **Put an app behind SSO:** create an `:access` permission in Auth and grant + it to a role. Add this Middleware to `apps//`, changing only the permission: + + ```yaml + apiVersion: traefik.io/v1alpha1 + kind: Middleware + metadata: + name: sso + spec: + forwardAuth: + address: http://oauth2-proxy.sso.svc.cluster.local:4180/?allowed_groups=:access + trustForwardHeader: true + authResponseHeaders: [X-Auth-Request-User, X-Auth-Request-Email, X-Auth-Request-Preferred-Username] + # Auth rotates refresh tokens: without this, sessions break after 4 minutes. + addAuthCookiesToResponse: [__Secure-pn_sso, __Secure-pn_sso_0, __Secure-pn_sso_1, __Secure-pn_sso_2] + maxResponseBodySize: 1048576 + ``` + + Reference it from the Ingress with + `traefik.ingress.kubernetes.io/router.middlewares: -sso@kubernetescrd`, add + `infrastructure-sso` to the app's `dependsOn`, and admit only Traefik with a + NetworkPolicy. Public paths go in a separate Ingress without the Middleware. - **Status:** Flux UI at `flux.polinetwork.org`. CI (`.github/workflows/ansible.yml`) syntax-checks the playbooks, builds the diff --git a/clusters/k3s/infrastructure-sso.yaml b/clusters/k3s/infrastructure-sso.yaml new file mode 100644 index 0000000..3bce680 --- /dev/null +++ b/clusters/k3s/infrastructure-sso.yaml @@ -0,0 +1,18 @@ +apiVersion: kustomize.toolkit.fluxcd.io/v1 +kind: Kustomization +metadata: + name: infrastructure-sso + namespace: flux-system +spec: + dependsOn: + - name: infrastructure-secret-stores + - name: infrastructure-traefik + interval: 30m + retryInterval: 1m + timeout: 5m + prune: true + wait: true + sourceRef: + kind: GitRepository + name: flux-system + path: ./infrastructure/sso diff --git a/clusters/k3s/kustomization.yaml b/clusters/k3s/kustomization.yaml index 8f2e583..62b5023 100644 --- a/clusters/k3s/kustomization.yaml +++ b/clusters/k3s/kustomization.yaml @@ -10,5 +10,6 @@ resources: - infrastructure-image-automation.yaml - infrastructure-node-exporter.yaml - infrastructure-flux-web.yaml + - infrastructure-sso.yaml - infrastructure-wg-portal.yaml - apps diff --git a/infrastructure/secret-stores/cluster-secret-stores.yaml b/infrastructure/secret-stores/cluster-secret-stores.yaml index c872981..55a288c 100644 --- a/infrastructure/secret-stores/cluster-secret-stores.yaml +++ b/infrastructure/secret-stores/cluster-secret-stores.yaml @@ -38,6 +38,7 @@ spec: - cloudflared - defguard - flux-system + - sso - wg-portal provider: azurekv: diff --git a/infrastructure/sso/deployment.yaml b/infrastructure/sso/deployment.yaml new file mode 100644 index 0000000..ec5024b --- /dev/null +++ b/infrastructure/sso/deployment.yaml @@ -0,0 +1,123 @@ +# oauth2-proxy as a Traefik forwardAuth server. Traefik sends each request to +# a protected host to "/?allowed_groups=": oauth2-proxy answers 202 +# when the session holds that permission, 403 when it doesn't, and redirects to +# auth.polinetwork.org when there is no session. +# +# - Sessions live in an encrypted cookie on .polinetwork.org, so one sign-in +# covers every protected host and the replicas share no state. +# - Permissions come from polinetwork_permission_list, the array copy of +# polinetwork_permissions. groups carries roles and must not authorize. +# - ID and access tokens last 5 minutes; refreshing after 4 re-reads +# permissions, so losing one takes effect within about 5 minutes. The IdP +# rotates refresh tokens, so the Middleware must hand the refreshed cookie +# back to the browser (addAuthCookiesToResponse). +# - The web entrypoint trusts X-Forwarded-* from the pod network (cloudflared), +# which keeps the https scheme for post-login redirects. +apiVersion: v1 +kind: Service +metadata: + name: oauth2-proxy + namespace: sso +spec: + selector: + app: oauth2-proxy + ports: + - name: http + port: 4180 + targetPort: http +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: oauth2-proxy + namespace: sso +spec: + replicas: 2 + strategy: + type: RollingUpdate + rollingUpdate: + maxUnavailable: 0 + maxSurge: 1 + selector: + matchLabels: + app: oauth2-proxy + template: + metadata: + labels: + app: oauth2-proxy + spec: + securityContext: + runAsNonRoot: true + seccompProfile: + type: RuntimeDefault + containers: + - name: oauth2-proxy + # v7.15.6 + image: quay.io/oauth2-proxy/oauth2-proxy@sha256:c3415e8660af553f5e18f5d90db050614c02493c0ac6260ac59442f6d9b12048 + args: + - --http-address=0.0.0.0:4180 + - --provider=oidc + - --provider-display-name=PoliNetwork + - --oidc-issuer-url=https://auth.polinetwork.org/api/auth + - --client-id=WCfLYlJHnnpjqfPahuvhlhotivVZGLgk + - --code-challenge-method=S256 + - --scope=openid profile email polinetwork:identity offline_access + - --oidc-groups-claim=polinetwork_permission_list + - --email-domain=* + - --redirect-url=https://sso.polinetwork.org/oauth2/callback + - --whitelist-domain=.polinetwork.org + - --cookie-domain=.polinetwork.org + - --cookie-name=__Secure-pn_sso + - --cookie-secure=true + - --cookie-samesite=lax + - --cookie-expire=24h + - --cookie-refresh=4m + - --cookie-csrf-per-request=true + - --cookie-csrf-expire=10m + - --reverse-proxy=true + - --trusted-proxy-ip=10.52.0.0/16 + - --upstream=static://202 + - --skip-provider-button=true + - --set-xauthrequest=true + - --silence-ping-logging=true + env: + - name: OAUTH2_PROXY_CLIENT_SECRET + valueFrom: + secretKeyRef: + name: oauth2-proxy + key: client-secret + - name: OAUTH2_PROXY_COOKIE_SECRET + valueFrom: + secretKeyRef: + name: oauth2-proxy + key: cookie-secret + ports: + - name: http + containerPort: 4180 + startupProbe: + httpGet: + path: /ping + port: http + periodSeconds: 2 + failureThreshold: 30 + readinessProbe: + httpGet: + path: /ready + port: http + periodSeconds: 5 + livenessProbe: + httpGet: + path: /ping + port: http + periodSeconds: 10 + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + capabilities: + drop: [ALL] + resources: + requests: + cpu: 10m + memory: 32Mi + limits: + memory: 128Mi diff --git a/infrastructure/sso/external-secret.yaml b/infrastructure/sso/external-secret.yaml new file mode 100644 index 0000000..f67ea52 --- /dev/null +++ b/infrastructure/sso/external-secret.yaml @@ -0,0 +1,23 @@ +# The client secret of the "SSO Gateway" application on auth.polinetwork.org +# and the key that encrypts the session cookie (32 random bytes, URL-safe +# base64). Rotating the cookie secret signs everyone out. +apiVersion: external-secrets.io/v1 +kind: ExternalSecret +metadata: + name: oauth2-proxy + namespace: sso +spec: + refreshInterval: 1h + secretStoreRef: + kind: ClusterSecretStore + name: kv-pn-infra + target: + name: oauth2-proxy + creationPolicy: Owner + data: + - secretKey: client-secret + remoteRef: + key: sso-oidc-client-secret + - secretKey: cookie-secret + remoteRef: + key: sso-cookie-secret diff --git a/infrastructure/sso/ingress.yaml b/infrastructure/sso/ingress.yaml new file mode 100644 index 0000000..c4ae144 --- /dev/null +++ b/infrastructure/sso/ingress.yaml @@ -0,0 +1,22 @@ +# The sign-in callback and sign-out (/oauth2/sign_out) for every protected +# host. Cloudflare Tunnel sends every HTTP hostname to Traefik. +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: oauth2-proxy + namespace: sso + annotations: + traefik.ingress.kubernetes.io/router.entrypoints: web +spec: + ingressClassName: traefik + rules: + - host: sso.polinetwork.org + http: + paths: + - path: / + pathType: Prefix + backend: + service: + name: oauth2-proxy + port: + name: http diff --git a/infrastructure/sso/kustomization.yaml b/infrastructure/sso/kustomization.yaml new file mode 100644 index 0000000..b9c3fa0 --- /dev/null +++ b/infrastructure/sso/kustomization.yaml @@ -0,0 +1,13 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +# SSO gateway: oauth2-proxy behind Traefik forwardAuth, signing people in +# through auth.polinetwork.org. No app uses it yet: apps opt in with a +# forwardAuth Middleware, as "Put an app behind SSO" in the README describes. + +resources: + - namespace.yaml + - external-secret.yaml + - deployment.yaml + - network-policy.yaml + - ingress.yaml diff --git a/infrastructure/sso/namespace.yaml b/infrastructure/sso/namespace.yaml new file mode 100644 index 0000000..95c3b2d --- /dev/null +++ b/infrastructure/sso/namespace.yaml @@ -0,0 +1,4 @@ +apiVersion: v1 +kind: Namespace +metadata: + name: sso diff --git a/infrastructure/sso/network-policy.yaml b/infrastructure/sso/network-policy.yaml new file mode 100644 index 0000000..4819f10 --- /dev/null +++ b/infrastructure/sso/network-policy.yaml @@ -0,0 +1,22 @@ +# Only Traefik calls oauth2-proxy: forwardAuth checks and sso.polinetwork.org. +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: allow-traefik + namespace: sso +spec: + podSelector: + matchLabels: + app: oauth2-proxy + policyTypes: [Ingress] + ingress: + - from: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: kube-system + podSelector: + matchLabels: + app.kubernetes.io/name: traefik + ports: + - port: 4180 + protocol: TCP diff --git a/infrastructure/traefik/traefik-service.yaml b/infrastructure/traefik/traefik-service.yaml index 29fe6c8..4bd766d 100644 --- a/infrastructure/traefik/traefik-service.yaml +++ b/infrastructure/traefik/traefik-service.yaml @@ -4,7 +4,16 @@ metadata: name: traefik namespace: kube-system spec: + # Traefik is ClusterIP, so traffic only arrives from pods (10.52.0.0/16), + # in practice cloudflared. Keep the X-Forwarded-* headers they send: + # Cloudflare's X-Forwarded-Proto (https) is what the SSO gateway builds + # post-login redirects from. valuesContent: |- service: spec: type: ClusterIP + ports: + web: + forwardedHeaders: + trustedIPs: + - 10.52.0.0/16 diff --git a/tests/workload-manifests.test.rb b/tests/workload-manifests.test.rb index d02e95d..2217790 100644 --- a/tests/workload-manifests.test.rb +++ b/tests/workload-manifests.test.rb @@ -112,6 +112,43 @@ def init_container(resource, name) raise "WireGuard Portal must read #{name} from a secret" unless wg_portal_env.dig(name, "valueFrom", "secretKeyRef") end +sso_deployment = deployment("infrastructure/sso/deployment.yaml", "oauth2-proxy") +sso = container(sso_deployment, "oauth2-proxy") +sso_args = sso["args"] +sso_env = sso["env"].to_h { |item| [item["name"], item] } +raise "oauth2-proxy image must be pinned" unless sso["image"].include?("@sha256:") +raise "oauth2-proxy must expose startup, readiness, and liveness probes" unless sso["startupProbe"] && sso["readinessProbe"] && sso["livenessProbe"] +raise "oauth2-proxy must declare resources" unless sso.dig("resources", "requests") && sso.dig("resources", "limits") +raise "oauth2-proxy must drop all capabilities" unless sso.dig("securityContext", "capabilities", "drop") == ["ALL"] +%w[OAUTH2_PROXY_CLIENT_SECRET OAUTH2_PROXY_COOKIE_SECRET].each do |name| + raise "oauth2-proxy must read #{name} from a secret" unless sso_env.dig(name, "valueFrom", "secretKeyRef") +end +[ + "--code-challenge-method=S256", + "--oidc-groups-claim=polinetwork_permission_list", + "--cookie-secure=true", + "--whitelist-domain=.polinetwork.org", + "--trusted-proxy-ip=10.52.0.0/16", +].each do |arg| + raise "oauth2-proxy must run with #{arg}" unless sso_args.include?(arg) +end +raise "oauth2-proxy must refresh sessions before the 5-minute tokens expire" unless sso_args.include?("--cookie-refresh=4m") + +# Every Middleware that calls the SSO gateway must name a permission, keep the +# https scheme for post-login redirects, and hand refreshed cookies back: Auth +# rotates refresh tokens on every use. +Dir.glob(File.join(ROOT, "apps/*/*.yaml")).each do |path| + YAML.load_stream(File.read(path)).compact.each do |doc| + next unless doc["kind"] == "Middleware" + auth = doc.dig("spec", "forwardAuth") + next unless auth && auth["address"].to_s.start_with?("http://oauth2-proxy.sso.svc.cluster.local:4180/") + name = "#{path.delete_prefix("#{ROOT}/")}: #{doc.dig("metadata", "name")}" + raise "#{name} must require a permission with ?allowed_groups=" unless auth["address"].match?(/\?allowed_groups=[^&]+/) + raise "#{name} must set trustForwardHeader" unless auth["trustForwardHeader"] == true + raise "#{name} must return refreshed session cookies" unless auth["addAuthCookiesToResponse"]&.include?("__Secure-pn_sso") + end +end + claims = Dir.glob(File.join(ROOT, "apps/*/*.yaml")).flat_map do |path| YAML.load_stream(File.read(path)).compact.select { |doc| doc["kind"] == "PersistentVolumeClaim" } end