Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/ansible.yml
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,7 @@ jobs:
go run sigs.k8s.io/kustomize/kustomize/v5@v5.7.1 build infrastructure/security >/dev/null
go run sigs.k8s.io/kustomize/kustomize/v5@v5.7.1 build infrastructure/storage >/dev/null
go run sigs.k8s.io/kustomize/kustomize/v5@v5.7.1 build infrastructure/traefik >/dev/null
go run sigs.k8s.io/kustomize/kustomize/v5@v5.7.1 build infrastructure/sso >/dev/null
go run sigs.k8s.io/kustomize/kustomize/v5@v5.7.1 build infrastructure/defguard >/dev/null
go run sigs.k8s.io/kustomize/kustomize/v5@v5.7.1 build infrastructure/wg-portal >/dev/null
for app in apps/*/; do
Expand Down
35 changes: 31 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ documented in [ansible/README.md](ansible/README.md).
```
ansible/ host provisioning and verification
clusters/k3s/ Flux entry point and reconciliation order
infrastructure/ storage, Traefik, External Secrets, cloudflared, image automation, Flux UI
infrastructure/ storage, Traefik, SSO gateway, External Secrets, cloudflared, image automation, Flux UI
apps/<ns>/ one folder per application namespace
tests/ manifest checks run in CI
bot-prod/ bot-rooms/ mariadb/ legacy AKS folders, disabled
Expand All @@ -23,11 +23,16 @@ bot-prod/ bot-rooms/ mariadb/ legacy AKS folders, disabled
denies all inbound traffic except WireGuard on UDP 48726, which reaches the
WireGuard Portal Pod through a hostPort; the tunnel cannot carry it.
Auth uses `AUTH_CLIENT_IP_HEADER=cf-connecting-ip` for per-address rate limits.
Cloudflare supplies that header; Traefik's default handling of `X-Forwarded-For`
does not preserve the visitor address without trusted proxy configuration.
Keep Auth behind the tunnel and ensure Cloudflare does not remove the header.
Cloudflare supplies that header. The `web` entrypoint trusts `X-Forwarded-*`
from the pod network, where cloudflared runs, but Auth keeps reading
`cf-connecting-ip`. Keep Auth behind the tunnel and ensure Cloudflare does
not remove the header.
Cluster-internal IdP calls carry no visitor address and share Auth's bounded
token budget of 60 requests per minute.
- **SSO gateway:** apps without their own OIDC login sit behind oauth2-proxy
([`infrastructure/sso`](infrastructure/sso)), which Traefik calls through a
`forwardAuth` Middleware. People sign in once at `auth.polinetwork.org`, and each
app's Middleware requires one permission (`?allowed_groups=<permission>`).
- **Secrets:** `ExternalSecret`s sync from Azure Key Vault (`kv-pn-apps`,
`kv-pn-infra`). No secrets live in Git.
- **Images:** PoliNetwork apps run `ghcr.io/polinetworkorg/<app>:latest`. The
Expand All @@ -46,6 +51,28 @@ bot-prod/ bot-rooms/ mariadb/ legacy AKS folders, disabled
- **New app:** add `apps/<ns>/` and `clusters/k3s/apps/<ns>.yaml`, list it in
`clusters/k3s/apps/kustomization.yaml`, and allow the namespace in
`infrastructure/secret-stores/` if it needs secrets.
- **Put an app behind SSO:** create an `<app>:access` permission in Auth and grant
it to a role. Add this Middleware to `apps/<ns>/`, changing only the permission:

```yaml
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
name: sso
spec:
forwardAuth:
address: http://oauth2-proxy.sso.svc.cluster.local:4180/?allowed_groups=<app>:access
trustForwardHeader: true
authResponseHeaders: [X-Auth-Request-User, X-Auth-Request-Email, X-Auth-Request-Preferred-Username]
# Auth rotates refresh tokens: without this, sessions break after 4 minutes.
addAuthCookiesToResponse: [__Secure-pn_sso, __Secure-pn_sso_0, __Secure-pn_sso_1, __Secure-pn_sso_2]
maxResponseBodySize: 1048576
```

Reference it from the Ingress with
`traefik.ingress.kubernetes.io/router.middlewares: <ns>-sso@kubernetescrd`, add
`infrastructure-sso` to the app's `dependsOn`, and admit only Traefik with a
NetworkPolicy. Public paths go in a separate Ingress without the Middleware.
- **Status:** Flux UI at `flux.polinetwork.org`.

CI (`.github/workflows/ansible.yml`) syntax-checks the playbooks, builds the
Expand Down
18 changes: 18 additions & 0 deletions clusters/k3s/infrastructure-sso.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: infrastructure-sso
namespace: flux-system
spec:
dependsOn:
- name: infrastructure-secret-stores
- name: infrastructure-traefik
interval: 30m
retryInterval: 1m
timeout: 5m
prune: true
wait: true
sourceRef:
kind: GitRepository
name: flux-system
path: ./infrastructure/sso
1 change: 1 addition & 0 deletions clusters/k3s/kustomization.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -10,5 +10,6 @@ resources:
- infrastructure-image-automation.yaml
- infrastructure-node-exporter.yaml
- infrastructure-flux-web.yaml
- infrastructure-sso.yaml
- infrastructure-wg-portal.yaml
- apps
1 change: 1 addition & 0 deletions infrastructure/secret-stores/cluster-secret-stores.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ spec:
- cloudflared
- defguard
- flux-system
- sso
- wg-portal
provider:
azurekv:
Expand Down
123 changes: 123 additions & 0 deletions infrastructure/sso/deployment.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,123 @@
# oauth2-proxy as a Traefik forwardAuth server. Traefik sends each request to
# a protected host to "/?allowed_groups=<permission>": oauth2-proxy answers 202
# when the session holds that permission, 403 when it doesn't, and redirects to
# auth.polinetwork.org when there is no session.
#
# - Sessions live in an encrypted cookie on .polinetwork.org, so one sign-in
# covers every protected host and the replicas share no state.
# - Permissions come from polinetwork_permission_list, the array copy of
# polinetwork_permissions. groups carries roles and must not authorize.
# - ID and access tokens last 5 minutes; refreshing after 4 re-reads
# permissions, so losing one takes effect within about 5 minutes. The IdP
# rotates refresh tokens, so the Middleware must hand the refreshed cookie
# back to the browser (addAuthCookiesToResponse).
# - The web entrypoint trusts X-Forwarded-* from the pod network (cloudflared),
# which keeps the https scheme for post-login redirects.
apiVersion: v1
kind: Service
metadata:
name: oauth2-proxy
namespace: sso
spec:
selector:
app: oauth2-proxy
ports:
- name: http
port: 4180
targetPort: http
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: oauth2-proxy
namespace: sso
spec:
replicas: 2
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 0
maxSurge: 1
selector:
matchLabels:
app: oauth2-proxy
template:
metadata:
labels:
app: oauth2-proxy
spec:
securityContext:
runAsNonRoot: true
seccompProfile:
type: RuntimeDefault
containers:
- name: oauth2-proxy
# v7.15.6
image: quay.io/oauth2-proxy/oauth2-proxy@sha256:c3415e8660af553f5e18f5d90db050614c02493c0ac6260ac59442f6d9b12048
args:
- --http-address=0.0.0.0:4180
- --provider=oidc
- --provider-display-name=PoliNetwork
- --oidc-issuer-url=https://auth.polinetwork.org/api/auth
- --client-id=WCfLYlJHnnpjqfPahuvhlhotivVZGLgk
- --code-challenge-method=S256
- --scope=openid profile email polinetwork:identity offline_access
- --oidc-groups-claim=polinetwork_permission_list
- --email-domain=*
- --redirect-url=https://sso.polinetwork.org/oauth2/callback
- --whitelist-domain=.polinetwork.org
- --cookie-domain=.polinetwork.org
- --cookie-name=__Secure-pn_sso
- --cookie-secure=true
- --cookie-samesite=lax
- --cookie-expire=24h
- --cookie-refresh=4m
- --cookie-csrf-per-request=true
- --cookie-csrf-expire=10m
- --reverse-proxy=true
- --trusted-proxy-ip=10.52.0.0/16
- --upstream=static://202
- --skip-provider-button=true
- --set-xauthrequest=true
- --silence-ping-logging=true
env:
- name: OAUTH2_PROXY_CLIENT_SECRET
valueFrom:
secretKeyRef:
name: oauth2-proxy
key: client-secret
- name: OAUTH2_PROXY_COOKIE_SECRET
valueFrom:
secretKeyRef:
name: oauth2-proxy
key: cookie-secret
ports:
- name: http
containerPort: 4180
startupProbe:
httpGet:
path: /ping
port: http
periodSeconds: 2
failureThreshold: 30
readinessProbe:
httpGet:
path: /ready
port: http
periodSeconds: 5
livenessProbe:
httpGet:
path: /ping
port: http
periodSeconds: 10
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop: [ALL]
resources:
requests:
cpu: 10m
memory: 32Mi
limits:
memory: 128Mi
23 changes: 23 additions & 0 deletions infrastructure/sso/external-secret.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# The client secret of the "SSO Gateway" application on auth.polinetwork.org
# and the key that encrypts the session cookie (32 random bytes, URL-safe
# base64). Rotating the cookie secret signs everyone out.
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: oauth2-proxy
namespace: sso
spec:
refreshInterval: 1h
secretStoreRef:
kind: ClusterSecretStore
name: kv-pn-infra
target:
name: oauth2-proxy
creationPolicy: Owner
data:
- secretKey: client-secret
remoteRef:
key: sso-oidc-client-secret
- secretKey: cookie-secret
remoteRef:
key: sso-cookie-secret
22 changes: 22 additions & 0 deletions infrastructure/sso/ingress.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# The sign-in callback and sign-out (/oauth2/sign_out) for every protected
# host. Cloudflare Tunnel sends every HTTP hostname to Traefik.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: oauth2-proxy
namespace: sso
annotations:
traefik.ingress.kubernetes.io/router.entrypoints: web
spec:
ingressClassName: traefik
rules:
- host: sso.polinetwork.org
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: oauth2-proxy
port:
name: http
13 changes: 13 additions & 0 deletions infrastructure/sso/kustomization.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

# SSO gateway: oauth2-proxy behind Traefik forwardAuth, signing people in
# through auth.polinetwork.org. No app uses it yet: apps opt in with a
# forwardAuth Middleware, as "Put an app behind SSO" in the README describes.

resources:
- namespace.yaml
- external-secret.yaml
- deployment.yaml
- network-policy.yaml
- ingress.yaml
4 changes: 4 additions & 0 deletions infrastructure/sso/namespace.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
apiVersion: v1
kind: Namespace
metadata:
name: sso
22 changes: 22 additions & 0 deletions infrastructure/sso/network-policy.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# Only Traefik calls oauth2-proxy: forwardAuth checks and sso.polinetwork.org.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-traefik
namespace: sso
spec:
podSelector:
matchLabels:
app: oauth2-proxy
policyTypes: [Ingress]
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
app.kubernetes.io/name: traefik
ports:
- port: 4180
protocol: TCP
9 changes: 9 additions & 0 deletions infrastructure/traefik/traefik-service.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,16 @@ metadata:
name: traefik
namespace: kube-system
spec:
# Traefik is ClusterIP, so traffic only arrives from pods (10.52.0.0/16),
# in practice cloudflared. Keep the X-Forwarded-* headers they send:
# Cloudflare's X-Forwarded-Proto (https) is what the SSO gateway builds
# post-login redirects from.
valuesContent: |-
service:
spec:
type: ClusterIP
ports:
web:
forwardedHeaders:
trustedIPs:
- 10.52.0.0/16
37 changes: 37 additions & 0 deletions tests/workload-manifests.test.rb
Original file line number Diff line number Diff line change
Expand Up @@ -112,6 +112,43 @@ def init_container(resource, name)
raise "WireGuard Portal must read #{name} from a secret" unless wg_portal_env.dig(name, "valueFrom", "secretKeyRef")
end

sso_deployment = deployment("infrastructure/sso/deployment.yaml", "oauth2-proxy")
sso = container(sso_deployment, "oauth2-proxy")
sso_args = sso["args"]
sso_env = sso["env"].to_h { |item| [item["name"], item] }
raise "oauth2-proxy image must be pinned" unless sso["image"].include?("@sha256:")
raise "oauth2-proxy must expose startup, readiness, and liveness probes" unless sso["startupProbe"] && sso["readinessProbe"] && sso["livenessProbe"]
raise "oauth2-proxy must declare resources" unless sso.dig("resources", "requests") && sso.dig("resources", "limits")
raise "oauth2-proxy must drop all capabilities" unless sso.dig("securityContext", "capabilities", "drop") == ["ALL"]
%w[OAUTH2_PROXY_CLIENT_SECRET OAUTH2_PROXY_COOKIE_SECRET].each do |name|
raise "oauth2-proxy must read #{name} from a secret" unless sso_env.dig(name, "valueFrom", "secretKeyRef")
end
[
"--code-challenge-method=S256",
"--oidc-groups-claim=polinetwork_permission_list",
"--cookie-secure=true",
"--whitelist-domain=.polinetwork.org",
"--trusted-proxy-ip=10.52.0.0/16",
].each do |arg|
raise "oauth2-proxy must run with #{arg}" unless sso_args.include?(arg)
end
raise "oauth2-proxy must refresh sessions before the 5-minute tokens expire" unless sso_args.include?("--cookie-refresh=4m")

# Every Middleware that calls the SSO gateway must name a permission, keep the
# https scheme for post-login redirects, and hand refreshed cookies back: Auth
# rotates refresh tokens on every use.
Dir.glob(File.join(ROOT, "apps/*/*.yaml")).each do |path|
YAML.load_stream(File.read(path)).compact.each do |doc|
next unless doc["kind"] == "Middleware"
auth = doc.dig("spec", "forwardAuth")
next unless auth && auth["address"].to_s.start_with?("http://oauth2-proxy.sso.svc.cluster.local:4180/")
name = "#{path.delete_prefix("#{ROOT}/")}: #{doc.dig("metadata", "name")}"
raise "#{name} must require a permission with ?allowed_groups=" unless auth["address"].match?(/\?allowed_groups=[^&]+/)
raise "#{name} must set trustForwardHeader" unless auth["trustForwardHeader"] == true
raise "#{name} must return refreshed session cookies" unless auth["addAuthCookiesToResponse"]&.include?("__Secure-pn_sso")
end
end

claims = Dir.glob(File.join(ROOT, "apps/*/*.yaml")).flat_map do |path|
YAML.load_stream(File.read(path)).compact.select { |doc| doc["kind"] == "PersistentVolumeClaim" }
end
Expand Down
Loading