Skip to content

feat(regulations): deploy regulations-collab and latex-worker, publish the backend's /public prefix - #32

Merged
lorenzocorallo merged 4 commits into
mainfrom
feat/regulations-services
Nov 15, 2026
Merged

lorenzocorallo merged 4 commits into
mainfrom
feat/regulations-services

Conversation

@lorenzocorallo

@lorenzocorallo lorenzocorallo commented Nov 2, 2026 •

Copy link
Copy Markdown
Member

Deployment for the regulations editor (shared spec, decision 4 and the "Deployment" and "Public backend routes" contracts), aligned with the backend stack merged as PoliNetworkOrg/backend#110–#117 (a048a04).

Merging this is safe before any manual prerequisite exists. Nothing that needs a missing secret or image runs, fails or alerts until Lorenzo creates it, and then it starts by itself. That was verified on a throwaway k3s at prod's version with the repo's real Flux setup; details below.

What it deploys

Flux Kustomization Contents Starts when
backend (existing, from the backend ResourceSet) backend env for the editor, all secrets as optional refs; the /public route; Traefik → backend:3000 in allow-callers immediately: one rollout, always healthy
regulations-secrets (new, plain) every regulations ExternalSecret, plus the latex-worker namespace applied immediately; Ready only once the core entries exist (wait: false + healthChecks on the two core ExternalSecrets)
regulations-collab (new, from the backend ResourceSet, so the same image digest) Hocuspocus: Deployment, Service, Ingress collab.polinetwork.org, NetworkPolicy after regulations-secrets and backend are Ready
latex-worker (new ResourceSet) worker Deployment, Service, NetworkPolicy, LimitRange, ResourceQuota after its image is pullable and regulations-secrets is Ready

The secrets are split so that only the core ones gate anything:

  • regulations-core (backend namespace): internal-secret, worker-secret.
  • regulations (latex-worker namespace): worker-secret.
  • regulations-github: private-key, webhook-secret.
  • regulations-llm: api-key.

The GitHub and LLM entries can arrive weeks later without blocking anything.

The latex-worker-image provider sits with its ResourceSet under the root sync, which doesn't wait for health, instead of in infrastructure-image-automation (which does). A missing or private image therefore reports an error only on that one provider.

Services (unchanged from the reviewed design)

  • regulations-collab
    • Runs the backend image with bun run collab.js.
    • 4100 is the public websocket, reachable from Traefik only; 4101 is the internal API, reachable from the backend only.
    • Egress is limited to DNS, PostgreSQL and the worker.
    • uid 1000, read-only root, single replica with Recreate, REGULATIONS_COMPILE_CONCURRENCY=2.
  • latex-worker
    • Port 4200, ingress only from the backend and collab, no egress at all.
    • PSA restricted, read-only root, tmpfs /work 512Mi and /tmp 64Mi.
    • uid 10200, used by nothing else on the node.
    • Limits 1500m / 1536Mi, plus a namespace LimitRange and ResourceQuota.
    • Probes GET /health.
  • /public
    • One IngressRoute: Host(`api.polinetwork.org`) && PathPrefix(`/public/`) && PathRegexp(`^/public(/[A-Za-z0-9_-]+)+$`).
    • Middlewares: 1 MiB body limit, a per-Cf-Connecting-Ip and a total rate limit, and the X-PoliNetwork-Ingress: public marker the backend relies on.
    • The webhook is POST /public/webhooks/github. Polling every 5 min (REGULATIONS_RECONCILE_SECONDS) is reconciliation only.

Tests (tests/workload-manifests.test.rb)

The tests enforce the sandbox, the exposure and the gate:

  • the backend reads every regulations Secret only through optional refs, and apps/backend holds no ExternalSecret except azure-kv;
  • every required secret of collab and the worker comes from an ExternalSecret that regulations-secrets health-checks;
  • both depend on regulations-secrets, and the backend doesn't;
  • regulations-secrets is listed in clusters/k3s/apps/kustomization.yaml;
  • in the backend namespace, exactly one router reaches backend-service: the /public route, with exactly that rule and those middlewares;
  • the path whitelist rejects .., %2e%2e, %2f, empty segments, a trailing slash and backslashes.

Each assertion was mutation-checked (break the manifest, the test must fail).

What happens at merge, before the prerequisites exist

These are the conditions the orchestrator described: no regulations-* Key Vault entries, no latex-worker image (the backend's latex-worker workflow currently skips because legal-texlive is unreadable, confirmed on run a048a04), no Cloudflare hostname or Access bypass, no GitHub App.

Verified on k3s v1.36.3+k3s1 with flux-operator v0.58.1, Flux 2.9.4 and the FluxInstance from ansible/roles/flux, syncing this repo from GitHub, first main and then this branch. The real ESO 2.12.0 HelmRelease ran with fake ClusterSecretStores standing in for Key Vault: same names and namespace conditions, every existing key, no regulations-*. The real Kustomization objects from clusters/k3s were applied for security, storage, traefik, external-secrets, image-automation, postgres, redis, backend, latex-worker and regulations-secrets. The production backend image ran as Flux resolved it.

  • Backend:

    • One rolling update to the new env, Kustomization Ready, GET / still hi (what Kuvasz checks).
    • An in-cluster prober saw 1 failed request in 490: the second the old Pod was stopped. Every backend release does the same today, since the Deployment has no preStop delay.
  • regulations-secrets: not Ready, "health check failed … ExternalSecret/backend/regulations-core". regulations-collab: "dependency 'regulations-secrets' is not ready". No collab or worker Pod, Deployment or Service exists, so PodNotRunning, DeploymentReplicasMismatch and PodCrashLooping have nothing to fire on.

  • latex-worker-image provider: not Ready (GHCR DENIED). The latex-worker ResourceSet is Ready and generates nothing. infrastructure-image-automation stays Ready; before the provider was moved, it went not-Ready, and that is fixed in this PR.

  • Every other Kustomization (security, storage, traefik, external-secrets, secret-stores, image-automation, postgres, redis, backend, root) is Ready on the new revision. Nothing dependsOn the new Kustomizations, and no change rolls back anything.

  • Receiver: a signed GitHub package event still annotated and reconciled every provider, including those listed after the failing one.

  • /public in this state:

    • Through Traefik as api.polinetwork.org, only /public/... reaches the backend, and the current image answers it with its own 404 because it has no /public routes.
    • /, /trpc, /internal/events, /public/../trpc, /public/%2e%2e/trpc and other hosts get Traefik's 404.
    • In prod, Cloudflare Access still sends every api.polinetwork.org request to its login (302) until the bypass exists.
  • Image first, then secrets: I made the worker's provider resolve an existing package before any secret existed. The ResourceSet generated the latex-worker Kustomization, which waited on regulations-secrets, and no worker object was created.

  • Self-activation: after adding the two core entries to the store, with no other action:

    Step Time
    ESO synced them (after backing off since the merge) +379 s
    The gate opened +384 s
    regulations-collab and latex-worker applied +416 s

    The GitHub and LLM ExternalSecrets stayed failing and blocked nothing.

  • Backend restart: the backend Pod that predated the secret lacked REGULATIONS_INTERNAL_SECRET. One rollout restart brought both core variables in, and GET / still answered hi.

Reasoned, not reproduced:

  • No paging. Alertmanager sends every alert to Telegram. All rules are about nodes, Pods, Deployments, PVCs, scrape targets and backups. Flux has no Alert or Provider objects, and no rule reads the flux-operator or ESO metrics, so a red Kustomization, provider or ExternalSecret pages no one.
  • Kuvasz only probes the backend's / (hi), which stays up. Collab and the worker aren't monitored.
  • Cloudflare: with no DNS for collab.polinetwork.org, the collab Ingress is unreachable. The /public route stays behind Access until the bypass.
  • Test artifacts:
    • After activation, collab crashed with Module not found "collab.js", because the test ran the previous production backend image. Prod will have run a048a04 long before. It does mean the core entries must not be created while an image without collab.js is live (checklist step 2).
    • The worker showed ErrImagePull only because of my test redirection to another package.

Lorenzo's checklist after the merge

  1. Check that it landed (no action): the backend rolled once and is healthy. regulations-secrets shows "ExternalSecret/backend/regulations-core" not healthy, regulations-collab waits on it, and the latex-worker-image provider is red. That's expected until the steps below.

  2. Core secrets. First make sure the live backend image includes the regulations code (backend main ≥ a048a04; its Docker workflow must be green). Then create in kv-pn-apps:

    • regulations-internal-secret = openssl rand -hex 32
    • regulations-worker-secret = openssl rand -hex 32

    Collab starts by itself within about 10 minutes (ESO backoff). To skip the wait:

    kubectl -n backend annotate externalsecret regulations-core force-sync="$(date +%s)" --overwrite && kubectl -n latex-worker annotate externalsecret regulations force-sync="$(date +%s)" --overwrite

  3. Restart the backend once, so it reads those secrets: kubectl -n backend rollout restart deployment/backend.

  4. Cloudflare tunnel: add collab.polinetwork.org to the k3s01 tunnel → Traefik, like the other hostnames. Keep WebSockets on and no Access. The editor's live sync then works.

  5. Worker image:

    • Make sure legal-texlive is built for amd64 and arm64 and includes fontconfig (the worker Dockerfile runs fc-cache).
    • On the legal-texlive package: Package settings → Manage Actions access → add PoliNetworkOrg/backend with Read.
    • Run the backend's "LaTeX worker image" workflow, or wait for its hourly run.
    • Once ghcr.io/polinetworkorg/latex-worker exists, set that package to public; the cluster pulls without credentials.

    Flux picks it up within 6 h, or immediately with kubectl -n flux-system annotate resourcesetinputprovider latex-worker-image reconcile.fluxcd.io/requestedAt="$(date +%s)" --overwrite, and the worker starts by itself. PDF previews then work.

  6. GitHub App on PoliNetworkOrg:

    • Permissions: contents and pull requests write, metadata read. Installed only on legal.
    • Webhook URL https://api.polinetwork.org/public/webhooks/github, content type JSON, events Pull request, Check suite and Release.
    • In kv-pn-apps, create regulations-github-app-private-key (the PEM, via az keyvault secret set --file) and regulations-github-webhook-secret.
  7. Cloudflare Access: add an Access application for api.polinetwork.org/public/ with Bypass / Everyone, and keep the existing one on the rest of the host.

  8. LLM: create regulations-llm-api-key in kv-pn-apps.

  9. One small PR setting the TODO values in apps/backend/deployment.yaml:

    • REGULATIONS_GITHUB_APP_ID;
    • REGULATIONS_LLM_PROVIDER: "anthropic" and REGULATIONS_LLM_MODEL;
    • REGULATIONS_TELEGRAM_CHAT_ID and REGULATIONS_TELEGRAM_TOPIC_ID;
    • the final REGULATIONS_EMAIL_SENDER.

    Merge it after steps 6 and 8. Its rollout also restarts the backend, so it picks up the GitHub and LLM secrets.

Follow-ups (not in this PR)

  • preStop delay: a few seconds on the backend would remove the one-request gap every rollout has.
  • Pid cap: an optional node-wide kubelet podPidsLimit.
  • DB role for collab: collab uses the PostgreSQL root credentials, like the backend.

@coderabbitai

coderabbitai Bot commented Nov 2, 2026 •

Copy link
Copy Markdown

Explain Change →

Caution

Review failed

The pull request is closed.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@lorenzocorallo
lorenzocorallo force-pushed the feat/regulations-services branch from cdf727a to deef6c5 Compare November 2, 2026 19:44
…ing is harmless

Before Lorenzo creates the regulations Key Vault entries, a collab Pod stuck in
CreateContainerConfigError would fire PodNotRunning and
DeploymentReplicasMismatch (both page Telegram) and keep the backend
Kustomization unhealthy.

- apps/regulations-secrets holds every regulations ExternalSecret, split by
  purpose: regulations-core (internal and worker secrets, plus the worker's
  copy), regulations-github and regulations-llm. Its Flux Kustomization checks
  only the core ones, so it is Ready exactly when those entries exist.
- regulations-collab (own Kustomization from the backend ResourceSet, same
  image digest) and latex-worker depend on it: none of their Pods exist until
  the secrets do, then they start by themselves.
- The backend Kustomization keeps only the backend, its public route and its
  NetworkPolicy, and reads every regulations Secret with optional refs, so it
  stays healthy in any state.
@lorenzocorallo
lorenzocorallo force-pushed the feat/regulations-services branch from 463c6a7 to a15fae7 Compare November 15, 2026 02:16
…ed infrastructure

infrastructure-image-automation waits for health, so a provider that cannot
resolve its image (before the backend publishes latex-worker, or while the
package is private) left it not Ready. The provider now sits with its
ResourceSet under the root sync, which does not wait; only the provider itself
reports the error. Verified on k3s with the real Flux setup.
@lorenzocorallo lorenzocorallo changed the title feat(regulations): deploy regulations-collab and latex-worker feat(regulations): deploy regulations-collab and latex-worker, publish the backend's /public prefix Nov 15, 2026
@lorenzocorallo
lorenzocorallo marked this pull request as ready for review November 15, 2026 03:16
@lorenzocorallo
lorenzocorallo merged commit fca20fe into main Nov 15, 2026
1 of 2 checks passed
@lorenzocorallo
lorenzocorallo deleted the feat/regulations-services branch November 15, 2026 03:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant