Repository navigation
feat(regulations): deploy regulations-collab and latex-worker, publish the backend's /public prefix - #32
Merged
Conversation
|
Caution Review failedThe pull request is closed.
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
lorenzocorallo
force-pushed
the
feat/regulations-services
branch
from
November 2, 2026 19:44
cdf727a to
deef6c5
Compare
…h the backend's /public prefix
…ing is harmless Before Lorenzo creates the regulations Key Vault entries, a collab Pod stuck in CreateContainerConfigError would fire PodNotRunning and DeploymentReplicasMismatch (both page Telegram) and keep the backend Kustomization unhealthy. - apps/regulations-secrets holds every regulations ExternalSecret, split by purpose: regulations-core (internal and worker secrets, plus the worker's copy), regulations-github and regulations-llm. Its Flux Kustomization checks only the core ones, so it is Ready exactly when those entries exist. - regulations-collab (own Kustomization from the backend ResourceSet, same image digest) and latex-worker depend on it: none of their Pods exist until the secrets do, then they start by themselves. - The backend Kustomization keeps only the backend, its public route and its NetworkPolicy, and reads every regulations Secret with optional refs, so it stays healthy in any state.
lorenzocorallo
force-pushed
the
feat/regulations-services
branch
from
November 15, 2026 02:16
463c6a7 to
a15fae7
Compare
…ed infrastructure infrastructure-image-automation waits for health, so a provider that cannot resolve its image (before the backend publishes latex-worker, or while the package is private) left it not Ready. The provider now sits with its ResourceSet under the root sync, which does not wait; only the provider itself reports the error. Verified on k3s with the real Flux setup.
lorenzocorallo
marked this pull request as ready for review
November 15, 2026 03:16
This was referenced Nov 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Deployment for the regulations editor (shared spec, decision 4 and the "Deployment" and "Public backend routes" contracts), aligned with the backend stack merged as PoliNetworkOrg/backend#110–#117 (
a048a04).Merging this is safe before any manual prerequisite exists. Nothing that needs a missing secret or image runs, fails or alerts until Lorenzo creates it, and then it starts by itself. That was verified on a throwaway k3s at prod's version with the repo's real Flux setup; details below.
What it deploys
backend(existing, from the backend ResourceSet)/publicroute; Traefik → backend:3000 inallow-callersregulations-secrets(new, plain)latex-workernamespacewait: false+healthCheckson the two core ExternalSecrets)regulations-collab(new, from the backend ResourceSet, so the same image digest)collab.polinetwork.org, NetworkPolicyregulations-secretsandbackendare Readylatex-worker(new ResourceSet)regulations-secretsis ReadyThe secrets are split so that only the core ones gate anything:
regulations-core(backend namespace):internal-secret,worker-secret.regulations(latex-worker namespace):worker-secret.regulations-github:private-key,webhook-secret.regulations-llm:api-key.The GitHub and LLM entries can arrive weeks later without blocking anything.
The
latex-worker-imageprovider sits with its ResourceSet under the root sync, which doesn't wait for health, instead of ininfrastructure-image-automation(which does). A missing or private image therefore reports an error only on that one provider.Services (unchanged from the reviewed design)
regulations-collabbun run collab.js.Recreate,REGULATIONS_COMPILE_CONCURRENCY=2.latex-workerrestricted, read-only root, tmpfs/work512Mi and/tmp64Mi.GET /health./publicHost(`api.polinetwork.org`) && PathPrefix(`/public/`) && PathRegexp(`^/public(/[A-Za-z0-9_-]+)+$`).Cf-Connecting-Ipand a total rate limit, and theX-PoliNetwork-Ingress: publicmarker the backend relies on.POST /public/webhooks/github. Polling every 5 min (REGULATIONS_RECONCILE_SECONDS) is reconciliation only.Tests (
tests/workload-manifests.test.rb)The tests enforce the sandbox, the exposure and the gate:
apps/backendholds no ExternalSecret exceptazure-kv;regulations-secretshealth-checks;regulations-secrets, and the backend doesn't;regulations-secretsis listed inclusters/k3s/apps/kustomization.yaml;backend-service: the/publicroute, with exactly that rule and those middlewares;..,%2e%2e,%2f, empty segments, a trailing slash and backslashes.Each assertion was mutation-checked (break the manifest, the test must fail).
What happens at merge, before the prerequisites exist
These are the conditions the orchestrator described: no
regulations-*Key Vault entries, nolatex-workerimage (the backend'slatex-workerworkflow currently skips becauselegal-texliveis unreadable, confirmed on runa048a04), no Cloudflare hostname or Access bypass, no GitHub App.Verified on k3s v1.36.3+k3s1 with flux-operator v0.58.1, Flux 2.9.4 and the FluxInstance from
ansible/roles/flux, syncing this repo from GitHub, firstmainand then this branch. The real ESO 2.12.0 HelmRelease ran with fake ClusterSecretStores standing in for Key Vault: same names and namespace conditions, every existing key, noregulations-*. The real Kustomization objects fromclusters/k3swere applied for security, storage, traefik, external-secrets, image-automation, postgres, redis, backend, latex-worker and regulations-secrets. The production backend image ran as Flux resolved it.Backend:
GET /stillhi(what Kuvasz checks).preStopdelay.regulations-secrets: not Ready, "health check failed … ExternalSecret/backend/regulations-core".regulations-collab: "dependency 'regulations-secrets' is not ready". No collab or worker Pod, Deployment or Service exists, soPodNotRunning,DeploymentReplicasMismatchandPodCrashLoopinghave nothing to fire on.latex-worker-imageprovider: not Ready (GHCRDENIED). Thelatex-workerResourceSet is Ready and generates nothing.infrastructure-image-automationstays Ready; before the provider was moved, it went not-Ready, and that is fixed in this PR.Every other Kustomization (security, storage, traefik, external-secrets, secret-stores, image-automation, postgres, redis, backend, root) is Ready on the new revision. Nothing
dependsOnthe new Kustomizations, and no change rolls back anything.Receiver: a signed GitHub
packageevent still annotated and reconciled every provider, including those listed after the failing one./publicin this state:api.polinetwork.org, only/public/...reaches the backend, and the current image answers it with its own 404 because it has no/publicroutes./,/trpc,/internal/events,/public/../trpc,/public/%2e%2e/trpcand other hosts get Traefik's 404.api.polinetwork.orgrequest to its login (302) until the bypass exists.Image first, then secrets: I made the worker's provider resolve an existing package before any secret existed. The ResourceSet generated the
latex-workerKustomization, which waited onregulations-secrets, and no worker object was created.Self-activation: after adding the two core entries to the store, with no other action:
regulations-collabandlatex-workerappliedThe GitHub and LLM ExternalSecrets stayed failing and blocked nothing.
Backend restart: the backend Pod that predated the secret lacked
REGULATIONS_INTERNAL_SECRET. Onerollout restartbrought both core variables in, andGET /still answeredhi.Reasoned, not reproduced:
/(hi), which stays up. Collab and the worker aren't monitored.collab.polinetwork.org, the collab Ingress is unreachable. The/publicroute stays behind Access until the bypass.Module not found "collab.js", because the test ran the previous production backend image. Prod will have runa048a04long before. It does mean the core entries must not be created while an image withoutcollab.jsis live (checklist step 2).ErrImagePullonly because of my test redirection to another package.Lorenzo's checklist after the merge
Check that it landed (no action): the backend rolled once and is healthy.
regulations-secretsshows "ExternalSecret/backend/regulations-core" not healthy,regulations-collabwaits on it, and thelatex-worker-imageprovider is red. That's expected until the steps below.Core secrets. First make sure the live backend image includes the regulations code (backend
main≥a048a04; its Docker workflow must be green). Then create inkv-pn-apps:regulations-internal-secret=openssl rand -hex 32regulations-worker-secret=openssl rand -hex 32Collab starts by itself within about 10 minutes (ESO backoff). To skip the wait:
kubectl -n backend annotate externalsecret regulations-core force-sync="$(date +%s)" --overwrite && kubectl -n latex-worker annotate externalsecret regulations force-sync="$(date +%s)" --overwriteRestart the backend once, so it reads those secrets:
kubectl -n backend rollout restart deployment/backend.Cloudflare tunnel: add
collab.polinetwork.orgto thek3s01tunnel → Traefik, like the other hostnames. Keep WebSockets on and no Access. The editor's live sync then works.Worker image:
legal-texliveis built for amd64 and arm64 and includes fontconfig (the worker Dockerfile runsfc-cache).legal-texlivepackage: Package settings → Manage Actions access → addPoliNetworkOrg/backendwith Read.ghcr.io/polinetworkorg/latex-workerexists, set that package to public; the cluster pulls without credentials.Flux picks it up within 6 h, or immediately with
kubectl -n flux-system annotate resourcesetinputprovider latex-worker-image reconcile.fluxcd.io/requestedAt="$(date +%s)" --overwrite, and the worker starts by itself. PDF previews then work.GitHub App on PoliNetworkOrg:
legal.https://api.polinetwork.org/public/webhooks/github, content type JSON, events Pull request, Check suite and Release.kv-pn-apps, createregulations-github-app-private-key(the PEM, viaaz keyvault secret set --file) andregulations-github-webhook-secret.Cloudflare Access: add an Access application for
api.polinetwork.org/public/with Bypass / Everyone, and keep the existing one on the rest of the host.LLM: create
regulations-llm-api-keyinkv-pn-apps.One small PR setting the TODO values in
apps/backend/deployment.yaml:REGULATIONS_GITHUB_APP_ID;REGULATIONS_LLM_PROVIDER: "anthropic"andREGULATIONS_LLM_MODEL;REGULATIONS_TELEGRAM_CHAT_IDandREGULATIONS_TELEGRAM_TOPIC_ID;REGULATIONS_EMAIL_SENDER.Merge it after steps 6 and 8. Its rollout also restarts the backend, so it picks up the GitHub and LLM secrets.
Follow-ups (not in this PR)
preStopdelay: a few seconds on the backend would remove the one-request gap every rollout has.podPidsLimit.