Make step-ca reachable from MDC1 → flip reprovision-runner to cert_mode: step_renew

[rcurranmoz]

Goal

Kill the recurring manual runner-cert reissue for good by moving the reprovision-runner (m4-81) from cert_mode: vault (static ≤24h cert, hand-minted) to the module's default cert_mode: step_renew — step ca renew --daemon auto-rotates a short-lived cert, key generated on-host, never in vault/git, centrally revocable. The 24h provisioner cap is a non-issue for step_renew (it renews within the window).

The one prerequisite is that m4-81 can reach step-ca to renew. Investigation shows this is mostly already built — it's DNS/naming + a one-time bootstrap, not new infra.

What already exists (evidence)

• step-ca: GCE VM, static external IP 34.61.3.27 (terraform/step_ca.tf, network.tf google_compute_address.step_ca), listens on :443. Cert SANs include step-ca.relops.mozilla and the static IP (step ca init --dns step-ca.relops.mozilla --dns <static-ip>, per step_ca.tf header).
• Firewall already permits MDC1 → step-ca: network.tf allow-step-ca-inbound allows tcp:443,8443 from trusted_source_cidrs (the MDC1 worker network) + Google LB ranges, target_tags = ["step-ca"]. Comment: "Inbound from MDC1 worker network only."
• The forge LB already fronts step-ca for enrollment: terraform/lb.tf:188-190 routes /scep/* and /acme/* to the step-ca backend; everything else → vault-broker. Reached from MDC1 at forge.relops.mozilla.com with the same trusted_source_cidrs allowlist + mTLS.

So EACS re-enrollment (SCEP) already reaches step-ca from MDC1 through the LB. The renewal path is what's not wired for the runner yet.

Why renewal needs the DIRECT path, not the L7 LB

step ca renew --daemon authenticates by presenting the current client cert via mTLS directly to step-ca and validates step-ca's server cert against the step-ca root. The forge L7 LB terminates TLS (Google-managed cert) and does its own ALLOW_INVALID_OR_MISSING mTLS + header forwarding — which is fine for SCEP/ACME (HTTP-based) but breaks native step ca renew mTLS. So renewal must use direct L4 to step-ca :443 (already firewall-allowed from MDC1), not the /acme,/scep L7 routes.

Work (small)

• Confirm reachability from a runner host. On m4-81: nc -vz 34.61.3.27 443 (and curl -sv https://step-ca.relops.mozilla/health --resolve step-ca.relops.mozilla:443:34.61.3.27). If it connects, the firewall/CIDR path is already good; if not, add m4-81's subnet to trusted_source_cidrs in terraform.tfvars + apply.
• Resolve the name from MDC1. step-ca.relops.mozilla is not a real DNS zone. Either (a) add an MDC1 DNS record step-ca.relops.mozilla → 34.61.3.27, or (b) drop an /etc/hosts entry on the runner via the puppet role (simplest, no DNS dependency), or (c) point reprovision_runner.step_ca_url at https://34.61.3.27 directly (the IP is a cert SAN; may be finicky vs. a DNS name — prefer a/b).
• Bootstrap the initial cert on m4-81 (modules/reprovision_runner/README.md): step ca bootstrap --ca-url https://step-ca.relops.mozilla --fingerprint f2d8032e35566ecb41b3f1dd5ec7b550fb73f6c8fa855870ca3f35638d777cf6 + step ca certificate ..., or the single-use enrollment-token path.
• Flip the role cert_mode: vault → step_renew + uncomment ca_fingerprint — ronin_puppet PR #1275 (already staged, one-step).
• Run puppet on m4-81; confirm the certrenew LaunchDaemon rotates and the runner reconnects.
• Remove client_cert/client_key from m4-81's /var/root/vault.yaml once step_renew is proven.

Alternative (if direct L4 is undesirable)

Switch the runner to an ACME provisioner (already LB-fronted at /acme/*) and renew via ACME over the forge L7 LB — avoids exposing step-ca :443 directly. Bigger change (new provisioner + challenge mechanism, http-01/tls-alpn on a headless mini is awkward), so the direct-L4 path above is preferred.

Context

From the 2026-07-10 reprovision security review. Stopgap in place: scripts/reissue-runner-cert.sh (one-command reissue, PRs #27/#29/#30). Ronin flip prepped: PR #1275.