ansible: add certbot role and serve origins from Let's Encrypt

[ryanaslett]

Summary

Adds a certbot role that issues and renews Let's Encrypt certificates over HTTP-01 (certbot certonly --webroot), using the distro-packaged certbot and its certbot.timer. No API token, DNS delegation or extra tooling is involved.

• New certbot role, wired into the ci, ci-release, www (direct.nodejs.org) and rsync (unencrypted.nodejs.org) plays, each with its own certificate.
• The port 80 vhosts now serve /.well-known/acme-challenge/ from /var/www/letsencrypt. ci-release's redirect moved into a location block so the challenge path is reachable.
• jenkins-nginx and nginx get a reload nginx handler, and the nginx site tasks get an nginx-sites tag, so vhost changes can be deployed without running the whole nginx role.
• The ci and ci-release templates now serve the certbot-issued certificates and drop OCSP stapling, which Let's Encrypt no longer supports.

Refs #4038, #4482. This does not close either; docs and cleanup are tracked in #4038.

Already applied

These were deployed ahead of the 2026-10-23 expiry in #4482, and all four origins serve Let's Encrypt now:

• rsync, ci-release and ci were deployed with the plays in this PR (--tags nginx-sites or jenkins-nginx, then --tags certbot).
• www is not managed by the nginx role (its sites use a different layout from the role's), so it was done by hand: the challenge location in sites-available/nodejs.org, certbot certonly --webroot for direct.nodejs.org and origin.nodejs.org, and the cert paths in /etc/nginx/conf.d/ssl-defaults.conf.
• rsync's /etc/nginx/conf.d/ssl-defaults.conf was also edited by hand (it is not in Ansible).
• The Sectigo custom certificate was removed from the Cloudflare zone, so proxied names use Universal SSL.

Things to know

• ci-release's certificate is ECDSA and chains to ISRG Root X2. If a release agent's Java cannot build that chain, reissue with --key-type rsa, which chains to ISRG Root X1.
• certbot_domains is set per host in the playbooks. origin.nodejs.org is still a live Cloudflare load balancer used by the release Worker, so it is covered by the www certificate.
• The old wildcard files under /etc/nginx/ssl/ are still on the hosts and are not removed here.

Test plan

• ansible-playbook --syntax-check on the three playbooks
• --check --diff before each real run, one host at a time
• certbot renew --dry-run passes on rsync, ci-release and ci
• openssl s_client from outside shows a Let's Encrypt certificate with a verified chain on all four origins
• Renewal reloads nginx on the first real renewal (about 30 days before 2027-01-14)

🤖 Generated with Claude Code

[ryanaslett]

Additional things: This is using the HTTP-01 verification method (LetsEncrypt hits a .well-known/acme-challenge/ http endpoint to prove you own the domain)

Other options I looked at is the DNS-01 verification method, which uses dns to verify and has the advantage of allowing wildcard certs to be issued, however, it required a cloudflare API key to live on the servers running certbot. There is a trick to giving the servers a well scoped key to a non threating domain (like node.dev) by proxying a CNAME from nodejs.org -> node.dev and allowing the servers to make dns mods that way, however certbot didnt have any support for that particular type of indirection, opting to wait until the newer verification method, detailed below, was available.

There is a third, newer verification method DNS-PERSIST-01 ->(https://letsencrypt.org/2026/02/18/dns-persist-01) however, that appears to be not actually usable yet (but removes the need to have API keys, so its the someday best way).. (https://www.certkit.io/blog/whats-taking-dns-persist-01-so-long) details the best status I can find on it so far.