🐛 Point llms.txt and feed.xml at the site they are served from

[taras]

Motivation

llms.txt and feed.xml hardcoded https://frontside.com/effection, so every copy advertised production regardless of where it was served from. Locally that meant llms.txt pointed agents away from the dev server they were reading it on; in a PR preview it pointed reviewers at production.

Everything else on the site escapes this because staticalize rewrites self-referencing absolute urls — but only inside HTML. downloader.ts takes that branch on Content-Type?.includes("html") and streams everything else to disk untouched, so text/plain and application/rss+xml are copied verbatim and their urls have to be correct when the response is generated.

Approach

• useSiteUrl() in www/plugins/current-request.ts. With no SITE_URL it returns useAbsoluteUrlFactory(), the existing origin-based helper; with one set it builds urls from that base, preserving its path.
• llms.txt and feed.xml use it in place of the hardcoded constant.
• www.yaml sets SITE_URL per deploy:
  • production → https://frontside.com/effection, the canonical url, byte-identical to what these files contain today
  • preview → https://pr-<N>--effection.netlify.app, computed before the server starts, since alias deploys land on a predictable url; fork PRs deploy anonymously to a url nobody knows in advance, so they fall back to the netlify production url
  • a ::warning:: fires if the site name netlify reports back differs from NETLIFY_SITE_NAME, so the prediction cannot silently drift
• environment.url for previews now points at the alias deploy, so the link in the checks panel shares an origin with the urls baked into that build. The per-commit snapshot url is still in the sticky comment.
• deno task dev sets SITE_URL=http://localhost:8000, so development exercises the same code path as a deploy rather than the fallback.

Could staticalize do this instead?

Asked, and tested against a miniature site — an HTML page, an llms.txt and an AGENTS.md, each carrying an absolute url back to itself — crawled by staticalize 0.2.7 (latest) with --base https://frontside.com/effection:

file	result
index.html	<link href> rewritten to https://frontside.com/llms.txt
llms.txt	untouched — still http://localhost:8321/api.md
AGENTS.md	untouched — still http://localhost:8321/api.md

Two things stand out. Text files are not rewritten, which is the reason this PR exists. And --base drops its path: I passed https://frontside.com/effection and got https://frontside.com/llms.txt back — staticalize copies only scheme, host and port from the base. So even if it learned to rewrite text, it could not produce urls for a site mounted under /effection; that is why the build already passes --base=https://effection.netlify.app and why useCanonicalUrl() carries the path separately.

Teaching staticalize to rewrite text and to honour the base path would let us delete SITE_URL later, and it would be a better fix, since --base would become the single source of truth. That is a change in another repository, so this PR takes the local route and leaves that door open.

Two other things the same experiment turned up, neither of them this PR's business but both worth filing:

• staticalize does not rewrite <a href>, only link[href], [src] and [content].
• the sitemap it writes wraps entries in <urls> rather than <url>, which is not the sitemaps.org schema — the deployed sitemap has 519 of them, so search engines are reading a malformed file today.

Verification

Served the site under each configuration and read the output: unset → http://localhost:8000/x/bdd; SITE_URL=https://frontside.com/effection → urls byte-identical to the current production ones, which matters most for feed.xml, where a changed guid would re-flag every post as unread in subscribers' readers; SITE_URL=https://pr-42--effection.netlify.app → preview urls. Requesting through 127.0.0.1 under deno task dev still returns localhost:8000, confirming SITE_URL is in effect rather than the origin fallback. The workflow's compute step, run directly, gives https://pr-42--effection.netlify.app for a branch PR and https://effection.netlify.app for a fork.

deno fmt --check (239 files), deno lint (199 files) and deno task test (38 passed, 244 steps) are clean. The test task gains --allow-env, because the new test reads and clears SITE_URL.

Second of three PRs carved out of #1241.

[pkg-pr-new]

Open in StackBlitz

npm i https://pkg.pr.new/effection@1246

commit: 37140ea

[codspeed]

Merging this PR will not alter performance

✅ 6 untouched benchmarks

---

_{Comparing tm/site-url (37140ea) with v4 (7b86d8d)}

[github-actions]

🚀 Deploy Preview Ready!

• Stable Review Link: https://pr-1246--effection.netlify.app
• Latest Commit Snapshot: https://6ab7b0fac9d95d1e66521d1e--effection.netlify.app

[taras]

Superseded by #1248.

staticalize 0.3.0 landed with the two fixes this PR was working around — it honours the path component of --base and rewrites textual bodies — so the site no longer has to be told its own public url. #1248 does the same job by emitting the origin that served the document and letting --base rewrite it at build time: no useSiteUrl(), no SITE_URL in the workflow or the dev task.

It also fixes something this PR did not. Previews have been building with --base=https://effection.netlify.app, so every rewritten absolute url in a preview pointed at production; #1248 points --base at the preview's own alias, which settles the html as well as the text.