🤖 Keep llms.txt, feed.xml, AGENTS.md and the guides on their own site

[taras]

Motivation

Three problems, all of them the same shape: a resource served by this site told its reader to go somewhere else.

llms.txt and feed.xml hardcoded https://frontside.com/effection, so every copy advertised production no matter where it was served from. Locally llms.txt pointed agents away from the dev server they were reading it on; in a PR preview it pointed reviewers at production. The rest of the site does not have this problem because staticalize rewrites self-referencing absolute urls — but only inside HTML. downloader.ts takes that branch on Content-Type?.includes("html") and streams everything else to disk untouched, so text/plain and application/rss+xml are copied verbatim and their urls have to be right when the response is generated.

llms.txt sent agents to raw.githubusercontent.com for AGENTS.md and for all seven guides, pinned to the v4 branch, and to rendered HTML pages for the packages. The documents that govern how an agent writes Effection code came from a different origin than the documentation they describe, and from whatever v4 happened to be rather than the snapshot the rest of the site was built from.

The root AGENTS.md mixed two audiences — the behavioral contract that governs anyone writing Effection code, and the rules for contributing to this repository — so an agent writing an application had to read past gitmoji and pre-commit commands to reach the invariants.

Approach

Urls follow the site

• useSiteUrl() in www/plugins/current-request.ts. With no SITE_URL it returns useAbsoluteUrlFactory(), the existing origin-based helper; with one set it builds urls from that base, preserving its path.
• llms.txt and feed.xml use it in place of the hardcoded constant, and the series in llms.txt now comes from the site config rather than a hardcoded v4.
• www.yaml sets SITE_URL per deploy: production → https://frontside.com/effection, the canonical url, byte-identical to what these files contain today; preview → https://pr-<N>--effection.netlify.app, computed before the server starts, since alias deploys land on a predictable url, with fork PRs falling back to the netlify production url because they deploy anonymously to a url nobody knows in advance. A ::warning:: fires if the site name netlify reports back differs from NETLIFY_SITE_NAME, so the prediction cannot silently drift.
• environment.url for previews now points at the alias deploy, so the link in the checks panel shares an origin with the urls baked into that build. The per-commit snapshot url is still in the sticky comment.
• deno task dev sets SITE_URL=http://localhost:8000, so development exercises the same code path as a deploy rather than the fallback.

Everything llms.txt points at is markdown on this site

• docs/agents.md is the public behavioral contract, moved verbatim out of the root AGENTS.md: core invariants, operations/futures/tasks, entry points, spawn(), Task.halt(), scope vs task, context, the concurrency operations, promise interoperability, resource(), ensure(), useAbortSignal(), streams and the rest. Only the three-line introduction changed, to address agents writing applications rather than agents editing this repository. The root AGENTS.md keeps the repository-only rules and opens by sending repository agents to the contract, both as the copy checked out on their branch and as its published url. www/AGENTS.md stays the scoped guide for the website; only its references to "the root AGENTS.md" for API correctness were repointed at where that material now lives.
• /AGENTS.md serves that file, read from the checkout per request, and rewrites the canonical links inside it through the same useSiteUrl(), so an agent on a preview stays on the preview. Urls that are not part of this site, such as the Frontside blog post linked from useAbortSignal(), are left alone, and the pattern is anchored so …/effectionx is not caught.
• /guides/:series/:id.md serves each guide's markdown source from the same worktree the rendered pages are built from — the loader already keeps it, so nothing is read twice or copied.
• /x/:workspacePath.md serves each package's README.md through the getReadme() the page already uses, and appends the npm install command, since a readme read on its own is missing what the page carries beside it. The six readmes that already give the command are left as they are.
• /api.md and /api/<series>/<symbol>.md serve the API reference, experimental symbols under their own segment as their pages already are. Both read the same pkg.docs() the HTML routes render: a symbol page is its declarations, each rendered through the existing Type component and taken as text so the signature cannot drift from the page, its documentation with {@link} resolved to markdown pages, and where the code lives. The index lists what the HTML index lists, newest series first, and llms.txt points [API] there rather than repeating the catalog. An unknown symbol answers 404, where the page route throws.
• The .md routes are registered before their page routes, since path-to-regexp would otherwise read .md as part of the id, and both list their documents in the sitemap so a static build captures them. All three markdown endpoints build their responses through one helper in www/lib/markdown-response.ts, which also settles the cache header: no-cache rather than max-age=3600, because that header only ever reaches a browser talking to the dev server — a static build copies the body and netlify supplies its own — and an hour of caching meant an editor who had opened one of these before a change kept being shown the copy from before it. The etag plugin already answers the revalidation with a 304.
• The footer's "AI Agent Resources" link points at the hosted contract, and [Browse all guides][docs/], a reference that never had a definition, now resolves to the [Guides] index.

Verification

Served the site in both configurations and read what came back. Every reference definition in llms.txt:

	no SITE_URL	SITE_URL=https://frontside.com/effection
[AGENTS.md]	http://localhost:8000/AGENTS.md	https://frontside.com/effection/AGENTS.md
[API]	http://localhost:8000/api/	https://frontside.com/effection/api/
[Operations]	http://localhost:8000/guides/v4/operations.md	https://frontside.com/effection/guides/v4/operations.md

…and the same for the other six guides. The package catalog now reads - [@effectionx/bdd](…/x/bdd.md): …. No github.com anywhere in the document.

• /AGENTS.md returns the contract as text/markdown with its API link rewritten and the Frontside blog link untouched. /guides/v4/operations.md is byte-identical to docs/operations.mdx, all seven guides return 200, and the sitemap carries 33 guide entries. /x/task-buffer.md is its readme plus the install section; across all 27 packages every document contains exactly one npm install @effectionx/<name> — none missing, none duplicated. Unknown ids return 404 on both routes, and the HTML pages at /guides/v4/operations and /x/task-buffer still return text/html, so the route ordering does not shadow them.
• /api.md lists 99 symbols across 4.1.1 and 3.6.1, every one of which returns 200; /api/v4/main.md carries the signature, the documentation with {@link exit} resolved to /api/v4/exit.md on the same host, and the source link; /api/v4/does-not-exist.md returns 404; the sitemap carries 156 API markdown entries including /api.md; and /api, /api/v4/main, /api/v3/main and /api/v4/experimental/createApi still return text/html.
• feed.xml under the production base is byte-identical to the current production feed, which matters most for guid, where a change would re-flag every post as unread in subscribers' readers.
• Requesting through 127.0.0.1 under deno task dev still returns localhost:8000, confirming SITE_URL is in effect rather than the origin fallback.
• The workflow's compute step, run directly, gives https://pr-42--effection.netlify.app for a branch PR and https://effection.netlify.app for a fork.
• This PR's own preview confirmed the first change end to end: pr-1241--effection.netlify.app/llms.txt points at itself rather than production.

Tests cover the routes (status, exact body, content type, cache header, sitemap entries, 404), the contract's contents (the public sections it has, the repository sections it must not), link rewriting under both environments, the install-appending rule including the readmes that already give the command and the one that merely mentions npm install in an example, the llms.txt definitions including that none point at GitHub and that every reference it uses is defined, the footer markup, and the shape of all three instruction files. The reference check was confirmed to bite by restoring the dangling label.

deno fmt --check (252 files), deno lint (211 files), deno check and the full suite (51 passed, 269 steps) are all clean.

One gap worth naming: /x/:workspacePath.md has no unit test, because exercising it means standing up useWorkspaces, which clones effectionx over the network. Its pure part, the install-appending rule, is tested; the route itself is covered by the live checks above. #1242 adds EFFECTIONX_DIR, which would make a fixture-based test cheap once it lands.

www/deno.json's test task gets --allow-env, because importing the route modules pulls in helpers that read OSTYPE at module load. Worth knowing for anything added to www later: the root task picks up the www workspace and grants --allow-env but not --allow-write.

[pkg-pr-new]

Open in StackBlitz

npm i https://pkg.pr.new/effection@1241

commit: 8c10ac9

[codspeed]

Merging this PR will improve performance by 20.25%

⚡ 1 improved benchmark
✅ 5 untouched benchmarks

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
⚡	Memory	effection-inline.recursion	5.9 KB	4.9 KB	+20.25%

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing tm/llms-txt-site-url (8c10ac9) with v4 (32a31da)}

[github-actions]

🚀 Deploy Preview Ready!

• Stable Review Link: https://pr-1241--effection.netlify.app
• Latest Commit Snapshot: https://6ab11f1450450613291ff700--effection.netlify.app

[cowboyd]

There are a couple things that I'm looking at

1. there are just a lot of changes outside scope
2. looking at this, I feel like we should ask "how can we fix this in staticalize?" because to be honest the SITE_URL was always a hack, and staticalize was written for the most part to solve that.

[taras]

Superseded by #1245, #1246, #1247