Skip to content

Keep crawlers off query pages, prefetch on intent, lighter markup (2.8.5) - #325

Merged
assafbar2 merged 4 commits into
mainfrom
cursor/cut-origin-transfer-8d76
Nov 3, 2026
Merged

assafbar2 merged 4 commits into
mainfrom
cursor/cut-origin-transfer-8d76

Conversation

@assafbar2

Copy link
Copy Markdown
Owner

What kind of change?

  • New agent listing (content/agents/<slug>.json added)
  • Correction to an existing entry (URL fix, renamed product, new access method…)
  • Removal (product discontinued)
  • Code / site change

Why

On Nov 2 at 5:08 PM PST, Vercel paused the Hobby team because Fast Origin Transfer reached 305% of its 10 GB monthly limit: 17.4 GB of function transfer plus 14.3 GB of ISR reads over 30 days. Two things drove it:

  • Crawlers on query pages. /compare?a=…&b=…, /stack?items=… and /operator?q=… render per request, and every one links to more mixes, so a crawler never runs out. On Nov 2, PerplexityBot fetched /compare 255,336 times, GoogleOther 149,385 times and ClaudeBot 67,336 times.
  • What each page view pulls from the origin. Next prefetched every link as soon as it scrolled into view, and the pages were heavy. A CDN miss on a prerendered page is billed as an ISR read of the uncompressed page. Listing pages make up 53% of ISR reads, category pages 28% and the changelog 8.5%.

Agents' full e2e runs against production also caused about 23% of cache misses in the week of Oct 26.

What this does

Crawlers (app/robots.ts, vercel.json, tests/robots.test.ts):

  • In robots.txt, the * group now disallows /browse?, /compare?, /stack?, /operator? and the three dynamic share cards. The bare pages stay open, and the group adds Crawl-delay: 5.
  • Link-preview bots (Twitterbot, facebookexternalhit, LinkedInBot, Slackbot, Discordbot and others) get their own group, so shared links still unfurl with their cards.
  • SEO-tool crawlers (AhrefsBot, SemrushBot, MJ12bot, DotBot, BLEXBot, DataForSeoBot, Barkrowler and serpstatbot) get Disallow: /.
  • For crawlers that ignore robots.txt, vercel.json routes with mitigate: deny turn away crawler user agents on /compare?a=, /stack?items=, /operator?q= and the three cards.
    • Link previews and assistants fetching a page for a person (ChatGPT-User, Perplexity-User, Claude-User and MistralAI-User) are exempt.
    • This is a Vercel firewall rule in config, which is free on every plan.

Prefetch on intent (components/Link.tsx, 52 import swaps, an ESLint rule):

  • Every page link goes through a wrapper around next/link that prefetches only once a pointer, touch or focus reaches the link.
  • Links with a query never prefetch, since each prefetch would run a function.
  • ESLint now rejects direct next/link imports anywhere else.

Lighter markup (app/globals.css, cards, badges, footer, changelog, category rail, dotted leaders):

  • Server components send their markup twice per page: once in the HTML and once in the RSC payload. The utility strings that every card, badge and footer link repeated are now short component classes built with @apply.
  • The Verified and Official icons are now CSS masks instead of inline SVGs.
  • Category pages' JSON-LD item lists no longer repeat each listing's description.

e2e (e2e/target.mjs, e2e/playwright.config.mjs, e2e/tests/05-edge.spec.mjs, README):

  • The suite now runs against a local production build on port 3100, which the config builds and starts.
  • With E2E_BASE_URL set to a deployment, only the smoke and edge specs run, unless E2E_FULL=1.
  • The new 05-edge spec checks what Vercel's edge adds:
    • HSTS, and the http and www redirects (these moved here from 95-seo-meta-headers);
    • crawlers getting 403 on query pages and their cards;
    • people and link previews still getting 200.
  • Six tests that only passed on production now check the site they run against.

Release: version 2.8.5, a patch, with a CHANGELOG entry.

Measured

Page weight (uncompressed HTML, local production builds of main and this branch):

Page main this branch change
/ 289 KB 239 KB −17%
/agents/claude-code 107 KB 74 KB −31%
/agents/linear 115 KB 83 KB −28%
/categories/code-devtools 2,052 KB 1,042 KB −49%
/categories/data-analytics 661 KB 342 KB −48%
/changelog 2,961 KB 2,134 KB −28%
/browse 6,195 KB 3,197 KB −48%
/browse?task=query-a-database 434 KB 230 KB −47%

Background requests in Chrome at 1440×900, counting from page load through a scroll to the bottom:

Page main this branch
/ 279 (1.2 MB) 0
/categories/code-devtools 851 (2.5 MB) 0
/agents/claude-code 116 (1.4 MB) 0
/changelog 974 (5.7 MB) 0

Hovering a card prefetches its listing, and the click then renders the page in 48 ms with no further requests.

Testing

  • tsc; lint (no errors, only the 4 warnings already on main); vitest (684 passed); validate-content; check-try-links; build.
  • Full e2e against a local production build of this branch: 923 passed, and 10 skipped (the edge checks, which need Vercel's edge).
  • Visual comparison of main and this branch:
    • 90 screenshot pairs: 19 pages at desktop and phone sizes in light and dark, plus card, rail, footer, changelog and inline-link hover and focus states.
    • Every pair has the same dimensions.
    • The only differing pixels are anti-aliasing inside the 12 px Verified and Official icons, and a one-step shade (RGB 16 vs 17) between leader dots in dark mode.
  • To check on the preview and again after merge: robots.txt, then E2E_BASE_URL=<deployment> npm run test:e2e (smoke and edge only).

Checklist

  • npx tsx scripts/validate-content.ts passes locally
  • Added a line to content/changelog.json (no listing changed)
  • All URLs load
  • Skills describe real, documented capabilities (not aspirations)
  • Any Use it now field is copied from the product's own docs, with no placeholders
  • For vendor submissions: "verified": false (maintainers verify & flip)
  • official left unset (maintainers set it; if you think it applies, link the evidence above. See CONTRIBUTING.md → Official listings)
  • Listing and catalog changes are contributed under CC BY-NC 4.0 (see LICENSE-DATA); code under MIT

cursoragent and others added 4 commits November 3, 2026 04:15
robots.txt now closes /browse?, /compare?, /stack? and /operator? and the
three dynamic share cards to crawlers (the bare pages stay open: /browse
is in the sitemap), lets link-preview bots fetch them, and shuts out
SEO-tool crawlers. vercel.json denies the crawlers that ignore it, by
user agent, at Vercel's firewall before a function runs; link previews
and assistants fetching a page for a person are let through.

On Nov 2 PerplexityBot, GoogleOther and ClaudeBot fetched /compare with
endless query mixes about 470,000 times, 16 GB of origin transfer in a
day, which paused the Hobby team.

Co-authored-by: assaf.barnir <assaf.barnir@sentry.io>
npm run test:e2e builds and starts this checkout on localhost:3100 unless a
server already answers there. E2E_BASE_URL points it at a deployment, where
only 00-smoke and the new 05-edge spec run unless E2E_FULL=1: every request
to a deployment counts against the team's Vercel usage, and a full run makes
thousands. The edge spec holds what only Vercel's edge does (HSTS, the http
and www redirects, the vercel.json crawler rules); tests that assumed the
production host now use the target's.

Co-authored-by: assaf.barnir <assaf.barnir@sentry.io>
components/Link wraps next/link: it prefetches once a pointer, touch or focus
reaches a link, and never for an address with a query. Viewport prefetching
fetched every link in view: about 800 requests for one look at a category
page, and a function run for each /browse?… or /compare?… link. ESLint now
points next/link imports at the wrapper.

Listing cards, tags, the footer's links, the category rail and the
changelog's rows use short classes from globals.css instead of repeating
their utilities, and the Verified and Official tags draw their icons with
CSS masks instead of inline SVGs. Category pages' JSON-LD no longer repeats
each listing's description.

Co-authored-by: assaf.barnir <assaf.barnir@sentry.io>
Co-authored-by: assaf.barnir <assaf.barnir@sentry.io>
@vercel

vercel Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
agentswitchboard.dev Ready Ready Preview Nov 3, 2026 4:36am UTC

Request Review

@assafbar2
assafbar2 merged commit e8520fb into main Nov 3, 2026
3 checks passed

This branch was successfully deployed

1 active deployment
Preview — 5260af15 Deployed Nov 3, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants