Repository navigation
Keep crawlers off query pages, prefetch on intent, lighter markup (2.8.5) - #325
Merged
Merged
Conversation
robots.txt now closes /browse?, /compare?, /stack? and /operator? and the three dynamic share cards to crawlers (the bare pages stay open: /browse is in the sitemap), lets link-preview bots fetch them, and shuts out SEO-tool crawlers. vercel.json denies the crawlers that ignore it, by user agent, at Vercel's firewall before a function runs; link previews and assistants fetching a page for a person are let through. On Nov 2 PerplexityBot, GoogleOther and ClaudeBot fetched /compare with endless query mixes about 470,000 times, 16 GB of origin transfer in a day, which paused the Hobby team. Co-authored-by: assaf.barnir <assaf.barnir@sentry.io>
npm run test:e2e builds and starts this checkout on localhost:3100 unless a server already answers there. E2E_BASE_URL points it at a deployment, where only 00-smoke and the new 05-edge spec run unless E2E_FULL=1: every request to a deployment counts against the team's Vercel usage, and a full run makes thousands. The edge spec holds what only Vercel's edge does (HSTS, the http and www redirects, the vercel.json crawler rules); tests that assumed the production host now use the target's. Co-authored-by: assaf.barnir <assaf.barnir@sentry.io>
components/Link wraps next/link: it prefetches once a pointer, touch or focus reaches a link, and never for an address with a query. Viewport prefetching fetched every link in view: about 800 requests for one look at a category page, and a function run for each /browse?… or /compare?… link. ESLint now points next/link imports at the wrapper. Listing cards, tags, the footer's links, the category rail and the changelog's rows use short classes from globals.css instead of repeating their utilities, and the Verified and Official tags draw their icons with CSS masks instead of inline SVGs. Category pages' JSON-LD no longer repeats each listing's description. Co-authored-by: assaf.barnir <assaf.barnir@sentry.io>
Co-authored-by: assaf.barnir <assaf.barnir@sentry.io>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
4 of 12 tasks
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What kind of change?
content/agents/<slug>.jsonadded)Why
On Nov 2 at 5:08 PM PST, Vercel paused the Hobby team because Fast Origin Transfer reached 305% of its 10 GB monthly limit: 17.4 GB of function transfer plus 14.3 GB of ISR reads over 30 days. Two things drove it:
/compare?a=…&b=…,/stack?items=…and/operator?q=…render per request, and every one links to more mixes, so a crawler never runs out. On Nov 2, PerplexityBot fetched /compare 255,336 times, GoogleOther 149,385 times and ClaudeBot 67,336 times.Agents' full e2e runs against production also caused about 23% of cache misses in the week of Oct 26.
What this does
Crawlers (
app/robots.ts,vercel.json,tests/robots.test.ts):robots.txt, the*group now disallows/browse?,/compare?,/stack?,/operator?and the three dynamic share cards. The bare pages stay open, and the group addsCrawl-delay: 5.Disallow: /.robots.txt,vercel.jsonroutes withmitigate: denyturn away crawler user agents on/compare?a=,/stack?items=,/operator?q=and the three cards.Prefetch on intent (
components/Link.tsx, 52 import swaps, an ESLint rule):next/linkthat prefetches only once a pointer, touch or focus reaches the link.next/linkimports anywhere else.Lighter markup (
app/globals.css, cards, badges, footer, changelog, category rail, dotted leaders):@apply.e2e (
e2e/target.mjs,e2e/playwright.config.mjs,e2e/tests/05-edge.spec.mjs, README):E2E_BASE_URLset to a deployment, only the smoke and edge specs run, unlessE2E_FULL=1.05-edgespec checks what Vercel's edge adds:95-seo-meta-headers);Release: version 2.8.5, a patch, with a CHANGELOG entry.
Measured
Page weight (uncompressed HTML, local production builds of main and this branch):
//agents/claude-code/agents/linear/categories/code-devtools/categories/data-analytics/changelog/browse/browse?task=query-a-databaseBackground requests in Chrome at 1440×900, counting from page load through a scroll to the bottom:
//categories/code-devtools/agents/claude-code/changelogHovering a card prefetches its listing, and the click then renders the page in 48 ms with no further requests.
Testing
tsc; lint (no errors, only the 4 warnings already on main); vitest (684 passed);validate-content;check-try-links; build.robots.txt, thenE2E_BASE_URL=<deployment> npm run test:e2e(smoke and edge only).Checklist
npx tsx scripts/validate-content.tspasses locallycontent/changelog.json(no listing changed)"verified": false(maintainers verify & flip)officialleft unset (maintainers set it; if you think it applies, link the evidence above. See CONTRIBUTING.md → Official listings)