Repository navigation
pitbot: pit.world's own crawler and index (0.13.0) - #91
Merged
Merged
Conversation
A BullMQ spider in a new services/ workspace: up to 1,000 pages per site, JavaScript-only pages rendered in Obscura, each page enriched (language, keywords, excerpt, counts, text hash) and published as an OpenSite record to nichedb.dev/c/sites through POST /api/v1/sites/records (nichedb 0.57.0). - Four queues on pit.world's own Redis: site (robots.txt, sitemaps, home), page (shallow first; a site inside its request gap is delayed, not held), render (one page per Obscura server), publish (outbox to nichedb every 5 s, hourly recrawl of sites whose 30-day cycle ran out). - Exact per-site admission and page cap in one Lua call; any number of workers on any number of boxes share the crawl. - robots.txt (pitbot group, Crawl-delay), noindex/nofollow, no private addresses at any redirect hop, media/traps skipped. - Backfill: seed --from nichedb (every site /c/sites keeps) or --from ring. - Search: 'pit' engine fused with the others, reading nichedb's sites collection (PIT_INDEX_URL, on by default; PIT_INDEX_URL=off disables). - /bot page; privacy page names the index. - Deploy: deploy/crawler docker compose (crawler + Redis + 2 Obscura), poll.sh rebuilds on new commits to main. - CI: Redis service for the crawler's integration tests, crawler typecheck; ThreatCrush false positive on deploy.yml's key armour suppressed (main was red on it since #90). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
The crawler now keeps its counters in minute and hour buckets and a 200-line event log in Redis, and answers /stats on loopback with queue depths, hour/day rates, the hourly curve, the log tail and its settings. pit.world reads that (CRAWLER_STATUS_URL, default 127.0.0.1:8807, 3 s leash, 5 s cache) and draws /status: a health verdict (healthy, idle, degraded, stalled, offline), a stat grid, pages per hour for 24 hours, a jobs table with backlog, rate, errors and time to clear, and the live log with each page linked to its nichedb record. Refreshes every 15 s; /api/status is the same as JSON. A crawler that does not answer shows as offline, never as an error page. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Nov 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
services/crawler(a Bun workspace): pitbot, a BullMQ spider that builds pit.world's own index and publishes it to nichedb.dev, plus apitengine that fuses that index into every search.Crawl
pit-crawl-site(robots.txt, sitemaps, home page),pit-crawl-page(shallow first),pit-crawl-render(JS shells re-read in Obscura, one page per server),pit-crawl-publish(outbox →POST nichedb.dev/api/v1/sites/recordsevery 5 s; hourly recrawl after the 30-day cycle).pitbotgroup, Crawl-delay ≤30 s), noindex/nofollow, no nofollow/sponsored/ugc links, no private addresses at any redirect hop (name + DNS), media/traps skipped./botpage explains it and how to opt out.topic:tags), main-text excerpt (→ summary), word/link counts, rendered flag, depth, text hash (unchanged pages are not rewritten).seed --from nichedb(every site nichedb's /c/sites keeps),--from ring, files, names.Search:
pitprovider ("pit.world index") last in PRIMARY, readingnichedb.dev/api/v1/search?collection=sites. On by default;PIT_INDEX_URL=offdisables,PIT_INDEX_KEYadds a bearer./status (like rssamplifier's /crawlstats): health verdict (healthy/idle/degraded/stalled/offline), stat grid, pages per hour for 24 h, jobs table (backlog, rate, errors, time to clear), live log with each page linked to its nichedb record; refreshes every 15 s, JSON at
/api/status. The crawler keeps minute/hour counter buckets and a 200-line event log in Redis and serves them on loopback/stats; pit.world reads that (CRAWLER_STATUS_URL, default 127.0.0.1:8807).Deploy:
deploy/crawler/docker-compose.yml(crawler + Redis + 2 Obscura, loopback only, healthcheck),poll.shrebuilds on new commits to main. Already booted from this branch on dev2 (~/www/pit-crawler, /healthz on 127.0.0.1:8807).Tests: 242 pass (
bun test), including the real-Redis frontier/queue tests (CI gets a Redis service) and a real Obscura render of react.dev locally. Both typechecks clean. ThreatCrush--deps --fail-on mediumclean; this also suppresses the false positive that has kept main red since #90 (deploy.yml rebuilds the secret's key armour; nothing committed).Depends on profullstack/niche-db#265 (merged, v0.57.0).
🤖 Generated with Claude Code