Repository navigation
Sites: POST /api/v1/sites/records for records a crawler already read (0.57.0) - #265
Merged
Merged
Conversation
…(0.57.0) pit.world's crawler reads pages itself (up to 1,000 per site, JavaScript pages through Obscura) and pushes OpenSite records here in batches of up to 200. Admin API key only. Each record is validated on its own (public address, known kind/status, spec size) and upserted by canonical address into a new disabled source, pit-crawler. Enrichment rides along under data.enrichment; keywords become topic: tags and the excerpt the summary, since search is title + summary + tags. The crawler's text hash becomes the content hash, so a re-crawl of an unchanged page writes nothing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
POST /api/v1/sites/records: an admin API key pushes up to 200 OpenSite records a crawler already read (pit.world's new spider). Records are validated one by one and upserted by canonical address into a new disabled sourcepit-crawler; enrichment (language, keywords, excerpt, word/link counts, rendered flag) goes todata.enrichment, keywords becometopic:tags and the excerpt fills the summary so search finds the body. The crawler's text hash is the content hash, so an unchanged re-crawl writes nothing.Version 0.57.0. Tests: sites route + opensite adapter (29 pass); full suite green apart from the adapter slug list, updated here.
🤖 Generated with Claude Code