Repository navigation
Refresh button + tiered crawl queue + archive silent feeds - #257
Conversation
- POST /api/feeds/<slug>/refresh + Refresh feed button: runs crawlFeed in the request (5-min guard, social sources jump their tier instead) - dueFeeds takes each crawl tier's share (hot 40 / active 30 / slow 18 / dormant 10 / archived 2), oldest due first within a tier, unused slots handed on: a live show no longer waits behind a dormant backlog after a stall - cadence from @profullstack/scraper-daemon 0.3.0: 15 min (dated feeds) to a week by rhythm, tighter around the next expected post (weekly show: 6 h on release day, not 3.5 days); silent 180 days = archived, read every 90 - feeds_due_tier_idx (built concurrently on prod), per-tier backlog in the queue-sample log, archived flag on /api/feeds/<slug> and the page Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
|
Warning Review the following alerts detected in dependencies. According to your organization's Security Policy, it is recommended to resolve "Warn" alerts. Learn more about Socket for GitHub.
|
Fixes stale live feeds such as https://rssamplifier.com/this-week-in-tech-audio (last read 10-21, due 10-25, still waiting on 10-29).
Why it was stale: after the 10-24..28 crawler wedge, 260k feeds were due at once and the queue was oldest-due-first, so 144k live feeds (posted in the last 30 days) sat more than a day late behind dormant ones. The cadence also read a weekly show every 3.5 days.
What changes
POST /api/feeds/<slug>/refresh, which runs the crawler's owncrawlFeedin the request (5-min guard; social sources move to the front of their tier).dueFeedsgives each tier (hot/active/slow/dormant/archived) a share of the batch, oldest due first within a tier, unused slots handed on. Indexfeeds_due_tier_idx(already built concurrently on prod; the migration is a no-op there).@profullstack/scraper-daemon@0.3.0(shared with p0dcasters): 15 min to 1 week by posting rhythm, tighter around the next expected post; silent 180 days -> archived (kept, read every 90 days, 2% of the queue).archivedon/api/feeds/<slug>, a note on the page, per-tier backlog in the poller log.Tests: db 347, ingest (cadence/changes/recrawl updated to the new policy), web 510, poller 30, all passing locally.
🤖 Generated with Claude Code