Skip to content

Refresh button + tiered crawl queue + archive silent feeds - #257

Merged
sh1pt-actions-fleet[bot] merged 1 commit into
mainfrom
feat/refresh-tiered-crawl
Oct 29, 2026
Merged

sh1pt-actions-fleet[bot] merged 1 commit into
mainfrom
feat/refresh-tiered-crawl

Conversation

@sh1pt-actions-fleet

Copy link
Copy Markdown
Contributor

Fixes stale live feeds such as https://rssamplifier.com/this-week-in-tech-audio (last read 10-21, due 10-25, still waiting on 10-29).

Why it was stale: after the 10-24..28 crawler wedge, 260k feeds were due at once and the queue was oldest-due-first, so 144k live feeds (posted in the last 30 days) sat more than a day late behind dormant ones. The cadence also read a weekly show every 3.5 days.

What changes

  • Refresh feed button on every feed page -> POST /api/feeds/<slug>/refresh, which runs the crawler's own crawlFeed in the request (5-min guard; social sources move to the front of their tier).
  • Tiered due queue: dueFeeds gives each tier (hot/active/slow/dormant/archived) a share of the batch, oldest due first within a tier, unused slots handed on. Index feeds_due_tier_idx (already built concurrently on prod; the migration is a no-op there).
  • Cadence from @profullstack/scraper-daemon@0.3.0 (shared with p0dcasters): 15 min to 1 week by posting rhythm, tighter around the next expected post; silent 180 days -> archived (kept, read every 90 days, 2% of the queue).
  • archived on /api/feeds/<slug>, a note on the page, per-tier backlog in the poller log.

Tests: db 347, ingest (cadence/changes/recrawl updated to the new policy), web 510, poller 30, all passing locally.

🤖 Generated with Claude Code

- POST /api/feeds/<slug>/refresh + Refresh feed button: runs crawlFeed in the
  request (5-min guard, social sources jump their tier instead)
- dueFeeds takes each crawl tier's share (hot 40 / active 30 / slow 18 /
  dormant 10 / archived 2), oldest due first within a tier, unused slots handed
  on: a live show no longer waits behind a dormant backlog after a stall
- cadence from @profullstack/scraper-daemon 0.3.0: 15 min (dated feeds) to a
  week by rhythm, tighter around the next expected post (weekly show: 6 h on
  release day, not 3.5 days); silent 180 days = archived, read every 90
- feeds_due_tier_idx (built concurrently on prod), per-tier backlog in the
  queue-sample log, archived flag on /api/feeds/<slug> and the page

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@socket-security

Copy link
Copy Markdown

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff Package Supply Chain
Security
Vulnerability Quality Maintenance License
Updated@​profullstack/​scraper-daemon@​0.2.2 ⏵ 0.3.077 +110010094 +3100

View full report

@socket-security

Copy link
Copy Markdown

Warning

Review the following alerts detected in dependencies.

According to your organization's Security Policy, it is recommended to resolve "Warn" alerts. Learn more about Socket for GitHub.

Priority Alert  (click "▶" to expand/collapse) Action
Low priority
Low adoption: npm @profullstack/scraper-daemon

Location: Package overview

From: apps/poller/package.json → npm/@profullstack/scraper-daemon@0.3.0

ℹ Read more on: This package | This alert | What are unpopular packages?

Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev.

Suggestion: Unpopular packages may have less maintenance and contain other problems.

Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore npm/@profullstack/scraper-daemon@0.3.0. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.

Warn

View full report

@sh1pt-actions-fleet
sh1pt-actions-fleet Bot merged commit 627190e into main Oct 29, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant