hnt-content

Article crawling and extraction pipeline for Firefox New Tab content recommendations. Crawls publisher pages, discovers articles, extracts content via Zyte, and streams results to BigQuery for ML ranking.

Development

nvm use  # Node 24
pnpm install
pnpm build
pnpm test

Run a service locally (no build step, uses tsx):

pnpm --filter crawl-scheduler dev
pnpm --filter crawl-worker dev

Command	Description
pnpm build	Build all packages and services
pnpm test	Run all tests
pnpm lint	Lint all packages
pnpm format	Format source files with Prettier
pnpm format:check	Check formatting (CI)
pnpm clean	Remove all build artifacts and node_modules

Architecture

See the Article Crawler Technical Spec for the full design. In brief:

• Crawl Scheduler runs a tick loop every 60s, checking which publisher pages and live articles need crawling based on Redis state, then enqueues jobs to Pub/Sub.
• Crawl Worker consumes from two Pub/Sub queues: crawl-article-discovery (page crawling) and crawl-article (article extraction). Results stream to BigQuery via Pub/Sub subscriptions.
• Redis (Memorystore) tracks crawl timestamps, prevents duplicate fetches, and provides distributed locking.

Repository structure

hnt-content/
├── services/
│   ├── crawl-scheduler/  # Enqueues crawl jobs on configured intervals
│   └── crawl-worker/     # Discovers articles and extracts content
├── packages/
│   ├── crawl-common/     # Shared types, utilities, Corpus API client
│   ├── metrics/          # OpenTelemetry metrics client
│   ├── pubsub/           # Pub/Sub publisher and subscriber
│   ├── redis-state/      # Redis crawl state and locks
│   ├── sentry/           # Sentry init and handler wrapper
│   └── zyte/             # Zyte API client
├── Dockerfile            # Multi-stage build with turbo prune + pnpm deploy
├── turbo.json
└── pnpm-workspace.yaml

Deployment

The services deploy when a pull request merges to main. GitHub Actions builds the image and pushes it to Artifact Registry, then ArgoCD Image Updater notices the new digest and rolls it out. Merging to main publishes the latest tag, which stage and prod track. Pushing to the dev branch (e.g. git push -f origin HEAD:dev) publishes the dev tag, which only hnt-dev tracks, so dev can run a branch ahead of main.

Per MozCloud convention the pods run on a shared platform cluster in moz-fx-webservices-high-prod, while this project's own infrastructure, including Pub/Sub, Redis and BigQuery, lives in moz-fx-hnt-prod, so the chart passes PROJECT_ID for the Google client libraries to use; dev and stage mirror that split in the nonprod projects.

One image contains both entry points. The shared mozcloud Helm chart chooses which one each workload runs and sets WORKER_ROLE to select the discovery or article role, so all three workloads come from one build.

For how the system fits together, see docs/crawl/ARCHITECTURE.md.