[debpalash]

What & why

Builds on the Firecrawl implementation merged in #118. This PR targets main and contains Spicrawl's additions and the requested sponsor README update.

Operators can opt into Spicrawl page/PDF-to-markdown reads for research columns with SPICRAWL_API_KEY. An unset key preserves the browser fetch pipeline. With a keyed search engine also configured, research can run without the browser sidecar.

Default requests use main-content markdown, PDF parsing, no link extraction or JavaScript rendering, and a one-credit ceiling. Explicit JavaScript calls cap at three credits. The client exposes selector options and billing metadata; research previews stay bounded to 2,000 Unicode characters.

The README adds the strategic-sponsor wordmark, tracked official signup/pricing links, and the published free-beta allowance. Demo GIF sections remain expanded. No additional OpenGTM-exclusive bonus is asserted without approved offer details.

Type

• New feature
• Docs

Validation

• Signed-off commit (DCO).
• CLA status to be confirmed before merge.
• go vet ./....
• go test -race -p 2 ./internal/leadweb ./internal/config ./internal/api ./cmd/worker ./internal/aicol.
• Frontend frozen install, lint, 70 tests, and production build.
• API build with embedded frontend assets and worker build.
• Provider remains inactive without its own server-side key.
• No session, cookie or authenticated-target scraping features added.

The full make test check reaches the same existing macOS/ARM one-ULP assertion failure in internal/provaccuracy/TestNormalizeMatchFreshness as #118 and the untouched merged baseline. All other Go unit packages passed; frontend checks were run separately. PostgreSQL/Redis integration checks are left to CI.

Local HTTP fixtures cover markdown/PDF handling, request flags, target status/credit validation, cost ceilings, response and preview limits, redirects, and private/mixed-DNS rejection. Worker tests exercise Spicrawl-only and combined API-only research plus credential-specific client reuse. No paid live requests were made.

Security-sensitive?

Yes: authenticated outbound fetches and LLM page content. The API endpoint is fixed, provider credentials stay in bearer headers and redacting types, and API redirects cannot replay credentials or target URLs. Target syntax and DNS are checked before handoff; the service remains responsible for its remote DNS and redirect hops.

Platform HTTP 200 is insufficient: target status must be 2xx, credits must be valid and within the requested ceiling, and content must be nonempty UTF-8 markdown/plain text within 4 MiB. Invalid metadata and coerced JSON/HTML responses fail closed. No automatic paid retries or JavaScript escalation are added. Provider credits remain separate from the LLM token budget. See docs/providers/spicrawl.md.