[JuampiHernandez]

Hey, I'm Juampi from the Firecrawl team.

What & why

This adds Firecrawl as one more opt-in keyed engine in the search fallback chain behind DuckDuckGo. It's built the same way as the keyed adapters added in #14 and #39: same 5 files, same spots in each. With FIRECRAWL_API_KEY unset, nothing changes.

The chain is there for site: lookups that come back empty from DDG. Firecrawl search handles site: operators in the query, so it covers that case (live output below), and it returns the plain title, URL and snippet for each result with no page content fetched.

Changes

• apps/api/services/leadgen/search_engines.py: new _firecrawl adapter, registered as firecrawl in _ADAPTERS and appended last in the default SEARCH_FALLBACK_ORDER, so existing setups route exactly as before. It sends POST https://api.firecrawl.dev/v2/search with the query in the JSON body and the key in an Authorization: Bearer header. limit is capped at 100, the API's per-request max, and highlights is off so description is the plain result snippet. data.web[] (title / url / description) maps to the shared title / href / body shape, and links that aren't http(s) are skipped. A new _http_post_json helper sits next to _http_get_json with the same contract: same timeout, None on any failure, nothing raised to the caller.
• apps/api/services/leadgen/config.py: FIRECRAWL_API_KEY.
• .env.example: commented FIRECRAWL_API_KEY line.
• tests/test_search_multi_engine.py: Firecrawl in the shape-parity test and the clean-env lists, plus test_firecrawl_posts_query_with_bearer_key_and_caps_limit (POST only, endpoint, bearer header, key not in the URL or body, limit cap and pass-through, non-URL link dropped, no page content or highlights requested).
• tests/test_search_cache.py: FIRECRAWL_API_KEY in the clean-env list.

Type

• Bug fix
• New feature
• New enrichment provider / source
• Docs
• Refactor / chore

Checklist

• Commits are signed off (git commit -s) per the DCO
• Backend tests pass (uv run pytest): 2360 passed, 134 skipped, 8 deselected with CI's -m 'not live and not postgres'
• Monorepo lint and build pass (bun run lint && bun run build): 4 of 4 tasks successful
• I did not add scraping of paywalled / pirated / login-gated content
• Credentialed providers are opt-in and inert by default

Security-sensitive?

Yes, it adds one outbound fetch to a fixed host (api.firecrawl.dev) over HTTPS, only when FIRECRAWL_API_KEY is set. You asked about these points on #14, so here they are for this provider:

• What leaves the instance: the search query string and a result limit. Nothing else from the lead or workspace is sent. It goes by POST, so the query isn't in the URL and doesn't show up in logs that record URLs. The key travels in the Authorization header, and failures return [] without logging it.
• Retention: Firecrawl doesn't publish a fixed retention period for search queries on standard plans. Its Privacy Policy keeps personal data until you ask in writing for it to be deleted, and data is processed in the US. Firecrawl search uses an upstream search provider. Search zero data retention is an Enterprise option that needs enterprise: ["zdr"] on the request, enforced at Firecrawl and at the upstream provider. This adapter doesn't send it, so users should assume standard retention. Details are on the security overview.
• DPA and subprocessors: Enterprise customers can sign a DPA, and the subprocessor list is in the Trust Center linked from the same page.
• Terms: each operator uses their own key under Firecrawl's Terms of Service. If you need written confirmation for a specific use, such as caching returned results, help@firecrawl.dev handles those questions.

If you want a short note like this next to the key in .env.example or the docs, or an opt-in setting that sends the ZDR flag for teams that have it, I can add it here.

Screenshots / notes

How I tested it

uv run pytest tests/test_search_multi_engine.py tests/test_search_cache.py: 33 passed. Then a live run with a real key and SEARCH_FALLBACK_ORDER=firecrawl:

>>> se.configured_engines()
['firecrawl']
>>> se.search_fallback("site:crunchbase.com/organization stripe", max_results=3)
- Stripe - Crunchbase Company Profile & Funding | https://www.crunchbase.com/organization/stripe
- Financial Details - Stripe - Crunchbase | https://www.crunchbase.com/organization/stripe/financial_details
- Stripe - Crunchbase Company Profile & Funding | https://www.crunchbase.com/organization/stripe-841c
>>> se.search_fallback("site:linkedin.com/company posthog", max_results=3)
- PostHog - LinkedIn | https://www.linkedin.com/company/posthog
- PostHog | LinkedIn | https://www.linkedin.com/company/posthog/?trk=products_details_guest_organization_page
- PostHog | LinkedIn | https://ar.linkedin.com/company/posthog
(each body above is the plain result snippet, 134 to 160 characters here)
>>> FIRECRAWL_API_KEY=<invalid> se._firecrawl("site:crunchbase.com/organization stripe", 3)
[]

Links: firecrawl.dev, search API docs.

[debpalash]

Thanks for contributing the Firecrawl integration. The native Go implementation is now merged in #118, carrying the opt-in fallback behavior proposed here with Go-side request/response guards and tests.

Closing this Python-backend PR as superseded by #118. The accompanying Spicrawl integration and sponsor README are also merged in #119.