triage: WebSearch gets DuckDuckGo's 202 challenge page; proposal for an opt-in keyless API backend #3364

Initial classification

actionable-feature

Evidence

• execute_web_search in rust/crates/tools/src/lib.rs always GETs https://html.duckduckgo.com/html/?q=... (build_search_url). CLAWD_WEB_SEARCH_BASE_URL can only point it at another page that serves the same HTML.
• On 2026-10-23 three such GETs with the tool's user agent (clawd-rust-tools/0.1) from one connection all got HTTP 202 with DuckDuckGo's challenge page and no result__a links. extract_search_hits_from_generic_links then turns that page into two hits, https://duckduckgo.com/html/ and https://duckduckgo.com/. This matches [BUG] WebSearch returns empty results or DuckDuckGo redirect pages, while WebFetch works with direct URLs #2820, which was closed in triage.
• Proposal: an opt-in CLAWD_WEB_SEARCH_PROVIDER=keenable; unset keeps DuckDuckGo exactly as now. It POSTs {"query": ..., "max_results": 8} to https://api.keenable.ai/v1/search/public with X-Keenable-Title: claw-code and no key (keyless limit: 1,000 requests/hour per IP). If KEENABLE_API_KEY is set it calls /v1/search with X-API-Key, which only lifts the limits. Results map to the existing SearchHit { title, url } and go through the same domain filters, dedupe_hits and truncate(8). It uses the blocking reqwest client already in the crate, so no new dependency.
• Keyless check on 2026-10-23 for reqwest blocking client redirect policy (a query that got the 202 page from DuckDuckGo): HTTP 200, 10 results in 0.40 s.

Safe next action

Owner review of the variable names and of whether a provider switch fits the tools crate. If yes, I will send the PR: the switch in build_search_url / execute_web_search in rust/crates/tools/src/lib.rs, inline tests against a local mock server in the style of web_search_extracts_and_filters_results, and both variables in USAGE.md.

Disclosure: I work at Keenable, which runs this API.

Guardrails

• I did not close, merge, or mutate remote state as part of this triage-only report.
• I checked for duplicates or related PRs/issues before recommending action.
• If this touches credentials, security, or private data, I avoided public reproduction details and routed to the appropriate private/security path.