Enhance extract.ai schema guidance and add web search

[ebhills]

Summary

• expand the extract.ai recipe schema hints so the WranglesXL editor explains the supported output schema, runtime parameters, and example forms
• add optional OpenAI Responses API web search with automatic per-row web_search_sources provenance
• distinguish top-level record_examples from field-level examples
• rename the public messages option to instructions

Behavior and API impact

• web_search: true enables the native Responses API web-search tool and always adds a web_search_sources output column
• each source is represented as {title, url}; duplicates are removed while preserving source order, and rows with no sources receive an empty list
• web_search_sources is reserved when web search is enabled
• top-level record_examples describe complete input/output records
• nested field examples continue to accept scalar or list value guidance and may also use paired input/output objects
• both field and record pairs accept optional model-visible name and notes
• instructions supplies guidance that applies to every input row

Compatibility

• web search is off by default and is supported only by the Responses protocol
• legacy Chat Completions behavior remains available when web search is disabled
• nested scalar/list examples remain backward compatible
• direct Python examples= remains an alias for record_examples=
• messages remains accepted by Python and recipe execution as an alias for instructions, but is no longer advertised in the recipe schema
• providing both a public name and its compatibility alias raises a clear error

Validation

• 72 passed across the focused definition compiler, Responses transport, cache, web-search recipe wrapper, and schema contract tests
• Python compilation and notebook JSON validation passed
• the generated Draft-07 recipe schema was regenerated and inspected successfully
• live OpenAI Responses API validation passed for web search with record/field example metadata and returned 17 sources
• live OpenAI Responses API validation passed for the new instructions key with web search and returned 17 sources
• git diff --check origin/main...HEAD passed

Risk and rollback

Web search is opt-in, so existing calls do not change unless they use a renamed schema option. Source provenance is returned alongside cached extraction results and can be bypassed with cache: false when freshness is required. Rollback is the three feature commits on this branch.

[Copilot]

Pull request overview

Adds richer extract.ai schema guidance, record examples, renamed instructions, and optional OpenAI web-search provenance.

Changes:

• Adds Responses API web search and per-row source metadata.
• Introduces record_examples, metadata, and compatibility aliases.
• Expands schema documentation and focused tests.

Recommended disposition: Request changes

Next steps

1. PR assignee: Initialize response_json before parsing and add a malformed-JSON response regression test.
2. Reviewer: Verify the fix, resolve the thread, and submit a fresh approval.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
wrangles/extract.py	Adds web-search orchestration and API aliases.
wrangles/openai_responses.py	Extracts provenance from Responses results.
wrangles/ai_definition.py	Supports example names and notes.
wrangles/recipe_wrangles/extract.py	Exposes recipe options and source columns.
tests/test_openai_extract_ai.py	Tests transport, caching, and aliases.
tests/test_ai_definition.py	Tests example compilation and recipes.
tests/recipes/wrangles/test_extract.py	Tests recipe source-column behavior.
tests/recipes/wrangles/test_main.py	Validates schema documentation.
tests/samples/extract ai judge example.wrgl.yml	Updates sample parameter names.
docs/extract_ai_configuration.md	Documents instructions and examples.
docs/examples/extract_ai/extract_ai_recipe_examples.ipynb	Adds usage examples and web search.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.