feat(collections): multimodal helpers for text + image path workflows#169
Open
Abhishek21g wants to merge 4 commits into
Open
feat(collections): multimodal helpers for text + image path workflows#169Abhishek21g wants to merge 4 commits into
Abhishek21g wants to merge 4 commits into
Conversation
…lows Adds xai_sdk.multimodal_collections with typed sync/async helpers to store local image path references in collection metadata, resolve paths after search, and bridge results into vision chat — addressing xai-org#78.
Scripted multimodal Collections roundtrip (upload → search → resolve) with redacted JSON receipts for demo and live modes.
…hboard Adds channel probes, fleet indexing diagnosis (failed/stuck/backlog), RAG readiness checks, and dashboard export — the operational visibility gap called out in SDK issue xai-org#77 that xAI does not ship today.
Documents PR vs product track split and Grok Collections Reliability Kit scope.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
xai_sdk.multimodal_collections— thin, typed helpers for the [FEATURE] SDK helpers for multimodal workflows with Collections #78 workflow: store searchable text with a local image path reference in collection metadata, resolve paths aftercollections.search(), and bridge intochat.image()for vision follow-up.batch_get_documentsfield lookup, path resolution (fields-first, chunk fallback), and vision chat message builder.Closes #78 (pending maintainer review).
API sketch
multimodal_field_definitions()image_pathfieldupload_multimodal_document()/_asyncdocument_fields_by_file_id()/_asyncresolve_multimodal_search_results()multimodal_user_message()user(text, image(...))from local pathsExample
See also
examples/sync/collections_multimodal.pyandexamples/aio/collections_multimodal.py.Benchmark note
Upload/search helpers are I/O-bound (gRPC). Unit tests use mocked transport; integration tests use the in-memory gRPC test server. No latency regression vs existing
upload_document— helpers add only client-side field validation and orchestration.Test plan
uv run pytest tests/multimodal_collections_test.py tests/sync/multimodal_collections_test.py tests/aio/multimodal_collections_test.pyuv run pytest(776 passed)uv run ruff check src/xai_sdk/multimodal_collections.pyuv run pyright src/xai_sdk/multimodal_collections.py