Skip to content

feat(collections): multimodal helpers for text + image path workflows#169

Open
Abhishek21g wants to merge 4 commits into
xai-org:mainfrom
Abhishek21g:feature/collections-multimodal
Open

feat(collections): multimodal helpers for text + image path workflows#169
Abhishek21g wants to merge 4 commits into
xai-org:mainfrom
Abhishek21g:feature/collections-multimodal

Conversation

@Abhishek21g

Copy link
Copy Markdown

Summary

  • Adds xai_sdk.multimodal_collections — thin, typed helpers for the [FEATURE] SDK helpers for multimodal workflows with Collections #78 workflow: store searchable text with a local image path reference in collection metadata, resolve paths after collections.search(), and bridge into chat.image() for vision follow-up.
  • Includes sync + async upload wrappers, batch_get_documents field lookup, path resolution (fields-first, chunk fallback), and vision chat message builder.
  • 13 new tests; full suite green (776 passed).

Closes #78 (pending maintainer review).

API sketch

Helper Purpose
multimodal_field_definitions() Collection schema with image_path field
upload_multimodal_document() / _async Upload text + path metadata
document_fields_by_file_id() / _async Fetch fields for search matches
resolve_multimodal_search_results() Attach resolved paths to matches
multimodal_user_message() user(text, image(...)) from local paths

Example

from xai_sdk import Client
from xai_sdk.multimodal_collections import (
    multimodal_field_definitions,
    upload_multimodal_document,
    document_fields_by_file_id,
    resolve_multimodal_search_results,
    multimodal_user_message,
)

client = Client()
col = client.collections.create(name="catalog", field_definitions=multimodal_field_definitions())

upload_multimodal_document(
    client.collections, col.collection_id,
    name="item.txt", text="Red widget, glossy finish.",
    image_path="/data/widget.jpg", wait_for_indexing=True,
)

results = client.collections.search("glossy widget", [col.collection_id], limit=3)
fields = document_fields_by_file_id(client.collections, col.collection_id, results.matches)
hits = resolve_multimodal_search_results(results.matches, fields)

chat = client.chat.create(model="grok-4.20-non-reasoning")
chat.append(multimodal_user_message(f"Context:\n{hits[0]['chunk_content']}", hits[0]['image_paths']))

See also examples/sync/collections_multimodal.py and examples/aio/collections_multimodal.py.

Benchmark note

Upload/search helpers are I/O-bound (gRPC). Unit tests use mocked transport; integration tests use the in-memory gRPC test server. No latency regression vs existing upload_document — helpers add only client-side field validation and orchestration.

Test plan

  • uv run pytest tests/multimodal_collections_test.py tests/sync/multimodal_collections_test.py tests/aio/multimodal_collections_test.py
  • uv run pytest (776 passed)
  • uv run ruff check src/xai_sdk/multimodal_collections.py
  • uv run pyright src/xai_sdk/multimodal_collections.py

…lows

Adds xai_sdk.multimodal_collections with typed sync/async helpers to store
local image path references in collection metadata, resolve paths after search,
and bridge results into vision chat — addressing xai-org#78.
@Abhishek21g Abhishek21g requested a review from a team as a code owner July 6, 2026 11:49
Scripted multimodal Collections roundtrip (upload → search → resolve) with
redacted JSON receipts for demo and live modes.
…hboard

Adds channel probes, fleet indexing diagnosis (failed/stuck/backlog), RAG
readiness checks, and dashboard export — the operational visibility gap
called out in SDK issue xai-org#77 that xAI does not ship today.
Documents PR vs product track split and Grok Collections Reliability Kit scope.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEATURE] SDK helpers for multimodal workflows with Collections

1 participant