Skip to content

feat(weaviate): add the Weaviate provider, read-only over its REST API - #1949

Open
kaya-abdullah wants to merge 1 commit into
mainfrom
feat/weaviate-provider
Open

kaya-abdullah wants to merge 1 commit into
mainfrom
feat/weaviate-provider

Conversation

@kaya-abdullah

Copy link
Copy Markdown
Member

Description

The Weaviate provider, weaviate: read-only over Weaviate's REST API, and over GraphQL for the searches of a server older than 1.39.7, tested on Weaviate 1.40.2, 1.38.22 and 1.37.4.
Browse collections with their properties and vectors, see tenants and aliases in a collection's Source, and run Weaviate's own REST read requests (object reads, BM25, hybrid at alpha 0, near-vector and near-object searches, aggregates) from a closed console of 17 routes over a REST client of Studio's own.
A search is written in the REST shape on every server; the provider picks REST or GraphQL from the server's version and falls back once when a path is switched off.
Studio sends no text to a model on Weaviate (every search that would embed text on the server is refused before anything is sent), never wakes a COLD tenant, and sends a key without TLS only to this machine or through an SSH tunnel.

The design is the body of #1566; the measurements behind each rule are in docs/providers/weaviate.md.

Release note

New: Weaviate, as the weaviate connection type, tested on Weaviate 1.40.2, 1.38.22 and 1.37.4 (the line DigitalOcean Managed Weaviate runs).
Browse collections with their properties, vectors, tenants and aliases; read objects; and run BM25, hybrid, near-vector and near-object searches and aggregates from a console that takes Weaviate's own METHOD /v1/path requests with one JSON body.
Studio only reads on Weaviate, refuses every search that would send text to a model provider, refuses a request that names a tenant that is not HOT, accepts an API key or a pasted OIDC access token, and shows MCP clients the declared properties and vectors alone.
Published types widened: DatabaseType, QueryTab["type"] and ProviderCapabilities.queryDialect gain weaviate, and the QueryEditor language prop gains weaviate.

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Code refactoring
  • Performance improvement
  • Test addition or update

Related Issue

Closes #1566. Part of #424 and #1089.

Changes Made

Provider (src/lib/db/providers/vector/weaviate/, 24 modules)

  • client.ts, routes.ts, rest-client.ts: the 19 operations the client can send, the 17 console routes plus the readiness probe and the GraphQL read; every other route is refused by name before the wire.
  • versions.ts: the version table (REST search from 1.39.7, REST near-vector from 1.39.8, GraphQL below), measured on ten releases from 1.37.4 to 1.40.2.
  • request.ts, guard.ts: the two-phase request rules: closed body and filter keys at every level, alpha must be 0, no near-text, generative or rerank search, tenant and name rules, bounds.
  • graphql.ts, execute.ts, results.ts: one REST-shaped request encoded for either path, the one-time fallback when a path is switched off, the tenant guard, and the same columns and cell types on both paths.
  • schema.ts, objects.ts, source.ts, monitoring*.ts, labels.ts, generators.ts: the tree, columns, Source (Schema and State), Tables and index panels, and the generated requests.
  • connection-options.ts, errors.ts: the bearer header, the plaintext-key rule, TLS material, and the error table with the secret withheld from every server text.
  • Registry rows beside the provider: connection-string.ts, execution-ui.ts, sql-grammar.ts, sql-writing.ts, migration-ddl.ts.

Registration (every shared table gains a weaviate row; no other engine's row changes)

  • src/lib/types.ts, src/lib/db/types.ts, compatibility.ts (shipped, external, read-only enforced, MCP), factory.ts, connection-fields.ts, connection-form.ts (appended to the picker), credential-warnings.ts, destructive-commands.ts, query-dialects.ts, src/lib/agent/agent-manifest.ts (no agent language).
  • UI: db-ui-config.ts, db-icons.tsx, db-showcase.ts, QueryEditor.tsx, AnswerCard.tsx, and a zinc hue in theme.css (light, dark and @theme), mapped in the VS Code extension's theme bridge.

Fixtures and live tools

  • database-compose.yml and docker/weaviate/: six servers (1.40.2, 1.38.22, 1.37.4, 1.40.2 with GraphQL disabled, a keyed server with a viewer user, and a TLS server), their seeds and key generation.
  • tests/live/weaviate-evidence.ts writes the 17 captures under tests/fixtures/weaviate/; tests/live/weaviate-live-check.ts holds a live run to them; tests/live/weaviate-path-parity.ts compares REST and GraphQL answers.

Security

  • docs/SECURITY.md: control 3.25 (the Weaviate console's read routes, no text to a model, no COLD tenant) with its tests, and the Known limits entry; scripts/security-check.mjs lists the control.

Docs

  • docs/providers/weaviate.md, the providers index, ARCHITECTURE.md, DATABASE_PROVIDERS.md, FEATURES.md, API_DOCS.md, MCP.md, SCHEMA_DIFF.md, editor/query-optimization.md, AGENT_GUIDE.md, AGENT_DATA_FLOW.md, AGENT.md, SEED_CONNECTIONS.md, ADDING_A_PROVIDER.md, BRAND_MESSAGING.md.
  • Engine counts and lists: the README and its eight translations, DOCKERHUB.md, the chart description, keywords and README (chart 0.1.103), the operator CSV and its bundle copy, the AWS, Azure, CapRover, DigitalOcean, Koyeb, Railway and Rancher listings, AUR, nfpm, winget, Chocolatey, snap, Flatpak and the desktop entry.

Testing

  • I have tested this locally
  • I have added/updated tests
  • All existing tests pass

Gates

  • format, lint (no new warning in any changed file), typecheck, knip, the four drift guards, every test file, build, build:lib and attw pass.
  • Coverage: 184,765 of 184,765 lines, after rebasing onto the current main.
  • Known local failures, present on main as well: the Databend wire-heap test on Node 24.11 (passes on Node 24.16), and the Kafka TLS handshake, Snowflake key-runtimes and Docker build-context tests under parallel load (each passes on its own).

New tests

  • 23 unit files under tests/unit/db/weaviate/, all 24 modules at 100% line coverage on their own.
  • tests/integration/db/weaviate-provider.test.ts replays 17 captures written against the six live servers: every request the provider sends and every outcome it shows must match what the servers answered.
  • e2e/weaviate-provider.spec.ts (5 tests, own Playwright project).

Live checks against the six servers

  • Live check: every scenario matched its capture, and the schema, aliases and every tenant's status were unchanged after each run.
  • Path parity: 72 requests (every where operator and value type, every search kind, the aggregates) answered the same rows with the same cell types over REST on 1.40.2 and over GraphQL on 1.38.22 and 1.37.4.
  • The version table on 1.39.0, 1.39.6, 1.39.7, 1.39.8 and 1.39.13 with GraphQL disabled, and 1.39.7 with GraphQL on: each search ran on the path the table names, and where neither path is served it is refused in one sentence while browsing still works.
  • Egress: with a collection whose text2vec-openai base URL is a listener on the Docker bridge, nothing Studio sends reached the listener on 1.37.4 or 1.40.2.

In the browser (the built app and Chromium on the fixture network; each step screenshotted and checked)

  • The full Weaviate surface on the four plaintext servers, 388 steps: the tree, every collection's click, Generate Request run as generated, View Source (Schema and State), every console route, the refusals, Monitoring, export as CSV, JSON, Markdown and HTML, Format, History, Saved, Charts, Pivot, Docs, light theme and a phone-width viewport.
  • Connections: the keyed server with a viewer key and a read-only seed, the TLS server in every SSL mode, seven Test Connection cases (no key, wrong key, a key without TLS to a remote host, plain HTTP to the TLS port, nothing listening), Paste URL of a Weaviate Cloud address, and a user's own connection created, kept across a reload, edited, duplicated and deleted.
  • SSH tunnel through an OpenSSH bastion: a key over plain HTTP reaches the keyed server only through the tunnel, TLS through the tunnel verifies the far end's name, a wrong bastion password and an unknown far end are refused in their own sentences; Qdrant and PostgreSQL through the same bastion.
  • OIDC: a Weaviate trusting Keycloak; a pasted access token connects and searches, an expired one is refused by name in the dialog and on a saved connection, and the token never appears in a message.
  • Cancel and timeout behind a proxy that holds each search for 45 s: CANCEL closes the socket within 3.1 s, and a connection timeout of 5000 ms ends the search at 5.9 s, the same as Qdrant.
  • Bounds on a server with 253 collections, 3,000 objects of about 12 KB, 200,000-character texts and 1,500 tenants: the 8 MiB result budget, the cut text cell, the 100-group notice, the 200-collection Tables bound and the 1,024-tenant Source cut each show their notice.
  • 80 concurrent requests reach the server at most four at a time and all answer; three signed-in browsers searching at once each get their own rows.
  • The AI rail with a stand-in OpenAI-compatible model: Plan mode drafts a request that Apply to editor runs; Agent mode says it has no read-only statement path on Weaviate, as on Qdrant.
  • MCP enabled with an mcp: true seed: list_connections, inspect_schema (properties and vectors, no tenants or aliases) and the run_read_query refusal.
  • The production Docker image built from this branch, on the same four servers.
  • Through all of it, no COLD tenant was woken.

Other engines, this branch against main

  • The same browser plan on both builds for the 18 other engines whose console is JSON (MongoDB, Redis, Firestore, LibreDB, etcd, Oxia, MQTT, Kafka, RabbitMQ, Milvus, Qdrant, S3 on MinIO, OPC UA, MTConnect, Modbus, NETCONF, SNMP, Sparkplug B), PostgreSQL, MySQL, the SQLite sample, every admin and settings page, the login page and a non-admin user: rows, errors, confirmation dialogs and trees match step for step; the remaining differences are live counters and server ordering.
  • The connection dialog's form for each of the other 42 engines is identical on both builds; the picker gains Weaviate at the end and nothing else moves.
  • The full Playwright suite: the three tests that fail on this branch locally fail on main too (a PostgreSQL connection on localhost blocked by the local VPN, and two WebKit tests).

Test Environment

  • LibreDB Studio Version: main at 1d5f63b plus this branch
  • Browser: Chromium (Playwright 1.64.1, headless)
  • OS: Linux
  • Node.js/Bun Version: Bun 1.4.2, Node 24.11.0 (24.16.0 for the extension and the Node-version-bound tests)
  • Database Type: Weaviate 1.40.2, 1.39.0 to 1.39.13, 1.38.22, 1.37.4; for comparison the 18 JSON-console engines above, PostgreSQL, MySQL and SQLite

Screenshots (if applicable)

Not attached; the browser pass above saved a screenshot of every step, available on request.

Checklist

  • My code follows the project's code style guidelines
  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have updated the documentation accordingly
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • The required CI test job passes the 100% line-coverage gate (bun run test:coverage and bun run coverage:check)
  • If I changed src/lib/db/providers/, I updated the matching docs/providers/ documentation and tests/integration/db/ tests in the same PR (provider triad)
  • If the type-id has a non-null AGENT_STATEMENT_LANGUAGE row, I kept its agent leg in step (the weaviate row is null: no agent leg)
  • Any dependent changes have been merged and published

Additional Notes

Decisions worth a look during review:

  • The 1.37 line is supported although it left Weaviate's support window, because DigitalOcean Managed Weaviate runs 1.37.4; a server below 1.37 connects with a warning.
  • Hybrid runs only at alpha 0: measured, it then sends nothing to the vectorizer, while any other alpha and near-text do.
  • A tree click on a multi-tenant collection is refused with the request that lists its tenants, rather than picking a tenant.
  • Generate Request writes a runnable BM25 search when no object has shown a vector's length, because a guessed length is a server error.
  • An aggregate with groupBy and no limit is sent with 100, the number both paths return then, and says when the groups reached it.
  • Weaviate 1.37 counts an aggregate filtered by WithinGeoRange as 0 while a search finds the objects (1.38 and 1.40 count them); such a count carries a note.
  • Found by the testing above and fixed here: Generate Request on a vectorized collection, the next-page notice after the result budget cut a page, a value-list filter over GraphQL, a BM25 search sending targetVector, and offset=0.
  • Seen on main and left for separate issues: the Qdrant scroll notice can skip points the result budget dropped, and a seed's queryTimeout field is ignored (the dialog's value works).

@kaya-abdullah kaya-abdullah self-assigned this Oct 24, 2026
@kaya-abdullah
kaya-abdullah requested a review from cevheri October 24, 2026 09:12
@codecov

codecov Bot commented Oct 24, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

Add the weaviate connection type. Studio browses a Weaviate server's
collections with their properties and vectors, shows tenants and aliases
in a collection's Source, and runs Weaviate's own REST read requests from
a closed console of 17 routes: object reads, BM25, hybrid at alpha 0,
near-vector and near-object searches, and aggregates.

A search is written in the REST shape on every server. The provider sends
it over REST search from 1.39.7 (near-vector from 1.39.8) and over GraphQL
below, falls back once when a path is switched off, and shows the same
columns and cell types on both paths. Tested on 1.40.2, 1.38.22 and
1.37.4, the line DigitalOcean Managed Weaviate runs.

Every request is a read. Searches that would make the server send text to
a model (near-text, generative, rerank, hybrid with alpha other than 0)
are refused before anything is sent; a request naming a tenant that is not
HOT is refused after reading the tenant's own record, so no read wakes a
COLD tenant; an API key or OIDC token goes without TLS only to this
machine or through an SSH tunnel.

Register the type in every shared table, add the provider document,
security control 3.25, six fixture servers with seeds, 17 captures with
the harness that writes them, a live check and a REST/GraphQL parity
check, unit, integration and end-to-end tests, and update every engine
count and list in the docs, READMEs, chart, operator and listings.

Closes #1566
@kaya-abdullah
kaya-abdullah force-pushed the feat/weaviate-provider branch from f114775 to 169845c Compare October 24, 2026 09:26
@sonarqubecloud

Copy link
Copy Markdown

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

database-provider enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Weaviate provider: read-only browsing and vector search over HTTP

1 participant