Skip to content

Document agent integration, evaluation and provider setup - #1074

Open
zlareb1 wants to merge 53 commits into
mainfrom
docs/agent-answerability
Open

zlareb1 wants to merge 53 commits into
mainfrom
docs/agent-answerability

Conversation

@zlareb1

@zlareb1 zlareb1 commented Nov 20, 2026 •

Copy link
Copy Markdown
Member

Changelog category (leave one):

  • Documentation (changelog entry is not required)

Changelog entry

Not required for documentation.

Why

Agents need a direct path from integration and repository questions to valid requests, access outcomes and provider diagnostics. Canonical guides connect those workflows with explicit evaluation and release boundaries.

Scope and risk

Adds an agent-platform guide and links it from the documentation and generated agent indexes. Extends repository Q&A, linked-memory, evaluation and repository-access guides with concrete examples, observable outcomes and limitations. The proposed paired evaluation records actual tool use, objective task outcomes, total cost and latency without claiming a measured benefit. Customer delivery guidance distinguishes technical integration from support and commercial agreements.

Corrects operation descriptions at their source and regenerates the affected references and SDK comments. Uncached retrieval-only requests need both synthesize: false and no_cache: true. Repository listings can include permitted request-only metadata with readable: false; that metadata grants no code reads. Namespace authorization, indexed-source freshness and asynchronous memory-link boundaries remain explicit.

Adds a native-provider setup recipe, corrects embedding-provider selection and dimension descriptions, and makes proxy provider and model selection explicit. Dashboard guidance distinguishes displayed jobs from the conflict resolver's separate setting. The health checklist distinguishes configured fakes, passive process-local outcomes and actual inference, including compatible-embedding coverage limits.

Clarifies standard token-claim or grant access, restricted namespace prerequisites, worker defaults and model-based capture filtering. Specifies Python compiler-worker ownership, native-provider request negotiation and repository-retirement cache and telemetry retention. Clarifies verifier failure handling and memory vector publication during database re-embedding. Documents persisted repository access overrides, manual cache retirement and ordinary reuse checks; answer-cache hits can still incur query-embedding cost. Distinguishes full backfill for erased operator-configured indexes from self-service bootstrap, and replaces a fixed link-completion estimate with configurable scheduling and actual result checks. Cost guidance distinguishes reported usage and numeric estimates from unknown provider spend. It states that daily caps do not guard transcript extraction and distinguishes default fail-open budget reads from selected fail-closed background work. LLM-free reads can still call embedding or selected reranker providers; cached answers can contain prior synthesis. Local embedding inference may need model-file downloads, and compiler builds require an explicit mandatory-sandbox policy to refuse plain-process execution.

Compatibility, rollout and rollback

  • Wire: no operation, field, setting or default changes. Retrieval, authorization and provider execution remain unchanged.
  • Operator steps: none for the documentation update. Recipes state credential, scope, worker-version and embedding-rebuild prerequisites.
  • Rollback: revert the documentation and generated descriptions and index rows.

Tests

  • Current SDK/reference generator checks, operation reach census, changed-Python lint and formatting, pinned Markdown formatting, added-prose and diff checks pass.
  • Full unmodified documentation-link, public-copy, type and test suites are pending in hosted CI. No broad local suite was run for this composition.
  • Six HTTP request examples and the Python example retain their previously validated literal bytes. JSON/Python syntax and declared request fields were rechecked statically. Current retrieval-budget defaults and constraints differ from the release; whole input-model parity is not claimed.
  • The SDK constructor and direct operation interface match the published wheel's unchanged Agent implementation. Provider recipes are checked against implementation source, with deployment and release limits stated in the guides.
  • Local checks made no live model calls or application requests. Account permissions, regional availability and useful answer quality were not exercised.
  • Fresh pushed-head technical and documentation reviews and full hosted CI are required.

AI assistance in the code (leave one; writing this description with AI is fine):

  • AI-generated: Codex wrote the documentation; operation contracts, SDK interfaces and provider behavior were checked against implementation source.

  • I can explain every line of this diff and how it interacts with the rest of Loom.

Checklist

  • Local format, lint, type checks and focused tests pass. The full make test-unit and applicable integration results are reported by CI; a full local run is not required before pushing.
  • Generated files are regenerated, not hand-edited.
  • No migration, plugin or frontend changes.
  • Documentation states prerequisites and no wire changes.
  • Examples use placeholders and public text contains no deployment-specific details.
  • Commits are signed off.

Connect canonical workflows with verified examples, access boundaries and a paired evaluation procedure so platform builders can find and check the answers.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
@vercel

vercel Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
loom-docs Ready Ready Preview Nov 22, 2026 4:26pm UTC

Request Review

Document the existing cache and access metadata contracts in the API sources, generated references and agent examples.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
@zlareb1 zlareb1 changed the title Document agent platform integration and repository evaluation Document agent integration, evaluation and provider setup Nov 20, 2026
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Validate the guide together with current contribution and translation sources.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Check caller and run-history guidance with the agent integration documentation.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Provider setup can replace cached clients without a restart. State the cache reset and embedding rebuild boundary in the architecture guide.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Describe local embeddings, worker defaults, model-based capture filtering and
standard versus restricted namespace access from their implementation. Unknown
provider usage and prices leave accounting gaps instead of proving free calls.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Set the provider alongside the local embedding model so existing overrides do
not change the recipe. Keep cost estimates, model comparisons, timeout and
repository access descriptions within their implemented and measured limits.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
The namespace spend check guards selected steps and does not stop inline or
queued transcript extraction. State that coverage instead of promising a stop.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Read flags and cached answers do not exclude embedding requests. State the cache, model-file and compiler-build prerequisites alongside their configuration.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Spend reads default to fail open, while conflict judgments and knowledge gap embeddings select a fail-closed guard. Distinguish direct request embeddings and present the guarded-work table as examples.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
zlareb1 added a commit that referenced this pull request Nov 22, 2026
Related: #1122
Related: #1074

### Changelog category

- CI Fix or Improvement

### Changelog entry

Not required.

### Why

The analysis-isolation test starts erasure when a new row becomes readable. With the Postgres authority, that read can precede publication acknowledgement. Erasure then waits for the outstanding publisher and can outlast the test client's deadline.

Wait for the exact setup revision to finish publication before requesting erasure. The full identity, memory ID and version must match committed rows, and the revision must have no delivery row. A retained settled delivery still blocks.

### Scope and risk

One integration fixture changes. The existing write, visibility check, request deadline and isolation assertions remain. The ClickHouse-authority path opens no Postgres connection. No production erasure, lease or publisher bounds change.

### Compatibility, rollout and rollback

- Wire, generated files and migrations: unchanged.
- Operator steps: none.
- Rollback: revert the fixture correction.

### Tests

- Existing inert controls: five tests passed; the original fixture produced seven failures.
- Native Postgres 16 and 18: 12 helper/SQL controls passed on each, including exact identity and version matching, retained deliveries, delayed acknowledgement, the unchanged 15-second polling deadline, read errors and connection cleanup. Each private database was removed. These controls qualify the helper and SQL, rather than the full erasure pipeline.
- Scoped Ruff format, lint, whitespace and PR-prose checks passed. This test-only change touches none of the server, SDK or extractor paths selected by the type gate.
- The existing isolation case exercises the complete request sequence in CI: `python -m pytest tests/integration/test_analysis_isolation.py -k an_erased_user_is_gone_from_the_next_statement -q` against the configured authenticated test server.
- Full integration and unit suites run in CI; no hosted result is claimed here.

### AI assistance in the code

- AI-generated: Codex wrote the fixture correction and checked revision matching and request order.

- [x] I can explain every line of this diff and how it interacts with the rest of Loom.

### Checklist

- [x] Local format, lint and focused checks pass; the type-gate paths are unchanged. Full suite results remain pending in CI.
- [x] No generated files, migrations, plugin, frontend or wire changes.
- [x] Commits are signed off.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Require observed provider calls for diagnostics and scope indexing claims to the workers and embedding options that implement them.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Keep the budget summary consistent with gated background embeddings and make the repository evaluation target the organization authorized by its token.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
The overview diagnostic excludes readers authorized by live grants. Link the canonical access rules so token ceilings and restricted namespace requirements remain explicit.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant