diff --git a/docs/architecture/llm-architecture.md b/docs/architecture/llm-architecture.md index 756350893e..9e3c9e0a2a 100644 --- a/docs/architecture/llm-architecture.md +++ b/docs/architecture/llm-architecture.md @@ -1,45 +1,54 @@ # LLM and embedding architecture Loom has two pluggable providers: an embedder (`server/embedding/`) that turns text into -vectors, and an LLM client (`server/llm/`) that completes prompts. Each is chosen once -per process from settings. Each falls back to a deterministic fake when its key is -missing, so a keyless install boots and answers. Operator setup is in +vectors, and an LLM client (`server/llm/`) that completes prompts. Their active +configuration comes from server settings. API-key providers can fall back to a +deterministic fake when their key is missing; native Bedrock uses deployment AWS +credentials and propagates credential failures. Operator setup is in [deployment](../operations/deployment.md); every setting is in the [configuration reference](../reference/configuration.md). ## Goals and non-goals -| Goals | Non-goals | -| --------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- | -| Every correctness gate on the write path works without an LLM. | A general LLM gateway. Loom calls models for its own features only. | -| A keyless install boots and says clearly that it runs on fakes. | Provider-agnostic prompts. Prompts are tuned to a model and re-benched on a change. | -| Swapping the embedder does not silently miscalibrate recall. | Mixing vector spaces. Vectors from different models are never compared. | -| Every provider call is attributed to the op or worker pass that made it, with its cost. | Runtime prompt management. Prompts are code. | +| Goals | Non-goals | +| ------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- | +| Authorization and direct memory-write admission work without an LLM. | A general LLM gateway. Loom calls models for its own features only. | +| A keyless install boots and says clearly that it runs on fakes. | Provider-agnostic prompts. Prompts are tuned to a model and re-benched on a change. | +| Swapping the embedder does not silently miscalibrate recall. | Mixing vector spaces. Vectors from different models are never compared. | +| Every provider call is attributed to its op or worker pass, with usage and cost when known. | Runtime prompt management. Prompts are code. | ## Embedders ### Selection -`server/embedding/embedder.py` (`provider_for`, `get_embedder`) picks the served embedder -once per process: +`server/embedding/embedder.py` (`provider_for`, `get_embedder`) selects and caches the +served embedder from the active settings. Applying or clearing dashboard provider +configuration resets that cache. Changing the embedding space requires the +[rebuild procedure](../getting-started/models-and-keys.md). + +Selection follows this order: 1. A label resolves first. `prefix::` and `ctx[]:` are views over the served embedder and cannot be the served model. -2. `LOOM_EMBEDDING_PROVIDER` (`fake`, `local`, `openai`, `gemini`) when set; any other - value raises. +2. `LOOM_EMBEDDING_PROVIDER` (`fake`, `local`, `openai`, `gemini`, `compatible`, + `bedrock`) when set; any other value raises. 3. Otherwise inferred from `LOOM_EMBEDDING_MODEL`: `fake-ngram*` is fake, `local` or - `local:*` is local, `text-embedding-*` is OpenAI, `gemini*` is Gemini. Any other name - raises `unknown embedding model`. - -| Provider | Module | Model names | Needs | Notes | -| -------- | ---------------------------- | -------------------------------------------------------------------------- | ---------------------------------------- | ------------------------------------------------------------------------- | -| `fake` | `server/embedding/fake.py` | `fake-ngram-` | nothing | Hashed character trigrams, L2-normalised. Deterministic; the dev default. | -| `local` | `server/embedding/local.py` | `local` (`bge-small`), `local:bge-small`, `local:bge-base`, `local:minilm` | the `local-embedder` extra (`fastembed`) | CPU ONNX; 384, 768 and 384 dimensions. No network. | -| `openai` | `server/embedding/openai.py` | `text-embedding-3-small`, `text-embedding-3-large` | `OPENAI_API_KEY` | Requests `dimensions=LOOM_EMBEDDING_DIM`. | -| `gemini` | `server/embedding/gemini.py` | `gemini-embedding-001` | `GEMINI_API_KEY` or `GOOGLE_API_KEY` | Requests `outputDimensionality=LOOM_EMBEDDING_DIM` (up to 3072). | - -**Width.** The served width is `LOOM_EMBEDDING_DIM` (default 3072). OpenAI and Gemini are -asked for that width; choose a model that supports it. Local models emit their native width. Boot probes the real + `local:*` is local, `text-embedding-*` is OpenAI, `gemini*` is Gemini and `bedrock:*` + is Bedrock. Any other name raises `unknown embedding model`; a compatible endpoint + needs an explicit provider. + +| Provider | Module | Model names | Needs | Notes | +| ------------ | -------------------------------- | -------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ | +| `fake` | `server/embedding/fake.py` | `fake-ngram-` | nothing | Hashed character trigrams, L2-normalised. Deterministic; the dev default. | +| `local` | `server/embedding/local.py` | `local` (`bge-small`), `local:bge-small`, `local:bge-base`, `local:minilm` | the `local-embedder` extra (`fastembed`) | CPU ONNX; 384, 768 and 384 dimensions. Inference is local; missing model files download on first use. | +| `openai` | `server/embedding/openai.py` | `text-embedding-3-small`, `text-embedding-3-large` | `OPENAI_API_KEY` | Requests `dimensions=LOOM_EMBEDDING_DIM`. | +| `gemini` | `server/embedding/gemini.py` | `gemini-embedding-001` | `GEMINI_API_KEY` or `GOOGLE_API_KEY` | Requests `outputDimensionality=LOOM_EMBEDDING_DIM` (up to 3072). | +| `compatible` | `server/embedding/compatible.py` | the endpoint's embedding model ID | `LOOM_EMBEDDING_BASE_URL`, `LOOM_EMBEDDING_API_KEY` or configured workload-token authentication | [OpenAI-compatible embeddings](../getting-started/models-and-keys.md#claude-for-the-llm); optional dimensions parameter. | +| `bedrock` | `server/embedding/bedrock.py` | Cohere Embed v4 model or inference-profile ID | deployment AWS credentials | [Native Bedrock](#native-bedrock); no API-key fallback. | + +**Width.** The served width is `LOOM_EMBEDDING_DIM` (default 3072, or 1536 for native +Bedrock when unset). OpenAI and Gemini are asked for that width; choose a model that +supports it. Local models emit their native width. Boot probes the real width (`probe_embedding_dim`); a mismatch shows as `embedding_dim_mismatch` on `/v1/health`. @@ -101,10 +110,11 @@ other column. In `memory_embeddings` and `conversation_embeddings` the model is sorting key, so old and new rows coexist until `--delete-old`. `entity_vectors` (`ReplacingMergeTree(created_at)`) is replaced in place. Flags: `--org`, `--namespace`, `--batch-size`, `--dry-run`, `--key-only`, `--resume`, and `--in-database `, which runs one `INSERT … SELECT` per table through ClickHouse's `aiEmbed` -(it needs a ClickHouse release that ships it, and is refused while a customer or identity -header is set or the gateway is required, since those calls name no one and skip the -gateway's check). See the [CLI reference](../reference/cli.md). +collection>`, which computes embeddings through ClickHouse's `aiEmbed`. Memory vectors +are published through embedding admission in bounded pages; other tables use one +`INSERT … SELECT` each. This mode needs a ClickHouse release that ships `aiEmbed` +and is refused while a customer or identity header is set or the gateway is required, +since those calls name no one and skip the gateway's check. See the [CLI reference](../reference/cli.md). `embedding_worker` covers the rest. It embeds `embedding_dim = 0` stubs newest first; a stub labelled with an older model is embedded under the served one and the old stub @@ -294,8 +304,9 @@ models keep the caller's requested temperature. An opaque gateway alias or appli inference-profile ARN does not identify its backing model and receives no inferred restriction. -Each client builds the rest of its request (the lowest reasoning effort and an -output cap), and provider responses teach other traits once per model per process: +The OpenAI-compatible and Anthropic clients build the rest of their requests (the +lowest reasoning effort and an output cap). Provider responses teach other traits +once per model per process: - a 400 naming a field the model does not take (`temperature`, `reasoning_effort`, an `output_config`, the cap field under its other name) drops that field: the call is @@ -308,6 +319,10 @@ output cap), and provider responses teach other traits once per model per proces `LOOM_LLM_HIDDEN_REASONING_RESERVE_MAX`), added to every caller's cap. A reply that hit the cap with no visible text is re-sent once with the ceiling as its reserve. +The native Bedrock client sends the caller's cap as `maxTokens` and learns only a +rejected `temperature` field. It does not add the reasoning reserves or negotiate +effort and cap limits described above. + An unrecognized model learns traits from its responses. Documented sampling restrictions and provider-default sampling avoid an initial temperature request; learned traits supplement them. `LOOM_LLM_ANTHROPIC_EFFORT` sends a thinking effort to Claude models. @@ -317,15 +332,18 @@ them. `LOOM_LLM_ANTHROPIC_EFFORT` sends a thinking effort to Claude models. - **Retry** (`server/llm/_retry.py`): transport errors and 429, 500, 502, 503, 504, 529; `LOOM_LLM_RETRY_ATTEMPTS` (4) attempts, exponential backoff (0.5 s base, `LOOM_LLM_RETRY_MAX_WAIT_S` cap, 20 s). The request timeout, - `LOOM_LLM_REQUEST_TIMEOUT_S`, is 300 s: it only catches a hung connection, so a slow - model or a long output never hits it. Callers that need less wrap the call (HyDE on + `LOOM_LLM_REQUEST_TIMEOUT_S`, defaults to 300 s. OpenAI and Anthropic clients apply + HTTP request timeouts; Bedrock also bounds request and retry work with an elapsed + deadline. Slow inference or long outputs can hit these limits. Callers that need less wrap the call (HyDE on `memory.search` waits `LOOM_MEMORY_SEARCH_HYDE_TIMEOUT_S`, 6 s). - **Fallback model.** `LOOM_LLM_FALLBACK_MODEL` wraps the client in `FallbackLLMClient` (`server/llm/fallback.py`). It hands the call to the second model on an outage (transport error, retryable status, timeout) or a rejected request (400, 413, 422); 401, 403 and 404 propagate. Each hand-over is logged and counted (`loom.llm.fallback`), and the result names the model that answered. Fallbacks do not chain. -- **Missing key.** The client is the fake (`_build_llm_client`), logged at WARNING. +- **Missing API key.** An OpenAI or Anthropic client without its key uses the fake + (`_build_llm_client`), logged at WARNING, unless configured workload-token + authentication supplies credentials. Native Bedrock credential failures propagate instead. - **One identity refused.** With identity headers set, an answer from a gateway host that `LOOM_INFERENCE_IDENTITY_REFUSALS_JSON` lists is the gateway refusing the call's identity or asking it to wait ([identity headers](#identity-headers)): never retried, never handed to the @@ -535,16 +553,18 @@ conversation titler). ## LLM-driven and deterministic parts -Every gate that decides what is stored or who may read it is deterministic: -authorization, the blacklist, sensitivity detection, conflict and duplicate verdicts, -ranking and the abstain floor. LLM stages produce or enrich content (facts, answers, -summaries, titles). Each is optional, bounded, and fails open to the deterministic path. +Authorization and direct `memory.set` admission do not call an LLM. The blacklist, +sensitivity checks, conflict and duplicate verdicts, ranking and abstain thresholds +use deterministic rules over the supplied content and retrieval scores. -**Why.** An LLM-decided gate cannot be reproduced, audited or unit-tested, and it adds -cost and latency to every write. The deterministic gates run on the fake provider in CI -through the production code path. The cost: a cosine threshold cannot tell a revision -from a contradiction, which is why the conflict resolver exists. It judges after the -write, on the leader, so the write's own verdict stays deterministic. +Model jobs can change the content that reaches storage. Extraction produces facts; +the optional durability verifier filters coding-capture facts using model votes before +persistence. If the primary verifier fails, the batch is kept. If a secondary verifier +fails, it blocks non-durable rejection, but a successful primary can still drop a +fact in favor of a retained representative. The conflict resolver judges after a +write, on the leader, and can revise stored content. These model judgments do not +replace authorization or the direct write's admission checks. Answers, summaries and +titles also use models; each feature's failure behavior is listed below. | Feature | LLM? | Switch (default) | | --------------------------------------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | @@ -572,11 +592,13 @@ descriptions (`server/transport/mcp_descriptions.py`) describe intent, not a syn ### LLM-free reads -`LOOM_READS_LLM_FREE=true` makes every read op (`Operation.read_only`) run with no LLM -call, so a deployment can promise that reads never reach a model. The HTTP and MCP transports mark the -request while the handler runs (`server/core/llm_free.py`); the memory search reads the mark through the -same gate as the fast posture and `context.pack` (`_is_llm_free`), and every other read step that could -call a model reads `reads_llm_free()`: +`LOOM_READS_LLM_FREE=true` suppresses the generative read steps listed below. +Query embeddings and the explicitly selected rerankers described below can still +call providers, send query or candidate content and incur charges. The HTTP and MCP +transports mark each read request (`Operation.read_only`) while its handler runs +(`server/core/llm_free.py`). Memory search reads this mark through `_is_llm_free`, +alongside the fast posture and `context.pack`; the other listed guards use +`reads_llm_free()`: - `memory.search`, `search.query` (the MCP `loom_search` tool), `context.pack` and every read built on them: no HyDE, graph triples, retry judge, LLM count intent or verb synonyms. The @@ -590,10 +612,11 @@ call a model reads `reads_llm_free()`: - `tracker.duplicate_check` and `tracker.answer`: no rerank or uncertainty judge; `code.bisect_candidates`: no rerank. Each keeps the deterministic answer it already falls back to. - `memory.ask` and `code.ask` answer 409 `READS_LLM_FREE` naming the setting unless the request - sets `allow_synthesis=false` / `synthesize=false`, even when a deterministic route could answer it; - with that flag they return the recalled evidence with no LLM call. - A model's answer is the purpose of those calls, so the setting refuses it rather than answering - without one. `memory.brief` asks for the evidence alone and returns it with its unknowns, no answer. + sets `allow_synthesis=false` / `synthesize=false`, even when a deterministic route could answer it. + These flags disable fresh synthesis. A `code.ask` cache hit can still return a previously + synthesized answer. For an uncached retrieval-only `code.ask` response, set both + `synthesize=false` and `no_cache=true`. `memory.brief` asks for the evidence alone and returns + it with its unknowns, no answer. - The sampled retrieval-quality judge queues nothing: both transports record a search's quality inside the same mark. @@ -637,11 +660,16 @@ parses its output. Editing a prompt needs a deploy. ## Cost accounting -Every provider call reports usage to a per-op accumulator +Provider clients report available usage to a per-op accumulator (`server/observability/cost_accumulator.py`). Both transports bind it per request; the worker supervisor binds it per pass. Prices come from `PRICE_PER_1K_TOKENS` in the same -module, with cache-read and cache-write multipliers. An unlisted model bills $0 and logs -one warning. Gemini embeddings report no usage and bill $0. +module, with cache-read and cache-write multipliers. An unlisted model contributes a +zero cost estimate and logs one warning. Missing embedding usage also leaves the +numeric ledger incomplete. These zeros do not establish zero provider spend. + +Generation spans omit unpriced cost and set `loom.llm.cost_unknown`. Embedding spans +set `loom.embed.cost_unknown` or `loom.embed.usage_unknown` when price or usage is +unavailable. Treat those gaps as unknown when comparing total deployment cost. The op's `loom.agent_events` row carries `upstream_cost_usd`, `embedding_tokens`, `input_tokens`, `output_tokens`, `llm_calls` and the dominant model (`emit_agent_event` @@ -653,7 +681,7 @@ in `server/observability/ledger.py`). Cache savings appear per answer The dev defaults are the fakes. Production sets `LOOM_EMBEDDING_MODEL` and `LOOM_LLM_MODEL` explicitly; the commented provider block in `.env.example` uses -`text-embedding-3-small` at 1536 dimensions and `gpt-6-luna`. +`text-embedding-3-large` at 3072 dimensions and `gpt-6-luna`. **Run `make bench-quick` before changing either default** or opting into the Anthropic provider. It runs a 30-question sample of the long-memory benchmark (`bench/longmemeval.py`) @@ -662,16 +690,18 @@ real providers. See [running benchmarks](../benchmarks/running.md). ## Invariants -- A process serves one embedder and one default LLM client, chosen at startup. +- A process serves one embedder and one default LLM client at a time. Dashboard + configuration can replace the cached clients; changing the embedding space requires a rebuild. - Vectors of different `embedding_model` values are never compared. - The served embedder falls back to the fake only when its key is missing, and says so on `/v1/health`. A secondary embedder never falls back. - A search always has a threshold: namespace calibration, else boot calibration, else the scalar settings. -- No gate that decides storage or access calls an LLM. +- Authorization and direct memory-write admission do not call an LLM; capture + verification and later conflict resolution can filter or revise content. - A fallback model's answer is never cached. -- Every provider call's usage lands on the `agent_events` row of the op or pass that made - it. +- Provider usage reported to the accumulator lands on the `agent_events` row of the + op or pass that made it; unavailable usage and prices leave accounting gaps. ## Source entry points diff --git a/docs/features/code-index/overview.md b/docs/features/code-index/overview.md index fda7829084..1ecb893baf 100644 --- a/docs/features/code-index/overview.md +++ b/docs/features/code-index/overview.md @@ -11,7 +11,8 @@ the index is built is in The [language support table](#language-support) distinguishes fast-tier symbols and edges from compiler-tier coverage. -The code workers are off by default; a deployment turns them on per repository +Worker defaults vary by tier; check the [workers reference](../../reference/workers.md). +A worker also needs an admitted supervisor and a repository configured for its tier ([repositories](repositories.md)). The default `core` MCP toolset lists no code tools unless `LOOM_MCP_CODE_TOOLS=true`; `?toolset=coding` always does. @@ -87,8 +88,11 @@ Structure is extracted twice. Each symbol and edge names its tier (`tier`: `scip [code index operations](../../operations/code-index-operations.md#compiler-tier)). An external producer can upload with `code.scip_ingest`. It gives exact call, override and inheritance edges. Its C/C++, Go and TypeScript steps execute the repository's own - build code; they run inside the - [repository-job sandbox](../../operations/code-index-operations.md#repository-job-sandbox). + build code. They use the + [repository-job sandbox](../../operations/code-index-operations.md#repository-job-sandbox) + when available. With `LOOM_CODE_SANDBOX_REQUIRED=true` they refuse to run without + it; otherwise, operator-configured repositories can run those builds as plain + processes. Why two: the fast tier is cheap, so a merge is searchable after one ingest pass. The compiler tier needs a build environment and trails. Serving both, labelled, gives an @@ -317,18 +321,18 @@ generated from the same schemas. Start with `code.index_status` (watermarks, lags, tiers, worker errors) and `code.coverage` (what the namespace holds). -| Symptom | Cause | Fix | -| ------------------------------------------ | --------------------------------------------------------- | --------------------------------------------------------------------- | -| a symbol you just added is not found | uncommitted, unpushed, or not yet ingested | grep your tree; `code.index_status` shows the lag | -| `lag_commits: 0` but merges are missing | the mirror stopped fetching | `clone_fetched_at` / `fetch_overdue`, then `/v1/admin/workers.status` | -| a fix you know exists is not found | wrong branch namespace | `code.grep mode=presence namespaces=[…]` | -| `AUTH_INSUFFICIENT` on every `code.*` call | the token does not name the code namespace | join the repository, or mint with `--namespaces` | -| a call that names no namespace is refused | no `X-Loom-Code-Scope` header and no deployment default | pass `namespace`, or set `LOOM_CODE_ORG` / `LOOM_CODE_NAMESPACE` | -| no code tools in the MCP list | the `core` toolset without `LOOM_MCP_CODE_TOOLS=true` | mount `?toolset=coding`, or set the flag | -| `code.search` says lexical-only | embedding coverage below `LOOM_CODE_READY_EMBED_COVERAGE` | enable `code_embed_recovery`; watch `embedding_coverage` | -| callers look incomplete for a language | the language has only the fast tier | read `epistemic` / `causes`; `code.coverage` shows the `scip` column | -| every namespace re-indexing after a deploy | an extractor module changed (the stamp moved) | expected; progress is in `code_index_state` | -| a self-service repository stuck in `error` | clone or first index failed | read `stage_detail` on `code.repos` | +| Symptom | Cause | Fix | +| ------------------------------------------ | --------------------------------------------------------- | ------------------------------------------------------------------------------------------ | +| a symbol you just added is not found | uncommitted, unpushed, or not yet ingested | grep your tree; `code.index_status` shows the lag | +| `lag_commits: 0` but merges are missing | the mirror stopped fetching | `clone_fetched_at` / `fetch_overdue`, then `/v1/admin/workers.status` | +| a fix you know exists is not found | wrong branch namespace | `code.grep mode=presence namespaces=[…]` | +| `AUTH_INSUFFICIENT` on every `code.*` call | the caller lacks sufficient access for the operation | check token scope and live grants against [the access rules](repositories.md#who-can-read) | +| a call that names no namespace is refused | no `X-Loom-Code-Scope` header and no deployment default | pass `namespace`, or set `LOOM_CODE_ORG` / `LOOM_CODE_NAMESPACE` | +| no code tools in the MCP list | the `core` toolset without `LOOM_MCP_CODE_TOOLS=true` | mount `?toolset=coding`, or set the flag | +| `code.search` says lexical-only | embedding coverage below `LOOM_CODE_READY_EMBED_COVERAGE` | enable `code_embed_recovery`; watch `embedding_coverage` | +| callers look incomplete for a language | the language has only the fast tier | read `epistemic` / `causes`; `code.coverage` shows the `scip` column | +| every namespace re-indexing after a deploy | an extractor module changed (the stamp moved) | expected; progress is in `code_index_state` | +| a self-service repository stuck in `error` | clone or first index failed | read `stage_detail` on `code.repos` | Operator signals (tier lag, stale files, failed passes) are in [code index operations](../../operations/code-index-operations.md). diff --git a/docs/features/code-index/repositories.md b/docs/features/code-index/repositories.md index f6fba29631..f59d3fb9f5 100644 --- a/docs/features/code-index/repositories.md +++ b/docs/features/code-index/repositories.md @@ -136,9 +136,11 @@ Rules: - A private repository's token is stored encrypted with a key derived from `LOOM_JWT_SECRET` and never returned. Rotating that secret makes stored tokens unusable until a maintainer enters them again. -- `code.repo.remove` retires the registry rows and purges every `code_*` row of the - namespace through a resumable erasure saga. The mirror is deleted when no other branch - uses it. An operator-configured entry is refused with `REPO_FROM_CONFIG`. +- `code.repo.remove` makes the registry rows inactive and starts a resumable namespace + erasure saga. It purges indexed code and search telemetry; cached answers follow the + [retention rules](../../operations/code-index-operations.md#rollout-and-shutdown). + The mirror is deleted when no other branch uses it. An operator-configured entry is + refused with `REPO_FROM_CONFIG`. - `LOOM_CODE_SELF_SERVICE_ENABLED=false` removes the feature (`SELF_SERVICE_OFF`). - `LOOM_CODE_SELF_SERVICE_PLATFORM_ADMIN_ONLY=true` leaves `code.repo.remote_branches`, `code.repo.add` and `code.repo.add_branch` to platform admins, beyond the role each op @@ -177,12 +179,12 @@ or SQL. `compiler_candidates` is inferred from build manifests. A candidate without a watermark appears in `quality_warnings` with the workers to enable: -| Candidate | Preflight in the worker environment | Worker | -| --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------- | -| C/C++ | `scip-clang --version` (in the amd64 image); the repository's CMake configures with clang 21 | `code_scip_full`, then `code_scip_delta` | -| TypeScript/JavaScript | Indexer and package managers included in Docker; native servers run `sh scripts/install-code-toolchains.sh`. Repository dependencies are prepared automatically. | `code_scip_ts_delta` | -| Go | `scip-go --version`; `go env` succeeds for every module | `code_scip_go_delta` | -| Python | `scip-python --version`; project imports resolve in its environment | `code_scip_py_delta`, then `code_scip_apply` | +| Candidate | Preflight in the worker environment | Worker | +| --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------- | +| C/C++ | `scip-clang --version` (in the amd64 image); the repository's CMake configures with clang 21 | `code_scip_full`, then `code_scip_delta` | +| TypeScript/JavaScript | Indexer and package managers included in Docker; native servers run `sh scripts/install-code-toolchains.sh`. Repository dependencies are prepared automatically. | `code_scip_ts_delta` | +| Go | `scip-go --version`; `go env` succeeds for every module | `code_scip_go_delta` | +| Python | `scip-python --version`; project imports resolve in its environment | `code_scip_py_delta` | The fast tier remains usable when compiler preflight fails. `code.index_status` and the Health tab expose worker errors, per-language coverage, unresolved edges, embedding @@ -204,18 +206,80 @@ until `loom code backfill` runs. Every setting is in the Code namespaces follow the usual grant rules ([identities and namespaces](../../concepts/identities-and-namespaces.md), -[tokens and grants](../../guides/tokens-and-grants.md)): a token must name the code -namespace (or `*`), or the call fails with `AUTH_INSUFFICIENT`. Three switches change -that: +[tokens and grants](../../guides/tokens-and-grants.md)). A standard code namespace +accepts a sufficient token claim or a live reader grant for the caller's org; +existing grant revocations and role caps still apply. Three switches change access: | Switch | Effect | | ----------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | -| `LOOM_CODE_ORG_READABLE=true`, or the entry's `org_readable` | every authenticated principal of the org reads without a grant (default off); writes stay grant-driven | +| `LOOM_CODE_ORG_READABLE=true`, or the entry's `org_readable` | authenticated org members read standard namespaces without a grant (default off); write access is unchanged | | `LOOM_CODE_REPO_PUBLIC=true`, or the entry's `public` | the namespace registers `join_policy=open` and discoverable, so any org member joins as a reader at once; otherwise the policy is `approval` | | `classification: "restricted"` (entry, or `code.repo.set_access`) | never org-readable or public; a maintainer approves each reader, and every read needs a token naming the namespace plus a live direct grant | A `code.repo.set_access` row wins over the configured classification and join policy -and survives restarts. `code.refresh` (writer) and the admin ops always need a grant. +and survives restarts. `code.refresh` requires `writer`; admin operations require +`admin`. For standard namespaces, token claims or grants can supply those roles, +subject to scoped-token ceilings and live grant caps. Restricted namespaces require +both the token's own role and a live direct grant, with literal namespace naming. + +### Separate repositories for two teams + +Keep repositories with different readers in different code namespaces. For example, +index a billing repository as `code-billing` and an operations repository as +`code-operations`, both `classification: "restricted"` with `join_policy: "approval"`. +Use the [operator-configured entry format](../../operations/code-index-operations.md) +or the [self-service access controls](#self-service-repositories) for your deployment. +Bootstrap maintainer grants before restricting an existing namespace, following the +[restricted-namespace procedure](../../guides/tokens-and-grants.md#restricted-namespaces-classificationrestricted). + +Give a billing service a live viewer grant and a token explicitly naming only +`code-billing`. Its expected read behavior is: + +| Request | Expected scope outcome | +| ---------------------------------- | ------------------------------------------------------------------------------ | +| `code.search` in `code-billing` | Authorized; useful hits still depend on the index and query | +| `code.search` in `code-operations` | Denied; the billing grant and token do not cover it | +| `code.repos` | Can include request-only metadata for the other repository; inspect `readable` | + +Check the allowed request with a symbol you know is indexed, then repeat an equivalent +request against the other namespace. Over HTTP, a denied scoped code read returns +`AUTH_INSUFFICIENT`; a known object ID can instead read as missing to avoid revealing its +existence ([authorization diagnostics](../../operations/troubleshooting.md#an-id-you-know-exists-reads-as-missing)). +Test revocation separately from an empty search result. Repeat for each indexed branch: +namespace naming and maintainer inheritance do not give every reader access to every +branch. + +A same-organization repository with approval-based joining can appear in `code.repos` +as `readable: false`. That row exposes metadata for requesting access, with empty index +counts and watermarks. It does not authorize reading code. Do not treat a listing entry +as a successful access check. + +For example, with the billing service's token, replace the query with a known billing +symbol. Repeat with `namespace` set to `code-operations` and expect an authorization +failure, not an empty successful result: + +```sh +curl --fail-with-body -X POST "$LOOM_BASE_URL/v1/code.search" \ + -H "Authorization: Bearer $LOOM_API_KEY" \ + -H 'Content-Type: application/json' \ + --data '{"org":"example","namespace":"code-billing","query":"invoice authorization","top_k":3}' +``` + +The Git token or SSH identity that fetches source is separate from the Loom token and +grants used by readers. A chat or document connector login does not grant a native Loom +service repository access. Agent names and repository tags record attribution or +association; they are not access boundaries. + +### Check indexing isolation separately from read access + +Repository read grants do not sandbox indexing jobs. Self-service indexing requires +the sandbox unless an operator explicitly enables its restricted unsandboxed mode; +operator-configured repositories can run without it. To require isolation for every +repository job, set `LOOM_CODE_SANDBOX_REQUIRED=true` and satisfy the host requirements. +When the sandbox is unavailable, jobs are refused or skipped rather than silently run +outside it. The [sandbox runbook](../../operations/code-index-operations.md#repository-job-sandbox) +describes namespaces, mounts, egress, process limits, supported stacks and remaining +container/host responsibilities; Loom's per-process limits do not impose a job cgroup. ## Failure modes @@ -226,7 +290,7 @@ and survives restarts. `code.refresh` (writer) and the admin ops always need a g | Adding a repository answers `AUTH_INSUFFICIENT` | the message names `LOOM_CODE_SELF_SERVICE_PLATFORM_ADMIN_ONLY`: only platform admins add repositories and branches on this deployment, with a token of their own that is not scoped to namespaces | | A self-service card stays at `error` | read `stage_detail` in `code.repos`; a failed clone retries next pass (timeout `LOOM_CODE_BOOTSTRAP_CLONE_TIMEOUT_S`) | | A private self-service repository stops fetching | `LOOM_JWT_SECRET` was rotated; re-enter the token | -| An entry is missing from `code.repos` | check `loom.code.repos_config_errors`; `code.repos` also lists only what the caller may read | +| An entry is missing from `code.repos` | check `loom.code.repos_config_errors` and the caller's listing eligibility; request-only entries have `readable: false` and do not grant code access | | Revoking or lowering an admin answers busy | a revoke on a code namespace first re-reads `loom.code_repos`, so it sees the branches added here from another process, and answers busy while that read fails; retry once the read succeeds | | A restricted branch added here has no admin | its admins were all copies from branches added here, which the sync revokes; grant admin on one of the repository's operator entries (with none, the operator lists a branch of it in `LOOM_CODE_REPOS_JSON` first) | | A new maintainer lacks the other branches | the admin grant was made before this process read `loom.code_repos` (that process logs `not synced` the first time); the next admin grant or approval on a branch of the repository, or a branch added, copies it; or grant it again | diff --git a/docs/features/observability/cost-and-usage.md b/docs/features/observability/cost-and-usage.md index 134b940294..6f1dbdc452 100644 --- a/docs/features/observability/cost-and-usage.md +++ b/docs/features/observability/cost-and-usage.md @@ -1,10 +1,11 @@ # Cost and usage -Loom records every upstream dollar it spends on embedding and LLM calls against the -operation that spent it, in the per-operation ledger `loom.agent_events`. The `cost.*` ops +Loom records reported token usage and estimated provider costs for embedding and LLM +calls against the operation that made them, in the per-operation ledger `loom.agent_events`. +Missing usage or prices leave gaps; a zero estimate does not prove zero provider spend. The `cost.*` ops read that ledger through a one-minute roll-up. A separate series, `loom.agent_turn_usage`, holds what a coding agent spent on its own model turns. The two never overlap: one is -Loom's provider bill, the other is the agent's. +Loom's provider usage estimates, the other is the agent's. This page also covers memory **usage** (recalls, evidence, reliance), which the `usage_fold` worker folds back into ranking. @@ -29,7 +30,11 @@ flowchart LR 1. **Accumulator.** The transport binds a cost accumulator (`server/observability/cost_accumulator.py`) per operation; embedders and LLM clients report tokens and dollars into it. Prices come from `PRICE_PER_1K_TOKENS` in the same - module. **A model missing from the table bills $0.** + module. A model missing from the table contributes a zero cost estimate, and missing + embedding usage also leaves the numeric ledger incomplete. Generation spans flag + unpriced cost with `loom.llm.cost_unknown`; embedding spans use + `loom.embed.cost_unknown` or `loom.embed.usage_unknown` and omit unavailable cost or + usage. Include those gaps and provider billing records when measuring total cost. 2. **Ledger row.** When the op finishes, the accumulator drains into its `agent_events` row: `input_tokens`, `output_tokens`, `embedding_tokens`, `upstream_cost_usd`, `model`, `llm_calls`, plus identity, latency and outcome ([the ledger](./telemetry.md#the-ledger)). @@ -58,8 +63,9 @@ flowchart LR Each row carries `ops`, `errors`, `tokens` (LLM input plus output), `embed_tokens`, `cost_usd` and `latency_p50_ms` / `p95` / `p99`. The output adds `total_ops`, `total_errors`, `total_cost_usd`, and `cache_hits` / `cache_saved_usd`: LLM calls served -from the response cache and the price-table dollars they did not spend. `total_cost_usd` -is what _was_ spent; the two never overlap. +from the response cache and their estimated price-table savings. `total_cost_usd` +sums recorded cost estimates, excluding those cache savings. Missing usage or pricing +must be accounted for separately when comparing provider bills. Grouping or filtering by a dimension the roll-up lacks (`memory_key`, `user_id`, `model`, `traffic_class`, `instance_label`, `build_commit`) switches to a scan of `agent_events`: @@ -111,16 +117,18 @@ older builds cannot enforce user caps or protect stored private warnings. Save t policies and reapply them after upgrading the whole fleet again. The guard (`namespace_over_cap` / `namespaces_over_cap` in -`server/observability/budget.py`) gates LLM work. Over the cap, each of these skips its -paid step: +`server/observability/budget.py`) gates selected model work. Over the cap, guarded +callers skip their paid step. Examples include: | Where | Gated | | ------------ | -------------------------------------------------------------------------------------------------------------------------- | | workers | entity extractor, code summarizer, code ingest | | request path | `memory.ask` and `code.ask` synthesis (retrieval still answers), `knowledge.synthesize`, the `expand` leg of `code.search` | -Embeddings (a `memory.set`, a search) are not gated. The guard is **fail-open**: if the -spend read errors, the namespace reads as under its cap. +Embeddings in direct `memory.set` and search requests do not check these caps. +The guard defaults to **fail-open**: a failed spend read counts as under the cap. +Conflict judgments and knowledge gap embeddings instead fail closed: a failed +spend read skips that model work. `cost.budget_status` reports the same computation: @@ -128,7 +136,7 @@ spend read errors, the namespace reads as under its cap. | ------------------------ | -------------------------------------------------------------------------------------------------------------------------- | | `spent_24h_usd` | Rolling 24h spend in USD; user mode withholds it as `null` except for a platform admin without namespace-role restrictions | | `cap_usd`, `cap_source` | the effective cap and where it came from: `namespace`, `global`, `uncapped`, or `org` for the org-wide form | -| `over_cap`, `pct_of_cap` | Whether real spend reaches the selected cap; share spent (`null` when uncapped or user spend is withheld) | +| `over_cap`, `pct_of_cap` | Whether recorded estimated cost reaches the selected cap; share used (`null` when uncapped or user spend is withheld) | | `namespaces_with_spend` | org-wide form only | Omit `namespace` for the org-wide form, which is always uncapped and sums the namespaces @@ -136,7 +144,8 @@ you can read. Supply both `namespace` and `user_id` for one user's allowance and `cap_source` then reads `namespace_user`, `global_user`, or `uncapped`. `namespace_over_cap` separately reports whether the namespace ceiling blocks that user. A token pinned to one user cannot select another user's status. Caps and routing count -all actual spend even when the response withholds its amount. +recorded estimated cost even when the response withholds its amount. Unreported usage +and unpriced calls can make provider billing higher than the amount used by the cap. Set `budget_alert_inbox` with `namespace.register` or the namespace settings dialog to an existing task inbox owned by that namespace. Empty disables warnings. A point budget @@ -156,8 +165,8 @@ Updates and watch streams show only warnings inside the caller's user and sessio scope. A scope without the required identity receives no warnings. Other task events retain their shared inbox visibility. -The $1 default stops a runaway extraction loop on a laptop. A production namespace that -ingests heavily needs its own cap, or extraction stops partway through the day. +The daily cap applies to the guarded steps listed above. Inline and queued transcript +extraction do not check this cap and can continue after it is reached. ## The coding agent's own spend @@ -248,7 +257,7 @@ See the [configuration reference](../../reference/configuration.md#runtime) for crash can lose the unflushed queue (`LOOM_EVENTS_STREAM_FLUSH_INTERVAL_S` bounds the window when the events stream is on). - **A cap is not a hard limit.** The guard reads spend before a batch, so one batch can - overshoot, and embeddings are not gated. + overshoot. Direct memory-write and search embeddings are not gated. - **Failed requests can have spent.** `loom.ops.paid_errors` counts requests that failed after spending upstream; their cost is still on the ledger row. diff --git a/docs/getting-started/evaluate.md b/docs/getting-started/evaluate.md index b8047d2da5..ab0aa12563 100644 --- a/docs/getting-started/evaluate.md +++ b/docs/getting-started/evaluate.md @@ -140,13 +140,14 @@ the repository Loom was tuned on. Label: - out-of-repository questions that should abstain; - the tests or dependants expected from an impact query. -Run the code-agent acceptance report against an indexed repository: +Run the code-agent acceptance report against an indexed repository. Replace the +organization and namespace with values the token can read: ```bash export PYTHONPATH=$PWD .venv/bin/python bench/acceptance/code_agent.py \ --base-url http://localhost:7777 --token-file /path/to/token \ - --namespace + --org --namespace ``` The built-in fixture targets ClickHouse and is useful for checking a deployment. For a @@ -155,6 +156,34 @@ agent tasks with ordinary repository tools only. Report task success first, then tokens, cost and wall time. Tool-schema overhead depends on the client, so record whether it defers MCP tool definitions. +#### Label your own repository before querying + +The built-in acceptance report is not a repository-neutral answer-quality test. Make a +task sheet for your own repository with these columns before collecting results: + +| Column | What to record | +| ---------------------- | ---------------------------------------------------------------------------------------------- | +| Task and frozen source | Question, repository commit, branch and task category | +| Expected evidence | Definition path and qualified symbol, accepted source ranges, or a reviewed caller/test set | +| Expected answer | Claims the answer must support and claims it must not make | +| Expected refusal | Unanswerable, wrong-repository and denied-access cases | +| Retrieval outcome | Returned identifiers, ranks, citations, freshness and completeness limits | +| Agent outcome | Actual calls and evidence used, answer or patch, objective test result, latency and total cost | + +This is a proposed evaluation record, not an input schema accepted by the supplied +harness. Call `code.symbol`, `code.callers`, `code.impact`, `code.search` or `code.ask` +for the corresponding labelled cases using the +[operation reference](../reference/operations/code.md). Read each result's uncertainty: +an incomplete graph should not be graded as an exhaustive caller set, and changed or +missing watermarks leave a freshness comparison unverified. Keep stale, failed and +permission-denied requests visible alongside successful retrieval. + +For task-level comparisons and a CTO decision scorecard, use the +[paired agent-platform evaluation](../guides/agent-platforms.md#give-a-cto-an-evaluation-they-can-check). +It includes application, hook and model-selected calls, zero-use tasks, citation checks, +objective tests, construction cost and uncertainty. That procedure supplies no measured +result until you run and report it on the chosen workload. + ### Keep the comparison fair - Use the same model, prompt, data snapshot and question order in paired arms. diff --git a/docs/getting-started/models-and-keys.md b/docs/getting-started/models-and-keys.md index 611aba7cf1..41dd301dea 100644 --- a/docs/getting-started/models-and-keys.md +++ b/docs/getting-started/models-and-keys.md @@ -21,15 +21,15 @@ The released Docker installation embeds with `local` (bge-small, 384 dimensions, from files the image carries) and selects the `fake-echo-1` LLM, unless the installer set a local model server ([a model server on your machine](#a-model-server-on-your-machine-no-api-key)). It boots without provider keys or -network: storage, the dashboard, structural code indexing and semantic search work, -though a hosted embedder retrieves better on paraphrase-heavy questions. The fake LLM +network: storage, the dashboard, structural code indexing and semantic search work. The fake LLM does not produce meaningful extracted facts, summaries or generated answers. A checkout's `.env` selects `fake-ngram-3072` embeddings, which match on character overlap. An upgraded installation keeps the embedder its `~/.loom/.env` names; switching it to `local` is a model change like any other, and re-embeds what is stored. -For stronger retrieval over paraphrase-heavy content, configure a hosted embedding -provider. For automatic fact extraction, summaries and generated answers, configure an +To choose another embedding model, configure a supported local or hosted provider and +[evaluate retrieval on your content](evaluate.md). For automatic fact extraction, +summaries and generated answers, configure an LLM too. An LLM alone does not replace an embedding model; an embedding model alone does not generate answers. With no embedding configured, a checkout's server uses its fake embedder; it does not skip embedding. @@ -43,26 +43,33 @@ keyless providers a degraded-health condition. ## Choose a setup -| What your organization has | Loom LLM | Embeddings | -| ---------------------------------------------- | ------------------------- | ------------------------------------------------------ | -| OpenAI API access | An available OpenAI model | OpenAI; one API key can serve both roles. | -| Anthropic plus an approved embedding provider | An available Claude model | OpenAI, Gemini or a compatible endpoint, with its key. | -| Anthropic API access only | An available Claude model | Add an embedding provider and its API key. | -| No provider key; a model server on the machine | A local model (`local`) | The `local` embedder the image carries. | +| What your organization has | Loom LLM | Embeddings | +| ---------------------------------------------- | ---------------------------------------------- | ---------------------------------------------------------------- | +| OpenAI API access | An available OpenAI model | OpenAI; one API key can serve both roles. | +| Anthropic plus an approved embedding provider | An available Claude model | Configure that embedder and its supported authentication. | +| Anthropic API access only | An available Claude model | The bundled `local` embedder needs no additional API key. | +| AWS role with Bedrock model access | A permitted Bedrock model or inference profile | Cohere Embed v4 through Bedrock, or another configured embedder. | +| No provider key; a model server on the machine | A local model (`local`) | The `local` embedder the image carries. | **One OpenAI API key is enough for both roles; an Anthropic key is not required.** -In Models & keys, choose one main LLM and leave per-job overrides blank. All enabled -LLM jobs use that model, including verification and judging. Advanced overrides are -optional and require another provider's key only when you explicitly select one. +In Models & keys, choose one main LLM and leave the displayed per-job overrides blank +to use it for those jobs, including capture verification and retrieval judging. +Background settings such as the conflict resolver are configured separately; use the +[per-job recipe](#keep-every-llm-job-on-your-chosen-provider) to route them explicitly. -Choose OpenAI, Google Gemini or an **OpenAI-compatible embedding endpoint**. OpenAI embeddings use +Choose the bundled `local` embedder, OpenAI, Google Gemini, [Bedrock](#amazon-bedrock) or an +**OpenAI-compatible embedding endpoint**. OpenAI embeddings use an embedding model such as `text-embedding-3-large`, not a GPT chat model. -Claude can handle Loom's LLM jobs, but requires a separate embedding provider and key. +Claude can handle Loom's LLM jobs with a separately configured embedder. The bundled +`local` embedder needs no provider key; hosted embedders use their supported API-key, +workload-token or AWS authentication. Anthropic does not provide its own embedding model; Voyage is not a built-in Loom provider. [Anthropic's embedding guidance](https://platform.claude.com/docs/en/build-with-claude/embeddings) -Loom's internal model jobs use provider API keys, not coding-agent CLI logins or -subscription credentials. Your external agent's choice of provider is independent. +Loom's internal model jobs can authenticate with provider API keys, gateway workload +tokens or deployment AWS credentials. Local models can run without provider credentials. +Coding-agent CLI logins and subscriptions do not configure Loom's internal providers; +your external agent's choice of provider is independent. ## Set the server configuration @@ -70,7 +77,8 @@ subscription credentials. Your external agent's choice of provider is independen As a platform administrator, open **Settings → Models & keys** (also available during Setup). Choose your LLM and embedding provider, enter their model IDs and API keys, then click -**Test & apply**. Leave per-job overrides blank to use one LLM for every enabled job. +**Test & apply**. Blank per-job overrides apply the main LLM to the displayed jobs; +the [conflict resolver](#keep-every-llm-job-on-your-chosen-provider) keeps its separate setting. Saved dashboard settings override the environment and are encrypted in ClickHouse. **No Docker restart or Compose command is needed.** Key and LLM updates apply live; embedding changes start the rebuild described below. **Use environment @@ -169,8 +177,42 @@ key and content unencrypted. Compatibility requires indexed float-array response with the configured width; a provider using a different API needs its own adapter. Changing the endpoint, model or width requires rebuilding existing embeddings. -If your organization currently has only Anthropic API access, obtain access to an -embedding provider before enabling semantic search. +With only Anthropic API access, configure the bundled semantic embedder with +`LOOM_EMBEDDING_PROVIDER=local`, `LOOM_EMBEDDING_MODEL=local` and +`LOOM_EMBEDDING_DIM=384`. To use a hosted embedder, +configure its supported authentication and dimensions. Changing an existing embedding +space requires [rebuilding embeddings](../architecture/llm-architecture.md#changing-the-embedder). + +### Amazon Bedrock + +Native Amazon Bedrock uses the server's AWS SDK credential chain, including deployment +roles, and needs permission to invoke the selected models or inference profiles. The +server image includes the AWS SDK; a source install needs its `aws` extra. Activate the +provider only after every server and worker supports it. AWS credentials stay in the +deployment; they are not entered as model-provider API keys in Models & keys. + +```dotenv +LOOM_LLM_PROVIDER=bedrock +LOOM_LLM_MODEL= +LOOM_BEDROCK_REGION=us-east-1 +LOOM_REQUIRE_REAL_PROVIDERS=true +``` + +Replace the model ID and region with ones your deployment can invoke; there is no +implicit Bedrock model. Keep the current embedder, or configure Cohere Embed v4 +separately, using an available model or inference profile, for example: + +```dotenv +LOOM_EMBEDDING_PROVIDER=bedrock +LOOM_EMBEDDING_MODEL=us.cohere.embed-v4:0 +LOOM_EMBEDDING_DIM=1536 +``` + +Changing the embedding provider requires rebuilding existing vectors. Missing or expired +AWS credentials fail rather than selecting a fake provider. Native Bedrock cannot be +used when an inference gateway is required. See [native Bedrock](../architecture/llm-architecture.md#native-bedrock) +for IAM resources, regional availability, supported dimensions and request limits, and +[per-job settings](#keep-every-llm-job-on-your-chosen-provider) for model overrides. ### Keep every LLM job on your chosen provider @@ -180,6 +222,7 @@ normally provider-specific jobs use your selected model: ```dotenv LOOM_MEMORY_EXTRACT_VERIFY_MODELS=${LOOM_LLM_MODEL} +LOOM_CONFLICT_RESOLVER_MODEL=${LOOM_LLM_MODEL} LOOM_SESSION_BRIEF_LLM_MODEL= LOOM_CODE_BISECT_LLM_RERANK_MODEL= LOOM_TRACKER_DUPCHECK_RERANK_MODEL= @@ -191,6 +234,10 @@ The empty values above inherit the main LLM. This keeps the capture verifier ena an empty verifier list would disable it. Remove any other cross-provider model overrides you added previously, including fallback, synthesis and reranking models. +The conflict resolver can retire superseded memories. Validate a replacement judge +against your own conflicts before automatic resolution; set +`LOOM_MEMORY_CONFLICT_RESOLVER_ENABLED=false` while evaluating it. + These settings matter: some shipped advanced defaults name Claude or OpenAI models explicitly, and an override with no provider key can fall back to the fake client. Using one model across jobs is a convenience configuration, not the configuration behind @@ -297,27 +344,32 @@ setting names the same model. `LOCAL_LLM_BASE_URL` to the assistant's Secret; the chart's policy lets LibreChat reach only DNS, its database, Loom and TCP 443, so the model server needs a NetworkPolicy of your own that selects the LibreChat pod. -- **Quality and speed.** Loom's prompts and thresholds are tuned on hosted models, and no - published benchmark covers a local one, so treat a small model as a way to try Loom - with no key or to keep everything on the machine, not as an equal: expect extraction, - `memory.ask` answers and the assistant's tool use to be weaker, by an amount nobody - has measured. A small model on a CPU is slow on long prompts such as extraction, so a - busy capture queue backs up; a GPU, or a larger model on a machine with the memory for - it, helps. For a team server, a hosted model answers better. +- **Quality and capacity.** Evaluate each model's extraction, `memory.ask` answers and + assistant tool use on your workload; deployment location alone does not establish + quality. Measure latency and capture throughput on the hardware you plan to use, + including long prompts and the expected arrival rate. Use the + [evaluation procedure](evaluate.md) to compare named local and hosted candidates. ### Through an inference gateway -A gateway that speaks the OpenAI or Anthropic API in front of the providers -(for budgets, cost attribution or one key for many services) takes Loom's -calls once its API roots point there; the model names stay what the gateway -serves: +An OpenAI-compatible proxy or inference gateway can serve Loom's chat-completions +requests. Select the provider and a model the gateway serves, as well as its API root +and key; a key alone does not change an explicitly configured fake provider: -```bash -LOOM_OPENAI_BASE_URL=https://gateway.example/v1 # chat and embeddings +```dotenv +LOOM_LLM_PROVIDER=openai +LOOM_LLM_MODEL= +LOOM_OPENAI_BASE_URL=https://gateway.example/v1 # chat and OpenAI embeddings LOOM_ANTHROPIC_BASE_URL=https://gateway.example # Messages: /v1/messages OPENAI_API_KEY= ``` +For an Anthropic-format gateway, select `LOOM_LLM_PROVIDER=anthropic`, its model ID +and `ANTHROPIC_API_KEY` instead. Configure embeddings separately using a supported +provider and its model; a chat endpoint does not imply embedding support. Check +[per-job overrides](#keep-every-llm-job-on-your-chosen-provider) too. Both API roots +above must use an allowed host when the gateway is required, even for an unused provider. + Off its default, the Anthropic key is also sent as `Authorization: Bearer`, the scheme gateways take. With a gateway key, require the gateway, so a wrong root can never send prompts and the key anywhere else. A token exchange requires it by itself. @@ -452,9 +504,10 @@ LOOM_INFERENCE_FEATURE_MAP_JSON='{"hyde": "search", "*": "memory"}' # Loom's fe forgot its org, and the gateway would bill it to no one. - An org id is sent percent-encoded where it needs to be. -Cached answers make no call: an embedding or a model answer served from Loom's -cache is not metered again. The customer is still checked before the cache, so a -path that names none is refused even on a hit. +An embedding or completion served from Loom's embedding or completion cache +avoids that provider call and is not metered again by the gateway. Serving a +cached `code.ask` answer can still require a query-embedding call. The customer +is checked before these caches, so a path that names none is refused even on a hit. #### Naming the agent on every call @@ -651,3 +704,31 @@ If dimensions change, check the existing schema's vector constraints before appl the new configuration. Preserve your data and follow [maintenance](../operations/maintenance.md#re-embed-and-re-extract) and [code-index operations](../operations/code-index-operations.md). + +### Diagnose silent model or provider failures + +Read the response body, not just the HTTP status: + +```bash +curl -sS "$LOOM_BASE_URL/v1/health" | jq '{status, fake_llm, fake_embedding, providers_degraded, providers, problems}' +``` + +The HTTP status checks ClickHouse availability; it does not prove model access. +`fake_llm` and `fake_embedding` expose configured fakes or missing-key fallbacks. +They can both be false while a real provider fails: inspect `providers_degraded`, each +provider's `failing` and `last_error`, and the fixes in `problems`. + +Provider outcomes are passive, process-local observations of actual calls; polling +health makes no model request. No recorded outcome is not proof of working inference. +A later recorded success clears that provider's failure. The `compatible` embedding +provider does not populate `providers`, so `providers_degraded: false` does not validate +that endpoint. Check its request errors and exercise retrieval. + +After checking configuration, use the [end-to-end smoke](../operations/monitoring.md#smoke-test) +with fresh authorized input and confirm a new embedding call. To check the LLM, use +[repository Q&A](../guides/repository-questions.md) on a readable indexed namespace with +`synthesize: true` and `no_cache: true`. An abstention can skip synthesis, so confirm a +new provider outcome or request trace tied to the probe on the serving process. +A missing observation leaves inference unverified; returned prose alone can come from +cache. These probes make model calls and can incur charges. For a stored memory missing +from search, also check the embedding backlog described in the smoke guide. diff --git a/docs/guides/agent-platforms.md b/docs/guides/agent-platforms.md new file mode 100644 index 0000000000..354f6de621 --- /dev/null +++ b/docs/guides/agent-platforms.md @@ -0,0 +1,174 @@ +# Offer Loom in an agent platform or factory + +An agent platform or factory can offer Loom as an optional shared memory and code +retrieval service through MCP, HTTP, or the Python and TypeScript SDKs. Start with a +running [Loom deployment](../operations/deployment.md), credentials for the intended +namespaces, and an explicit choice of which application stages call it. + +## Choose what the integration owns + +Keep the framework's execution state and Loom's shared knowledge distinct. A framework +can retain its conversation, checkpoint and orchestration state while Loom holds facts +or documents that other agents and later sessions need. A native adapter can also put +the state its interface supports in Loom; each +[framework adapter](../features/integrations/frameworks.md) documents that boundary +and its optional dependencies. Installing an adapter does not make every framework +memory operation use Loom. + +Choose an interface for the application you deliver: + +| Integration | Application responsibility | Starting point | +| ------------------------ | ------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------- | +| MCP | Configure the endpoint, token, tool listing and model's tool loop | [MCP clients](../features/integrations/mcp-clients.md) | +| Native framework adapter | Install the matching dependency and bind the supported interface | [Frameworks and orchestrators](../features/integrations/frameworks.md) | +| Direct SDK or HTTP calls | Await recall, validation or recording at defined stages; handle errors and abstention | [Agent integration](agent-integration.md) | +| Model-selected SDK tools | Supply definitions, execute selected calls and return their results to the model | [Code tools](code-tools.md) and [SDK tool loops](../reference/python-sdk.md) | + +For a factory, place retrieval before planning or implementation when its result is +required. Let investigation tools answer additional questions during the run. Record +the outcome after the task's own checks. The existing +[invocation paths](agent-integration.md#who-invokes-loom) distinguish application calls, +plugin hooks and model choices; no particular factory controller is required. + +## Call Loom explicitly from an SDK + +Install the [Python SDK](../reference/python-sdk.md) and set `LOOM_BASE_URL` to the +deployment and `LOOM_API_KEY` to a token with read access to `team-context`. Replace +the example organization and namespace with scopes that token permits: + +```python +import asyncio +import os + +from loom import Agent + + +async def main() -> None: + async with Agent( + org="example", + namespace="team-context", + agent="triage-worker", + base_url=os.environ["LOOM_BASE_URL"], + api_key=os.environ["LOOM_API_KEY"], + ) as agent: + result = await agent.call( + "memory.search", + agent="*", + query="Which checks are required before a change?", + top_k=3, + ) + print(result) + + +asyncio.run(main()) +``` + +This application call runs without asking a model to select a tool. Inspect the returned +hits or `abstain` before adding evidence to the prompt; an abstention is an expected +outcome when no memory qualifies. Authentication or transport errors need their own +handling. `agent="*"` searches across writers within the token's permitted scope; +it does not widen namespace or user access. The [memory operation reference](../reference/operations/memory.md#memorysearch) +defines the response. Use `call_with_meta` when recording the request ID and timings +([request correlation](../features/integrations/index.md#joining-a-request-across-systems)). + +For code investigation, use the existing Python and TypeScript +[`loom_code_lookup` examples](code-tools.md). The application fixes the code namespace; +the model chooses `symbol`, `search`, `callers` or `impact`. The model's ordinary tool +loop must still dispatch the selected call. An application can instead call the +[code operations](../reference/operations/code.md) directly by their dotted names. + +## Diagnose agents that never call the tools + +Check a representative task from connection through use of evidence: + +1. Confirm that the client can discover the intended tool and that the token can read + its namespace. A successful connection alone does not test either condition. +2. Make one explicit, relevant call. Check for a usable result, abstention, denied access + or a provider error before changing the model's instructions. +3. Check whether the running client supplied tool definitions and executed selected + calls. A raw model request does not run coding-client lifecycle hooks. +4. Give the model a short policy saying when to use each tool and when existing evidence + is sufficient. Reuse the [coding-agent policy and diagnostics](../features/integrations/claude-code.md#claude-does-not-call-loom-tools). +5. Record application calls, hook retrieval and model-selected calls separately. Confirm + that the returned evidence reached the model. Count errors, unused results and + zero-call tasks as outcomes, then [evaluate answer quality](../getting-started/evaluate.md). + +If a stage must retrieve or validate something, enforce that dependency in application +code. Instructions guide discretionary tool use; they do not enforce a required gate. + +## Give a CTO an evaluation they can check + +Evaluate the infrastructure decision on the platform's own repositories and tasks. +The [benchmark coverage map](../benchmarks/coverage.md) separates published results from +harnesses without published measurements. Follow its links to the committed artifacts, +models, datasets, reproduction commands and limitations. A published retrieval score +does not establish a task-success or cost gain for a different agent platform. + +Use the existing [evaluation guide](../getting-started/evaluate.md) for indexing, +labelled retrieval cases and deployment smoke tests. Then write a paired protocol +before collecting results: + +1. Freeze repository commits, task inputs, models, prompts, test commands and execution + budgets. Include triage, design and implementation tasks, unanswerable questions, + changed code, permission-denied cases and tasks needing no shared context. Label + expected source ranges or caller sets without exposing reference answers to the agent. +2. Run a baseline using the platform's usual checkout, framework memory and local tools. + Run the treatment with the same inputs and access plus the intended Loom integration. + Keep reference fixes and held-out tests outside both agent checkouts. Use fresh + sessions; prevent one arm's writes from teaching the other arm evaluation answers. +3. Record each task's actual calls, request IDs, outcomes and evidence delivered to the + model. Separate explicit application calls, hooks and model-selected tools. A run with + no calls remains a non-use outcome. A failed or misconfigured request is an integration + failure; report it separately from a grounded retrieval miss. +4. Score task success with the same objective tests or a fixed human rubric. Check answer + claims and citations against the frozen source, and score retrieval separately from + synthesis. Review false answers, stale lessons, denied reads and appropriate abstention. +5. Measure end-to-end wall time and model tokens for both arms. Include tool-definition + overhead, failed requests, retries, Loom's provider work and validation time. Report + indexing, storage and operation costs separately from per-task cost. Mark unavailable + usage or prices as unknown; an absent cost field is not a zero-cost result. +6. Report task count, paired outcomes, category results, repeated-run variation and + uncertainty. Keep raw failures and non-use cases. Small samples and mixed outcomes + support a bounded decision or another experiment, not a general accuracy claim. + +Give the decision maker a short scorecard with links to the underlying task records: + +| Decision | Evidence to retain | +| ---------------------------------------- | ------------------------------------------------------------------------------------------------- | +| Are answers or completed tasks better? | Baseline and treatment outputs, objective test results, valid citations, misses and uncertainty | +| Did agents use the intended integration? | Per-task calls and evidence receipt, including non-use and unusable results | +| Is the benefit worth the cost? | Total task latency and model cost, construction cost, operating cost and expected workload volume | +| Can the team operate it? | Allowed and denied reads, provider failures, recovery, retention, erasure and upgrade checks | + +Set acceptance thresholds before running the protocol. No improvement follows merely +from installing Loom or completing this procedure. Real model and indexing providers +can incur charges; the protocol is a proposed evaluation, not a measured result. + +## Deliver a customer deployment + +A platform provider can integrate a customer-operated Loom service or package the +documented [container or Kubernetes deployment](../operations/deployment.md) with its +application. Agree who operates storage, model providers, backups, upgrades and +credential rotation. Test the same release and adapter dependencies that will ship; +main-branch documentation can describe changes absent from an installed release. Use +the [release history](https://github.com/ClickHouse/loom/releases) and +[supported versions](../operations/supported-versions.md) to choose the matching source. + +Give each service identity only the namespaces and roles it needs +([tokens and grants](tokens-and-grants.md)). Repository indexing credentials and Loom +read authorization are separate controls. A chat connector login does not grant a +native service token access to a repository. Test allowed reads, denied reads, revocation +and [repository-job isolation](../operations/code-index-operations.md#repository-job-sandbox) +before delivery. Set retention and +[erasure](../features/memory/retention-and-erasure.md) requirements for shared content. + +Make optional integration behavior explicit in the platform: whether a customer has +enabled it, which stages depend on it, and what users see when it is unavailable. Treat +required validation failures as failures; report omitted optional context rather than +silently claiming it was used. Evaluate the deployment with the customer's own tasks +and [published evidence](../benchmarks/coverage.md). + +These are technical delivery options. They do not establish a partner program, joint +delivery commitment, support agreement, SLA, pricing or commercial terms. Those terms, +including who owns customer support and escalation, require a separate agreement with +the parties involved. diff --git a/docs/guides/repository-questions.md b/docs/guides/repository-questions.md index 6c7eaddfc5..f6564b3237 100644 --- a/docs/guides/repository-questions.md +++ b/docs/guides/repository-questions.md @@ -4,6 +4,47 @@ Use `code.search` to retrieve source and documentation, and `code.ask` to produc cited answer from that evidence. Both can serve a reader who has no local checkout. The repository must be indexed in the code namespace passed to the request. +## Serve triage agents, design assistants and team chatbots + +Run the assistant against the indexed repository, with a Loom token that can read its +code namespace. The application supplies the repository scope; the user supplies the +question. Git fetch credentials used to build the index do not grant a chatbot's users +access to it. Follow the [repository access rules](../features/code-index/repositories.md#who-can-read) +and preserve each caller's authorized scope. + +| Assistant | Evidence to request | Result to check | +| ---------------- | ------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------- | +| Triage agent | `code.search` for the symptom, then `code.symbol` and `code.callers` for the relevant definition | Cited implementation and callers that explain the reported path | +| Design assistant | `code.impact` for a proposed symbol change and [linked team decisions](../use-cases/memory-linked-to-code.md) | Affected definitions, test mappings and lessons with their freshness state | +| Team chatbot | `code.ask` for a cited answer, or `code.search` for the application's own reader | Source ranges supporting each claim, or an explicit abstention | + +For example, replace the organization, namespace and question with those of your indexed +repository. Server-side synthesis needs a configured real LLM: + +```sh +curl --fail-with-body -X POST "$LOOM_BASE_URL/v1/code.ask" \ + -H "Authorization: Bearer $LOOM_API_KEY" \ + -H 'Content-Type: application/json' \ + --data '{"org":"example","namespace":"code-example","question":"Where are incoming requests authorized?","synthesize":true}' +``` + +A grounded answer returns `answer` and `citations`; an unanswerable question can return +`abstain` and `abstain_reason`. Inspect the completeness signals below before displaying +an answer. For an uncached retrieval-only response, pass both `synthesize: false` and +`no_cache: true`; `answer` is empty and `citations` holds the retrieved units. With cache +lookup enabled, a matching cached synthesized answer can still be returned. These flags +do not disable any +embedding or reranking provider used by retrieval. The +[code operation reference](../reference/operations/code.md#codeask) gives the full contract. + +The answer concerns the indexed branch, not an uncommitted local checkout. Check +[freshness](../features/code-index/freshness.md), [language and compiler-tier support](../features/code-index/overview.md#language-support), +and returned truncation or lower-bound metadata. An empty caller list does not prove +there are no runtime callers, and an impact test mapping does not mean a test ran. +Use the [SDK intent table](code-tools.md#choose-evidence-by-intent) to expose bounded +lookups to a model, and [evaluate on your own repository](../getting-started/evaluate.md#code-intelligence) +before relying on its answers. + ## Calibrate the embedding model After indexing, run `code.calibrate` as an administrator for the indexed namespace. @@ -85,6 +126,12 @@ is a separate result. Keep both measures when evaluating a repository assistant. ## Using retrieval inside a coding agent +An existing checkout already provides local files and tools. Loom can add remote +cross-file relationships, cited retrieval and shared lessons that are not present in +that checkout. Whether these reduce work or improve answers depends on the task and +index coverage. Use local tools for uncommitted files and insufficient or unsupported +indexed results; check the indexed commit before treating evidence as current. + For a coding agent, also inspect whether the model requested tools and received useful results. A connected MCP server does not establish tool adoption. Hook-supplied evidence and model-selected lookups are separate paths; report them separately. diff --git a/docs/guides/tokens-and-grants.md b/docs/guides/tokens-and-grants.md index ee3cc8bb73..ce0648c1de 100644 --- a/docs/guides/tokens-and-grants.md +++ b/docs/guides/tokens-and-grants.md @@ -450,9 +450,11 @@ complete the self-service surface. reading needs a grant or a token that names it. - `LOOM_CODE_ORG_READABLE=true` (or `"org_readable": true` on an entry of - `LOOM_CODE_REPOS_JSON`) lets every authenticated principal of the org read every code - namespace with the token it holds. -- Refreshing (`writer`) and administering (`admin`) always come from grants. + `LOOM_CODE_REPOS_JSON`) lets authenticated org members read standard code + namespaces with the token it holds. +- Refreshing requires `writer` and administering requires `admin`. Standard code + namespaces accept sufficient token claims or grants, subject to scoped-token + ceilings and live grant caps. Restricted namespaces require both (below). - An unowned code namespace is adopted with `POST /v1/admin/namespace.adopt` (or `loom namespaces-adopt`); a restricted one never is (below). @@ -460,10 +462,11 @@ See [code index repositories](../features/code-index/repositories.md). ## Restricted namespaces (`classification=restricted`) -Everything above widens access. `classification` narrows it: a namespace whose -`loom.namespaces.classification` is `restricted`, or whose `LOOM_CODE_REPOS_JSON` entry says -`"classification": "restricted"` (either input restricts), switches every default off. A -read or write passes only when **all** hold: +A namespace whose directory row or effective code repository classification is +`restricted` switches automatic access off. For an operator-configured repository, +a persisted `code.repo.set_access` row overrides the configured classification and +join policy. Widening to `standard` requires confirmation and the caller's own live +admin grant. A read or write on a restricted namespace passes only when **all** hold: 1. the token belongs to the namespace's org (`org="*"` does not count); 2. the token names the namespace literally, in `namespaces` or `ns_roles` @@ -477,7 +480,10 @@ refusal is the generic "cannot access" message, so it does not reveal whether th exists. If the classification cannot be read, access is denied. - Search fan-out, context packs and org-wide rollups skip the restricted namespaces the - caller cannot read, and `code.repos` hides them. + caller cannot read. `code.repos` can return permitted join, access-request or adoption + metadata with `readable: false`; those rows omit populated index counts and watermarks + and do not authorize code reads. Other inaccessible entries are omitted. See + [repository listing and access](../features/code-index/repositories.md#who-can-read). - Each request admitted to the restricted namespace it names writes a `namespace.restricted_access` audit row before the answer (`LOOM_RESTRICTED_AUDIT_INLINE=false` writes it beside the request instead, diff --git a/docs/index.md b/docs/index.md index 8be60892b8..dfa85e1a23 100644 --- a/docs/index.md +++ b/docs/index.md @@ -110,6 +110,7 @@ client sees one the server no longer reads. ## Guides [Agent integration](guides/agent-integration.md) · +[Offer Loom in an agent platform or factory](guides/agent-platforms.md) · [Headless bots](guides/headless-bots.md) · [Tokens and grants](guides/tokens-and-grants.md) · [Trace and resume long runs](guides/trace-and-resume.md) · @@ -146,6 +147,7 @@ client sees one the server no longer reads. [Overview](features/code-index/overview.md) · [Repositories](features/code-index/repositories.md) · +[Repository Q&A for agents and chatbots](guides/repository-questions.md) · [Freshness](features/code-index/freshness.md) · [History and PRs](features/code-index/history-and-prs.md) · [Operations](features/code-index/operations.md) · diff --git a/docs/operations/code-index-operations.md b/docs/operations/code-index-operations.md index f0f290dfd8..c6d3c4bc42 100644 --- a/docs/operations/code-index-operations.md +++ b/docs/operations/code-index-operations.md @@ -1,9 +1,13 @@ # Code index operations -This page covers how to enable the code index workers, configure repositories and keep -the index current; the code index runs only when an operator enables its workers and -configures repositories. Every `code_*` worker except `code_coverage_notify` and -`code_backport_gaps` is off by default. What the `code.*` operations answer is in the +This page covers how to configure code index workers and repositories and keep the +index current. Workers run when the supervisor is admitted and their settings and +repository prerequisites permit it. Some compiler workers are enabled by default; +they still need a configured repository, its first indexed revision, sandbox admission +and the required checkout and toolchain. Check each worker's defaults in the +[workers reference](../reference/workers.md) and the +[configuration reference](../reference/configuration.md#code). +What the `code.*` operations answer is in the [code index overview](../features/code-index/overview.md); how the index is built is in [code index architecture](../architecture/code-index-architecture.md). @@ -160,7 +164,9 @@ configured restricted namespace, so a private repository is set up in this order finish step 3 before handing out `*` tokens of the org. 3. The maintainer restricts it with `code.repo.set_access` (`classification: "restricted"` and `join_policy: "approval"` or `"closed"`; both are required); then set - `classification: "restricted"` on the entry as well, so either input restricts, and run + `classification: "restricted"` on the entry as well to align the configured default + with the stored policy. A stored access decision takes precedence over the + configured classification and join policy. Run `loom code backfill`. Mirrors and compiler-tier checkouts are created at mode 2770 when they belong to `code_repos_shared_gid`, open to the server's user and that group only, and 0700 otherwise (see "Writable git mirrors"); one an @@ -189,8 +195,9 @@ with `docker compose --env-file .env -f dev/docker-compose.cloud.yml config`. Who can read an index is governed by the namespace's join policy and grants ([tokens and grants](../guides/tokens-and-grants.md)). `LOOM_CODE_ORG_READABLE=true` -makes every code namespace readable by every principal in the org without a grant; the -default is off. +lets authenticated org members read standard code namespaces without a grant; the +default is off. Restricted namespaces still need a token naming the namespace and a +live direct grant; see [restricted namespaces](../guides/tokens-and-grants.md#restricted-namespaces-classificationrestricted). ## First index @@ -213,9 +220,10 @@ docker exec loom-workers loom code backfill-prs --namespace code-myrepo --org ac an operator `git_dir` must be writable by the `loom` user. Network: a host with an empty disk makes one full clone per indexed repository; keep `LOOM_CODE_REPOS_ROOT` on a persistent volume so a rollout does not re-clone every mirror on every pod. -- `backfill` parses, chunks, extracts symbols and edges, and embeds each file. It is - resumable: re-run it after an interruption. `--phase index` then `--phase embed` splits - the work, since the unembedded rows are the embed phase's queue. `--commit ` pins +- `backfill` parses, chunks, and extracts symbols and edges. With `--embed`, it also + embeds each file. It is resumable: re-run it after an interruption. + `--phase index` then `--phase embed` splits the work, since the unembedded rows are + the embed phase's queue. `--commit ` pins the tree (and implies `--no-fetch`); `--concurrency N` parallelises files; `--limit N` indexes a sample. - The first full backfill of a namespace also re-derives every file's edges after the @@ -423,13 +431,14 @@ memory the workers process has. The full index of such a tree takes about an hou `git gc --no-prune` every `LOOM_CODE_MIRROR_GC_INTERVAL_H`, under the same rules as the mirrors (below). Every git command a tier runs in a checkout has auto-gc and background maintenance off, so no fetch starts a repack. -- **Entry locks.** A full pass and a delta never work on the same entry at once: each - takes the entry's lock file in `LOOM_CODE_SCIP_LOCK_DIR` (`var/cron/locks`), +- **C/C++ entry locks.** The `code_scip_full` and `code_scip_delta` workers never work + on the same entry at once: each takes the entry's lock file in + `LOOM_CODE_SCIP_LOCK_DIR` (`var/cron/locks`), `scip...lock` with both names percent-encoded (`loom code scip-lock-path --org ORG --namespace NS` prints it). A full pass waits for a delta already on its entry; a delta skips an entry a full pass holds and names it under `skipped` ("entry lock held (full pass running)"). Different entries run side by - side. Every pass also takes the entry's key in ClickHouse (`scip_entry:/` + side. These passes also take the entry's key in ClickHouse (`scip_entry:/` in `loom._migration_lock`), so passes on two hosts exclude each other too; a dead holder's key expires after 120 s. A pass that cannot take the key (ClickHouse or Keeper unreachable) fails instead of @@ -779,7 +788,7 @@ The setting also resumes the git jobs of entries that members added before `LOOM_CODE_SELF_SERVICE_PLATFORM_ADMIN_ONLY` was set. They keep the remotes their adders named and are cloned and fetched the same way. Before turning it on, list the branches in the registry (the active rows of `loom.code_repos` whose `row_kind` is not `access`; -`code.repos` lists only what its caller may read) and have those a platform admin would +`code.repos` is caller-filtered and can include request-only metadata) and have those a platform admin would not add removed with `code.repo.remove`. `/v1/health` and `loom doctor` keep reporting `repo_jobs_unsandboxed` as a warning, with @@ -1053,8 +1062,8 @@ copy. | Command | Does | | -------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `loom code compact [--table T] [--apply]` | measures, and with `--apply` runs, `OPTIMIZE … FINAL` on a code vector table once its physical copies per key reach `LOOM_CODE_COMPACT_MIN_RATIO` (1.5), and waits for the merge; bulk passes do it themselves after `LOOM_CODE_COMPACT_AFTER_BULK_ROWS` rows ([maintenance](./maintenance.md#compaction-and-optimize)) | -| `loom code warm ` | asks each question once so `code.ask` answers are cached for everyone (text lines or JSONL with `question`); hits are free, so re-running is safe | -| `loom code answers-purge` | retires cached `code.ask` answers after a synthesis-model change (the cache otherwise invalidates only when a cited file changes); dry run until `--apply` | +| `loom code warm ` | asks each question once to populate shared `code.ask` answers (text lines or JSONL with `question`); a valid answer hit avoids fresh synthesis, but query embedding can still cost | +| `loom code answers-purge` | retires stored `code.ask` answers; dry run until `--apply`. Normal reuse already checks the model and prompt, answer-key settings, maximum age and recorded source hashes | | `loom code repack` | rebuilds context packs for recently merged PRs in the current shape | | `loom code crossref` | the SQL/shell test → implementing C++ reference pass (idempotent) | | `loom code coverage-import [--dry-run]` | imports the newest per-test line coverage run of the configured source into the test map ([measured coverage](#measured-coverage)) | @@ -1078,16 +1087,24 @@ has a gold question set; it is never run in CI, whose embedder is fake the mirror host loop; every other code worker keeps its own switch ([workers](../reference/workers.md)). - **Retire a repository.** A maintainer uses **Retire…** on the repository's Access tab - (`code.repo.remove` with `confirm` set to the namespace). The entry goes inactive, an - erasure saga purges every `loom.code_*` table for the namespace, and the bare mirror is - deleted when no other branch uses it. Search telemetry is kept. An operator-configured + (`code.repo.remove` with `confirm` set to the namespace). The registry and directory + entries go inactive, and a resumable namespace erasure saga purges selected index + tables plus search traces and outcomes. The inactive registry row remains. The index + purge excludes `code_answers`; a separate step deletes answers whose quoted + `memory_namespace` matches the erased namespace, so other cached code answers can + remain. The bare mirror is deleted when no other branch uses it. An operator-configured entry is refused with `REPO_FROM_CONFIG`: remove it from `LOOM_CODE_REPOS_JSON` instead. With `LOOM_CODE_INTEL_ENABLED=false` the page cannot list repositories, but `code.repo.remove` over HTTP, MCP or the SDK is still served; the other replicas then sweep their copy of the mirror only once code intelligence is on again. - **Erase an index.** `erasure.request` with `scope_type = namespace` on a code namespace - also purges the `loom.code_*` tables. An entry that is still configured is re-indexed - from scratch on the next pass, so retire it first. + also purges indexed content and the `git_head` watermark. An active self-service + entry can bootstrap again; an operator-configured entry needs a full + `loom code backfill --org --namespace ` before incremental ingestion + resumes. A targeted `--paths-file` repair does not restore that watermark. Add + `--embed` to request vector rebuilding in the same backfill. To keep the namespace + unindexed, first retire the self-service entry or remove the operator entry from + `LOOM_CODE_REPOS_JSON`. ## When a tier fails diff --git a/docs/reference/operations.md b/docs/reference/operations.md index 9211aa7f43..e6252700df 100644 --- a/docs/reference/operations.md +++ b/docs/reference/operations.md @@ -224,7 +224,7 @@ Fields for every `code` op: [operations/code.md](operations/code.md). | [`code.repo.remove`](operations/code.md#codereporemove) | Retire a repository index a person added. | admin | idempotent | yes | yes | | [`code.repo.set_access`](operations/code.md#codereposet_access) | A maintainer sets a repository's access — its classification (`standard` \| `restricted`) and join policy (`open` \| `approval` \| `closed`). | admin | idempotent | no | yes | | [`code.repo.set_source`](operations/code.md#codereposet_source) | A maintainer sets the trackers this repository reads (records of issues, PRs, tests and incidents registered with `tracker.source.add`), in order. | admin | idempotent | no | yes | -| [`code.repos`](operations/code.md#coderepos) | Lists the code namespaces (repo@branch entries) configured on this deployment that the caller is authorized to read, across orgs — each row carries its org. | viewer | read | yes | | +| [`code.repos`](operations/code.md#coderepos) | List readable repositories and permitted access-request metadata. | viewer | read | yes | | | [`code.review_brief`](operations/code.md#codereview_brief) | A merged PR's review brief: the point read of what the ingest built at merge — touched symbols with callers and fan-in, coverage verdicts, setting facts, regression history, blame of the deleted… | viewer | read | no | | | [`code.scip_ingest`](operations/code.md#codescip_ingest) | Accept one part of a SCIP index upload (compiler-tier symbols and references) for a batch. | writer | idempotent | no | yes | | [`code.scip_status`](operations/code.md#codescip_status) | Report the progress of a SCIP upload batch: parts received and apply status. | viewer | read | no | | diff --git a/docs/reference/operations/code.md b/docs/reference/operations/code.md index 15528526f5..c2e5049a43 100644 --- a/docs/reference/operations/code.md +++ b/docs/reference/operations/code.md @@ -22,19 +22,19 @@ Ask a QUESTION about this codebase and get a grounded answer with [file:line] ci | `user_id` | `str` | `'-'` | The end user the row belongs to; `'-'` for none. | | `consumer` | `str` | `''` | trace attribution: which integration is calling (clickgap\|claude-code\|…) | | `question` | `str` | required | The question to answer about the indexed code (grounded on a code.search of it). | -| `synthesize` | `bool` | `True` | False = retrieval-only (agent synthesizes); True = grounded server-side answer with citations, memoized across agents and invalidated when any cited file changes at merge. | +| `synthesize` | `bool` | `True` | False disables fresh synthesis; use no_cache=True to exclude cached answers. | | `rerank` | `str` | `''` | Reranker for the retrieval: "" = the deployment default (LOOM_CODE_SEARCH_RERANK_DEFAULT), "none" = off for this request, "builtin:openai" / "builtin:cohere", or the name of an endpoint in LOOM_RERANK_ENDPOINTS_JSON (not a URL). | -| `no_cache` | `bool` | `False` | Skip the cache lookup and synthesize fresh (the fresh answer still refreshes the cache entry). | +| `no_cache` | `bool` | `False` | Skip answer and pre-synthesis prompt-cache lookups; fresh synthesis, when enabled, still refreshes the answer cache. | | `token_budget` | `int \| None` | `None` | Retrieval context handed to synthesis, in tokens. | | Output field | Type | Meaning | | --- | --- | --- | -| `answer` | `str` | The grounded answer text; "" on an abstain or when synthesize=false. | +| `answer` | `str` | The grounded answer text; empty on abstention or an uncached retrieval-only call. | | `partial` | `bool` | True when retrieval or provider limits prevent a complete response. | | `partial_reason` | `str` | Sanitized reason for an incomplete response; empty when complete. | | `note` | `str` | Caller-facing warning about missing retrieval or synthesis. | | `coverage` | `Literal['', 'full', 'partial', 'none']` | Synthesis self-assessment, not verified correctness; empty on cached answers. | -| `citations` | `list[dict[str, Any]]` | The units the answer used (or was shown when none is cited): {path, start_line, end_line, symbol, unit_id} each; the retrieved units when synthesize=false. | +| `citations` | `list[dict[str, Any]]` | The units the answer used (or was shown when none is cited): {path, start_line, end_line, symbol, unit_id} each; retrieved units on an uncached retrieval-only call. | | `source` | `Literal['cached', 'synthesized', 'abstained']` | cached = served from the answer cache \| synthesized = answered now (or retrieval-only when synthesize=false) \| abstained = no grounded answer (`abstain_reason` says why). | | `cache_similarity` | `float` | Cosine between this question and the served cached question; 0.0 unless source=cached. | | `cache` | `str` | What the answer cache did. hit (a cached answer whose key, identifiers and files all matched) \| miss \| stale (the nearest entry's files, prompt version or age invalidated it) \| guarded (the nearest cached question names different… | @@ -1316,7 +1316,7 @@ A maintainer sets the trackers this repository reads (records of issues, PRs, te ## `code.repos` -Lists the code namespaces (repo@branch entries) configured on this deployment that the caller is authorized to read, across orgs — each row carries its org. +List readable repositories and permitted access-request metadata. `POST /v1/code.repos` · MCP `loom_code_repos` · role `viewer` · read @@ -1333,7 +1333,7 @@ Lists the code namespaces (repo@branch entries) configured on this deployment th | Output field | Type | Meaning | | --- | --- | --- | -| `repos` | `list[CodeRepoSummary]` | One row per entry the caller reads (plus unreadable rows an org admin may adopt), narrowed to the `repo` / `repos` match when either was given; [] when none. | +| `repos` | `list[CodeRepoSummary]` | Readable entries and permitted request-only or adoption metadata, narrowed to the `repo` / `repos` match when either was given; [] when none. | ## `code.review_brief` diff --git a/docs/use-cases/memory-linked-to-code.md b/docs/use-cases/memory-linked-to-code.md index cc3da089b1..539e669ed8 100644 --- a/docs/use-cases/memory-linked-to-code.md +++ b/docs/use-cases/memory-linked-to-code.md @@ -1,7 +1,8 @@ # Memory linked to code for coding agents Loom links coding agents' memories to the functions, classes and files they name, follows -them through refactors, and flags a memory when its code changes. By default, search leaves +them through refactors, and flags a memory when its code changes. Team decisions and notes +saved as memories use the same links. By default, search leaves out a memory whose code is all gone unless a person pinned, locked or verified it, and every read that serves a lesson says when its code changed or is gone. It runs over MCP, HTTP and the SDKs on the repositories a self-hosted Loom indexes (the @@ -77,10 +78,43 @@ hooks capture with `repo:`, and the session primer names the id agent saves itself with `loom_memory_set` links when it carries that tag (`tags: ["repo:"]`) or is written in the repository's code namespace; any client can do either. Over HTTP (or `agent.call` in the Python SDK), a lesson saved in your own namespace -is read only when the call names that namespace in `lesson_namespaces`. Each link pass links -writes at least 60 seconds old and passes run every 60 seconds, so once trunk is indexed, -`code.symbol` on that definition returns the lesson under `lessons` within about two -minutes. +is read only when the call names that namespace in `lesson_namespaces`. Writes become +eligible after `LOOM_MEMORY_CODE_LINKS_SETTLE_SECONDS` (60 seconds by default). An +admitted worker processes them on later passes; `LOOM_MEMORY_CODE_LINKS_INTERVAL_SECONDS` +defaults to 60 seconds. Queue backlogs, pass duration and worker failures can delay +completion. Once trunk is indexed, read `code.symbol` on the definition and check +`lessons` to confirm that the link resolved. + +### Save and retrieve a linked team decision + +Replace the repository tag, symbol and claim with a real, verified lesson from your +indexed repository. The token needs write access to `team-context` and read access to +`code-example`. This example saves a memory outside the code namespace, so the symbol +read explicitly names its memory namespace: + +```sh +curl --fail-with-body -X POST "$LOOM_BASE_URL/v1/memory.set" \ + -H "Authorization: Bearer $LOOM_API_KEY" \ + -H 'Content-Type: application/json' \ + --data '{"org":"example","namespace":"team-context","agent":"design-assistant","memory_key":"design/retry-tests","value":"Changes to `pkg.Reader.fetch_page` require retry tests.","tags":["repo:"]}' +``` + +Check the write receipt before claiming it was saved +([write outcomes](../guides/agent-integration.md#4-read-the-writes-answer)). After the +link worker has processed it, request the actual indexed definition: + +```sh +curl --fail-with-body -X POST "$LOOM_BASE_URL/v1/code.symbol" \ + -H "Authorization: Bearer $LOOM_API_KEY" \ + -H 'Content-Type: application/json' \ + --data '{"org":"example","namespace":"code-example","name":"pkg.Reader.fetch_page","include_lessons":true,"lesson_namespaces":["team-context"]}' +``` + +Look for the saved memory under `lessons` and inspect `code_state`. A saved memory is +not proof that a link resolved. An ambiguous name, pending indexing or link work, +permissions and response caps can prevent it appearing; use `memory.get` to inspect +the memory's own `code` links. A tag associates content with a repository; it grants +no read access. ## Check a lesson after a code edit @@ -118,4 +152,19 @@ tool selection are not tested. For removal, held-memory exceptions and model-backed revision, see the [link policies](../concepts/memory.md#memories-linked-to-code). +To list review work for the example memory, call the read-only integrity report: + +```sh +curl --fail-with-body -X POST "$LOOM_BASE_URL/v1/namespace.validate" \ + -H "Authorization: Bearer $LOOM_API_KEY" \ + -H 'Content-Type: application/json' \ + --data '{"org":"example","namespace":"team-context","keys":["design/retry-tests"]}' +``` + +Inspect `code_links_stale` findings and any skipped checks. A report with unavailable +evidence is not a freshness proof. Review the current source and revise the same memory +key when the decision changes; +[history and provenance](../features/memory/history-and-provenance.md) retain the prior +versions. A code change alone does not authorize rewriting the decision. + Some ops on this page are marked experimental on the [status page](../reference/status.md). diff --git a/llms.txt b/llms.txt index 4b32f3b00c..6cd692c67a 100644 --- a/llms.txt +++ b/llms.txt @@ -79,7 +79,7 @@ Start with [the docs map](docs/index.md). To install Loom, a coding agent reads - [Working memory and profiles](docs/features/memory/working-memory-and-profiles.md): Two planes hold text an agent is given every time instead of finding it by search: - [Writing memory](docs/features/memory/writing.md): A memory is one claim stored under a key in a namespace. - [Audit log](docs/features/observability/audit.md): Audit rows, the `audit_verifier` worker and the checkpoint chain are on by default. -- [Cost and usage](docs/features/observability/cost-and-usage.md): Loom records every upstream dollar it spends on embedding and LLM calls against the operation that spent it, in the per-operation ledger `loom.agent_events`. +- [Cost and usage](docs/features/observability/cost-and-usage.md): Loom records reported token usage and estimated provider costs for embedding and LLM calls against the operation that made them, in the per-operation ledger `loom.agent_events`. - [Telemetry](docs/features/observability/telemetry.md): Loom reports on itself in three layers: - [Issue duplicate checks and regression candidates for agents](docs/features/tracker/overview.md): With `LOOM_TRACKER_ORG` set and a registered source, Loom lets agents check whether a bug is already reported and rank changes that may have introduced a regression, using tracker records and indexed PR history. - [Maintain a GitHub tracker record](docs/features/tracker/github-feed.md): `loom tracker-feed` copies repository issues, pull requests, issue comments and inline review comments into a team's dedicated ClickHouse database. @@ -88,6 +88,7 @@ Start with [the docs map](docs/index.md). To install Loom, a coding agent reads ## Guides - [Integrate Loom into an agent](docs/guides/agent-integration.md): This guide covers what an agent sends to Loom and how it reads the answers: the client surface, the identity, what to write, and how to read. +- [Offer Loom in an agent platform or factory](docs/guides/agent-platforms.md): An agent platform or factory can offer Loom as an optional shared memory and code retrieval service through MCP, HTTP, or the Python and TypeScript SDKs. - [Code tools for model integrations](docs/guides/code-tools.md): An indexed code namespace and a token with read access let an SDK tool loop investigate definitions, behavior and relationships with `loom_code_lookup`. - [Debug a request](docs/guides/debugging.md): Loom tags everything one request does with the same id, so tracing a request is a join. - [Customize the extraction pipeline](docs/guides/extraction-pipeline.md): `memory.set_from_messages` runs its transcript through an ordered list of steps that share one state. @@ -111,7 +112,7 @@ Start with [the docs map](docs/index.md). To install Loom, a coding agent reads - [Cell control upgrades and recovery](docs/operations/cell-control.md): Cell control mutations require generation-aware clients and router handlers, and direct access to the cell's Postgres. - [Update a cell's configuration or addresses](docs/operations/cell-update.md): `loom cell update` reserves a configuration before the cell accepts it. - [ClickHouse Cloud and Managed Postgres](docs/operations/clickhouse-cloud.md): This guide runs Loom on Kubernetes over two managed services: ClickHouse Cloud for projections, vectors, history and analytics, and ClickHouse Managed Postgres as the memory authority. -- [Code index operations](docs/operations/code-index-operations.md): This page covers how to enable the code index workers, configure repositories and keep the index current; the code index runs only when an operator enables its workers and configures repositories. +- [Code index operations](docs/operations/code-index-operations.md): This page covers how to configure code index workers and repositories and keep the index current. - [Deployment](docs/operations/deployment.md): Loom is one server image plus ClickHouse (and Postgres, under `LOOM_STORAGE_AUTHORITY=postgres`), with the document extractor service as an image of its own (container images). - [Kubernetes](docs/operations/kubernetes.md): `deploy/helm/loom` runs Loom as N identical replicas on Kubernetes: one Deployment of web pods over one ClickHouse service, each pod a full replica. - [Maintenance](docs/operations/maintenance.md): Every command here is a `loom` subcommand; run it inside the container that has the deployment's environment (`docker exec loom-server loom …`, or `loom-workers` in a split deployment). diff --git a/sdk/python/loom/ops.py b/sdk/python/loom/ops.py index 5296dde77a..5bbf06c2d0 100644 --- a/sdk/python/loom/ops.py +++ b/sdk/python/loom/ops.py @@ -7363,7 +7363,7 @@ class CodeRepoSetSourceOutput(TypedDict, total=False): class CodeReposInput(TypedDict, total=False): - "Lists the code namespaces (repo@branch entries) configured on this" + "List readable repositories and permitted access-request metadata." org: NotRequired[str] namespace: NotRequired[str] @@ -7376,7 +7376,7 @@ class CodeReposInput(TypedDict, total=False): class CodeReposBody(TypedDict, total=False): - "Lists the code namespaces (repo@branch entries) configured on this" + "List readable repositories and permitted access-request metadata." org: NotRequired[str] namespace: NotRequired[str] diff --git a/sdk/ruby/lib/clickhouse/loom/generated/ops.rb b/sdk/ruby/lib/clickhouse/loom/generated/ops.rb index 20f4c49d6c..30ad0f371a 100644 --- a/sdk/ruby/lib/clickhouse/loom/generated/ops.rb +++ b/sdk/ruby/lib/clickhouse/loom/generated/ops.rb @@ -693,7 +693,7 @@ def repo_set_source(body = nil, request_options: nil, **fields) end # `code.repos` (viewer, read-only). - # Lists the code namespaces (repo@branch entries) configured on this + # List readable repositories and permitted access-request metadata. def repos(body = nil, request_options: nil, **fields) call_op("code.repos", body, fields, request_options) end diff --git a/server/code/api/models_repos.py b/server/code/api/models_repos.py index 6ac41fc2d2..51abda2e6b 100644 --- a/server/code/api/models_repos.py +++ b/server/code/api/models_repos.py @@ -15,17 +15,17 @@ _GIT_PROXY_IDENTITY_OR_EMPTY = f"^(?:{GIT_PROXY_IDENTITY.pattern})?$" # ============================================================ -# code.repos — every configured repo the caller may read +# code.repos — readable repositories and permitted access metadata # ============================================================ class CodeReposInput(_CodeIdentity): - """Lists the code namespaces (repo@branch entries) configured on this - deployment that the caller is authorized to read, across orgs — each row - carries its org. Transport authz targets the default code namespace when - there is one to use; the handler then checks each entry separately, so - unreadable entries are omitted, never 403'd, and the dashboard renders an - empty state.""" + """List readable repositories and permitted access-request metadata. + + Each row names its organization. Rows with readable=False expose namespace + metadata without populated index counts or watermarks; other inaccessible + entries are omitted. + """ # A caller standing in one checkout (the plugin's code shunt, from the origin # remote) names it, and the answer is that repository's entries only — one status @@ -70,10 +70,8 @@ class CodeRepoSummary(ProjectedRow): indexed_at: str = "" # git_head watermark's updated_at last_refresh_at: str = "" counts: dict[str, int] = Field(default_factory=dict) - # False only for org-admin-shaped callers (flat ADMIN+) who hold no grant - # on the entry: they see it exists so they can adopt it (bootstrap of an - # auto-registered, ownerless code namespace). Everyone else never sees - # entries they cannot read. + # Whether the caller may read code; False rows expose permitted join/adoption + # metadata without populated index counts or watermarks. readable: bool = True # How the namespace is joined and where the caller stands, so the Code view can # render Join / Request access / Cancel without a second call. `my_status` is @@ -138,8 +136,8 @@ class CodeRepoSummary(ProjectedRow): class CodeReposOutput(BaseModel): - # One row per entry the caller reads (plus unreadable rows an org admin may adopt), - # narrowed to the `repo` / `repos` match when either was given; [] when none. + # Readable entries and permitted request-only or adoption metadata, narrowed to + # the `repo` / `repos` match when either was given; [] when none. repos: list[CodeRepoSummary] diff --git a/server/code/api/models_search.py b/server/code/api/models_search.py index d9b9f5421a..c574b9a900 100644 --- a/server/code/api/models_search.py +++ b/server/code/api/models_search.py @@ -305,18 +305,14 @@ def _drop_empty_fanout(self, handler: SerializerFunctionWrapHandler) -> dict[str class CodeAskInput(_CodeIdentity): # The question to answer about the indexed code (grounded on a code.search of it). question: str = Field(min_length=1, max_length=2000) - # False = retrieval-only (agent synthesizes); True = grounded server-side - # answer with citations, memoized across agents and invalidated when any - # cited file changes at merge. + # False disables fresh synthesis; use no_cache=True to exclude cached answers. synthesize: bool = True # Reranker for the retrieval: "" = the deployment default # (LOOM_CODE_SEARCH_RERANK_DEFAULT), "none" = off for this request, "builtin:openai" / # "builtin:cohere", or the name of an endpoint in LOOM_RERANK_ENDPOINTS_JSON (not a URL). rerank: RerankName = "" - # Skip the cache lookup and synthesize fresh (the fresh answer still - # refreshes the cache entry). For eval and for agents that suspect a - # stale answer. Also skips the exact-prompt cache of the pre-synthesis - # LLM calls. + # Skip answer and pre-synthesis prompt-cache lookups; fresh synthesis, when + # enabled, still refreshes the answer cache. no_cache: bool = False # Retrieval context handed to synthesis, in tokens. None = the # deployment default (settings.code_ask_token_budget). @@ -324,7 +320,7 @@ class CodeAskInput(_CodeIdentity): class CodeAskOutput(BaseModel): - # The grounded answer text; "" on an abstain or when synthesize=false. + # The grounded answer text; empty on abstention or an uncached retrieval-only call. answer: str = "" # True when retrieval or provider limits prevent a complete response. partial: bool = False @@ -335,7 +331,7 @@ class CodeAskOutput(BaseModel): # Synthesis self-assessment, not verified correctness; empty on cached answers. coverage: Literal["", "full", "partial", "none"] = "" # The units the answer used (or was shown when none is cited): {path, start_line, - # end_line, symbol, unit_id} each; the retrieved units when synthesize=false. + # end_line, symbol, unit_id} each; retrieved units on an uncached retrieval-only call. citations: list[dict[str, Any]] = Field(default_factory=list) # cached = served from the answer cache | synthesized = answered now (or retrieval-only # when synthesize=false) | abstained = no grounded answer (`abstain_reason` says why).