Skip to content

Loom traces record invocation origin, outcome and benchmark context - #1112

Open
zlareb1 wants to merge 9 commits into
mainfrom
feat/prompt-observability-20261119
Open

zlareb1 wants to merge 9 commits into
mainfrom
feat/prompt-observability-20261119

Conversation

@zlareb1

@zlareb1 zlareb1 commented Nov 21, 2026 •

Copy link
Copy Markdown
Member

Changelog category (leave one):

  • New Feature

Changelog entry (one or two sentences a user of Loom can read without the diff; the release pull request collects these into the release notes):

Loom traces distinguish tool, automatic-hook and background-worker origins, with request outcomes, duration and optional benchmark correlation. Hook invocation headers need the next server release.

Why

Transport and session identity do not explain why Loom was called. Optional caller metadata lets evaluators connect actual invocations to a benchmark run, case and variant while leaving absent origin and usage unknown.

Scope and risk

HTTP operations and MCP tool calls receive one attempt observation, including catalog and dispatch-screen refusals. Nested retrieval and provider observations retain their own timing and measured usage. Origin metadata never grants access or replaces authenticated tenant identity. Host tool-call IDs can correlate several separate Loom requests. Combining retries requires a caller guarantee that an ID uniquely identifies one logical Loom request within its scope and stays stable across those retries. Concurrent calls keep separate context; private diagnostics retain their export boundary. Cached replays retain admitted tenant tags for trace discovery and erasure; early authenticated refusals use verified token identity with namespace unresolved.

The canonical hook transport identifies hook requests without requiring a prompt ID. Its generated mirror follows the same implementation. Both plugin manifests use 1.2.48.

Compatibility, rollout and rollback

  • Wire: optional telemetry headers and MCP _meta.loom fields; no operation arguments, stored identity or authorization defaults change.
  • Operator steps: none. Existing tracing configuration, sampling, filters and content-capture policy apply. Parent context, conversation/turn/tool IDs and benchmark IDs are optional.
  • Rollback: revert the instrumentation and plugin changes. No migration, backfill or stored-row rewrite is required. Previously exported observations follow the tracing backend's retention and erasure policy.

Tests

  • Current-head focused transport, publication, sender-provenance and public-copy checks: 106 cases pass with .venv/bin/python -m pytest -q tests/unit/test_prompt_context.py tests/unit/test_mcp_malformed_requests.py tests/unit/test_mirror_publication_batch.py tests/unit/test_public_copy.py tests/unit/test_session_content_senders.py.
  • Two unit shards exposed fixtures patching an inactive convenience wrapper after dispatch instrumentation. The four affected cases failed locally, then passed after targeting the core dispatcher. Batch limits, pre-send proof failures and notification proof disposal assertions are unchanged; no production behavior changed in that correction.
  • The sender-provenance guard also detected the two changed client module digests. An independent review checked the bounded, redacted metadata headers and unchanged payload builders before renewing exactly those two pins. All sender exceptions, classifications and negative controls from current main are preserved; this PR changes only the two reviewed client digests in that manifest. The exact failing case was reproduced locally before the correction.
  • A prior composed head passed all 65 hosted checks (63 successful, two skipped) across seven workflows, including native Cell acceptance and the installer interruption test. The preceding run failed the existing Vue macro-member scaling case: a measured CPU ratio of 8.760606 exceeded the unchanged limit of 8. Its test, timing helper and parser source match the base and previous green head. That run subsequently passed native Cell acceptance and cleanup; the overall result remains failed. The cause remains unproven. Earlier PROJECTION_BEHIND and installer failures also remain recorded; later success does not establish their causes or prove a flake. Full CI must clear before merge.
  • After composing current main, the focused transport, provenance and public-copy group passed 106 cases; privacy, tenant identity, auth, replay and admission checks passed 51 cases; transport, package and plugin contracts passed 23 cases; and the four relevant Node test files passed 36 cases. Strict typing of the metadata module, generated plugin mirror consistency, function length, changed-file lint/format and whitespace checks passed. The executable instrumentation and plugin files match the preceding reviewed version. The sender manifest preserves current main and the two reviewed client digests; upstream changes are preserved.
  • On the composed source commit, ten synthetic nonmodel requests through the supported local MCP HTTP endpoint and canonical hook client were exported and read back with the tracing CLI. Exact attempt IDs, origins, operation, outcome, timing, benchmark fields, supplied correlation IDs, distinct attempt IDs and a supplied parent were reconciled. Empty usage and null cost remain unknown; usage stays on provider observations. The exact source hashes and commit are checked again at publication.
  • Fresh whole-change review and full hosted CI are required on every pushed head. Broad local suites were not run.

AI assistance in the code (leave one; writing this description with AI is fine):

  • AI-generated (an agent wrote most of the diff): Codex. Focused regressions, privacy/auth behavior, generated mirrors and actual trace readback were checked.

  • I can explain every line of this diff and how it interacts with the rest of Loom.

Checklist

  • Local format, lint, type checks and focused tests pass. The full make test-unit and applicable integration results are reported by CI; a full local run is not required before pushing. Full CI remains required.
  • Generated plugin files are regenerated through the normal generator. No SDK wire models, reference tables or schema changed.
  • No migration is added or edited.
  • Both plugin versions are bumped; no other checked open PR uses 1.2.48. Recheck before publication.
  • No frontend source changes.
  • Documentation describes the invocation metadata, coverage and accounting limits.
  • Repository text contains no deployment-specific details.
  • Commits are signed off.

@vercel

vercel Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
loom-docs Ready Ready Preview Nov 21, 2026 7:15pm UTC

Request Review

@zlareb1
zlareb1 force-pushed the feat/prompt-observability-20261119 branch 3 times, most recently from 6129bf9 to b9ef5b2 Compare November 21, 2026 16:54
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Share operation name resolution with routing so refused aliases retain their target, and classify unsupported per-call protocol versions as refusals.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Use the routing help aliases for invocation attribution and retain a bounded generic label for unknown or malformed loom_op requests.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Stamp resolved identity before cache replay so tenant trace discovery and erasure find each attempt. Early authenticated refusals carry verified token identity without trusting the requested namespace.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
The HTTP observer calls the core dispatcher directly. Keep batch admission and pre-send proof assertions on the executed path.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Admitted replay spans carry the authorized tenant identity. Describe that order in the dispatcher documentation.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
The transport adds bounded, redacted invocation headers without changing request bodies or admission. Renew only the two reviewed client module digests.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
One host tool can trigger several Loom requests. Logical request counting needs a caller guarantee of unique IDs that persist across retries.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
@zlareb1
zlareb1 force-pushed the feat/prompt-observability-20261119 branch from b9ef5b2 to 6e04d61 Compare November 21, 2026 19:13

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant