Skip to content

Report sample and ingestion outcomes when fixture checks fail - #1088

Merged
zlareb1 merged 13 commits into
mainfrom
test/linear-cost-sample-context
Nov 21, 2026
Merged

zlareb1 merged 13 commits into
mainfrom
test/linear-cost-sample-context

Conversation

@zlareb1

@zlareb1 zlareb1 commented Nov 20, 2026 •

Copy link
Copy Markdown
Member

Related: #1090
Related: #1096

Changelog category

  • CI Fix or Improvement (changelog entry is not required)

Changelog entry

Not required.

Why

Cost assertions need the process state around each original sample as well as the selected minima. Ingestion fixture failures need the worker attempt's outcome when a dataset read returns the wrong number of datasets.

Record numeric process, wall and thread clocks, allocation counts, GC state, and tracing/profiling state around the original cost calls. Scalar checks, registered lanes, the independent lane-walk child and the SQL ratio check retain their sample values and order. Failure output includes that context.

The workbook fixture's two-dataset assertion reports the actual worker return and observed dataset count. The metadata-edit fixture reports the same fields when its one-dataset assertion fails. A worker return describes that attempt; it does not establish persisted state at the later dataset read. The retrieval-floor fixture reads fixed persisted status fields only when its original readiness assertion fails.

Scope and risk

Only test code changes. The hook-contract prerequisite waits for its owned detached flusher before removing the sandbox, using the existing isolated-home and child-environment helpers. Its hook deadlines and assertions stay unchanged.

Sample observation runs outside the original timed calls. Boundary deltas include observer work and collection or tracer restoration. Observation can perturb later samples; its fields are diagnostic context, not a cost measure or acceptance criterion. Input text and exception messages are omitted. Context errors record their class without changing the judge.

The lane-walk child emits its original two CPU minima and all six ordered samples. Input sizes, operations, call order, sample counts, minima, warmup, subprocess deadlines, comparison, floor and retry policy stay unchanged. The causes of the Python, Lua, SQL and setup.cfg cost failures remain unknown. This change claims no parser performance fix.

Dataset diagnostics use the existing worker return and dataset response, with no additional I/O, connection, status read, worker attempt or retry. The workbook still requires exactly two datasets, then runs its original unpacking and every sheet, count, rejected-row and served-row assertion. The metadata-edit fixture retains its exact cardinality, metadata, deletion and served-snapshot checks. Only fixed worker-state strings and dataset counts enter the added messages.

The retrieval diagnostic performs one same-scope status lookup with a one-second timeout. It emits whitelisted state, stage and error-code values plus a bounded integer attempt count. Missing or unavailable observations use fixed labels. Stored error messages, source content, metadata and identifiers are excluded from the added messages.

The preserved workbook failure reaches an empty dataset result after successful upload, ingest, extraction and dataset-read HTTP responses. Those responses do not establish the ingestion cause. This change adds failure context and changes no production ingestion behavior.

The retrieval fixture also observes the existing inline retry helper through one test-scoped dispatcher. Per-attempt context matches the trusted scope and originating task, delegates the same objects and awaitable exactly once, and resets before cleanup. Failure messages contain closed exception labels and bounded native numeric codes through at most four cause links. No new I/O, worker attempt, retry or deadline change is introduced. Original ingestion causation remains unknown.

Compatibility, rollout and rollback

  • Wire changes and operator steps: none.
  • Rollback: revert the test diagnostics and fixture prerequisites.

Tests

The current-main composition includes the reviewed scanner action pin from #960. Its workflow file matches main exactly; every preceding concern, generated entry and unchanged support remains byte-identical. Current scoped commands below are new-head results. Earlier CI results and review verdicts do not transfer.

On this current-main composition, all 64 directly related ingestion-observer controls pass. Scoped Ruff, formatting, public prose and committed-range whitespace pass. Three canonical diagnostic files match the independently reviewed ingestion-context source exactly.

Targeted parser-context controls also pass on this head. The prior 47-case aggregate and six Node controls remain retained receipts, not repeated aggregates. Seven cost and Node concern entries remain exact; the shared lane helper combines the unchanged diagnostic sampler nodes with current main's Thrift selectors and samples. A whole-module AST proof verifies that exact composition. The full tree equals current main plus the retained cost diagnostics and three canonical ingestion files. The CPU sampler, judges, floors, thresholds and workloads remain unchanged. Earlier failure-only sample-context and Node deadline controls retain their original evidence; no cause or performance repair is inferred from them.

The retrieval controls preserve strict readiness, scope/task isolation, delegate identity, cancellation, bounded numeric codes, privacy, the ready path and cleanup. Dataset and workbook assertions remain strict. Original hosted ingestion and cost failures are not rerun locally; their causes remain unknown. Earlier CI and review verdicts do not transfer. Fresh independent entire review and full new exact-head hosted CI remain required.

AI assistance in the code

  • AI-generated: Codex wrote the diagnostics and checked the unchanged assertion and timing contracts.

  • I can explain every line of this diff and how it interacts with the rest of Loom.

Checklist

  • Focused qualification of the final candidate is complete.
  • Full required CI passes on the eventual public head.
  • A fresh independent whole review is complete on the eventual public head.
  • Commits are signed off.

@vercel

vercel Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
loom-docs Ready Ready Preview Nov 21, 2026 7:45am UTC

Request Review

@zlareb1
zlareb1 force-pushed the test/linear-cost-sample-context branch 4 times, most recently from be775cb to e12f192 Compare November 21, 2026 02:14
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Keep each test sandbox until its detached flusher removes its final job marker. Preserve hook deadlines and verify unfinished jobs refuse cleanup.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit 2e63764)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit 334a924)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
@zlareb1 zlareb1 changed the title Linear cost failures include each sample’s process context Report sample and ingestion outcomes when fixture checks fail Nov 21, 2026
@zlareb1
zlareb1 force-pushed the test/linear-cost-sample-context branch from e12f192 to e4838a6 Compare November 21, 2026 04:38
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>

# Conflicts:
#	tests/integration/test_knowledge_retrieval_floors.py
#	tests/unit/test_knowledge_retrieval_failure_context.py
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
@zlareb1
zlareb1 merged commit 5b63495 into main Nov 21, 2026
57 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant