Skip to content

Report ingestion outcomes when fixture setup fails - #1096

Open
zlareb1 wants to merge 3 commits into
mainfrom
test/ingestion-failure-context
Open

zlareb1 wants to merge 3 commits into
mainfrom
test/ingestion-failure-context

Conversation

@zlareb1

@zlareb1 zlareb1 commented Nov 21, 2026 •

Copy link
Copy Markdown
Member

Related: #1071
Related: #1087

Changelog category

  • CI Fix or Improvement (changelog entry is not required)

Changelog entry

Not required.

Why

Fixture failures need to distinguish an ingestion retry from a dataset read that follows a successful worker return. Persisted pending/embed/retrying fields omit the caught exception class and native error code. The retained hosted failures do not establish an ingestion cause.

The retrieval-floor fixture reports fixed persisted status fields and the caught retry class and numeric native code when available. Dataset fixtures retain the existing worker return and report it when their one-dataset or two-sheet workbook expectation fails. A worker return describes that attempt; it does not establish the state at the later dataset read.

Scope and risk

Three test files change. A function-scoped pytest fixture installs one dispatcher around the existing retry helper and restores it once at teardown. Per-attempt context storage surrounds only the existing inline worker call and resets in finally. Exact org, namespace, ingestion-id and task comparisons keep overlapping helper calls separate, including calls with the same tuple.

The dispatcher returns the saved delegate's identical awaitable, passing the same job and exception objects exactly once. It adds no await, I/O, connection, status lookup, worker attempt or retry. The original readiness assertion and one-second failure-status lookup remain. An explicit string message keeps pytest from truncating the fixed fields.

Caught classes use closed labels. Native codes come only from exact known driver types, through at most four cause links with cycle detection. Codes must be strict integers from zero through one million. A missing code may use the existing parser on at most 4096 characters of an already plain string argument. Unknown or unavailable observations use fixed labels. No arbitrary exception formatting, stored error message, source content, SQL, URL or identifier enters the added fields.

Dataset messages contain only the dataset count and a whitelisted worker state. The workbook still requires exactly two datasets before its original unpack. All sheet, row, metadata and deletion assertions remain strict. The shared worker helper retains its non-None assertion and connection cleanup.

No production source, ingestion retry, readiness exception, retrieval floor, workload, query order, deadline or CI gate changes. The ingestion cause remains unknown.

Compatibility, rollout and rollback

  • Wire and operator steps: none.
  • Rollback: revert the test-only diagnostics.

Tests

LOOM_ENV_FILE= .venv/bin/python -m pytest tests/unit/test_knowledge_retrieval_failure_context.py -q

The command passes 64 direct cases. The retained status, dataset and worker-helper controls remain strict. Added controls cover caught classes, native numeric bounds, bounded strings and causes, cycles, no read beyond four causes, unsafe objects, observation failure, scope/task isolation, overlapping calls, delegate identity, cancellation, fixture restoration and the ready path.

Before the observer implementation, two controls fail because the original assertion omits the caught class. The final qualification records the focused result plus scoped Ruff, formatting, prose and whitespace checks. Workbook integration and the original hosted failures are not rerun locally; the workbook extension retains its exact approved source proof.

All older hosted CI and reviews apply to earlier heads. Full exact-head hosted CI and a fresh independent entire review remain required. This change collects evidence and claims no ingestion root-cause fix.

AI assistance in the code

  • AI-generated: Codex wrote the fixture diagnostics and controlled regressions.

  • I can explain every line of this diff and how it interacts with the rest of Loom.

Checklist

  • Commits are signed off.
  • No generated interface, migration, setting or operator step changes.
  • Full exact-head hosted CI passes.
  • Fresh independent entire review has no unresolved findings or nits.

@vercel

vercel Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
loom-docs Ready Ready Preview Nov 21, 2026 5:32am UTC

Request Review

@zlareb1 zlareb1 changed the title Report bounded status after fixture ingestion fails Report ingestion outcomes when fixture setup fails Nov 21, 2026
@zlareb1
zlareb1 force-pushed the test/ingestion-failure-context branch 2 times, most recently from 45bc622 to 2dc04e7 Compare November 21, 2026 04:00
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
@zlareb1
zlareb1 force-pushed the test/ingestion-failure-context branch from 2dc04e7 to c0f1d98 Compare November 21, 2026 05:30
zlareb1 added a commit that referenced this pull request Nov 21, 2026
Related: #1090
Related: #1096

### Changelog category

- CI Fix or Improvement (changelog entry is not required)

### Changelog entry

Not required.

### Why

Cost assertions need the process state around each original sample as well as the selected minima. Ingestion fixture failures need the worker attempt's outcome when a dataset read returns the wrong number of datasets.

Record numeric process, wall and thread clocks, allocation counts, GC state, and tracing/profiling state around the original cost calls. Scalar checks, registered lanes, the independent lane-walk child and the SQL ratio check retain their sample values and order. Failure output includes that context.

The workbook fixture's two-dataset assertion reports the actual worker return and observed dataset count. The metadata-edit fixture reports the same fields when its one-dataset assertion fails. A worker return describes that attempt; it does not establish persisted state at the later dataset read. The retrieval-floor fixture reads fixed persisted status fields only when its original readiness assertion fails.

### Scope and risk

Only test code changes. The hook-contract prerequisite waits for its owned detached flusher before removing the sandbox, using the existing isolated-home and child-environment helpers. Its hook deadlines and assertions stay unchanged.

Sample observation runs outside the original timed calls. Boundary deltas include observer work and collection or tracer restoration. Observation can perturb later samples; its fields are diagnostic context, not a cost measure or acceptance criterion. Input text and exception messages are omitted. Context errors record their class without changing the judge.

The lane-walk child emits its original two CPU minima and all six ordered samples. Input sizes, operations, call order, sample counts, minima, warmup, subprocess deadlines, comparison, floor and retry policy stay unchanged. The causes of the Python, Lua, SQL and setup.cfg cost failures remain unknown. This change claims no parser performance fix.

Dataset diagnostics use the existing worker return and dataset response, with no additional I/O, connection, status read, worker attempt or retry. The workbook still requires exactly two datasets, then runs its original unpacking and every sheet, count, rejected-row and served-row assertion. The metadata-edit fixture retains its exact cardinality, metadata, deletion and served-snapshot checks. Only fixed worker-state strings and dataset counts enter the added messages.

The retrieval diagnostic performs one same-scope status lookup with a one-second timeout. It emits whitelisted state, stage and error-code values plus a bounded integer attempt count. Missing or unavailable observations use fixed labels. Stored error messages, source content, metadata and identifiers are excluded from the added messages.

The preserved workbook failure reaches an empty dataset result after successful upload, ingest, extraction and dataset-read HTTP responses. Those responses do not establish the ingestion cause. This change adds failure context and changes no production ingestion behavior.

The retrieval fixture also observes the existing inline retry helper through one test-scoped dispatcher. Per-attempt context matches the trusted scope and originating task, delegates the same objects and awaitable exactly once, and resets before cleanup. Failure messages contain closed exception labels and bounded native numeric codes through at most four cause links. No new I/O, worker attempt, retry or deadline change is introduced. Original ingestion causation remains unknown.

### Compatibility, rollout and rollback

- Wire changes and operator steps: none.
- Rollback: revert the test diagnostics and fixture prerequisites.

### Tests

The current-main composition includes the reviewed scanner action pin from #960. Its workflow file matches main exactly; every preceding concern, generated entry and unchanged support remains byte-identical. Current scoped commands below are new-head results. Earlier CI results and review verdicts do not transfer.

On this current-main composition, all 64 directly related ingestion-observer controls pass. Scoped Ruff, formatting, public prose and committed-range whitespace pass. Three canonical diagnostic files match the independently reviewed ingestion-context source exactly.

Targeted parser-context controls also pass on this head. The prior 47-case aggregate and six Node controls remain retained receipts, not repeated aggregates. Seven cost and Node concern entries remain exact; the shared lane helper combines the unchanged diagnostic sampler nodes with current main's Thrift selectors and samples. A whole-module AST proof verifies that exact composition. The full tree equals current main plus the retained cost diagnostics and three canonical ingestion files. The CPU sampler, judges, floors, thresholds and workloads remain unchanged. Earlier failure-only sample-context and Node deadline controls retain their original evidence; no cause or performance repair is inferred from them.

The retrieval controls preserve strict readiness, scope/task isolation, delegate identity, cancellation, bounded numeric codes, privacy, the ready path and cleanup. Dataset and workbook assertions remain strict. Original hosted ingestion and cost failures are not rerun locally; their causes remain unknown. Earlier CI and review verdicts do not transfer. Fresh independent entire review and full new exact-head hosted CI remain required.

### AI assistance in the code

- AI-generated: Codex wrote the diagnostics and checked the unchanged assertion and timing contracts.

- [x] I can explain every line of this diff and how it interacts with the rest of Loom.

### Checklist

- [x] Focused qualification of the final candidate is complete.
- [ ] Full required CI passes on the eventual public head.
- [ ] A fresh independent whole review is complete on the eventual public head.
- [x] Commits are signed off.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant