Skip to content

Drive trigger cleanup checks with one private clock - #1095

Merged
zlareb1 merged 3 commits into
mainfrom
fix/cell-attach-trigger-deadline-clock
Nov 21, 2026
Merged

zlareb1 merged 3 commits into
mainfrom
fix/cell-attach-trigger-deadline-clock

Conversation

@zlareb1

@zlareb1 zlareb1 commented Nov 21, 2026 •

Copy link
Copy Markdown
Member

Related: #1094

Changelog category (leave one):

  • CI Fix or Improvement (changelog entry is not required)

Changelog entry

Not required.

Why

The trigger-try unit fixture gives work a short event-loop timeout and freezes its cleanup clock. Its outer watchdog also uses the event-loop clock. Scheduling can consume the watchdog before the work timeout and cleanup finish their separate turns. A controlled 250 ms loop-clock advance reproduces the reset case's stuck-task assertion; its zero-jump neighbour passes. The historical CI scheduling cause remains unknown.

The six stall cases explicitly request the same function-scoped loop as their clock fixture. A runtime identity assertion checks that the test and fixture use that loop before setting phase clocks. The cases share a private idle clock between the event loop, owner phase and Postgres close. Work and cleanup still run their real cancellation code. The watchdog, phase bound and all original cancellation rounds remain unchanged. This fixture does not qualify real wall-clock latency.

Scope and risk

One existing unit module changes. Every original test body, all six stall parameters, the 200 ms watchdog and three 10 ms cancellation rounds remain unchanged. The clock fixture activates only for this parameterized case and its new controls.

A genuinely blocked public close must still time out, terminate its fake connection and cancel its pending close task. An orphan-task control requires the fixture to cancel, drain and reject newly created tasks on its private function-scoped loop. Scoped patches restore all clock and helper bindings. The fixture relies on the selector loop used by the unit test platform.

Compatibility, rollout and rollback

  • Wire: none.
  • Operator steps: none.
  • Rollback: revert the test-only change; no stored data or generated files change.

Prior-source regression evidence

  • Controlled scheduling regression: one failure and one passing neighbour before the fixture; both controls pass with the fixture.
  • Collected loop-identity regression: all six original parameter cases fail before the explicit function-scoped test marker; the identical command passes all six after it.
  • .venv/bin/python -m pytest tests/unit/test_cell_attach.py -q -k 'a_trigger_try_deadline_covers_transaction_and_lock_cleanup or trigger_try_clock' --override-ini=addopts=: 10 passed, including all six original stall cases.
  • Ruff lint and format checks, public body and added fixture text prose checks, and git diff --check pass.
  • AST comparison preserves all 52 original test bodies and all 119 original assertions.
  • Full hosted CI and a fresh independent whole-diff review remain required.

Tests

All three new clock-control functions explicitly request function-scoped loops, matching the six collected original stalls. A separate temporary collection check rejects the four collected control cases before the marker correction, then accepts all ten selected cases afterward. The same ten cases pass execution. Every published and original test body, all original assertions, phase and watchdog deadlines remain unchanged. Full-module prose remains failed on the same three baseline literals; added text and this body pass their scoped prose checks.

On this current-main composition, all ten directly related original stall and clock control cases pass. Scoped Ruff lint and formatting, added-text prose, public-body prose and committed-range whitespace checks pass. All 55 published test bodies retain identical ASTs and the reviewed loop marker and identity guard match their exact source commit. The full-module prose checker has three unchanged baseline test literals; the preserved baseline equivalence receipt records those failures. Full hosted CI and a fresh independent whole review remain required.

AI assistance in the code

  • AI-generated: Codex wrote the fixture and controls. The original test bodies, deadlines, production source and focused results were checked.

  • I can explain every line of this diff and how it interacts with the rest of Loom.

Checklist

  • Full required hosted CI passes; local checks are limited to this concern.
  • Generated files: not applicable.
  • Migrations: not applicable.
  • Shipped plugin files and versions: not applicable.
  • Frontend: not applicable.
  • Docs and operator steps: not applicable.
  • Repository text contains no private operational details.
  • Commits are signed off.

@vercel

vercel Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
loom-docs Ready Ready Preview Nov 21, 2026 1:50am UTC

Request Review

@zlareb1
zlareb1 force-pushed the fix/cell-attach-trigger-deadline-clock branch from f6ab8fd to 559c141 Compare November 21, 2026 01:32
The work watchdog and cleanup clocks share one function-scoped event loop. Preserve all original stall cases and bounds, and require a genuinely blocked close to time out and cancel.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
The test marker uses the same function-scoped loop as the idle clock fixture. Assert their runtime loop identity before setting phase clocks so scope drift fails each collected stall case.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit 9f7988578ecc0654720996156bff919ae33199e5)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
@zlareb1
zlareb1 force-pushed the fix/cell-attach-trigger-deadline-clock branch from 559c141 to 8c5c767 Compare November 21, 2026 01:47
@zlareb1
zlareb1 merged commit 0976998 into main Nov 21, 2026
49 checks passed
zlareb1 added a commit that referenced this pull request Nov 21, 2026
Related: #1095

Related: #998

### Changelog category (leave one):

- CI Fix or Improvement (changelog entry is not required)

### Changelog entry

Not required.

### Why

The first-content-leg unit case expects a fake lexical read to finish after 300 ms inside a 400 ms request budget. The budget starts before setup, so unrelated CPU scheduling can consume the margin before either fake read starts. A controlled 120 ms pre-leg clock jump reproduces the assertion that both legs time out; its zero-jump neighbour passes. The original CI scheduling cause is unknown.

The case uses a private event loop whose clock advances only while idle. Fake sleeps, request budgets and elapsed-time assertions share that clock. Production deadline code remains unchanged, and this fixture does not qualify wall-clock latency.

### Scope and risk

Two existing unit modules change. The search private clock applies to the first-content-leg case and its new controls. The exact reviewed trigger-deadline fixture is also composed from #1095. It places all six stalls and all four collected controls on their function-scoped private loops, with the runtime loop-identity check intact. Its deadlines, cancellation rounds and original assertions are unchanged. Every original test body, fake delay, request deadline and assertion remains unchanged. A blocked-leg control still requires an empty partial result and cancellation of both legs. Another control drains and rejects a task left on the private loop.

Module-local facades restore their bindings on exit. The standard clock and asyncio modules remain intact. The fixture relies on the selector loop used by the unit test platform.

### Compatibility, rollout and rollback

- Wire: none.
- Operator steps: none.
- Rollback: revert the test-only change; no stored data or generated files change.

### Tests

Current composition: all 29 search-deadline cases and 10 selected trigger-deadline cases pass. Scoped Ruff, formatting, body prose and committed-range whitespace checks pass. The whole tree equals current main plus the exact prior search fixture and exact reviewed trigger fixture. Full exact-head hosted CI and a new independent whole review remain required. Final merge also requires #1095 to land on main.

The following regression receipts precede this composition; they remain evidence for the unchanged search fixture.

- Controlled regression: one failure and one passing neighbour before the clock fixture; the same two cases pass after it.
- `.venv/bin/python -m pytest tests/unit/test_code_search_deadline.py -q --override-ini=addopts=`: 29 passed.
- Ruff lint and format checks and `git diff --check` pass.
- AST comparison retains all 25 original test bodies and all 74 original assertions.
- Full hosted CI and a fresh independent whole-diff review remain required.

### AI assistance in the code

- AI-generated: Codex wrote the fixture and controls. The original test bodies, timing inputs, production source and focused results were checked.

- [x] I can explain every line of this diff and how it interacts with the rest of Loom.

### Checklist

- [ ] Full required hosted CI passes; local checks are limited to this concern.
- [x] Generated files: not applicable.
- [x] Migrations: not applicable.
- [x] Shipped plugin files and versions: not applicable.
- [x] Frontend: not applicable.
- [x] Docs and operator steps: not applicable.
- [x] Repository text contains no private operational details.
- [x] Commits are signed off.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant