Skip to content

Clear key-lock health after successful lease acquisition - #998

Open
zlareb1 wants to merge 20 commits into
mainfrom
fix/key-lock-health-recovery
Open

zlareb1 wants to merge 20 commits into
mainfrom
fix/key-lock-health-recovery

Conversation

@zlareb1

@zlareb1 zlareb1 commented Nov 19, 2026 •

Copy link
Copy Markdown
Member

Related: #1095
Related: #1072

Changelog category

  • Bug Fix (user-visible misbehavior in a release)

Changelog entry

Health clears a previous memory key-lock failure after a later lease acquisition succeeds. Busy locks and acquisition failures continue to refuse the protected operation.

Why

The lock records backend failures in provider health but never records successful acquisition. A recovered lock therefore leaves health degraded. Record success after receiving a lease, before entering the caller's body, so an unrelated body error cannot be counted as an acquisition failure.

Scope and risk

The server change records success for one health entry after confirmed lease acquisition. Its regression cases cover recovery, a later body error, busy acquisition, another provider's failure and lease release. Lease acquisition, refusal, release and per-process cleanup retain their existing behavior.

The remaining test-support changes compare Cell proof defaults as a single typed canonical JSON mapping and give the Cell cleanup fixture its reviewed private idle clock. Expected defaults, field-mutation assertions, numeric type rejection, cleanup bounds and cancellation remain unchanged. These controls do not establish end-to-end Cell performance.

The SCIM observation, search-deadline and trigger-deadline prerequisites are already on main and match their earlier composed source exactly. They add no remaining file delta here. The failure-only cancellation observer is inherited from current main without an edit.

Compatibility, rollout and rollback

  • Wire: no operations, fields or defaults change.
  • Schema, settings and operator steps: none.
  • Rollback: reverting restores the previous health reporting; no stored data changes.

Prior-source qualification

This composition includes the complete reviewed test prerequisites on current main. Every original feature entry is preserved; entries already supplied identically by main remain inherited support. No previous CI result or review verdict qualifies this head.

The lock/provider checks and SCIM, sparse-default and cleanup controls retain their original assertions, bounded clocks and refusal behavior. The earlier scoped commands and outcomes remain bound to their recorded heads on unchanged corresponding source. Full hosted CI and a new independent whole review remain required.

Earlier hosted GraphQL-cost and native-worker failures remain recorded and unexplained. The local controls do not establish Cell end-to-end performance. No live provider, benchmark, runtime or CI retry was run.

Tests

This composition includes main's grep response fixture and private retirement clock controls. The committed tree preserves every prior concern entry outside those three main paths, with its exact blob and mode. The three main entries match main exactly.

Current local checks pass for the retirement clock's blocked-wait control, syntax of both grep test files, scoped Ruff and formatting of the retirement unit file, public prose and whitespace in the three-path composition. Feature qualification listed above remains historical on unchanged source; those checks were not repeated unless named here. No prior whole review or hosted CI verdict qualifies this head. Full new exact-head hosted CI and a fresh independent whole review remain required before merge. Existing failures and unproven causes remain unresolved; no performance or runtime repair is claimed.

AI assistance in the code

  • AI-generated: Codex wrote the health correction and regression fixtures, then checked acquisition ordering, refusal paths, provider isolation and release after body errors.

  • I can explain every line of this diff and how it interacts with the rest of Loom.

Checklist

  • No generated interfaces, migrations, plugins or frontend files change.
  • No operator or wire changes require documentation updates.
  • Commits are signed off.
  • Full exact-head hosted CI passes.

Current-main composition preserves independent human administrator approval for proposals and the stored review-provenance field. Focused local checks on this source are recorded separately; fresh whole-source review and complete exact-head hosted CI are required. Earlier CI failures and their unproven causes remain recorded. No previous review or CI pass qualifies this composition.

Default-comparison refusal coverage includes all 44 protocol defaults, including requires_review, with the existing assertions and encoder bound retained.

@zlareb1
zlareb1 force-pushed the fix/key-lock-health-recovery branch from 08ab7b8 to 1055d3c Compare November 20, 2026 08:20
@zlareb1
zlareb1 force-pushed the fix/key-lock-health-recovery branch from 1055d3c to 30ee114 Compare November 20, 2026 12:49
@vercel

vercel Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
loom-docs Ready Ready Preview Nov 21, 2026 11:33am UTC

Request Review

Exercise each default-field refusal and JSON numeric distinction before consolidating repeated default comparisons.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit 73ef63b)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Keep every default-field and JSON type check while avoiding a separate encoder setup for each field.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit 0bfc712)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Shared client schema reads run with an independent context. Record only decision-tagged statements while retaining unbounded decision reads as failures.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit 57327d5)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Keep the blocked close, deadline observers and loop timers on one private
clock. A controlled independent clock advance reproduces the elapsed-time
dependency while the original cleanup and refusal assertions stay intact.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit 4b03db4)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
@zlareb1
zlareb1 force-pushed the fix/key-lock-health-recovery branch from 58d8f0c to 84d6257 Compare November 21, 2026 00:08
@zlareb1
zlareb1 force-pushed the fix/key-lock-health-recovery branch from 84d6257 to 58b9de6 Compare November 21, 2026 00:53
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Shared client schema reads run with an independent context. Record only decision-tagged statements while retaining unbounded decision reads as failures.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit 57327d5)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Exercise each default-field refusal and JSON numeric distinction before consolidating repeated default comparisons.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit 73ef63b)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit cb594d0)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Keep every default-field and JSON type check while avoiding a separate encoder setup for each field.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit 0bfc712)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit a94d2f9)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Keep the blocked close, deadline observers and loop timers on one private
clock. A controlled independent clock advance reproduces the elapsed-time
dependency while the original cleanup and refusal assertions stay intact.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit 4b03db4)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
The fake leg delays and request budget share a private event loop clock. Preserve the original timing inputs and assertions, and require blocked legs to time out and cancel.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit bfb9d40)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
@zlareb1
zlareb1 force-pushed the fix/key-lock-health-recovery branch from 58b9de6 to 85a3323 Compare November 21, 2026 02:31
zlareb1 added a commit that referenced this pull request Nov 21, 2026
Related: #1095

Related: #998

### Changelog category (leave one):

- CI Fix or Improvement (changelog entry is not required)

### Changelog entry

Not required.

### Why

The first-content-leg unit case expects a fake lexical read to finish after 300 ms inside a 400 ms request budget. The budget starts before setup, so unrelated CPU scheduling can consume the margin before either fake read starts. A controlled 120 ms pre-leg clock jump reproduces the assertion that both legs time out; its zero-jump neighbour passes. The original CI scheduling cause is unknown.

The case uses a private event loop whose clock advances only while idle. Fake sleeps, request budgets and elapsed-time assertions share that clock. Production deadline code remains unchanged, and this fixture does not qualify wall-clock latency.

### Scope and risk

Two existing unit modules change. The search private clock applies to the first-content-leg case and its new controls. The exact reviewed trigger-deadline fixture is also composed from #1095. It places all six stalls and all four collected controls on their function-scoped private loops, with the runtime loop-identity check intact. Its deadlines, cancellation rounds and original assertions are unchanged. Every original test body, fake delay, request deadline and assertion remains unchanged. A blocked-leg control still requires an empty partial result and cancellation of both legs. Another control drains and rejects a task left on the private loop.

Module-local facades restore their bindings on exit. The standard clock and asyncio modules remain intact. The fixture relies on the selector loop used by the unit test platform.

### Compatibility, rollout and rollback

- Wire: none.
- Operator steps: none.
- Rollback: revert the test-only change; no stored data or generated files change.

### Tests

Current composition: all 29 search-deadline cases and 10 selected trigger-deadline cases pass. Scoped Ruff, formatting, body prose and committed-range whitespace checks pass. The whole tree equals current main plus the exact prior search fixture and exact reviewed trigger fixture. Full exact-head hosted CI and a new independent whole review remain required. Final merge also requires #1095 to land on main.

The following regression receipts precede this composition; they remain evidence for the unchanged search fixture.

- Controlled regression: one failure and one passing neighbour before the clock fixture; the same two cases pass after it.
- `.venv/bin/python -m pytest tests/unit/test_code_search_deadline.py -q --override-ini=addopts=`: 29 passed.
- Ruff lint and format checks and `git diff --check` pass.
- AST comparison retains all 25 original test bodies and all 74 original assertions.
- Full hosted CI and a fresh independent whole-diff review remain required.

### AI assistance in the code

- AI-generated: Codex wrote the fixture and controls. The original test bodies, timing inputs, production source and focused results were checked.

- [x] I can explain every line of this diff and how it interacts with the rest of Loom.

### Checklist

- [ ] Full required hosted CI passes; local checks are limited to this concern.
- [x] Generated files: not applicable.
- [x] Migrations: not applicable.
- [x] Shipped plugin files and versions: not applicable.
- [x] Frontend: not applicable.
- [x] Docs and operator steps: not applicable.
- [x] Repository text contains no private operational details.
- [x] Commits are signed off.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…acquisition

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant