Skip to content

Recheck renewed supervisor leases after a wait expires - #1002

Open
zlareb1 wants to merge 15 commits into
mainfrom
fix/supervisor-renewal-timeout-race
Open

zlareb1 wants to merge 15 commits into
mainfrom
fix/supervisor-renewal-timeout-race

Conversation

@zlareb1

@zlareb1 zlareb1 commented Nov 19, 2026 •

Copy link
Copy Markdown
Member

Related: #1041
Related: #1072
Related: #1077
Related: #1087

Changelog category

  • Bug Fix (user-visible misbehavior in a release)

Changelog entry

The worker supervisor rechecks lease freshness after a wait times out, so a renewal that completed during the wait keeps its workers running.

Why

A renewal can complete while the wait for the previous deadline resumes. Recheck the lease's current freshness after an empty wait before deciding to stop workers.

Scope and risk

The supervisor continues waiting when the lease is fresh. Missing and expired leases keep the existing stop path. Regression controls cover a renewal during the wait and the missing and expired cases. Lease duration, renewal cadence, cancellation and stop behavior are retained.

The qualification composition includes related Cell proof encoder setup, SCIM query-recording controls, replicated-erasure fixture regressions and cleanup-clock controls. The erasure fixture already on main remains unchanged. All original cleanup deadlines and assertions are retained. Cell workload, proof pages, resource census, global deadlines and latency and throughput bounds remain unchanged. Complete acceptance, latency and throughput require hosted qualification on this composition.

Compatibility, rollout and rollback

  • Wire, schema, settings, defaults and operator steps: none.
  • Rollback: revert the freshness recheck and qualification changes. No persisted data changes.

Prior-source qualification

The prior feature command records name their recorded heads but lack per-run immutable working-source hashes. Their outcomes do not independently establish the executed feature bytes. Committed-file comparisons establish source equality only.

Historical command records report that the supervisor renewal, canonical default comparison, person-query observation, erasure-fixture and private cleanup controls passed. A historical strict-typing command checks the worker module. The original expiry and cancellation assertions remain unchanged. The earlier Cell latency failure remains a failed acceptance result with an unproven cause; these source checks do not establish an end-to-end performance fix.

Current main adds stored review provenance to the proof protocol. This composition retains its 62 memory fields, 44 sparse defaults and stored-provenance requirement, and extends the default-field refusal coverage to the new field. Other original source entries retain their exact modes and bytes. Current main supplies the person-query observations and remaining merged prerequisites. Earlier records include scoped Ruff, formatting, public prose and committed-range whitespace results. Their commands and scoped outcomes remain historical. No previous review or CI result qualifies this head; a fresh independent entire review and all required full exact-head hosted workflows must pass before merge. No runtime, provider call, performance probe or hosted retry was run.

Tests

This composition includes main's grep response fixture and private retirement clock controls. The committed tree preserves every prior concern entry outside those three main paths, with its exact blob and mode. The three main entries match main exactly.

Current local checks pass for the retirement clock's blocked-wait control, syntax of both grep test files, scoped Ruff and formatting of the retirement unit file, public prose and whitespace in the three-path composition. Feature qualification listed above remains historical. Committed-source equality is separately verified; those checks were not repeated unless named here. No prior whole review or hosted CI verdict qualifies this head. Full new exact-head hosted CI and a fresh independent whole review remain required before merge. Existing failures and unproven causes remain unresolved; no performance or runtime repair is claimed.

AI assistance in the code

  • AI-generated: Codex wrote the recheck and controls, then checked fresh, missing and expired leases, cancellation and unchanged failure bounds.

  • I can explain every line of this diff and how it interacts with the rest of Loom.

Checklist

  • Existing lease, workload and deadline bounds are retained.
  • No generated interfaces, migrations or operator settings change.
  • Commits are signed off.
  • Full exact-head GitHub CI passes.

@zlareb1
zlareb1 force-pushed the fix/supervisor-renewal-timeout-race branch 2 times, most recently from 0184f15 to 1f01367 Compare November 20, 2026 20:36
@vercel

vercel Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
loom-docs Ready Ready Preview Nov 21, 2026 11:33am UTC

Request Review

@zlareb1
zlareb1 force-pushed the fix/supervisor-renewal-timeout-race branch from 1f01367 to 05a5528 Compare November 20, 2026 23:07
Exercise each default-field refusal and JSON numeric distinction before consolidating repeated default comparisons.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit 73ef63b)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Keep every default-field and JSON type check while avoiding a separate encoder setup for each field.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit 0bfc712)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Shared client schema reads run with an independent context. Record only decision-tagged statements while retaining unbounded decision reads as failures.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit 57327d5)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Keep the blocked close, deadline observers and loop timers on one private
clock. A controlled independent clock advance reproduces the elapsed-time
dependency while the original cleanup and refusal assertions stay intact.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit 4b03db4)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
@zlareb1
zlareb1 force-pushed the fix/supervisor-renewal-timeout-race branch 2 times, most recently from 2bd301c to 5531d47 Compare November 21, 2026 02:06
A confirmed renewal can extend the deadline while the previous wait resumes. Use the current freshness before stopping workers, and retain genuine expiry controls.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Exercise each default-field refusal and JSON numeric distinction before consolidating repeated default comparisons.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit 73ef63b)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Keep every default-field and JSON type check while avoiding a separate encoder setup for each field.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit 0bfc712)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Shared client schema reads run with an independent context. Record only decision-tagged statements while retaining unbounded decision reads as failures.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit 57327d5)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Keep the blocked close, deadline observers and loop timers on one private
clock. A controlled independent clock advance reproduces the elapsed-time
dependency while the original cleanup and refusal assertions stay intact.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit 4b03db4)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
@zlareb1
zlareb1 force-pushed the fix/supervisor-renewal-timeout-race branch from 5531d47 to 28bd5de Compare November 21, 2026 03:14
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant