Skip to content

Fix deadlock after partial health-worker startup failure - #694

Merged
matteius merged 1 commit into
mainfrom
fix/health-worker-startup-failure
Oct 31, 2026
Merged

matteius merged 1 commit into
mainfrom
fix/health-worker-startup-failure

Conversation

@matteius

Copy link
Copy Markdown
Contributor

A later pthread_create failure could hang health startup: cleanup joined an already-created worker while holding the lifecycle mutex that the worker needed to enter collection.

Keep workers at a startup gate until all four threads exist. Partial failure clears the running flag and wakes/joins gated workers without releasing lifecycle ownership. The initialized runtime remains retryable.

  • Scope: one production file, one regression test, existing CMake registration and native CI selection (four files total).
  • Tests exercise actual threads, every creation-failure position, retry, and concurrent start/shutdown. Bounded child processes contain failures; no hardware collectors or application server run.
  • Verified independently from main: the new suite reproduces five failures before the fix; afterward all nine new cases plus four existing type cases pass under ASan/UBSan. Actionlint passes.

Test selection: ctest --test-dir build/test --output-on-failure --no-tests=error -R '^test_system_health_(startup|types)$'.

Extracted from #693. This PR has no dependency on its foundation tooling, plans or generated artifacts.

Keep workers at a startup gate until all threads exist. On a later pthread_create failure, gated workers can exit and be joined without waiting for the lifecycle lock held by startup. Preserve same-runtime retries and concurrent start/shutdown ownership.

Nine real-thread fault/ownership cases plus four existing type controls pass under ASan/UBSan on main. The new suite reproduced five failures before the fix. Register the regression in the existing native CI job.
@codecov

codecov Bot commented Oct 31, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@matteius
matteius merged commit 8cb1859 into main Oct 31, 2026
13 checks passed
@matteius
matteius deleted the fix/health-worker-startup-failure branch October 31, 2026 14:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant