Add worker health monitoring with automatic restart on stall

[RUKAYAT-CODER]

Overview

src/workers/health/worker-health-check.service.ts exists but there is no evidence it triggers automatic recovery when a worker stalls (stops processing jobs without crashing). Silent worker stalls can halt notification delivery or subscription renewals indefinitely.

Specifications

Features:

• Monitor each worker's last-processed-job timestamp.
• Trigger a graceful worker restart when the timestamp exceeds a configurable stall threshold.

Tasks:

• In each worker, update a Redis key worker:heartbeat:{workerId} on every successful job.
• Create a WorkerStalledDetector scheduled task that checks heartbeats every 60 seconds.
• If heartbeat is older than WORKER_STALL_THRESHOLD_SECONDS (default 300), emit a worker.stalled event and initiate graceful restart.
• Add a Prometheus counter worker_restarts_total{worker_name}.

Impacted Files:

• src/workers/health/worker-health-check.service.ts
• All processor files.

Acceptance Criteria

• Stalled worker is restarted within 2x the stall threshold.
• Prometheus counter increments on each automatic restart.
• Test simulates a stall by freezing the heartbeat key.

[barnabasolutayo-lgtm]

@barnabasolutayo-lgtm has applied to work on this issue as part of the Stellar Wave Program's 6th wave.

Hello Maintainer. I would love to work on this issue. Please assign to me. Thank you

ℹ️ Repo Maintainers: To accept this application, review their application or assign @barnabasolutayo-lgtm to this issue.

[ykargeee-bit]

@ykargeee-bit has applied to work on this issue as part of the Stellar Wave Program's 6th wave.

Hi Maintainer,

I’m a full-stack developer, and I’d like to contribute to the task. I’ll implement it carefully and efficiently thoroughly understand requirements and edge cases
Communicate promptly about suggestions or improvements. And I will provide regular progress updates and address blockers
Align with project goals through feedback and clarification.

Thank you,

ℹ️ Repo Maintainers: To accept this application, review their application or assign @ykargeee-bit to this issue.

[CHKM001]

@CHKM001 has applied to work on this issue as part of the Stellar Wave Program's 6th wave.

Bikoo nu oga Rinaf
Give me this one as well
I can do this and deliver it, ETA within the day of assigning

ℹ️ Repo Maintainers: To accept this application, review their application or assign @CHKM001 to this issue.

[drips-wave]

Congratulations, @ykargeee-bit! 🎉 Your application was accepted by the repo's maintainers, and the issue is due on June 30, 2026.

🧑‍💻 @ykargeee-bit: Please resolve the issue such that the repo's maintainers have enough time to review your contribution before the due date. You'll earn Points for completing the issue on-time, which will make you eligible for a share of the Stellar Wave Program's reward pool.

Warning

When opening a PR, please link it to this issue to ensure it gets tracked accurately. Points are awarded when this issue is marked as completed by the maintainer.

🤠 Repo maintainers: Please keep an eye on the contributor's progress and review their work before the due date. You can manage this issue, including adjusting its complexity and points, here.

🌊 Happy Wave 🌊