Skip to content

feat: expose email_queue_depth as Prometheus metric with Grafana panel and alert#1014

Open
euniceamoni wants to merge 2 commits into
solutions-plug:mainfrom
euniceamoni:feat/email-queue-depth-metric
Open

feat: expose email_queue_depth as Prometheus metric with Grafana panel and alert#1014
euniceamoni wants to merge 2 commits into
solutions-plug:mainfrom
euniceamoni:feat/email-queue-depth-metric

Conversation

@euniceamoni

Copy link
Copy Markdown
Contributor

Summary

  • Add email_queue_depth Prometheus gauge to track current jobs in the main Redis queue
  • Emit the metric in the email worker loop after each dequeue cycle
  • Also update on each /api/v1/email/queue/stats request
  • Add Grafana panel for queue depth visualization
  • Add Prometheus alert firing when depth exceeds 100 for more than 5 minutes

Changes

  • services/api/src/metrics.rs — added gauge, registration, and setter
  • services/api/src/email/queue.rs — added get_queue_depth() and worker loop emission
  • services/api/src/main.rs — pass metrics into the worker
  • services/api/src/handlers.rs — update gauge on stats endpoint
  • performance/config/grafana-dashboard.json — new panel
  • performance/config/alerts.yaml — new alert rule

Resolves #961

…lug#960)

- Move ecs_tasks SG to root module to avoid circular dependencies
- ALB SG: restrict egress to container port → ecs_tasks SG only
- ecs_tasks SG: egress limited to 5432 (RDS), 6379 (Redis), 443 (AWS APIs)
- RDS SG: replace 10.0.0.0/8 ingress with ecs_tasks SG reference, remove broad egress
- Redis SG: replace 10.0.0.0/8 ingress with ecs_tasks SG reference, remove broad egress
- Add Checkov CI scan job that fails on HIGH/CRITICAL findings before terraform plan
…l and alert

- Add email_queue_depth IntGauge to Metrics struct
- Emit the gauge in the email worker loop after each dequeue cycle
- Also update the gauge on each /api/v1/email/queue/stats request
- Add Grafana panel for queue depth visualization
- Add Prometheus alert when queue depth exceeds 100 for 5 minutes

Resolves solutions-plug#961
@drips-wave

drips-wave Bot commented Jun 27, 2026

Copy link
Copy Markdown

@euniceamoni Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Email queue depth not exposed as Prometheus metric

1 participant