CCM-14044 Adding eventpub anom alarms config

Author: aidenvaines-cgi

infrastructure/terraform/modules/eventpub/README.md

@@ -1,3 +1,44 @@
+# EventPub Module
+
+## Overview
+
+The `eventpub` module provides a centralized event publishing infrastructure for NHS Notify bounded contexts. It creates an SNS topic with configurable subscribers (Lambda, Firehose, SQS) and includes comprehensive monitoring via CloudWatch alarms.
+
+```
+┌─────────────────┐
+│ Service Lambda  │
+│   (Publisher)   │
+└────────┬────────┘
+         │ publishes to
+         ▼
+┌─────────────────────────┐
+│     SNS Topic           │
+│  (eventpub module)      │
+│                         │
+│ - Anomaly Detection     │
+│ - Delivery Logging      │
+│ - KMS Encryption        │
+└─────────┬───────────────┘
+          │ fan-out to:
+          ├─────────────────────────┐
+          │                         │
+          ▼                         ▼
+┌─────────────────┐     ┌──────────────────┐
+│ Kinesis         │     │ EventBridge      │
+│ Firehose        │     │ Rules            │
+│   ↓ S3          │     │   ↓ Subscribers  │
+│ (Event Cache)   │     │   (SQS/Lambda)   │
+└─────────────────┘     └──────────────────┘
+          │                         │
+          ▼                         ▼
+┌─────────────────┐     ┌──────────────────┐
+│ CloudWatch      │     │ CloudWatch       │
+│ - DLQ Alarm     │     │ - Anomaly        │
+│ - Delivery      │     │   Detection      │
+│   Failures      │     │                  │
+└─────────────────┘     └──────────────────┘
+```
+
 <!-- BEGIN_TF_DOCS -->
 <!-- markdownlint-disable -->
 <!-- vale off -->
@@ -19,6 +60,7 @@
 | <a name="input_default_tags"></a> [default\_tags](#input\_default\_tags) | Default tag map for application to all taggable resources in the module | `map(string)` | `{}` | no |
 | <a name="input_enable_event_cache"></a> [enable\_event\_cache](#input\_enable\_event\_cache) | Enable caching of events to an S3 bucket | `bool` | `false` | no |
 | <a name="input_enable_firehose_raw_message_delivery"></a> [enable\_firehose\_raw\_message\_delivery](#input\_enable\_firehose\_raw\_message\_delivery) | Enables raw message delivery on firehose subscription | `bool` | `false` | no |
+| <a name="input_enable_event_publishing_anomaly_detection"></a> [enable\_publishing\_anomaly\_detection](#input\_enable\_publishing\_anomaly\_detection) | Enable CloudWatch anomaly detection alarm for SNS message publishing. Detects abnormal drops or spikes in event publishing volume. | `bool` | `true` | no |
 | <a name="input_enable_sns_delivery_logging"></a> [enable\_sns\_delivery\_logging](#input\_enable\_sns\_delivery\_logging) | Enable SNS Delivery Failure Notifications | `bool` | `false` | no |
 | <a name="input_environment"></a> [environment](#input\_environment) | The name of the terraformscaffold environment the module is called for | `string` | n/a | yes |
 | <a name="input_event_cache_buffer_interval"></a> [event\_cache\_buffer\_interval](#input\_event\_cache\_buffer\_interval) | The buffer interval for data firehose | `number` | `500` | no |
@@ -31,6 +73,9 @@
 | <a name="input_log_retention_in_days"></a> [log\_retention\_in\_days](#input\_log\_retention\_in\_days) | The retention period in days for the Cloudwatch Logs events generated by the lambda function | `number` | n/a | yes |
 | <a name="input_name"></a> [name](#input\_name) | A unique name to distinguish this module invocation from others within the same CSI scope | `string` | n/a | yes |
 | <a name="input_project"></a> [project](#input\_project) | The name of the terraformscaffold project calling the module | `string` | n/a | yes |
+| <a name="input_event_publishing_anomaly_band_width"></a> [publishing\_anomaly\_band\_width](#input\_publishing\_anomaly\_band\_width) | The width of the anomaly detection band. Higher values (e.g., 4-6) reduce sensitivity and noise, lower values (e.g., 2-3) increase sensitivity. Recommended: 2-4 depending on traffic patterns. | `number` | `3` | no |
+| <a name="input_event_publishing_anomaly_evaluation_periods"></a> [publishing\_anomaly\_evaluation\_periods](#input\_publishing\_anomaly\_evaluation\_periods) | Number of evaluation periods for the publishing anomaly alarm. Each period is defined by event_publishing_anomaly_period. | `number` | `2` | no |
+| <a name="input_event_publishing_anomaly_period"></a> [publishing\_anomaly\_period](#input\_publishing\_anomaly\_period) | The period in seconds over which the specified statistic is applied for anomaly detection. Minimum 300 seconds (5 minutes). Recommended: 300-600 for event-driven workloads. | `number` | `300` | no |
 | <a name="input_region"></a> [region](#input\_region) | The AWS Region | `string` | n/a | yes |
 | <a name="input_sns_success_logging_sample_percent"></a> [sns\_success\_logging\_sample\_percent](#input\_sns\_success\_logging\_sample\_percent) | Enable SNS Delivery Successful Sample Percentage | `number` | `0` | no |
 ## Modules
@@ -42,6 +87,7 @@
 
 | Name | Description |
 |------|-------------|
+| <a name="output_publishing_anomaly_alarm"></a> [publishing\_anomaly\_alarm](#output\_publishing\_anomaly\_alarm) | CloudWatch anomaly detection alarm details for SNS publishing |
 | <a name="output_s3_bucket_event_cache"></a> [s3\_bucket\_event\_cache](#output\_s3\_bucket\_event\_cache) | S3 Bucket ARN and Name for event cache |
 | <a name="output_sns_topic"></a> [sns\_topic](#output\_sns\_topic) | SNS Topic ARN and Name |
 <!-- vale on -->

infrastructure/terraform/modules/eventpub/cloudwatch_metric_alarm_publishing_anomaly.tf

@@ -0,0 +1,42 @@
+resource "aws_cloudwatch_metric_alarm" "publishing_anomaly" {
+  count = var.enable_event_publishing_anomaly_detection ? 1 : 0
+
+  alarm_name          = "${local.csi}-sns-publishing-anomaly"
+  alarm_description   = "RELIABILITY: Anomaly detection alarm for abnormal SNS message publishing patterns. Detects unexpected drops or spikes in event publishing volume that may indicate service degradation or misconfiguration."
+  comparison_operator = "LessThanLowerOrGreaterThanUpperThreshold"
+  evaluation_periods  = var.event_publishing_anomaly_evaluation_periods # Number of evaluation periods for the publishing anomaly alarm.
+  threshold_metric_id = "ad1"
+  treat_missing_data  = "notBreaching"
+  actions_enabled     = true
+
+  tags = merge(
+    local.default_tags,
+    {
+      AlarmType    = "AnomalyDetection"
+      AlarmPurpose = "EventPublishingAbnormality"
+    }
+  )
+
+  metric_query {
+    id          = "m1"
+    return_data = true
+
+    metric {
+      metric_name = "NumberOfMessagesPublished"
+      namespace   = "AWS/SNS"
+      period      = var.event_publishing_anomaly_period # The period in seconds over which the specified statistic is applied for anomaly detection.
+      stat        = "Sum"
+
+      dimensions = {
+        TopicName = aws_sns_topic.main.name
+      }
+    }
+  }
+
+  metric_query {
+    id          = "ad1"
+    expression  = "ANOMALY_DETECTION_BAND(m1, ${var.event_publishing_anomaly_band_width})" # The width of the anomaly detection band. Higher values (e.g. 4-6) reduce sensitivity and noise, lower values (e.g. 2-3) increase sensitivity.
+    label       = "NumberOfMessagesPublished (expected)"
+    return_data = true
+  }
+}

infrastructure/terraform/modules/eventpub/outputs.tf

@@ -13,3 +13,11 @@ output "s3_bucket_event_cache" {
     bucket = module.s3bucket_event_cache[0].bucket
   } : {}
 }
+
+output "publishing_anomaly_alarm" {
+  description = "CloudWatch anomaly detection alarm details for SNS publishing"
+  value = var.enable_event_publishing_anomaly_detection ? {
+    arn  = aws_cloudwatch_metric_alarm.publishing_anomaly[0].arn
+    name = aws_cloudwatch_metric_alarm.publishing_anomaly[0].alarm_name
+  } : null
+}

infrastructure/terraform/modules/eventpub/variables.tf

@@ -129,3 +129,27 @@ variable "additional_policies_for_event_cache_bucket" {
   description = "A list of JSON policies to use to build the bucket policy"
   default     = []
 }
+
+variable "enable_event_publishing_anomaly_detection" {
+  type        = bool
+  description = "Enable CloudWatch anomaly detection alarm for SNS message publishing. Detects abnormal drops or spikes in event publishing volume."
+  default     = true
+}
+
+variable "event_publishing_anomaly_evaluation_periods" {
+  type        = number
+  description = "Number of evaluation periods for the publishing anomaly alarm. Each period is defined by event_publishing_anomaly_period."
+  default     = 2
+}
+
+variable "event_publishing_anomaly_period" {
+  type        = number
+  description = "The period in seconds over which the specified statistic is applied for anomaly detection. Minimum 300 seconds (5 minutes). Recommended: 300-600."
+  default     = 300
+}
+
+variable "event_publishing_anomaly_band_width" {
+  type        = number
+  description = "The width of the anomaly detection band. Higher values (e.g. 4-6) reduce sensitivity and noise, lower values (e.g. 2-3) increase sensitivity. Recommended: 2-4."
+  default     = 3
+}