CCM-16644: Add the eventsub module

Author: gareth-allan

infrastructure/terraform/modules/eventsub/README.md

@@ -0,0 +1,52 @@
+<!-- BEGIN_TF_DOCS -->
+<!-- markdownlint-disable -->
+<!-- vale off -->
+
+## Requirements
+
+| Name | Version |
+|------|---------|
+| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 1.9.0 |
+## Inputs
+
+| Name | Description | Type | Default | Required |
+|------|-------------|------|---------|:--------:|
+| <a name="input_access_logging_bucket"></a> [access\_logging\_bucket](#input\_access\_logging\_bucket) | Name of S3 bucket to use for access logging | `string` | `""` | no |
+| <a name="input_aws_account_id"></a> [aws\_account\_id](#input\_aws\_account\_id) | The AWS Account ID (numeric) | `string` | n/a | yes |
+| <a name="input_component"></a> [component](#input\_component) | The name of the terraformscaffold component calling this module | `string` | n/a | yes |
+| <a name="input_default_tags"></a> [default\_tags](#input\_default\_tags) | Default tag map for application to all taggable resources in the module | `map(string)` | `{}` | no |
+| <a name="input_enable_event_anomaly_detection"></a> [enable\_event\_anomaly\_detection](#input\_enable\_event\_anomaly\_detection) | Enable CloudWatch anomaly detection alarm for SNS topic message publishing | `bool` | `true` | no |
+| <a name="input_enable_event_cache"></a> [enable\_event\_cache](#input\_enable\_event\_cache) | Enable caching of events to an S3 bucket | `bool` | `true` | no |
+| <a name="input_enable_firehose_raw_message_delivery"></a> [enable\_firehose\_raw\_message\_delivery](#input\_enable\_firehose\_raw\_message\_delivery) | Enables raw message delivery on firehose subscription | `bool` | `false` | no |
+| <a name="input_enable_sns_delivery_logging"></a> [enable\_sns\_delivery\_logging](#input\_enable\_sns\_delivery\_logging) | Enable SNS Delivery Failure Notifications | `bool` | `true` | no |
+| <a name="input_environment"></a> [environment](#input\_environment) | The name of the terraformscaffold environment the module is called for | `string` | n/a | yes |
+| <a name="input_event_anomaly_band_width"></a> [event\_anomaly\_band\_width](#input\_event\_anomaly\_band\_width) | The width of the anomaly detection band. Higher values (e.g. 4-6) reduce sensitivity and noise, lower values (e.g. 2-3) increase sensitivity. Recommended: 2-4. | `number` | `3` | no |
+| <a name="input_event_anomaly_evaluation_periods"></a> [event\_anomaly\_evaluation\_periods](#input\_event\_anomaly\_evaluation\_periods) | Number of evaluation periods for the anomaly alarm. Each period is defined by event\_anomaly\_period. | `number` | `2` | no |
+| <a name="input_event_anomaly_period"></a> [event\_anomaly\_period](#input\_event\_anomaly\_period) | The period in seconds over which the specified statistic is applied for anomaly detection. Minimum 300 seconds (5 minutes). Recommended: 300-600. | `number` | `300` | no |
+| <a name="input_event_cache_buffer_interval"></a> [event\_cache\_buffer\_interval](#input\_event\_cache\_buffer\_interval) | The buffer interval for data firehose | `number` | `500` | no |
+| <a name="input_event_cache_expiry_days"></a> [event\_cache\_expiry\_days](#input\_event\_cache\_expiry\_days) | s3 archiving expiry in days | `number` | `30` | no |
+| <a name="input_force_destroy"></a> [force\_destroy](#input\_force\_destroy) | When enabled will force destroy event-cache S3 bucket | `bool` | `false` | no |
+| <a name="input_glue_role_arn"></a> [glue\_role\_arn](#input\_glue\_role\_arn) | ARN of the Glue execution role from the parent | `string` | n/a | yes |
+| <a name="input_group"></a> [group](#input\_group) | The name of the tfscaffold group | `string` | `null` | no |
+| <a name="input_kms_key_arn"></a> [kms\_key\_arn](#input\_kms\_key\_arn) | KMS key arn to use for this function | `string` | n/a | yes |
+| <a name="input_log_level"></a> [log\_level](#input\_log\_level) | The log level to be used in lambda functions within the component. Any log with a lower severity than the configured value will not be logged: https://docs.python.org/3/library/logging.html#levels | `string` | `"WARN"` | no |
+| <a name="input_log_retention_in_days"></a> [log\_retention\_in\_days](#input\_log\_retention\_in\_days) | The retention period in days for the Cloudwatch Logs events generated by the lambda function | `number` | n/a | yes |
+| <a name="input_name"></a> [name](#input\_name) | A unique name to distinguish this module invocation from others within the same CSI scope | `string` | n/a | yes |
+| <a name="input_project"></a> [project](#input\_project) | The name of the terraformscaffold project calling the module | `string` | n/a | yes |
+| <a name="input_region"></a> [region](#input\_region) | The AWS Region | `string` | n/a | yes |
+| <a name="input_shared_infra_account_id"></a> [shared\_infra\_account\_id](#input\_shared\_infra\_account\_id) | The AWS Account ID of the shared infrastructure account | `string` | `"000000000000"` | no |
+| <a name="input_sns_success_logging_sample_percent"></a> [sns\_success\_logging\_sample\_percent](#input\_sns\_success\_logging\_sample\_percent) | Enable SNS Delivery Successful Sample Percentage | `number` | `0` | no |
+## Modules
+
+| Name | Source | Version |
+|------|--------|---------|
+| <a name="module_s3bucket_event_cache"></a> [s3bucket\_event\_cache](#module\_s3bucket\_event\_cache) | https://github.com/NHSDigital/nhs-notify-shared-modules/releases/download/3.1.2/terraform-s3bucket.zip | n/a |
+## Outputs
+
+| Name | Description |
+|------|-------------|
+| <a name="output_s3_bucket_event_cache"></a> [s3\_bucket\_event\_cache](#output\_s3\_bucket\_event\_cache) | S3 Bucket ARN and Name for event cache |
+| <a name="output_sns_topic"></a> [sns\_topic](#output\_sns\_topic) | SNS Topic ARN and Name |
+<!-- vale on -->
+<!-- markdownlint-enable -->
+<!-- END_TF_DOCS -->

infrastructure/terraform/modules/eventsub/cloudwatch_log_group_kinesis_data_firehose.tf

@@ -0,0 +1,14 @@
+resource "aws_cloudwatch_log_group" "kinesis_data_firehose" {
+  count = var.enable_event_cache ? 1 : 0
+
+  name              = "/aws/firehose/${local.csi}"
+  kms_key_id        = var.kms_key_arn
+  retention_in_days = var.log_retention_in_days
+}
+
+resource "aws_cloudwatch_log_stream" "kinesis_data_firehose_extended_s3" {
+  count = var.enable_event_cache ? 1 : 0
+
+  name           = "extended_s3"
+  log_group_name = aws_cloudwatch_log_group.kinesis_data_firehose[0].name
+}

infrastructure/terraform/modules/eventsub/cloudwatch_log_group_sns_delivery_logging_failure.tf

@@ -0,0 +1,9 @@
+resource "aws_cloudwatch_log_group" "sns_delivery_logging_failure" {
+  count = var.enable_sns_delivery_logging ? 1 : 0
+
+  # SNS doesn't allow specifying a log group and is derived as: sns/${region}/${account_id}/${name_of_sns_topic}/Failure
+  # (for failure logs)
+  name              = "sns/${var.region}/${var.aws_account_id}/${local.csi}/Failure"
+  kms_key_id        = var.kms_key_arn
+  retention_in_days = var.log_retention_in_days
+}

infrastructure/terraform/modules/eventsub/cloudwatch_log_group_sns_delivery_logging_success.tf

@@ -0,0 +1,9 @@
+resource "aws_cloudwatch_log_group" "sns_delivery_logging_success" {
+  count = var.enable_sns_delivery_logging ? 1 : 0
+
+  # SNS doesn't allow specifying a log group and is derived as: sns/${region}/${account_id}/${name_of_sns_topic}
+  # (for success logs)
+  name              = "sns/${var.region}/${var.aws_account_id}/${local.csi}"
+  kms_key_id        = var.kms_key_arn
+  retention_in_days = var.log_retention_in_days
+}

infrastructure/terraform/modules/eventsub/cloudwatch_metric_alarm_sns_delivery_failures.tf

@@ -0,0 +1,16 @@
+resource "aws_cloudwatch_metric_alarm" "sns_delivery_failures" {
+  alarm_name          = "${local.csi}-sns-delivery-failures"
+  alarm_description   = "RELIABILITY: Alarm for SNS topic delivery failures"
+  comparison_operator = "GreaterThanThreshold"
+  evaluation_periods  = 1
+  metric_name         = "NumberOfNotificationsFailed"
+  namespace           = "AWS/SNS"
+  period              = 300
+  statistic           = "Sum"
+  threshold           = 0
+  treat_missing_data  = "notBreaching"
+
+  dimensions = {
+    TopicName = aws_sns_topic.main.name
+  }
+}

infrastructure/terraform/modules/eventsub/cloudwatch_metric_alarm_subscriber_anomaly.tf

@@ -0,0 +1,40 @@
+resource "aws_cloudwatch_metric_alarm" "subscriber_anomaly" {
+  count = var.enable_event_anomaly_detection ? 1 : 0
+
+  alarm_name          = "${local.csi}-subscriber-anomaly"
+  alarm_description   = "ANOMALY: Detects anomalous patterns in messages published to the SNS fanout topic"
+  comparison_operator = "LessThanLowerOrGreaterThanUpperThreshold"
+  evaluation_periods  = var.event_anomaly_evaluation_periods
+  threshold_metric_id = "ad1"
+  treat_missing_data  = "notBreaching"
+
+  metric_query {
+    id          = "m1"
+    return_data = true
+
+    metric {
+      metric_name = "NumberOfNotificationsDelivered"
+      namespace   = "AWS/SNS"
+      period      = var.event_anomaly_period
+      stat        = "Sum"
+
+      dimensions = {
+        TopicName = aws_sns_topic.main.name
+      }
+    }
+  }
+
+  metric_query {
+    id          = "ad1"
+    expression  = "ANOMALY_DETECTION_BAND(m1, ${var.event_anomaly_band_width})"
+    label       = "NumberOfNotificationsDelivered (expected)"
+    return_data = true
+  }
+
+  tags = merge(
+    var.default_tags,
+    {
+      Name = "${local.csi}-subscriber-anomaly"
+    }
+  )
+}

infrastructure/terraform/modules/eventsub/iam_policy_sns_delivery_logging_cloudwatch.tf

@@ -0,0 +1,44 @@
+resource "aws_iam_policy" "sns_delivery_logging_cloudwatch" {
+  count = var.enable_sns_delivery_logging ? 1 : 0
+
+  name        = "${local.csi}-${var.name}-sns-delivery"
+  description = "Policy for ${local.csi}-${var.name} SNS Delivery Logging"
+  policy      = data.aws_iam_policy_document.sns_delivery_logging_cloudwatch[0].json
+}
+
+data "aws_iam_policy_document" "sns_delivery_logging_cloudwatch" {
+  count = var.enable_sns_delivery_logging ? 1 : 0
+
+  statement {
+    sid    = "KMSCloudwatchKeyAccess"
+    effect = "Allow"
+
+    actions = [
+      "kms:GenerateDataKey",
+      "kms:Decrypt",
+    ]
+
+    resources = [
+      var.kms_key_arn
+    ]
+  }
+
+  statement {
+    sid    = "AllowSNSDeliveryNotifications"
+    effect = "Allow"
+
+    actions = [
+      "logs:CreateLogStream",
+      "logs:PutLogEvents",
+      "logs:PutMetricFilter",
+      "logs:PutRetentionPolicy",
+    ]
+
+    resources = [
+      aws_cloudwatch_log_group.sns_delivery_logging_success[0].arn,
+      "${aws_cloudwatch_log_group.sns_delivery_logging_success[0].arn}:log-stream:*",
+      aws_cloudwatch_log_group.sns_delivery_logging_failure[0].arn,
+      "${aws_cloudwatch_log_group.sns_delivery_logging_failure[0].arn}:log-stream:*",
+    ]
+  }
+}

infrastructure/terraform/modules/eventsub/iam_role_firehose_role.tf

@@ -0,0 +1,59 @@
+resource "aws_iam_role" "firehose_role" {
+  count = var.enable_event_cache ? 1 : 0
+
+  name               = "${local.csi}-firehose-role"
+  assume_role_policy = data.aws_iam_policy_document.firehose_assume_role[0].json
+}
+
+data "aws_iam_policy_document" "firehose_assume_role" {
+  count = var.enable_event_cache ? 1 : 0
+
+  statement {
+    effect = "Allow"
+
+    principals {
+      type        = "Service"
+      identifiers = ["firehose.amazonaws.com"]
+    }
+
+    actions = ["sts:AssumeRole"]
+  }
+}
+
+resource "aws_iam_role_policy_attachment" "s3_write_object" {
+  count = var.enable_event_cache ? 1 : 0
+
+  role       = aws_iam_role.firehose_role[0].name
+  policy_arn = aws_iam_policy.s3_write_object[0].arn
+}
+
+resource "aws_iam_policy" "s3_write_object" {
+  count = var.enable_event_cache ? 1 : 0
+
+  name        = "${local.csi}-${var.name}-s3-write-object"
+  description = "S3 Put Object policy for ${local.csi}-${var.name} Firehose"
+  policy      = data.aws_iam_policy_document.s3_write_object[0].json
+}
+
+data "aws_iam_policy_document" "s3_write_object" {
+  count = var.enable_event_cache ? 1 : 0
+
+  statement {
+    sid    = "AllowWriteObject"
+    effect = "Allow"
+
+    actions = [
+      "s3:AbortMultipartUpload",
+      "s3:GetBucketLocation",
+      "s3:GetObject",
+      "s3:ListBucket",
+      "s3:ListBucketMultipartUploads",
+      "s3:PutObject",
+      "s3:PutObject",
+    ]
+
+    resources = [
+      "${module.s3bucket_event_cache[0].arn}/*",
+    ]
+  }
+}

infrastructure/terraform/modules/eventsub/iam_role_sns.tf

@@ -0,0 +1,51 @@
+resource "aws_iam_role" "sns_role" {
+  name               = "${local.csi}-sns-role"
+  assume_role_policy = data.aws_iam_policy_document.sns_assume_role.json
+}
+
+resource "aws_iam_policy" "firehose_delivery" {
+  count = var.enable_event_cache ? 1 : 0
+
+  name        = "${local.csi}-${var.name}-firehose-delivery"
+  description = "Delivery Policy for ${local.csi}-${var.name} Firehose"
+  policy      = data.aws_iam_policy_document.firehose_delivery[0].json
+}
+
+resource "aws_iam_role_policy_attachment" "firehose_delivery" {
+  count = var.enable_event_cache ? 1 : 0
+
+  role       = aws_iam_role.sns_role.name
+  policy_arn = aws_iam_policy.firehose_delivery[0].arn
+}
+
+
+data "aws_iam_policy_document" "sns_assume_role" {
+  statement {
+    effect = "Allow"
+
+    principals {
+      type        = "Service"
+      identifiers = ["sns.amazonaws.com"]
+    }
+
+    actions = ["sts:AssumeRole"]
+  }
+}
+
+data "aws_iam_policy_document" "firehose_delivery" {
+  count = var.enable_event_cache ? 1 : 0
+
+  statement {
+    sid    = "AllowFirehoseDelivery"
+    effect = "Allow"
+
+    actions = [
+      "firehose:PutRecord",
+      "firehose:PutRecordBatch"
+    ]
+
+    resources = [
+      "${aws_kinesis_firehose_delivery_stream.main[0].arn}",
+    ]
+  }
+}

infrastructure/terraform/modules/eventsub/iam_role_sns_delivery_logging.tf

@@ -0,0 +1,21 @@
+resource "aws_iam_role" "sns_delivery_logging_role" {
+  count = var.enable_sns_delivery_logging ? 1 : 0
+
+  name               = "${local.csi}-sns-delivery-logging"
+  assume_role_policy = data.aws_iam_policy_document.sns_delivery_logging_assume_role[0].json
+}
+
+data "aws_iam_policy_document" "sns_delivery_logging_assume_role" {
+  count = var.enable_sns_delivery_logging ? 1 : 0
+
+  statement {
+    effect = "Allow"
+
+    principals {
+      type        = "Service"
+      identifiers = ["sns.amazonaws.com"]
+    }
+
+    actions = ["sts:AssumeRole"]
+  }
+}