From 14256bc0fc30bd001e8794e2f2fc593122635e5f Mon Sep 17 00:00:00 2001 From: Derek Chamorro Date: Tue, 27 Oct 2026 22:24:55 -0500 Subject: [PATCH 1/2] Add behavioural anomalies and EPSS/KEV enrichment (long tail, part 1) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Completion item 5 from the re-review (~/Documents/Testing/plans/2026-10-27-predictive-completion.md), first of two parts: the UEBA breadth that does not require an external dataset, and the EPSS/KEV half of exploitability enrichment. Behavioural anomalies: - Two new bounded CloudTrail analyzer queries share the established shape (7-day evaluation against a 30-day baseline, partition projection, the same exclusions): NEW_API_FOR_PRINCIPAL (calls a principal never made in the baseline — a never-touched service shows through its calls) and UNUSUAL_HOUR_ACTIVITY (UTC hours never used). Both rows carry a detail column (new APIs / hours, capped at 200 chars) that the anomaly nodes record. - RULE-035 (medium; first-use APIs are a lead) and RULE-036 (low; timezone moves trip it) surface them; neither auto-tickets. ASN attribution is deferred with its reason recorded: it needs an external IP-to-ASN dataset and a licensing decision. EPSS/KEV: - threat-intel types gain cve (known-exploited membership, e.g. a CISA KEV export) and epss (CVE,score lines — the FIRST CSV's first two columns), with validation, dedup, caps and TTL unchanged; the store carries the score; Vulnerabilities' cve_id candidates both lookups. - A KEV match marks the exploit available and EPSS rides exploitability through one shared helper — max(CVSS/10, EPSS) + 0.2 for a known exploit, clamped — now used by both pathLikelihood and computeRiskScore so the likelihood and the score can no longer disagree. - Terraform variables/outputs, chart values/schema/configmap and the OPERATIONAL runbook wire the two new list URIs. Load-bearing updates, explicit: the rule catalogue goes 34 -> 36 (pinned counts in rules.test.ts, README, ARCHITECTURE updated); REVIEWED_MAPPING gains reviewed entries for both rules (-> KH-DET-1); RULE_TECHNIQUES gains T1078 for each; getRemediationHint gains both hints; KH-DET-1 is broadened from credential-use to identity anomalies with the new rules and evidence text (control counts unchanged). Why this is safe: both queries are read-only, window-bounded and exclusion-filtered like the existing three; the new rules report behavioural leads with auto-ticketing off; the threat-intel additions reuse the reviewed import path (validation, cap, TTL) and only raise exploitability through values the score already carried (CVSS, exploit-known); no new write surface and no identifiers. --- ARCHITECTURE.md | 35 +++-- CHANGELOG.md | 18 +++ OPERATIONAL.md | 15 +- README.md | 28 ++-- deploy/helm/khalifa/templates/configmap.yaml | 2 + deploy/helm/khalifa/values.schema.json | 4 + deploy/helm/khalifa/values.yaml | 2 + deploy/terraform/khalifa-data/outputs.tf | 2 + deploy/terraform/khalifa-data/variables.tf | 12 ++ lambdas/cloudtrail-analyzer/index.test.ts | 86 ++++++++++- .../cloudtrail-analyzer/src/anomalies.test.ts | 112 ++++++++++++-- lambdas/cloudtrail-analyzer/src/anomalies.ts | 138 +++++++++++++++++- lambdas/threat-intel-import/index.test.ts | 2 +- lambdas/threat-intel-import/index.ts | 6 +- packages/risk-engine/src/attack-techniques.ts | 2 + packages/risk-engine/src/compliance-types.ts | 8 +- .../src/control-rule-mapping.test.ts | 8 + packages/risk-engine/src/exploitability.ts | 24 +++ packages/risk-engine/src/likelihood.ts | 13 +- packages/risk-engine/src/rules.test.ts | 8 +- packages/risk-engine/src/rules.ts | 38 +++++ packages/risk-engine/src/runner.test.ts | 46 ++++++ packages/risk-engine/src/runner.ts | 10 ++ packages/risk-engine/src/scoring.ts | 12 +- packages/risk-engine/src/threat-intel.test.ts | 61 ++++++++ packages/risk-engine/src/threat-intel.ts | 61 ++++++-- packages/risk-engine/src/types.ts | 4 +- 27 files changed, 671 insertions(+), 86 deletions(-) create mode 100644 packages/risk-engine/src/exploitability.ts diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 544f6d3e..d80dc191 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -91,7 +91,7 @@ flowchart LR subgraph DETECT["Detection pipelines"] direction TB - CTA["CloudTrail Analyzer (daily)
usage profile + anomaly queries:
ROOT_USAGE, SECRET_READ_NEW_SOURCE_IP,
ACCESS_DENIED_SPIKE, CANARY_USED"] + CTA["CloudTrail Analyzer (daily)
usage profile + anomaly queries:
ROOT_USAGE, SECRET_READ_NEW_SOURCE_IP,
ACCESS_DENIED_SPIKE, CANARY_USED,
NEW_API_FOR_PRINCIPAL, UNUSUAL_HOUR_ACTIVITY"] DSPM["DSPM Scanner
S3 objects + Lambda env vars
(in-memory classification)"] end @@ -99,7 +99,7 @@ flowchart LR direction TB PE["Policy Evaluator (6h):
effective permissions,
escalation + pass-role paths"] LPS["Least-Privilege Sweep (daily):
rightsizing vs CloudTrail usage"] - RE["Risk Engine (hourly):
34 rules, attack paths, scoring"] + RE["Risk Engine (hourly):
36 rules, attack paths, scoring"] COMP["Compliance:
CIS v3.0.0 + Khalifa baseline
(Security Hub reconciliation)"] end @@ -292,18 +292,25 @@ deployment without the table simply has no observed traffic. ## Threat-intelligence matching Customer-supplied intelligence arrives as plain S3 lists (one indicator per -line): `threat-intel-import` validates them (IPs v4/v6, 12-digit account ids), -caps each list, and writes `ThreatIndicators` with a TTL — freshness is the -TTL, so an indicator dropped from a list drains itself. Nothing here calls the -internet: AWS's own managed reputation lists need no import because GuardDuty -applies them and the findings they drive already feed RULE-018/RULE-029. The -rule runner joins the store against the facts rules already match: an anomaly's -`source_ip` and a trusted `ExternalAccount`'s account id. A match adds a flat -**+10 threat-intelligence factor** to the risk score (a fact about the path, -not a proportional strength; recorded as `threatIntelMatches` on the issue and -labelled on the score card) and matches clear on refresh once the indicator is -gone. Matched issues do **not** auto-ticket until `/rules/quality` shows the -measured precision P6 requires; the local +line): `threat-intel-import` validates them (IPs v4/v6, 12-digit account ids, +CVE ids), caps each list, and writes `ThreatIndicators` with a TTL — freshness +is the TTL, so an indicator dropped from a list drains itself. Four types: +`ip` and `account`, plus `cve` (known-exploited membership — an export of the +CISA KEV catalogue's CVE ids) and `epss` (lines of `CVE-YYYY-NNNN,score`, +the first two columns of the FIRST EPSS CSV). Nothing here calls the internet: +AWS's own managed reputation lists need no import because GuardDuty applies +them and the findings they drive already feed RULE-018/RULE-029. The rule +runner joins the store against the facts rules already match: an anomaly's +`source_ip`, a trusted `ExternalAccount`'s account id, and a `Vulnerability` +vertex's `cve_id` (candidate for both CVE lookups). An ip/account match adds a +flat **+10 threat-intelligence factor** to the risk score (a fact about the +path, not a proportional strength; recorded as `threatIntelMatches` on the +issue and labelled on the score card); a `cve` match marks the exploit +available and an `epss` score rides the shared exploitability value +(`max(CVSS/10, EPSS)`, +0.2 for a known exploit — the same helper the +likelihood and the score both use, so they cannot disagree). Matches clear on +refresh once the indicator is gone. Matched issues do **not** auto-ticket +until `/rules/quality` shows the measured precision P6 requires; the local `scripts/risk-plan` gate is unaffected (it has no store to read, so it never claims a match). diff --git a/CHANGELOG.md b/CHANGELOG.md index 8fe3497c..300bf25b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,24 @@ All notable changes to Khalifa. +## [Unreleased] - UEBA breadth and EPSS/KEV enrichment (long tail, part 1) + +### Added + +- **Behavioural anomalies in the CloudTrail analyzer**: two new bounded queries (7-day window against a 30-day baseline, same partition projection and exclusions as the existing three) — `NEW_API_FOR_PRINCIPAL` (API calls a principal never made in the baseline; a service it never touched shows through its calls) and `UNUSUAL_HOUR_ACTIVITY` (UTC hours the principal never used). Rows carry a `detail` column (the new APIs, or the hours; capped at 200 chars), the anomaly nodes record it, and two new medium/low rules (RULE-035/RULE-036) surface them — neither auto-tickets (behavioural leads need `/rules/quality` evidence first). ASN attribution is deliberately deferred: it needs an external IP→ASN dataset and a licensing decision, recorded in the plan. +- **CVE and EPSS threat-intel types**: `threat-intel-import` now imports known-exploited CVE lists (`THREAT_INTEL_CVE_LIST_S3_URI`, e.g. a CISA KEV export) and EPSS score lists (`THREAT_INTEL_EPSS_LIST_S3_URI`, `CVE-YYYY-NNNN,score` — the FIRST CSV's first two columns). The runner candidates `Vulnerability` vertices' `cve_id` for both; a KEV match marks the exploit available and an EPSS score rides the exploitability factor — now one shared helper, `max(CVSS/10, EPSS)` +0.2 for a known exploit, used by both the likelihood and the score so they cannot disagree. + +### Load-bearing updates (explicit) + +- Rule catalogue **34 → 36 rules** (both enabled, auto-ticketing off): pinned counts in `rules.test.ts`, README and the ARCHITECTURE diagram updated. +- `REVIEWED_MAPPING` (`control-rule-mapping.test.ts`) gains reviewed entries for RULE-035/RULE-036 (both → KH-DET-1, with reasons); `RULE_TECHNIQUES` gains T1078 for each; `getRemediationHint` gains both hints. +- KH-DET-1 (`compliance-types.ts`) broadened from "credential-use anomalies" to "identity anomalies": `relatedRules` now includes RULE-035/RULE-036 and the evidence text names the new types (control counts unchanged). +- Terraform (variables/outputs), chart values/schema/configmap and the OPERATIONAL threat-intel runbook carry the two new list URIs. + +### Notes + +- Still open from the re-review (`~/Documents/Testing/plans/2026-10-27-predictive-completion.md`): long-tail part 2 — trust-center publication of projections, per-issue risk window and SLO-burn thresholds, exposure-by-type series. + ## [Unreleased] - Observed-traffic hop citations and shift-left v2 ### Added diff --git a/OPERATIONAL.md b/OPERATIONAL.md index b88fa6e3..58f939de 100644 --- a/OPERATIONAL.md +++ b/OPERATIONAL.md @@ -207,10 +207,14 @@ kubectl create job --from=cronjob/khalifa-rule-runner rule-runner-manual -n secu 1. Put plain lists in an account-owned S3 bucket: one indicator per line, `#` comments and blanks ignored. IPs (v4/v6) in one object, 12-digit - account ids in another. + account ids in another, CVE ids in a third, and EPSS scores in a fourth as + `CVE-YYYY-NNNN,score` (the first two columns of the FIRST EPSS CSV work + as-is; an export of the CISA KEV catalogue's CVE ids works for the CVE + list). 2. In `khalifa-data`: set `threat_intel_list_bucket` (grants the rule-runner - role `s3:GetObject` on it) and `threat_intel_ip_list_s3_uri` / - `threat_intel_account_list_s3_uri` (flow into the chart's config), plus + role `s3:GetObject` on it) and any of `threat_intel_ip_list_s3_uri`, + `threat_intel_account_list_s3_uri`, `threat_intel_cve_list_s3_uri`, + `threat_intel_epss_list_s3_uri` (flow into the chart's config), plus `threat_intel_ttl_days` (default 90). Apply. 3. Set `threatIntelImport.image` (the rule-runner digest) and unsuspend or run it once: @@ -219,7 +223,9 @@ kubectl create job --from=cronjob/khalifa-rule-runner rule-runner-manual -n secu invalid lines are skipped and counted in the log. 4. Verify: `ThreatIndicators` gains rows with a TTL; the next rule run scores matching anomaly IPs / trusted external accounts with the +10 - threat-intelligence factor and records the match on the issue. + threat-intelligence factor, marks a path exploit-available when one of its + CVEs is on the known-exploited list, takes the higher of CVSS/EPSS for + exploitability, and records matches on the issue. 5. Freshness is the TTL: an indicator dropped from the list expires by itself up to `threat_intel_ttl_days` after it was last imported. AWS's own managed reputation lists need no import (GuardDuty applies them; RULE-018/RULE-029 @@ -230,6 +236,7 @@ kubectl create job --from=cronjob/khalifa-rule-runner rule-runner-manual -n secu The recompute worker is rendered only when `collection.recomputeQueueUrl` is set (the complete example/Terraform wires it). It is always-on by design: + - Producer: the incremental collector sends one message per five-minute window when a batch touched IAM. Watch `Queued an event-triggered recompute` in the incremental collector logs. diff --git a/README.md b/README.md index 724aba24..27ee7d0f 100644 --- a/README.md +++ b/README.md @@ -22,20 +22,20 @@ One member role. Evaluation stays in-account. ## Features -| Surface | What you get | -| ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| **Collection** | 30+ AWS services via [`SecurityGraphCollectorRole`](templates/SecurityGraphCollectorRole.yaml). A hub account collects; each member account gets the spoke role. | -| **Graph** | Typed vertices and edges in Neptune. Same input, same Gremlin traversal. | -| **CIEM** | IAM decomposition, effective permissions, escalation paths (assume chains and `iam:PassRole` to compute, RULE-034), CloudTrail-grounded rightsizing. | -| **Least privilege** | Sweep of roles and users. Deployable policy recommendations. | -| **Secrets** | Secrets Manager and SSM parameter inventory with who-can-read analysis from identity and resource policies, rotation and access age, sharing detection (RULE-010, RULE-032, RULE-033), and admin users holding active access keys (RULE-031). | -| **Risk** | Scheduled rules and attack-path traversals on the live graph (34 rules), with an exploitability factor that raises scores for paths carrying Amazon Inspector findings, an explainable likelihood on every path (tempered down when the graph it read is aging or stale), escalation paths recomputed within minutes of an IAM change (event-triggered and rate-bounded), a per-fix simulation that counts what each edge removal would clear, matching against customer-supplied threat-intel lists (additive score bonus, TTL-fresh), curated MITRE ATT&CK techniques on every finding, graph-only adversary scenarios that re-run technique chains over the live graph, and a per-rule quality report (suppression vs resolution) that calibrates which rules to trust. | -| **Detection** | GuardDuty detector coverage per region (including `MISSING`), Security Hub findings joined to internet-exposed resources (RULE-017, RULE-018), CloudTrail anomaly signals: root activity, secret reads from IPs outside each principal's own 30-day baseline, AccessDenied deviations from each principal's own daily baseline (RULE-020 to RULE-022), canary credential tripwires (RULE-023), secret scanning of Lambda environment variables plus IMDSv1-with-role detection (RULE-024), and Amazon Inspector CVEs on internet-exposed workloads (RULE-003). | -| **Exposure** | Public EKS control planes (RULE-019) plus public S3/RDS and world-open security groups, derived from collected endpoint and network config, with optional observed traffic from VPC flow logs confirming where traffic actually goes. | -| **Compliance** | CIS AWS Foundations v3.0 and the Khalifa baseline, per-control evidence, mapped SOC 2 / ISO 27001 / PCI DSS / NIST 800-53 requirements, and CIS reconciliation against Security Hub before anything is publishable. | -| **Trust center** | Nightly static trust-center page built from the publishable posture (statuses only, in-repo site). | -| **Runtime** | EKS Helm + `khalifa-eks` / `khalifa-data` / `khalifa-registry`. CDK Lambda remains for the in-repo pipeline. | -| **Shift-left** | `npm run risk:plan -- ` predicts which rules a Terraform plan would feed (world-open security groups, account-root trusts, public buckets/RDS/EKS endpoints, secret policies, function URLs, IMDSv1) and fails the gate, naming the rule and the predicted path before anything is deployed. | +| Surface | What you get | +| ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Collection** | 30+ AWS services via [`SecurityGraphCollectorRole`](templates/SecurityGraphCollectorRole.yaml). A hub account collects; each member account gets the spoke role. | +| **Graph** | Typed vertices and edges in Neptune. Same input, same Gremlin traversal. | +| **CIEM** | IAM decomposition, effective permissions, escalation paths (assume chains and `iam:PassRole` to compute, RULE-034), CloudTrail-grounded rightsizing. | +| **Least privilege** | Sweep of roles and users. Deployable policy recommendations. | +| **Secrets** | Secrets Manager and SSM parameter inventory with who-can-read analysis from identity and resource policies, rotation and access age, sharing detection (RULE-010, RULE-032, RULE-033), and admin users holding active access keys (RULE-031). | +| **Risk** | Scheduled rules and attack-path traversals on the live graph (36 rules), with an exploitability factor that raises scores for paths carrying Amazon Inspector findings or imported known-exploited (KEV) CVEs and takes the higher of CVSS/EPSS, an explainable likelihood on every path (tempered down when the graph it read is aging or stale), escalation paths recomputed within minutes of an IAM change (event-triggered and rate-bounded), a per-fix simulation that counts what each edge removal would clear, matching against customer-supplied threat-intel lists (IPs, accounts, CVE/KEV, EPSS — additive score bonus, TTL-fresh), behavioural anomalies (first-use APIs, unusual hours), curated MITRE ATT&CK techniques on every finding, graph-only adversary scenarios that re-run technique chains over the live graph, and a per-rule quality report (suppression vs resolution) that calibrates which rules to trust. | +| **Detection** | GuardDuty detector coverage per region (including `MISSING`), Security Hub findings joined to internet-exposed resources (RULE-017, RULE-018), CloudTrail anomaly signals: root activity, secret reads from IPs outside each principal's own 30-day baseline, AccessDenied deviations from each principal's own daily baseline (RULE-020 to RULE-022), canary credential tripwires (RULE-023), secret scanning of Lambda environment variables plus IMDSv1-with-role detection (RULE-024), and Amazon Inspector CVEs on internet-exposed workloads (RULE-003). | +| **Exposure** | Public EKS control planes (RULE-019) plus public S3/RDS and world-open security groups, derived from collected endpoint and network config, with optional observed traffic from VPC flow logs confirming where traffic actually goes. | +| **Compliance** | CIS AWS Foundations v3.0 and the Khalifa baseline, per-control evidence, mapped SOC 2 / ISO 27001 / PCI DSS / NIST 800-53 requirements, and CIS reconciliation against Security Hub before anything is publishable. | +| **Trust center** | Nightly static trust-center page built from the publishable posture (statuses only, in-repo site). | +| **Runtime** | EKS Helm + `khalifa-eks` / `khalifa-data` / `khalifa-registry`. CDK Lambda remains for the in-repo pipeline. | +| **Shift-left** | `npm run risk:plan -- ` predicts which rules a Terraform plan would feed (world-open security groups, account-root trusts, public buckets/RDS/EKS endpoints, secret policies, function URLs, IMDSv1) and fails the gate, naming the rule and the predicted path before anything is deployed. | --- diff --git a/deploy/helm/khalifa/templates/configmap.yaml b/deploy/helm/khalifa/templates/configmap.yaml index 2aa20b02..65fc8efc 100644 --- a/deploy/helm/khalifa/templates/configmap.yaml +++ b/deploy/helm/khalifa/templates/configmap.yaml @@ -17,6 +17,8 @@ data: ACCESS_ANALYZER_TABLE: {{ .Values.config.accessAnalyzerTable | quote }} TRAFFIC_OBSERVATIONS_TABLE: {{ .Values.config.trafficObservationsTable | quote }} THREAT_INDICATORS_TABLE: {{ .Values.config.threatIndicatorsTable | quote }} + THREAT_INTEL_CVE_LIST_S3_URI: {{ .Values.config.threatIntelCveListS3Uri | quote }} + THREAT_INTEL_EPSS_LIST_S3_URI: {{ .Values.config.threatIntelEpssListS3Uri | quote }} THREAT_INTEL_IP_LIST_S3_URI: {{ .Values.config.threatIntelIpListS3Uri | quote }} THREAT_INTEL_ACCOUNT_LIST_S3_URI: {{ .Values.config.threatIntelAccountListS3Uri | quote }} THREAT_INTEL_TTL_DAYS: {{ .Values.config.threatIntelTtlDays | quote }} diff --git a/deploy/helm/khalifa/values.schema.json b/deploy/helm/khalifa/values.schema.json index 5f99199b..48e9e360 100644 --- a/deploy/helm/khalifa/values.schema.json +++ b/deploy/helm/khalifa/values.schema.json @@ -89,6 +89,8 @@ "accessAnalyzerTable", "trafficObservationsTable", "threatIndicatorsTable", + "threatIntelCveListS3Uri", + "threatIntelEpssListS3Uri", "threatIntelIpListS3Uri", "threatIntelAccountListS3Uri", "threatIntelTtlDays", @@ -117,6 +119,8 @@ "accessAnalyzerTable": { "type": "string", "minLength": 1 }, "trafficObservationsTable": { "type": "string" }, "threatIndicatorsTable": { "type": "string" }, + "threatIntelCveListS3Uri": { "type": "string", "pattern": "^(|s3://.+/.+)$" }, + "threatIntelEpssListS3Uri": { "type": "string", "pattern": "^(|s3://.+/.+)$" }, "threatIntelIpListS3Uri": { "type": "string", "pattern": "^(|s3://.+/.+)$" }, "threatIntelAccountListS3Uri": { "type": "string", "pattern": "^(|s3://.+/.+)$" }, "threatIntelTtlDays": { "type": "integer", "minimum": 1, "maximum": 365 }, diff --git a/deploy/helm/khalifa/values.yaml b/deploy/helm/khalifa/values.yaml index ce2b644c..8b51cd36 100644 --- a/deploy/helm/khalifa/values.yaml +++ b/deploy/helm/khalifa/values.yaml @@ -82,6 +82,8 @@ config: accessAnalyzerTable: '' trafficObservationsTable: '' threatIndicatorsTable: '' + threatIntelCveListS3Uri: '' + threatIntelEpssListS3Uri: '' threatIntelIpListS3Uri: '' threatIntelAccountListS3Uri: '' threatIntelTtlDays: 90 diff --git a/deploy/terraform/khalifa-data/outputs.tf b/deploy/terraform/khalifa-data/outputs.tf index aad94946..c54842f8 100644 --- a/deploy/terraform/khalifa-data/outputs.tf +++ b/deploy/terraform/khalifa-data/outputs.tf @@ -96,6 +96,8 @@ output "helm_values" { accessAnalyzerTable = aws_dynamodb_table.access_analyzer.name trafficObservationsTable = aws_dynamodb_table.traffic_observations.name threatIndicatorsTable = aws_dynamodb_table.threat_indicators.name + threatIntelCveListS3Uri = var.threat_intel_cve_list_s3_uri + threatIntelEpssListS3Uri = var.threat_intel_epss_list_s3_uri threatIntelIpListS3Uri = var.threat_intel_ip_list_s3_uri threatIntelAccountListS3Uri = var.threat_intel_account_list_s3_uri threatIntelTtlDays = var.threat_intel_ttl_days diff --git a/deploy/terraform/khalifa-data/variables.tf b/deploy/terraform/khalifa-data/variables.tf index 4ae2baef..124b695d 100644 --- a/deploy/terraform/khalifa-data/variables.tf +++ b/deploy/terraform/khalifa-data/variables.tf @@ -194,6 +194,18 @@ variable "threat_intel_list_bucket" { default = "" } +variable "threat_intel_cve_list_s3_uri" { + description = "S3 URI of a plain CVE list (known-exploited membership, e.g. a CISA KEV export); empty disables it" + type = string + default = "" +} + +variable "threat_intel_epss_list_s3_uri" { + description = "S3 URI of an EPSS score list (lines of CVE-YYYY-NNNN,score); empty disables it" + type = string + default = "" +} + variable "threat_intel_ip_list_s3_uri" { description = "s3://bucket/key of the newline-separated IP indicator list. Empty disables that list." type = string diff --git a/lambdas/cloudtrail-analyzer/index.test.ts b/lambdas/cloudtrail-analyzer/index.test.ts index 46e98f1e..a84c2a8f 100644 --- a/lambdas/cloudtrail-analyzer/index.test.ts +++ b/lambdas/cloudtrail-analyzer/index.test.ts @@ -213,7 +213,56 @@ describe('CloudTrail analyzer', () => { }, }; const deniedRows = { ResultSet: { Rows: [anomalyHeader] } }; - const athenaSend = athena([usagePage, rootRows, secretRows, deniedRows]); + const detailHeader = [ + 'principal_arn', + 'source_ip', + 'event_count', + 'first_seen', + 'last_seen', + 'detail', + ].map((VarCharValue) => ({ VarCharValue })); + const newApiRows = { + ResultSet: { + Rows: [ + detailHeader, + { + Data: [ + 'arn:aws:iam::111122223333:user/deploy', + '203.0.113.9', + '4', + '2026-09-10T02:00:00Z', + '2026-09-10T02:30:00Z', + 'CreateUser, AttachUserPolicy', + ].map((VarCharValue) => ({ VarCharValue })), + }, + ], + }, + }; + const unusualHourRows = { + ResultSet: { + Rows: [ + detailHeader, + { + Data: [ + 'arn:aws:iam::111122223333:user/deploy', + '203.0.113.9', + '3', + '2026-09-10T03:00:00Z', + '2026-09-10T03:20:00Z', + '3', + ].map((VarCharValue) => ({ VarCharValue })), + }, + ], + }, + }; + const athenaSend = athena([ + usagePage, + rootRows, + secretRows, + newApiRows, + unusualHourRows, + deniedRows, + ]); type AnomalyNode = { id: string; properties: Record }; type AnomalyWriteBatch = { nodes: AnomalyNode[]; @@ -230,12 +279,12 @@ describe('CloudTrail analyzer', () => { { ...environment, GRAPH_WRITER_FUNCTION_NAME: 'graph-writer-fn' } ); - expect(result.anomalies).toBe(2); + expect(result.anomalies).toBe(4); const nodeBatches = writeAnomalies.mock.calls .map((call) => call[0]) .filter((batch) => batch.nodes.length > 0); - expect(nodeBatches).toHaveLength(2); + expect(nodeBatches).toHaveLength(4); const rootNode = nodeBatches[0].nodes[0]; expect(rootNode.properties.anomaly_type).toBe('ROOT_USAGE'); @@ -253,6 +302,14 @@ describe('CloudTrail analyzer', () => { expect(secretNode.properties.anomaly_type).toBe('SECRET_READ_NEW_SOURCE_IP'); expect(secretNode.properties.account_id).toBe('111122223333'); + const newApiNode = nodeBatches[2].nodes[0]; + expect(newApiNode.properties.anomaly_type).toBe('NEW_API_FOR_PRINCIPAL'); + expect(newApiNode.properties.detail).toBe('CreateUser, AttachUserPolicy'); + + const unusualHourNode = nodeBatches[3].nodes[0]; + expect(unusualHourNode.properties.anomaly_type).toBe('UNUSUAL_HOUR_ACTIVITY'); + expect(unusualHourNode.properties.detail).toBe('3'); + const reconcileBatch = writeAnomalies.mock.calls .map((call) => call[0]) .find((batch) => batch.reconcile); @@ -265,7 +322,7 @@ describe('CloudTrail analyzer', () => { const startedQueries = athenaSend.mock.calls.filter( ([command]) => command instanceof StartQueryExecutionCommand ); - expect(startedQueries).toHaveLength(4); + expect(startedQueries).toHaveLength(6); }); it('detects canary credential usage when canary principals are configured', async () => { @@ -294,7 +351,24 @@ describe('CloudTrail analyzer', () => { }, }; const emptyRows = { ResultSet: { Rows: [anomalyHeader] } }; - const athenaSend = athena([usagePage, emptyRows, emptyRows, emptyRows, canaryRows]); + const emptyDetailRows = { + ResultSet: { + Rows: [ + ['principal_arn', 'source_ip', 'event_count', 'first_seen', 'last_seen', 'detail'].map( + (VarCharValue) => ({ VarCharValue }) + ), + ], + }, + }; + const athenaSend = athena([ + usagePage, + emptyRows, + emptyRows, + emptyDetailRows, + emptyDetailRows, + emptyRows, + canaryRows, + ]); const writeAnomalies = jest.fn(async (batch: { nodes: unknown[] }) => ({ written: batch.nodes.length, })); @@ -338,7 +412,7 @@ describe('CloudTrail analyzer', () => { expect(canaryQuery).toBeDefined(); expect( athenaSend.mock.calls.filter(([c]) => c instanceof StartQueryExecutionCommand) - ).toHaveLength(5); + ).toHaveLength(7); }); it('rejects invalid canary ARNs before starting any queries', async () => { diff --git a/lambdas/cloudtrail-analyzer/src/anomalies.test.ts b/lambdas/cloudtrail-analyzer/src/anomalies.test.ts index 4de0f817..45b27b70 100644 --- a/lambdas/cloudtrail-analyzer/src/anomalies.test.ts +++ b/lambdas/cloudtrail-analyzer/src/anomalies.test.ts @@ -51,12 +51,14 @@ describe('buildAnomalyQueries input guards', () => { }); describe('buildAnomalyQueries', () => { - it('builds the three detection queries with the database and window bound in', () => { + it('builds the five detection queries with the database and window bound in', () => { const queries = buildAnomalyQueries(DATABASE, 90); expect(queries.map((q) => q.type)).toEqual([ 'ROOT_USAGE', 'SECRET_READ_NEW_SOURCE_IP', + 'NEW_API_FOR_PRINCIPAL', + 'UNUSUAL_HOUR_ACTIVITY', 'ACCESS_DENIED_SPIKE', ]); @@ -82,11 +84,74 @@ describe('buildAnomalyQueries', () => { // Denied spikes: daily counts for the baseline evaluator, no fixed // threshold in SQL. - expect(queries[2].sql).toContain("errorcode = 'AccessDenied'"); - expect(queries[2].sql).toContain('AS day'); - expect(queries[2].sql).toContain("'%Y-%m-%d'"); - expect(queries[2].sql).not.toContain('HAVING'); - expect(queries[2].sql).not.toContain(`>= ${ACCESS_DENIED_SPIKE_THRESHOLD}`); + expect(queries[4].sql).toContain("errorcode = 'AccessDenied'"); + expect(queries[4].sql).toContain('AS day'); + expect(queries[4].sql).toContain("'%Y-%m-%d'"); + expect(queries[4].sql).not.toContain('HAVING'); + expect(queries[4].sql).not.toContain(`>= ${ACCESS_DENIED_SPIKE_THRESHOLD}`); + }); + + it('parses a detail-bearing row and caps an oversized detail', () => { + const data = (values: string[]) => values.map((VarCharValue) => ({ VarCharValue })); + const record = parseAnomalyRow( + 'NEW_API_FOR_PRINCIPAL', + data([ + 'arn:aws:iam::111122223333:user/deploy', + '203.0.113.9', + '4', + '2026-09-10T02:00:00Z', + '2026-09-10T02:30:00Z', + 'CreateUser, AttachUserPolicy', + ]), + 2 + ); + expect(record.detail).toBe('CreateUser, AttachUserPolicy'); + + const long = parseAnomalyRow( + 'UNUSUAL_HOUR_ACTIVITY', + data([ + 'arn:aws:iam::111122223333:user/deploy', + '203.0.113.9', + '4', + '2026-09-10T02:00:00Z', + '2026-09-10T02:30:00Z', + 'x'.repeat(500), + ]), + 3 + ); + expect(long.detail).toHaveLength(200); + expect(long.detail?.endsWith('…')).toBe(true); + }); + + it('still requires five columns for the older query types', () => { + const data = (values: string[]) => values.map((VarCharValue) => ({ VarCharValue })); + expect(() => + parseAnomalyRow( + 'ROOT_USAGE', + data([ + 'arn:aws:iam::111122223333:root', + '203.0.113.50', + '5', + '2026-09-09T10:00:00Z', + '2026-09-10T10:00:00Z', + 'unexpected', + ]), + 2 + ) + ).toThrow('expected 5 columns'); + expect(() => + parseAnomalyRow( + 'NEW_API_FOR_PRINCIPAL', + data([ + 'arn:aws:iam::111122223333:user/deploy', + '203.0.113.9', + '4', + '2026-09-10T02:00:00Z', + '2026-09-10T02:30:00Z', + ]), + 2 + ) + ).toThrow('expected 6 columns'); }); it('adds a CANARY_USED query only when canary principals are configured', () => { @@ -96,17 +161,40 @@ describe('buildAnomalyQueries', () => { 'arn:aws:iam::111122223333:user/khalifa-canary-2', ]); - expect(queries).toHaveLength(4); - expect(queries[3].type).toBe('CANARY_USED'); - expect(queries[3].sql).toContain( + expect(queries).toHaveLength(6); + expect(queries[5].type).toBe('CANARY_USED'); + expect(queries[5].sql).toContain( "useridentity.arn IN ('arn:aws:iam::111122223333:user/khalifa-canary-1', 'arn:aws:iam::111122223333:user/khalifa-canary-2')" ); - expect(queries[3].sql).toContain("interval '90' day"); + expect(queries[5].sql).toContain("interval '90' day"); }); it('never emits a CANARY_USED query without canary principals', () => { - expect(buildAnomalyQueries(DATABASE, 90)).toHaveLength(3); - expect(buildAnomalyQueries(DATABASE, 90, [])).toHaveLength(3); + expect(buildAnomalyQueries(DATABASE, 90)).toHaveLength(5); + expect(buildAnomalyQueries(DATABASE, 90, [])).toHaveLength(5); + }); + + it('bounds the behavioural-breadth queries with their own baseline window', () => { + const queries = buildAnomalyQueries(DATABASE, 90); + const newApi = queries.find((query) => query.type === 'NEW_API_FOR_PRINCIPAL')!; + const hour = queries.find((query) => query.type === 'UNUSUAL_HOUR_ACTIVITY')!; + + for (const query of [newApi, hour]) { + expect(query.sql).toContain('WITH baseline_'); + expect(query.sql).toContain('SELECT DISTINCT'); + expect(query.sql).toContain("interval '7' day"); + expect(query.sql).toContain("interval '37' day"); + expect(query.sql).toContain('LEFT JOIN baseline_'); + expect(query.sql).toContain('WHERE baseline_'); + expect(query.sql).toContain( + "'elasticloadbalancing.amazonaws.com', 'monitoring.amazonaws.com', 'logs.amazonaws.com'" + ); + } + expect(newApi.sql).toContain('AND reads.eventname = baseline_events.eventname'); + expect(hour.sql).toContain('hour(from_iso8601_timestamp(eventtime)) AS activity_hour'); + expect(hour.sql).toContain( + 'array_join(array_agg(DISTINCT CAST(reads.activity_hour AS varchar))' + ); }); }); diff --git a/lambdas/cloudtrail-analyzer/src/anomalies.ts b/lambdas/cloudtrail-analyzer/src/anomalies.ts index 33a3e58b..5cb96e7a 100644 --- a/lambdas/cloudtrail-analyzer/src/anomalies.ts +++ b/lambdas/cloudtrail-analyzer/src/anomalies.ts @@ -10,7 +10,12 @@ import { export { ACCESS_DENIED_SPIKE_THRESHOLD }; export type AnomalyType = - 'ROOT_USAGE' | 'SECRET_READ_NEW_SOURCE_IP' | 'ACCESS_DENIED_SPIKE' | 'CANARY_USED'; + | 'ROOT_USAGE' + | 'SECRET_READ_NEW_SOURCE_IP' + | 'ACCESS_DENIED_SPIKE' + | 'CANARY_USED' + | 'NEW_API_FOR_PRINCIPAL' + | 'UNUSUAL_HOUR_ACTIVITY'; export interface AnomalyRecord { type: AnomalyType; @@ -26,6 +31,12 @@ export interface AnomalyRecord { basis?: 'baseline' | 'cold-start'; /** How many baseline standard deviations above the mean the spike is. */ deviation?: number; + /** + * The behaviour that made the record new, for the query types that carry it + * (NEW_API_FOR_PRINCIPAL: the API calls absent from the baseline; + * UNUSUAL_HOUR_ACTIVITY: the UTC hours the principal never used before). + */ + detail?: string; } export interface AnomalyQuery { @@ -37,6 +48,15 @@ const EXCLUDED_EVENT_SOURCES = `'elasticloadbalancing.amazonaws.com', 'monitorin export const SECRET_READ_NEW_IP_WINDOW_DAYS = 7; export const SECRET_READ_BASELINE_MARGIN_DAYS = 30; +// The behavioural-breadth queries (API mix, hour-of-day) share one window: +// seven days evaluated against the thirty before them. +export const UEBA_WINDOW_DAYS = 7; +export const UEBA_BASELINE_DAYS = 30; +export const MAX_ANOMALY_DETAIL = 200; +const QUERY_TYPES_WITH_DETAIL = new Set([ + 'NEW_API_FOR_PRINCIPAL', + 'UNUSUAL_HOUR_ACTIVITY', +]); const MAX_QUERY_DAYS = 365; export function assertAthenaDatabase(value: string): string { @@ -148,6 +168,97 @@ export function buildAnomalyQueries( ON reads.principal_arn = baseline_ips.principal_arn AND reads.source_ip = baseline_ips.source_ip WHERE baseline_ips.principal_arn IS NULL + `, + }, + { + type: 'NEW_API_FOR_PRINCIPAL', + // API mix: calls this principal made in the evaluation window that it + // never made in the baseline window. A service the principal never + // touched shows through its API calls (all of them are new). + sql: ` + WITH baseline_events AS ( + SELECT DISTINCT + useridentity.arn AS principal_arn, + eventname + FROM ${database}.cloudtrail_logs + WHERE ${partitionWindow(UEBA_WINDOW_DAYS + UEBA_BASELINE_DAYS)} + AND from_iso8601_timestamp(eventtime) >= current_timestamp - interval '${UEBA_WINDOW_DAYS + UEBA_BASELINE_DAYS}' day + AND from_iso8601_timestamp(eventtime) < current_timestamp - interval '${UEBA_WINDOW_DAYS}' day + AND useridentity.arn IS NOT NULL + AND useridentity.arn != '' + ) + SELECT + reads.principal_arn, + reads.source_ip, + COUNT(*) AS event_count, + from_iso8601_timestamp(MIN(reads.eventtime)) AS first_seen, + from_iso8601_timestamp(MAX(reads.eventtime)) AS last_seen, + array_join(array_agg(DISTINCT reads.eventname), ', ') AS detail + FROM ( + SELECT + useridentity.arn AS principal_arn, + sourceipaddress AS source_ip, + eventname, + eventtime + FROM ${database}.cloudtrail_logs + WHERE ${partitionWindow(UEBA_WINDOW_DAYS)} + AND from_iso8601_timestamp(eventtime) >= current_timestamp - interval '${UEBA_WINDOW_DAYS}' day + AND useridentity.arn IS NOT NULL + AND useridentity.arn != '' + AND sourceipaddress IS NOT NULL + AND eventsource NOT IN (${EXCLUDED_EVENT_SOURCES}) + ) reads + LEFT JOIN baseline_events + ON reads.principal_arn = baseline_events.principal_arn + AND reads.eventname = baseline_events.eventname + WHERE baseline_events.principal_arn IS NULL + GROUP BY reads.principal_arn, reads.source_ip + `, + }, + { + type: 'UNUSUAL_HOUR_ACTIVITY', + // Time of day: UTC hours in which the principal acted during the + // evaluation window but never during the baseline window. A principal + // running at 03:00 for the first time is either a human in a new + // timezone or automation that is not the principal's own. + sql: ` + WITH baseline_hours AS ( + SELECT DISTINCT + useridentity.arn AS principal_arn, + hour(from_iso8601_timestamp(eventtime)) AS activity_hour + FROM ${database}.cloudtrail_logs + WHERE ${partitionWindow(UEBA_WINDOW_DAYS + UEBA_BASELINE_DAYS)} + AND from_iso8601_timestamp(eventtime) >= current_timestamp - interval '${UEBA_WINDOW_DAYS + UEBA_BASELINE_DAYS}' day + AND from_iso8601_timestamp(eventtime) < current_timestamp - interval '${UEBA_WINDOW_DAYS}' day + AND useridentity.arn IS NOT NULL + AND useridentity.arn != '' + ) + SELECT + reads.principal_arn, + reads.source_ip, + COUNT(*) AS event_count, + from_iso8601_timestamp(MIN(reads.eventtime)) AS first_seen, + from_iso8601_timestamp(MAX(reads.eventtime)) AS last_seen, + array_join(array_agg(DISTINCT CAST(reads.activity_hour AS varchar)), ', ') AS detail + FROM ( + SELECT + useridentity.arn AS principal_arn, + sourceipaddress AS source_ip, + eventtime, + hour(from_iso8601_timestamp(eventtime)) AS activity_hour + FROM ${database}.cloudtrail_logs + WHERE ${partitionWindow(UEBA_WINDOW_DAYS)} + AND from_iso8601_timestamp(eventtime) >= current_timestamp - interval '${UEBA_WINDOW_DAYS}' day + AND useridentity.arn IS NOT NULL + AND useridentity.arn != '' + AND sourceipaddress IS NOT NULL + AND eventsource NOT IN (${EXCLUDED_EVENT_SOURCES}) + ) reads + LEFT JOIN baseline_hours + ON reads.principal_arn = baseline_hours.principal_arn + AND reads.activity_hour = baseline_hours.activity_hour + WHERE baseline_hours.principal_arn IS NULL + GROUP BY reads.principal_arn, reads.source_ip `, }, { @@ -210,8 +321,9 @@ export function parseAnomalyRow( data: { VarCharValue?: string }[], rowNumber: number ): AnomalyRecord { - if (!data || data.length !== 5) { - throw new Error(`Invalid Athena anomaly row ${rowNumber}: expected 5 columns`); + const expectedColumns = QUERY_TYPES_WITH_DETAIL.has(type) ? 6 : 5; + if (!data || data.length !== expectedColumns) { + throw new Error(`Invalid Athena anomaly row ${rowNumber}: expected ${expectedColumns} columns`); } const principalArn = requiredColumn(data[0].VarCharValue, 'principal_arn', rowNumber); @@ -236,7 +348,24 @@ export function parseAnomalyRow( } } - return { type, principalArn, sourceIp, eventCount, firstSeen, lastSeen }; + let detail: string | undefined; + if (QUERY_TYPES_WITH_DETAIL.has(type)) { + const rawDetail = requiredColumn(data[5].VarCharValue, 'detail', rowNumber); + detail = + rawDetail.length > MAX_ANOMALY_DETAIL + ? `${rawDetail.slice(0, MAX_ANOMALY_DETAIL - 1)}…` + : rawDetail; + } + + return { + type, + principalArn, + sourceIp, + eventCount, + firstSeen, + lastSeen, + ...(detail ? { detail } : {}), + }; } export function anomalyAccountId(principalArn: string): string { @@ -275,6 +404,7 @@ export function buildAnomalyNodes( ...(record.baselineDays !== undefined ? { baseline_days: record.baselineDays } : {}), ...(record.basis !== undefined ? { basis: record.basis } : {}), ...(record.deviation !== undefined ? { deviation: record.deviation } : {}), + ...(record.detail !== undefined ? { detail: record.detail } : {}), collection_scope: 'cloudtrail-anomalies', collection_run_id: runId, active: true, diff --git a/lambdas/threat-intel-import/index.test.ts b/lambdas/threat-intel-import/index.test.ts index 968727fc..da8adf83 100644 --- a/lambdas/threat-intel-import/index.test.ts +++ b/lambdas/threat-intel-import/index.test.ts @@ -71,7 +71,7 @@ describe('importThreatIntel', () => { it('requires at least one list', async () => { await expect(importThreatIntel({}, {}, {})).rejects.toThrow( - 'THREAT_INTEL_IP_LIST_S3_URI or THREAT_INTEL_ACCOUNT_LIST_S3_URI is required' + 'At least one of THREAT_INTEL_IP_LIST_S3_URI, THREAT_INTEL_ACCOUNT_LIST_S3_URI, THREAT_INTEL_CVE_LIST_S3_URI or THREAT_INTEL_EPSS_LIST_S3_URI is required' ); }); diff --git a/lambdas/threat-intel-import/index.ts b/lambdas/threat-intel-import/index.ts index c11c27c7..0b6b379c 100644 --- a/lambdas/threat-intel-import/index.ts +++ b/lambdas/threat-intel-import/index.ts @@ -70,12 +70,16 @@ function loadConfig(environment: NodeJS.ProcessEnv): ImportConfig { for (const [type, name] of [ ['ip', 'THREAT_INTEL_IP_LIST_S3_URI'], ['account', 'THREAT_INTEL_ACCOUNT_LIST_S3_URI'], + ['cve', 'THREAT_INTEL_CVE_LIST_S3_URI'], + ['epss', 'THREAT_INTEL_EPSS_LIST_S3_URI'], ] as const) { const uri = environment[name]?.trim(); if (uri) lists.push({ type, uri }); } if (lists.length === 0) { - throw new Error('THREAT_INTEL_IP_LIST_S3_URI or THREAT_INTEL_ACCOUNT_LIST_S3_URI is required'); + throw new Error( + 'At least one of THREAT_INTEL_IP_LIST_S3_URI, THREAT_INTEL_ACCOUNT_LIST_S3_URI, THREAT_INTEL_CVE_LIST_S3_URI or THREAT_INTEL_EPSS_LIST_S3_URI is required' + ); } return { tableName: environment.THREAT_INDICATORS_TABLE || 'ThreatIndicators', diff --git a/packages/risk-engine/src/attack-techniques.ts b/packages/risk-engine/src/attack-techniques.ts index 6663eab7..a854bedb 100644 --- a/packages/risk-engine/src/attack-techniques.ts +++ b/packages/risk-engine/src/attack-techniques.ts @@ -93,6 +93,8 @@ export const RULE_TECHNIQUES: Record = { 'RULE-032': ['T1552', 'T1199'], 'RULE-033': ['T1552', 'T1199'], 'RULE-034': ['T1078.004', 'T1098'], + 'RULE-035': ['T1078'], + 'RULE-036': ['T1078'], }; export function techniquesForRule(ruleId: string): AttackTechnique[] { diff --git a/packages/risk-engine/src/compliance-types.ts b/packages/risk-engine/src/compliance-types.ts index f60a596d..28fcbb80 100644 --- a/packages/risk-engine/src/compliance-types.ts +++ b/packages/risk-engine/src/compliance-types.ts @@ -1368,14 +1368,14 @@ export const KHALIFA_BASELINE_CONTROLS: ComplianceControl[] = [ id: 'KH-DET-1', framework: 'KHALIFA_BASELINE', section: 'Detection', - title: 'Ensure there are no unresolved credential-use anomalies', + title: 'Ensure there are no unresolved identity anomalies', description: - 'Reworded from "credential-use anomalies are detected", which its rules could not measure: they fire when an anomaly occurs, so zero was the healthy state and a finding read as a control failure. Stated as an outcome, each finding is a violation.', + 'Reworded from "credential-use anomalies are detected", which its rules could not measure: they fire when an anomaly occurs, so zero was the healthy state and a finding read as a control failure. Stated as an outcome, each finding is a violation. Identity anomalies cover credential use (secret reads, AccessDenied spikes) and behavioural drift (first-use APIs, unusual hours).', severity: 'critical', automated: true, - relatedRules: ['RULE-021', 'RULE-022'], + relatedRules: ['RULE-021', 'RULE-022', 'RULE-035', 'RULE-036'], evidenceRequirements: [ - 'Open CloudTrailAnomaly findings of type SECRET_READ_NEW_SOURCE_IP (RULE-021) or ACCESS_DENIED_SPIKE (RULE-022).', + 'Open CloudTrailAnomaly findings of type SECRET_READ_NEW_SOURCE_IP (RULE-021), ACCESS_DENIED_SPIKE (RULE-022), NEW_API_FOR_PRINCIPAL (RULE-035) or UNUSUAL_HOUR_ACTIVITY (RULE-036).', ], remediationGuidance: 'Investigate the principal and source IP; rotate the secret if the access was not expected.', diff --git a/packages/risk-engine/src/control-rule-mapping.test.ts b/packages/risk-engine/src/control-rule-mapping.test.ts index e285aa73..6595984c 100644 --- a/packages/risk-engine/src/control-rule-mapping.test.ts +++ b/packages/risk-engine/src/control-rule-mapping.test.ts @@ -107,6 +107,14 @@ const REVIEWED_MAPPING: Record = { why: 'unresolved AccessDenied spike; not 4.1, which asks for an alarm', }, 'RULE-023': { controls: ['KH-DET-2'], why: 'a canary credential has been used' }, + 'RULE-035': { + controls: ['KH-DET-1'], + why: 'unresolved first-use API activity; a lead, so auto-ticketing stays off', + }, + 'RULE-036': { + controls: ['KH-DET-1'], + why: 'unresolved unusual-hour activity; informational, never auto-ticketed', + }, 'RULE-024': { controls: ['5.6'], why: 'IMDSv1 instances, those holding a role' }, 'RULE-025': { controls: ['1.20'], why: 'Access Analyzer not ACTIVE, including missing' }, 'RULE-026': { controls: ['3.6'], why: 'customer CMK without rotation' }, diff --git a/packages/risk-engine/src/exploitability.ts b/packages/risk-engine/src/exploitability.ts new file mode 100644 index 00000000..0fb1def8 --- /dev/null +++ b/packages/risk-engine/src/exploitability.ts @@ -0,0 +1,24 @@ +import type { PathExploitability } from './types'; + +// One exploitability value shared by the likelihood and the score, so the two +// can never disagree: the CVSS base score scaled to 0..1, or the module's +// mid-value when the path carries no score at all; an EPSS probability takes +// the higher of the two (it is the same 0..1 scale by construction); a known +// exploit — Inspector's opinion *or* membership of an imported known-exploited +// list (KEV) — adds 0.2, clamped at 1. +const UNKNOWN = 0.5; + +export function exploitabilityValue(exploitability: PathExploitability | undefined): number { + if (!exploitability) return 0; + const cvss = exploitability.cvssBaseScore; + const cvssValue = typeof cvss === 'number' && Number.isFinite(cvss) ? cvss / 10 : UNKNOWN; + const epss = + typeof exploitability.epss === 'number' && Number.isFinite(exploitability.epss) + ? exploitability.epss + : 0; + let value = Math.max(cvssValue, epss); + if (exploitability.exploitAvailable) value += 0.2; + return Math.min(1, Math.max(0, value)); +} + +export { UNKNOWN as EXPLOITABILITY_UNKNOWN }; diff --git a/packages/risk-engine/src/likelihood.ts b/packages/risk-engine/src/likelihood.ts index 2d6e6536..b9d1b82a 100644 --- a/packages/risk-engine/src/likelihood.ts +++ b/packages/risk-engine/src/likelihood.ts @@ -1,4 +1,5 @@ import { coverageFactor, type CoverageLevel } from './coverage'; +import { exploitabilityValue } from './exploitability'; import type { RiskScoreInput } from './types'; // Explainable v1 path likelihood: the product of four independent 0..1 @@ -12,7 +13,7 @@ export interface PathLikelihoodFactors { exposure: number; /** Admin 1, wildcard 0.9, cross-account 0.7, service-linked 0.4, none 0.5. */ privilege: number; - /** CVSS/10, +0.2 when an exploit is known, capped at 1; no data 0.5. */ + /** max(CVSS/10, EPSS), +0.2 when an exploit is known, capped at 1; no data 0.5. */ exploitability: number; /** * 1 when exposure was derived (or observed traffic proves it), 0.5 when a @@ -59,13 +60,9 @@ export function pathLikelihood( const exposure = EXPOSURE[input.exposureLevel]; const privilege = input.identityBlastRadius ? PRIVILEGE[input.identityBlastRadius.type] : UNKNOWN; - let exploitability = UNKNOWN; - if (input.exploitability) { - const cvss = input.exploitability.cvssBaseScore; - let value = typeof cvss === 'number' && Number.isFinite(cvss) ? cvss / 10 : UNKNOWN; - if (input.exploitability.exploitAvailable) value += 0.2; - exploitability = Math.min(1, Math.max(0, value)); - } + // Shared with the score (exploitability.ts): CVSS or EPSS, +0.2 for a known + // exploit (Inspector or an imported KEV list). + const exploitability = input.exploitability ? exploitabilityValue(input.exploitability) : UNKNOWN; // Observed traffic (the flow-log analyzer's facts on a workload) proves // reachability even when a restricted read left the configuration unknown: diff --git a/packages/risk-engine/src/rules.test.ts b/packages/risk-engine/src/rules.test.ts index 82eb0f76..2ddb0aab 100644 --- a/packages/risk-engine/src/rules.test.ts +++ b/packages/risk-engine/src/rules.test.ts @@ -4,8 +4,8 @@ import { ADMIN_PORT_RANGE_PREDICATE } from './admin-ports'; const MANAGED_HOP = "optional(outE('CONTAINS').inV().hasLabel('IamPolicyDocument'))"; describe('riskRules', () => { - test('should have exactly 34 rules', () => { - expect(riskRules.length).toBe(34); + test('should have exactly 36 rules', () => { + expect(riskRules.length).toBe(36); }); test('each rule should have required fields', () => { @@ -68,7 +68,7 @@ describe('getEnabledRules', () => { test('should return 34 enabled rules', () => { const enabledRules = getEnabledRules(); - expect(enabledRules.length).toBe(34); + expect(enabledRules.length).toBe(36); }); }); @@ -138,6 +138,8 @@ describe('getRuleById', () => { ['RULE-021', 'Secrets Read', ['CloudTrailAnomaly', 'SECRET_READ_NEW_SOURCE_IP']], ['RULE-022', 'AccessDenied', ['CloudTrailAnomaly', 'ACCESS_DENIED_SPIKE']], ['RULE-023', 'Canary', ['CloudTrailAnomaly', 'CANARY_USED']], + ['RULE-035', 'Never Used Before', ['CloudTrailAnomaly', 'NEW_API_FOR_PRINCIPAL']], + ['RULE-036', 'Unusual Hour', ['CloudTrailAnomaly', 'UNUSUAL_HOUR_ACTIVITY']], ['RULE-024', 'IMDS', ['Ec2Instance', 'metadata_http_tokens', 'HAS_IAM_ROLE']], [ 'RULE-029', diff --git a/packages/risk-engine/src/rules.ts b/packages/risk-engine/src/rules.ts index ecabb962..9c93abe8 100644 --- a/packages/risk-engine/src/rules.ts +++ b/packages/risk-engine/src/rules.ts @@ -607,6 +607,44 @@ export const riskRules: RiskRule[] = [ priority: 'P0', }, }, + { + id: 'RULE-035', + name: 'Principal Used an API It Never Used Before', + description: + 'Detects a principal calling an API in the last seven days that it had not called in the thirty before (the analyzer records the new calls on the anomaly). First use of identity-, policy- or key-admin APIs is how a compromised identity persists; first use of benign APIs is common, so this is a lead and auto-ticketing stays off', + severityHint: 'medium', + riskFactors: [...baseRiskFactors], + gremlinQueryTemplate: ` + g.V().hasLabel('CloudTrailAnomaly') + .has('anomaly_type', 'NEW_API_FOR_PRINCIPAL') + .as('anomaly') + .${PATH_PROJECTION} + `, + ownerTeam: 'detection-response', + enabled: true, + autoTicketConfig: { + enabled: false, + }, + }, + { + id: 'RULE-036', + name: 'Principal Active at an Unusual Hour', + description: + 'Detects an identity acting in UTC hours it never used during the thirty-day baseline (the analyzer records the hours on the anomaly). Night-time activity from a daytime-only identity suggests automation with stolen credentials; humans moving timezones trip it too, so this is informational', + severityHint: 'low', + riskFactors: [...baseRiskFactors], + gremlinQueryTemplate: ` + g.V().hasLabel('CloudTrailAnomaly') + .has('anomaly_type', 'UNUSUAL_HOUR_ACTIVITY') + .as('anomaly') + .${PATH_PROJECTION} + `, + ownerTeam: 'detection-response', + enabled: true, + autoTicketConfig: { + enabled: false, + }, + }, { id: 'RULE-024', name: 'EC2 Instance with IMDSv1 and an IAM Role', diff --git a/packages/risk-engine/src/runner.test.ts b/packages/risk-engine/src/runner.test.ts index da6705e9..b211a4bf 100644 --- a/packages/risk-engine/src/runner.test.ts +++ b/packages/risk-engine/src/runner.test.ts @@ -233,6 +233,52 @@ describe('RiskRuleRunner stale-issue fix', () => { expect(created?.metadata.scoringFactors).toMatchObject({ threatIntelContribution: 10 }); }); + test('runRule threads KEV and EPSS matches into exploitability', async () => { + const { Path } = Gremlin.structure; + const { t } = Gremlin.process; + const element = (id: string, label: string, props: Record = {}) => + new Map([[t.id, id], [t.label, label], ...Object.entries(props)]); + const path = new Path( + ['i', 'cve'], + [ + element('i-1', 'Ec2Instance'), + element('cve-1', 'Vulnerability', { cve_id: ['CVE-2024-1234'] }), + ] + ); + const threatStore = { + find: jest.fn().mockResolvedValue([ + { type: 'cve', value: 'CVE-2024-1234', source: 'kev.txt' }, + { type: 'epss', value: 'CVE-2024-1234', source: 'epss.csv', score: 0.9 }, + ]), + }; + const runner = new RiskRuleRunner( + 'localhost:8182', + undefined, + undefined, + undefined, + threatStore as never + ); + const store = (runner as any).issueStore; + const neptune = (runner as any).neptuneClient; + neptune.executeQuery = jest.fn().mockResolvedValue([path]); + store.getIssueByRuleAndResources = jest.fn().mockResolvedValue(null); + store.getActiveIssuesByRule = jest.fn().mockResolvedValue([]); + let created: Issue | undefined; + store.createIssue = jest.fn().mockImplementation(async (issue: Issue) => { + created = issue; + return true; + }); + + await runner.runRule(rule); + + // EPSS 0.9 + 0.2 for known-exploited membership, clamped at 1. + expect(created?.metadata.likelihoodFactors).toMatchObject({ exploitability: 1 }); + expect(created?.metadata.threatIntelMatches).toEqual([ + { type: 'cve', value: 'CVE-2024-1234', source: 'kev.txt' }, + { type: 'epss', value: 'CVE-2024-1234', source: 'epss.csv', score: 0.9 }, + ]); + }); + test("runRule records the rule's ATT&CK techniques on the issue", async () => { const { Path } = Gremlin.structure; const { t } = Gremlin.process; diff --git a/packages/risk-engine/src/runner.ts b/packages/risk-engine/src/runner.ts index 632c6265..c965aa3e 100644 --- a/packages/risk-engine/src/runner.ts +++ b/packages/risk-engine/src/runner.ts @@ -489,8 +489,18 @@ export class RiskRuleRunner { private riskInputWithThreatIntel(path: RulePath, threatIntel: ThreatIndicator[]): RiskScoreInput { const riskInput = this.buildRiskInput(path.vertices); if (threatIntel.length === 0) return riskInput; + const knownExploited = threatIntel.some((indicator) => indicator.type === 'cve'); + const epss = threatIntel + .filter((indicator) => indicator.type === 'epss' && typeof indicator.score === 'number') + .reduce((max, indicator) => Math.max(max, indicator.score as number), 0); + const exploitability = { + ...riskInput.exploitability, + ...(knownExploited ? { exploitAvailable: true } : {}), + ...(epss > 0 ? { epss } : {}), + }; return { ...riskInput, + ...(Object.keys(exploitability).length > 0 ? { exploitability } : {}), threatIntel: { indicators: threatIntel.map((indicator) => `${indicator.type}:${indicator.value}`), }, diff --git a/packages/risk-engine/src/scoring.ts b/packages/risk-engine/src/scoring.ts index 622a1ebf..312bdaa2 100644 --- a/packages/risk-engine/src/scoring.ts +++ b/packages/risk-engine/src/scoring.ts @@ -1,3 +1,4 @@ +import { exploitabilityValue } from './exploitability'; import type { DataClassification, Environment, @@ -98,11 +99,8 @@ export function computeRiskScore(input: RiskScoreInput): RiskScoreOutput { let exploitabilityContribution = 0; if (input.exploitability) { - const cvss = input.exploitability.cvssBaseScore; - let value = typeof cvss === 'number' && Number.isFinite(cvss) ? cvss / 10 : 0.5; - if (input.exploitability.exploitAvailable) value += 0.2; - value = Math.min(1, Math.max(0, value)); - exploitabilityContribution = value * 100 * EXPLOITABILITY_WEIGHT; + exploitabilityContribution = + exploitabilityValue(input.exploitability) * 100 * EXPLOITABILITY_WEIGHT; } const totalScore = Math.min( @@ -222,6 +220,10 @@ export function getRemediationHint(ruleId: string, _context: Record 'Confirm who used root and why. If it was not approved break-glass use, rotate the root password, check root MFA, delete any root access keys and review the session in CloudTrail.', 'RULE-021': "Confirm the new source IP belongs to the principal's expected infrastructure. If not, rotate the secrets it read and the principal's credentials, and review its CloudTrail activity.", + 'RULE-035': + 'Review the new API calls on the anomaly. If they are administrative (user, policy, key or role changes), treat the identity as compromised until proven otherwise: rotate its credentials and check what the calls changed.', + 'RULE-036': + 'Confirm whether the identity is a human who changed timezone or automation that legitimately runs then. If neither, rotate its credentials and review the activity in the flagged hours.', 'RULE-022': "Review the principal's denied calls: a broken deployment or probing with stolen credentials. Disable the credentials if the activity is not expected.", 'RULE-023': diff --git a/packages/risk-engine/src/threat-intel.test.ts b/packages/risk-engine/src/threat-intel.test.ts index 1490d02e..410e0619 100644 --- a/packages/risk-engine/src/threat-intel.test.ts +++ b/packages/risk-engine/src/threat-intel.test.ts @@ -84,3 +84,64 @@ describe('pathIndicators', () => { expect(indicatorKey('ip', '203.0.113.7')).toBe('ip#203.0.113.7'); }); }); + +describe('CVE and EPSS indicators', () => { + it('accepts CVE ids (either case) for known-exploited and EPSS lists', () => { + expect(isValidIndicator('cve', 'CVE-2024-1234')).toBe(true); + expect(isValidIndicator('cve', 'cve-2024-12345')).toBe(true); + expect(isValidIndicator('epss', 'CVE-2021-44228')).toBe(true); + expect(isValidIndicator('cve', 'RHSA-2024:1234')).toBe(false); + expect(isValidIndicator('epss', 'CVE-24-1')).toBe(false); + }); + + it('parses EPSS lines as "cve,score" and uppercases ids', () => { + const parsed = parseIndicatorList( + [ + 'cve-2024-1234,0.97', + 'CVE-2021-44228, 0.5 # a comment', + 'CVE-2020-0001 # no score: unusable', + 'CVE-2019-0001,1.4 # out of range', + '', + ].join('\n'), + 'epss', + 's3://lists.example.com/epss.csv' + ); + + expect(parsed.indicators).toEqual([ + { + type: 'epss', + value: 'CVE-2024-1234', + source: 's3://lists.example.com/epss.csv', + score: 0.97, + }, + { + type: 'epss', + value: 'CVE-2021-44228', + source: 's3://lists.example.com/epss.csv', + score: 0.5, + }, + ]); + expect(parsed.invalid).toBe(2); + }); + + it('parses a plain known-exploited CVE list', () => { + const parsed = parseIndicatorList('CVE-2023-1111\nCVE-2023-1111\njunk\n', 'cve', 'kev.txt'); + expect(parsed.indicators).toEqual([{ type: 'cve', value: 'CVE-2023-1111', source: 'kev.txt' }]); + expect(parsed.invalid).toBe(1); + }); + + it('candidates a vulnerability CVE for both the KEV and EPSS lookups', () => { + const candidates = pathIndicators({ + vertices: [ + { + label: 'Vulnerability', + properties: { cve_id: 'cve-2024-9999' }, + }, + ], + }); + expect(candidates).toEqual([ + { type: 'cve', value: 'CVE-2024-9999' }, + { type: 'epss', value: 'CVE-2024-9999' }, + ]); + }); +}); diff --git a/packages/risk-engine/src/threat-intel.ts b/packages/risk-engine/src/threat-intel.ts index 1d060b77..50f1cb00 100644 --- a/packages/risk-engine/src/threat-intel.ts +++ b/packages/risk-engine/src/threat-intel.ts @@ -11,14 +11,19 @@ import { isIP } from 'node:net'; // per line, '#' comments). AWS-native intelligence needs no import: GuardDuty // applies AWS's managed reputation lists itself and the findings they drive // already feed RULE-018/RULE-029 — this store adds the customer's own view -// ("is this IP or account on our list?") so matches can change ranking. -export type ThreatIndicatorType = 'ip' | 'account'; +// ("is this IP, account, or CVE on our list?") so matches can change ranking. +// `cve` carries known-exploited membership (e.g. an export of the CISA KEV +// catalogue's CVE ids); `epss` carries a score per CVE ("CVE-2024-1234,0.97") +// and feeds the exploitability factor directly. +export type ThreatIndicatorType = 'ip' | 'account' | 'cve' | 'epss'; export interface ThreatIndicator { type: ThreatIndicatorType; value: string; /** Where the indicator came from, for the record (e.g. the list URI). */ source: string; + /** EPSS-indicator only: the probability score (0..1). */ + score?: number; } export const THREAT_INDICATOR_MAX_LINES = 50_000; @@ -31,8 +36,11 @@ export function indicatorKey(type: ThreatIndicatorType, value: string): string { return `${type}#${value}`; } +const CVE_PATTERN = /^CVE-\d{4}-\d{4,}$/i; + export function isValidIndicator(type: ThreatIndicatorType, value: string): boolean { if (type === 'ip') return isIP(value) !== 0; + if (type === 'cve' || type === 'epss') return CVE_PATTERN.test(value); return /^\d{12}$/.test(value); } @@ -59,25 +67,45 @@ export function parseIndicatorList( for (const rawLine of text.split('\n')) { const line = rawLine.split('#')[0].trim(); if (!line) continue; - if (!isValidIndicator(type, line)) { + const parsed = type === 'epss' ? parseEpssLine(line) : { value: line }; + if (!parsed || !isValidIndicator(type, parsed.value)) { invalid += 1; continue; } - const key = indicatorKey(type, line); + // CVE ids are case-insensitive identifiers; the store keeps one casing. + const value = parsed.value.toUpperCase(); + const key = indicatorKey(type, value); if (seen.has(key)) continue; seen.add(key); if (indicators.length >= THREAT_INDICATOR_MAX_LINES) { truncated = true; break; } - indicators.push({ type, value: line, source }); + indicators.push({ + type, + value, + source, + ...(parsed.score !== undefined ? { score: parsed.score } : {}), + }); } return { indicators, invalid, truncated }; } -// Graph facts the matcher joins on: an anomaly's source IP and a trusted -// external account. Kept here (not in the runner) so the join is testable and -// shared with anything else that reads the store. +// EPSS lines are "CVE-2024-1234,0.97" (a CSV's first two columns work as-is); +// a CVE without its score is not a usable EPSS indicator. +function parseEpssLine(line: string): { value: string; score?: number } | undefined { + const [rawValue, rawScore] = line.split(','); + const value = rawValue?.trim(); + if (!value || rawScore === undefined) return undefined; + const score = Number(rawScore.trim()); + if (!Number.isFinite(score) || score < 0 || score > 1) return undefined; + return { value, score }; +} + +// Graph facts the matcher joins on: an anomaly's source IP, a trusted +// external account, and a vulnerability's CVE id (candidate for both the +// known-exploited list and EPSS scores). Kept here (not in the runner) so the +// join is testable and shared with anything else that reads the store. export function pathIndicators(path: { vertices: Array<{ label: string; properties: Record }>; }): Array<{ type: ThreatIndicatorType; value: string }> { @@ -95,6 +123,14 @@ export function pathIndicators(path: { candidates.set(indicatorKey('account', account), { type: 'account', value: account }); } } + if (vertex.label === 'Vulnerability') { + const cve = vertex.properties.cve_id; + if (typeof cve === 'string' && isValidIndicator('cve', cve)) { + const value = cve.toUpperCase(); + candidates.set(indicatorKey('cve', value), { type: 'cve', value }); + candidates.set(indicatorKey('epss', value), { type: 'epss', value }); + } + } } return [...candidates.values()]; } @@ -130,6 +166,7 @@ export class ThreatIndicatorStore { type: indicator.type, value: indicator.value, source: indicator.source, + ...(indicator.score !== undefined ? { score: indicator.score } : {}), added_at: now.toISOString(), expires_at: expiresAt, }) as Record, @@ -175,7 +212,13 @@ export class ThreatIndicatorStore { const type = item.type?.S as ThreatIndicatorType | undefined; const value = item.value?.S; if (!type || !value) continue; - found.push({ type, value, source: item.source?.S ?? 'unknown' }); + const score = Number(item.score?.N); + found.push({ + type, + value, + source: item.source?.S ?? 'unknown', + ...(item.score?.N !== undefined && Number.isFinite(score) ? { score } : {}), + }); } } return found; diff --git a/packages/risk-engine/src/types.ts b/packages/risk-engine/src/types.ts index 08e7873f..be96a3fd 100644 --- a/packages/risk-engine/src/types.ts +++ b/packages/risk-engine/src/types.ts @@ -129,8 +129,10 @@ export interface IdentityBlastRadius { export interface PathExploitability { /** Highest CVSS base score among the path's vulnerability vertices. */ cvssBaseScore?: number; - /** An Amazon Inspector finding on the path reports a known exploit. */ + /** Inspector reports a known exploit, or an imported known-exploited (KEV) list does. */ exploitAvailable?: boolean; + /** Highest EPSS probability among the path's CVEs, from an imported score list. */ + epss?: number; } export interface RiskScoreInput { From ec161146fc29850fcf1f12865897e6da5cdef2c4 Mon Sep 17 00:00:00 2001 From: Derek Chamorro Date: Tue, 27 Oct 2026 22:36:18 -0500 Subject: [PATCH 2/2] Cover RULE-035/036 in the Gremlin graph tests rules.graph.test.ts's "every rule has a graph test" guard scans the file for rule ids; the two new anomaly rules need their real Gremlin coverage. Extend the anomaly-type matrix with both types and both rules. Why this is safe: test-only change; runs in the graph-queries job. --- packages/risk-engine/src/rules.graph.test.ts | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/packages/risk-engine/src/rules.graph.test.ts b/packages/risk-engine/src/rules.graph.test.ts index d97649fc..dc0153c9 100644 --- a/packages/risk-engine/src/rules.graph.test.ts +++ b/packages/risk-engine/src/rules.graph.test.ts @@ -662,12 +662,16 @@ describe('detection and posture rules', () => { ['RULE-021', 'SECRET_READ_NEW_SOURCE_IP'], ['RULE-022', 'ACCESS_DENIED_SPIKE'], ['RULE-023', 'CANARY_USED'], + ['RULE-035', 'NEW_API_FOR_PRINCIPAL'], + ['RULE-036', 'UNUSUAL_HOUR_ACTIVITY'], ])('%s matches only %s anomalies', async (ruleId, type) => { for (const anomaly of [ 'ROOT_USAGE', 'SECRET_READ_NEW_SOURCE_IP', 'ACCESS_DENIED_SPIKE', 'CANARY_USED', + 'NEW_API_FOR_PRINCIPAL', + 'UNUSUAL_HOUR_ACTIVITY', ]) { await vertex('CloudTrailAnomaly', `anomaly-${anomaly}`, { anomaly_type: anomaly }); }