Add behavioural anomalies and EPSS/KEV enrichment (long tail, part 1)

ARCHITECTURE.md

@@ -91,15 +91,15 @@ flowchart LR
 
     subgraph DETECT["Detection pipelines"]
         direction TB
-        CTA["CloudTrail Analyzer (daily)<br/>usage profile + anomaly queries:<br/>ROOT_USAGE, SECRET_READ_NEW_SOURCE_IP,<br/>ACCESS_DENIED_SPIKE, CANARY_USED"]
+        CTA["CloudTrail Analyzer (daily)<br/>usage profile + anomaly queries:<br/>ROOT_USAGE, SECRET_READ_NEW_SOURCE_IP,<br/>ACCESS_DENIED_SPIKE, CANARY_USED,<br/>NEW_API_FOR_PRINCIPAL, UNUSUAL_HOUR_ACTIVITY"]
         DSPM["DSPM Scanner<br/>S3 objects + Lambda env vars<br/>(in-memory classification)"]
     end
 
     subgraph EVAL["Evaluation engines"]
         direction TB
         PE["Policy Evaluator (6h):<br/>effective permissions,<br/>escalation + pass-role paths"]
         LPS["Least-Privilege Sweep (daily):<br/>rightsizing vs CloudTrail usage"]
-        RE["Risk Engine (hourly):<br/>34 rules, attack paths, scoring"]
+        RE["Risk Engine (hourly):<br/>36 rules, attack paths, scoring"]
         COMP["Compliance:<br/>CIS v3.0.0 + Khalifa baseline<br/>(Security Hub reconciliation)"]
     end
 
@@ -292,18 +292,25 @@ deployment without the table simply has no observed traffic.
 ## Threat-intelligence matching
 
 Customer-supplied intelligence arrives as plain S3 lists (one indicator per
-line): `threat-intel-import` validates them (IPs v4/v6, 12-digit account ids),
-caps each list, and writes `ThreatIndicators` with a TTL — freshness is the
-TTL, so an indicator dropped from a list drains itself. Nothing here calls the
-internet: AWS's own managed reputation lists need no import because GuardDuty
-applies them and the findings they drive already feed RULE-018/RULE-029. The
-rule runner joins the store against the facts rules already match: an anomaly's
-`source_ip` and a trusted `ExternalAccount`'s account id. A match adds a flat
-**+10 threat-intelligence factor** to the risk score (a fact about the path,
-not a proportional strength; recorded as `threatIntelMatches` on the issue and
-labelled on the score card) and matches clear on refresh once the indicator is
-gone. Matched issues do **not** auto-ticket until `/rules/quality` shows the
-measured precision P6 requires; the local
+line): `threat-intel-import` validates them (IPs v4/v6, 12-digit account ids,
+CVE ids), caps each list, and writes `ThreatIndicators` with a TTL — freshness
+is the TTL, so an indicator dropped from a list drains itself. Four types:
+`ip` and `account`, plus `cve` (known-exploited membership — an export of the
+CISA KEV catalogue's CVE ids) and `epss` (lines of `CVE-YYYY-NNNN,score`,
+the first two columns of the FIRST EPSS CSV). Nothing here calls the internet:
+AWS's own managed reputation lists need no import because GuardDuty applies
+them and the findings they drive already feed RULE-018/RULE-029. The rule
+runner joins the store against the facts rules already match: an anomaly's
+`source_ip`, a trusted `ExternalAccount`'s account id, and a `Vulnerability`
+vertex's `cve_id` (candidate for both CVE lookups). An ip/account match adds a
+flat **+10 threat-intelligence factor** to the risk score (a fact about the
+path, not a proportional strength; recorded as `threatIntelMatches` on the
+issue and labelled on the score card); a `cve` match marks the exploit
+available and an `epss` score rides the shared exploitability value
+(`max(CVSS/10, EPSS)`, +0.2 for a known exploit — the same helper the
+likelihood and the score both use, so they cannot disagree). Matches clear on
+refresh once the indicator is gone. Matched issues do **not** auto-ticket
+until `/rules/quality` shows the measured precision P6 requires; the local
 `scripts/risk-plan` gate is unaffected (it has no store to read, so it never
 claims a match).
 

CHANGELOG.md

@@ -2,6 +2,24 @@
 
 All notable changes to Khalifa.
 
+## [Unreleased] - UEBA breadth and EPSS/KEV enrichment (long tail, part 1)
+
+### Added
+
+- **Behavioural anomalies in the CloudTrail analyzer**: two new bounded queries (7-day window against a 30-day baseline, same partition projection and exclusions as the existing three) — `NEW_API_FOR_PRINCIPAL` (API calls a principal never made in the baseline; a service it never touched shows through its calls) and `UNUSUAL_HOUR_ACTIVITY` (UTC hours the principal never used). Rows carry a `detail` column (the new APIs, or the hours; capped at 200 chars), the anomaly nodes record it, and two new medium/low rules (RULE-035/RULE-036) surface them — neither auto-tickets (behavioural leads need `/rules/quality` evidence first). ASN attribution is deliberately deferred: it needs an external IP→ASN dataset and a licensing decision, recorded in the plan.
+- **CVE and EPSS threat-intel types**: `threat-intel-import` now imports known-exploited CVE lists (`THREAT_INTEL_CVE_LIST_S3_URI`, e.g. a CISA KEV export) and EPSS score lists (`THREAT_INTEL_EPSS_LIST_S3_URI`, `CVE-YYYY-NNNN,score` — the FIRST CSV's first two columns). The runner candidates `Vulnerability` vertices' `cve_id` for both; a KEV match marks the exploit available and an EPSS score rides the exploitability factor — now one shared helper, `max(CVSS/10, EPSS)` +0.2 for a known exploit, used by both the likelihood and the score so they cannot disagree.
+
+### Load-bearing updates (explicit)
+
+- Rule catalogue **34 → 36 rules** (both enabled, auto-ticketing off): pinned counts in `rules.test.ts`, README and the ARCHITECTURE diagram updated.
+- `REVIEWED_MAPPING` (`control-rule-mapping.test.ts`) gains reviewed entries for RULE-035/RULE-036 (both → KH-DET-1, with reasons); `RULE_TECHNIQUES` gains T1078 for each; `getRemediationHint` gains both hints.
+- KH-DET-1 (`compliance-types.ts`) broadened from "credential-use anomalies" to "identity anomalies": `relatedRules` now includes RULE-035/RULE-036 and the evidence text names the new types (control counts unchanged).
+- Terraform (variables/outputs), chart values/schema/configmap and the OPERATIONAL threat-intel runbook carry the two new list URIs.
+
+### Notes
+
+- Still open from the re-review (`~/Documents/Testing/plans/2026-10-27-predictive-completion.md`): long-tail part 2 — trust-center publication of projections, per-issue risk window and SLO-burn thresholds, exposure-by-type series.
+
 ## [Unreleased] - Observed-traffic hop citations and shift-left v2
 
 ### Added

OPERATIONAL.md

@@ -207,10 +207,14 @@ kubectl create job --from=cronjob/khalifa-rule-runner rule-runner-manual -n secu
 
 1. Put plain lists in an account-owned S3 bucket: one indicator per line,
    `#` comments and blanks ignored. IPs (v4/v6) in one object, 12-digit
-   account ids in another.
+   account ids in another, CVE ids in a third, and EPSS scores in a fourth as
+   `CVE-YYYY-NNNN,score` (the first two columns of the FIRST EPSS CSV work
+   as-is; an export of the CISA KEV catalogue's CVE ids works for the CVE
+   list).
 2. In `khalifa-data`: set `threat_intel_list_bucket` (grants the rule-runner
-   role `s3:GetObject` on it) and `threat_intel_ip_list_s3_uri` /
-   `threat_intel_account_list_s3_uri` (flow into the chart's config), plus
+   role `s3:GetObject` on it) and any of `threat_intel_ip_list_s3_uri`,
+   `threat_intel_account_list_s3_uri`, `threat_intel_cve_list_s3_uri`,
+   `threat_intel_epss_list_s3_uri` (flow into the chart's config), plus
    `threat_intel_ttl_days` (default 90). Apply.
 3. Set `threatIntelImport.image` (the rule-runner digest) and unsuspend or
    run it once:
@@ -219,7 +223,9 @@ kubectl create job --from=cronjob/khalifa-rule-runner rule-runner-manual -n secu
    invalid lines are skipped and counted in the log.
 4. Verify: `ThreatIndicators` gains rows with a TTL; the next rule run scores
    matching anomaly IPs / trusted external accounts with the +10
-   threat-intelligence factor and records the match on the issue.
+   threat-intelligence factor, marks a path exploit-available when one of its
+   CVEs is on the known-exploited list, takes the higher of CVSS/EPSS for
+   exploitability, and records matches on the issue.
 5. Freshness is the TTL: an indicator dropped from the list expires by itself
    up to `threat_intel_ttl_days` after it was last imported. AWS's own managed
    reputation lists need no import (GuardDuty applies them; RULE-018/RULE-029
@@ -230,6 +236,7 @@ kubectl create job --from=cronjob/khalifa-rule-runner rule-runner-manual -n secu
 
 The recompute worker is rendered only when `collection.recomputeQueueUrl` is set
 (the complete example/Terraform wires it). It is always-on by design:
+
 - Producer: the incremental collector sends one message per five-minute window
   when a batch touched IAM. Watch `Queued an event-triggered recompute` in the
   incremental collector logs.