Agentic NetOps

Closed-loop AI network remediation on AWS. When a VPC drifts from its declared intent, the platform detects it, proves it, measures the impact, drafts a repair, routes it to the right sign-off, executes it under scoped credentials and verifies the fix. Every decision lands in a tamper-evident ledger.

Note

The code is private, and I'm happy to demo it. On a call I can walk through the code and run the replay of the live deployment, which replays the recorded incidents, re-checks their hash chains in the browser and shows a forged entry getting caught. Email haeganm@gmail.com or message me on LinkedIn.

The idea

The LLM proposes, deterministic systems dispose. The model drafts diagnoses and repair rationales, and its guess at the fault class is scored against a deterministic classifier, never trusted. Control flow runs on config diffs, so a wrong model answer can't change what gets executed.

DETECT → PROVE → MEASURE → GENERATE-REPAIR → APPROVE → EXECUTE → VERIFY

flowchart LR
    CT[CloudTrail<br/>write-only mgmt events] --> EB[EventBridge] --> SQS[SQS + DLQ<br/>30s debounce] --> DET[detector λ]
    DET --> SFN{{Step Functions<br/>incident loop}}
    SFN --> PR[PROVE<br/>baseline differ]
    SFN --> ME[MEASURE<br/>CFN drift + Reachability Analyzer + in-VPC probe + LLM]
    SFN --> GR[GENERATE-REPAIR<br/>converge-to-baseline planner + policy gate]
    GR --> TR{TierRouter<br/>deterministic risk tier}
    TR -->|LOW| VW[veto window ~60s<br/>auto-executes unless cancelled]
    TR -->|MEDIUM| A1[one approval]
    TR -->|HIGH| A2[two DISTINCT approvers<br/>maker-checker]
    VW --> EX[EXECUTE<br/>role ∩ boundary ∩ session policy]
    A1 --> EX
    A2 --> EX
    EX --> VF[VERIFY<br/>re-diff empty + oracles re-run]
    VF --> AN[anchor chain head<br/>out-of-band SNS]
    SFN --> DDB[(DynamoDB<br/>hash-chained decision ledger)]

Loading

Measured on the live deployment

The stack ran in July 2026 and is now torn down to a $0 bill. Every decision it made was exported before teardown.

• LOW tier: a seeded fault self-healed in 191 s. The veto window passed with nobody intervening, so no human touched it.
• MEDIUM tier: seed to resolved in 129 s, with one approval.
• HIGH tier: the same user's second approval was refused with a 403, and a distinct second approver resolved it.
• All 9 seeded fault classes remediate.
• 70 incidents were exported. All 34 that carry a hash chain verify, and 31 of them match a head anchored out of band. The other 36 were recorded before the chain existed.

Design decisions

• Converge to baseline, nothing else. A versioned policy gate bounds every repair to restoring the declared state, so a free-form fix is structurally impossible.
• Autonomy tiers that fail closed. A deterministic risk tier decides who signs off: LOW auto-executes behind a 60 s veto window, MEDIUM takes one approval, HIGH needs two distinct approvers. A kill switch sends everything to a human.
• Segregation of duties as recusal. Whoever caused the drift is removed from approve, reject and veto on their own change. Approve-only SoD would still let them reject or veto the fix, which keeps the hole open.
• Least privilege per incident. The executor runs on STS credentials scoped to the intersection of its role, a permissions boundary and a session policy listing only the approved plan's ARNs.
• Tamper-evident ledger. Each decision is a link in a per-incident SHA-256 chain, and the head is anchored out of band through SNS, so a chain re-hashed by an insider with table access still fails the check.
• $0 idle. No NAT, no EC2, no always-on compute. The one metered check, Reachability Analyzer, is hard-capped in code and fails closed. A teardown check proves nothing billable is left.

Engineering

• 301 tests in credential-free CI, moto-mocked, including 11 headless-browser tests of the replay and a standing red-team suite. One test forges every hashed field of every preserved entry, 1,836 forgeries, and requires each to be caught at its own sequence number.
• 18 architecture decision records, and a security findings register from repeated adversarial audits, one of them run against the deployed stack.
• Live testing caught what mocks couldn't: a green 9/9 eval once coexisted with every console approval returning 500, because the eval harness completed approvals with admin credentials. A regression test now guards the IAM that caused it.
• The last audit found operators could rewrite their own Cognito email, the claim recusal and two-party approval key on. The template now allows one harmless writable attribute, and a test pins it.

Stack

AWS SAM Step Functions Lambda DynamoDB EventBridge SQS SNS CloudTrail Cognito Bedrock IAM CloudFront VPC Reachability Analyzer Python