ops(dev): enable VPC CNI NetworkPolicy on tb-client-dev-templates + audit EKS fleets [#102 prerequisite]

[saadqbal]

Context

tb-client-dev-templates (EKS, arn:aws:eks:eu-central-1:872515254613:cluster/tb-client-dev-templates) runs a self-managed VPC CNI with NetworkPolicy disabled (aws-eks-nodeagent --enable-network-policy=false). Confirmed empirically: a DNS-only egress NetworkPolicy does not block https://example.com from a pod. So the §8.2 egress lockdown (client-runtime#102) cannot enforce on this cluster today.

Flipping the DaemonSet flags alone is not enough — the policyendpoints.networking.k8s.aws CRD is present but there's no PolicyEndpoint controller running (it ships with the EKS managed vpc-cni addon, which this cluster doesn't use). A bare DS patch was tried and reverted (no effect).

What

1. Enable VPC CNI NetworkPolicy via the managed addon, in a maintenance window:

   aws eks create-addon --cluster-name tb-client-dev-templates --region eu-central-1 \
     --addon-name vpc-cni --configuration-values '{"enableNetworkPolicy":"true"}' \
     --resolve-conflicts OVERWRITE
   

   (adopts/rolls the self-managed CNI as a managed addon + activates the control-plane PolicyEndpoint controller).
2. Re-run the pre-flight egress probe — it must now go BLOCKED.
3. Audit the other EKS fleets for the same gap (self-managed VPC CNI / NetworkPolicy off) so the lockdown rollout isn't silently cosmetic anywhere.

Acceptance criteria

• Egress NetworkPolicy is enforced on tb-client-dev-templates (probe → BLOCKED).
• EKS fleets audited; any with NetworkPolicy disabled are listed for remediation.

Refs tracebloc/client-runtime#102. Owner: whoever manages the EKS clusters (infra).

[saadqbal]

✅ VPC CNI NetworkPolicy now enabled + confirmed enforcing on tb-client-dev-templates.

• vpc-cni managed addon v1.20.5-eksbuild.1, status ACTIVE, configurationValues={"enableNetworkPolicy":"true"}; aws-eks-nodeagent --enable-network-policy=true.

• Enforcement verified (exit-code probe in a throwaway ns): https://example.com REACHED with no policy → BLOCKED on the first check after a DNS-only egress NetworkPolicy was applied; DNS still resolves (precise egress deny, not a blanket block). The PolicyEndpoint controller (previously absent) now creates PolicyEndpoints.

• Egress NetworkPolicy is enforced on tb-client-dev-templates (probe → BLOCKED).

• Audit the other EKS fleets for the same gap (self-managed VPC CNI / NetworkPolicy off).

The cluster is now §8.2 lockdown-ready. Actually running the lockdown here still needs chart 1.7.0 on the cluster (not yet on gh-pages — release it, or a transient local-chart install).

[saadqbal]

✅ Lockdown validated end-to-end on dev EKS after the NP enablement. Transient client-1.7.0 upgrade of the tracebloc release (--reset-then-reuse-values + networkPolicy.training.enabled=true + routeWorkloads=true + allowExternalHttps=false), auto-upgrade cronjobs suspended for the window. A real tracebloc.io/workload: training-labelled pod under the chart's training-egress NetworkPolicy:

• direct → example.com / dev-api: BLOCKED (netpol; no 0.0.0.0/0 rule)
• via gateway → example.com: BLOCKED — squid TCP_DENIED/403
• via gateway → dev-api.tracebloc.io: REACHED — squid TCP_TUNNEL/200

So the §8.2 lockdown enforces correctly under EKS's VPC CNI eBPF/PolicyEndpoint engine (distinct from the k3s/kube-router path proven in client-runtime#102). Cluster rolled back to 1.6.1 (transient test); VPC CNI NetworkPolicy left enabled.

Remaining on this ticket: audit the other EKS fleets for the same enableNetworkPolicy gap.

[saadqbal]

EKS fleet audit (account 872515254613, eu-central-1)

Two tracebloc-managed EKS clusters:

cluster	vpc-cni	enableNetworkPolicy	enforces egress NP?
tb-client-dev-templates	managed addon v1.20.5	true	✅ yes (ready)
tracebloc-clients-prod	managed addon v1.19.2	unset → false	❌ NO

tracebloc-clients-prod (production) does not enforce egress NetworkPolicy → the §8.2 lockdown there would render but not block. It's a managed addon, so enabling is clean (no DaemonSet surgery):

aws eks update-addon --cluster-name tracebloc-clients-prod --region eu-central-1 \
  --addon-name vpc-cni --configuration-values '{"enableNetworkPolicy":"true"}' --resolve-conflicts OVERWRITE

Caveats before doing it: rolls aws-node cluster-wide (brief CNI restart per node) → maintenance window + owner approval; first confirm no existing NetworkPolicies would suddenly start enforcing; then re-probe (curl pod + DNS-only egress policy → example.com must fail).

Customer/edge fleets run on customer infrastructure (not this AWS account) → not auditable via AWS. The scalable readiness signal is the client self-reporting egress-enforcement status (client-runtime#104) — that's the real audit mechanism for the broader fleet.

• tb-client-dev-templates — enforces
• tracebloc-clients-prod — enable enableNetworkPolicy (windowed, owner-approved) before any lockdown flip
• customer fleets — self-report enforcement via client-runtime#104

[saadqbal]

Refinement after a read-only prod check:

• tracebloc-clients-prod hosts the hasan-prod + stg client releases — both already auto-upgraded to client-1.7.0 (rev 13, 12:23 UTC) with the egress-proxy deployed inert. Phase 1 (inert mechanism) has landed here, not just dev.
• No NetworkPolicies exist on the cluster → enabling enableNetworkPolicy is behavior-neutral (nothing suddenly enforces; only future netpols do). So the enablement risk reduces to the brief aws-node CNI roll — still windowed/owner-approved, but lower-risk than a cluster carrying live netpols.
• No rush: the prod lockdown flip is gated on client-runtime#104 (drain/sequencing) anyway, so enabling NP can be scheduled deliberately.