Skip to content

Commit b1781da

Browse files
committed
Add lists, tables, and highlighter to chaos-engineering-failure-injection-low-cost-experiments articles
1 parent 3a29a4d commit b1781da

5 files changed

Lines changed: 114 additions & 46 deletions

File tree

‎_TODO.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -288,7 +288,7 @@ src/content/articles/internal-developer-portal-platform-self-service-actions/pdf
288288

289289
<Diagram
290290
magnifyButtonPosition="bottom-right"
291-
size="2xl"
291+
size="full"
292292
/>
293293

294294
[&>span]:
@@ -301,6 +301,7 @@ src/content/articles/internal-developer-portal-platform-self-service-actions/pdf
301301
/>
302302

303303
<Table
304+
fullWidth={false}
304305
classes={{
305306
figure: "sm:ml-8 sm:mr-12 mb-6",
306307
}}
Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
---
2+
export type Props = {
3+
color: string
4+
classes?: string
5+
size: number
6+
accessible: boolean
7+
focusable: boolean
8+
isListMarker?: boolean
9+
id?: string
10+
}
11+
12+
const { color, classes, size, accessible, focusable, isListMarker, id } = Astro.props
13+
---
14+
15+
<svg
16+
id={id}
17+
xmlns="http://www.w3.org/2000/svg"
18+
class:list={["shrink-0", `w-${size} h-${size}`, `text-${color}`, classes, isListMarker ? "mt-0.5" : ""]}
19+
fill="currentColor"
20+
stroke="none"
21+
viewBox="0 0 24 24"
22+
aria-hidden={accessible ? "false" : "true"}
23+
focusable={focusable ? "true" : "false"}
24+
>
25+
{accessible && <title>Availability Icon</title>}
26+
<path
27+
stroke-width="0.08"
28+
d="m 13.819602,13.80294 c 0,-0.0571 0.0028,-0.113611 0.0043,-0.170407 H 2.9158474 C 1.3080416,13.632533 0,14.940577 0,16.548382 c 0,1.607806 1.3080416,2.915847 2.9158474,2.915847 H 18.024709 C 15.594483,18.729208 13.819602,16.469843 13.819602,13.80294 Z M 6.1725083,17.305744 H 3.521738 c -0.41761,0 -0.7573632,-0.339753 -0.7573632,-0.757362 0,-0.417611 0.3397532,-0.757364 0.7573632,-0.757364 h 2.6507703 c 0.4176101,0 0.757363,0.339753 0.757363,0.757364 0,0.417609 -0.3397529,0.757362 -0.757363,0.757362 z"
29+
/>
30+
<path
31+
stroke-width="0.08"
32+
d="M 2.9158474,12.647961 H 13.933661 c 0.538789,-2.7098429 2.934251,-4.7590405 5.799962,-4.7590405 0.722524,0 1.414906,0.1307894 2.055407,0.3689862 C 21.281824,7.3961797 20.345042,6.8162678 19.274888,6.8162678 H 2.9158474 C 1.3080416,6.8162678 0,8.1243083 0,9.7321144 0,11.339921 1.3080416,12.647961 2.9158474,12.647961 Z M 3.521738,8.9747507 h 2.6507703 c 0.4176101,0 0.757363,0.339753 0.757363,0.7573637 0,0.4176106 -0.3397529,0.7573636 -0.757363,0.7573636 H 3.521738 c -0.41761,0 -0.7573632,-0.339753 -0.7573632,-0.7573636 0,-0.4176107 0.3397532,-0.7573637 0.7573632,-0.7573637 z"
33+
/>
34+
<path
35+
stroke-width="0.08"
36+
d="M 2.9158474,5.8316952 H 19.274888 c 1.607729,0 2.915848,-1.3080427 2.915848,-2.9158486 C 22.190736,1.3080407 20.882617,0 19.274888,0 H 2.9158474 C 1.3080416,0 0,1.3080407 0,2.9158466 0,4.5236525 1.3080416,5.8316952 2.9158474,5.8316952 Z M 17.987371,1.9502106 c 0.532426,0 0.965638,0.43321 0.965638,0.965636 0,0.5324281 -0.433212,0.9656381 -0.965638,0.9656381 -0.532426,0 -0.965638,-0.43321 -0.965638,-0.9656381 0,-0.532426 0.433212,-0.965636 0.965638,-0.965636 z M 3.521738,2.1584846 h 2.6507703 c 0.4176101,0 0.757363,0.3397534 0.757363,0.757362 0,0.4176107 -0.3397529,0.757364 -0.757363,0.757364 H 3.521738 c -0.41761,0 -0.7573632,-0.3397533 -0.7573632,-0.757364 0,-0.4176086 0.3397532,-0.757362 0.7573632,-0.757362 z"
37+
/>
38+
<path
39+
stroke-width="0.08"
40+
d="m 19.733623,9.5364876 c -2.356308,0 -4.266453,1.9101444 -4.266453,4.2664524 0,2.356309 1.910145,4.266454 4.266453,4.266454 C 22.089931,18.069394 24,16.159249 24,13.80294 24,11.446632 22.089931,9.5364876 19.733623,9.5364876 Z m 2.27936,3.3626164 -2.700303,2.700226 c -0.0571,0.05719 -0.133219,0.08876 -0.214182,0.08876 -0.08096,0 -0.157076,-0.03151 -0.214258,-0.08876 l -1.354164,-1.354165 c -0.05726,-0.05718 -0.08876,-0.133307 -0.08876,-0.214183 0,-0.08089 0.03151,-0.156996 0.08876,-0.214258 l 0.267727,-0.267727 c 0.0571,-0.05718 0.13322,-0.08876 0.214183,-0.08876 0.08088,0 0.157,0.03151 0.214182,0.08869 l 0.87233,0.872256 2.218392,-2.218241 c 0.0571,-0.05718 0.133225,-0.08876 0.214183,-0.08876 0.08095,0 0.157078,0.03159 0.214256,0.08876 l 0.267729,0.267727 c 0.05719,0.0571 0.08876,0.133225 0.08876,0.214183 -7.5e-5,0.08095 -0.03159,0.157078 -0.08884,0.214258 z"
41+
/>
42+
</svg>
Loading

‎src/content/articles/chaos-engineering-failure-injection-low-cost-experiments/index.mdx‎

Lines changed: 26 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -26,10 +26,11 @@ This article covers what chaos engineering actually is (hint: it's not random de
2626

2727
The term "chaos engineering" sounds destructive, which leads to a common misconception: that it's about randomly breaking things to see what happens. That's not chaos engineering — that's just chaos.
2828

29-
Real chaos engineering follows the scientific method. You form a hypothesis about how your system should behave under specific failure conditions, you design a controlled experiment to test that hypothesis, and you observe whether reality matches your expectations. The goal isn't to cause outages; it's to build confidence that your system handles failures gracefully — or to learn exactly how it doesn't.
29+
<Highlighter>Real chaos engineering follows the scientific method.</Highlighter> You form a hypothesis about how your system should behave under specific failure conditions, you design a controlled experiment to test that hypothesis, and you observe whether reality matches your expectations. The goal isn't to cause outages; it's to build confidence that your system handles failures gracefully — or to learn exactly how it doesn't.
3030

3131
<Table
32-
variant="vertical-column-delineation-table"
32+
variant="grid-and-accent-header-table"
33+
fullWidth={false}
3334
content={{
3435
figure: "The distinction that separates chaos engineering from breaking things.",
3536
thead: {
@@ -69,15 +70,27 @@ The simplest and most valuable first experiment is killing a pod. If you've neve
6970
**Prerequisites:**
7071

7172
<List
72-
variant="check-icons-list"
73+
variant="three-column-icon-list"
74+
classes={{
75+
text: "!text-base",
76+
}}
7377
items={[
7478
{
79+
icon: "database",
80+
color: "page-inverse",
81+
bgColor: "note-inverse",
7582
text: "Your deployment has replicas > 1 (so traffic continues on surviving pods during recovery)",
7683
},
7784
{
85+
icon: "availability",
86+
color: "page-inverse",
87+
bgColor: "note-inverse",
7888
text: "Health checks (liveness and readiness probes) are configured",
7989
},
8090
{
91+
icon: "observability-and-telemetry",
92+
color: "page-inverse",
93+
bgColor: "note-inverse",
8194
text: "You have monitoring in place to observe the experiment",
8295
},
8396
]}
@@ -107,6 +120,9 @@ The simplest and most valuable first experiment is killing a pod. If you've neve
107120

108121
<List
109122
variant="check-icons-list"
123+
classes={{
124+
wrapper: "sm:ml-6 sm:mr-12 mb-6",
125+
}}
110126
items={[
111127
{
112128
text: "New pod running within 60 seconds",
@@ -139,24 +155,26 @@ Code: Simple pod termination experiment with recovery observation.
139155
**What teams commonly discover from this experiment:**
140156

141157
<List
142-
variant="check-icons-list"
158+
variant="chevron-list"
159+
classes={{
160+
wrapper: "sm:ml-6 sm:mr-18 mb-6",
161+
}}
143162
items={[
144163
{
145-
lead: "Slow container startup delays recovery.",
164+
lead: "Slow container startup delays recovery",
146165
text: "If your container takes 45 seconds to start, that's 45 seconds of degraded capacity. Optimize image size, add startup probes, or adjust replica counts.",
147166
},
148167
{
149-
lead: "Missing readiness probes cause traffic to unhealthy pods.",
168+
lead: "Missing readiness probes cause traffic to unhealthy pods",
150169
text: "Without a readiness probe, Kubernetes sends traffic to pods that aren't ready to handle it, causing errors during startup.",
151170
},
152171
{
153-
lead: "No alerts fired despite degradation.",
172+
lead: "No alerts fired despite degradation",
154173
text: "This is the \"aha\" moment — your monitoring gap was invisible until you tested it.",
155174
},
156175
]}
157176
/>
158177

159-
160178
These findings alone justify the experiment. And you can run it in staging today, with no budget approval, no platform purchase, and no dedicated chaos team.
161179

162180
<Newsletter />

‎src/content/articles/chaos-engineering-failure-injection-low-cost-experiments/pdf.mdx‎

Lines changed: 44 additions & 37 deletions
Original file line numberDiff line numberDiff line change
@@ -47,7 +47,8 @@ The term "chaos engineering" sounds destructive, which leads to a common misconc
4747
Real chaos engineering follows the scientific method. You form a hypothesis about how your system should behave under specific failure conditions, design a controlled experiment to test that hypothesis, and observe whether reality matches your expectations. The goal isn't to cause outages; it's to build confidence that your system handles failures gracefully — or to learn exactly how it doesn't.
4848

4949
<Table
50-
variant="vertical-column-delineation-table"
50+
variant="grid-and-accent-header-table"
51+
fullWidth={false}
5152
content={{
5253
figure: "The distinction that separates real chaos engineering from breaking things randomly.",
5354
thead: {
@@ -84,6 +85,8 @@ The scientific cycle looks like this: form a hypothesis about expected behavior,
8485

8586
<Diagram
8687
src={theScientificCycleDiagram}
88+
magnifyButtonPosition="bottom-left"
89+
size="full"
8790
title="The scientific cycle of chaos engineering"
8891
description="Flowchart showing the chaos engineering loop: form a hypothesis, define steady state, plan the experiment, contain blast radius, run the experiment, observe behavior, and learn and improve before repeating the cycle."
8992
/>
@@ -142,6 +145,8 @@ From there, you can progress to experiments during business hours with real traf
142145
143146
<Diagram
144147
src={progressiveChaosDiagram}
148+
magnifyButtonPosition="top-left"
149+
size="full"
145150
title="Progressive chaos engineering maturity levels"
146151
description="Flowchart showing four maturity levels progressing from non-production experiments to low-risk production experiments, production experiments with live traffic, and finally automated chaos, with each level annotated by the kind of learning it enables."
147152
/>
@@ -415,33 +420,33 @@ Code: LitmusChaos network latency experiment.
415420
The tradeoff with LitmusChaos is complexity. You need to install the LitmusChaos operator, set up service accounts with appropriate permissions, and understand the CRD-based workflow. For teams already deep in Kubernetes and GitOps, this feels natural. For teams just starting with chaos engineering, Chaos Toolkit's simpler CLI-based approach might be easier to adopt.
416421

417422
<Table
418-
variant="vertical-column-delineation-table"
423+
variant="criteria-highlight-table"
419424
content={{
420425
figure: "Comparison of free chaos engineering tools.",
421426
thead: {
422-
th: ["Tool", "Best For", "Complexity", "Kubernetes Required"],
427+
th: ["Tool", "Complexity", "Best For", "Kubernetes Required"],
423428
},
424429
tbody: {
425430
tr: [
426431
{
427432
th: "kubectl",
428-
td: ["First experiments, simple pod/node chaos", "Low", "Yes"],
433+
td: ["Low", "First experiments, simple pod/node chaos", "Yes"],
429434
},
430435
{
431436
th: "tc/iptables",
432-
td: ["Network latency and packet loss", "Medium", "No (Linux only)"],
437+
td: ["Medium", "Network latency and packet loss", "No (Linux only)"],
433438
},
434439
{
435440
th: "toxiproxy",
436-
td: ["Surgical dependency failure injection", "Medium", "No"],
441+
td: ["Medium", "Surgical dependency failure injection", "No"],
437442
},
438443
{
439444
th: "Chaos Toolkit",
440-
td: ["Structured experiments with hypothesis validation", "Medium", "No (has K8s extension)"],
445+
td: ["Medium", "Structured experiments with hypothesis validation", "No (has K8s extension)"],
441446
},
442447
{
443448
th: "LitmusChaos",
444-
td: ["GitOps-managed, Kubernetes-native chaos", "High", "Yes"],
449+
td: ["High", "GitOps-managed, Kubernetes-native chaos", "Yes"],
445450
},
446451
],
447452
},
@@ -464,6 +469,7 @@ Impact limits define what customer-facing consequences you're willing to accept.
464469

465470
<Diagram
466471
src={threeDimensionsDiagram}
472+
magnifyButtonPosition="top-left"
467473
title="Three dimensions of blast radius control"
468474
description="Flowchart showing blast radius control divided into scope limiting, time limiting, and impact limiting, with examples such as targeting a single pod, setting maximum duration and automatic aborts, and starting with low-traffic non-critical paths."
469475
/>
@@ -654,24 +660,23 @@ Watch for the new pod scheduling, the health check passing, and traffic resuming
654660
What teams commonly discover from this experiment:
655661

656662
<List
657-
variant="check-icons-list"
663+
variant="zebra-list"
658664
items={[
659665
{
660-
lead: "_Slow container startup delays recovery._",
666+
lead: "Slow container startup delays recovery.",
661667
text: "If your container takes 45 seconds to start, that's 45 seconds of degraded capacity. Consider optimizing image size, adding startup probes, or adjusting replica counts.",
662668
},
663669
{
664-
lead: "_Missing readiness probes cause traffic to unhealthy pods._",
670+
lead: "Missing readiness probes cause traffic to unhealthy pods.",
665671
text: "Without a readiness probe, Kubernetes sends traffic to pods that aren't ready, causing errors during startup.",
666672
},
667673
{
668-
lead: "_No alerts fired despite degradation._",
674+
lead: "No alerts fired despite degradation.",
669675
text: "This is the \"aha\" moment — your monitoring gap was invisible until you tested it.",
670676
},
671677
]}
672678
/>
673679

674-
675680
### Experiment 2: Dependency Latency
676681

677682
Most production incidents aren't clean crashes — they're slowdowns. A database that responds in 500ms instead of 10ms is technically "up," but your service might as well be down if it can't handle the latency gracefully. This experiment reveals whether your timeouts, circuit breakers, and connection pools are configured correctly.
@@ -708,7 +713,7 @@ Success looks like: service remains responsive (P99 under 3 seconds), no cascadi
708713
What teams commonly discover:
709714

710715
<List
711-
variant="check-icons-list"
716+
variant="zebra-list"
712717
items={[
713718
{
714719
lead: "Missing timeouts cause request pileup.",
@@ -764,18 +769,18 @@ Success criteria: service continues responding (even if slowly), fallback to dat
764769
What teams commonly discover:
765770

766771
<List
767-
variant="check-icons-list"
772+
variant="zebra-list"
768773
items={[
769774
{
770-
lead: "_Hard dependency on cache, no fallback._",
775+
lead: "Hard dependency on cache, no fallback.",
771776
text: "The code assumes cache is always available and throws exceptions on connection failure instead of falling back.",
772777
},
773778
{
774-
lead: "_Timeouts too long, requests queue._",
779+
lead: "Timeouts too long, requests queue.",
775780
text: "Connection timeouts of 30 seconds mean requests pile up waiting for a cache that will never respond.",
776781
},
777782
{
778-
lead: "_Cache reconnection logic is buggy._",
783+
lead: "Cache reconnection logic is buggy.",
779784
text: "After the partition heals, the service doesn't reconnect properly and continues failing.",
780785
},
781786
]}
@@ -821,18 +826,18 @@ Success looks like: no OOM kills, graceful degradation where the service sheds e
821826
What teams commonly discover:
822827

823828
<List
824-
variant="check-icons-list"
829+
variant="zebra-list"
825830
items={[
826831
{
827-
lead: "_No memory limits set._",
832+
lead: "No memory limits set.",
828833
text: "Without limits, one pod's memory pressure affects the entire node — the classic noisy neighbor problem.",
829834
},
830835
{
831-
lead: "_OOM kill with no graceful handling._",
836+
lead: "OOM kill with no graceful handling.",
832837
text: "The application gets killed abruptly with no opportunity to drain connections or save state.",
833838
},
834839
{
835-
lead: "_Memory leaks exposed under pressure._",
840+
lead: "Memory leaks exposed under pressure.",
836841
text: "Baseline memory usage creeps up over time, and the stress test just accelerates the inevitable OOM.",
837842
},
838843
]}
@@ -850,6 +855,7 @@ A mature chaos practice looks different. Experiments run on a regular cadence
850855

851856
<Diagram
852857
src={evolutionDiagram}
858+
magnifyButtonPosition="top-left"
853859
title="Evolution from ad-hoc experiments to mature chaos practice"
854860
description="Flowchart showing a progression from ad-hoc experiments to regular experiments, then game days, and finally automated chaos, with each stage annotated by the learning or organizational capability it adds."
855861
/>
@@ -859,42 +865,42 @@ The progression takes time — typically a year or more to go from first experim
859865
Here's a realistic four-quarter roadmap for building a chaos engineering practice from scratch:
860866

861867
<Table
862-
variant="vertical-column-delineation-table"
868+
variant="criteria-highlight-table"
863869
content={{
864870
figure: "Four-quarter roadmap for building a chaos engineering practice.",
865871
thead: {
866-
th: ["Quarter", "Focus", "Activities", "Success Criteria"],
872+
th: ["Focus", "Quarter", "Activities", "Success Criteria"],
867873
},
868874
tbody: {
869875
tr: [
870876
{
871-
th: "Q1",
877+
th: "Establish basics",
872878
td: [
873-
"Establish basics",
879+
"Q1",
874880
"Run first pod-kill experiments, document and fix findings, train team on experiment design (hypothesis, abort conditions, stakeholder communication)",
875881
"3+ experiments completed, 5+ bugs fixed",
876882
],
877883
},
878884
{
879-
th: "Q2",
885+
th: "Expand coverage",
880886
td: [
881-
"Expand coverage",
887+
"Q2",
882888
"Move to latency injection and network partitions, create reusable experiment library, test all critical services",
883889
"Comprehensive critical path coverage, updated runbooks",
884890
],
885891
},
886892
{
887-
th: "Q3",
893+
th: "Build team capability",
888894
td: [
889-
"Build team capability",
895+
"Q3",
890896
"Host first game day, enable multiple teams to run experiments independently, share learnings org-wide",
891897
"Successful game day, 3+ teams practicing chaos",
892898
],
893899
},
894900
{
895-
th: "Q4",
901+
th: "Automate foundations",
896902
td: [
897-
"Automate foundations",
903+
"Q4",
898904
"Integrate chaos into CI/CD for staging deployments, automate steady-state verification, schedule recurring experiments",
899905
"Weekly automated experiments without manual intervention",
900906
],
@@ -960,7 +966,8 @@ Game days build team muscle memory for incident response. The goal isn't to find
960966
How do you know if your chaos practice is working? You need metrics that measure both activity (are you actually running experiments?) and impact (are you finding and fixing problems?).
961967

962968
<Table
963-
variant="vertical-column-delineation-table"
969+
variant="grid-and-accent-header-table"
970+
fullWidth={false}
964971
content={{
965972
figure: "Metrics for measuring chaos engineering practice effectiveness.",
966973
thead: {
@@ -1029,10 +1036,10 @@ Chaos engineering doesn't require expensive platforms or dedicated teams. You ca
10291036

10301037
The principles are simple: form a hypothesis before you break anything, define abort conditions so experiments don't become incidents, control the blast radius so you're learning rather than causing outages, and follow through on findings by actually fixing them.
10311038

1032-
Once you've mastered the four experiments in this guide, the path forward opens up. You can explore zone and region failures to test geographic redundancy. You can inject clock skew to find time-sensitive bugs. You can simulate certificate expiration, DNS failures, or cloud provider API throttling. Each new experiment type reveals a different class of bugs — and each one can be done with free tools.
1033-
1034-
The barrier to starting isn't tooling or budget. It's deciding to run that first experiment. Pick a service. Form a hypothesis. Open your monitoring dashboard. Kill a pod. Watch what happens.
1035-
10361039
<Callout type="success">
10371040
You don't need an enterprise chaos platform to start chaos engineering. You need a hypothesis, a safe experiment, monitoring to observe the results, and the discipline to fix what you find. Start small, learn, and expand.
10381041
</Callout>
1042+
1043+
Once you've mastered the four experiments in this guide, the path forward opens up. You can explore zone and region failures to test geographic redundancy. You can inject clock skew to find time-sensitive bugs. You can simulate certificate expiration, DNS failures, or cloud provider API throttling. Each new experiment type reveals a different class of bugs — and each one can be done with free tools.
1044+
1045+
The barrier to starting isn't tooling or budget. It's deciding to run that first experiment. Pick a service. Form a hypothesis. Open your monitoring dashboard. Kill a pod. Watch what happens.

0 commit comments

Comments
 (0)