You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit b1781da
Browse filesBrowse the repository at this point in the historyBrowse files
d="m 13.819602,13.80294 c 0,-0.0571 0.0028,-0.113611 0.0043,-0.170407 H 2.9158474 C 1.3080416,13.632533 0,14.940577 0,16.548382 c 0,1.607806 1.3080416,2.915847 2.9158474,2.915847 H 18.024709 C 15.594483,18.729208 13.819602,16.469843 13.819602,13.80294 Z M 6.1725083,17.305744 H 3.521738 c -0.41761,0 -0.7573632,-0.339753 -0.7573632,-0.757362 0,-0.417611 0.3397532,-0.757364 0.7573632,-0.757364 h 2.6507703 c 0.4176101,0 0.757363,0.339753 0.757363,0.757364 0,0.417609 -0.3397529,0.757362 -0.757363,0.757362 z"
29
+
/>
30
+
<path
31
+
stroke-width="0.08"
32
+
d="M 2.9158474,12.647961 H 13.933661 c 0.538789,-2.7098429 2.934251,-4.7590405 5.799962,-4.7590405 0.722524,0 1.414906,0.1307894 2.055407,0.3689862 C 21.281824,7.3961797 20.345042,6.8162678 19.274888,6.8162678 H 2.9158474 C 1.3080416,6.8162678 0,8.1243083 0,9.7321144 0,11.339921 1.3080416,12.647961 2.9158474,12.647961 Z M 3.521738,8.9747507 h 2.6507703 c 0.4176101,0 0.757363,0.339753 0.757363,0.7573637 0,0.4176106 -0.3397529,0.7573636 -0.757363,0.7573636 H 3.521738 c -0.41761,0 -0.7573632,-0.339753 -0.7573632,-0.7573636 0,-0.4176107 0.3397532,-0.7573637 0.7573632,-0.7573637 z"
33
+
/>
34
+
<path
35
+
stroke-width="0.08"
36
+
d="M 2.9158474,5.8316952 H 19.274888 c 1.607729,0 2.915848,-1.3080427 2.915848,-2.9158486 C 22.190736,1.3080407 20.882617,0 19.274888,0 H 2.9158474 C 1.3080416,0 0,1.3080407 0,2.9158466 0,4.5236525 1.3080416,5.8316952 2.9158474,5.8316952 Z M 17.987371,1.9502106 c 0.532426,0 0.965638,0.43321 0.965638,0.965636 0,0.5324281 -0.433212,0.9656381 -0.965638,0.9656381 -0.532426,0 -0.965638,-0.43321 -0.965638,-0.9656381 0,-0.532426 0.433212,-0.965636 0.965638,-0.965636 z M 3.521738,2.1584846 h 2.6507703 c 0.4176101,0 0.757363,0.3397534 0.757363,0.757362 0,0.4176107 -0.3397529,0.757364 -0.757363,0.757364 H 3.521738 c -0.41761,0 -0.7573632,-0.3397533 -0.7573632,-0.757364 0,-0.4176086 0.3397532,-0.757362 0.7573632,-0.757362 z"
37
+
/>
38
+
<path
39
+
stroke-width="0.08"
40
+
d="m 19.733623,9.5364876 c -2.356308,0 -4.266453,1.9101444 -4.266453,4.2664524 0,2.356309 1.910145,4.266454 4.266453,4.266454 C 22.089931,18.069394 24,16.159249 24,13.80294 24,11.446632 22.089931,9.5364876 19.733623,9.5364876 Z m 2.27936,3.3626164 -2.700303,2.700226 c -0.0571,0.05719 -0.133219,0.08876 -0.214182,0.08876 -0.08096,0 -0.157076,-0.03151 -0.214258,-0.08876 l -1.354164,-1.354165 c -0.05726,-0.05718 -0.08876,-0.133307 -0.08876,-0.214183 0,-0.08089 0.03151,-0.156996 0.08876,-0.214258 l 0.267727,-0.267727 c 0.0571,-0.05718 0.13322,-0.08876 0.214183,-0.08876 0.08088,0 0.157,0.03151 0.214182,0.08869 l 0.87233,0.872256 2.218392,-2.218241 c 0.0571,-0.05718 0.133225,-0.08876 0.214183,-0.08876 0.08095,0 0.157078,0.03159 0.214256,0.08876 l 0.267729,0.267727 c 0.05719,0.0571 0.08876,0.133225 0.08876,0.214183 -7.5e-5,0.08095 -0.03159,0.157078 -0.08884,0.214258 z"
Copy file name to clipboardExpand all lines: src/content/articles/chaos-engineering-failure-injection-low-cost-experiments/index.mdx
+26-8Lines changed: 26 additions & 8 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -26,10 +26,11 @@ This article covers what chaos engineering actually is (hint: it's not random de
26
26
27
27
The term "chaos engineering" sounds destructive, which leads to a common misconception: that it's about randomly breaking things to see what happens. That's not chaos engineering — that's just chaos.
28
28
29
-
Real chaos engineering follows the scientific method. You form a hypothesis about how your system should behave under specific failure conditions, you design a controlled experiment to test that hypothesis, and you observe whether reality matches your expectations. The goal isn't to cause outages; it's to build confidence that your system handles failures gracefully — or to learn exactly how it doesn't.
29
+
<Highlighter>Real chaos engineering follows the scientific method.</Highlighter> You form a hypothesis about how your system should behave under specific failure conditions, you design a controlled experiment to test that hypothesis, and you observe whether reality matches your expectations. The goal isn't to cause outages; it's to build confidence that your system handles failures gracefully — or to learn exactly how it doesn't.
30
30
31
31
<Table
32
-
variant="vertical-column-delineation-table"
32
+
variant="grid-and-accent-header-table"
33
+
fullWidth={false}
33
34
content={{
34
35
figure: "The distinction that separates chaos engineering from breaking things.",
35
36
thead: {
@@ -69,15 +70,27 @@ The simplest and most valuable first experiment is killing a pod. If you've neve
69
70
**Prerequisites:**
70
71
71
72
<List
72
-
variant="check-icons-list"
73
+
variant="three-column-icon-list"
74
+
classes={{
75
+
text: "!text-base",
76
+
}}
73
77
items={[
74
78
{
79
+
icon: "database",
80
+
color: "page-inverse",
81
+
bgColor: "note-inverse",
75
82
text: "Your deployment has replicas > 1 (so traffic continues on surviving pods during recovery)",
76
83
},
77
84
{
85
+
icon: "availability",
86
+
color: "page-inverse",
87
+
bgColor: "note-inverse",
78
88
text: "Health checks (liveness and readiness probes) are configured",
79
89
},
80
90
{
91
+
icon: "observability-and-telemetry",
92
+
color: "page-inverse",
93
+
bgColor: "note-inverse",
81
94
text: "You have monitoring in place to observe the experiment",
82
95
},
83
96
]}
@@ -107,6 +120,9 @@ The simplest and most valuable first experiment is killing a pod. If you've neve
107
120
108
121
<List
109
122
variant="check-icons-list"
123
+
classes={{
124
+
wrapper: "sm:ml-6 sm:mr-12 mb-6",
125
+
}}
110
126
items={[
111
127
{
112
128
text: "New pod running within 60 seconds",
@@ -139,24 +155,26 @@ Code: Simple pod termination experiment with recovery observation.
139
155
**What teams commonly discover from this experiment:**
140
156
141
157
<List
142
-
variant="check-icons-list"
158
+
variant="chevron-list"
159
+
classes={{
160
+
wrapper: "sm:ml-6 sm:mr-18 mb-6",
161
+
}}
143
162
items={[
144
163
{
145
-
lead: "Slow container startup delays recovery.",
164
+
lead: "Slow container startup delays recovery",
146
165
text: "If your container takes 45 seconds to start, that's 45 seconds of degraded capacity. Optimize image size, add startup probes, or adjust replica counts.",
147
166
},
148
167
{
149
-
lead: "Missing readiness probes cause traffic to unhealthy pods.",
168
+
lead: "Missing readiness probes cause traffic to unhealthy pods",
150
169
text: "Without a readiness probe, Kubernetes sends traffic to pods that aren't ready to handle it, causing errors during startup.",
151
170
},
152
171
{
153
-
lead: "No alerts fired despite degradation.",
172
+
lead: "No alerts fired despite degradation",
154
173
text: "This is the \"aha\" moment — your monitoring gap was invisible until you tested it.",
155
174
},
156
175
]}
157
176
/>
158
177
159
-
160
178
These findings alone justify the experiment. And you can run it in staging today, with no budget approval, no platform purchase, and no dedicated chaos team.
Copy file name to clipboardExpand all lines: src/content/articles/chaos-engineering-failure-injection-low-cost-experiments/pdf.mdx
+44-37Lines changed: 44 additions & 37 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -47,7 +47,8 @@ The term "chaos engineering" sounds destructive, which leads to a common misconc
47
47
Real chaos engineering follows the scientific method. You form a hypothesis about how your system should behave under specific failure conditions, design a controlled experiment to test that hypothesis, and observe whether reality matches your expectations. The goal isn't to cause outages; it's to build confidence that your system handles failures gracefully — or to learn exactly how it doesn't.
48
48
49
49
<Table
50
-
variant="vertical-column-delineation-table"
50
+
variant="grid-and-accent-header-table"
51
+
fullWidth={false}
51
52
content={{
52
53
figure: "The distinction that separates real chaos engineering from breaking things randomly.",
53
54
thead: {
@@ -84,6 +85,8 @@ The scientific cycle looks like this: form a hypothesis about expected behavior,
84
85
85
86
<Diagram
86
87
src={theScientificCycleDiagram}
88
+
magnifyButtonPosition="bottom-left"
89
+
size="full"
87
90
title="The scientific cycle of chaos engineering"
88
91
description="Flowchart showing the chaos engineering loop: form a hypothesis, define steady state, plan the experiment, contain blast radius, run the experiment, observe behavior, and learn and improve before repeating the cycle."
89
92
/>
@@ -142,6 +145,8 @@ From there, you can progress to experiments during business hours with real traf
description="Flowchart showing four maturity levels progressing from non-production experiments to low-risk production experiments, production experiments with live traffic, and finally automated chaos, with each level annotated by the kind of learning it enables."
The tradeoff with LitmusChaos is complexity. You need to install the LitmusChaos operator, set up service accounts with appropriate permissions, and understand the CRD-based workflow. For teams already deep in Kubernetes and GitOps, this feels natural. For teams just starting with chaos engineering, Chaos Toolkit's simpler CLI-based approach might be easier to adopt.
416
421
417
422
<Table
418
-
variant="vertical-column-delineation-table"
423
+
variant="criteria-highlight-table"
419
424
content={{
420
425
figure: "Comparison of free chaos engineering tools.",
@@ -464,6 +469,7 @@ Impact limits define what customer-facing consequences you're willing to accept.
464
469
465
470
<Diagram
466
471
src={threeDimensionsDiagram}
472
+
magnifyButtonPosition="top-left"
467
473
title="Three dimensions of blast radius control"
468
474
description="Flowchart showing blast radius control divided into scope limiting, time limiting, and impact limiting, with examples such as targeting a single pod, setting maximum duration and automatic aborts, and starting with low-traffic non-critical paths."
469
475
/>
@@ -654,24 +660,23 @@ Watch for the new pod scheduling, the health check passing, and traffic resuming
654
660
What teams commonly discover from this experiment:
text: "If your container takes 45 seconds to start, that's 45 seconds of degraded capacity. Consider optimizing image size, adding startup probes, or adjusting replica counts.",
662
668
},
663
669
{
664
-
lead: "_Missing readiness probes cause traffic to unhealthy pods._",
670
+
lead: "Missing readiness probes cause traffic to unhealthy pods.",
665
671
text: "Without a readiness probe, Kubernetes sends traffic to pods that aren't ready, causing errors during startup.",
666
672
},
667
673
{
668
-
lead: "_No alerts fired despite degradation._",
674
+
lead: "No alerts fired despite degradation.",
669
675
text: "This is the \"aha\" moment — your monitoring gap was invisible until you tested it.",
670
676
},
671
677
]}
672
678
/>
673
679
674
-
675
680
### Experiment 2: Dependency Latency
676
681
677
682
Most production incidents aren't clean crashes — they're slowdowns. A database that responds in 500ms instead of 10ms is technically "up," but your service might as well be down if it can't handle the latency gracefully. This experiment reveals whether your timeouts, circuit breakers, and connection pools are configured correctly.
@@ -708,7 +713,7 @@ Success looks like: service remains responsive (P99 under 3 seconds), no cascadi
708
713
What teams commonly discover:
709
714
710
715
<List
711
-
variant="check-icons-list"
716
+
variant="zebra-list"
712
717
items={[
713
718
{
714
719
lead: "Missing timeouts cause request pileup.",
@@ -764,18 +769,18 @@ Success criteria: service continues responding (even if slowly), fallback to dat
764
769
What teams commonly discover:
765
770
766
771
<List
767
-
variant="check-icons-list"
772
+
variant="zebra-list"
768
773
items={[
769
774
{
770
-
lead: "_Hard dependency on cache, no fallback._",
775
+
lead: "Hard dependency on cache, no fallback.",
771
776
text: "The code assumes cache is always available and throws exceptions on connection failure instead of falling back.",
772
777
},
773
778
{
774
-
lead: "_Timeouts too long, requests queue._",
779
+
lead: "Timeouts too long, requests queue.",
775
780
text: "Connection timeouts of 30 seconds mean requests pile up waiting for a cache that will never respond.",
776
781
},
777
782
{
778
-
lead: "_Cache reconnection logic is buggy._",
783
+
lead: "Cache reconnection logic is buggy.",
779
784
text: "After the partition heals, the service doesn't reconnect properly and continues failing.",
780
785
},
781
786
]}
@@ -821,18 +826,18 @@ Success looks like: no OOM kills, graceful degradation where the service sheds e
821
826
What teams commonly discover:
822
827
823
828
<List
824
-
variant="check-icons-list"
829
+
variant="zebra-list"
825
830
items={[
826
831
{
827
-
lead: "_No memory limits set._",
832
+
lead: "No memory limits set.",
828
833
text: "Without limits, one pod's memory pressure affects the entire node — the classic noisy neighbor problem.",
829
834
},
830
835
{
831
-
lead: "_OOM kill with no graceful handling._",
836
+
lead: "OOM kill with no graceful handling.",
832
837
text: "The application gets killed abruptly with no opportunity to drain connections or save state.",
833
838
},
834
839
{
835
-
lead: "_Memory leaks exposed under pressure._",
840
+
lead: "Memory leaks exposed under pressure.",
836
841
text: "Baseline memory usage creeps up over time, and the stress test just accelerates the inevitable OOM.",
837
842
},
838
843
]}
@@ -850,6 +855,7 @@ A mature chaos practice looks different. Experiments run on a regular cadence
850
855
851
856
<Diagram
852
857
src={evolutionDiagram}
858
+
magnifyButtonPosition="top-left"
853
859
title="Evolution from ad-hoc experiments to mature chaos practice"
854
860
description="Flowchart showing a progression from ad-hoc experiments to regular experiments, then game days, and finally automated chaos, with each stage annotated by the learning or organizational capability it adds."
855
861
/>
@@ -859,42 +865,42 @@ The progression takes time — typically a year or more to go from first experim
859
865
Here's a realistic four-quarter roadmap for building a chaos engineering practice from scratch:
860
866
861
867
<Table
862
-
variant="vertical-column-delineation-table"
868
+
variant="criteria-highlight-table"
863
869
content={{
864
870
figure: "Four-quarter roadmap for building a chaos engineering practice.",
"Run first pod-kill experiments, document and fix findings, train team on experiment design (hypothesis, abort conditions, stakeholder communication)",
875
881
"3+ experiments completed, 5+ bugs fixed",
876
882
],
877
883
},
878
884
{
879
-
th: "Q2",
885
+
th: "Expand coverage",
880
886
td: [
881
-
"Expand coverage",
887
+
"Q2",
882
888
"Move to latency injection and network partitions, create reusable experiment library, test all critical services",
"Host first game day, enable multiple teams to run experiments independently, share learnings org-wide",
891
897
"Successful game day, 3+ teams practicing chaos",
892
898
],
893
899
},
894
900
{
895
-
th: "Q4",
901
+
th: "Automate foundations",
896
902
td: [
897
-
"Automate foundations",
903
+
"Q4",
898
904
"Integrate chaos into CI/CD for staging deployments, automate steady-state verification, schedule recurring experiments",
899
905
"Weekly automated experiments without manual intervention",
900
906
],
@@ -960,7 +966,8 @@ Game days build team muscle memory for incident response. The goal isn't to find
960
966
How do you know if your chaos practice is working? You need metrics that measure both activity (are you actually running experiments?) and impact (are you finding and fixing problems?).
961
967
962
968
<Table
963
-
variant="vertical-column-delineation-table"
969
+
variant="grid-and-accent-header-table"
970
+
fullWidth={false}
964
971
content={{
965
972
figure: "Metrics for measuring chaos engineering practice effectiveness.",
966
973
thead: {
@@ -1029,10 +1036,10 @@ Chaos engineering doesn't require expensive platforms or dedicated teams. You ca
1029
1036
1030
1037
The principles are simple: form a hypothesis before you break anything, define abort conditions so experiments don't become incidents, control the blast radius so you're learning rather than causing outages, and follow through on findings by actually fixing them.
1031
1038
1032
-
Once you've mastered the four experiments in this guide, the path forward opens up. You can explore zone and region failures to test geographic redundancy. You can inject clock skew to find time-sensitive bugs. You can simulate certificate expiration, DNS failures, or cloud provider API throttling. Each new experiment type reveals a different class of bugs — and each one can be done with free tools.
1033
-
1034
-
The barrier to starting isn't tooling or budget. It's deciding to run that first experiment. Pick a service. Form a hypothesis. Open your monitoring dashboard. Kill a pod. Watch what happens.
1035
-
1036
1039
<Callout type="success">
1037
1040
You don't need an enterprise chaos platform to start chaos engineering. You need a hypothesis, a safe experiment, monitoring to observe the results, and the discipline to fix what you find. Start small, learn, and expand.
1038
1041
</Callout>
1042
+
1043
+
Once you've mastered the four experiments in this guide, the path forward opens up. You can explore zone and region failures to test geographic redundancy. You can inject clock skew to find time-sensitive bugs. You can simulate certificate expiration, DNS failures, or cloud provider API throttling. Each new experiment type reveals a different class of bugs — and each one can be done with free tools.
1044
+
1045
+
The barrier to starting isn't tooling or budget. It's deciding to run that first experiment. Pick a service. Form a hypothesis. Open your monitoring dashboard. Kill a pod. Watch what happens.
0 commit comments