[mpereira]

Over the last couple of days I've been playing around with ideas for running OpenShell fault-injection tests. I came up with a prototype harness that executes declarative experiments. An experiment has workloads and operations (a fault such as kill or freeze, or a policy change), sequenced by a flow, and checks on the state of the system that decide the verdict: did the experiment pass or fail. e2e/chaos/README.md documents all of this.

• Declarative experiments: one YAML file says what to run, what to disrupt and what must hold afterwards.
• Visualizations of experiment runs: a Gantt chart of the flow and every sandbox, an HTML report and a Grafana dashboard.
• Self-contained output with telemetry: each run is one directory with its inputs, timeline, logs, gateway metrics and traces, and the verdict.

Summary

e2e/chaos is a fault-injection harness for OpenShell. Its experiments reproduce NVIDIA#4910, NVIDIA#4828, NVIDIA#4981 and NVIDIA#4409 in one command each. The branch adds the harness under e2e/chaos in four commits, and it touches nothing under crates/ or docs/.

The OpenShell gateway creates sandboxes and reconciles their state, and each sandbox runs with a supervisor that enforces its policy. The failures that matter happen between those three parts: a gateway that dies while sandboxes run, a supervisor whose processes are frozen, a policy that changes under a running sandbox. An experiment reaches them because it runs real sandboxes against a real gateway, disrupts them on purpose and records everything that happened. The e2e suites in tests/ check what a sandbox does in normal operation. The harness adds operations that change or disrupt the parts of the system, a flow that times them, evidence recorded whether the run passes or not, and a verdict that can expect a known failure.

Because an experiment is one declarative file with a verdict, it serves as three things.

• A reproducible bug report. Anyone runs the file and gets the same kind of run directory to look at. Each of the four issues above has one.
• Verification that a fix works. The verdict decides that the bug is fixed, not the developer or the agent that changed the code.
• A regression test afterwards. The experiment names the issue in expectations, so a run tells a known failure (XFAIL) from a new one (FAIL), and a landed fix shows up as XPASS. Empty the expectation's checks and the experiment guards the fix from then on.

The Python engine under e2e/chaos/src is a prototype that shows the model working. The model is the part I am putting forward: the experiment manifest, the adapter manifest and the run directory are the contract, and an engine built on the Rust testkit in tests/testkit/, or any other engine, can run the same experiments and write the same runs. The README documents the model, and its section What a Rust engine would reuse says which parts such an engine would keep.

The diagram below, the same one that opens the README, shows what one run takes in and what it leaves behind.

%%{init: {"flowchart": {"wrappingWidth": 600}, "themeVariables": {"fontSize": "20px"}}}%%
flowchart LR
  subgraph experiment ["experiment manifest"]
    e["<pre>id:#nbsp;supervisor-kill#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;<br/>default-adapter:#nbsp;openshell-docker#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;<br/>workloads:#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;<br/>#nbsp;#nbsp;-#nbsp;name:#nbsp;counter#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;<br/>#nbsp;#nbsp;#nbsp;#nbsp;kind:#nbsp;shell#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;<br/>#nbsp;#nbsp;#nbsp;#nbsp;instances:#nbsp;3#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;<br/>#nbsp;#nbsp;#nbsp;#nbsp;run:#nbsp;|#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;<br/>#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;for#nbsp;i#nbsp;in#nbsp;$(seq#nbsp;1#nbsp;#quot;$TICKS#quot;);#nbsp;do#nbsp;...#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;<br/>#nbsp;#nbsp;#nbsp;#nbsp;checks:#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;<br/>#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;-#nbsp;name:#nbsp;exited-ok#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;<br/>#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;kind:#nbsp;exit-code#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;<br/>#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;equals:#nbsp;0#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;<br/>operations:#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;<br/>#nbsp;#nbsp;-#nbsp;name:#nbsp;kill-supervisors#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;<br/>#nbsp;#nbsp;#nbsp;#nbsp;uses:#nbsp;kill#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;<br/>#nbsp;#nbsp;#nbsp;#nbsp;target:#nbsp;{workload:#nbsp;counter,#nbsp;role:#nbsp;supervisor}#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;<br/>#nbsp;#nbsp;#nbsp;#nbsp;select:#nbsp;{count:#nbsp;2}#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;<br/>flow:#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;<br/>#nbsp;#nbsp;parallel:#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;<br/>#nbsp;#nbsp;#nbsp;#nbsp;-#nbsp;counter#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;<br/>#nbsp;#nbsp;#nbsp;#nbsp;-#nbsp;serial:#nbsp;[{wait:#nbsp;...},#nbsp;{delay:#nbsp;10s},#nbsp;kill-supervisors]</pre>"]
  end
  subgraph adapter ["adapter manifest"]
    a["<pre>id:#nbsp;openshell-docker#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;<br/>backend:#nbsp;docker#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;<br/>instance:#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;<br/>#nbsp;#nbsp;create:#nbsp;|#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;<br/>#nbsp;#nbsp;#nbsp;#nbsp;openshell#nbsp;sandbox#nbsp;create#nbsp;...#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;<br/>#nbsp;#nbsp;collect:#nbsp;|#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;<br/>#nbsp;#nbsp;#nbsp;#nbsp;openshell#nbsp;sandbox#nbsp;download#nbsp;...#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;<br/>targets:#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;<br/>#nbsp;#nbsp;gateway:#nbsp;{type:#nbsp;process,#nbsp;pid-file:#nbsp;...}#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;<br/>#nbsp;#nbsp;roles:#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;<br/>#nbsp;#nbsp;#nbsp;#nbsp;sandbox:#nbsp;{type:#nbsp;container,#nbsp;labels:#nbsp;...}#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;<br/>#nbsp;#nbsp;#nbsp;#nbsp;supervisor:#nbsp;{type:#nbsp;container,#nbsp;labels:#nbsp;...}#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;<br/>operations:#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;<br/>#nbsp;#nbsp;openshell-policy-set:#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;<br/>#nbsp;#nbsp;#nbsp;#nbsp;run:#nbsp;|#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;<br/>#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;openshell#nbsp;policy#nbsp;set#nbsp;#quot;$(sandbox_name)#quot;#nbsp;--policy#nbsp;...#nbsp;#nbsp;#nbsp;#nbsp;<br/>metrics:#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;<br/>#nbsp;#nbsp;-#nbsp;{name:#nbsp;gateway,#nbsp;url:#nbsp;http://127.0.0.1:.../metrics}#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;<br/>watch:#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;<br/>#nbsp;#nbsp;-#nbsp;{name:#nbsp;docker-events,#nbsp;run:#nbsp;docker#nbsp;events#nbsp;...}#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;<br/>#nbsp;#nbsp;-#nbsp;{name:#nbsp;otel,#nbsp;run:#nbsp;docker#nbsp;run#nbsp;...#nbsp;opentelemetry-collector}</pre>"]
  end
  experiment -- "the argument" --> chaos["<pre>chaos#nbsp;run#nbsp;experiments/supervisor-kill.yaml</pre>"]
  adapter -- "named by default-adapter,<br/>or by --adapter ID" --> chaos
  subgraph ops ["what runs against the system"]
    direction TB
    h["adapter hooks<br/><pre>openshell#nbsp;sandbox#nbsp;create#nbsp;#nbsp;<br/>openshell#nbsp;sandbox#nbsp;download<br/>openshell#nbsp;policy#nbsp;set#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;</pre>"]
    f["built-in faults<br/><pre>kill#nbsp;-TERM,#nbsp;kill#nbsp;-STOP#nbsp;#nbsp;#nbsp;<br/>docker#nbsp;kill,#nbsp;docker#nbsp;pause</pre>"]
    o["observers<br/><pre>GET#nbsp;/metrics#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;<br/>docker#nbsp;events#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;<br/>OTLP#nbsp;spans#nbsp;from#nbsp;the#nbsp;gateway</pre>"]
  end
  chaos --> ops
  ops <--> sut
  subgraph sut ["system under test"]
    direction TB
    gw["OpenShell gateway process<br/>on the Docker backend"]
    subgraph inst ["per instance"]
      direction LR
      sb["sandbox container"] --- sv["supervisor container"]
    end
    gw --- inst
  end
  chaos -- "creates" --> run
  subgraph run ["experiment run, one directory"]
    r["<pre>runs/#lt;experiment#gt;/#lt;run#nbsp;id#gt;/#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;<br/>├──#nbsp;experiment.yaml#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;<br/>├──#nbsp;adapter.yaml#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;<br/>├──#nbsp;manifest.json#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;<br/>├──#nbsp;timeline.jsonl#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;<br/>├──#nbsp;verdict.json#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;<br/>├──#nbsp;report.html#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;<br/>├──#nbsp;logs/#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;<br/>│#nbsp;#nbsp;#nbsp;├──#nbsp;harness.log#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;<br/>│#nbsp;#nbsp;#nbsp;└──#nbsp;gateway.log#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;<br/>├──#nbsp;system/#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;<br/>│#nbsp;#nbsp;#nbsp;├──#nbsp;metrics/gateway.jsonl#nbsp;#nbsp;#nbsp;<br/>│#nbsp;#nbsp;#nbsp;└──#nbsp;watch/#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;<br/>│#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;├──#nbsp;docker-events.jsonl#nbsp;<br/>│#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;└──#nbsp;otel.jsonl#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;<br/>└──#nbsp;workloads/counter/instances/<br/>#nbsp;#nbsp;#nbsp;#nbsp;└──#nbsp;counter-0/#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;<br/>#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;├──#nbsp;logs/sandbox.log#nbsp;#nbsp;#nbsp;#nbsp;<br/>#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;├──#nbsp;logs/supervisor.log#nbsp;<br/>#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;└──#nbsp;output/journal#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;#nbsp;</pre>"]
  end

Loading

In the diagram:

• An experiment is one YAML file: the workloads for the system to run, each as one or more instances, the operations to apply while they run, a flow that orders them, and the checks that decide from recorded evidence whether the system did what the experiment claims. It says what, not how.
• An operation changes the system while the workloads run. A built-in fault such as restart, kill, outage, restart-in-place or freeze disrupts one part of the system through the backend, Docker or Kubernetes. An adapter operation such as openshell-policy-set runs a hook that calls the system's own commands.
• An adapter is a second YAML file that binds the harness to one system on one backend: what an instance is, the hooks that create, read, collect and delete one, what the faults act on, and what to observe. The harness knows OpenShell only through its two adapters, openshell-docker and openshell-kubernetes, and an experiment runs unchanged on either.
• chaos run takes one experiment and one adapter and writes one experiment run: a self-contained directory with every input, the timeline, logs, metrics, the files each instance wrote and the verdict. chaos report, chaos share, chaos dashboard and chaos evaluate read only that directory, so a verdict can be decided again offline.

Related Issue

This PR targets my fork, so the state:accepted gate does not apply. The experiments reproduce NVIDIA#4910, NVIDIA#4828, NVIDIA#4981 and NVIDIA#4409, and they close none of them.

Results

Below is the latest run of each shipped experiment, with the chart chaos share draws for it. All ran on the openshell-docker adapter (the harness's binding to OpenShell on the Docker compute driver) with Docker Desktop 29.8.2 on macOS on Apple silicon. The runs came from earlier states of this branch, before it was squashed into one commit, and each run's manifest.json records the commit it ran at. Each run directory is under e2e/chaos/runs/<experiment>/<run>/, and the gateway under test was built from main at 9f1cdb4. The verdict is PASS when every check holds, XFAIL (exit 0) when the only failures are ones an expectation names, and FAIL (exit 1) otherwise. ERROR overrides all three when the harness could not finish the run cleanly, for example when a sandbox could not be deleted.

Experiment	Verdict	Duration	What it reproduces	Gantt chart of the run
hello-world A sandbox says hello.	PASS	34 s
smoke Counters run to completion with no faults.	PASS	40 s
supervisor-pause A workload runs to completion while its supervisor is frozen.	PASS	1 min 20 s
gateway-graceful-restart Counter instances resume after a graceful gateway restart.	PASS	2 min 08 s
supervisor-kill A sandbox survives its supervisor being killed.	FAIL	1 min 34 s	No issue is filed yet
gateway-kill-keeps-state Sandboxes survive a hard gateway kill.	XFAIL	1 min 50 s	#4910
policy-tighten-survives-gateway-restart A tightened policy stays in effect across a gateway restart.	XFAIL	1 min 27 s	#4910
sandbox-container-restart A sandbox returns to Ready after its container restarts.	XFAIL	5 min 34 s	#4828
sandbox-stop-start A stopped sandbox starts again every time.	XFAIL (flaky)	40 min 56 s	#4981
gateway-sigterm-during-recovery A SIGTERM during the gateway's startup recovery stops it gracefully.	ERROR (4 of 4 runs)	8 min 36 s	#4409

In each chart the top section is the flow, with the faults in red, and every section under it is one sandbox instance from Created to Succeeded or Failed. In supervisor-kill, a killed supervisor stops the workload, --restart-policy on-failure does not start it again, and no exit code is recorded. In gateway-kill-keeps-state the three counters are Unknown to the harness while the gateway is down (their workloads keep ticking), come back as Running once the gateway is up although the restarted gateway has already removed their supervisors, and then end Pending and Failed. A flaky expectation is one the failure need not hit on every run. The verdict records which instances it hit, and in the sandbox-stop-start run two of the eight sandboxes did, so its chart collapses the twenty restarts into one row and keeps only those two sandboxes. For what the two experiments for #4910 observed, see the comment on NVIDIA#4910: .

gateway-sigterm-during-recovery came later than the other nine. Its four runs used the third commit, against a gateway built from main at 3a5dfcd. The chart is the latest run and shows four of the eight sandboxes, failed ones first; the README section charts the first run. A second SIGTERM lands while the restarted gateway's startup recovery is starting the sandboxes again, before the gateway installs its SIGTERM handler. The gateway exits without logging a shutdown, the next gateway keeps the interrupted start's pending driver operation, and that sandbox can then be neither started nor deleted. Every run hit it on one sandbox. The failed delete is a run error, so the verdict is ERROR and not XFAIL, and the sandbox's container is left running. In every run the gateway after the kill also failed to start one or two of the other sandboxes with sandbox runtime is already bound to another supervisor process, which no issue covers yet.

The HTML report for the gateway-kill-keeps-state run, and the same run loaded into Grafana, look like this.

How to run one experiment

The three commands below clone the branch and run the experiment that reproduces #4910. The mise task builds and starts a local Docker gateway, runs the experiment against it, and writes a run directory with a timeline, a verdict and an HTML report. The run takes about 2 minutes after the first build, and it needs local Docker, uv and jq.

git clone -b feat/chaos-experiment-harness/mpereira https://github.com/mpereira/OpenShell
cd OpenShell
mise run e2e:chaos e2e/chaos/experiments/gateway-kill-keeps-state.yaml

The run exits 0 with the verdict XFAIL and writes its directory under e2e/chaos/runs/gateway-kill-keeps-state/.

Changes

The commits add these pieces.

• e2e/chaos/ holds the chaos CLI (validate, preflight, run, evaluate, report, share, export, dashboard, list, show), two adapters (openshell-docker and openshell-kubernetes), ten experiments with their latest runs, a Grafana loader, and the README, REFERENCE and ADAPTER docs.
• tasks/test.toml gains e2e:chaos, e2e:chaos:preflight, e2e:chaos:check, e2e:chaos:share and e2e:chaos:grafana.
• e2e/with-docker-gateway.sh now always exposes the gateway's Prometheus metrics port. With OPENSHELL_E2E_GATEWAY_OTLP=1 it points the gateway's OTLP trace export at a port the harness's collector listens on, so an experiment can wait for the gateway's reconcile sweep (the periodic pass that re-checks every sandbox) instead of sleeping.
• CONTRIBUTING.md gains one line for the new directory.

Release Notes

This branch changes no user-facing behaviour.

Testing

Here is what I checked and what I did not.

• e2e/chaos/scripts/test-and-lint.sh passes on the branch head: 253 unit tests, ruff check and format, and shellcheck on the scripts and the embedded shell.
• The nine runs in the table ran with the Docker adapter at earlier states of this branch. The four XFAIL runs came from a clean tree, and the other five had uncommitted harness edits in the tree. The four gateway-sigterm-during-recovery runs used the third commit on a clean tree.
• uv run --project e2e/chaos chaos validate --all-adapters passes. Every experiment validates on openshell-docker. Seven validate on openshell-kubernetes, and three are marked not applicable by design, because in-place restart, freeze and the reconcile-sweep signal have no Kubernetes equivalent.
• I have not run the Kubernetes adapter against a cluster.
• Nothing in this branch runs in CI. e2e:chaos:check, which requires hello-world and smoke to PASS, is the candidate.
• Unit tests are added, all new, under e2e/chaos/tests.
• E2E tests are added: the experiments are the e2e tests, and none is wired into an existing suite.

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture docs are not updated. e2e/chaos/README.md documents the harness.

Questions

Three questions for OpenShell maintainers who read this from the issue comments:

1. When the supervisor of a Ready sandbox dies while the workload is healthy, is the intended outcome that the sandbox ends in Error, or should --restart-policy on-failure cover it?
2. After a gateway restart, should a Docker sandbox that ran through the restart come back Ready on its own, as #4910 and #4828 expect, or is recreating it the intended path?
3. Does #4983 settle the race in #4981, so that a sandbox start during the reconcile sweep should now always reach Ready, or is some residual flakiness expected?