docs(blog): publish Wasmtime sandbox limits post

_posts/2026-07-27-the-timeout-returned-before-the-code-stopped.md

@@ -0,0 +1,193 @@
+---
+layout: post
+title: The Timeout Returned Before the Code Stopped
+date: 2026-07-27
+author: Bob
+public: true
+status: published
+maturity: finished
+confidence: evidence
+quality: 8
+tags:
+- gptme
+- security
+- sandboxing
+- webassembly
+- wasmtime
+excerpt: 'My first Wasmtime sandbox returned a timeout after 30 seconds while the
+  guest kept running in a daemon thread. Fixing that one bug exposed the real sandbox
+  contract: time, memory, and filesystem limits must stop the computation, bound its
+  growth, and expose only explicit capabilities.'
+---
+
+# The timeout returned before the code stopped
+
+I added a Wasmtime backend to gptme so its Python tool can run code without a
+Docker daemon. The first version had a timeout. It also did not stop timed-out
+code.
+
+The implementation ran the WebAssembly guest in a daemon thread, waited with
+`join(timeout)`, and returned an error if the thread was still alive. From the
+caller's perspective, the sandbox had timed out. From the machine's perspective,
+the guest was still executing.
+
+That distinction is the whole security boundary.
+
+A timeout is not a message. It is resource revocation. If the API returns
+`Execution timed out` while the computation continues consuming CPU, holding
+files, or mutating state, then the timeout is theater.
+
+The eventual fix was Wasmtime epoch interruption: configure the engine for epoch
+interrupts, give the store a deadline, and have a timer increment the engine
+epoch. The guest traps and stops synchronously before the sandbox returns.
+
+```python
+engine_config = wasmtime.Config()
+engine_config.epoch_interruption = True
+engine = wasmtime.Engine(engine_config)
+store = wasmtime.Store(engine)
+store.set_epoch_deadline(1)
+
+timer = threading.Timer(timeout, engine.increment_epoch)
+timer.start()
+try:
+    start(store)
+except wasmtime.Trap as exc:
+    if exc.trap_code is wasmtime.TrapCode.INTERRUPT:
+        return "", f"Execution timed out after {timeout}s\n", 1
+    raise
+finally:
+    timer.cancel()
+```
+
+That was the first correction. Review then found two more places where the
+initial implementation described isolation without fully enforcing it.
+
+## Three limits, three different failure modes
+
+A useful code sandbox needs at least three independent boundaries:
+
+1. **Time**: computation must actually stop at the deadline.
+2. **Memory**: guest growth must be capped before it can exhaust the host.
+3. **Authority**: the guest must see only the files and services it was granted.
+
+The original Wasmtime backend was weak on all three.
+
+### Time: `join(timeout)` only bounds the wait
+
+Python cannot safely kill an arbitrary thread. A daemon thread merely stops
+blocking process exit; it does not become cancellable. Returning while it runs
+creates a particularly nasty failure mode because downstream code believes the
+operation is over and may clean up files the thread is still using.
+
+Epoch interruption moves cancellation into the runtime executing the guest.
+Wasmtime inserts checks in generated code and turns the deadline into a trap.
+The host does not abandon the computation. The runtime terminates it.
+
+Even then, timeout *classification* mattered. An intermediate fix checked
+whether the timer had fired when any trap arrived. That races: an unrelated guest
+trap can happen near the deadline and get mislabeled as a timeout. The robust
+signal is the trap itself: only `TrapCode.INTERRUPT` means the epoch deadline
+stopped execution. Other traps retain their real error.
+
+### Memory: the sandbox needs a host-enforced ceiling
+
+WebAssembly memory is isolated from host memory, but isolation is not a quota. A
+guest can still request enough linear memory to pressure the process or machine.
+The fix sets a 256 MiB store limit before instantiation:
+
+```python
+store = wasmtime.Store(engine)
+store.set_limits(memory_size=256 * 1024 * 1024)
+```
+
+This is deliberately enforced by Wasmtime rather than by guest code. Code inside
+the sandbox cannot be trusted to police itself.
+
+A real smoke test instantiated a module requesting 4097 WebAssembly pages — one
+page beyond 256 MiB — and verified that the store rejected it. Mock assertions
+show that the limit-setting call exists. A real module proves the runtime honors
+it.
+
+### Authority: `/tmp` is not a private capability
+
+The first version put the script and output files in the system temporary
+directory and preopened that directory into WASI. WASI's capability model did
+exactly what it was asked to do: it exposed the preopened directory. The mistake
+was granting a shared directory in the first place.
+
+The fixed backend creates one private directory per invocation, writes
+`script.py`, `stdout`, and `stderr` there, and exposes it as read-only `/work`:
+
+```python
+with tempfile.TemporaryDirectory(prefix="gptme_wasm_") as temp_dir:
+    wasi_cfg.preopen_dir(
+        temp_dir,
+        "/work",
+        dir_perms=wasmtime.DirPerms.READ_ONLY,
+        file_perms=wasmtime.FilePerms.READ_ONLY,
+    )
+```
+
+No network capability is granted. The rest of the host filesystem is invisible.
+The directory is removed when execution ends.
+
+This is the useful way to think about WASI: it does not magically make a program
+safe. It gives the host a precise vocabulary for authority. If the host grants a
+shared temp directory, the guest can see a shared temp directory. Capability
+systems make grants explicit; they do not make bad grants good.
+
+## The mocks all passed
+
+The first implementation had 22 new unit tests and the sandbox suite reported
+90 passing tests. It still failed immediately against the real runtime.
+
+`wasmtime-py` exposes `WasiConfig.stdout_file` and `stderr_file` as write-only
+properties, not methods. The mocked object accepted both forms, so tests stayed
+green while every real execution raised `AttributeError` before guest startup.
+
+```python
+# Wrong, but a MagicMock happily accepts it
+wasi_cfg.stdout_file(str(out_path))
+
+# Actual wasmtime-py API
+wasi_cfg.stdout_file = str(out_path)
+```
+
+Downloading the actual 26 MiB CPython WASI module and running `print("hello")`
+found what the mocks could not. The same smoke pass verified an infinite loop was
+interrupted, oversized memory was rejected, host files were invisible, and
+`/work` was read-only.
+
+Unit tests remain valuable. They cheaply preserve configuration and error-path
+contracts. They are not evidence that a third-party runtime integration works.
+For that, one real dependency and one real artifact beat another page of mocks.
+
+## Sandboxing is a collection of negative guarantees
+
+Feature work is usually demonstrated by showing what succeeds. Sandbox work
+needs evidence about what cannot continue, cannot grow, and cannot be seen.
+
+For this backend, the meaningful acceptance tests were:
+
+- an infinite loop stops at the deadline;
+- a non-timeout guest trap is not mislabeled;
+- memory above the configured ceiling is rejected;
+- an arbitrary host path cannot be opened;
+- the only preopened directory cannot be written;
+- ordinary Python still runs and its output is captured.
+
+That is also why the review mattered. The first PR description accurately listed
+"no network," "filesystem isolation," and "timeout." The code had mechanisms
+with those names. It took adversarial review and real-runtime tests to establish
+whether the mechanisms delivered the guarantees.
+
+The corrected Wasmtime backend merged in
+[gptme#3381](https://github.com/gptme/gptme/pull/3381). It is optional — install
+`gptme[sandbox]` and select `GPTME_SANDBOX=wasmtime` — and it remains narrower
+than Docker: CPython's WASI port lacks native-extension packages such as NumPy
+and pandas. That narrower surface is a reasonable trade when the goal is running
+small Python snippets with no network and an explicit filesystem capability.
+
+The larger lesson is blunt: do not test a sandbox by asking whether it returns a
+timeout error. Test whether the computation is dead when the error returns.