Context
Two fault cases are covered today:
- the retry path — an HTTP proxy injects 503s and dropped connections and the
hydrate must still complete (tests/mst2_retry_e2e.rs, proxy in the
development harness);
- local-store tampering — a corrupted CAS object is refused mid-session, a
corrupted store fails to mount, and the client refuses to serve wrong bytes
(covered by mst2_lease_e2e / the acceptance script's tamper step and by the
durable-store unit tests).
Spec 16 requires a wider fault matrix, and several rows of it are not covered at
all: truncated responses, half-written files, restart with an empty client,
lease revocation during a running operation, and a server that changes its mind
mid-stream.
Scope
Add a fault-injection suite that asserts behaviour, not just "it errored":
| Injection |
Required behaviour |
| Truncated response body (connection closed mid-body) |
Typed error; no partial file reaches the store; retry (if applicable) does not double-count bytes as useful |
| Response with correct length but wrong bytes |
Digest mismatch; the bad object is never written to the CAS nor journaled |
| Chunk frame with a valid frame digest but wrong chunk bytes |
Chunk digest check rejects it; the whole file is not reported verified |
| Server returns 5xx mid-hydration after some files |
Already-hydrated files stay valid; the operation retries or fails typed; no DURABLE_COMPLETE |
| Lease revoked while a hydration is running |
The running operation fails typed (410/EACCES class); files already persisted remain readable offline |
| Process killed between blob write and journal append |
Restart resumes; the file is re-fetched and re-verified, never assumed |
| Process killed after journal append, before the marker |
Same as above at the marker level: no completion is claimed |
| Empty client (fresh store) against a lock-free server |
Hydrate completes, mount serves bytes identical to git clone |
CURSOR_STALE during enumeration |
Enumeration restarts and returns the complete, correct set |
| Store directory truncated (CAS blob removed) after completion |
The store refuses to claim completeness; a reopen reports the defect rather than serving short files |
Requirements on the suite
- One command, deterministic, no manual steps; inject via the proxy/fixture
layer rather than by editing the client.
- Each case prints what it injected and what it observed, so a failure names its
cause.
- Cases that cannot run in CI (need
/dev/fuse, need process kill) must skip
with an explicit message, never silently pass.
- No case may pass by asserting only "an error occurred": assert the error class
and the resulting local state (store contents, marker presence, readable
files).
Acceptance
- The matrix above is implemented and green, or each unimplemented row has a
tracked reason in the issue thread.
- A deliberately broken implementation (for example, writing the marker before
the last file is verified) must make at least one case fail — the suite has to
be able to fail for the right reason.
References
- Spec 16 §1 (conformance and fault injection), spec 11 §§4, 8 (local publish,
durable commit), spec 07 §7 (verification gates)
src/snapshot/{durable,client,reader,coordinator,range}.rs,
tests/mst2_retry_e2e.rs, tests/mst2-e2e/
Context
Two fault cases are covered today:
hydrate must still complete (
tests/mst2_retry_e2e.rs, proxy in thedevelopment harness);
corrupted store fails to mount, and the client refuses to serve wrong bytes
(covered by
mst2_lease_e2e/ the acceptance script's tamper step and by thedurable-store unit tests).
Spec 16 requires a wider fault matrix, and several rows of it are not covered at
all: truncated responses, half-written files, restart with an empty client,
lease revocation during a running operation, and a server that changes its mind
mid-stream.
Scope
Add a fault-injection suite that asserts behaviour, not just "it errored":
DURABLE_COMPLETEgit cloneCURSOR_STALEduring enumerationRequirements on the suite
layer rather than by editing the client.
cause.
/dev/fuse, need process kill) must skipwith an explicit message, never silently pass.
and the resulting local state (store contents, marker presence, readable
files).
Acceptance
tracked reason in the issue thread.
the last file is verified) must make at least one case fail — the suite has to
be able to fail for the right reason.
References
durable commit), spec 07 §7 (verification gates)
src/snapshot/{durable,client,reader,coordinator,range}.rs,tests/mst2_retry_e2e.rs,tests/mst2-e2e/