Skip to content

MST/2 client: fault-injection matrix (truncation, wrong bytes, kill between write and journal, revocation mid-run) #63

Description

@Ivanbeethoven

Context

Two fault cases are covered today:

  • the retry path — an HTTP proxy injects 503s and dropped connections and the
    hydrate must still complete (tests/mst2_retry_e2e.rs, proxy in the
    development harness);
  • local-store tampering — a corrupted CAS object is refused mid-session, a
    corrupted store fails to mount, and the client refuses to serve wrong bytes
    (covered by mst2_lease_e2e / the acceptance script's tamper step and by the
    durable-store unit tests).

Spec 16 requires a wider fault matrix, and several rows of it are not covered at
all: truncated responses, half-written files, restart with an empty client,
lease revocation during a running operation, and a server that changes its mind
mid-stream.

Scope

Add a fault-injection suite that asserts behaviour, not just "it errored":

Injection Required behaviour
Truncated response body (connection closed mid-body) Typed error; no partial file reaches the store; retry (if applicable) does not double-count bytes as useful
Response with correct length but wrong bytes Digest mismatch; the bad object is never written to the CAS nor journaled
Chunk frame with a valid frame digest but wrong chunk bytes Chunk digest check rejects it; the whole file is not reported verified
Server returns 5xx mid-hydration after some files Already-hydrated files stay valid; the operation retries or fails typed; no DURABLE_COMPLETE
Lease revoked while a hydration is running The running operation fails typed (410/EACCES class); files already persisted remain readable offline
Process killed between blob write and journal append Restart resumes; the file is re-fetched and re-verified, never assumed
Process killed after journal append, before the marker Same as above at the marker level: no completion is claimed
Empty client (fresh store) against a lock-free server Hydrate completes, mount serves bytes identical to git clone
CURSOR_STALE during enumeration Enumeration restarts and returns the complete, correct set
Store directory truncated (CAS blob removed) after completion The store refuses to claim completeness; a reopen reports the defect rather than serving short files

Requirements on the suite

  • One command, deterministic, no manual steps; inject via the proxy/fixture
    layer rather than by editing the client.
  • Each case prints what it injected and what it observed, so a failure names its
    cause.
  • Cases that cannot run in CI (need /dev/fuse, need process kill) must skip
    with an explicit message, never silently pass.
  • No case may pass by asserting only "an error occurred": assert the error class
    and the resulting local state (store contents, marker presence, readable
    files).

Acceptance

  • The matrix above is implemented and green, or each unimplemented row has a
    tracked reason in the issue thread.
  • A deliberately broken implementation (for example, writing the marker before
    the last file is verified) must make at least one case fail — the suite has to
    be able to fail for the right reason.

References

  • Spec 16 §1 (conformance and fault injection), spec 11 §§4, 8 (local publish,
    durable commit), spec 07 §7 (verification gates)
  • src/snapshot/{durable,client,reader,coordinator,range}.rs,
    tests/mst2_retry_e2e.rs, tests/mst2-e2e/

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions