Skip to content

test: spruce up oracle contracts, metamorphic laws, and async coverage #1808

Description

@KyleAMathews

Goal

Strengthen our existing oracle suites without growing production code or building a universal testing framework. Prefer several small independent checks over increasingly elaborate reference implementations.

Proposed follow-up PR

  • Add metamorphic laws where the contract supports them: incremental versus fresh settled query results; backend page-size invariance for completed UI windows; idempotent authoritative replay; independent-key commutation; pending optimistic snapshot stability under unrelated sync/recomputation.
  • Standardize public-observation checks: direct reads versus event-reconstructed rows and downstream results at publication boundaries, no torn transactions, no obsolete-session effects, balanced acquisition/release, and redundant-publication budgets where guaranteed.
  • Generate meaningful async boundaries, not just operation order. Include begin → truncate → mutation settles → commit, startup/release reentrancy, replay overlap, cleanup, and late settlement. Reuse controlled promises and existing fast-check scheduler support.
  • Test the checker with deliberate faults: dropped publication, stale completion, incorrect sibling release, and forbidden rebasing of optimistic snapshots. Each important law should reject an appropriate broken subject.
  • Measure executed transitions and meaningful combinations, not just generated commands or test counts. Preserve fixed seeds, random campaigns, shrinking, and replayable minimal histories. Consider fast-check commands only where their preconditions and shrink tracking simplify an existing driver.
  • Record each suite's intended contract, observation boundary, independent model inputs, supported history grammar, and explicit exclusions. Do not infer new product guarantees from a reference model's convenient behavior.

Constraints and traps

  • Optimistic mutations own whole-row snapshots. Do not merge newer synced fields into a pending mutation merely because a patch-based reference model expects it.
  • Metamorphic equivalences need explicit preconditions. Fresh-query comparison checks settled results; it cannot replace continuous-observer checks for partial publication or lifecycle corruption.
  • Control scheduling where possible. Permit multiple outcomes only when the actual contract allows them, with whole-outcome and causal constraints—not arbitrary per-field old/new choices. No general linearizability checker is proposed.
  • Reference engines are useful only on a genuinely compatible semantic subset. A passing comparison can share a bug, and a disagreement is not proof that production is wrong.
  • Keep valuable existing regression tests. Fold minimal counterexamples into a shared oracle driver when practical, rather than deleting coverage to improve test counts.
  • Higher run counts cannot repair an incorrect model or unreachable generator dimension.

Starting points

Existing includes cross-formulation and temporal oracles, D2 source-reconciliation commutation laws, subscription replay/lifecycle oracles, pagination and adapter refinement suites, and the optimistic history work in #1807.

Acceptance

For each touched suite, document the law and its limits, demonstrate a relevant deliberately broken subject fails, and show fixed/random runs plus exact replay pass for the correct implementation. Report executed-history coverage and test cost. Keep all additions test-only unless a separately demonstrated production defect needs a narrow fix.

References: fast-check model-based testing, fast-check advanced testing.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions