Skip to content

Add reproducible local benchmark and Unity compatibility matrix harness #21

Description

@wallstop

Context

The modernization plan now has focused correctness coverage, but the release gate still lacks repeatable performance and cross-version evidence. Current validation is local Unity 6000.4.6f1 on the confirmed host; Unity 2021.3/2022.3, other local platforms, and measured Play-entry budgets remain unverified.

Scope

Add a disposable local driver under scripts/ that accepts explicit host-project, Unity-version, fixture-size, suite, and output-directory arguments. It must support the existing Unity MCP bridge and direct host Unity execution with explicit path translation.

The fixture and measurements should cover:

  • deterministic 100, 1,000, 10,000, and 50,000 main ScriptableObject datasets
  • BaseDataObject and ordinary ScriptableObject controls, labels, descriptions, nested fields, shared references, duplicate short type names, custom ordering, and saved selections
  • shell construction, first visible rows, selection readiness, complete indexing, search, selection, filtering, scrolling, type switching, refresh, allocations, retained references, and realized rows
  • cold import, warm restart, warm in-session, incremental edits, and Play entry with the package absent, closed, open/idle, and open/indexing
  • machine-readable raw samples and readable comparisons recording revision, fixture, Unity, OS, CPU, memory, storage, graphics, DPI, theme, window size, and reload settings

Acceptance criteria

  • Generated fixture projects, datasets, caches, and results remain outside the distributed package.
  • Timing excludes process launch, MCP transport, and command dispatch; Unity measurements use monotonic timers and focused profiler markers.
  • Warm UI and Play-entry scenarios use five warm-ups and at least 30 alternating matched repetitions; expensive restart/compile scenarios use at least five samples and report all samples.
  • The driver verifies fixture counts/content before timing and restores any changed project settings, including after failures.
  • A compatibility report covers Unity 2021.3, 2022.3, and the current host editor, with player compile/build smoke results for ordinary and BaseDataObject assets.
  • The report identifies whether Play-entry p95 overhead meets the 20 ms target and separates window-open from window-closed cases.

Related: #19

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions