perf lab: freeze the canonical IL2CPP profile and add a shipping-fidelity player

[wallstop]

Part of #500.

Question

Can every published result prove its effective IL2CPP configuration, and does a minimal stripped consumer preserve supported behavior without relying on test assemblies or accidental AOT roots?

Mechanism

Build settings, stripping roots, symbols, native toolchains, and cached products change generic sharing, code layout, first-touch work, binary size, and throughput. The current test player is needed for discovery, but it is not a shipping-size or stripping oracle. The canonical profile now pins OptimizeSpeed; generated configuration logs and configured, prebuild, and postbuild evidence record the effective mode. Remaining provenance and campaign factors are tracked below.

Contract

Maintain two separate profiles:

• Canonical verdict player: pinned Windows x64 Standalone IL2CPP Release, .NET Standard 2.1, with the test infrastructure required by the benchmark.
• Shipping-fidelity player: a minimal consumer with no test assemblies, production-compatible stripping, and explicit AOT roots.

Never substitute one profile's result for the other. Preserve Unity 2021.3 support, the public API, all message kinds, private and nested message support, untyped dispatch, zero-allocation warm dispatch, and current ordering, mutation, reset, reentrancy, exception, diagnostics, and reclamation behavior.

Factors and workloads

• Unity 2021.3 and the current pinned Unity line.
• Clean and incremental builds as separate factors.
• Test player versus shipping-fidelity player.
• Every supported non-Disabled stripping level, including Minimal, Low, Medium, and High where present.
• Class and readonly-struct messages; public, private, and nested types.
• Typed and untyped dispatch.
• 1, 16, 256, and 1,000 closed message shapes.
• Cold startup, first registration, first dispatch, warm dispatch, trim, player size, and build time.

Record architecture, backend, API compatibility, IL2CPP compiler configuration and code-generation mode, stripping, exceptions, stack traces, Development/debug/profiler/deep-profile flags, assertions, incremental GC, test-assembly inclusion, editor/compiler/linker versions, native flags, symbols, and cache state.

Primary response

Exact conformance against a versioned machine-readable profile identified by a SHA-256 hash.

Independent unit

One clean IL2CPP build and its resulting player. Repeated launches of one player are nested observations.

Effect threshold

Profile acceptance is exact: no missing, ambiguous, or mismatched required settings, and complete shipping-player behavioral coverage. A rate or latency difference beyond 3%, or player-size difference beyond 1%, prevents treating the two harnesses as interchangeable until explained.

RED proof

• Toggle each pinned setting and prove conformance fails closed.
• Include test assemblies in the shipping player and prove validation rejects it.
• Remove an AOT root and prove a private, nested, or untyped shipping scenario fails.
• Reuse incremental output where a clean build is required and prove provenance validation fails.

GREEN suites

Pass focused profile-schema tests, generated-host tests, shipping-player smoke tests, typed and untyped scenarios, public-surface checks, allocation coverage, and Standalone IL2CPP runs on Unity 2021.3 and the current line.

Stop rule

Do not publish size, stripping, cold-start, or AOT-root claims if the effective profile cannot be reconstructed or the shipping player relies on test assemblies. Do not compare binaries whose profile hashes differ outside the preregistered factor.

Immutable evidence

Retain source/tree/profile/player hashes, package lock, BuildReport, effective settings, editor and toolchain versions, clean-build declaration, generated C++ and symbol indexes, player size, logs, and shipping smoke result. Store large sealed bundles as immutable GitHub prereleases using the campaign evidence schema.

Dependencies

Implementation can start immediately. Evidence acceptance depends on #508. This task blocks confirmatory claims in #501 through #505.

Completion checklist

• Add and validate the canonical profile specification.
• Pin and log effective IL2CPP code generation.
• Add the minimal shipping-fidelity player.
• Prove setting drift and accidental test inclusion fail closed.
• Prove stripping/AOT behavior for every supported message shape.
• Run clean builds on Unity 2021.3 and the current line.
• Retain content-addressed build and result evidence.
• Link the completed protocol from perf: calibrate the IL2CPP measurement system and attribute dispatch cost #500 and PLAN.md.

Profile version integrity in PR #552

PR #552, source 3e67087da13cf3d191a739d4342d204141de24c5, binds every configuration, build-options, and runtime evidence file to the exact Unity version requested by the runner. Previously, a non-empty version from another editor passed validation if its settings and profile hash matched. A red test demonstrated acceptance of 2021.3.45f1 evidence for a requested 6000.3.16f1 run.

The validator now requires ExpectedUnityVersion for evidence, and all seven runner call sites pass it. The three evidence kinds reject version, case, and whitespace drift and missing expectations. An AST guard protects every caller. The focused PowerShell suite and all 1,269 script tests pass locally. No editor version, native build setting, test filter, or benchmark window changes.

All PR checks pass at the same source: CI, four-version Unity matrix, IL2CPP performance, and devcontainer. The internal and comparison IL2CPP artifacts (10006223200, 10006503574) verify against archive SHA-256 199e3f71df0a0755271e7ea152daad35050d4be63fb4feec35816e0f02d10a62 and 7850d00239c51634948190215a33e04555be2403b804f57fd8c18f50ce5acba0. The production validator accepts all ten actual configuration/build/runtime evidence files with expected editor 6000.5.2f1 and canonical profile hash e1e525d1a80eb4dbd8e547853224e3175187e1050f014423116d4bb6f20a4ac2. These remain ordinary expiring CI artifacts. This completes the version-binding defect; full profile/toolchain/cache provenance, stripping coverage, and durable campaign evidence remain open.

Build cache provenance in PR #554

PR #554, source 5afd1aabc2470d8bcf721f66a9a9f28a10d2e818, records build-options evidence schema 2. The generated editor observes Library, Bee, Il2cppBuildCache, and player-output directory state immediately before the build. Final BuildReport options determine whether the player used CleanBuildCache. A warm Library therefore cannot be mistaken for an incremental player build, or a clean player option for a fresh project.

The validator rejects absent or contradictory provenance, stale player output, and incremental evidence under the reviewed clean profile. Both test-player and shipping-player paths share the observer. Profile hashes, Unity settings, cache reuse, test filters, and build counts remain unchanged. The added work is four bounded directory probes per build.

The old validator accepted missing provenance in the red fixture. The new profile suite tests the actual generated C# observer with both LF and CRLF source. Profile, shipping, generated-host, and all 1,273 script tests pass locally. Independent implementation review is clear. The internal IL2CPP job passes at this source with all 202 prior outcomes unchanged. Artifact 10023665143 matches GitHub archive SHA-256 bc544bad4e379cfbbff45979ab171c29e11e1d92a5f7e13e021443fcd022fa6a. The current production validator accepts all five actual profile files for Unity 6000.5.2f1. Build-options schema 2 records clean, populated Library and Bee, missing Il2cppBuildCache, and empty player output. The canonical profile hash remains e1e525d1a80eb4dbd8e547853224e3175187e1050f014423116d4bb6f20a4ac2. This artifact contains text evidence; binary hashes do not prove retained native payloads. The complete Unity matrix and both IL2CPP performance jobs now pass. Comparison artifact 10024960291 matches archive SHA-256 8ce8d18bbcfb60947dbb978fc74c8935acb79633075acfc6c06b708899541aea. Independent review verifies all ten actual profile stages across the two performance players with the current validator and unchanged canonical/CPU profiles. Both schema-2 observations agree. Shipping was skipped in the ordinary correctness jobs, so this does not establish new shipping-player acceptance. An actual preregistered incremental experiment, native-output retention, and independent remote restoration remain open.

Checklist evidence audit

The four checked mechanisms are implemented in the canonical profile, generated runner, shipping player, and profile/shipping validation tests. The current profile pins OptimizeSpeed, including Unity 2021 compatibility. Retained native configured and postbuild evidence from artifact 10006223200 records that mode for Unity 6000.5.2f1 and canonical hash e1e525d1a80eb4dbd8e547853224e3175187e1050f014423116d4bb6f20a4ac2; its retained verification hashes were independently recomputed. Current-source script CI passes on Windows, macOS, and Linux, including per-setting drift and accidental test-inclusion rejection. These completed mechanisms do not establish every stripping/message-shape factor, complete native/toolchain provenance, or durable retention. Those unchecked acceptance requirements remain open.

Native build-input retention in PR #555

PR #555, source fc3927cf63f7061b01b8808cf3aa2a6b1fdcc930, adds bounded capture of the actual build-log-selected Bee player graph and input JSON. Cold generator and backend identities must agree. Cached backend-only builds retain the matching JSON companions. The binary DAG is hashed. Log-referenced response files are retained as scanned text.

The source manifest binds original file hashes to the profile, requested editor, and build log. The existing final artifact scan records retained-byte hashes in a separate acceptance manifest. It removes reachable stale acceptance on inventory failure, validates every group before publication, and rolls back acceptance files after a write failure. Missing, ambiguous, empty, linked, oversized, and unsafe inputs fail closed. Capture runs inside existing profiled builds; it adds no Unity launch or timing window.

All 421 redactor tests and the profile suite pass locally. Negative fixtures reproduce stale-manifest failures before the fix. A six-log audit covers actual historical cold and cached build selection; historical input bytes were absent, so it does not prove successful native capture. Independent review is clear. The final native performance run is pending.

The retained response scope is explicitly build-log-references-only. Actual CI capture acceptance, transitive response closure, native executable versions and payload retention, the full stripping matrix, and independent immutable restoration remain open.

Native acceptance investigation

The earlier PR-source internal job 101828680328 passes all 202 tests and captures the logged Bee DAG. Final artifact validation rejects that DAG with the generic encoded-data/format-control warning and correctly blocks upload. License return and lock cleanup pass. The exact Windows rejection category is still unknown.

Two real local macOS Bee files pass structural inspection and safe redaction. They do not reproduce the Windows failure. A reviewed diagnostic change adds only fixed reason codes and byte size, with no acceptance-rule relaxation or exposure of offending values. All 1,298 script tests pass. This native artifact rejection must be resolved before the retention increment is accepted.

Resolution: format-control neutralization (PR #555, commits caca299 and 56143fc)

The diagnostic head named the category: Playereb61d75c.dag.json was refused with scalar-format-control, so a decoded JSON scalar held a literal Unicode format-control character. After literal scalars were made visible, the next run still refused with unsafe-structured-data, which narrowed to two remaining sources: a structured scalar beginning with a byte-order mark that the offset-0 carve-out preserved, and escape text that the serialized-shadow decoder resolves back into an invisible character.

Redaction now rewrites the whole class into visible [cf:xxxx] markers: literal format-control characters and lone surrogates, JSON \uXXXX escape runs with valid surrogate pairs preserved verbatim, and XML numeric entity chains, each counted as scalar-format-control. The byte-order-mark carve-out now applies only at document level. Encoded-Cf shadows, control-character keys, and structural defects still refuse, so no acceptance rule was relaxed. Two vector rows moved from refusal to exact-byte acceptance and their raw sources also block immutable sealing. The JS budget history entry 087 records the reviewed raise to 24060.

CI acceptance at 56143fc3 (run 34160405988): all 46 checks pass. The internal perf leg redacted the exact previously rejected file (Playereb61d75c.dag.json: scalar-format-control) and uploaded perf-6000.5.2f1-standalone-internal.zip (198,665 bytes); the comparisons leg and the performance aggregate gate pass. Local verification: npm 1298/1298, validate:all, spelling, and the budget gate. Timing against the merged-main baseline keeps editor Unity execution within noise (126-146s versus 126-150s); the perf redaction phase grows about 10 seconds per perf job from scanning the newly retained graph and input files.

Native upload is unblocked; actual retained-artifact identity checks, transitive response closure, the stripping matrix, and independent immutable restoration remain open.

[wallstop]

PR #517 advances the closed-message-cardinality portion of this protocol.

Completed evidence:

• Added separate closed cardinality topologies for exactly 1, 16, 256, and 1,000 public readonly struct messages; the existing 18-shape semantic topology remains independent.
• Built every topology at Minimal, Low, Medium, and High managed stripping on Unity 2021.3.45f1 and 6000.5.2f1.
• Passed all 40 endpoint players: 20 per editor, including all 32 new cardinality players.
• Each cardinality player proves source-generated registration, typed dispatch, registered untyped dispatch, the expected missing-root failure, exact ordered type inventory, and an unchanged stripped binary between positive and mutant runs.
• Evidence schema v2 binds topology ID and exact message count across generated project inputs, runtime results, and final binary manifests. Exact fail-closed validators reject outer and nested schema, type, order, count, path, length, and hash drift.
• Windows PowerShell 5.1 script tests, the 518-test script suite, 257 source-generator tests, repository validation, performance jobs, all four Unity editor jobs, CI aggregates, and Cursor Bugbot are green on commit 3b08053e.

This completes the exact 1/16/256/1,000 AOT-root and provenance matrix item in PLAN.md. Issue #506 remains open for the distinct clean-versus-incremental build, cold/startup/build-time/player-size evidence, and content-addressed retention dependency in #508.

[wallstop]

PR #518 adds the build-time, player-size, and cold-start evidence this protocol asked for. Both
endpoint editors passed all 20 cells, so 40 stripped IL2CPP players were built and run.

Each cell now writes a build report from the same BuildReport that proves the final build options,
a timing block measured inside the stripped player, and a joined cell summary. The matrix wrapper
collects every completed cell into one file per endpoint editor and prints a table. All of it is
labeled characterization; none of it enters the published baseline.

Native code size scales with closed message count

GameAssembly.dll bytes at High stripping:

Message types	Unity 2021.3.45f1	Unity 6000.5.2f1
1	8,297,984	11,016,192
16	9,274,880	11,735,552
256	24,915,456	23,268,352
1,000	73,405,440	58,969,088

That is 8.8x on 2021.3 and 5.4x on 6000.5 from 1 to 1,000 types. This is the code-size cliff the
cardinality matrix exists to expose.

Build time scales with it

Time inside BuildPipeline.BuildPlayer at High stripping on 2021.3: 35.6 s at 1 type, 37.3 s at 16,
63.3 s at 256, and 166.7 s at 1,000. The same shape appears on 6000.5.

Stripping level moves size, not dispatch cost

At 1,000 types on 2021.3, GameAssembly.dll is 95.0 MB at Minimal, 95.0 MB at Low, 75.4 MB at
Medium, and 70.0 MB at High. Minimal and Low produce identical sizes.

Dispatch cost does not scale with cardinality

The timed dispatch loop reads 32.0 to 33.3 ns per operation on 2021.3 and 23.5 to 27.2 ns on 6000.5,
across every stripping level and every cardinality from 1 to 1,000 message types. Steady-state
dispatch does not get more expensive as the closed message set grows. Treat this as
characterization from one clean build and one fresh player launch, not a confirmatory result: it
uses a fixed-count average that includes the harness counting each delivery, and only cells that
loop the same message shape are comparable.

Observations that need repetition before interpretation

First typed dispatch is a single observation per cell and it is noisy. On 2021.3 the
High-stripping 1,000-type cell read 2,657 us against 211 to 422 us everywhere else, and on 6000.5
the Minimal and Low 16-type cells read about 1,200 us against 245 to 424 us elsewhere. These need
repeated launches before anyone draws a conclusion from them.

Engine start to first script is 1 ms on 2021.3 and 2 to 19 ms on 6000.5. The validator accepts 0
because a batchmode player can reach the first script before that clock advances.

Remaining for this issue: hand the content-addressed native outputs to the durable bundle tracked
by #508, and separate clean from incremental builds as a distinct factor. Every build in this leg is
already clean because the builder always sets BuildOptions.CleanBuildCache; only the editor
Library cache state varies, and each cell now records it.

[wallstop]

The clean/incremental audit found that toggling CleanBuildCache alone is insufficient: run-ci-tests.ps1 deletes the prior shipping-player output and requires a fresh executable timestamp. Existing warm-Library metadata does not prove incremental build behavior.

A bounded next implementation should keep canonical publication clean-only and preserve the default 40 endpoint cells. A shipping-only opt-in clean/incremental pair needs the same project/source/topology, a validated clean predecessor, explicit base-profile plus build-factor identity, retained clean native outputs, and a fresh nonce-bound BuildReport. Incremental evidence must never pass as the unchanged clean profile; dependent incremental runs must fail closed when the clean predecessor fails.

No runner change or native incremental experiment was made in this increment. The audited design and remaining native-output/restore requirements remain in scope; PR #524 does not close this issue.

[wallstop]

Independent retrieval of retained native outputs proven

The retained content-addressed native outputs from the accepted PR #555 perf run were independently retrieved and re-verified from a fresh clean checkout (git clone --no-hardlinks at 73627f7e), completing the retrieval leg of "Retain content-addressed native outputs through #508/#521 and prove independent retrieval" for the shipping-fidelity profile work.

Source: perf run 34160405988 at head 56143fc312c4d054e68233da1068677b4e22b39a.

• Artifact 10032601073 perf-6000.5.2f1-standalone-internal, ZIP SHA-256 f6a28480a5f1fb8ee2ca8fa6f3731e0cc906c2769273c33456097a147bc50228.
• Artifact 10032771893 perf-6000.5.2f1-standalone-comparisons, ZIP SHA-256 3fd1a21c470970de95da2435d3101cdb9c4c85466e201acd7d008106263eb123.

Both archives carry native-build-inputs/retained-manifest.json with captureState: redacted, profileId: canonical-il2cpp-verdict-player-v1, graphSelection: generator-and-backend-agree, and six retained files (inputdata, DAG JSON export, four response files). Every retainedLength and retainedSha256 re-verifies against the actual archive bytes: the internal redacted DAG JSON is 2,940,183 bytes (matching the session 272 acceptance record) and the comparisons DAG JSON is 4,090,344 bytes. A BOM-aware literal format-control scan over every JSON, TXT, and LOG entry in both archives found zero findings beyond any leading byte-order mark.

This is ordinary CI artifact retention, not immutable publication: the artifacts expire on schedule, and sealed-bundle publication stays behind the #521 approval gate. Native executable identity (player binary digests) remains an open item; the retained inputs bind the logged build, not the stripped binary.