Smaller iOS PR gate, automatic risk selection and full nightly coverage

Revision 14 — 4 November 2026. This replaces revision 9's smoke/shadow rollout and Android expansion. Earlier comments describe earlier revisions. Implementation is prepared in coordinated E2E/iOS draft PRs; default-branch cutover has not happened.

Drafts: E2E producer #291, iOS consumer #1199. Initial development-branch five-case validation: run 41709148174, all five cases passed on the first attempt at iOS f0a4aa3 / E2E 39f8964; test execution 27m36s, Mac job about 30 minutes. This predates the policy relocation and is not evidence of its hosted execution. Both PRs remain draft and neither is merged.

Scope

Keep Appium and reuse existing assertions and fixtures. Change iOS orchestration only. Android's workflows, PR gate, nightly selection and legacy retry behavior remain unchanged. Shared E2E reporting is opt-in; an optional immutable fixture-source input preserves existing Android callers.

There is no shadow workflow, no newly invented smoke suite and no requirement to put every job in one YAML file. The existing iOS e2e-tests.yml owns selection and the required e2e-status. Existing staging and migration workflows provide reusable execution lanes and retain standalone manual dispatch.

What runs automatically

Automatic runs still defer intermediate stacked PRs that do not target master. Manual PR-merge validation supports their actual base/head. Existing verified retarget reporting remains in place. PR events run the usual local/staging gate. A release push owns its migration run, including when there is no open PR.

Automatic selection

The iOS repository owns the policy in .github/e2e-policy.json, its selector in .github/scripts/e2e-selection.cjs, and policy tests/CI. The E2E repository owns case definitions, inventory, fixtures and execution/reporting. App paths change with their policy in the same iOS PR. Selection loads policy from the actual tested app checkout and records its SHA/digest separately from the E2E SHA.

There are no blanket Services/Models/Managers/Utilities full-suite rules. Feature directories and explicit service/model/helper files select relevant cases. Widget services add widgets; Paykit services add staging; shared recovery state selects recovery/migration and applicable profile restoration. Shared scan/payment paths retain their Pubky/Electrum/hardware consumers. On PRs, full local/staging coverage is reserved for explicitly named cross-cutting wallet state, lifecycle and build dependencies.

A cheap Linux E2E selection policy workflow runs on every PR, including drafts/stacks. It checks all tracked paths, stale entries and unavailable test IDs, tests actual source-path examples and previews the PR selection without launching device jobs. Main selection performs the same checks before builds. Unknown paths fail with an actionable update request. Renames use both old/new paths and base/tested policy mappings, preventing a policy edit from silently waiving relevant existing coverage on the same code change.

selection.json records exact IDs, reasons, policy/app/test identity and gaps. USDT-only changes keep the Bitcoin base regression gate and explicitly note missing USDT E2E coverage; unrelated Bitcoin tests cannot prove USDT behavior. Existing native/integration coverage remains necessary.

The nightly set contains all currently executable iOS local/UI/payment/hardware cases, pubky_profile_1–5 and paykit_1, plus native restore/upgrade and four RN migration variants. Fixture-preparation cases are prerequisites, not additional product passes.

Five existing skips (hardware_wallet_3, multi_address_2, transfer_1, transfer_max, settings_08) and six unsupported iOS mainnet cases remain explicit gaps. The interactive receive-payment utility stays manual. This work does not claim those gaps, or missing new-feature automation, are covered. Android/private mainnet orchestration is unchanged.

Attempts and trustworthy reporting

• Allow up to three attempts, stopping after the first successful complete attempt.
• Keep the first failure visible after recovery, with per-attempt runner logs, worker results, timing and diagnostics.
• Bind each plan to the selected app SHA, actual E2E checkout, downloaded app digest, backend, variant and run identity.
• Require every selected case to execute and pass. Missing reports, zero matches, unexpected skips, stale identities, setup/teardown errors and failed runner processes cannot produce a pass.
• Run selected spec files sequentially in one WDIO worker per simulator. Local/staging retries rerun the selected pack; migration preserves its existing per-attempt wallet preparation.
• The existing required e2e-status includes every selected lane and its build. No new ruleset/check-name cutover is needed.

Cost controls and two-shard experiment

Automatic runs use one local shard. Manual dispatch can request two disjoint local shards, keeping each spec's fixture lifecycle together. Each shard owns its simulator and temporary backend; both download the same local app build. Staging and migration share one network app build.

Keep draft/path/stack filtering, cancellation of superseded runs and backend cleanup. Builds are bounded to 40 minutes and diagnostics to seven days. Base-sized local selections and individual migration cases retain 40-minute attempts within 135-minute jobs. Local shards of 6–30 cases and the six-case staging pack allow 90-minute attempts within 285-minute jobs. Local shards above 30 cases allow 110-minute attempts within 345-minute jobs. Their backend expiry is six hours, compared with five hours for shards of 6–30 cases. Each shard destroys its backend on completion. These are ceilings, not reserved runtime: successful attempts stop retries immediately. Migration fixture preparation has a 90-minute bound.

Compare selection=full, shards=1 with selection=full, shards=2 on the same app/E2E revisions. Record total elapsed time, setup/test time, first-attempt reliability, retries and actual Namespace Mac/backend usage. Do not run both arms automatically on every PR or enable two shards by default without evidence. The initial partition uses spec/case grouping; duration-based balancing follows measurements. No billing savings or runtime target is claimed yet.

The earlier Namespace sample demonstrated concurrent runners, not unlimited capacity or a queue-free service. Account-level spending alerts/concurrency limits and invoice measurements are operational follow-up; this implementation does not claim to configure them.

Validation and merge order

1. Publish coordinated E2E producer and iOS consumer drafts. Merge the producer first after validation; merge the consumer to replace the current gate in place. The default branch's existing gate stays intact until that merge.
2. Before merging, dispatch the changed iOS workflow from its development branch, explicitly selecting the E2E development branch. selection=base checks the five cases; selection=full exercises local/staging coverage and selection=nightly reproduces the full nightly on a branch; selection=auto with pr=NUMBER tests a verified PR merge, including a stack. The fixture workflow is pinned to a producer commit supporting immutable source selection, so premerge full validation does not depend on an unreleased input on main.
3. Confirm actual hosted case execution, first-attempt/recovered outcomes, cleanup and retained artifacts. Exercise missing/failed selection and jobs through the automated contract tests. Local controlled-runner checks do not count as hosted Appium passes.
4. Run the optional matched two-shard comparison separately. Keep one shard if reliability or paid compute worsens.

Implementation validation currently includes 101 passing shared tests and 32 passing iOS policy tests, inventory validation, TypeScript checking, actionlint and executable aggregate-status checks. Existing native-unit evidence records 2,969 passes, zero failures and five skips at its original app revision; independent review assesses its applicability because native inputs are unchanged. Hosted redesigned-gate and cost measurements remain separate validation work and must name their actual revisions/results.

Current hosted validation

The revised policy is on iOS 5ba352f7 with shared documentation at ca9e731. All 32 selector tests and the workflow contract checks passed. Ordinary PRs run the five-case gate plus affected local/staging coverage. All six migration cases run every night at 03:00 UTC and independently on each push to an iOS release-* branch. A branch dispatch can explicitly reproduce nightly with selection=nightly. Full PR coverage and selection=full use the regular local and staging suites.

The one-shard nightly rehearsal is running at iOS 5ba352f7 and E2E ee7a429. The matched two-shard rehearsal follows a complete pass on the same revisions. Current validation

The preceding run restored the correct 449,502 sats in RN, but its upgrade and passphrase tests could not read the nested amount on iOS. The corrected test reads the balance from its parent accessibility label. A local iOS simulator using the hosted target build and funded RN fixture passed the passphrase upgrade, including balances, history and tags after relaunch. Preceding run

The previous full run passed all six staging cases on the first attempt and five of six migration cases within three attempts. All 12 RN source wallets were prepared. The RN passphrase case failed. One recording showed the correct 449,502 sats behind a USDT introduction. Another showed a truncated passphrase and an empty wallet before the upgrade. The harness now checks the exact passphrase and funded RN balance before installing the native app, then dismisses the introduction before verifying balances and history. Migration job

The previous local job reached its 90-minute limit three times, with 37, 38 and 39 of 42 cases passing on the respective attempts. Security and transfer each passed on a subsequent attempt. The first PIN reset showed a regtest log-deletion error and returned to home instead of onboarding. The transfer recording showed QuickPay covering the activity assertions. The security assertion remains in the suite. The transfer case handles the one-time introduction and still verifies the balance and both activity rows. Local job

That unsuccessful run consumed 271.72 Namespace Mac job-minutes and 271.00 observed backend minutes. Published rates imply $18.47 prepaid-equivalent or $27.71 overage for compute. The estimate excludes storage, credits, rounding and account discounts. Run · Namespace pricing

The initial diagnostic also exposed a redundant Android Back after RN send success that backgrounded the source wallet. Its correction was exercised by all three sweep wallets in the next run. Initial diagnostic

Rollback is the prior iOS workflow configuration using Namespace. No Mini restoration or Android change is required.