Skip to content

ci: shrink iOS gate and run full nightly coverage - #1199

Draft
piotr-iohk wants to merge 17 commits into
masterfrom
codex/264-inventory-shadow
Draft

piotr-iohk wants to merge 17 commits into
masterfrom
codex/264-inventory-shadow

Conversation

@piotr-iohk

@piotr-iohk piotr-iohk commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

Part of synonymdev/bitkit-e2e-tests#264.

Dependency: synonymdev/bitkit-e2e-tests#291

This PR replaces the broad iOS PR selection with five existing base cases plus automatically selected risk coverage, and runs the full applicable suite nightly.

Description

  • Uses send_1, send_2, lightning_1, transfer_2 and boost_2 as the base gate. The app-owned .github/e2e-policy.json maps actual feature paths and explicit services to existing cases. Named wallet/lifecycle/build dependencies select all local and staging cases on PRs. Unclassified paths fail cheap validation before builds.
  • Adds an inexpensive Linux policy check on every PR, including drafts/stacks: validates all tracked paths, stale mappings and case IDs, tests actual source examples, and previews selection. The policy follows the tested app commit; renames retain both old/new classifications.
  • Runs 42 local, six staging and six migration cases nightly, including unchanged revisions. e2e-tests.yml owns local/staging and its required e2e-status. The independently scheduled e2e_migration.yml owns migration and its result.
  • Keeps up to three attempts with first-failure reporting, exact selected-case coverage and pinned app/test/build identities.
  • Reuses one build per backend, bounds job/attempt time, retains diagnostics for seven days and offers a manual two-shard experiment. Automatic local execution defaults to one shard.
  • Supports manual validation from the development workflow branch, including verified stacked-PR merge commits. No shadow workflow is added. Native checks and Android workflows are unchanged.

Merge the E2E producer first, then this consumer after branch validation. The pinned migration fixture workflow supports premerge full validation without relying on new inputs already being on main.

Design

N/A — no UI changes.

Preview

N/A

QA Notes

Setup

Use the workflow from codex/264-inventory-shadow and explicitly select the E2E branch of the same name. The branch name is historical; its final tree has no shadow workflow. Docs/e2e-ci.md contains executable dispatch commands. Local cases use isolated Namespace backends; staging/migration use existing remote-regtest fixtures. All selected app and test revisions are recorded in the selection artifact.

Journeys

N/A — no user-visible behaviour change.

Manual Tests

N/A

Automated Checks

  • updated e2e-tests.yml — smaller base selection, automatic risk coverage, full nightly lanes and required aggregate (after merge).
  • ran workflow contract checks and actionlint — shared build settings/reuse, immutable fixture handoff and 19 aggregate-status scenarios, including selected failures, skips and cancellations.
  • ran 99 shared E2E unit/contract tests and 29 iOS policy tests — selector, retry evidence, reporting identity, installed WDIO scheduling and fixture-source compatibility.

Evidence

The earlier hosted iOS policy check passed all policy validation/tests and produced the actual PR selection preview, using PR merge 7513f821a0e157859453a161d4a19797136d486a and E2E ee9e8bed6147ce0cc6ccb446e7df598e9614ff31. The shared inventory/reporting check also passed. This verifies selection, not device execution.

Inventory validation, TypeScript checks, all 99 shared tests, 29 iOS policy tests and workflow validation passed at E2E ee9e8bed6147ce0cc6ccb446e7df598e9614ff31 / iOS 0eb027de7f2d24dd9e1a4134552c17a14997089e. Policy tests cover the actual tree, exact domain cases, caller dependencies (scanner/backup/Keychain/payment flows), renames and unknown/stale paths. Independent review verified the correction.

The actual-source audit changes full selection from 347/619 Swift files (all 76 services) to 28/619 (three explicitly shared services). This is a selection audit, not a measured billing reduction.

The initial five-case hosted run passed all five on its first attempt at iOS f0a4aa3 / E2E 39f8964: run 41709148174. Test execution was 27m36s, Mac job about 30 minutes, complete workflow about 36 minutes. This predates policy relocation; its coverage is not relabeled as current-head execution.

The earlier native-unit invocation built the app and recorded 2,969 passes, zero failures and five skips at iOS 5c79e56aa6e31c2fd197fd90bf926818ccf56153, on the local iOS 26.0 simulator. Independent review retained this evidence after inspecting the entire intervening delta: native source, project/build inputs, dependencies and unit-test selection are unchanged. This is historical native evidence, not a current hosted E2E pass.

Gaps

Hosted full-lane execution remains to be recorded before cutover; the cheap current-PR selection preview is checked separately. The five-case runtime pass remains at its original pins. The two-shard cost comparison has not run; no elapsed-time or billing improvement is claimed. Five existing skipped cases and six unsupported iOS mainnet cases remain documented gaps.

Current full-suite validation

At iOS 94088e98, the regular E2E workflow owns 42 local and six staging cases. Migration has an independent nightly and release-* push workflow, app build and result. The E2E workflow has no migration call or dependency. A regression check enforces that boundary.

The hosted one-shard baseline at iOS 5ba352f7 and E2E ee7a429 passed all 54 cases under the preceding combined orchestration. Its 42 local cases passed first try in 95m01s. Observed Namespace usage was 96.57 Mac job-minutes and 95.93 backend minutes, estimated at $6.56 prepaid-equivalent or $9.84 overage. All six migrations passed, including passphrase on its first attempt. Baseline

The first two-shard dispatch mistakenly repeated migration and was cancelled. Its additional estimated compute was $1.55 prepaid-equivalent or $2.32 overage. The replacement selected 42 local cases in two shards and six staging cases at the identical revisions. Its selection artifact confirms migration: false. Only local elapsed time and paid Namespace compute are comparable between the two runs. Cancelled dispatch · Regular E2E comparison

The preceding run restored the correct 449,502 sats in RN, but its upgrade and passphrase tests could not read the nested amount on iOS. The corrected test reads the balance from its parent accessibility label. A local iOS simulator using the hosted target build and funded RN fixture passed the passphrase upgrade, including balances, history and tags after relaunch. Preceding run

The previous full run passed all six staging cases on the first attempt and five of six migration cases within three attempts. All 12 RN source wallets were prepared. The RN passphrase case failed. One recording showed the correct 449,502 sats behind a USDT introduction. Another showed a truncated passphrase and an empty wallet before the upgrade. The harness now checks the exact passphrase and funded RN balance before installing the native app, then dismisses the introduction before verifying balances and history. Migration job

The previous local job reached its 90-minute limit three times, with 37, 38 and 39 of 42 cases passing on the respective attempts. Security and transfer each passed on a subsequent attempt. The first PIN reset showed a regtest log-deletion error and returned to home instead of onboarding. The transfer recording showed QuickPay covering the activity assertions. The security assertion remains in the suite. The transfer case handles the one-time introduction and still verifies the balance and both activity rows. Local job

That unsuccessful run consumed 271.72 Namespace Mac job-minutes and 271.00 observed backend minutes. Published rates imply $18.47 prepaid-equivalent or $27.71 overage for compute. The estimate excludes storage, credits, rounding and account discounts. Run · Namespace pricing

The initial diagnostic also exposed a redundant Android Back after RN send success that backgrounded the source wallet. Its correction was exercised by all three sweep wallets in the next run. Initial diagnostic

Models used

  • Planning/scoping: gpt-6-astra (reasoning: high), gpt-6-astra (reasoning: xhigh)
  • Implementation: gpt-6-astra (reasoning: high), gpt-6-astra (reasoning: xhigh)
  • Review: gpt-6-astra (reasoning: high)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant