test: rewrite the test suite, test plans and CI

.claude/skills/fix-issue/references/verification-apple.md

@@ -42,7 +42,7 @@ A raw `xcodebuild` failure comes back as a head-and-tail excerpt of about 10,000
 - **The accessibility tree differs between this machine and the CI runner.** A contextual `NSMenu` has its identifier locally and not on CI, so reach it with `app.windows.firstMatch.menus.firstMatch`. The data grid is identifier `data-grid`, since the window holds several tables. Click the grid at an offset from `data-grid`, never on a row or cell element, which XCUITest reads as obscured.
 - **`.accessibilityIdentifier` on a SwiftUI container replaces the identifier of every child** in that hosting tree. When the container needs one, put `.accessibilityElement(children: .contain)` before it.
 - **Diagnose by dumping the tree** (`print(app.windows.firstMatch.debugDescription)` inside a `UITestCase`), not by guessing.
-- **When UI tests cannot start on this machine** (`Timed out while enabling automation mode`, zero cases), reproduce it on an untouched suite, then say in the PR that CI runs the suite. When a test fails only on the runner for a reason you cannot establish, quarantine it in `.github/macos-ui-test-quarantine.txt` with the reason instead of guessing at a fix.
+- **When UI tests cannot start on this machine** (`Timed out while enabling automation mode`, zero cases), reproduce it on an untouched suite, then say in the PR that CI runs the suite. When a test fails only on the runner for a reason you cannot establish, mark it XCTExpectFailure with the reason and an issue link instead of guessing at a fix.
 
 ## Lint
 

.claude/skills/fix-issue/scripts/verify.sh

@@ -1,11 +1,10 @@
 #!/usr/bin/env bash
 #
-# Run one TablePro verification step, store the full output on disk, and print a short verdict.
+# Run one TablePro verification step, keep the full output on disk, and print a short verdict.
 #
-# The point is context economy: a Debug build prints thousands of lines and a test run enumerates
-# every case, which is unreadable and crowds out the work. This keeps the full log on disk and
-# prints at most ~30 lines: status, the real errors, test counts, and whether a failing case is
-# quarantined.
+# A Debug build prints thousands of lines and a test run lists every case. This keeps that log on
+# disk and prints about 30 lines: the status, the real errors and the executed counts. A test
+# verdict comes from the result bundle or the xunit file, never from the console banner.
 #
 # Usage:
 #   verify.sh [options] <step> [args]           # a step of the platform, listed below
@@ -22,16 +21,14 @@
 #   --no-wait       do not wait for a concurrent build of the same checkout to finish
 #   --offline       resolve dependencies without the network
 #
-# Exit: 0 pass, 1 fail, 2 inconclusive (environment, not the change), 3 usage error.
+# Exit: 0 pass, 1 fail, 2 inconclusive (a named environment cause, not the change), 3 usage error.
 
 set -uo pipefail
 
 SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 
 REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../../.." && pwd)"
-# Run from inside a worktree, the absolute path of the main checkout's copy of this script used to
-# verify the main checkout instead, which produced green runs of the wrong tree. When the current
-# directory is a checkout of this same repository, that checkout is the default.
+# Called by absolute path from a worktree, this script would otherwise verify the main checkout.
 caller_root="$(git rev-parse --show-toplevel 2> /dev/null)"
 if [ -n "$caller_root" ] && [ "$caller_root" != "$REPO_ROOT" ] \
     && [ "$(git -C "$caller_root" rev-parse --path-format=absolute --git-common-dir 2> /dev/null)" \
@@ -46,9 +43,7 @@ HELP=0
 WAIT_FOR_BUILD=1
 OFFLINE=0
 
-# Asking for help is not a usage error, so -h exits 0. Anything else exits 3, which a caller
-# running under `set -e` can distinguish from a real verdict. The steps are the platform's, so
-# they are printed from its backend, when one was found.
+# -h exits 0; any other usage error exits 3, which a caller can tell apart from a verdict.
 header_of() {
     awk 'NR > 2 { if (!/^#/) exit; sub(/^# ?/, ""); print }' "$1"
 }
@@ -63,18 +58,13 @@ need_value() {
     [ "$1" -ge 2 ] || { echo "$2 needs a value" >&2; usage; }
 }
 
-# Options are read wherever they appear, not only before the step. The loop used to stop at the
-# first non-option, so everything after the step name reached the step as an argument: `verify.sh
-# build --offline` built a scheme named `--offline` and reported FAIL rather than a usage error,
-# which reads like the change broke the build.
+# Options are read before or after the step, so a trailing option never becomes a step argument.
 POSITIONAL=()
 while [ $# -gt 0 ]; do
     case "$1" in
         --run) need_value $# "--run"; RUN_DIR="$2"; shift 2 ;;
         --root)
             need_value $# "--root"
-            # Without this check a bad path leaves REPO_ROOT empty, and the run then builds and
-            # reports on a tree that is not the one asked for.
             REPO_ROOT="$(cd "$2" 2> /dev/null && pwd)" || { echo "--root: no such directory: $2" >&2; exit 3; }
             [ -n "$REPO_ROOT" ] || { echo "--root: no such directory: $2" >&2; exit 3; }
             shift 2
@@ -90,8 +80,7 @@ done
 # bash 3.2 treats an unset array as unbound under `set -u`, so an options-only run needs the guard.
 set -- ${POSITIONAL[@]+"${POSITIONAL[@]}"}
 
-# A step belongs to a platform: xcodebuild cannot build the Linux app and meson cannot build the
-# macOS one. The default is the platform this machine can build, which is its operating system.
+# The default platform is the one this machine can build, which is its operating system.
 if [ -z "$PLATFORM" ]; then
     case "$(uname -s)" in
         Darwin) PLATFORM=apple ;;
@@ -110,8 +99,7 @@ if [ -z "$RUN_DIR" ]; then
     RUN_DIR="$REPO_ROOT/.analysis/${branch//\//-}"
 fi
 LOG_DIR="$RUN_DIR/logs"
-# Only the steps that produce a log create the directory. `parse` and `tail` read an existing
-# log and used to leave an empty logs/ tree wherever --run pointed.
+# `parse` and `tail` read an existing log, so they create no logs/ directory.
 case "$STEP" in
     parse | tail) ;;
     *) mkdir -p "$LOG_DIR" ;;
@@ -124,9 +112,7 @@ NOTES=("")
 
 note() { NOTES=("${NOTES[@]}" "$1"); }
 
-# The verdict is also written beside the log, because some of what decides it (the exit code, the
-# root, the agent-docs result) is not in the log, and `parse` replaying the log alone reached a
-# different verdict than the live run more than once.
+# The verdict is written beside the log: part of what decides it, like the exit code, is not in the log.
 emit() {
     local log="$1" exit_code="$2" receipt=""
     [ -n "$log" ] && [ "$STEP" != "parse" ] && receipt="$log.verdict"
@@ -150,8 +136,7 @@ emit() {
     esac
 }
 
-# Two quick steps can start in the same second, and the second one used to overwrite the first
-# one's log and receipt, so the earlier verdict could no longer be re-read.
+# Two steps can start in the same second; neither may overwrite the other's log.
 new_log() {
     local base n=1
     base="$LOG_DIR/$1-$(date +%H%M%S)"
@@ -160,7 +145,6 @@ new_log() {
     printf '%s-%s.log' "$base" "$n"
 }
 
-# What a backend asks about the options, so it never reads the option variables themselves.
 waits_for_other_builds() { [ "$WAIT_FOR_BUILD" -eq 1 ]; }
 is_offline() { [ "$OFFLINE" -eq 1 ]; }
 
@@ -170,8 +154,7 @@ run_logged() {
     return $?
 }
 
-# A tool this machine does not have is the machine, not the change. Without this the step ran
-# anyway, the shell printed "command not found", and the verdict was FAIL with exit 127.
+# A missing tool is the machine, not the change.
 require_tool() {
     command -v "$1" > /dev/null 2>&1 && return 0
     STATUS=INCONCLUSIVE
@@ -189,9 +172,8 @@ available_platforms() {
     printf '%s' "${names# }"
 }
 
-# A platform named on the command line that has no backend is a typo, so it is a usage error. One
-# picked from the operating system is a checkout that cannot be verified on this machine, which
-# says nothing about the change.
+# A platform named on the command line with no backend is a typo; one picked from the operating
+# system is a checkout this machine cannot verify, which says nothing about the change.
 load_platform() {
     if [ ! -f "$BACKEND" ]; then
         if [ "$PLATFORM_GIVEN" -eq 1 ]; then
@@ -212,7 +194,6 @@ load_platform() {
 case "$STEP" in
     tail)
         [ $# -ge 1 ] || usage
-        # A mistyped path used to print the shell's error and still exit 0, which reads as success.
         [ -f "$1" ] || { echo "no such log: $1" >&2; exit 3; }
         tail -n "${2:-60}" "$1"
         exit 0
@@ -236,9 +217,7 @@ case "$STEP" in
         ;;
 
     agent-docs)
-        # CLAUDE.md, .claude/rules and this skill, checked for paths, symbols and skills that no
-        # longer exist. It used to run inside every `lint`, which made
-        # every code lint red whenever main carried one stale reference nobody on the branch wrote.
+        # Kept out of `lint`, so one stale reference on main never turns a code lint red.
         log="$(new_log agent-docs)"
         run_logged "$log" "$REPO_ROOT/scripts/check-doc-symbols.sh"
         code=$?

.claude/skills/release/SKILL.md

@@ -45,19 +45,28 @@ Check each one. If any fails, stop and say what is wrong.
    in `Configs/Version.xcconfig`, and its tag is free (`git tag -l "v<version>"`). Ask if it is missing.
 2. The tree is clean (`git status --porcelain`), or the user says the changes belong in the release.
    `[Unreleased]` has entries; if not, say the release has no notes.
-3. `swiftlint lint --strict` is clean. Fix findings first, in their own commit.
+3. `swiftlint lint --strict` is clean. Fix findings first, through their own pull request.
 4. If `Plugins/TableProPluginKit/` changed since the last tag, settle the ABI now (stage 3, PluginKit),
    because a breaking bump publishes plugins before the app tag.
 5. `scripts/export-cloudkit-schema.sh --check` passes. A field in the snapshot but not in Production
    makes CloudKit reject every record that carries it, so the user deploys it in CloudKit Console
    before the tag. It needs the cktool management token on this Mac.
-6. Warn, without blocking, when you are not on `main`, and report the last finished full-suite
-   verdict on `main` with its commit. `main` is often red on merge skew; the tag's own run is the gate.
+6. Stop when you are not on `main` or it is not `origin/main`. The tag runs no suite: its `Tested on main` job takes, for
+   `macOS Tests` and `iOS and Package Tests` each, the newest `main` commit where that job ran, and
+   fails the release when it is red or when files the suite reads changed after it. Report both
+   verdicts below, and stop when either is a failure.
 
    ```bash
-   gh run list --workflow=macos-tests.yml --branch main --limit 30 --json conclusion,headSha,createdAt \
-     -q '[.[] | select(.conclusion == "success" or .conclusion == "failure")][0]
-         | "\(.conclusion) \(.headSha[0:9]) \(.createdAt)"'
+   for pair in 'macos-tests.yml|macOS Tests' 'ios-tests.yml|iOS and Package Tests'; do
+     for id in $(gh run list --workflow "${pair%%|*}" --branch main --event push --limit 20 \
+         --json databaseId -q '.[].databaseId'); do
+       verdict=$(gh run view "$id" --json headSha,jobs \
+         -q ".headSha[0:9] + \" \" + ([.jobs[] | select(.name == \"${pair#*|}\") | .conclusion][0] // \"\")")
+       case "$verdict" in *' ' | *skipped) continue ;; esac
+       echo "${pair#*|}: $verdict (run $id)"
+       break
+     done
+   done
    ```
 
 ### Credit the entries
@@ -164,13 +173,26 @@ release with the flag.
 
 ### Commit, tag, push
 
+`main` takes no direct push. The release commit is the one exception: it goes up with the release
+deploy key `~/.ssh/tablepro_release`, the only bypass in main's ruleset. The key has a passphrase and
+is never added to ssh-agent, so you cannot run that push: ask the user to run it in their own
+terminal, wait for them to confirm it landed, then push the tag. The key skips every check, so
+before you hand it over, prove the push carries the release commit and nothing else. Stop if the
+check prints nothing.
+
 ```bash
 git add Configs/Version.xcconfig CHANGELOG.md .github/release-flags.json \
     TablePro/Resources/WhatsNew.md
 scripts/check-banned-words.sh --staged
 git commit -m "release: v<version>"
+git fetch origin main
+[ "$(git branch --show-current)" = main ] \
+  && [ "$(git rev-parse HEAD^)" = "$(git rev-parse origin/main)" ] \
+  && [ -z "$(git diff --name-only HEAD^ HEAD | grep -vxE 'CHANGELOG\.md|Configs/Version\.xcconfig|TablePro/Resources/WhatsNew\.md|\.github/release-flags\.json')" ] \
+  && echo "only the release commit"
 git tag -a v<version> -m "v<version>"
-git push origin main
+# The user runs this one:
+GIT_SSH_COMMAND='ssh -i ~/.ssh/tablepro_release -o IdentitiesOnly=yes' git push origin main
 git push origin v<version>
 ```
 
@@ -186,24 +208,26 @@ git -C ../tablepro-docs push origin main
 The docs CI fails while the app's `CHANGELOG.md` carries a version the page does not, so push it in
 the same sitting as the tag.
 
-- Keep the subject exactly `release: v<version>`: `scripts/ci/detect-changed-paths.sh` matches it to
-  skip the duplicate test run on `main`, since the tag's run is the one that gates.
+- Keep the subject exactly `release: v<version>`.
 - Always `-a -m`: the owner's git has `tag.gpgSign=true`, and a plain `git tag <name>` fails with
   "no tag message?".
-- Keep unrelated work out of the release commit. A lint fix made on the way goes first, as its own
-  Conventional Commit of at most 72 characters.
+- Keep the release commit to the four files above. No suite reads them, so its push to `main` skips
+  both suites and the tag takes the verdict of the code it ships. Any other file runs the suites on
+  that push, and the tag waits for them. A lint fix made on the way lands first, through its own
+  pull request.
 
 The tag runs `.github/workflows/build.yml`: arm64 and x86_64 builds, DMG and tar.xz, Sparkle signatures,
 deltas from the previous five releases, the GitHub Release with notes from `CHANGELOG.md`, then
-`appcast.xml`. The release is created as a draft holding every asset and published in one step, only
-when `lint`, `test`, `build` and `registry-readiness` pass. The job fails when the tag disagrees with
-`MARKETING_VERSION`.
+`appcast.xml`, pushed with `APPCAST_DEPLOY_KEY` from the `release` environment. The release is created
+as a draft holding every asset and published in one step, only when `Tested on main`, both builds and
+`Registry Readiness` pass. The job fails when the tag disagrees with `MARKETING_VERSION`.
 
 A published release is immutable: its assets and its tag are locked. When a run fails:
 
-- Before **Publish the release**, only a draft exists. Fix forward on `main` and move the tag:
-  `git tag -f -a v<version> -m "v<version>"`, then `git push origin v<version> --force`. The next run
-  reuses the draft.
+- Before **Publish the release**, only a draft exists. Fix forward on `main` through a pull request,
+  then move the tag to the commit it lands as: `git fetch origin main`,
+  `git tag -f -a v<version> -m "v<version>" origin/main`, then `git push origin v<version> --force`.
+  The next run reuses the draft.
 - After it, at the `appcast.xml` push for instance, use **Re-run failed jobs**. The release job finds
   the same files already published and carries on. **Re-run all jobs** rebuilds, and the release job
   refuses the new files.
@@ -216,11 +240,14 @@ minutes away. Removing the version from `appcast.xml` is the only way to un-ship
 
 ```bash
 python3 scripts/ci/pull-release.py <version> --dry-run
-python3 scripts/ci/pull-release.py <version>
+# The user runs this one, for the same reason as the release push:
+GIT_SSH_COMMAND='ssh -i ~/.ssh/tablepro_release -o IdentitiesOnly=yes' \
+    python3 scripts/ci/pull-release.py <version>
 ```
 
-It commits and pushes to `main`. Leave the GitHub Release so direct DMG links resolve. Users who
-already installed the build are reached by the corrective release with `criticalUpdate` set.
+It commits and pushes to `main` with the release key. Leave the GitHub Release so direct DMG links
+resolve. Users who already installed the build are reached by the corrective release with
+`criticalUpdate` set.
 
 ## Stage 3: Plugins
 

.github/actionlint.yaml

@@ -0,0 +1,4 @@
+self-hosted-runner:
+  # GitHub's Xcode 27 preview image, newer than the label list in actionlint 1.7.12.
+  labels:
+    - xcode-27

.github/actions/setup-xcode/action.yml

@@ -1,26 +1,12 @@
 name: Select Xcode
-description: Selects the Xcode version every TablePro build and test job uses.
-
-# The version lives here and nowhere else. It used to be written out in four workflows, eight
-# times counting the pinned action SHA beside it, so bumping Xcode meant finding every copy and
-# a missed one silently built against a different toolchain.
-
-inputs:
-  xcode-version:
-    description: Override the pinned version. Leave unset; it exists for trying a toolchain on one job.
-    required: false
-    default: "26.4.1"
+description: Selects the one Xcode every TablePro build and test job uses.
 
+# The only place the version is written. runs-on names the image (xcode-27); this picks the build
+# inside it, the one developers install locally. The image has no Xcode 26.
 runs:
   using: composite
   steps:
-    - uses: maxim-lobanov/setup-xcode@ed7a3b1fda3918c0306d1b724322adc0b8cc0a90 # v1.7.0
-      with:
-        xcode-version: ${{ inputs.xcode-version }}
-
-    # Proves the pin was actually applied before a twenty minute build starts. build.yml carried
-    # this as a separate step in both of its jobs.
     - shell: bash
       run: |
-        echo "Active Xcode: $(xcode-select -p)"
+        sudo xcode-select --switch /Applications/Xcode_27.1.app
         xcodebuild -version