diff --git a/.flake8 b/.flake8 index bc2ba4ee..f3767590 100644 --- a/.flake8 +++ b/.flake8 @@ -1,3 +1,19 @@ [flake8] max-line-length = 120 -max-complexity = 10 \ No newline at end of file +max-complexity = 10 + +# flake8's own default exclusions do not cover a virtualenv, and `uv sync` puts one inside +# python/. Without this, linting ./python walks site-packages and reports hundreds of findings in +# other people's code - including under the E9/F63/F7/F82 selection that CI treats as fatal. +extend-exclude = .venv,venv,build,dist + +# The shared package re-exports its public names lazily, through a module-level __getattr__, so +# the imports in its TYPE_CHECKING block exist only for editors and type checkers. flake8 cannot +# see the connection and reports every one of them as unused. +# +# Both spellings, because flake8 matches the path as it was walked: from the repository root - as +# CI does, to pick up this file - it is python/opentagviewer_export/..., and from python/, as a +# person iterating does, it is opentagviewer_export/... with no leading segment for */ to match. +per-file-ignores = + */opentagviewer_export/__init__.py:F401 + opentagviewer_export/__init__.py:F401 diff --git a/.github/workflows/macos-exporter-python.yml b/.github/workflows/macos-exporter-python.yml index cb76e345..da5887d6 100644 --- a/.github/workflows/macos-exporter-python.yml +++ b/.github/workflows/macos-exporter-python.yml @@ -1,7 +1,13 @@ -# This workflow will install Python dependencies, run tests and lint with a variety of Python versions -# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-python +# Builds and publishes the desktop exporter, for every platform it now runs on. +# +# It used to be macOS-only, because the data came out of macOS-local files. It comes out of iCloud +# now, so the binary runs anywhere - and the release has to actually produce one for anywhere, +# which is what the matrix below is for. PyInstaller cannot cross-compile: each platform's binary +# is built on that platform, and there is no way around that. +# +# The filename is still macos-exporter-python.yml so its history stays attached to it. -name: MacOS Exporter App +name: Exporter App on: release: @@ -10,33 +16,36 @@ on: env: PYTHONPATH: "${{ github.workspace }}/python:$PYTHONPATH" PYINSTALLER_PYTHON_VER: 3.13 - APP_NAME: OpenTagViewer-ExportWizardMacOS + APP_NAME: OpenTagViewer-Exporter jobs: test-release-version: runs-on: macos-14 - # Only run this for 'macos-exporter-v' tags! + # Only run this for exporter release tags. `macos-exporter-v` is the old spelling, still + # accepted by scripts/release_version.py so that existing tags and existing habits keep + # working - see CONTRIBUTING.md -> Releasing the exporter. if: | github.event_name == 'release' && (github.event.release.prerelease == true || github.event.release.published_at != null) - && startsWith(github.event.release.tag_name, 'macos-exporter-v') + && (startsWith(github.event.release.tag_name, 'exporter-v') + || startsWith(github.event.release.tag_name, 'macos-exporter-v')) - # The version both build jobs name their artifacts with. It comes from here rather than - # from each job parsing the tag again, so there is exactly one place it can be wrong. + # The version every build job names its artifact with. It comes from here rather than from + # each job parsing the tag again, so there is exactly one place it can be wrong. outputs: app-version: ${{ steps.version.outputs.APP_VERSION }} - steps: # (reuses steps from full flow!) + steps: - name: Checkout code uses: actions/checkout@v4 - # Nothing in this pipeline rewrites VERSION in python/main/wizard.py: the tag names the + # Nothing in this pipeline rewrites VERSION in python/exporter/version.py: the tag names the # artifact, and the app keeps whatever was committed. So a tag that disagrees with the # source ships a build that calls itself the old version in its title bar and in the # `via:` field of every export it writes. This step is what makes that a failed release - # instead of a confusing zip. It runs before either build job (they `need` this one), so - # a mismatch costs one minute rather than two full PyInstaller builds. + # instead of a confusing zip. It runs before every build job (they `need` this one), so + # a mismatch costs one minute rather than four full PyInstaller builds. # # Deliberately not the reverse - patching the tag into the source - because the wizard # also runs from source, and those exports stamp `via:` too. See scripts/release_version.py. @@ -49,53 +58,34 @@ jobs: # Do everything strictly for the python version used in the release build... - name: Set up Python ${{ env.PYINSTALLER_PYTHON_VER }} - uses: actions/setup-python@v3 + uses: actions/setup-python@v5 with: python-version: ${{ env.PYINSTALLER_PYTHON_VER }} + - name: Install uv + uses: astral-sh/setup-uv@v5 + + # --frozen: exactly what uv.lock records, or fail. FindMy.py is pinned there by commit, + # because it is a branch rather than a release - so a release build cannot silently pick up + # whatever that branch happened to be that day. - name: Install dependencies working-directory: ./python - run: | - python -m pip install --upgrade pip - python -m pip install flake8 pytest - pip install -r requirements.txt + run: uv sync --frozen - name: Lint with flake8 - working-directory: . # run flake8 from repository root + working-directory: . # run flake8 from repository root, so it picks up .flake8 run: | - # Run flake8 on the 'python' directory, letting it pick up the .flake8 config file # stop the build if there are Python syntax errors or undefined names - flake8 ./python --count --select=E9,F63,F7,F82 --show-source --statistics - flake8 ./python --count --exit-zero --statistics + uv run --project python flake8 ./python --count --select=E9,F63,F7,F82 --show-source --statistics + uv run --project python flake8 ./python --count --exit-zero --statistics + # Both suites: the wizard's own, and the shared export package's, which live with the package. - name: Test with pytest working-directory: ./python - run: | - pytest ./test - -# !!!!! -# -# YES, THESE ARE DUPLICATED. -# I COULD NOT FIND A BETTER SOLUTION FOR THIS. -# -# THERE IS SOME KIND OF SCHEDULING PROBLEM WHERE -# IF YOU PROVIDE THEM BOTH IN 'RUNS-ON', THE JOB -# JUST STRAIGHT UP WILL NEVER START (AT LEAST NOT -# FOR 20 MINS). -# -# SEPARATING THEM FIXES IT. -# -# - YAML ANCHORS ARE NOT SUPPORTED BY GITHUB. -# - WORKFLOWS MESS UP PERMISSIONS. -# -# SO YES WE WILL HAVE TO TAKE THE DUPLICATION. -# UNTIL WE GET BETTER OPTIONS. -# -# !!!!! + run: uv run --frozen pytest ./test ./opentagviewer_export - build-apple-silicon: + build: needs: test-release-version - runs-on: macos-14 # Apple Silicon environment: 'MacOS Exporter App CI' # The repository default is read-only, which is what turned a missing token into a 403 at @@ -104,153 +94,141 @@ jobs: permissions: contents: write - # Only run this for 'macos-exporter-v' tags! if: | github.event_name == 'release' && (github.event.release.prerelease == true || github.event.release.published_at != null) - && startsWith(github.event.release.tag_name, 'macos-exporter-v') + && (startsWith(github.event.release.tag_name, 'exporter-v') + || startsWith(github.event.release.tag_name, 'macos-exporter-v')) + + # This was two hand-duplicated jobs, with a comment saying that naming both runners in + # `runs-on` made the job never start. That is because a list in `runs-on` is a set of labels a + # single runner must have *all* of, not a choice between runners - so it asked for a machine + # that is both Apple Silicon and Intel, and no such machine exists. A matrix is the construct + # that means "one job per entry", and it takes the duplication with it. + # + # fail-fast is off deliberately: one platform failing should not cancel the others and leave a + # release with some of its binaries attached, which is how 1.0.5 published with only ARM64. + strategy: + fail-fast: false + matrix: + include: + - runner: macos-14 # Apple Silicon + os: macos + arch: arm64 + - runner: macos-15-intel # macos-13 no longer exists; this is the current x64 label + os: macos + arch: x64 + - runner: windows-latest + os: windows + arch: x64 + # Deliberately not ubuntu-latest: PyInstaller links against the builder's glibc, so the + # oldest image available is the widest audience. A binary built on a newer one refuses + # to start on an older distribution with a message about GLIBC versions. + - runner: ubuntu-22.04 + os: linux + arch: x64 + + runs-on: ${{ matrix.runner }} - # The version is not parsed out of the tag here. It is whatever test-release-version - # confirmed the tag and python/main/wizard.py agree on, so the zip name, the release - # name, and the version the app reports about itself cannot come apart. env: + # Not parsed out of the tag here. It is whatever test-release-version confirmed the tag and + # python/exporter/version.py agree on, so the zip name, the release name, and the version the + # app reports about itself cannot come apart. APP_VERSION: ${{ needs.test-release-version.outputs.app-version }} - # The wizard only works on macOS 14 and earlier - 15 tightened keychain access so the - # BeaconStore key cannot be read - so every machine this binary has to run on is older - # than the machine building it. Setting the target explicitly stops the toolchain - # stamping the runner's own OS version onto the result and locking out every user. + # The macOS-local export route only works on macOS 14 and earlier - 15 tightened keychain + # access so the BeaconStore key cannot be read - so every Mac this binary has to run on is + # older than the machine building it. Setting the target explicitly stops the toolchain + # stamping the runner's own OS version onto the result and locking out every user. Ignored + # on the other platforms. MACOSX_DEPLOYMENT_TARGET: "11.0" steps: - name: Checkout code uses: actions/checkout@v4 - - name: Set up Python ${{ env.PYINSTALLER_PYTHON_VER }} (${{ runner.arch }}) - uses: actions/setup-python@v3 + - name: Set up Python ${{ env.PYINSTALLER_PYTHON_VER }} (${{ matrix.arch }}) + uses: actions/setup-python@v5 with: python-version: ${{ env.PYINSTALLER_PYTHON_VER }} + - name: Install uv + uses: astral-sh/setup-uv@v5 + + # tkinter is not part of a stock Python on macOS or Linux, and the wizard is a tkinter app - + # without this the build succeeds and the binary dies at import on a user's machine. Windows + # ships it with the interpreter. + - name: Install tkinter (macOS) + if: matrix.os == 'macos' + run: brew install python-tk + + - name: Install tkinter (Linux) + if: matrix.os == 'linux' + run: sudo apt-get update && sudo apt-get install -y python3-tk + + # --no-default-groups drops the dev group: pytest and the linters have no business in a + # bundle handed to a user. `build` carries PyInstaller, pinned like everything else. - name: Install dependencies working-directory: ./python - run: | - brew install python-tk - python -m pip install --upgrade pip - pip install pyinstaller -r requirements.txt - - - name: Run PyInstaller for MacOS (${{ runner.arch }}) + run: uv sync --frozen --no-default-groups --group build + + # --collect-all unicorn is not optional, and its absence is invisible until somebody runs the + # result. unicorn loads its native library through ctypes at runtime, so PyInstaller's analysis + # never sees it, and the build succeeds - producing a binary that dies at import with "Failed + # to load the Unicorn dynamic library". unicorn is there because local Anisette emulates + # Apple's ADI library, which is what lets sign-in work without a third-party server. + - name: Run PyInstaller (macOS, ${{ matrix.arch }}) + if: matrix.os == 'macos' working-directory: ./python run: | - pyinstaller \ - --onefile \ + uv run --frozen pyinstaller \ --windowed \ --name "OpenTagViewer" \ --osx-bundle-identifier "dev.wander.opentagviewer" \ --icon=OpenTagViewer.icns \ + --collect-all unicorn \ + --collect-data findmy \ + --collect-data anisette \ --noconfirm \ - ./main/wizard.py - - # Verifies rather than assumes. A dependency built on the runner's own macOS - Homebrew - # bottles especially, which are not backward compatible - can raise the whole bundle's - # floor to the builder's OS version. That produces a binary which installs fine and then - # refuses to launch, on a user's Mac, with no traceback and nothing in any build log. - # Runs before the upload so a build nobody could use never reaches the release page. - - name: Check the build can run on the macOS the wizard supports - run: python3 scripts/check_macos_min_version.py python/dist - - - name: Prepare and Upload MacOS App - working-directory: ./python - run: | - echo "Going to build zip with version: $APP_VERSION" - APP_ZIP_NAME="${{ env.APP_NAME }}-$APP_VERSION-${{ runner.arch }}.zip" - cd ./dist - zip -r $APP_ZIP_NAME OpenTagViewer.app/ OpenTagViewer - mkdir -p ${{ github.workspace }}/output - mv ${{ github.workspace }}/python/dist/$APP_ZIP_NAME ${{ github.workspace }}/output/$APP_ZIP_NAME - - echo "UPLOAD_ASSET=$APP_ZIP_NAME" >> $GITHUB_ENV - echo "UPLOAD_PATH=${{ github.workspace }}/output/$APP_ZIP_NAME" >> $GITHUB_ENV - - - name: Upload PyInstaller Artifact - uses: actions/upload-artifact@v4 - with: - name: ${{ env.APP_NAME }}-${{ env.APP_VERSION }}-${{ runner.arch }}.zip - path: ${{ github.workspace }}/output/${{ env.APP_NAME }}-${{ env.APP_VERSION }}-${{ runner.arch }}.zip - - # The token goes in `with:`, not `env:`. This action's `token` input defaults to - # ${{ github.token }}, and its own docs say "a non-empty explicit token overrides - # GITHUB_TOKEN" - so the default is always non-empty and the env var was ignored - # entirely. It authenticated as the built-in token, whose contents permission is read-only - # here, and failed with 403 "Resource not accessible by integration" after both binaries - # had already been built. The v1.0.5 release published with no assets attached. - - name: Upload Release Asset to GitHub Release - uses: softprops/action-gh-release@v2 - if: github.event_name == 'release' - with: - files: ${{ env.UPLOAD_PATH }} # Path to asset created in previous step - name: OpenTagViewer MacOS AirTag Exporter ${{ env.APP_VERSION }} - token: ${{ secrets.MACOS_EXPORTER_APP_GITHUB_TOKEN }} - - build-intel: - needs: test-release-version - # macos-13 no longer exists. GitHub's supported images are macOS 26, 15 and 14 (14 itself - # deprecated), so a job asking for macos-13 is never picked up - it sits queued until the - # six-hour timeout rather than failing, which is why the 1.0.5 release went out with only - # the ARM64 binary attached. macos-15-intel is the current standard-size x64 label. - runs-on: macos-15-intel # Intel - environment: 'MacOS Exporter App CI' - - # The repository default is read-only, which is what turned a missing token into a 403 at - # the very last step. Granting it here means the upload still works if the PAT is ever - # unset or expires - the action treats an empty token as unset and falls back to this one. - permissions: - contents: write - - # Only run this for 'macos-exporter-v' tags! - if: | - github.event_name == 'release' - && (github.event.release.prerelease == true || github.event.release.published_at != null) - && startsWith(github.event.release.tag_name, 'macos-exporter-v') - - # The version is not parsed out of the tag here. It is whatever test-release-version - # confirmed the tag and python/main/wizard.py agree on, so the zip name, the release - # name, and the version the app reports about itself cannot come apart. - env: - APP_VERSION: ${{ needs.test-release-version.outputs.app-version }} - - # The wizard only works on macOS 14 and earlier - 15 tightened keychain access so the - # BeaconStore key cannot be read - so every machine this binary has to run on is older - # than the machine building it. Setting the target explicitly stops the toolchain - # stamping the runner's own OS version onto the result and locking out every user. - MACOSX_DEPLOYMENT_TARGET: "11.0" - - steps: - - name: Checkout code - uses: actions/checkout@v4 - - - name: Set up Python ${{ env.PYINSTALLER_PYTHON_VER }} (${{ runner.arch }}) - uses: actions/setup-python@v3 - with: - python-version: ${{ env.PYINSTALLER_PYTHON_VER }} - - - name: Install dependencies + ./exporter/wizard.py + + # --collect-all unicorn is not optional, and its absence is invisible until somebody runs the + # result. unicorn loads its native library through ctypes at runtime, so PyInstaller's analysis + # never sees it, and the build succeeds - producing a binary that dies at import with "Failed + # to load the Unicorn dynamic library". unicorn is there because local Anisette emulates + # Apple's ADI library, which is what lets sign-in work without a third-party server. + - name: Run PyInstaller (Windows) + if: matrix.os == 'windows' working-directory: ./python run: | - brew install python-tk - python -m pip install --upgrade pip - pip install pyinstaller -r requirements.txt - - - name: Run PyInstaller for MacOS (${{ runner.arch }}) + uv run --frozen pyinstaller ` + --windowed ` + --name "OpenTagViewer" ` + --icon=OpenTagViewer.ico ` + --collect-all unicorn ` + --collect-data findmy ` + --collect-data anisette ` + --noconfirm ` + ./exporter/wizard.py + + # No --windowed on Linux: it means "no console window" on the platforms that have that + # concept, and there is no bundle format here for it to produce either way. + # --collect-all unicorn is not optional, and its absence is invisible until somebody runs the + # result. unicorn loads its native library through ctypes at runtime, so PyInstaller's analysis + # never sees it, and the build succeeds - producing a binary that dies at import with "Failed + # to load the Unicorn dynamic library". unicorn is there because local Anisette emulates + # Apple's ADI library, which is what lets sign-in work without a third-party server. + - name: Run PyInstaller (Linux) + if: matrix.os == 'linux' working-directory: ./python run: | - pyinstaller \ - --onefile \ - --windowed \ + uv run --frozen pyinstaller \ --name "OpenTagViewer" \ - --osx-bundle-identifier "dev.wander.opentagviewer" \ - --icon=OpenTagViewer.icns \ + --collect-all unicorn \ + --collect-data findmy \ + --collect-data anisette \ --noconfirm \ - ./main/wizard.py + ./exporter/wizard.py # Verifies rather than assumes. A dependency built on the runner's own macOS - Homebrew # bottles especially, which are not backward compatible - can raise the whole bundle's @@ -258,26 +236,77 @@ jobs: # refuses to launch, on a user's Mac, with no traceback and nothing in any build log. # Runs before the upload so a build nobody could use never reaches the release page. - name: Check the build can run on the macOS the wizard supports + if: matrix.os == 'macos' run: python3 scripts/check_macos_min_version.py python/dist - - name: Prepare and Upload MacOS App - working-directory: ./python + # Start the binary and have it print its version. Not a substitute for running the thing, but + # it catches a bundle missing an import - which is the failure PyInstaller actually produces, + # and it produces it at startup. + # + # **This ran everywhere except macOS, which is exactly where it was needed.** The first build + # of the iCloud exporter was missing unicorn's native library, built cleanly, and died at + # import; nothing in this workflow would have noticed. It costs a second, so it runs on all of + # them now. + # + # `--self-test` is the other half, and starting was not enough of a check on its own: the + # second build started, signed in, and *then* died, because FindMy.py keeps Apple's pinned + # root certificates as files beside its code and PyInstaller bundles code. That failure was + # several minutes and one Apple ID into the flow. The self-test reaches the same files with + # no network and no account. + - name: Check the built binary starts and can reach its data files + working-directory: ./python/dist + shell: bash run: | - echo "Going to build zip with version: $APP_VERSION" - APP_ZIP_NAME="${{ env.APP_NAME }}-$APP_VERSION-${{ runner.arch }}.zip" - cd ./dist - zip -r $APP_ZIP_NAME OpenTagViewer.app/ OpenTagViewer - mkdir -p ${{ github.workspace }}/output - mv ${{ github.workspace }}/python/dist/$APP_ZIP_NAME ${{ github.workspace }}/output/$APP_ZIP_NAME - - echo "UPLOAD_ASSET=$APP_ZIP_NAME" >> $GITHUB_ENV - echo "UPLOAD_PATH=${{ github.workspace }}/output/$APP_ZIP_NAME" >> $GITHUB_ENV + BINARY="./OpenTagViewer/OpenTagViewer" + [ "${{ matrix.os }}" = "macos" ] && BINARY="./OpenTagViewer.app/Contents/MacOS/OpenTagViewer" + "$BINARY" --version + "$BINARY" --self-test + + - name: Package (macOS) + if: matrix.os == 'macos' + working-directory: ./python/dist + shell: bash + run: | + ASSET="${{ env.APP_NAME }}-${{ env.APP_VERSION }}-macos-${{ matrix.arch }}.zip" + # The .app only. In onedir mode COLLECT and BUNDLE each hold a full copy, so zipping both + # ships every file twice. + zip -qr "$ASSET" OpenTagViewer.app + mkdir -p "${{ github.workspace }}/output" + mv "$ASSET" "${{ github.workspace }}/output/$ASSET" + echo "UPLOAD_ASSET=$ASSET" >> $GITHUB_ENV + echo "UPLOAD_PATH=${{ github.workspace }}/output/$ASSET" >> $GITHUB_ENV + + - name: Package (Windows) + if: matrix.os == 'windows' + working-directory: ./python/dist + shell: bash + run: | + ASSET="${{ env.APP_NAME }}-${{ env.APP_VERSION }}-windows-${{ matrix.arch }}.zip" + # The whole directory: onedir keeps the interpreter and the libraries beside the .exe, and + # the .exe alone does nothing. + 7z a "$ASSET" OpenTagViewer + mkdir -p "${{ github.workspace }}/output" + mv "$ASSET" "${{ github.workspace }}/output/$ASSET" + echo "UPLOAD_ASSET=$ASSET" >> $GITHUB_ENV + echo "UPLOAD_PATH=${{ github.workspace }}/output/$ASSET" >> $GITHUB_ENV + + - name: Package (Linux) + if: matrix.os == 'linux' + working-directory: ./python/dist + shell: bash + run: | + ASSET="${{ env.APP_NAME }}-${{ env.APP_VERSION }}-linux-${{ matrix.arch }}.tar.gz" + tar -czf "$ASSET" OpenTagViewer + mkdir -p "${{ github.workspace }}/output" + mv "$ASSET" "${{ github.workspace }}/output/$ASSET" + echo "UPLOAD_ASSET=$ASSET" >> $GITHUB_ENV + echo "UPLOAD_PATH=${{ github.workspace }}/output/$ASSET" >> $GITHUB_ENV - name: Upload PyInstaller Artifact uses: actions/upload-artifact@v4 with: - name: ${{ env.APP_NAME }}-${{ env.APP_VERSION }}-${{ runner.arch }}.zip - path: ${{ github.workspace }}/output/${{ env.APP_NAME }}-${{ env.APP_VERSION }}-${{ runner.arch }}.zip + name: ${{ env.UPLOAD_ASSET }} + path: ${{ env.UPLOAD_PATH }} # The token goes in `with:`, not `env:`. This action's `token` input defaults to # ${{ github.token }}, and its own docs say "a non-empty explicit token overrides @@ -289,6 +318,6 @@ jobs: uses: softprops/action-gh-release@v2 if: github.event_name == 'release' with: - files: ${{ env.UPLOAD_PATH }} # Path to asset created in previous step - name: OpenTagViewer MacOS AirTag Exporter ${{ env.APP_VERSION }} - token: ${{ secrets.MACOS_EXPORTER_APP_GITHUB_TOKEN }} \ No newline at end of file + files: ${{ env.UPLOAD_PATH }} + name: OpenTagViewer AirTag Exporter ${{ env.APP_VERSION }} + token: ${{ secrets.MACOS_EXPORTER_APP_GITHUB_TOKEN }} diff --git a/.github/workflows/macos-scripts-python.yml b/.github/workflows/macos-scripts-python.yml index 6a2e028c..d33ee2ab 100644 --- a/.github/workflows/macos-scripts-python.yml +++ b/.github/workflows/macos-scripts-python.yml @@ -1,4 +1,4 @@ -# This workflow will install Python dependencies, run tests and lint with a variety of Python versions +# Lints and tests everything under python/: the desktop exporter and the shared export package. # For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-python name: MacOS Scripts @@ -24,45 +24,49 @@ jobs: strategy: fail-fast: true matrix: - # 3.9 is what the Xcode Command Line Tools install, so it is the interpreter anyone - # running the wizard from source on a stock Mac actually gets. Dropping it means - # 3.10-only syntax can land and the wizard dies at import on a fresh machine, which - # is exactly how it broke once. - python-version: ["3.9", "3.10", "3.11", "3.12", "3.13"] + # 3.9 used to be here, because it is what the Xcode Command Line Tools install and so what + # anyone running the wizard from source on a stock Mac gets. It is gone because FindMy.py + # requires 3.10, and the iCloud export route is the whole point of this exporter now. + # + # The consequence is real and belongs in the wiki rather than only here: running from + # source - the VM bootstrap - now needs a Python installed first. The built .app is + # unaffected; it bundles its own interpreter. + python-version: ["3.10", "3.11", "3.12", "3.13"] steps: - name: Checkout code uses: actions/checkout@v4 - name: Set up Python ${{ matrix.python-version }} - uses: actions/setup-python@v3 + uses: actions/setup-python@v5 with: python-version: ${{ matrix.python-version }} + - name: Install uv + uses: astral-sh/setup-uv@v5 + + # --frozen: install exactly what uv.lock records and fail if it disagrees with pyproject.toml, + # rather than quietly resolving something else. That is what makes the lock mean anything - + # FindMy.py is pinned there by commit, since it is a branch rather than a release. - name: Install dependencies working-directory: ./python - run: | - python -m pip install --upgrade pip - python -m pip install flake8 pytest - pip install -r requirements.txt + run: uv sync --frozen --python ${{ matrix.python-version }} - name: Lint with flake8 - working-directory: . # run flake8 from repository root + working-directory: . # run flake8 from repository root, so it picks up .flake8 run: | - # Run flake8 on the 'python' directory, letting it pick up the .flake8 config file # stop the build if there are Python syntax errors or undefined names - flake8 ./python --count --select=E9,F63,F7,F82 --show-source --statistics - flake8 ./python --count --exit-zero --statistics + uv run --project python flake8 ./python --count --select=E9,F63,F7,F82 --show-source --statistics + uv run --project python flake8 ./python --count --exit-zero --statistics + # Both suites: the wizard's own, and the shared export package's, which live with the package. - name: Test with pytest working-directory: ./python - run: | - pytest ./test + run: uv run --frozen pytest ./test ./opentagviewer_export - # The release check reads VERSION out of wizard.py by parsing the module, so a refactor - # that makes it unreadable - moving it into a class, computing it from parts - would - # otherwise only surface when a release failed. This is that same read, run whenever - # python/ changes. + # The release check reads VERSION out of main/version.py by parsing the module, so a refactor + # that makes it unreadable - moving it into a class, computing it from parts - would otherwise + # only surface when a release failed. This is that same read, run whenever python/ changes. - name: Check the exporter version is still readable working-directory: . run: | diff --git a/.gitignore b/.gitignore index fa9d68c0..45fdaa9c 100644 --- a/.gitignore +++ b/.gitignore @@ -35,3 +35,9 @@ tmp/ # in - none of which belongs to anybody else working on this. /.idea/androidTestResultsUserPreferences.xml /.idea/deploymentTargetSelector.xml + +# Fixture-investigation notebook. Its whole job is decoding real exports, so its saved cell +# outputs contain the very things the fixtures under app/src/test/resources/ exist to have +# stripped: device names ("'s iPad"), record UUIDs, key material. Committing it would +# publish, in a notebook, exactly what was redacted out of the plists next to it. +scripts/plist_explorer.ipynb diff --git a/AGENTS.md b/AGENTS.md index 013bd97e..80ae4827 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -108,23 +108,31 @@ rather than copying file contents into your own commit. Credit them in the PR de ### 9. Bump the exporter's `VERSION` before tagging a release -`VERSION` in `python/main/wizard.py` is the only place the macOS exporter's version is +`VERSION` in `python/exporter/version.py` is the only place the desktop exporter's version is written. It reaches the window title and, more importantly, every export it produces, as -`via: OpenTagViewer.app:` in `OPENTAGVIEWER.yml` — which is how anyone looking at a -zip later works out which exporter built it. +`via: :` in `OPENTAGVIEWER.yml` — which is how anyone looking at a zip later +works out what built it. + +**There is more than one producer, and they must not claim to be each other.** The windowed +exporter stamps `OpenTagViewer.wizard:`, its CLI stamps `OpenTagViewer.cli:`, and +the Android app will stamp its own. They share `VERSION` because they ship together; the name in +front of it is what makes a bug report answerable. The shared writer takes `via` as a parameter +and never invents one — see `python/opentagviewer_export/bundle.py`. Nothing patches it at build time, and nothing should: the wizard also runs from source (the -VM bootstrap, `python main/wizard.py`), and those exports stamp `via:` too, so a build-time +VM bootstrap, `python -m exporter.wizard`), and those exports stamp `via:` too, so a build-time patch would make two artifacts from one commit disagree. So releasing is two steps, in this order: 1. Commit the `VERSION` bump to `main` -2. Tag that commit `macos-exporter-v` and publish the release +2. Tag that commit `exporter-v` and publish the release `scripts/release_version.py --kind exporter --tag ` enforces it, and runs in `test-release-version` -before either build job. A tag that disagrees fails the release rather than shipping a build -that lies about itself. Full procedure: [CONTRIBUTING.md](./CONTRIBUTING.md#releasing-the-macos-exporter). +before any build job. A tag that disagrees fails the release rather than shipping a build that +lies about itself. `macos-exporter-v` is the old spelling and still resolves, because tags already +published keep their name — but it stopped being true when the exporter started building for +Windows and Linux. Full procedure: [CONTRIBUTING.md](./CONTRIBUTING.md#releasing-the-exporter). ### 10. Update the docs that index what you added @@ -144,6 +152,40 @@ The test for whether it belongs here rather than in a comment: would somebody hi reading the code that explains it? Anisette's machine-identity binding is the example — it presents as auth failing for no reason, hours away from the code responsible. +### 11. The app is one device, and every path must say the same thing + +Everything this app does against Apple is attributed to a device identity it invents, and the +user sees the result: an entry in their Apple account's device list, with a *Remove from Account* +button next to the words "If you do not recognise this device". + +**So the identity has to be one identity.** It is assembled in more than one place — the client +info string in `anisette/AdiDeviceIdentity.java`, the hardware headers, the serial the FindMy.py +providers take as a parameter — and paths that disagree do not fail. They register **separate +devices**, and the user is invited to remove things they cannot identify, which breaks the +session that was working. + +Under the iCloud export route the same identity reaches two further places a person reads: the +escrow record's metadata, which is the only way to recognise a record in a listing, and the +peer's `serialNumber` in the keychain trust circle. A serial that differs between them makes one +client look like several — and by +[findmy-export §5.2](./docs/findmy-export/03-keychain-trust.md), serials are exactly what +distinguishes devices there. + +Three things follow: + +- **One source of truth.** A path that needs the identity reads it; it does not compose its own. +- **The parts must agree with each other.** Model, OS version, build, CFNetwork and Darwin + describe one real release ([findmy-export §2.2](./docs/findmy-export/01-authentication.md)). + Claiming a Mac in one string and an iPhone in another is a contradiction Apple's own clients + never produce. +- **The serial is a label, and the only field here the user actually sees.** `0PENTAGVIEWR` is + confirmed accepted and displayed. Without one, Apple omits the row entirely, leaving an entry + with nothing to tell it apart from real hardware. + +**Changing it later adds an entry rather than renaming one**, and may require signing in again, +so it is not a thing to adjust casually once shipped. Document what the app registers as, so a +user reading their device list can recognise it — see the wiki. + --- ## Building and testing diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 00000000..2d585447 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,9 @@ +# CLAUDE.md + +The rules for this repo live in [AGENTS.md](./AGENTS.md), so that every agent reads the same +file rather than two copies that drift apart. This one exists only to pull it in — Claude Code +loads `CLAUDE.md` automatically, and the line below imports the real thing. + +**Put new rules in `AGENTS.md`, not here.** + +@AGENTS.md diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index f6aea092..f72fd90f 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -130,7 +130,8 @@ wizard). | Anisette tests | `app/src/androidTest/java/.../anisette/` | as above, **opt-in** | yes, and network | | UI tests | `AppleLoginFlowTest`, `app/src/androidTest/java/.../ui/` | Espresso, on the managed device | provisioned for you | | Chaquopy bridge tests | `app/src/test/python/` | pytest | no | -| Desktop wizard tests | `python/test/` | pytest | no | +| Desktop exporter tests | `python/test/` | pytest | no | +| Shared export package | `python/opentagviewer_export/tests/` | pytest | no | | Tooling tests | `scripts/test/` | pytest | no | ### Run everything @@ -295,18 +296,45 @@ enforces that. Fixtures live in `app/src/test/resources/` — a redacted copy of a real export. See the README there for what was redacted and why. -### Desktop wizard tests +### Desktop exporter tests -For the macOS export wizard under `python/`. +For the exporter under `python/` — the CLI, the windowed wizard, and the shared export package +they both write bundles with. Dependencies are managed with [uv](https://docs.astral.sh/uv/), +which installs a suitable Python itself: ```bash cd python -python -m pip install -r requirements.txt flake8 pytest +uv sync -python -m pytest ./test -flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics +uv run pytest ./test ./opentagviewer_export +uv run flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics ``` +**Both paths, always.** `python/test/` covers the exporter — the iCloud mapping, the key-file +readers, the CLI's own decisions — and `python/opentagviewer_export/tests/` covers the shared +package that writes the bundle. The second is the one the Android app also depends on, and its +strongest test rebuilds the committed export fixtures in `app/src/test/resources/` +**byte for byte**, so a change in how a record is serialised fails here rather than on a phone. + +Two things it deliberately does not cover, because they need an Apple account: signing in, and +everything the account then yields. What happens to the records afterwards is covered in full. + +**Two files need Tk and skip without it** — `test_asyncui.py`, for what happens when somebody +closes the progress window, and `test_wizard_list.py`, for the list of accessories. Both drive +real Tk widgets, because what they cover *is* the event loop and the widget: a cancelled progress +window used to leave the wizard hung for ever, and that is not reproducible against a stand-in. + +**CI runs them on one job in four.** The matrix is 3.10 to 3.13 on `macos-14`, and only its 3.13 +has `_tkinter` — the other three skip both files at collection, so a green square there says +nothing about the window. `-rs` is what tells you which you got: + +```bash +uv run pytest ./test -rs +``` + +`361 passed, 4 skipped` means Tk was missing and neither file ran. `378 passed, 2 skipped` means +they did, and the two remaining skips are the macOS-14-only decryptor tests. + ### Tooling tests ```bash @@ -416,13 +444,13 @@ source .venv/bin/activate pip install -r requirements.txt # 4. Run it. -PYTHONPATH=. python3 main/wizard.py +PYTHONPATH=. python3 exporter/wizard.py ``` -`PYTHONPATH=.` is not optional: `wizard.py` does `from main.airtag_decryptor import ...`, and -running the file directly puts `python/main` on `sys.path` rather than `python/`, so the +`PYTHONPATH=.` is not optional: `wizard.py` does `from exporter.airtag_decryptor import ...`, and +running the file directly puts `python/exporter` on `sys.path` rather than `python/`, so the import fails. It is the same thing the CI workflow sets. Then check the export actually contains what it should: @@ -517,8 +545,8 @@ python scripts/make_test_beacon_plist.py out.plist --days-old 730 | --- | --- | --- | | `build-debug.yml` | push/PR to `main` | Translation check, pyright, string tooling tests, instrumented tests on an emulator, JVM tests, Chaquopy bridge tests, debug APK | | `build-release.yml` | on release | Translation check, JVM tests, Chaquopy bridge tests, release APK | -| `macos-scripts-python.yml` | `python/**` changes | Wizard tests across Python 3.10–3.13 on macOS 14 | -| `macos-exporter-python.yml` | on release | Tag/version check, wizard tests, the PyInstaller build for both architectures | +| `macos-scripts-python.yml` | `python/**` changes | Exporter and shared-package tests across Python 3.10–3.13 on macOS 14 | +| `macos-exporter-python.yml` | on release | Tag/version check, exporter tests, and the PyInstaller build for macOS (both architectures), Windows and Linux | | `update-contributors.yml` | weekly | Regenerates the contributor list on the Information page, opens a PR if it changed | | `check-adi-libraries.yml` | weekly | Checks Apple's ADI libraries still match what is checked in, opens an issue if they drifted | @@ -528,9 +556,9 @@ emulator definition in CI to keep in step with `app/build.gradle.kts`. --- -## Releasing the macOS exporter +## Releasing the exporter -The exporter's version lives in exactly one place — `VERSION` in `python/main/wizard.py`: +The exporter's version lives in exactly one place — `VERSION` in `python/exporter/version.py`: ```python VERSION = "1.0.5" @@ -543,7 +571,7 @@ be true. **Nothing rewrites it at build time.** The release tag names the zip and the GitHub release; the app keeps whatever was committed. That is deliberate rather than an oversight — the wizard -also runs straight from source (the VM bootstrap below, `python main/wizard.py`), and those +also runs straight from source (the VM bootstrap below, `python -m exporter.wizard`), and those runs stamp `via:` as well. If CI patched the tag into the source, a binary and a from-source export built from the same commit would claim different versions, which is the same drift in a place nobody would think to look. @@ -552,8 +580,8 @@ So a release is two steps, in this order: ```bash # 1. Bump it, commit it, push it -# (edit python/main/wizard.py -> VERSION = "1.0.6") -git commit -am "Bump the macOS exporter to 1.0.6" +# (edit python/exporter/version.py -> VERSION = "1.1.1") +git commit -am "Bump the exporter to 1.1.1" git push origin main # 2. Write the changes for this version, then create the release as a draft @@ -579,23 +607,28 @@ same for the app releases. The draft is deliberate: the release workflow triggers on `published`, so nothing builds or ships until someone clicks the button. -The tag must be `macos-exporter-v` followed by exactly what `VERSION` says. Before either -build job starts, `test-release-version` runs: +The tag must be `exporter-v` followed by exactly what `VERSION` says. Before any build job +starts, `test-release-version` runs: ```bash -python scripts/release_version.py --kind exporter --tag macos-exporter-v1.0.6 +python scripts/release_version.py --kind exporter --tag exporter-v1.1.0 ``` which fails the release, with instructions, if the two disagree — so the mistake costs a -minute rather than an incorrectly labelled build on the releases page. Both build jobs then -take the version from that job's output instead of parsing the tag themselves, so the zip +minute rather than an incorrectly labelled build on the releases page. Every build job then +takes the version from that job's output instead of parsing the tag themselves, so the zip name, the release title, and the version the app reports cannot come apart. +> **The tag used to be `macos-exporter-v`, and still resolves.** The exporter reads accessories +> out of iCloud now and builds for Windows and Linux as well, so the name stopped being true — +> but tags already published keep theirs forever, and nobody's habits update with a rename. Use +> `exporter-v` for anything new. + You can run the same check locally before tagging: ```bash -python scripts/release_version.py --kind exporter --print # what the source declares -python scripts/release_version.py --kind exporter --tag macos-exporter-v1.0.6 # would this tag be accepted? +python scripts/release_version.py --kind exporter --print # what the source declares +python scripts/release_version.py --kind exporter --tag exporter-v1.1.0 # would this tag be accepted? ``` If you tagged before bumping, the fix is to push the bump, delete the release and its tag, and diff --git a/README.md b/README.md index f2495f29..b5b36103 100644 --- a/README.md +++ b/README.md @@ -43,7 +43,7 @@ This project is a relatively polished looking Android/Java UI-wrapper around the 1. An Android phone with [the `OpenTagViewer` app installed](https://github.com/parawanderer/OpenTagViewer/wiki/How-To:-Install-App) 2. A (free) [Apple Account](https://account.apple.com/) with 2FA enabled to be via either `SMS` or `Trusted Device` 3. One or more **AirTags**, which need to be already registered to some Apple account via the `FindMy` app -4. A Mac or a [MacOS Virtual Machine](https://github.com/parawanderer/OpenTagViewer/wiki/How-To:-Export-AirTags-From-Mac#prerequisites) (only needed once/initially) +4. Any computer to run the exporter on — Windows, Linux or a Mac (only needed once/initially). See [the export guide](https://github.com/parawanderer/OpenTagViewer/wiki/How-To:-Export-AirTags#prerequisites) ### How to view my AirTag on my Android Phone?! @@ -51,7 +51,7 @@ This project is a relatively polished looking Android/Java UI-wrapper around the See [📖 wiki](https://github.com/parawanderer/OpenTagViewer/wiki) for more details: 1. [Install the app](https://github.com/parawanderer/OpenTagViewer/wiki/How-To:-Install-App) and log in to your Apple Account -2. Create an export `.zip` file by following [this wiki guide](https://github.com/parawanderer/OpenTagViewer/wiki/How-To:-Export-AirTags-From-Mac#opentagviewer-macos-export-app--recommended) +2. Create an export `.zip` file by following [this wiki guide](https://github.com/parawanderer/OpenTagViewer/wiki/How-To:-Export-AirTags#the-export-wizard--recommended) 3. Import the `.zip` file in the app 4. Profit: you can now track your AirTags on your Android Phone indefinitely! diff --git a/app/build.gradle.kts b/app/build.gradle.kts index 2e983e2c..1e92756d 100644 --- a/app/build.gradle.kts +++ b/app/build.gradle.kts @@ -261,7 +261,24 @@ chaquopy { pip { // SEE: https://chaquo.com/chaquopy/doc/current/android.html#android-requirements install(unicornStubWheel.get().asFile.absolutePath) - install("FindMy==0.9.8") + + // TEMPORARY: a fork, until the iCloud keychain export work is merged upstream + // and released to PyPI. Then this goes back to a plain `install("FindMy==")`. + // + // A branch, deliberately, while that branch is under active development - a + // commit pin would need moving on every change to it. The cost is that the build + // is not reproducible: two builds of the same commit of *this* repo can ship + // different Python. Acceptable now, not acceptable to release. + // + // **Pin to a commit before any build that leaves this machine**, by appending + // `@` in place of the branch name. + // + // Note for whoever bumps it: 0.10.x adds protobuf, which publishes a native + // wheel for desktop platforms and a pure-Python `py3-none-any` one as well. + // There is no Android wheel, so pip falls back to the pure-Python build - which + // is correct but markedly slower. The messages here are small enough not to care. + install("git+https://github.com/parawanderer/FindMy.py@feat/icloud-keychain-export") + install("NSKeyedUnArchiver==1.5") } } diff --git a/app/src/main/python/main.py b/app/src/main/python/main.py index 184a40fd..5d330baa 100644 --- a/app/src/main/python/main.py +++ b/app/src/main/python/main.py @@ -872,3 +872,45 @@ def getReports( err = traceback.format_exc() print(f"Failed to fetch all reports due to error: {err}") return None + + +def identifyHardware(plistXml: str) -> str | None: + """ + Say what an accessory is, from the `OwnedBeacons` plist the app already holds. + + **The heuristic lives in `opentagviewer_export.hardware`, not here and not in Java.** It is + guesswork over half a dozen fields - product and vendor ids, the model, the shape of + `stableIdentifier` - and the same guesswork runs in the desktop exporter when it asks which + accessories to export. Two copies of it would drift, and the symptom of drift is one tag + described as an AirTag in one place and as a hex number in the other. + + Returns None when nothing recognises the record, which is a real answer: the caller should + then show what it already has rather than a guess. + + :param plistXml: The accessory's plist, as the app stores it. + """ + try: + from opentagviewer_export.hardware import identify + + return identify(util_files.read_data_plist(plistXml.encode("utf-8"))) + except Exception: + # Never fatal: this is a label. An import failing here should cost a nicer row in a list, + # not the accessory. + print(f"identifyHardware failed, carrying on without it: {traceback.format_exc()}") + return None + + +def whereToLookUpHardware(plistXml: str) -> str | None: + """ + How the user could find out what an unrecognised accessory is, in one line. + + Offered rather than guessed at - see `opentagviewer_export.hardware.where_to_look_up`. Returns + None when there is nothing worth saying, which is the common case. + """ + try: + from opentagviewer_export.hardware import where_to_look_up + + return where_to_look_up(util_files.read_data_plist(plistXml.encode("utf-8"))) + except Exception: + print(f"whereToLookUpHardware failed, carrying on without it: {traceback.format_exc()}") + return None diff --git a/app/src/test/python/conftest.py b/app/src/test/python/conftest.py index fcedab57..2af80f8b 100644 --- a/app/src/test/python/conftest.py +++ b/app/src/test/python/conftest.py @@ -11,7 +11,18 @@ APP_PYTHON = Path(__file__).resolve().parents[2] / "main" / "python" RESOURCES = Path(__file__).resolve().parents[1] / "resources" +# The shared export package. Put on the path here so these tests can exercise it, and **the app +# cannot yet**: Chaquopy is not given it as a second source directory, so `import +# opentagviewer_export` fails inside the APK and main.py's two bridge functions return None there. +# +# That is a known gap rather than an oversight - `../python` also holds the tkinter wizard and a +# top-level package called `test`, which would shadow the standard library's on a phone. See +# "Wire the shared package into Chaquopy" in docs/android-import-handover.md for the two ways out. +# Whichever is taken changes the layout, and this line with it. +SHARED_PYTHON = Path(__file__).resolve().parents[4] / "python" + sys.path.insert(0, str(APP_PYTHON)) +sys.path.insert(0, str(SHARED_PYTHON)) def pytest_configure(config): diff --git a/app/src/test/python/test_hardware_bridge.py b/app/src/test/python/test_hardware_bridge.py new file mode 100644 index 00000000..69a0554c --- /dev/null +++ b/app/src/test/python/test_hardware_bridge.py @@ -0,0 +1,70 @@ +""" +The app's side of the shared hardware heuristic. + +The heuristic itself is tested in `python/opentagviewer_export/tests/test_hardware.py`. What is +tested here is the bridge: that the app can reach it at all, that it reads the plist the app +actually stores, and that a failure costs a label rather than an accessory. +""" + +from __future__ import annotations + +import pytest + +import main + +PLIST = """ + + + + identifier725A989D-D871-49A7-B2FE-948C24F356AB + model + productId21760 + vendorId76 + stableIdentifier2001~#001234a12345aaac~#A02BCDEFG1AB + + +""" + + +def plist_with(body: str) -> str: + return PLIST.replace( + "productId21760\n vendorId76", + body, + ) + + +class TestIdentifying: + def test_recognises_an_airtag_from_the_plist_the_app_stores(self): + assert main.identifyHardware(PLIST) == "AirTag" + + def test_recognises_one_of_the_owners_own_devices(self): + as_ipad = plist_with("productId-1").replace( + "model", "modeliPad13,18") + + assert main.identifyHardware(as_ipad) == "iPad (iPad13,18)" + + def test_says_nothing_rather_than_guessing_when_it_does_not_know(self): + bare = plist_with("productId-1") + + assert main.identifyHardware(bare) is None + + def test_a_broken_plist_costs_a_label_and_not_an_accessory(self): + # This is decoration on a row in a list. It must never be the reason a tag fails to load. + assert main.identifyHardware("not a plist at all") is None + + +class TestPointingAtTheRegistry: + def test_offers_the_lookup_for_a_maker_nobody_here_knows(self): + unknown = plist_with( + "productId1\n vendorId2457") + + advice = main.whereToLookUpHardware(unknown) + + assert advice is not None + assert "0x0999" in advice + + def test_says_nothing_for_a_maker_it_names(self): + assert main.whereToLookUpHardware(PLIST) is None + + def test_a_broken_plist_is_not_fatal_here_either(self): + assert main.whereToLookUpHardware("not a plist at all") is None diff --git a/app/src/test/resources/09082026_2/BeaconNamingRecord/725A989D-D871-49A7-B2FE-948C24F356AB/59AFE75B-8BD8-4522-85BC-F9B0BECD257A.plist b/app/src/test/resources/09082026_2/BeaconNamingRecord/725A989D-D871-49A7-B2FE-948C24F356AB/59AFE75B-8BD8-4522-85BC-F9B0BECD257A.plist new file mode 100644 index 00000000..87b295fd --- /dev/null +++ b/app/src/test/resources/09082026_2/BeaconNamingRecord/725A989D-D871-49A7-B2FE-948C24F356AB/59AFE75B-8BD8-4522-85BC-F9B0BECD257A.plist @@ -0,0 +1,52 @@ + + + + + associatedBeacon + 725A989D-D871-49A7-B2FE-948C24F356AB + cloudKitMetadata + + YnBsaXN0MDDUAQIDBAUGBz9YJHZlcnNpb25ZJGFyY2hpdmVyVCR0b3BYJG9iamVjdHMS + AAGGoF8QD05TS2V5ZWRBcmNoaXZlct8QLAgJCgsMDQ4PEBESExQVFhcYGRobHB0eHyAh + IiMkJSYnKCkqKywtLi8wMTIzNDU0NjQ0NTc0NDQ1NDQ0NTg5NDQ0NDQ6NDs0NTQ0NDQ1 + PDc1NDU0PT40NTRfEBZUb21ic3RvbmVkUHVibGljS2V5SURzXxAZSGFzVXBkYXRlZFBh + cmVudFJlZmVyZW5jZV8QE0NoYWluUHJvdGVjdGlvbkRhdGFdS25vd25Ub1NlcnZlcl8Q + EURpc3BsYXllZEhvc3RuYW1lWUJhc2VUb2tlbl8QEFdhbnRzQ2hhaW5QQ1NLZXlbUmVj + b3JkQ3RpbWVaUm91dGluZ0tleV8QElByb3RlY3Rpb25EYXRhRXRhZ15FeHBpcmF0aW9u + RGF0ZV8QGU1lcmdlYWJsZVZhbHVlRGVsdGFSZWNvcmRfECZQcmV2aW91c1Byb3RlY3Rp + b25EYXRhRXRhZ0Zyb21Vbml0VGVzdF8QEkNvbmZsaWN0TG9zZXJFdGFnc18QGlByZXZp + b3VzUHJvdGVjdGlvbkRhdGFFdGFnXxAUSGFzVXBkYXRlZEV4cGlyYXRpb25aUmVjb3Jk + VHlwZV8QE0NyZWF0b3JVc2VyUmVjb3JkSURfEA9QYXJlbnRSZWZlcmVuY2VZU2hhcmVF + dGFnWFBDU0tleUlEXFpvbmVpc2hLZXlJRF8QIE11dGFibGVFbmNyeXB0ZWRQdWJsaWNT + aGFyaW5nS2V5VEVUYWdfEBZQcmV2aW91c1NoYXJlUmVmZXJlbmNlXxAQTW9kaWZpZWRC + eURldmljZV5Qcm90ZWN0aW9uRGF0YV8QEVVzZUxpZ2h0d2VpZ2h0UENTXlNoYXJlUmVm + ZXJlbmNlXxARVXBkYXRlZEV4cGlyYXRpb25TVVJMXxAWQ2hhaW5QYXJlbnRQdWJsaWNL + ZXlJRFdFeHBpcmVkXxAYTGFzdE1vZGlmaWVkVXNlclJlY29yZElEW1JlY29yZE10aW1l + XxAVV2FudHNQdWJsaWNTaGFyaW5nS2V5XxAWWm9uZVByb3RlY3Rpb25EYXRhRXRhZ1lX + YXNDYWNoZWRfEA9DaGFpblByaXZhdGVLZXlaUGVybWlzc2lvblhSZWNvcmRJRFxBbGxQ + Q1NLZXlJRHNfEBhIYXNVcGRhdGVkU2hhcmVSZWZlcmVuY2VfEBdQcmV2aW91c1BhcmVu + dFJlZmVyZW5jZYAACAmACYABgAuAE4ASgBAQAYACrxAUQEFCSUpTVFVbXmJlVGhrVGxU + bm9VJG51bGxfEBJCZWFjb25OYW1pbmdSZWNvcmTTQ0RFRkdIViRjbGFzc1pSZWNvcmRO + YW1lVlpvbmVJRIAIgAOABF8QJDU5QUZFNzVCLThCRDgtNDUyMi04NUJDLUY5QjBCRUNE + MjU3QdVLTE1OQ080UFFSXxAQZGF0YWJhc2VTY29wZUtleV8QEWFub255bW91c0NLVXNl + cklEWW93bmVyTmFtZVhab25lTmFtZRAAgAaABYAHW0JlYWNvblN0b3JlXxAQX19kZWZh + dWx0T3duZXJfX9JWV1hZWiRjbGFzc25hbWVYJGNsYXNzZXNeQ0tSZWNvcmRab25lSUSi + WFpYTlNPYmplY3TSVldcXVpDS1JlY29yZElEolxa0l9DYGFXTlMudGltZSNBxrh/uF+d + soAK0lZXY2RWTlNEYXRlomNa00NERUZmZ4AMgA3VS0xNTkNPNGlqUoAPgA5cX2RlZmF1 + bHRab25l00NERUZtZ4ARXEV4YW1wbGUgaVBhZFFtAAgAEQAaACQAKQAyADcASQCkAL0A + 2QDvAP0BEQEbAS4BOgFFAVoBaQGFAa4BwwHgAfcCAgIYAioCNAI9AkoCbQJyAosCngKt + AsEC0ALkAugDAQMJAyQDMANIA2EDawN9A4gDkQOeA7kD0wPVA9YD1wPZA9sD3QPfA+ED + 4wPlA+cD/gQEBBkEIAQnBDIEOQQ7BD0EPwRmBHEEhASYBKIEqwStBK8EsQSzBL8E0gTX + BOIE6wT6BP0FBgULBRYFGQUeBSYFLwUxBTYFPQVABUcFSQVLBVYFWAVaBWcFbgVwBX0A + AAAAAAACAQAAAAAAAABwAAAAAAAAAAAAAAAAAAAFfw== + + emoji + 🐈 + identifier + 59AFE75B-8BD8-4522-85BC-F9B0BECD257A + name + cat + roleId + 9 + + diff --git a/app/src/test/resources/09082026_2/KeyAlignmentRecords/725A989D-D871-49A7-B2FE-948C24F356AB/D2C5F114-987C-48DB-9900-1E5B6D98671D.plist b/app/src/test/resources/09082026_2/KeyAlignmentRecords/725A989D-D871-49A7-B2FE-948C24F356AB/D2C5F114-987C-48DB-9900-1E5B6D98671D.plist new file mode 100644 index 00000000..3c36725c --- /dev/null +++ b/app/src/test/resources/09082026_2/KeyAlignmentRecords/725A989D-D871-49A7-B2FE-948C24F356AB/D2C5F114-987C-48DB-9900-1E5B6D98671D.plist @@ -0,0 +1,51 @@ + + + + + beaconIdentifier + 725A989D-D871-49A7-B2FE-948C24F356AB + cloudKitMetadata + + YnBsaXN0MDDUAQIDBAUGB0BYJHZlcnNpb25ZJGFyY2hpdmVyVCR0b3BYJG9iamVjdHMS + AAGGoF8QD05TS2V5ZWRBcmNoaXZlct8QLAgJCgsMDQ4PEBESExQVFhcYGRobHB0eHyAh + IiMkJSYnKCkqKywtLi8wMTIzNDU0NjQ0NTc0NDQ1NDQ0NTg5NDQ0NDQ6NDs0NTQ0NDQ1 + PD01NDU0Pj80NTRfEBZUb21ic3RvbmVkUHVibGljS2V5SURzXxAZSGFzVXBkYXRlZFBh + cmVudFJlZmVyZW5jZV8QE0NoYWluUHJvdGVjdGlvbkRhdGFdS25vd25Ub1NlcnZlcl8Q + EURpc3BsYXllZEhvc3RuYW1lWUJhc2VUb2tlbl8QEFdhbnRzQ2hhaW5QQ1NLZXlbUmVj + b3JkQ3RpbWVaUm91dGluZ0tleV8QElByb3RlY3Rpb25EYXRhRXRhZ15FeHBpcmF0aW9u + RGF0ZV8QGU1lcmdlYWJsZVZhbHVlRGVsdGFSZWNvcmRfECZQcmV2aW91c1Byb3RlY3Rp + b25EYXRhRXRhZ0Zyb21Vbml0VGVzdF8QEkNvbmZsaWN0TG9zZXJFdGFnc18QGlByZXZp + b3VzUHJvdGVjdGlvbkRhdGFFdGFnXxAUSGFzVXBkYXRlZEV4cGlyYXRpb25aUmVjb3Jk + VHlwZV8QE0NyZWF0b3JVc2VyUmVjb3JkSURfEA9QYXJlbnRSZWZlcmVuY2VZU2hhcmVF + dGFnWFBDU0tleUlEXFpvbmVpc2hLZXlJRF8QIE11dGFibGVFbmNyeXB0ZWRQdWJsaWNT + aGFyaW5nS2V5VEVUYWdfEBZQcmV2aW91c1NoYXJlUmVmZXJlbmNlXxAQTW9kaWZpZWRC + eURldmljZV5Qcm90ZWN0aW9uRGF0YV8QEVVzZUxpZ2h0d2VpZ2h0UENTXlNoYXJlUmVm + ZXJlbmNlXxARVXBkYXRlZEV4cGlyYXRpb25TVVJMXxAWQ2hhaW5QYXJlbnRQdWJsaWNL + ZXlJRFdFeHBpcmVkXxAYTGFzdE1vZGlmaWVkVXNlclJlY29yZElEW1JlY29yZE10aW1l + XxAVV2FudHNQdWJsaWNTaGFyaW5nS2V5XxAWWm9uZVByb3RlY3Rpb25EYXRhRXRhZ1lX + YXNDYWNoZWRfEA9DaGFpblByaXZhdGVLZXlaUGVybWlzc2lvblhSZWNvcmRJRFxBbGxQ + Q1NLZXlJRHNfEBhIYXNVcGRhdGVkU2hhcmVSZWZlcmVuY2VfEBdQcmV2aW91c1BhcmVu + dFJlZmVyZW5jZYAACAmACYABgAyAFIATgBGACxABgAKvEBVBQkNKS1RVVlxfY2ZoVWtu + VW9VcXJVJG51bGxfEBJLZXlBbGlnbm1lbnRSZWNvcmTTREVGR0hJViRjbGFzc1pSZWNv + cmROYW1lVlpvbmVJRIAIgAOABF8QJDcyNUE5ODlELUQ4NzEtNDlBNy1CMkZFLTk0OEMy + NEYzNTZBQtVMTU5PRFA0UVJTXxAQZGF0YWJhc2VTY29wZUtleV8QEWFub255bW91c0NL + VXNlcklEWW93bmVyTmFtZVhab25lTmFtZRAAgAaABYAHW0JlYWNvblN0b3JlXxAQX19k + ZWZhdWx0T3duZXJfX9JXWFlaWiRjbGFzc25hbWVYJGNsYXNzZXNeQ0tSZWNvcmRab25l + SUSiWVtYTlNPYmplY3TSV1hdXlpDS1JlY29yZElEol1b0mBEYWJXTlMudGltZSNBxrh/ + uJtkWoAK0ldYZGVWTlNEYXRlomRb0mBEZ2IjQcgFvG98SbrTREVGR2lqgA2ADtVMTU5P + RFA0bG1TgBCAD1xfZGVmYXVsdFpvbmXTREVGR3BqgBJcRXhhbXBsZSBpUGFkUmR6AAgA + EQAaACQAKQAyADcASQCkAL0A2QDvAP0BEQEbAS4BOgFFAVoBaQGFAa4BwwHgAfcCAgIY + AioCNAI9AkoCbQJyAosCngKtAsEC0ALkAugDAQMJAyQDMANIA2EDawN9A4gDkQOeA7kD + 0wPVA9YD1wPZA9sD3QPfA+ED4wPlA+cD6QQBBAcEHAQjBCoENQQ8BD4EQARCBGkEdASH + BJsEpQSuBLAEsgS0BLYEwgTVBNoE5QTuBP0FAAUJBQ4FGQUcBSEFKQUyBTQFOQVABUMF + SAVRBVgFWgVcBWcFaQVrBXgFfwWBBY4AAAAAAAACAQAAAAAAAABzAAAAAAAAAAAAAAAA + AAAFkQ== + + identifier + D2C5F114-987C-48DB-9900-1E5B6D98671D + lastIndexObservationDate + 2026-07-13T18:45:10Z + lastIndexObserved + 48090 + + diff --git a/app/src/test/resources/09082026_2/OPENTAGVIEWER.yml b/app/src/test/resources/09082026_2/OPENTAGVIEWER.yml new file mode 100644 index 00000000..a13905a8 --- /dev/null +++ b/app/src/test/resources/09082026_2/OPENTAGVIEWER.yml @@ -0,0 +1,4 @@ +exportTimestamp: 1786282770586 +sourceUser: user +version: 0.0.2 +via: OpenTagViewer.app:1.0.5 diff --git a/app/src/test/resources/09082026_2/OwnedBeacons/725A989D-D871-49A7-B2FE-948C24F356AB.plist b/app/src/test/resources/09082026_2/OwnedBeacons/725A989D-D871-49A7-B2FE-948C24F356AB.plist new file mode 100644 index 00000000..cfbeb8bb --- /dev/null +++ b/app/src/test/resources/09082026_2/OwnedBeacons/725A989D-D871-49A7-B2FE-948C24F356AB.plist @@ -0,0 +1,106 @@ + + + + + batteryLevel + 1 + cloudKitMetadata + + YnBsaXN0MDDUAQIDBAUGB0BYJHZlcnNpb25ZJGFyY2hpdmVyVCR0b3BYJG9iamVjdHMS + AAGGoF8QD05TS2V5ZWRBcmNoaXZlct8QLAgJCgsMDQ4PEBESExQVFhcYGRobHB0eHyAh + IiMkJSYnKCkqKywtLi8wMTIzNDU0NjQ0NTc0NDQ1NDQ0NTg5NDQ0NDQ6NDs0NTQ0NDQ1 + PD01NDU0Pj80NTRfEBZUb21ic3RvbmVkUHVibGljS2V5SURzXxAZSGFzVXBkYXRlZFBh + cmVudFJlZmVyZW5jZV8QE0NoYWluUHJvdGVjdGlvbkRhdGFdS25vd25Ub1NlcnZlcl8Q + EURpc3BsYXllZEhvc3RuYW1lWUJhc2VUb2tlbl8QEFdhbnRzQ2hhaW5QQ1NLZXlbUmVj + b3JkQ3RpbWVaUm91dGluZ0tleV8QElByb3RlY3Rpb25EYXRhRXRhZ15FeHBpcmF0aW9u + RGF0ZV8QGU1lcmdlYWJsZVZhbHVlRGVsdGFSZWNvcmRfECZQcmV2aW91c1Byb3RlY3Rp + b25EYXRhRXRhZ0Zyb21Vbml0VGVzdF8QEkNvbmZsaWN0TG9zZXJFdGFnc18QGlByZXZp + b3VzUHJvdGVjdGlvbkRhdGFFdGFnXxAUSGFzVXBkYXRlZEV4cGlyYXRpb25aUmVjb3Jk + VHlwZV8QE0NyZWF0b3JVc2VyUmVjb3JkSURfEA9QYXJlbnRSZWZlcmVuY2VZU2hhcmVF + dGFnWFBDU0tleUlEXFpvbmVpc2hLZXlJRF8QIE11dGFibGVFbmNyeXB0ZWRQdWJsaWNT + aGFyaW5nS2V5VEVUYWdfEBZQcmV2aW91c1NoYXJlUmVmZXJlbmNlXxAQTW9kaWZpZWRC + eURldmljZV5Qcm90ZWN0aW9uRGF0YV8QEVVzZUxpZ2h0d2VpZ2h0UENTXlNoYXJlUmVm + ZXJlbmNlXxARVXBkYXRlZEV4cGlyYXRpb25TVVJMXxAWQ2hhaW5QYXJlbnRQdWJsaWNL + ZXlJRFdFeHBpcmVkXxAYTGFzdE1vZGlmaWVkVXNlclJlY29yZElEW1JlY29yZE10aW1l + XxAVV2FudHNQdWJsaWNTaGFyaW5nS2V5XxAWWm9uZVByb3RlY3Rpb25EYXRhRXRhZ1lX + YXNDYWNoZWRfEA9DaGFpblByaXZhdGVLZXlaUGVybWlzc2lvblhSZWNvcmRJRFxBbGxQ + Q1NLZXlJRHNfEBhIYXNVcGRhdGVkU2hhcmVSZWZlcmVuY2VfEBdQcmV2aW91c1BhcmVu + dFJlZmVyZW5jZYAACAmACYABgAyAFIATgBGACxABgAKvEBVBQkNKS1RVVlxfY2ZoVWtu + VW9VcXJVJG51bGxfEBJNYXN0ZXJCZWFjb25SZWNvcmTTREVGR0hJViRjbGFzc1pSZWNv + cmROYW1lVlpvbmVJRIAIgAOABF8QJDcyNUE5ODlELUQ4NzEtNDlBNy1CMkZFLTk0OEMy + NEYzNTZBQtVMTU5PRFA0UVJTXxAQZGF0YWJhc2VTY29wZUtleV8QEWFub255bW91c0NL + VXNlcklEWW93bmVyTmFtZVhab25lTmFtZRAAgAaABYAHW0JlYWNvblN0b3JlXxAQX19k + ZWZhdWx0T3duZXJfX9JXWFlaWiRjbGFzc25hbWVYJGNsYXNzZXNeQ0tSZWNvcmRab25l + SUSiWVtYTlNPYmplY3TSV1hdXlpDS1JlY29yZElEol1b0mBEYWJXTlMudGltZSNBxrh/ + uIBiToAK0ldYZGVWTlNEYXRlomRb0mBEZ2IjQca4gvOFP33TREVGR2lqgA2ADtVMTU5P + RFA0bG1TgBCAD1xfZGVmYXVsdFpvbmXTREVGR3BqgBJcRXhhbXBsZSBpUGFkUXIACAAR + ABoAJAApADIANwBJAKQAvQDZAO8A/QERARsBLgE6AUUBWgFpAYUBrgHDAeAB9wICAhgC + KgI0Aj0CSgJtAnICiwKeAq0CwQLQAuQC6AMBAwkDJAMwA0gDYQNrA30DiAORA54DuQPT + A9UD1gPXA9kD2wPdA98D4QPjA+UD5wPpBAEEBwQcBCMEKgQ1BDwEPgRABEIEaQR0BIcE + mwSlBK4EsASyBLQEtgTCBNUE2gTlBO4E/QUABQkFDgUZBRwFIQUpBTIFNAU5BUAFQwVI + BVEFWAVaBVwFZwVpBWsFeAV/BYEFjgAAAAAAAAIBAAAAAAAAAHMAAAAAAAAAAAAAAAAA + AAWQ + + identifier + 725A989D-D871-49A7-B2FE-948C24F356AB + isZeus + + model + + pairingDate + 2025-02-27T20:15:44Z + privateKey + + key + + data + + J1AAk7qStLSbMhZT/XEve6by7hI0H7CslD/Oh7SrOc+mlmLnAO8c + 5FGnhi/s3TDlWNiL3SMy19NQuCWg6oTS+YfBZN79RiUmZtssTp9f + UvZjmqMX3g== + + + + productId + 21760 + publicKey + + key + + data + + k6fWaOxFGbClYV6tu/ZK4vXdyWl2joSbJhbzu12Pfmf5p09w5LxK + IvnABRfysSFkOAlo/F3Ii9Dq + + + + secondarySharedSecret + + key + + data + + 1pWMT+FI3flAWmgbUEW5H6omZy+yZOzp30zZGxEa2A8= + + + + sharedSecret + + key + + data + + vM2ZjU/sKW/novHcwzTlY5xwGLOUOZjpgcZa9cNx2Y8= + + + + stableIdentifier + + 2001~#001234a12345aaac~#A02BCDEFG1AB + + systemVersion + 2.0.73 + vendorId + 76 + + diff --git a/app/src/test/resources/09082026_3/BeaconNamingRecord/725A989D-D871-49A7-B2FE-948C24F356AB/59AFE75B-8BD8-4522-85BC-F9B0BECD257A.plist b/app/src/test/resources/09082026_3/BeaconNamingRecord/725A989D-D871-49A7-B2FE-948C24F356AB/59AFE75B-8BD8-4522-85BC-F9B0BECD257A.plist new file mode 100644 index 00000000..87b295fd --- /dev/null +++ b/app/src/test/resources/09082026_3/BeaconNamingRecord/725A989D-D871-49A7-B2FE-948C24F356AB/59AFE75B-8BD8-4522-85BC-F9B0BECD257A.plist @@ -0,0 +1,52 @@ + + + + + associatedBeacon + 725A989D-D871-49A7-B2FE-948C24F356AB + cloudKitMetadata + + YnBsaXN0MDDUAQIDBAUGBz9YJHZlcnNpb25ZJGFyY2hpdmVyVCR0b3BYJG9iamVjdHMS + AAGGoF8QD05TS2V5ZWRBcmNoaXZlct8QLAgJCgsMDQ4PEBESExQVFhcYGRobHB0eHyAh + IiMkJSYnKCkqKywtLi8wMTIzNDU0NjQ0NTc0NDQ1NDQ0NTg5NDQ0NDQ6NDs0NTQ0NDQ1 + PDc1NDU0PT40NTRfEBZUb21ic3RvbmVkUHVibGljS2V5SURzXxAZSGFzVXBkYXRlZFBh + cmVudFJlZmVyZW5jZV8QE0NoYWluUHJvdGVjdGlvbkRhdGFdS25vd25Ub1NlcnZlcl8Q + EURpc3BsYXllZEhvc3RuYW1lWUJhc2VUb2tlbl8QEFdhbnRzQ2hhaW5QQ1NLZXlbUmVj + b3JkQ3RpbWVaUm91dGluZ0tleV8QElByb3RlY3Rpb25EYXRhRXRhZ15FeHBpcmF0aW9u + RGF0ZV8QGU1lcmdlYWJsZVZhbHVlRGVsdGFSZWNvcmRfECZQcmV2aW91c1Byb3RlY3Rp + b25EYXRhRXRhZ0Zyb21Vbml0VGVzdF8QEkNvbmZsaWN0TG9zZXJFdGFnc18QGlByZXZp + b3VzUHJvdGVjdGlvbkRhdGFFdGFnXxAUSGFzVXBkYXRlZEV4cGlyYXRpb25aUmVjb3Jk + VHlwZV8QE0NyZWF0b3JVc2VyUmVjb3JkSURfEA9QYXJlbnRSZWZlcmVuY2VZU2hhcmVF + dGFnWFBDU0tleUlEXFpvbmVpc2hLZXlJRF8QIE11dGFibGVFbmNyeXB0ZWRQdWJsaWNT + aGFyaW5nS2V5VEVUYWdfEBZQcmV2aW91c1NoYXJlUmVmZXJlbmNlXxAQTW9kaWZpZWRC + eURldmljZV5Qcm90ZWN0aW9uRGF0YV8QEVVzZUxpZ2h0d2VpZ2h0UENTXlNoYXJlUmVm + ZXJlbmNlXxARVXBkYXRlZEV4cGlyYXRpb25TVVJMXxAWQ2hhaW5QYXJlbnRQdWJsaWNL + ZXlJRFdFeHBpcmVkXxAYTGFzdE1vZGlmaWVkVXNlclJlY29yZElEW1JlY29yZE10aW1l + XxAVV2FudHNQdWJsaWNTaGFyaW5nS2V5XxAWWm9uZVByb3RlY3Rpb25EYXRhRXRhZ1lX + YXNDYWNoZWRfEA9DaGFpblByaXZhdGVLZXlaUGVybWlzc2lvblhSZWNvcmRJRFxBbGxQ + Q1NLZXlJRHNfEBhIYXNVcGRhdGVkU2hhcmVSZWZlcmVuY2VfEBdQcmV2aW91c1BhcmVu + dFJlZmVyZW5jZYAACAmACYABgAuAE4ASgBAQAYACrxAUQEFCSUpTVFVbXmJlVGhrVGxU + bm9VJG51bGxfEBJCZWFjb25OYW1pbmdSZWNvcmTTQ0RFRkdIViRjbGFzc1pSZWNvcmRO + YW1lVlpvbmVJRIAIgAOABF8QJDU5QUZFNzVCLThCRDgtNDUyMi04NUJDLUY5QjBCRUNE + MjU3QdVLTE1OQ080UFFSXxAQZGF0YWJhc2VTY29wZUtleV8QEWFub255bW91c0NLVXNl + cklEWW93bmVyTmFtZVhab25lTmFtZRAAgAaABYAHW0JlYWNvblN0b3JlXxAQX19kZWZh + dWx0T3duZXJfX9JWV1hZWiRjbGFzc25hbWVYJGNsYXNzZXNeQ0tSZWNvcmRab25lSUSi + WFpYTlNPYmplY3TSVldcXVpDS1JlY29yZElEolxa0l9DYGFXTlMudGltZSNBxrh/uF+d + soAK0lZXY2RWTlNEYXRlomNa00NERUZmZ4AMgA3VS0xNTkNPNGlqUoAPgA5cX2RlZmF1 + bHRab25l00NERUZtZ4ARXEV4YW1wbGUgaVBhZFFtAAgAEQAaACQAKQAyADcASQCkAL0A + 2QDvAP0BEQEbAS4BOgFFAVoBaQGFAa4BwwHgAfcCAgIYAioCNAI9AkoCbQJyAosCngKt + AsEC0ALkAugDAQMJAyQDMANIA2EDawN9A4gDkQOeA7kD0wPVA9YD1wPZA9sD3QPfA+ED + 4wPlA+cD/gQEBBkEIAQnBDIEOQQ7BD0EPwRmBHEEhASYBKIEqwStBK8EsQSzBL8E0gTX + BOIE6wT6BP0FBgULBRYFGQUeBSYFLwUxBTYFPQVABUcFSQVLBVYFWAVaBWcFbgVwBX0A + AAAAAAACAQAAAAAAAABwAAAAAAAAAAAAAAAAAAAFfw== + + emoji + 🐈 + identifier + 59AFE75B-8BD8-4522-85BC-F9B0BECD257A + name + cat + roleId + 9 + + diff --git a/app/src/test/resources/09082026_3/CustomAccessories/tag-0a1b2c3d4e5f.json b/app/src/test/resources/09082026_3/CustomAccessories/tag-0a1b2c3d4e5f.json new file mode 100644 index 00000000..ead3e14b --- /dev/null +++ b/app/src/test/resources/09082026_3/CustomAccessories/tag-0a1b2c3d4e5f.json @@ -0,0 +1,10 @@ +{ + "identifier": "tag-0a1b2c3d4e5f", + "name": "Bike (OpenHaystack)", + "private_keys": [ + "000102030405060708090a0b0c0d0e0f101112131415161718191a1b", + "28292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f40414243", + "505152535455565758595a5b5c5d5e5f606162636465666768696a6b" + ], + "type": "custom_rolling_key_accessory" +} diff --git a/app/src/test/resources/09082026_3/KeyAlignmentRecords/725A989D-D871-49A7-B2FE-948C24F356AB/D2C5F114-987C-48DB-9900-1E5B6D98671D.plist b/app/src/test/resources/09082026_3/KeyAlignmentRecords/725A989D-D871-49A7-B2FE-948C24F356AB/D2C5F114-987C-48DB-9900-1E5B6D98671D.plist new file mode 100644 index 00000000..3c36725c --- /dev/null +++ b/app/src/test/resources/09082026_3/KeyAlignmentRecords/725A989D-D871-49A7-B2FE-948C24F356AB/D2C5F114-987C-48DB-9900-1E5B6D98671D.plist @@ -0,0 +1,51 @@ + + + + + beaconIdentifier + 725A989D-D871-49A7-B2FE-948C24F356AB + cloudKitMetadata + + YnBsaXN0MDDUAQIDBAUGB0BYJHZlcnNpb25ZJGFyY2hpdmVyVCR0b3BYJG9iamVjdHMS + AAGGoF8QD05TS2V5ZWRBcmNoaXZlct8QLAgJCgsMDQ4PEBESExQVFhcYGRobHB0eHyAh + IiMkJSYnKCkqKywtLi8wMTIzNDU0NjQ0NTc0NDQ1NDQ0NTg5NDQ0NDQ6NDs0NTQ0NDQ1 + PD01NDU0Pj80NTRfEBZUb21ic3RvbmVkUHVibGljS2V5SURzXxAZSGFzVXBkYXRlZFBh + cmVudFJlZmVyZW5jZV8QE0NoYWluUHJvdGVjdGlvbkRhdGFdS25vd25Ub1NlcnZlcl8Q + EURpc3BsYXllZEhvc3RuYW1lWUJhc2VUb2tlbl8QEFdhbnRzQ2hhaW5QQ1NLZXlbUmVj + b3JkQ3RpbWVaUm91dGluZ0tleV8QElByb3RlY3Rpb25EYXRhRXRhZ15FeHBpcmF0aW9u + RGF0ZV8QGU1lcmdlYWJsZVZhbHVlRGVsdGFSZWNvcmRfECZQcmV2aW91c1Byb3RlY3Rp + b25EYXRhRXRhZ0Zyb21Vbml0VGVzdF8QEkNvbmZsaWN0TG9zZXJFdGFnc18QGlByZXZp + b3VzUHJvdGVjdGlvbkRhdGFFdGFnXxAUSGFzVXBkYXRlZEV4cGlyYXRpb25aUmVjb3Jk + VHlwZV8QE0NyZWF0b3JVc2VyUmVjb3JkSURfEA9QYXJlbnRSZWZlcmVuY2VZU2hhcmVF + dGFnWFBDU0tleUlEXFpvbmVpc2hLZXlJRF8QIE11dGFibGVFbmNyeXB0ZWRQdWJsaWNT + aGFyaW5nS2V5VEVUYWdfEBZQcmV2aW91c1NoYXJlUmVmZXJlbmNlXxAQTW9kaWZpZWRC + eURldmljZV5Qcm90ZWN0aW9uRGF0YV8QEVVzZUxpZ2h0d2VpZ2h0UENTXlNoYXJlUmVm + ZXJlbmNlXxARVXBkYXRlZEV4cGlyYXRpb25TVVJMXxAWQ2hhaW5QYXJlbnRQdWJsaWNL + ZXlJRFdFeHBpcmVkXxAYTGFzdE1vZGlmaWVkVXNlclJlY29yZElEW1JlY29yZE10aW1l + XxAVV2FudHNQdWJsaWNTaGFyaW5nS2V5XxAWWm9uZVByb3RlY3Rpb25EYXRhRXRhZ1lX + YXNDYWNoZWRfEA9DaGFpblByaXZhdGVLZXlaUGVybWlzc2lvblhSZWNvcmRJRFxBbGxQ + Q1NLZXlJRHNfEBhIYXNVcGRhdGVkU2hhcmVSZWZlcmVuY2VfEBdQcmV2aW91c1BhcmVu + dFJlZmVyZW5jZYAACAmACYABgAyAFIATgBGACxABgAKvEBVBQkNKS1RVVlxfY2ZoVWtu + VW9VcXJVJG51bGxfEBJLZXlBbGlnbm1lbnRSZWNvcmTTREVGR0hJViRjbGFzc1pSZWNv + cmROYW1lVlpvbmVJRIAIgAOABF8QJDcyNUE5ODlELUQ4NzEtNDlBNy1CMkZFLTk0OEMy + NEYzNTZBQtVMTU5PRFA0UVJTXxAQZGF0YWJhc2VTY29wZUtleV8QEWFub255bW91c0NL + VXNlcklEWW93bmVyTmFtZVhab25lTmFtZRAAgAaABYAHW0JlYWNvblN0b3JlXxAQX19k + ZWZhdWx0T3duZXJfX9JXWFlaWiRjbGFzc25hbWVYJGNsYXNzZXNeQ0tSZWNvcmRab25l + SUSiWVtYTlNPYmplY3TSV1hdXlpDS1JlY29yZElEol1b0mBEYWJXTlMudGltZSNBxrh/ + uJtkWoAK0ldYZGVWTlNEYXRlomRb0mBEZ2IjQcgFvG98SbrTREVGR2lqgA2ADtVMTU5P + RFA0bG1TgBCAD1xfZGVmYXVsdFpvbmXTREVGR3BqgBJcRXhhbXBsZSBpUGFkUmR6AAgA + EQAaACQAKQAyADcASQCkAL0A2QDvAP0BEQEbAS4BOgFFAVoBaQGFAa4BwwHgAfcCAgIY + AioCNAI9AkoCbQJyAosCngKtAsEC0ALkAugDAQMJAyQDMANIA2EDawN9A4gDkQOeA7kD + 0wPVA9YD1wPZA9sD3QPfA+ED4wPlA+cD6QQBBAcEHAQjBCoENQQ8BD4EQARCBGkEdASH + BJsEpQSuBLAEsgS0BLYEwgTVBNoE5QTuBP0FAAUJBQ4FGQUcBSEFKQUyBTQFOQVABUMF + SAVRBVgFWgVcBWcFaQVrBXgFfwWBBY4AAAAAAAACAQAAAAAAAABzAAAAAAAAAAAAAAAA + AAAFkQ== + + identifier + D2C5F114-987C-48DB-9900-1E5B6D98671D + lastIndexObservationDate + 2026-07-13T18:45:10Z + lastIndexObserved + 48090 + + diff --git a/app/src/test/resources/09082026_3/OPENTAGVIEWER.yml b/app/src/test/resources/09082026_3/OPENTAGVIEWER.yml new file mode 100644 index 00000000..fbd7233a --- /dev/null +++ b/app/src/test/resources/09082026_3/OPENTAGVIEWER.yml @@ -0,0 +1,4 @@ +exportTimestamp: 1786282770586 +sourceUser: user +version: 0.0.3 +via: OpenTagViewer.cli:1.1.0 diff --git a/app/src/test/resources/09082026_3/OwnedBeacons/725A989D-D871-49A7-B2FE-948C24F356AB.plist b/app/src/test/resources/09082026_3/OwnedBeacons/725A989D-D871-49A7-B2FE-948C24F356AB.plist new file mode 100644 index 00000000..cfbeb8bb --- /dev/null +++ b/app/src/test/resources/09082026_3/OwnedBeacons/725A989D-D871-49A7-B2FE-948C24F356AB.plist @@ -0,0 +1,106 @@ + + + + + batteryLevel + 1 + cloudKitMetadata + + YnBsaXN0MDDUAQIDBAUGB0BYJHZlcnNpb25ZJGFyY2hpdmVyVCR0b3BYJG9iamVjdHMS + AAGGoF8QD05TS2V5ZWRBcmNoaXZlct8QLAgJCgsMDQ4PEBESExQVFhcYGRobHB0eHyAh + IiMkJSYnKCkqKywtLi8wMTIzNDU0NjQ0NTc0NDQ1NDQ0NTg5NDQ0NDQ6NDs0NTQ0NDQ1 + PD01NDU0Pj80NTRfEBZUb21ic3RvbmVkUHVibGljS2V5SURzXxAZSGFzVXBkYXRlZFBh + cmVudFJlZmVyZW5jZV8QE0NoYWluUHJvdGVjdGlvbkRhdGFdS25vd25Ub1NlcnZlcl8Q + EURpc3BsYXllZEhvc3RuYW1lWUJhc2VUb2tlbl8QEFdhbnRzQ2hhaW5QQ1NLZXlbUmVj + b3JkQ3RpbWVaUm91dGluZ0tleV8QElByb3RlY3Rpb25EYXRhRXRhZ15FeHBpcmF0aW9u + RGF0ZV8QGU1lcmdlYWJsZVZhbHVlRGVsdGFSZWNvcmRfECZQcmV2aW91c1Byb3RlY3Rp + b25EYXRhRXRhZ0Zyb21Vbml0VGVzdF8QEkNvbmZsaWN0TG9zZXJFdGFnc18QGlByZXZp + b3VzUHJvdGVjdGlvbkRhdGFFdGFnXxAUSGFzVXBkYXRlZEV4cGlyYXRpb25aUmVjb3Jk + VHlwZV8QE0NyZWF0b3JVc2VyUmVjb3JkSURfEA9QYXJlbnRSZWZlcmVuY2VZU2hhcmVF + dGFnWFBDU0tleUlEXFpvbmVpc2hLZXlJRF8QIE11dGFibGVFbmNyeXB0ZWRQdWJsaWNT + aGFyaW5nS2V5VEVUYWdfEBZQcmV2aW91c1NoYXJlUmVmZXJlbmNlXxAQTW9kaWZpZWRC + eURldmljZV5Qcm90ZWN0aW9uRGF0YV8QEVVzZUxpZ2h0d2VpZ2h0UENTXlNoYXJlUmVm + ZXJlbmNlXxARVXBkYXRlZEV4cGlyYXRpb25TVVJMXxAWQ2hhaW5QYXJlbnRQdWJsaWNL + ZXlJRFdFeHBpcmVkXxAYTGFzdE1vZGlmaWVkVXNlclJlY29yZElEW1JlY29yZE10aW1l + XxAVV2FudHNQdWJsaWNTaGFyaW5nS2V5XxAWWm9uZVByb3RlY3Rpb25EYXRhRXRhZ1lX + YXNDYWNoZWRfEA9DaGFpblByaXZhdGVLZXlaUGVybWlzc2lvblhSZWNvcmRJRFxBbGxQ + Q1NLZXlJRHNfEBhIYXNVcGRhdGVkU2hhcmVSZWZlcmVuY2VfEBdQcmV2aW91c1BhcmVu + dFJlZmVyZW5jZYAACAmACYABgAyAFIATgBGACxABgAKvEBVBQkNKS1RVVlxfY2ZoVWtu + VW9VcXJVJG51bGxfEBJNYXN0ZXJCZWFjb25SZWNvcmTTREVGR0hJViRjbGFzc1pSZWNv + cmROYW1lVlpvbmVJRIAIgAOABF8QJDcyNUE5ODlELUQ4NzEtNDlBNy1CMkZFLTk0OEMy + NEYzNTZBQtVMTU5PRFA0UVJTXxAQZGF0YWJhc2VTY29wZUtleV8QEWFub255bW91c0NL + VXNlcklEWW93bmVyTmFtZVhab25lTmFtZRAAgAaABYAHW0JlYWNvblN0b3JlXxAQX19k + ZWZhdWx0T3duZXJfX9JXWFlaWiRjbGFzc25hbWVYJGNsYXNzZXNeQ0tSZWNvcmRab25l + SUSiWVtYTlNPYmplY3TSV1hdXlpDS1JlY29yZElEol1b0mBEYWJXTlMudGltZSNBxrh/ + uIBiToAK0ldYZGVWTlNEYXRlomRb0mBEZ2IjQca4gvOFP33TREVGR2lqgA2ADtVMTU5P + RFA0bG1TgBCAD1xfZGVmYXVsdFpvbmXTREVGR3BqgBJcRXhhbXBsZSBpUGFkUXIACAAR + ABoAJAApADIANwBJAKQAvQDZAO8A/QERARsBLgE6AUUBWgFpAYUBrgHDAeAB9wICAhgC + KgI0Aj0CSgJtAnICiwKeAq0CwQLQAuQC6AMBAwkDJAMwA0gDYQNrA30DiAORA54DuQPT + A9UD1gPXA9kD2wPdA98D4QPjA+UD5wPpBAEEBwQcBCMEKgQ1BDwEPgRABEIEaQR0BIcE + mwSlBK4EsASyBLQEtgTCBNUE2gTlBO4E/QUABQkFDgUZBRwFIQUpBTIFNAU5BUAFQwVI + BVEFWAVaBVwFZwVpBWsFeAV/BYEFjgAAAAAAAAIBAAAAAAAAAHMAAAAAAAAAAAAAAAAA + AAWQ + + identifier + 725A989D-D871-49A7-B2FE-948C24F356AB + isZeus + + model + + pairingDate + 2025-02-27T20:15:44Z + privateKey + + key + + data + + J1AAk7qStLSbMhZT/XEve6by7hI0H7CslD/Oh7SrOc+mlmLnAO8c + 5FGnhi/s3TDlWNiL3SMy19NQuCWg6oTS+YfBZN79RiUmZtssTp9f + UvZjmqMX3g== + + + + productId + 21760 + publicKey + + key + + data + + k6fWaOxFGbClYV6tu/ZK4vXdyWl2joSbJhbzu12Pfmf5p09w5LxK + IvnABRfysSFkOAlo/F3Ii9Dq + + + + secondarySharedSecret + + key + + data + + 1pWMT+FI3flAWmgbUEW5H6omZy+yZOzp30zZGxEa2A8= + + + + sharedSecret + + key + + data + + vM2ZjU/sKW/novHcwzTlY5xwGLOUOZjpgcZa9cNx2Y8= + + + + stableIdentifier + + 2001~#001234a12345aaac~#A02BCDEFG1AB + + systemVersion + 2.0.73 + vendorId + 76 + + diff --git a/app/src/test/resources/19032025/OwnedBeacons/F612A183-492B-45A8-A5A2-233CA9062A94.plist b/app/src/test/resources/19032025/OwnedBeacons/F612A183-492B-45A8-A5A2-233CA9062A94.plist index 7ef2f397..f1a0f671 100644 --- a/app/src/test/resources/19032025/OwnedBeacons/F612A183-492B-45A8-A5A2-233CA9062A94.plist +++ b/app/src/test/resources/19032025/OwnedBeacons/F612A183-492B-45A8-A5A2-233CA9062A94.plist @@ -24,7 +24,7 @@ J1AAk7qStLSbMhZT/XEve6by7hI0H7CslD/Oh7SrOc+mlmLnAO8c 5FGnhi/s3TDlWNiL3SMy19NQuCWg6oTS+YfBZN79RiUmZtssTp9f - UvZjmqMX3o== + UvZjmqMX3g== @@ -57,7 +57,7 @@ data - vM2ZjU/sKW/novHcwzTlY5xwGLOUOZjpgcZa9cNx2Y+= + vM2ZjU/sKW/novHcwzTlY5xwGLOUOZjpgcZa9cNx2Y8= diff --git a/app/src/test/resources/README.md b/app/src/test/resources/README.md index 056b0f5c..45359223 100644 --- a/app/src/test/resources/README.md +++ b/app/src/test/resources/README.md @@ -1,5 +1,13 @@ # Test resources +**Export bundles** — the zip contents the Android app imports. Shared deliberately: the Java tests +read them as the thing being imported, and the shared export package's tests in +`python/opentagviewer_export/tests/` rebuild them to prove the writer still produces exactly this. + +What does *not* belong here is anything the exporter reads on the way in — the sample OpenHaystack +and Macless Haystack key files live in `python/test/resources/keyfiles/`, because a key file is +raw material consumed before a bundle exists and Java never sees one. + ## `19032025/` A redacted copy of a real OpenTagViewer export (`OPENTAGVIEWER.yml` reports @@ -43,6 +51,61 @@ That is what `_narrowAlignmentIfNeeded` in `main.py` exists to avoid. These files are committed to a public repo. The filler bytes are not derived from the originals; only lengths carried over. +## `09082026_2/` + +A second redacted export, `version: 0.0.2`, `via: OpenTagViewer.app:1.0.5`. Same layout as +`19032025/` plus the directory that version added: + +``` +09082026_2/ + OPENTAGVIEWER.yml + OwnedBeacons/.plist + BeaconNamingRecord//.plist + KeyAlignmentRecords//.plist +``` + +**It exists for the key alignment record**, which `19032025/` predates. Without one, an imported +accessory searches its whole key history on first locate; with one, a few hundred indices. Both +fixtures are read by `python/opentagviewer_export/tests/test_macos_fixture.py`, which rebuilds +them from their own contents and compares **bytes**, so the pair covers the format with and +without alignment. + +### Two things were corrected in these fixtures + +- **`19032025`'s key blobs were not valid canonical base64.** Redaction substituted characters + from the base64 alphabet while preserving length, which leaves padding bits set — the blob + decodes and re-encodes to something different, so nothing could round-trip it. They have been + re-encoded; the decoded bytes are unchanged, so every reader sees exactly what it saw before. +- **`09082026_2`'s naming record named a different beacon** than the directory it sits in. The + importer keys off the directory so it imported fine, but the record contradicted its own + filing. `associatedBeacon` now matches. + +## `09082026_3/` + +Format `0.0.3`: the same accessory as `09082026_2`, plus a **self-generated tag**. + +``` +09082026_3/ + OPENTAGVIEWER.yml <- version: 0.0.3 + OwnedBeacons/... + BeaconNamingRecord/... + KeyAlignmentRecords/... + CustomAccessories/tag-0a1b2c3d4e5f.json +``` + +**No Mac could have written this one**, which is the point of it. An OpenHaystack-style tag's keys +are a plain list rather than the inputs to Apple's rolling-key derivation, so the plist layout has +nowhere to put one - it goes in as FindMy.py's own `custom_rolling_key_accessory` JSON, in its own +directory, and `OPENTAGVIEWER.yml` says `0.0.3` **only because** the bundle contains one. A bundle +of Apple-paired tags from the same exporter still says `0.0.2`. + +So it exists for the Android importer to be built against: today's importer skips the unknown +directory silently. See [docs/android-import-handover.md](../../../../docs/android-import-handover.md). + +Generated by the exporter itself rather than assembled by hand, so it is exactly what the writer +produces. The three private keys are filler of the right length - 28 bytes each, as a SECP224R1 +private key is - and are not derived from anything real. + ## `beacons/` Reserved for additional fixtures of differing tag ages, for testing how the diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 00000000..1b570e28 --- /dev/null +++ b/docs/README.md @@ -0,0 +1,24 @@ +# docs + +Longer-form documents that are neither rules nor setup instructions. + +The rules for changing this project live in [AGENTS.md](../AGENTS.md); how to build and test it +lives in [CONTRIBUTING.md](../CONTRIBUTING.md). What lands here is the other thing: protocol +specifications and investigation records — the material that is expensive to rediscover and has +no natural home in a source comment. + +| Document | What it is | +| --- | --- | +| [how-to-export-with-the-cli.md](./how-to-export-with-the-cli.md) | How to export AirTags out of iCloud with no Mac. Written for whoever is running it rather than for whoever is changing it, which is why it is the one page here addressed to a user. | +| [android-import-handover.md](./android-import-handover.md) | What the Android app has to do to catch up with the exporter: the shared package's Chaquopy wiring, encrypted bundles, self-generated tags. A task brief - delete it when the work lands. | +| [anisette-native-android.md](./anisette-native-android.md) | Proof-of-concept log for running Apple's ADI libraries in-process on Android, and the measurements that came out of it. Historical; the implementation shipped. | +| [findmy-export/](./findmy-export/) | Clean-room specification of the protocol for reading FindMy accessory keys from iCloud without a Mac. Stage 1 written; five stages to go. | + +## If you add a document here + +Add a row above in the same commit. A list that nothing checks goes stale silently, which is +the failure mode [AGENTS.md rule 10](../AGENTS.md) exists to prevent. + +Records of investigations are worth keeping even when they describe work that has since +shipped, provided they say so at the top. What makes them worth the disk is the measurements and +the dead ends — the things a reader would otherwise pay for twice. diff --git a/docs/android-import-handover.md b/docs/android-import-handover.md new file mode 100644 index 00000000..9806dd0b --- /dev/null +++ b/docs/android-import-handover.md @@ -0,0 +1,177 @@ +# Handover: the Android side of the iCloud exporter + +> **A task brief, not documentation.** Delete it when the work lands. +> +> Written by whoever built the exporter, on a machine that **cannot build the Android app**. Every +> Python change described here is tested and passing; nothing Java or Gradle has been compiled, +> which is why some of it is described rather than done. + +The desktop exporter now reads accessories out of iCloud and writes bundles the app imports. Three +things it emits are ahead of what the app can read, and one thing it shares needs wiring. + +--- + +## 1. Wire the shared package into Chaquopy + +`python/opentagviewer_export/` is the one implementation of the bundle format and of the +accessory-identification heuristic. The app's Python layer already calls into it - +`identifyHardware` and `whereToLookUpHardware` in `app/src/main/python/main.py` - and those work, +with tests in `app/src/test/python/test_hardware_bridge.py` that pass on desktop CPython. + +**What is missing is the Chaquopy source directory**, so the import resolves in the APK too: + +```kotlin +chaquopy { + sourceSets { + getByName("main") { + srcDir("../python") + } + } +} +``` + +> [!WARNING] +> **Do not merge that line as written.** `../python` also contains `exporter/` (the desktop wizard, +> which imports tkinter) and `test/` - and a top-level package called `test` on the Python path +> **shadows the standard library's `test` module**. That is a real hazard in a phone app and the +> reason this was left undone rather than committed unverified. +> +> Two ways out, either fine: +> +> - **Move the shared package under a directory of its own** - `python/shared/opentagviewer_export/` +> - and point `srcDir` at `python/shared`, which then contains exactly one thing. The exporter +> reaches it by adding `shared` to its path; `python/pyproject.toml` and the PyInstaller spec both +> need to know. +> - **Filter the source set**, if Chaquopy's `srcDir` honours AGP's `exclude` patterns. Cheaper if +> it works; unverified here, which is the whole problem. +> +> Whichever, `app/src/test/python/conftest.py` puts `python/` on `sys.path` today and must be +> changed to match, or the tests will exercise a layout the app does not have. + +`main.py`'s two bridge functions never raise: an accessory is not worth losing over a label, so a +failed import there costs the label and logs. + +## 2. Let Java use the heuristic instead of its own + +`BeaconInformation.isAirTag()` and `isIpad()` are the older, narrower version of what +`opentagviewer_export/hardware.py` now does. They are not wrong; they are a subset: + +| | Java today | The shared heuristic | +| --- | --- | --- | +| AirTag | `productId == 21760` | the same test | +| iPad | `model.contains("iPad")` | every Apple model family, with the identifier kept alongside | +| AirPods | — | recognised, **including which unit** - left, right or case | +| Third-party tags | — | named by maker where the Bluetooth SIG registry knows them | +| Anything else | — | the vendor and product ids, plus where to look them up | + +**Prefer calling the bridge over porting the table.** The vendor list came out of the SIG's +registry and will need adding to as accessories turn up; two copies of it means two things to +update and one of them will be forgotten. The costs of a wrong answer are asymmetric and worth +remembering: a wrong name is believed, a hex number gets looked up. + +Keep `isAirTag()` if something depends on the boolean - it agrees with the shared version by +construction. + +## 3. Read a locked bundle + +**The exporter locks bundles by default** (AES-256, WinZip scheme, a generated 12-character code). +`java.util.zip.ZipInputStream` in `AppleZipImporterUtil` cannot decrypt anything at all - not AES, +not the legacy ZipCrypto - so a locked bundle currently fails to import with a message about the +zip rather than about a missing code. + +What this needs: + +- **zip4j**, or another reader that does AES. `net.lingala.zip4j:zip4j`. +- **A prompt.** `ZipFile.isEncrypted()` answers before anything is read, so the app can ask for the + code rather than failing. +- **The same normalisation the exporter uses**, or the code a user types will not match the bytes + the zip was locked with. It is in `opentagviewer_export/passcode.py`: strip spaces and hyphens, + uppercase, then fold `O` to `0` and `I`/`L` to `1`. Those letters are excluded from the alphabet + *because* people write them for the digits, so a code read off paper depends on this. + +Until then, exports for released app versions need `--no-password`. The +[CLI guide](./how-to-export-with-the-cli.md) says so prominently. + +## 4. Import a self-generated tag + +An OpenHaystack-style tag has no `privateKey`, `sharedSecret` or `secondarySharedSecret` - its keys +are a plain list - so the plist layout cannot hold one. The exporter writes them as: + +``` +CustomAccessories/.json +``` + +carrying FindMy.py's own `custom_rolling_key_accessory` mapping, and `OPENTAGVIEWER.yml` then +declares `version: 0.0.3` **only when the bundle contains one** - so a plist-only export is +byte-identical to what the previous version wrote, and a reader meeting `0.0.3` knows there is +something in it that `0.0.2` could not express. + +Today's importer skips the unknown directory silently. What it needs: + +- The new directory in `AppleZipImporterUtil.MATCHERS`. Note the identifier is **not** a UUID - + these come from files other tools wrote - so the existing v4-UUID regex does not apply. +- Storage for a key-list accessory. `OwnedBeacon.content` assumes a plist; `accessoryJson` is + already FindMy.py's own JSON and is the natural home, but the fetch path calls + `FindMyAccessory.from_json` specifically and a custom tag is a `FixedRollingKeyPairAccessory`. +- **A Room migration** if the schema moves - rule 1, and there is no going back from getting that + wrong. + +## 5. The app becomes an exporter too — and then most people stop needing an export + +Everything above is about the app *importing* better. This is the other direction, and it changes +what the zip is for. + +**The app can run this whole pipeline itself.** It already embeds FindMy.py through Chaquopy, and +the iCloud route is pure Python with no desktop in it: sign in, recover the keychain from an escrow +record, fetch, decrypt. The desktop exporter is a UI over `exporter/icloud.py`, and that module +imports nothing the app cannot have. + +Two consequences, and the second is the bigger one. + +**A third producer.** The app exports bundles of its own, for sharing a tag with somebody else. It +stamps its own `via:` — `OpenTagViewer.android:`, beside the desktop's +`OpenTagViewer.wizard` and `OpenTagViewer.cli` — and it writes them through +`opentagviewer_export`, the same shared package, so there is still one implementation of the +format. That is most of why the package exists. + +**And an owner does not need an export at all.** Today every user imports a zip, including people +exporting their own tags to their own phone — a round trip through a desktop for data the phone +could fetch directly. Once the app can read the account, that step disappears for anyone who owns +the tags: + +| Who | Needs an export today | Needs one after | +| --- | --- | --- | +| **Owner, own phone** | yes — desktop, zip, transfer, import | **no.** Sign in and the tags are there | +| **Owner, sharing with somebody** | yes | yes — the zip *is* the sharing mechanism | +| **Recipient** | yes | yes — they have no account with those tags on it | + +So the zip stops being a prerequisite and becomes what it always should have been: **how you give +a tag to another person.** [export-modes.md](./export-modes.md) calls these the *connected* and +*export* roles and says why connected is the default — the device passcode is spent once, so after +it syncing costs nothing and is what makes this a connection rather than an import. + +**Do not document any of this until it works.** The wiki currently tells every user to export, and +that is correct until the app can do without one. The paragraph to change is the prerequisite on +the wiki's Home page, and it should change when the feature ships and not before. + +Worth knowing while building it: the app must present `0PENTAGVIEWR` and not the exporter's +`0PENTAGXPORT` (§2 above, and rule 11), and a connected app writes to the account in a way the +exporter never does — it may enrol as a peer rather than only recovering. That is the one place +this stops being read-only, and it deserves its own decision rather than arriving as a side effect. + +## 6. Two smaller things + +**The FindMy pin is out of step with itself.** `app/build.gradle.kts` installs the +`feat/icloud-keychain-export` branch, while `app/src/test/python/requirements.txt` pins +`FindMy==0.9.8`. Whatever `test_pinned_versions_match_the_app_build` is checking, it is not those +two agreeing - so the bridge tests exercise a different library than the app ships. The exporter +solved the same problem with a lockfile that records the resolved commit; the app still needs +`@` appended before any build that leaves the machine, which its own comment already says. + +**The app registers as `0FINDMYPY001`.** Nothing passes a serial to FindMy.py, so the device the +user sees in their Apple account list is named after the library rather than after this app - +which is what [rule 11](../AGENTS.md) exists to prevent, and not what the docs describe. FindMy.py +now takes `serial=` on the Anisette provider and defaults CloudKit to it, so this is a one-line +fix at the point the provider is built, plus a decision that `0PENTAGVIEWR` is the app's. The +exporter presents `0PENTAGXPORT`, deliberately different: two installs, two entries, each +removable without breaking the other. diff --git a/docs/anisette-native-android.md b/docs/anisette-native-android.md new file mode 100755 index 00000000..f52ac8c6 --- /dev/null +++ b/docs/anisette-native-android.md @@ -0,0 +1,469 @@ +# PoC: native Anisette on Android — what is verified + +> **Status: historical record, kept for its measurements.** This is the proof-of-concept log +> that preceded the app's native Anisette implementation, and it is retained because the numbers +> in it — download sizes, load timings, the stub surface, the eleven entry points — cost real +> device time to obtain and would be expensive to rediscover. +> +> It is a lab notebook, not a description of the shipped code. Two kinds of reference in it have +> since moved on: +> +> - **The measurement scripts were throwaway** and are not in the tree. Each is described well +> enough to rewrite, and `python -m zipfile` plus an ELF parser is the whole toolkit. +> - **The probe and its tests were implemented properly** and renamed. See +> `app/src/main/cpp/adi.cpp`, `app/src/main/java/.../anisette/`, and the tests in +> `app/src/androidTest/.../anisette/`. +> +> For the authentication flow this Anisette work feeds, see +> [findmy-export/01-authentication.md](./findmy-export/01-authentication.md). + +2026-08-10. Everything below was run from a Windows desktop with no Android device involved, +so it verifies the parts that do not need one. It stops exactly where a device becomes +necessary. + +Reference implementation: [Dadoum/Provision](https://github.com/Dadoum/Provision) — +`lib/provision/adi.d` and `docs/ADI.md`, plus +[anisette-v3-server](https://github.com/Dadoum/anisette-v3-server) `source/app.d` for the +server wrapper around it. + +--- + +## 1. The libraries are still published, and reachable + +``` +HTTP/1.1 200 OK +Content-Length: 142139820 +Content-Type: application/vnd.android.package-archive +``` + +`https://apps.mzstatic.com/content/android-apple-music-apk/applemusic.apk` — Apple's own CDN, +no account, no auth. Nothing is redistributed by us, which is the same position every public +Anisette server is in. + +## 2. We do not need to download 142 MB + +`anisette-v3-server` downloads the whole APK, unzips it, and keeps two files. That is fine for +a server doing it once and unacceptable on a phone. + +A zip's central directory is at the end, so HTTP range requests can read the directory, locate +the two members, and fetch only those. Measured: + +``` +APK is 142 MB +central directory: 272 kB at offset 141867668 + lib/arm64-v8a/libCoreADI.so: 1.79 MB (deflated), ELF=True + lib/arm64-v8a/libstoreservicescore.so: 2.39 MB (deflated), ELF=True + +Downloaded 2.6 MB of 142 MB (1.8%) +``` + +**1.8% of the bytes.** On a phone that is the difference between a feature people accept and +one they cancel. Script: `scratchpad/poc_fetch_adi.py`. + +The APK also carries `armeabi-v7a`, `x86` and `x86_64`, so the ABI is a runtime choice — on a +device, simply the device's own. + +## 2a. But two libraries is not the real number — `DT_NEEDED` is + +2026-08-10. `libstoreservicescore.so` does not stand alone: + +``` +libstoreservicescore.so NEEDED libmediaplatform.so, libCoreFoundation.so, libc++_shared.so, ... +libCoreADI.so NEEDED liblog, libdl, libc <- clean +``` + +`dlopen()` follows `DT_NEEDED`. Dadoum's hand-written loader does not, which is why +anisette-v3-server gets away with keeping two files. The full transitive closure, measured +(`scratchpad/poc_closure.py`): + +``` +libstoreservicescore.so 2.39 MB -> libmediaplatform, libCoreFoundation, libc++_shared +libmediaplatform.so 2.36 MB -> libcurl, libCoreFoundation, libicu{i18n,uc,data}_sv_apple, + libdispatch, libxml2, libc++_shared +libCoreFoundation.so 2.01 MB -> libicu*, libcurl, libxml2, libBlocksRuntime, libdispatch +libicudata_sv_apple.so 10.23 MB +libcurl.so 3.40 MB libicui18n_sv_apple.so 2.60 MB +libicuuc_sv_apple.so 1.75 MB libxml2.so 2.00 MB +libc++_shared.so 1.03 MB libdispatch.so 0.38 MB +libBlocksRuntime.so 0.01 MB + +Closure: 11 libraries, 28.2 MB unpacked, 11.3 MB downloaded (7.9% of the APK) +``` + +Apple shipped their own CoreFoundation, libdispatch and ICU for Android, and ADI sits on top +of them. All four ABIs carry the complete set, so no ABI is a special case. + +This splits the feature into two shapes: + +| Path | Cost | Needs | +| --- | --- | --- | +| **A — honour `DT_NEEDED`** | 11.3 MB download, 28 MB on disk, 11 libraries | nothing; plain `dlopen` in dependency order | +| **B — ignore them** (Provision's approach) | 2.6 MB, 2 libraries | the manual ELF mapper | + +B is the shipping form. A is the one worth building first, because it needs no mapper and so +can answer *"do the eleven ADI calls work on Android at all"* immediately. If A works, the +mapper stops being a blocker and becomes a size optimisation — a far better project shape than +writing an ELF loader and only then finding out. + +Ordering is what makes A work without any linker trickery: bionic resolves `DT_NEEDED` against +already-loaded libraries by SONAME, so opening the closure bottom-up satisfies each library's +dependencies before it is opened. Nothing is on the default search path and nothing needs to be. + +## 3. `libCoreADI.so` is built to be loaded by an Android JVM + +It exports exactly three symbols: + +``` +JNI_OnLoad +cvu8io98wun +vdfut768ig +``` + +`JNI_OnLoad` is the entry point Android's `System.loadLibrary` calls. This library expects to +live in a JVM process. That is the whole premise of doing this in-app rather than over the +network, confirmed at the binary level. + +The two scrambled names are CoreADI's real entry points, and per `docs/ADI.md` they are +deliberately hard to reverse. **We do not call them.** We call the wrappers exported by +`libstoreservicescore.so`, which call into them. + +## 4. All eleven ADI entry points exist in today's build + +From `lib/provision/adi.d`, with signatures. Every one verified present in the +`libstoreservicescore.so` pulled today: + +| Function | Symbol | Signature | +| --- | --- | --- | +| `ADILoadLibraryWithPath` | `kq56gsgHG6` | `int(const char*)` | +| `ADISetAndroidID` | `Sph98paBcz` | `int(const char*, uint)` | +| `ADISetProvisioningPath` | `nf92ngaK92` | `int(const char*)` | +| `ADIProvisioningErase` | `p435tmhbla` | `int(ulong)` | +| `ADISynchronize` | `tn46gtiuhw` | `int(ulong, ubyte*, uint, ubyte**, uint*, ubyte**, uint*)` | +| `ADIProvisioningDestroy` | `fy34trz2st` | `int(uint)` | +| `ADIProvisioningEnd` | `uv5t6nhkui` | `int(uint, ubyte*, uint, ubyte*, uint)` | +| `ADIProvisioningStart` | `rsegvyrt87` | `int(ulong, ubyte*, uint, ubyte**, uint*, uint*)` | +| `ADIGetLoginCode` | `aslgmuibau` | `int(ulong)` | +| `ADIDispose` | `jk24uiwqrg` | `int(void*)` | +| `ADIOTPRequest` | `qi864985u0` | `int(ulong, ubyte**, uint*, ubyte**, uint*)` | + +``` + FOUND kq56gsgHG6 FOUND fy34trz2st + FOUND Sph98paBcz FOUND uv5t6nhkui + FOUND nf92ngaK92 FOUND rsegvyrt87 + FOUND p435tmhbla FOUND aslgmuibau + FOUND tn46gtiuhw FOUND jk24uiwqrg + FOUND qi864985u0 +``` + +Symbol dumper: `scratchpad/poc_elf_symbols.py` (pure Python, no NDK). + +**These names are not stable across builds** — `docs/ADI.md` says obfuscation differs per +build. So the loader has to fail legibly when a symbol is missing, and the version of the APK +it pulled should be recorded. + +--- + +## 5. RESOLVED (2026-08-10): it loads, and ADI initialises + +Run on the Gradle managed device (Pixel 6, API 34, x86_64, aosp-atd) by a load test against a +native probe. All three steps passed, `skipped="0"`. (Both have since been implemented as +`app/src/main/cpp/adi.cpp` and the tests in `app/src/androidTest/.../anisette/`.) + +**W^X does not block it.** SELinux said so itself: + +``` +avc: granted { execute } for path=".../files/adi-poc/x86_64/libc++_shared.so" + scontext=u:r:untrusted_app:s0 tcontext=u:object_r:app_data_file:s0 tclass=file +``` + +All eleven libraries `dlopen`'d out of `filesDir` in dependency order, RTLD_NOW | RTLD_GLOBAL, +no linker tricks and nothing on the search path. All eleven ADI entry points resolved. Then: + +``` +ADILoadLibraryWithPath(/data/user/0/.../files/adi-poc/x86_64) = 0 +``` + +0 is ADI's success code. Note what that implies: `libCoreADI.so` was never opened by us - +`libstoreservicescore.so` found and loaded it itself. Apple's CoreFoundation, libdispatch, ICU +and CoreADI all initialise inside an ordinary Android app process. + +**Consequence: the manual mmap+relocate ELF loader is not needed.** That was the only +open item that could have made this a multi-week project, and it is gone. Path B (ignoring +DT_NEEDED to save ~25 MB) is now purely a size optimisation and can wait indefinitely. + +### Confirmed on real hardware the same day + +Re-run on a Galaxy S25 (SM-S938B), **arm64-v8a, Android 16**, from Studio with +`-e adiPoc true` as an instrumentation extra param. All three steps passed, and there is no +`avc: denied` anywhere in logcat: + +``` +downloaded 12.6 MB of 142 MB (8.9%) +dlopen(.../arm64-v8a/libstoreservicescore.so) ok +resolved kq56gsgHG6 ... qi864985u0 (all eleven) +ADILoadLibraryWithPath(.../arm64-v8a) = 0 +``` + +So the emulator result was not a policy artefact of aosp-atd. Two numbers worth keeping: + +- **The download took 2.1 seconds.** This is a brief one-time spinner, not a progress-bar + problem. Design for the slow case rather than around it. +- **All eleven libraries loaded in 11 ms.** Dependency-ordered loading costs nothing. + +Correction to section 2a: the real cost is **12.6 MB, not 11.3 MB**. That script counted only +the DT_NEEDED graph, and `libCoreADI.so` (1.79 MB) is not a declared dependency of anything - +libstoreservicescore opens it by path at runtime. 12.6 MB is the measured figure. + +## 7. RESOLVED (2026-08-10): provisioning works end to end + +`AdiProvisioningTest` on the managed device, against Apple's live servers. +`tests="1" failures="0" skipped="0"`: + +``` +provisioning as 3ED7500C-29FA-4A26-99A2-47D8BD863E01 (ADI id bb1037520408cb27) +ADI initialised, persisting to .../files/adi-provisioning-state +provisioned: session 1458087437 completed +already provisioned <- second call correctly a no-op +``` + +Apple accepted an entirely invented machine identity, ADI swallowed the persistent token and +trust key, and ADIGetLoginCode then returned 0. **There is no research left in this feature.** + +Three things it took to get there, all of them things the reference does differently: + +**1. The identity lengths are checked.** See section on invented identity - `rndGen.take(2)` is +two uints, not two bytes. Wrong length gives -45001. + +**2. `gsa.apple.com` fails TLS against Android's trust store.** It chains to the original +"Apple Root CA", which Android does not ship: + +``` +SSLHandshakeException: Unacceptable certificate: CN=Apple Root CA, O=Apple Inc., C=US +``` + +Provision solves this with `request.sslSetVerifyPeer(false)`. **We do not** - turning off +certificate verification in an authentication flow inside a phone app makes it interceptable by +anything on the network path. Instead the one missing root is added for that one domain in +`network_security_config.xml`, alongside `system`, with the certificate stored as PEM text +(SHA-256 `b0b1730e...f024`, published at https://www.apple.com/appleca/). That is *stricter* +than the default, not looser. + +**3. The stub question is answered on the path that mattered.** `makeWorkQueue` was called, +returned NULL, and provisioning completed anyway. So stubbing holds through provisioning, and +that symbol is now recorded in `libmediaplatform.expected` - logged at INFO rather than raising +an alarm, because an alarm that fires on every successful run is one nobody reads. + +### What this does not yet prove + +- **ADIOTPRequest**, the per-login one-time password. Provisioning is the hard half; this is + the half that runs on every login. +- **The Python side** - a provider that hands FindMy.py the headers, alongside the existing + remote one. +- Historical note, now resolved: `ADILoadLibraryWithPath` returning 0 meant ADI initialised. The + ProvisioningStart -> Apple -> ProvisioningEnd round trip is untested. +- Apple can change the APK layout or re-obfuscate the symbols at any time, so the failure + paths matter as much as the happy one. + +## 6. The 28 MB can probably be stubbed away, without an ELF loader + +Measured by `scratchpad/poc_stub_surface.py`. `libstoreservicescore.so` has 405 undefined +symbols, and they come from only three libraries: + +``` +libc++_shared.so 141 symbols [1.0 MB] +libmediaplatform.so 125 symbols [2.4 MB] +libCoreFoundation.so 93 symbols [2.0 MB] +(platform / bionic) 46 symbols - already present +``` + +**ICU, curl, libxml2, libdispatch and BlocksRuntime are never referenced.** They are in the +closure only because CoreFoundation and mediaplatform depend on them. ICU alone is 14.6 MB. + +bionic satisfies DT_NEEDED by SONAME against already-loaded libraries and does not check who +built them. So two generated stub libraries - SONAME `libCoreFoundation.so` and +`libmediaplatform.so`, exporting those 218 names - satisfy the linker at a few kB each, and the +20 MB tail behind them stops being reachable because nothing declares a dependency on it. + +``` +today (path A): 11 libraries 11.3 MB downloaded 28.2 MB on disk +with stubs: 3 libraries 2.9 MB downloaded 5.2 MB on disk +``` + +Keep Apple's real `libc++_shared.so`: the mangled names are `__ndk1`-namespaced, so NDK 27's +own libc++ should not be assumed ABI-identical without checking. + +### CONFIRMED (2026-08-10): it works, and one function is called + +`StubbedAdiLoadTest` on the managed device. All three steps passed, `skipped="0"`. Three +libraries downloaded instead of twelve, all eleven ADI entry points still resolved, and +`ADILoadLibraryWithPath` still returned 0. + +Of **218 stubbed symbols, exactly one was reached**: + +``` +adi-stub: mediaplatform::WorkQueue::makeWorkQueue(std::string const&, WorkQueueType) +``` + +None of the 93 CoreFoundation symbols were touched at all. + +**And checking Provision shows this is lower risk than it looks.** Two things: + +*When it happens.* By timestamp, `makeWorkQueue` fired at `48.008` - the same millisecond as +`dlopen(libstoreservicescore.so) ok`, and before `ADILoadLibraryWithPath` at `48.026`. So it is +a **static constructor running during dlopen**, not something ADI called. + +*Why Provision never sees it.* Their symbol table (`lib/provision/symbols.d`) is 29 entries and +every one is libc/bionic - `dlopen`, `malloc`, `pthread_*` as `emptyStub`, `read`, `write`, +`__errno`, `__system_property_get`, `arc4random`. No CoreFoundation, no mediaplatform. Anything +else becomes a trampoline that *throws* `UndefinedSymbolException`. And their loader **never +runs `.init_array`** - `androidlibrary.d` has no constructor handling at all. + +So in the reference implementation that constructor never runs, and whatever global holds the +work queue stays zero: functionally identical to our stub returning NULL. Every public Anisette +server is a live demonstration that ADI's provisioning paths work with no work queue. Worth +watching during provisioning, but not an open question. + +This is why the stubs log and count instead of aborting. A stub that returns 0 produces a +*silent wrong answer*, which is worse than a crash, so each one logs an actionable error and +increments a counter that `_adi_stub_calls()` exposes - the hook the runtime fallback +to a remote Anisette server should hang off. + +## What is left, and where a device becomes necessary + +Everything above is acquisition and symbol resolution. The remaining work is the actual +integration: + +1. **JNI wrapper** over the eleven functions. `dlopen` `libstoreservicescore.so`, then + `ADILoadLibraryWithPath` pointing at the directory holding `libCoreADI.so`. +2. **Provisioning**: `ADISetAndroidID` with a generated identifier, `ADISetProvisioningPath` + into app storage, then `ADIProvisioningStart` → POST to Apple's `midStartProvisioning` → + `ADIProvisioningEnd`. One time, persisted. `adi.d` around lines 380–500 has the exact + request shape. +3. **Headers per login**: `ADIOTPRequest` produces the one-time password and machine ID that + the login flow needs. +4. **Python side**: a provider that calls back into Java, alongside `RemoteAnisetteProvider`. +5. **Fallback**: keep the remote provider. A local path that fails must degrade to today's + behaviour, not to a broken login. + +None of steps 1–3 can be tested without a device — the libraries are arm64 Android binaries, +and this desktop cannot load them. + +### The device identity is invented, not derived + +Worth settling, because it sounds like it should be the hard part and is not. The whole +identity a public Anisette server presents to Apple is randomly generated on first run: + +```d +v1Device.serverFriendlyDescription = clientInfo; // a UA-ish string +v1Device.uniqueDeviceIdentifier = randomUUID().toString().toUpper(); +v1Device.adiIdentifier = (cast(ubyte[]) rndGen.take(2).array()).toHexString().toLower(); +v1Device.localUserUUID = (cast(ubyte[]) rndGen.take(8).array()).toHexString().toUpper(); +``` + +**Careful with those lengths - I got them wrong first time round and ADI rejected it with +-45001 (invalid parameters).** `rndGen.take(2)` takes two *uints*, not two bytes: that is +8 bytes, 16 hex characters. `take(8)` is 32 bytes, 64 hex characters - which is what +`X-Apple-I-MD-LU` looks like in real Apple traffic. The values are invented, but the sizes are +checked. + +It is then persisted and reused, so the machine stays the same machine across logins. + +This matches what rustpush's author said independently in +[rustpush#18](https://github.com/OpenBubbles/rustpush/issues/18): + +> You still need apple hw IDs to send to Apple's server (X-MME-Client-Info or whatever) but you +> should be able to make something up, i don't think that stuff is strictly validated + +So there is no hardware attestation to defeat and nothing to extract from a real device. Every +public Anisette server in use today runs on invented identity, which is the working proof. + +What matters is that the identity is **generated once and kept** — regenerating per login would +make every session look like a new machine, which is exactly the pattern 2FA is designed to +notice. + +## Honest assessment + +The scope has not changed: this is still weeks, and the hard part was never acquisition. What +this PoC removes is uncertainty. Every question that could be answered without a device now is +answered, and none of the answers were bad: + +- the libraries are still published, and fetching them costs 2.6 MB +- they are Android-native, JNI-ready, and require no emulation +- all eleven entry points exist, with known signatures, in the current build + +There is no research left in the *sequence*. The identity is invented, the entry points are +known, the libraries are fetchable cheaply, and every public Anisette server is a live +demonstration that the calls work. + +### But there is no precedent for doing it on Android, and that matters + +Checked, expecting to find one: + +- **Provision / anisette-v3-server** (D) — `androidlibrary.d`, a hand-written ELF loader, for + running Android libraries on desktop Linux. +- **omnisette** (Rust, OpenBubbles) — depends on Dadoum's `android-loader`, and uses it in + `store_services_core/posix_macos.rs` and `posix_windows.rs`. **There is no Android file.** + Its only Anisette features are `remote-anisette-v3` and `remote-clearadi`; `clearadi` itself + is an empty submodule pointing at a closed-source stub. + +So **OpenBubbles on Android uses a remote Anisette server**, exactly as this app does today. The +whole ecosystem writes Android-library loaders *for desktop*, and on the one platform where the +libraries are native, everybody calls a server. + +That is not evidence it cannot work — these projects are desktop-first, and OpenBubbles' Android +client is a Flutter app that already had a server to talk to. But it does mean there is nothing +to copy for the loading step, and the one Android-specific obstacle is unanswered: + +**W^X.** Apps targeting API 29+ cannot execute code from writable app-home storage, and +`targetSdk` here is 35. `dlopen()` on a `.so` downloaded into the app's data directory is the +thing the platform is designed to refuse. If that holds, the options are: + +| Option | Consequence | +| --- | --- | +| `System.load()` from app data dir | free if it works; needs a fallback if it does not | +| **Map it ourselves** | always works, no file is ever executed — see below | +| Bundle in `jniLibs` | always works, but **we would be redistributing Apple's binaries** — the exact position fetching-from-Apple's-CDN avoids | + +### `dlopen` is not the only way to load a library + +W^X restricts executing **files** from app-writable storage. It does not stop an app mapping +anonymous memory and marking it executable — ART's own JIT depends on precisely that, which is +why `execmem` is granted to the `untrusted_app` SELinux domain. + +And a loader that does this already exists, in the reference implementation. `androidlibrary.d`: + +```d +allocation = MmapAllocator.instance.allocate(allocSize + shift); // anonymous, not file-backed +mprotect(..., PROT_READ | PROT_WRITE); // writable while loading +allocation[headerStart..headerEnd] = elfFile[fileStart..fileEnd]; // copy segments in +mprotect(..., prot); // then read+execute +// ... relocations applied by hand +``` + +No file is ever executed. The bytes are copied into anonymous pages and relocated there. + +**On Android this loader gets simpler, not harder.** Its companion `symbols.d` exists to map +bionic symbols (`open`, `pthread_create`, `__errno`, ...) onto glibc equivalents, because it +runs on desktop. On Android those symbols are the real ones — resolution is `dlsym` against the +platform's own libc, and that entire table becomes unnecessary. + +Implementation would be an NDK component: port the segment mapping and relocation logic to +C/C++ (Android's own linker is the reference), or reuse Dadoum's `android-loader` as a Rust +cdylib. Either is real work, but it is bounded, well-understood work with a working +implementation to read. + +### The experiment, in order + +1. **Try `System.load()` from `filesDir` first.** If it works, none of the above is needed. + Ten minutes on a device. +2. **If W^X refuses it**, map it manually. Known to work, code to copy, and simpler here than + in the project it comes from. +3. Bundling stays the last resort, because it is the only option that changes the licensing + position. + +Either way the feature is reachable without shipping Apple's binaries — which was the real +question. + +What remains is a JNI layer, a provisioning flow persisted to app storage, a Python provider +that calls back into Java, and a fallback to `RemoteAnisetteProvider` when any of it fails. +That is ordinary work — a fair amount of it, on a device, but ordinary. diff --git a/docs/assets/cli-choose-accessories.png b/docs/assets/cli-choose-accessories.png new file mode 100644 index 00000000..063cec6b Binary files /dev/null and b/docs/assets/cli-choose-accessories.png differ diff --git a/docs/export-modes.md b/docs/export-modes.md new file mode 100644 index 00000000..65d1f92e --- /dev/null +++ b/docs/export-modes.md @@ -0,0 +1,185 @@ +# How exporting works, and who runs it + +> **Status: plan, not description.** None of this is built yet. It records decisions taken while +> the protocol work was being specified, so that the UI work does not have to re-derive them. +> +> The protocol itself is in [findmy-export/](./findmy-export/). This document is about product +> shape: which flows exist, what each one costs the user, and which of them writes to their Apple +> account. + +--- + +## The change this plans for + +Today, getting tags out of an Apple account needs a **Mac** — the exporter reads the plists Find +My leaves on disk, which is why there is a VM bootstrap and why the wizard is macOS-only. The +protocol work replaces that with reading the same records out of iCloud. + +**So the Mac disappears, and with it the operating system requirement.** What is left is an Apple +ID, the screen-lock passcode of a device already on that account, and Python — which runs +anywhere, including inside the Android app through Chaquopy. + +The one thing it does not remove is the iPhone or iPad that paired the tag in the first place. +See [findmy-export/README.md](./findmy-export/README.md#the-one-thing-left-and-why-it-is-not-worth-starting). + +--- + +## Three roles, and only one of them writes + +| | Who | Needs | Writes to the account | +| --- | --- | --- | --- | +| **Owner, connected** | has an Apple account with tags, wants live tracking on Android | Apple ID, device passcode | **yes** — a peer and an escrow record | +| **Owner, exporting** | has tags, wants a zip — for themselves or a friend | Apple ID, device passcode | no | +| **Recipient** | has no tags of their own | a free Apple ID | no | +| **Sharee** *(not reachable yet)* | has tags **shared to them** through Apple's own sharing | Apple ID, device passcode | no | + +**The sharee is a real case and the one thing here that does not work yet.** Someone who was sent +an AirTag through Apple's own sharing sees the accessory in Find My on an iPhone, and this route +can see that the share exists — the records come back in the ordinary `BeaconStore` fetch — but a +shared accessory record **carries no key material**, so nothing can locate or export it. The keys +sit behind a sharing circle secret, which is a second key hierarchy and is not part of Stages 1 +to 6. See +[findmy-export/README.md](./findmy-export/README.md#shared-accessories-reachable-and-new) for what +it would take; the short version is that the records are already in hand, so the remaining work +is the unwrapping rather than another fetch path. + +**Worth knowing because it is the natural workaround people will reach for.** "Just have the owner +share it with me in Find My" does not substitute for an export today, and the app should say so +rather than showing a tag it cannot place. Until then, an owner sharing with an Android user +exports a bundle; Apple's sharing is not a route in. + +**The recipient's Apple ID is not optional and is often misremembered as being so.** Fetching +location reports is authenticated: any Apple ID may fetch reports for any tag's hashed keys, but +some Apple ID must. That is exactly why the app has a login flow today, and it is what makes an +export to a friend work at all — the service does not check that the requesting account owns the +tag. No Apple *device* is needed, and the account can be free and new. + +--- + +## Two exporters, one artefact + +The zip is the interface, and it does not care what produced it: + +- **The Android app**, for an owner who has an Android phone anyway. +- **The desktop wizard** (`python/`), repointed at the iCloud route, for an owner who does not — + or who would rather not put an Apple ID into an Android app. + +Both emit the layout in [findmy-export/06-output.md](./findmy-export/06-output.md), and both stamp +`via:` in `OPENTAGVIEWER.yml`, which is how anyone looking at a bundle later works out which +produced it. That stamping rule already exists — `AGENTS.md` rule 9 — and now has two producers to +distinguish rather than one to version. + +**Repointing the wizard is the larger user-visible win of the two.** It retires the VM bootstrap +and the macOS-only packaging, and it is the step people actually complain about. + +> ### The wizard persists nothing, and must keep not doing so +> +> Today it holds no credential at all: the password it asks for is macOS keychain access, used to +> unwrap the local plists, and nothing is written anywhere. Every run starts from nothing. +> +> **The new route raises the stakes of keeping that property**, because the credential stops being +> a local keychain password and becomes the user's **Apple ID password**, alongside a device +> passcode. Both should be read, used, and dropped. +> +> This needs saying because FindMy.py's default is the opposite: `to_json()` writes the account — +> Apple ID password included — to `account.json` in plaintext, and every example script in that +> library does exactly that. A wizard built by following those examples would start persisting a +> credential it has never held, on a desktop with no keystore behind it. The app is not exposed to +> this, because it encrypts the blob under a key held in `AndroidKeyStore` and never writes the +> library's own file. + +--- + +## Joining the trust circle is a separate decision from signing in + +**It is tempting to always join**, on the reasoning that a user who buys a tag later should see it +appear without being asked for anything. The outcome is right; the mechanism is not what it looks +like, and the difference decides the design. + +**A new tag does not need a join to be visible.** It arrives as a new record in the same zone, +protected under keys already held, and an incremental fetch finds it. + +**What a join actually buys is surviving key rotation.** A non-member receives no new key shares, +so when Apple's view keys roll — which happens for reasons outside this app, such as a device +being removed from the account — a non-member's keys go stale and the user is asked for the +device passcode again. A member is handed the new keys and notices nothing. + +So the choice is between *one passcode prompt, occasionally, at an unpredictable moment* and +*permanent artefacts on the user's Apple account*. That is a real trade, not an obvious one. + +> ### The route being replaced already joined, and nobody called it that +> +> The macOS export is easy to think of as the local, private, offline option. It is not. It reads +> files macOS wrote, and macOS wrote them by signing into iCloud, **joining the keychain trust +> circle as a peer**, and syncing the records down. The VM bootstrap does exactly that with a Mac +> that does not exist. +> +> So the status quo leaves a **full device peer and its escrow record** on the user's account — +> more than anything planned here — and it was invisible only because Apple's own software created +> it rather than ours. An export-only flow that writes nothing is a genuine improvement on that, +> not a compromise against it. + +### The decision + +**Join only when the user has asked for the connected experience.** Export is a read and must not +write anything. + +| Flow | Joins | +| --- | --- | +| One-off export, in the app or the wizard | **no** — recover keys, decrypt, write the zip, leave nothing | +| Live tracking of your own account's tags | yes, and say so before doing it | + +Bundling the join into sign-in would give every user who wanted a single export a peer in their +keychain trust circle and an escrow record that **no Apple interface displays**, acquired without +being asked. The residue rules in +[findmy-export/README.md](./findmy-export/README.md#account-side-residue-is-a-first-class-design-problem) +exist for precisely this. + +### What joining costs, in words a user should see + +- a **peer** in the keychain trust circle — the circle that protects every password on the account +- an **escrow record**, sealed under a passcode the user chooses, which is what recovers this + client later +- both permanent; only the record can be deleted afterwards + +The second passcode is only ever needed for this flow. **An export never asks for it**, because +nothing is created that would need protecting. + +--- + +## Edge cases worth handling deliberately + +**An account with no circle at all.** An Apple ID that has never had an Apple device has nothing +to recover from — no bottles, no peers, no keys. The only way in is `establish`, which creates a +circle, and this project never calls it (see +[findmy-export/03-keychain-trust.md](./findmy-export/03-keychain-trust.md) §6.7). It does not +matter in practice: such an account cannot have paired a tag, so it owns nothing to export. Detect +it and say so plainly rather than failing at a lower layer. + +**A recipient signing in.** Their account has tags of nobody's; the keychain path should never run +for them at all. Importing a zip must not require a passcode, a circle, or anything from Stage 3. + +**A user who exports and later wants live tracking.** They will be asked for the device passcode a +second time, because the first flow deliberately kept nothing. That is the correct trade and +should be explained at the point it happens, not treated as a bug. + +--- + +## Open decisions + +- **`sourceUser` in `OPENTAGVIEWER.yml`** names a person and travels inside a bundle that is meant + to be shared. Keep, drop, or make optional. +- **Tag selection in the app.** The wizard already does this — `_create_zip` takes a list of + beacon ids — so only the Android side is open, and it is a UI question rather than a design one. + The platform convention fits: long-press to enter a selection mode, tap to add, act on the + selection, as the mail and photo pickers do. **The default must be an explicit list, never + everything**, because sharing one tag with a friend is the common case and sharing all of them + by accident is the failure worth designing against. + + **[observed] This is sharper than it first looked.** The owner's own iPhone, iPad and Mac are in + the same zone as the tags, they carry private keys, and they are locatable — so they pass the + export rule and can end up in a bundle. Finding your own phone from your own Android phone is a + feature; handing a friend the ability to follow it is not. **The selection must therefore show + which entries are devices**, using the discriminators in + [findmy-export/06-output.md](./findmy-export/06-output.md) §3, so the choice is informed rather + than inferred from a name. diff --git a/docs/findmy-export/01-authentication.md b/docs/findmy-export/01-authentication.md new file mode 100644 index 00000000..f440149e --- /dev/null +++ b/docs/findmy-export/01-authentication.md @@ -0,0 +1,969 @@ +# Stage 1 — Apple ID authentication (Grand Slam / GSA) + +Specification of the authentication exchange between a client and Apple's Grand Slam +authentication service, from an Apple ID and password to a set of session tokens that later +stages consume. + +Read [README.md](./README.md) first. In particular: **implement from this document alone.** + +> **Verified against a live account on 2026-08-13.** A probe built from this document +> authenticated end to end: SRP handshake, server proof verified, session payload decrypted, +> trusted-device two-factor, 25 service tokens collected. Values from that run are marked +> **[observed]**. Everything unmarked is still derived by reading rather than running — the SMS +> path in particular was not exercised. +> +> That the server proof verified is the strongest single result: `M2` only matches if the +> client's entire SRP computation agrees with Apple's, so all three deviations in §4.6 are +> confirmed rather than merely plausible. +> +> The run also settled the question that shapes the whole feature: **signing in registers a +> device on the account.** See §13. + +--- + +## 1. What this stage produces + +On success the client holds: + +| Value | Origin | Used by | +| --- | --- | --- | +| `adsid` | decrypted session payload | every later stage; Apple's numeric account identifier (the "DSID") | +| `GsIdmsToken` | decrypted session payload | building the two-factor identity token | +| `acname` | decrypted session payload | the account's canonical Apple ID; may differ from what the user typed | +| **PET** (`com.apple.gs.idms.pet`) | response header, or the session payload | Stage 2. Short-lived — assume 5 minutes. | +| Service tokens, keyed by reverse-DNS service name | response headers, or the session payload | later stages; `com.apple.gs.idms.hb` is needed by several GSA endpoints | +| The password's SHA-256 digest | computed locally | silently re-authenticating when a token expires | + +The PET ("password-equivalent token") is the handover to Stage 2: it is presented as the +password in an HTTP Basic credential against the MobileMe delegate endpoint. Because it expires +in minutes, Stage 1 and Stage 2 must run back to back, or Stage 1 must be repeatable without +user interaction (see §11). + +## 2. Prerequisites + +### 2.1 Anisette + +Every request in this stage carries **Anisette** headers: a one-time password and machine +identifier produced by Apple's ADI libraries from provisioned, per-installation state. Without +valid Anisette, GSA rejects the request regardless of whether the password is correct. + +This document treats Anisette as a solved dependency and specifies only the header names it +must supply. OpenTagViewer generates these on-device from Apple's own Android ADI libraries — +see `app/src/main/java/.../anisette/` — falling back to a remote Anisette server when it +cannot. + +The Anisette provider must supply: + +| Header | Meaning | +| --- | --- | +| `X-Apple-I-MD` | The one-time password. Changes per request. | +| `X-Apple-I-MD-M` | The machine identifier from the provisioned ADI state. | +| `X-Apple-I-MD-RINFO` | Routing information, a decimal integer as a string. | +| `X-Apple-I-MD-LU` | The local user identifier — a stable hex string. | +| `X-Mme-Device-Id` | A stable UUID identifying this installation. | +| `X-Apple-I-Client-Time` | Current UTC time, ISO 8601, whole seconds, `Z` suffix — e.g. `2026-08-13T09:41:07Z`. | +| `X-Apple-I-TimeZone` | `UTC`. | +| `X-Apple-Locale` | `en_US`. | + +Headers are selected **by exact name**, and anything unrecognised is silently dropped rather +than rejected. So a provider that omits one — or spells it differently — does not fail at the +point of the mistake; authentication fails later, for a reason that looks unrelated. Verify the +provider's output against the six names above before debugging anything else. (Observed: +macOS's own Anisette omits `X-Apple-I-MD-RINFO` and `X-Apple-I-Client-Time` entirely.) + +Two hard requirements: + +- **The identity must be stable across logins.** `X-Apple-I-MD-LU` and `X-Mme-Device-Id` must be + derived from the same persisted state that provisioned ADI, and must not be regenerated. An + identity that changes per login makes every session look like a new machine, which is the + pattern two-factor authentication exists to detect. +- **A session is bound to the identity that established it.** Switching between a local Anisette + provider and a remote server, or between two different remote servers, presents a different + machine to Apple and invalidates the session. This is inherent to how Apple binds sessions. + +Anisette headers may be cached and reused for a short window; **60 seconds** is the interval the +reference implementations use before regenerating. The one-time password is genuinely one-time +in the sense that it is time-derived, not that Apple rejects a repeat within its validity window. + +### 2.2 Client identity + +The client presents itself as a device. The values are invented and are not validated against +real hardware, but they must be **internally consistent** and **stable across logins**. + +> **The iPhone below is a worked example, not a requirement.** A client presenting as a Mac works +> equally well, including against the `iosbuddy` endpoint of [Stage 2](./02-mobileme-delegate.md) +> despite the name — FindMy.py has long presented as a MacBook Pro and authenticates fine. What +> matters is that one client is consistent with itself. An implementation that already has an +> identity should **keep it**: introducing a second invalidates every existing session, and +> Stage 1 §13 explains what a changed identity costs. + +A worked example, using an iPhone 14 Pro on iOS 17.4: + +| Field | Example value | +| --- | --- | +| Hardware model | `iPhone15,2` | +| OS version | `17.4` | +| Build | `21E219` | +| CFNetwork version | `1494.0.7` | +| Darwin version | `23.4.0` | +| Device serial | **User-visible — choose it as a label.** Uppercase alphanumeric, 10–12 characters. `0PENTAGVIEWR` is confirmed accepted and displayed. See §13. | +| Device name | any user-visible string | +| Device UUID | a random v4 UUID, **uppercase**, generated once and persisted | +| UDID | 32 uppercase hex characters, generated once and persisted. **May be the same value as the device UUID**, with its hyphens removed — a UUID's hex form is exactly 32 uppercase hex characters. Nothing observed requires them to differ, and one persisted value is one fewer thing that can drift out of step with itself. | + +The OS version, build, CFNetwork version and Darwin version must correspond to one another — +they describe one real iOS release, and Apple's own clients never disagree with themselves. +Changing the hardware model means changing all of them together. + +From those, three composite strings are built: + +``` +mmeClientInfo = +mmeClientInfoAkd = +akdUserAgent = akd/1.0 CFNetwork/1494.0.7 Darwin/23.4.0 +``` + +The general shape is ` `, where the trailing bundle +identifies which Apple daemon or app is speaking. The client also uses a fixed browser +user-agent string for the two-factor web endpoints: + +``` +browserUserAgent = Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) +``` + +That it names macOS while everything else names iOS is not a mistake to correct — the +two-factor endpoints are web endpoints and are addressed as a browser would address them. + +The client also presents an application identity. Throughout this stage it claims to be +Messages: + +``` +akContextType = imessage +clientAppName = Messages +clientBundleId = com.apple.MobileSMS +``` + +Finally, a **hardware headers** map exists for values a genuine Apple device would send +(`X-Apple-I-SRL-NO` and similar). For an invented identity it is **empty**, and every header +set below that mentions it contributes nothing. + +### 2.3 TLS + +`gsa.apple.com` presents a certificate chaining to Apple's original **Apple Root CA**, which is +not in Android's system trust store. A plain HTTPS request fails the handshake: + +``` +Unacceptable certificate: CN=Apple Root CA, O=Apple Inc., C=US +``` + +**Do not disable certificate verification.** Add that single root, for those Apple hosts only, +alongside the system trust store — a strictly narrower trust decision than the default, not a +looser one. The certificate is published at ; its SHA-256 +fingerprint is: + +``` +B0:B1:73:0E:CB:C7:FF:45:05:14:2C:49:F1:29:5E:6E:DA:6B:CA:ED:7E:2C:68:C5:BE:91:B5:A1:10:01:F0:24 +``` + +OpenTagViewer already carries this root in its network security configuration for the Anisette +provisioning work. Reuse that rather than adding a second copy. + +### 2.4 HTTP client + +- **Cookies must persist across requests for the whole stage.** The two-factor endpoints depend + on cookies set during the SRP exchange. A stateless client fails partway through with errors + that do not mention cookies. +- Apple's servers are sensitive to header casing. Send headers in the conventional + `X-Apple-I-MD` title case rather than lowercasing them, and do not rely on an HTTP/2 client + that normalises them. + +--- + +## 3. Message envelope + +The GSA service endpoint is: + +``` +POST https://gsa.apple.com/grandslam/GsService2 +``` + +Requests and responses are **XML property lists**. Every request body is a dictionary with +exactly two keys: + +``` +Header → dictionary: { "Version": "1.0.1" } +Request → dictionary: the operation-specific payload +``` + +Every response body is a dictionary whose `Response` key holds the reply dictionary. + +### 3.1 Errors + +Inside the reply, a `Status` dictionary carries the outcome. If `Status` is absent, the reply +dictionary itself carries these keys: + +| Key | Type | Meaning | +| --- | --- | --- | +| `ec` | integer | Error code. `0` means success. | +| `em` | string | Human-readable message. | + +Check `ec` after **every** exchange, before reading any other field. A non-zero `ec` with its +`em` is the only diagnostic Apple gives, so surface both verbatim in logs and in whatever the +user sees; the messages are specific enough to act on and guessing at them wastes hours. + +Content type for these requests is `text/x-xml-plist`, and `Accept` is `*/*`. + +--- + +## 4. The SRP handshake + +Authentication is **SRP-6a** — a password-authenticated key exchange in which the password +never crosses the network and both sides end up holding a shared session key. It runs as two +round trips against the endpoint in §3. + +Apple's variant deviates from RFC 5054 in three specific ways. Each is called out below, and +each will produce an indistinguishable "authentication failed" if implemented to the letter of +the RFC instead. **§4.6 lists them together; read it before writing any of this.** + +### 4.1 Parameters + +- Group: the **2048-bit group from RFC 5054, appendix A** — modulus `N`, generator `g = 2`. Both + are public constants; take them from the RFC. +- Hash: **SHA-256** throughout. +- Client secret `a`: **32 random bytes** from a cryptographically secure source. +- `A = g^a mod N`. + +### 4.2 Round trip one — `init` + +Request payload: + +| Key | Type | Value | +| --- | --- | --- | +| `o` | string | `init` | +| `u` | string | The Apple ID as the user typed it | +| `A2k` | data | `A`, big-endian | +| `ps` | array of string | `["s2k", "s2k_fo"]` — the password-derivation schemes offered | +| `cpd` | dictionary | See §5 | + +Reply: + +| Key | Type | Meaning | +| --- | --- | --- | +| `s` | data | Salt. **[observed]** 16 bytes. | +| `B` | data | Server public value, big-endian. **[observed]** 256 bytes. | +| `i` | integer | PBKDF2 iteration count. **[observed]** 20082 — note it is neither round nor a power of two. Do not hardcode it; it is Apple's to choose and may vary by account or over time. | +| `c` | string | An opaque continuation value, echoed in round trip two | +| `sp` | string | The scheme the server chose: `s2k` or `s2k_fo`. **If absent, assume `s2k`.** **[observed]** present, and `s2k`. | + +### 4.3 Deriving the SRP password + +The raw password is never used directly. Let `P₀ = SHA-256(password_utf8)` — this digest is +also what gets cached in §11. + +Then, depending on `sp`: + +| `sp` | Input to PBKDF2 | +| --- | --- | +| `s2k` | `P₀` — the 32 raw digest bytes | +| `s2k_fo` | The **lowercase hexadecimal ASCII encoding** of `P₀` — 64 bytes, not 32 | + +Then: + +``` +P = PBKDF2-HMAC-SHA256(input, salt = s, iterations = i, output length = 32) +``` + +`P` is what the SRP computation below calls the password. Note that the `s2k_fo` case is a +double transformation: hash the password, then hex-encode that hash *as text*, then stretch it. + +**[observed]** Apple selected `s2k`. The `s2k_fo` branch is therefore specified but unexercised, +and should be treated as the less trustworthy of the two — it is offered in `ps`, so Apple may +select it for some accounts, but nothing here has seen it happen. + +### 4.4 Client computation + +``` +x = SHA-256( s ‖ SHA-256( "" ‖ ":" ‖ P ) ) ← deviation 1, see §4.6 +u = SHA-256( A ‖ B ) ← deviation 2, see §4.6 +k = SHA-256( N ‖ PAD(g) ) +S = (B − k·g^x) ^ (a + u·x) mod N +K = SHA-256(S) +M1 = SHA-256( (SHA-256(N) ⊕ SHA-256(PAD(g))) ‖ SHA-256(I) ‖ s ‖ A ‖ B ‖ K ) +M2 = SHA-256( A ‖ M1 ‖ K ) +``` + +where `I` is the Apple ID as typed, `PAD(g)` is `g` left-padded with zero bytes to the byte +length of `N` (256 bytes), and `‖` is concatenation. + +`M1` and `M2` are standard SRP-6a. `K` is the session key that decrypts the payload in §6. + +Reject the exchange if `B mod N == 0`. + +Computing `B − k·g^x` in modular arithmetic can go negative. Add `N` before subtracting: +`((N + B) − (k·g^x mod N)) mod N`. + +### 4.5 Round trip two — `complete` + +Request payload: + +| Key | Type | Value | +| --- | --- | --- | +| `o` | string | `complete` | +| `u` | string | The same Apple ID string sent in `init` | +| `M1` | data | `M1` | +| `c` | string | The `c` value from the `init` reply, echoed unchanged | +| `cpd` | dictionary | See §5 — **the same request UUID as `init`** | + +Reply: + +| Key | Type | Meaning | +| --- | --- | --- | +| `M2` | data | Server proof. Compare against the locally computed `M2` and abort on mismatch. | +| `spd` | data | The encrypted session payload — see §6 | +| `Status` | dictionary | `ec`/`em`, plus `au` if a second factor is required — see §7 | + +Verifying `M2` is what authenticates Apple to the client. It is not optional: skipping it means +a party who never knew the password can complete the exchange. + +### 4.6 The three deviations, and the serialisation trap + +**Deviation 1 — the identity in `x` is empty.** RFC 5054 computes +`x = H(s ‖ H(I ‖ ":" ‖ P))` with `I` the username. Apple computes the inner hash with the +username **omitted**: `H("" ‖ ":" ‖ P)`, which is `SHA-256(":" ‖ P)` — the colon is still there. +Note the asymmetry: `M1` *does* use the real username, as `SHA-256(I)`. Getting this backwards +is the single most likely cause of a handshake that fails with correct credentials. + +**Deviation 2 — `u` is not padded.** RFC 5054 computes `u = H(PAD(A) ‖ PAD(B))`. Apple hashes +`A` and `B` at their natural lengths, with no padding. `k` and `M1` *do* pad `g`. So padding +applies to `g` and not to `A` or `B`, within the same handshake. + +**Deviation 3 — big integers are serialised minimally.** `A`, `B` and the `g` inside `k` and +`M1` are big-endian byte strings **with no leading zero bytes and no sign byte**. Two things +follow, and both are silent failures rather than errors: + +- Whenever `A` or `B` has a leading zero byte in its natural 256-byte form — roughly one time in + 256 — a fixed-width encoding produces a different `u` and the handshake fails. This is an + intermittent, unreproducible login failure if implemented wrongly, so it will not show up in + testing. +- Many big-integer libraries emit a leading `0x00` on export to keep the value positive. That + byte must be stripped. Equally, `g = 2` must be encoded as the single byte `0x02` before + padding, not as a wider machine word. + +**[observed]** On the verified run both `A` and `B` were the full 256 bytes, so the +leading-zero case did not arise. That is the expected outcome roughly 255 times in 256 and is +emphatically not evidence the trap is absent — it is the reason the trap is dangerous. A wrong +implementation passes this test almost every time. + +Round-trip your encoder against a known value before debugging anything else. + +--- + +## 5. The `cpd` dictionary + +`cpd` is a dictionary carried inside both SRP request payloads. It repeats some of the Anisette +headers as payload fields and adds client capability flags. + +Generate **one** uppercase v4 UUID per login attempt and use it for `X-Apple-I-Request-UUID` in +both round trips. A different UUID in `complete` breaks the correlation with `init`. + +Copy these straight from the Anisette headers, unchanged: + +``` +X-Apple-I-Client-Time X-Apple-I-MD X-Apple-I-MD-LU +X-Apple-I-MD-M X-Apple-I-MD-RINFO X-Mme-Device-Id +``` + +Add, as strings: + +| Key | Value | +| --- | --- | +| `X-Apple-I-Device-Configuration-Mode` | `0` | +| `X-Apple-I-Request-UUID` | the per-login UUID | +| `X-Apple-Requested-Partition` | `0` | +| `X-Apple-Security-Upgrade-Context` | `com.apple.authkit.generic` | +| `capp` | `Messages` | +| `cbid` | `com.apple.MobileSMS` | +| `cou` | `US` | +| `loc` | `en_US` | +| `svct` | `imessage` | + +Add, as booleans and integers — the plist types matter, a boolean sent as the string `"true"` +is not the same value: + +| Key | Type | Value | +| --- | --- | --- | +| `X-Apple-Offer-Security-Upgrade` | boolean | true | +| `at` | integer | 0 | +| `bootstrap` | boolean | true | +| `ckgen` | boolean | true | +| `fcd` | boolean | true | +| `icdrsDisabled` | boolean | false | +| `icscrec` | boolean | true | +| `pbe` | boolean | false | +| `prkgen` | boolean | true | +| `webAccessEnabled` | boolean | false | + +Finally, merge in the hardware headers map (§2.2), which is empty for an invented identity. + +A push token field `ptkn` also exists here, carrying an uppercase hex APNs token. It belongs to +clients that maintain a push connection. This flow does not, so **omit the key entirely** — +do not send it empty. + +--- + +## 6. Decrypting the session payload + +`spd` from the `complete` reply is AES-encrypted under keys derived from the SRP session key +`K`. Derive them by HMAC, using `K` as the HMAC key and two fixed ASCII strings as messages: + +``` +key = HMAC-SHA256( K, "extra data key:" ) → 32 bytes, the AES key +iv = HMAC-SHA256( K, "extra data iv:" )[0 .. 16] → first 16 bytes only +``` + +Both trailing colons are part of the strings. + +Decrypt `spd` with **AES-256-CBC** and **PKCS#7** padding. The plaintext is an XML property list +dictionary: + +| Key | Type | Meaning | +| --- | --- | --- | +| `adsid` | string | The account's numeric identifier (DSID) | +| `GsIdmsToken` | string | Paired with `adsid` to form the two-factor identity token (§7.1) | +| `acname` | string | The account's canonical Apple ID. **Store this** in preference to what the user typed. | +| `fn`, `ln` | string | Given and family name | +| `t` | dictionary | Service tokens — see §6.1 | + +The payload is decrypted even when a second factor is still required; `adsid` and `GsIdmsToken` +are available at that point and are needed to request the second factor. + +**[observed]** The dictionary held **49 keys** from 4096 bytes of ciphertext on a first login, +and **50 keys** from 18160 bytes on a subsequent one — the second being larger because the full +token set is inlined. **Its keys are not fixed**: `url` appeared only when a second factor was +pending, while `ck` and `prk` appeared only when one was not. So treat it as an open-ended +structure, read by name, and never assume a key is present. Beyond those above, several +matter to later stages and are worth knowing exist now: + +| Key | Why it matters later | +| --- | --- | +| `DsPrsId` | a second form of the account identifier, alongside `adsid` | +| `sk`, `c` | session key and cookie for the separate app-token exchange, which this flow does not use but which exists | +| `SOSNeeded`, `SOSCompatibilityOptInNeeded`, `hasSOSActiveDevice` | "Secure Object Sharing" — Apple's name for the iCloud Keychain circle. Directly relevant to Stage 3, and `hasSOSActiveDevice` looks like the signal for whether the account has anything to recover from. | +| `isPasscodeAuth`, `passcodeAuthEnabled` | relates to the device-passcode authentication Stage 3 needs | +| `hasRK` | presence of a recovery key | +| `passkeyEligible`, `passkeyPresent` | an alternative authentication route this document does not cover | +| `primaryEmail`, `countryCode`, `authmode`, `ut` | account attributes | + +### 6.1 The token dictionary + +Each key of `t` is a service name; each value is a dictionary: + +| Key | Meaning | +| --- | --- | +| `token` | The token string | +| `expiry` | Absolute expiry, **milliseconds since the Unix epoch** | +| `duration` | Lifetime in **seconds** from now | + +**[observed]** `expiry` and `duration` are **not** alternatives — the one token present before +two-factor carried both. Prefer `expiry`, since it is absolute and needs no clock arithmetic, +and fall back to `now + duration`. Do not assume the presence of one implies the absence of the +other. + +**[observed]** Which tokens arrive here depends on whether a second factor was needed: + +| Situation | `t` contained | +| --- | --- | +| Second factor required | **one** entry, `com.apple.gs.appleid.auth`. The rest arrive later via response headers (§8). | +| No second factor (known machine) | **all 25**, including the PET and the heartbeat token. No header parsing needed. | + +So both routes must be implemented: an implementation that only reads tokens from headers works +on first run and silently gets nothing afterwards, and one that only reads the payload fails the +other way round. + +**[observed]** In the no-second-factor case every one of the 25 entries carried *both* `expiry` +and `duration`, reinforcing that they are not alternatives. + +--- + +## 7. Two-factor authentication + +> **[observed] A second login from the same machine identity required no second factor at all.** +> `au` was absent, the login completed in zero extra rounds, and the full token set arrived in +> the decrypted payload rather than in headers. This is the behaviour §11 depends on for silent +> refresh, and it is the practical reward for the identity stability demanded in §2.1 — Apple +> remembers the machine, and stops challenging it. +> +> It also means **the two-factor path is the first-run path**. An implementation that only ever +> tests against an already-known identity will not exercise §7 at all. + +After a successful `complete`, inspect `Status.au`: + +| `au` | Meaning | Go to | +| --- | --- | --- | +| absent | Login is complete. | §9 | +| `trustedDeviceSecondaryAuth` | A code can be pushed to the user's trusted devices. | §7.2 | +| `secondaryAuth` | A code must be sent by SMS. | §7.3 | +| anything else | An unhandled requirement. Report the value verbatim. | — | + +Do not hard-fail on an unrecognised `au` before checking whether a PET was nonetheless issued; +some variants complete anyway. + +The endpoints in this section are **not** the plist service of §3. They are web endpoints, they +mostly speak JSON, and they carry a different header set (§7.1). + +### 7.1 Two-factor header set + +Take from the Anisette headers: + +``` +X-Apple-I-MD X-Apple-I-MD-M X-Apple-I-MD-LU X-Apple-I-MD-RINFO X-Mme-Device-Id +``` + +Add the identity token, which is what authenticates these calls: + +``` +X-Apple-Identity-Token = base64( adsid + ":" + GsIdmsToken ) +``` + +Add the following fixed headers. Note that `X-MMe-Client-Info` here is the **Messages** variant +from §2.2, not the `akd` variant used in §8, and the user-agent is the **browser** one. + +| Header | Value | +| --- | --- | +| `X-Apple-Client-App-Name` | `Messages` | +| `X-Apple-I-Client-Bundle-Id` | `com.apple.MobileSMS` | +| `X-MMe-Client-Info` | `mmeClientInfo` | +| `X-Apple-AK-Context-Type` | `imessage` | +| `User-Agent` | `browserUserAgent` | +| `Accept-Language` | `en-US,en;q=0.9` | +| `X-Apple-I-Locale` | `en_US` | +| `X-Apple-I-TimeZone` | `UTC` | +| `X-Apple-I-TimeZone-Offset` | `0` | +| `X-MMe-Country` | `US` | +| `X-Apple-Requested-Partition` | `0` | +| `X-Apple-I-Device-Configuration-Mode` | `0` | +| `X-Apple-I-DeviceUserMode` | `0` | +| `X-Apple-Security-Upgrade-Context` | `com.apple.authkit.generic` | +| `X-Apple-I-CDP-Status` | `false` | +| `X-Apple-I-CDP-Circle-Status` | `false` | +| `X-Apple-I-OT-Status` | `false` | +| `X-Apple-I-ICSCREC` | `true` | +| `X-Apple-I-PRK-Gen` | `true` | +| `Sec-Fetch-Site` | `same-origin` | +| `Sec-Fetch-Mode` | `cors` | +| `Sec-Fetch-Dest` | `empty` | +| `X-Apple-I-CFU-State` | base64 of an XML plist whose root is an empty `` | + +Then merge the hardware headers map, and set content negotiation per endpoint: + +- **Plist endpoints** (`/grandslam/GsService2/validate`): `Content-Type: text/x-xml-plist` and + `Accept: text/x-xml-plist`. +- **JSON endpoints** (everything under `/auth`): `Accept: application/json`, and + `Content-Type: application/json` where there is a body. + +For reference, the `X-Apple-I-CFU-State` value is the base64 of this exact document, trailing +newline included: + +```xml + + + + + +``` + +### 7.2 Trusted-device flow + +**Request a code:** + +``` +GET https://gsa.apple.com/auth/verify/trusteddevice +``` + +with the §7.1 headers. Any non-2xx is a failure; report the status code. On success Apple pushes +a code to the user's trusted devices and the client prompts for it. + +**Submit the code:** + +``` +GET https://gsa.apple.com/grandslam/GsService2/validate +``` + +with the §7.1 headers plus a header carrying the code: + +``` +security-code: 123456 +``` + +The reply is a **property list**, not JSON. Check `ec`/`em` per §3.1. Tokens arrive in the +response headers — see §8. + +### 7.3 SMS flow + +**Discover the numbers.** SMS delivery needs the identifier of a specific trusted phone number, +obtained from: + +``` +GET https://gsa.apple.com/auth +``` + +with the §7.1 headers and `Accept: application/json`. The JSON reply carries +`trustedPhoneNumbers`, an array of: + +| Field | Meaning | +| --- | --- | +| `id` | Numeric identifier used to request delivery | +| `numberWithDialCode` | Display form, already partly masked by Apple | +| `lastTwoDigits` | Display aid | +| `pushMode` | Delivery mode | + +Two response statuses have specific meanings: + +- **`201`** — Apple has *already sent* a code to the first trusted number as a side effect of + this request. Go straight to submitting the code; requesting delivery again sends a second + code and invalidates the first. +- **`403`** — the two-factor configuration cannot be read. The reference implementations + attribute this to **Advanced Data Protection** being enabled on the account. That attribution + is unverified here, but it is the message to surface, because if it is right the user's only + remedy is to turn ADP off. Do not present it as a network error. + +> **Do not treat "no trusted phone numbers" as fatal on the trusted-device path.** This endpoint +> is the only source of phone numbers, so it is tempting to call it for both flows and abort +> when the list is empty — which breaks accounts that have trusted devices but no registered +> phone number. Only the SMS path needs a number. This is a known defect in at least one +> implementation of this protocol; do not reproduce it. + +Offer SMS whenever trusted numbers exist, even when `au` was `trustedDeviceSecondaryAuth`. A +user without a second Apple device to hand needs it, and Apple accepts it. + +**Request delivery:** + +``` +PUT https://gsa.apple.com/auth/verify/phone +``` + +§7.1 headers, `Accept: application/json`, JSON body: + +```json +{ "phoneNumber": { "id": 1 }, "mode": "sms" } +``` + +**Submit the code:** + +``` +POST https://gsa.apple.com/auth/verify/phone/securitycode +``` + +§7.1 headers, `Accept: application/json`, and the **same body as the delivery request** with the +code added: + +```json +{ "phoneNumber": { "id": 1 }, "mode": "sms", "securityCode": { "code": "123456" } } +``` + +The `phoneNumber` and `mode` fields must match what was sent to request delivery. Anything other +than HTTP 200 means the code was rejected. + +--- + +## 8. Harvesting tokens from response headers + +Both code-submission endpoints return tokens in **response headers**, not in the body. Three +header names matter, and a header may appear **more than once** — collect all values. + +| Header | Carries | +| --- | --- | +| `X-Apple-GS-Token` | A service token | +| `X-Apple-HB-Token` | The heartbeat token, service `com.apple.gs.idms.hb` | +| `X-Apple-PE-Token` | The PET | + +Each value is base64. Decode to ASCII and split on `:`. Field 0 is the service name, field 1 is +the token; what follows is inconsistent: + +| Source | Layout | Fields | +| --- | --- | --- | +| `validate` — PET | `identifier:token` — no lifetime at all | **2** [observed] | +| `validate` — heartbeat | `service:token` — **no lifetime either** | **2** [observed] | +| `validate` — service tokens | `service:token:durationSeconds` | **3** [observed] | +| `securitycode` — all | `service:token:durationSeconds:absoluteExpiry` | 4, unverified | + +**[observed] The heartbeat token carries no lifetime.** A reference implementation documents it +as `service:token:absoluteExpiry`; the live run returned two fields, not three, so it falls to +the one-year default like everything else. Trust the field count you actually receive rather +than any documented layout, this one included. + +**[observed] The first field of the PET is not a service name.** Service tokens key themselves +by their own first field, but the PET's is an account identifier — so it must be filed under +`com.apple.gs.idms.pet` by the code, not by what the header says. + +**[observed] `X-Apple-GS-Token` appeared 23 times in one response**, alongside one +`X-Apple-HB-Token` and one `X-Apple-PE-Token` — 25 tokens in total. An HTTP client that returns +only the first value for a header name will silently discard 22 of them. + +So the third field means different things depending on which endpoint answered. Rather than +special-casing the endpoint, disambiguate by magnitude — an absolute expiry in milliseconds +since 1970 is a far larger number than any plausible lifetime in seconds: + +``` +if value > 40 years expressed in milliseconds: + treat as milliseconds since the Unix epoch +else: treat as a lifetime in seconds, counted from now +``` + +For the `securitycode` layout, prefer field 3 and fall back to field 2. Defaults when no +lifetime is present: **300 seconds for the PET**, and one year (31,536,000 seconds) for +everything else. The one-year default is a guess made by the reference implementations, not +something Apple states — treat any token as possibly-expired and be ready to re-authenticate. + +### 8.1 After the second factor + +If the code-submission response carried an `X-Apple-PE-Token`, the login is complete. + +If it did not, the second factor succeeded but no session was issued. **Repeat the entire SRP +handshake from §4** with the same credentials. It will now complete without an `au` in `Status`, +and the token dictionary will arrive in the decrypted payload (§6.1) instead of in headers. This +second pass needs no user interaction — the cached password digest is enough. + +--- + +## 9. State machine + +``` + ┌──────────────┐ + │ SRP §4 │ + └──────┬───────┘ + │ Status.au? + ┌───────────────┼────────────────┐ + absent trustedDevice secondaryAuth + │ │ │ + │ ┌──────┴───────┐ ┌────┴─────┐ + │ │ offer choice: trusted device │ + │ │ or any trusted phone number │ + │ └──────┬───────┘ └────┬─────┘ + │ §7.2 request §7.3 request + │ └────────┬────────┘ + │ prompt for code + │ │ + │ submit code + │ │ + │ PET in headers? + │ ┌────┴────┐ + │ yes no + │ │ │ + └───────────────────┴──────────┴──→ repeat §4 once + │ + ┌───▼────┐ + │ Done │ + └────────┘ +``` + +Two rules that fall out of it: + +- **Bound the loop.** A repeat of §4 that again asks for a second factor is a failure, not a + reason to prompt again. Cap re-entry at one pass. +- **Present both second-factor options at once**, rather than following `au` blindly. The user + picks; the client already knows which endpoints each choice needs. + +--- + +## 10. Endpoint header sets, at a glance + +Three distinct header sets appear in this stage. Mixing them up is a plausible failure and a +tedious one to diagnose, so they are tabulated together. + +| | §3 plist service | §7.1 two-factor web | Anisette provisioning | +| --- | --- | --- | --- | +| Anisette headers copied | `MD`, `MD-M`, `MD-LU`, `MD-RINFO`, `X-Mme-Device-Id` | same five | n/a | +| `X-MMe-Client-Info` | `mmeClientInfoAkd` | `mmeClientInfo` | `mmeClientInfo` | +| `User-Agent` | `akdUserAgent` | `browserUserAgent` | `akdUserAgent` | +| `X-Apple-Client-App-Name` | `Messages` | `Messages` | `akd` | +| `X-Apple-I-Client-Bundle-Id` | ✓ | ✓ | — | +| `X-Apple-AK-Context-Type` | ✓ | ✓ | — | +| `Accept-Language` | `en-US,en;q=0.9` | `en-US,en;q=0.9` | `en-US,en;q=0.9` | +| `Sec-Fetch-*` | — | ✓ | — | +| `X-Apple-Identity-Token` | — | ✓ | — | +| Content type | `text/x-xml-plist` | per endpoint | — | + +The full §3 header set is: the five Anisette headers above, plus `X-Apple-AK-Context-Type: +imessage`, `X-Apple-Client-App-Name: Messages`, `X-Apple-I-Client-Bundle-Id: +com.apple.MobileSMS`, `X-MMe-Client-Info: mmeClientInfoAkd`, `Accept-Language: en-US,en;q=0.9`, +`User-Agent: akdUserAgent`, `Content-Type: text/x-xml-plist`, `Accept: */*`. + +--- + +## 11. Persistence, refresh and storage + +To avoid a full interactive login on every run, persist: + +| Item | Why | +| --- | --- | +| `acname` | the canonical Apple ID | +| The decrypted session payload | `adsid` and `GsIdmsToken` are needed by later stages | +| Every token, with its absolute expiry | so an unexpired token can be reused | +| **SHA-256 of the password** | so §4 can be replayed without prompting | + +Include a schema version and refuse to load anything else, so a format change fails loudly +rather than misreading old data. + +**Refreshing.** When a needed token has expired, re-run §4 with the stored Apple ID and stored +password digest. On an account with two-factor enabled this normally completes without a second +factor, because Apple has seen this machine identity before. That is exactly why §2.1 insists +the identity be stable. + +### 11.1 Security + +> **The stored password digest is password-equivalent.** SRP consumes `SHA-256(password)`, not +> the password, so anything holding that digest can authenticate as the user indefinitely. It is +> not a "hashed password" in the sense that protects a leaked database — there is no salt and no +> stretching before storage, and possession is sufficient. + +On Android this means: + +- Store it, the session payload and the tokens under hardware-backed encryption. Never in plain + `SharedPreferences` and never in a file the app's own export or backup paths can reach. +- Never log it, and never log the PET, the service tokens, `GsIdmsToken`, or `M1`/`K`. Truncated + is still too much: these are secrets, not identifiers. +- The raw password itself must never be written to disk in any form. +- Offer an explicit sign-out that deletes all of it. + +Apply the same care to Anisette provisioning state, which identifies the machine to Apple. + +--- + +## 12. Open questions + +Things this specification cannot answer without running against a real account. Each needs +confirming on first implementation, and the answer belongs back in this document. + +**Answered by the runs of 2026-08-13:** + +1. ~~Which scheme does Apple select?~~ **`s2k`.** The `s2k_fo` branch remains specified but + unexercised — Apple could still select it for another account. +2. ~~Does the invented client identity pass?~~ **Yes**, and it is reflected straight back to the + user in their device list. See §13. +3. ~~Does signing in register a device?~~ **Yes.** This was the most consequential unknown, and + the answer is the unwelcome one. See §13. + +**Still open:** + +4. **Is the 403-means-Advanced-Data-Protection attribution correct?** The run returned 200 from + `GET /auth`, so the 403 path never arose. It comes from another implementation's error text, + not from Apple. If wrong, the app will tell users to disable a feature that is not the + problem. +5. **Does the SMS path work as specified?** Untested end to end. The account had one trusted + number and the trusted-device path was taken instead. Both the request and the + code-submission bodies, and the four-field token layout in §8, are unverified. +6. **How does an account with no second factor at all behave?** The `au`-absent path is + specified but is the least likely to have been exercised by anyone. +7. **Do the two code-submission endpoints return different token sets?** `validate` returned 25 + tokens; what `securitycode` returns is unknown. Stage 2 needs the PET from either. +8. **What does GSA do on repeated sign-ins from the same synthetic machine, over time?** + Partially answered: several sign-ins across a day were accepted, and after the first the + machine was no longer challenged for a second factor. Nothing is known about rate limiting or + flagging over weeks, which is the timescale that would matter. +9. **Does the PBKDF2 iteration count vary?** Observed twice, both `20082`, on **one account**. + Whether it differs per account or drifts over time is unknown — which is the argument for + never hardcoding it. +10. **Is there a limit on how many devices one account may register?** Not encountered. The + account examined carried a dozen escrow records and several registrations without complaint, + but no limit was searched for and hitting one would present as an unexplained failure. + +--- + +## 13. Signing in registers a device [observed] + +Completing this stage adds an entry to the account's device list, visible at appleid.apple.com +and in iOS and macOS Settings. **No further call is required** — there is a separate endpoint by +which a client announces itself, and the verified run never touched it. Authentication alone was +enough. + +The entry is synthesised entirely from the client identity of §2.2: + +| Shown to the user | Comes from | On the verified run | +| --- | --- | --- | +| Model | the claimed hardware model | `iPhone15,2` → **"iPhone 14 Pro"** | +| Version | the claimed OS version | `17.4` → **"iOS 17.4"** | +| Name | **nothing this stage sends** | defaulted to a bare **"iPhone"** | + +So Apple resolves the model code into a marketing name and presents the result as an ordinary +device. The registered entry was, to the eye, indistinguishable from a real iPhone 14 Pro. + +**There is exactly one distinguishing marker**, on the device's detail page: + +> This device cannot be used to receive Apple Account verification codes. + +It is one line, on a subpage, and a user with a real iPhone 14 Pro must open both entries and +compare to know which is which. As a way for a person to find the right entry, that is thin. + +### Three display fields, three independent levers [observed] + +Comparing our entry against a device registered by the older macOS-VM export route — itself +synthetic, so the difference is purely what each flow sent: + +| Shown to the user | VM-registered entry | This stage alone | Set by | +| --- | --- | --- | --- | +| Name | `sb's iMac Pro` | bare `iPhone` | the device-name field of the announce call | +| Serial Number | `C02XXXXXXXXX` *(redacted)* | **absent entirely** | `X-Apple-I-SRL-NO`, among the hardware headers | +| Trusted for verification codes | **yes** | **no** | almost certainly the push token | + +**The serial is displayed when it is sent, and this stage can send it.** [observed] Supplying +`X-Apple-I-SRL-NO` among the hardware headers put the value straight onto the device's page as +its Serial Number, with no announce call and no other change. A serial is an arbitrary uppercase +alphanumeric string of roughly 10–12 characters, so it is the one identity field that can be +made to read as a label rather than as hardware. The verified run registered as: + +``` +iPhone +iPhone 14 Pro + Model iPhone 14 Pro + Version iOS 17.4 + Serial Number 0PENTAGVIEWR <- ours, and legible +This device cannot be used to receive Apple Account verification codes. +``` + +**So send one, and make it self-describing.** It costs a single header, it is the only field +this stage controls that the user actually sees, and without it Apple omits the row entirely — +leaving an entry with nothing to tell it apart from real hardware. The device page also carries +a *Remove from Account* button next to the text "If you do not recognise this device", so a +legible serial is the difference between a user confidently removing the right entry and not +daring to touch either. + +Beware the spelling: `X-Apple-I-SRL-NO` is the form the hardware-header map and remote Anisette +servers use, while macOS's own Anisette emits `X-Apple-SRL-NO`, without the `I`. The `-I-` form +is the one confirmed to work here. Note also that a serial arriving *from* an Anisette provider +is filtered out before the request is built (§10) — it must be supplied as a hardware header. + +### Do not make the device trusted + +The synthetic device is *not* trusted for verification codes, and the difference from the VM +entry points at the push token: this flow maintains no push connection and omits the field +(§5), while the VM had one. + +**Treat that as a boundary to hold rather than a gap to close.** A trusted device can receive +Apple Account verification codes — so an installation that became trusted would make itself a +second factor for the user's Apple ID. For an application whose purpose is reading accessory +locations, that is a large and unnecessary expansion of what a compromise of it would mean. +Implementing push in order to look more like a real device would buy legitimacy at exactly the +wrong price. + +### What follows for the implementation + +**The name is the problem, and this stage cannot set it.** The name field is carried by the +announce-yourself call, not by anything here. So an implementation that does only Stage 1 gets +an anonymous entry named after the hardware it claims. Two ways out, and the second is better: + +- **Choose the claimed model for recognisability.** Weak: every real model code resolves to a + name that looks like real hardware, and claiming an implausible one risks rejection. +- **Make the announce call once, with a deliberate name.** It carries a device-name field, and + is the mechanism by which an entry stops reading as "iPhone" and starts reading as + "OpenTagViewer". This needs specifying in Stage 2, and it is the recommended route. + +**One identity means one entry.** The registration keys on the machine identity, so a stable +identity re-registers nothing on later sign-ins. This is the concrete payoff of the stability +requirement in §2.1: it is not merely good hygiene, it is the difference between one entry and +one per run. + +**Removal is manual today.** The user can remove the device from appleid.apple.com or Settings. +A sign-out sequence exists in the protocol — a per-service sign-out event followed by a global +one — but whether it removes the account entry or merely signs services out is **unverified**, +and should not be claimed until it is. diff --git a/docs/findmy-export/02-mobileme-delegate.md b/docs/findmy-export/02-mobileme-delegate.md new file mode 100644 index 00000000..d9e72b07 --- /dev/null +++ b/docs/findmy-export/02-mobileme-delegate.md @@ -0,0 +1,522 @@ +# Stage 2 — MobileMe delegate + +Specification of the exchange that converts a Grand Slam session into iCloud service tokens. +This is the step between authenticating (Stage 1) and doing anything with iCloud. + +Read [README.md](./README.md) first. In particular: **implement from this document alone.** + +> **Verified against a live account on 2026-08-13**, and verified in the strongest available +> sense: the probe that exercised it was written **from this document**, not by calling any +> reference implementation. HTTP 200, top-level status 0, delegate status 0, service data +> returned, 11 tokens. So what follows is a description that someone has successfully built from. +> +> Values from that run are marked **[observed]**. One section — §6, the configuration +> dictionary — was **wrong** and has been corrected; the terms-of-service flow of §5.2 and the +> device lifecycle of §7 remain unexercised. + +--- + +## 1. What this stage produces + +A single request returns a set of named tokens and a service configuration dictionary. Three +tokens matter to this project: + +| Token | Needed by | +| --- | --- | +| `cloudKitToken` | Stage 4 — CloudKit record fetches | +| `mmeAuthToken` | Stage 4 — the general iCloud credential, paired with the numeric account id | +| `searchPartyToken` | the Find My "search party" service, which is what actually holds beacon data | + +**[observed] Eleven tokens were returned**, and all three above were among them. The full set is +in §5. + +What does **not** come back is any service configuration — see §6, and do not plan on getting +endpoints from this stage. + +## 2. Prerequisites + +From Stage 1: + +| Value | Where it came from | +| --- | --- | +| **PET** | `com.apple.gs.idms.pet`. Used as the HTTP Basic password. | +| `acname` | the account's canonical Apple ID. Used as the HTTP Basic username. | +| `adsid` | the account identifier, sent as a header | +| Anisette headers | the same provider as Stage 1, and it must be the *same machine identity* | + +> **The PET expires in about five minutes.** This is the binding constraint on how Stage 1 and +> Stage 2 fit together: either they run back to back, or Stage 1 must be replayable without user +> interaction from the cached password digest. There is no way to hold a PET across a session. + +A session is bound to the machine identity that established it (Stage 1 §2.1), so the Anisette +source must not change between the two stages. + +## 3. The request + +``` +POST https://setup.icloud.com/setup/iosbuddy/loginDelegates +``` + +**This URL encodes the client's claimed platform.** `iosbuddy` is the iOS Setup Assistant's +endpoint, and it matches the iPhone identity of Stage 1 §2.2. A client presenting as a Mac uses +a different path. Claiming iOS in the identity and macOS in the URL is the kind of mismatch that +produces an unhelpful error, so keep them consistent. + +### 3.1 Authentication + +HTTP Basic, with: + +- **username** — the Apple ID (`acname` from Stage 1, not necessarily what the user typed) +- **password** — the **PET** + +The PET is why this works without re-entering a password, and why it must be fresh. + +### 3.2 Headers + +Send **all** the Anisette headers — unlike Stage 1's GSA requests, this endpoint is not given a +filtered subset. Then add: + +| Header | Value | +| --- | --- | +| `X-Apple-ADSID` | `adsid` | +| `X-Mme-Client-Info` | `mmeClientInfoAkd` — the `com.apple.akd/1.0` variant from Stage 1 §2.2 | +| `User-Agent` | `com.apple.iCloudHelper/282 CFNetwork/ Darwin/` | +| `Accept-Encoding` | `gzip` | + +The user-agent follows the same ` CFNetwork/ Darwin/` shape as elsewhere, with the +item naming the Apple component being impersonated — here the iCloud helper rather than `akd`. +Its version must match the CFNetwork and Darwin versions of the claimed OS release. + +Two conditional headers: + +- `X-Mme-Nas-Qualify` — base64 **validation data**, an attestation blob a genuine Apple device + produces. This flow has none and **must omit the header entirely** rather than send it empty. + See §3.4. +- `Cookie` — only when re-sending after accepting terms of service. See §5.2. + +### 3.3 Body + +An XML property list. **There are two forms, and which one to send depends on whether you have +validation data.** With none — which is this project's case — send the legacy form, whose keys +are hyphenated: + +| Key | Type | Value | +| --- | --- | --- | +| `apple-id` | string | the Apple ID, same as the Basic username | +| `client-id` | string | a fresh v4 UUID, **lowercase**, per request | +| `password` | string | the **PET**, repeated here as well as in the Basic credential | +| `delegates` | dictionary | which services to request — see below | + +The newer form, for clients that do have validation data, uses camelCase `delegates`, +`protocolVersion` (`"1.0"`) and a `userInfo` sub-dictionary containing hyphenated `client-id` +(**uppercase** UUID), `language` (`en-US`) and `timezone`. It is documented here only so that an +implementer recognises it; **this project sends the legacy form.** + +`delegates` is a dictionary keyed by service identifier: + +| Key | Value | Wanted here | +| --- | --- | --- | +| `com.apple.mobileme` | empty dictionary | **yes** | +| `com.apple.private.ids` | `{ "protocol-version": "4" }` | no — iMessage identity, irrelevant to this project | + +Request only `com.apple.mobileme`. Asking for delegates you do not need means more of the +account's surface touched for no benefit. + +### 3.4 On validation data + +Validation data is an attestation blob that a real Apple device generates to prove it is one. +Producing it is a separate reverse-engineering problem, and this flow does not attempt it. + +**The endpoint accepts requests without it**, via the legacy request form — this is load-bearing +for the whole project, and is the reason the two body shapes exist. An implementation must +therefore never send an empty `X-Mme-Nas-Qualify` or an empty validation field; absence and +emptiness are not the same to this endpoint. + +## 4. The response + +An XML property list dictionary. Check it in this order — the first two checks are cheap and +produce far better diagnostics than the third. + +**1. `localizedError`.** If present, the request failed: + +| Value | Meaning | +| --- | --- | +| `UNAUTHORIZED` | the credential was rejected. Usually an expired PET — re-run Stage 1 rather than reporting an auth failure to the user. | +| anything else | report verbatim, along with the `description` key if present | + +**2. `status`.** An integer, and `0` means success. Anything else is a failure and the whole +response is the diagnostic. + +**`dsid`.** The response also carries a top-level `dsid`: the **numeric** account identifier. +This is the same value as `DsPrsId` in the Stage 1 session payload, and it is what +[Stage 4 §2.1](./04-cloudkit.md) needs as its HTTP Basic username. + +**Prefer this one.** Stage 1 §6 records that the session payload's keys vary between logins, so +depending on it for a value that is also available here is needless fragility. Taking `dsid` from +this response means Stage 4 needs nothing retained from Stage 1 at all. + +**3. `delegates`.** A dictionary keyed by the same service identifiers that were requested. Each +value is: + +| Key | Type | Meaning | +| --- | --- | --- | +| `service-data` | dictionary | the payload. **Absent when that delegate failed.** | +| `status` | integer | per-delegate status | +| `status-message` | string | per-delegate message | + +**Per-delegate status is separate from the top-level status.** The request as a whole can +succeed while an individual delegate fails, so a missing `service-data` is not a parse error — +it is that delegate's failure, and its `status` and `status-message` are the explanation. Report +them rather than dereferencing nothing. + +### 4.1 The MobileMe service data + +The `service-data` dictionary for `com.apple.mobileme` holds: + +| Key | Type | Contents | +| --- | --- | --- | +| `tokens` | dictionary | token name → token string. See §5. | +| everything else | — | the service configuration. See §6. | + +> **The configuration is the service-data dictionary itself, not a nested key inside it.** An +> earlier reading of this protocol expected the configuration under a `com.apple.mobileme` key +> *within* `service-data`, and that is wrong. Read the tokens from the `tokens` key, and treat +> the same dictionary as the configuration. Expect a nested `com.apple.mobileme` key not to +> exist, and do not fail if it does. + +## 5. Tokens + +`tokens` maps names to opaque strings. **[observed]** All eleven returned for an ordinary +account, requesting only the MobileMe delegate: + +| Name | Purpose | +| --- | --- | +| `mmeAuthToken` | the general iCloud credential. Presented as an HTTP Basic password with the **numeric** account id (`DsPrsId` from the Stage 1 session payload) as the username — note that is a *different* identifier from `adsid`. | +| `cloudKitToken` | CloudKit access, Stage 4 | +| `searchPartyToken` | **the one this project exists for** — the Find My search-party service, which holds beacon data | +| `mmeFMIPToken`, `mmeFMIPAppToken`, `mmeFMIPSiriToken` | Find My iPhone: service, application and Siri variants | +| `mmeFMFToken`, `mmeFMFAppToken`, `mmeFMFNotificationToken` | Find My Friends: service, application and notification variants | +| `keyTransparencyToken` | Apple's key-transparency service | +| `mapsToken` | Maps | + +Keep the whole map, look tokens up by name, and fail with the name that was missing rather than +an index error. The Find My families each have three variants whose distinctions are not +established here, so store all of them rather than guessing which one a later stage wants. + +### 5.1 Lifetime and refresh + +These tokens are long-lived relative to the PET — a reference implementation refreshes them +**weekly** — but the refresh path is a full repeat of this stage, which needs a fresh PET, which +needs Stage 1. So the dependency chain for any long-running use is: + +``` +cached password digest -> Stage 1 -> PET (~5 min) -> Stage 2 -> service tokens (~1 week) +``` + +Cache the delegate response with the time it was obtained. Do not try to infer expiry from the +tokens themselves; nothing in the response states one. + +### 5.2 Terms of service + +If the account has unaccepted iCloud terms, this stage fails until they are accepted. **This is +implemented in the app**, because sending the user to an Apple device to do it is a dead end for +the very users this feature exists for. + +#### How the failure arrives + +The delegate response carries **two independent error channels**, and the one that matters here is +the one a reader is least likely to check: + +| Where | Key | | +| --- | --- | --- | +| **Top level** of the response | `localizedError` | a short machine-ish string. `UNAUTHORIZED` is one known value | +| alongside it | `description` | the human-readable explanation | +| Top level | `status` | non-zero for a failure | +| Inside `delegates.com.apple.mobileme` | `status`, `status-message` | the per-delegate result | + +**Check `localizedError` before `status`.** A reader that only looks at the two `status` fields — +which is the obvious implementation, since those are what a success is defined by — will report a +generic failure for a response that said exactly what was wrong in a field it never read. + +> **[unestablished] Which `localizedError` value means "terms pending" is not known**, only that +> this is the channel it arrives on and that `UNAUTHORIZED` is a different value on the same +> channel. Do not branch on a guessed string. Surface `localizedError` and `description` verbatim, +> and offer the terms flow below as a remedy the user can choose — that behaves correctly whether +> or not the guess would have been right, and it records the real value the first time an account +> hits it. + +The one rule that makes this acceptable: **the terms are fetched and displayed, and the user +performs the acceptance.** The protocol returns the terms text, so there is no reason to accept +on their behalf, and doing so would be agreeing to a contract on someone else's account without +showing it to them. Fetch, render, let them read, let them decide. + +**Step 1 — fetch the terms.** + +``` +POST https://setup.icloud.com/setup/iosbuddy/ui/genericTermsUI +``` + +with the setup headers of §5.3, plus `Accept: application/x-buddyml`, +`Content-Type: text/plist` and `X-Apple-I-Appearance: 1`. Body is a plist: + +| Key | Value | +| --- | --- | +| `format` | `plist/buddyml` | +| `terms` | array of dictionaries, each `{ "name": "iCloud" }` | + +**Step 2 — read the response.** It is *not* a plist. It is **BuddyML**, an Apple XML dialect +used by Setup Assistant, and it must be parsed as XML: + +| Element | What to take | +| --- | --- | +| `clientInfo` | its `agreeUrl` attribute — the URL that records acceptance | +| `page` | its `id` attribute, identifying which terms document this is (`iCloud`) | +| `html` | character data — the terms themselves, as HTML, usually inside CDATA | + +Configure the XML reader to **treat CDATA as characters**, or the terms text arrives empty. Take +the HTML for the page whose id matches the terms requested, and render it. + +> ### Renew the PET before accepting, not before fetching only +> +> §5.3 authenticates these endpoints with the **PET**, and §5.1 puts a PET at about five minutes. +> This flow puts *a person reading a contract* in the middle of that. **The token will usually have +> expired by the time they agree.** +> +> The failure lands at the worst possible moment: after the user has read the terms and said yes. +> To them, they accepted a contract and were told to sign in again — and an app that loops them +> back through login has asked for agreement twice to record it once. +> +> **So re-authenticate immediately before each terms request**, the acceptance especially. It costs +> one GSA exchange in a flow that already contains a human, and the alternative is unrecoverable +> without asking them to agree a second time. +> +> **This is always possible here, and that is not a coincidence.** Renewing a PET means repeating +> the SRP exchange, which needs the password — and terms can only block a login that is *in +> progress*, so the password is necessarily in hand. A session restored from storage has already +> passed this stage. +> +> The exception is the token refresh of §5.1, which repeats this stage unattended a week later. If +> Apple has published new terms by then and no password is retained, the PET cannot be renewed and +> there is nothing to do but ask for a fresh sign-in. **Fall back to the token already held rather +> than failing at the renewal** — that lets the request fail on its own terms, which says more than +> a pre-emptive error would. + +**Step 3 — the user accepts.** Only after they have actually seen it: + +``` +POST +``` + +with the setup headers, `Content-Type: application/xml`, and no body. A non-success status means +acceptance failed and must be surfaced, not swallowed — an app that reports "accepted" when +Apple said otherwise will fail confusingly at the next step instead. + +**Step 4 — repeat this stage.** The delegate request of §3 should now succeed. + +A client presenting as a Mac uses a different route, re-sending the delegate request with a +`Cookie: termsAccepted=true` header. That path is irrelevant here — the identity of Stage 1 §2.2 +is an iPhone — and is noted only so it is not mistaken for the one to implement. + +### 5.3 Setup headers + +The terms endpoints take a different header set from §3.2. All the Anisette headers, plus: + +| Header | Value | +| --- | --- | +| `Authorization` | `Basic ` + base64 of the Apple ID + `:` + the **PET** | +| `User-Agent` | `iOS iPhone iPhone Setup Assistant` | +| `X-MMe-Client-Info` | ` ` | +| `X-MMe-Country` | `US` | +| `X-MMe-Language` | `en,en-US` | +| `Cookie` | `repairSteps=` | + +Note the bundle in `X-MMe-Client-Info` is Preferences here, not `akd` — these endpoints are +addressed as Settings would address them. + +## 6. The configuration dictionary + +> **Correction. [observed] There is no configuration.** An earlier draft of this section, taken +> from reading how other services use this protocol, said the service data carries per-dataclass +> endpoints keyed as `com.apple.Dataclass.`. For a request that asks only for +> `com.apple.mobileme`, it does not. The service data held `tokens` and exactly one other key: +> +> | Key | Value | +> | --- | --- | +> | `protocolVersion` | a version marker | +> +> No dataclass keys, no service URLs, and no nested `com.apple.mobileme` key either. + +Dataclass configuration does exist in this protocol — other clients read a quota URL and a +shared-streams URL from exactly such keys — so the likeliest explanation is that Apple returns +configuration only for dataclasses the client asks about or claims to support, and this request +asks for none. That is a hypothesis, not a finding. + +**What follows for Stage 4, now settled:** the CloudKit *credential* comes from here, but the +CloudKit *endpoint* does not — and it turns out not to be needed from here either. Opening the +container returns the service URLs directly. See +[04-cloudkit.md §2.3](./04-cloudkit.md). Nothing further is required of this stage. + +## 7. Naming the registered device + +[Stage 1 §13](./01-authentication.md#13-signing-in-registers-a-device-observed) established that +signing in registers a device on the account, and that this stage's predecessor cannot set its +name — the entry appears named after whatever hardware the client claims to be. A separate +endpoint carries a device name, and calling it once is what turns a bare `iPhone` into something +the user recognises. + +``` +POST https://gsas.apple.com/grandslam/GsService2/postdata +``` + +> **Note the host: `gsas.apple.com`, not `gsa.apple.com`.** One letter, a different host, and no +> useful error if you get it wrong. + +This is a Grand Slam endpoint, not an iCloud one, so it takes GSA-style headers rather than the +ones in §3.2: the Anisette subset, plus `Content-Type: text/x-xml-plist`, plus + +| Header | Value | +| --- | --- | +| `X-Apple-HB-Token` | base64 of `adsid` + `:` + the `com.apple.gs.idms.hb` token from Stage 1 | +| `X-Apple-I-UrlSwitch-Info` | base64 of `adsid` + `:postdata` | +| `X-MMe-Client-Info` | `mmeClientInfoAkd` | +| `User-Agent` | `akdUserAgent` | +| `X-Apple-I-Service-Type` | `itunesstore` | +| `X-Apple-I-CDP-Status`, `X-Apple-I-OT-Status`, `X-Apple-I-CK-Presence` | `true` | +| `X-Apple-AK-DataRecoveryService-Status` | `1` | +| `X-Apple-I-Device-Configuration-Mode`, `X-Apple-I-DeviceUserMode`, `X-Apple-Requested-Partition`, `X-Apple-I-TimeZone-Offset` | `0` | +| `x-apple-i-device-type` | `1` | + +Body is the usual `Header`/`Request` plist envelope. The `Request` dictionary announces the +client's state; the field that matters here is `dn`, the device name: + +| Key | Type | Value | +| --- | --- | --- | +| `dn` | string | **the device name** — what appears in the account's device list | +| `event` | string | `liveness` | +| `loc` | string | `en_US` | +| `services` | array | empty, for a client that provides none | +| `cdpStatus`, `circleStatus`, `otStatus`, `icscStatus`, `prkgen`, `denyICloudWebAccess` | boolean | `true`, except `icloudMailEnabled` and `stingrayDisabledIndicator` which are `false` | +| `rep`, `ut`, `signinPartition`, `isLegacyContactAssignee`, `isRecoveryContactAssignee` | integer | `1` | +| `reason` | integer | `5` | +| `usrt` | integer | `4` | +| `pkc` | string | `"1"` | +| `cfuids` | array | empty | +| `ptkn` | string | an APNs push token, **omitted entirely** by this flow — see below | + +**Never send `ptkn`.** The push token is the most likely reason a device becomes trusted for +verification codes, and Stage 1 §13 argues at length that this client must never become a second +factor for the user's Apple ID. Omit the key entirely rather than sending it empty. + +## 7.1 The device is the app's identity, not a temporary artefact + +**[observed, Stage 1 §7]** A second sign-in from the same machine identity required no second +factor. Apple remembers the machine, and stops challenging it. + +That matters because this feature is not a one-shot export. Noticing a newly-paired accessory +means re-reading the account periodically, which means re-authenticating periodically — and that +is only silent while a device registration exists for Apple to recognise. **Registering and +removing around each operation would demand a two-factor code every time**, which is fatal for a +background refresh and worse for the user than the entry it avoids. + +So the registration is created once and kept for the life of the installation. + +### Announce it, in the app, in plain terms + +The user should learn what exists on their account from the application, not from finding it +themselves: + +> Signing in registers this app as a device on your Apple account, so Apple will keep talking to +> it without asking for a verification code every time. It appears in your device list as +> **OpenTagViewer**, serial **0PENTAGVIEWR**. It cannot receive verification codes. Signing out +> removes it. + +Both identifiers must be the ones actually sent — the name from `dn` above, the serial from +`X-Apple-I-SRL-NO` in Stage 1 §2.2 — because those are the two fields visible in the device list, +and they are what the user will match against. + +### Remove it at sign-out, and only then + +Sign-out is the one moment removal is correct: the user has deliberately disconnected the +account, so the registration has no further purpose and a two-factor prompt is no longer a cost +anyone is paying. The protocol offers a sequence of sign-out events through the same endpoint, +with the request body changed: + +| Key | Value | +| --- | --- | +| `event` | `signout-`, and finally `signout-all` | +| `dn` | the same device name | +| `services` | the services **still** signed in — i.e. shrinking to empty across the sequence | +| `cdpStatus`, `circleStatus`, `otStatus` | `false` | +| `prkgen` | `true` | +| `rep`, `ut` | `1` | +| `loc` | `en_US` | + +and the headers to match: `X-Apple-I-CDP-Status` and `X-Apple-I-OT-Status` become `false`, and +`X-Apple-I-Service-Type` and `X-Apple-AK-DataRecoveryService-Status` are dropped entirely. + +The reference walks every service it might have used — iCloud, iMessage, FaceTime — signing each +out in turn with the remainder listed, then a final `signout-all`. This client only ever uses +iCloud, so `signout-icloud` followed by `signout-all` is the plausible minimum, **and that is an +inference, not an observation**. + +> **Two things here are unverified and must not be stated to the user as fact.** Whether this +> sequence removes the device from the account list at all, rather than merely signing services +> out. And how far `signout-all` reaches — it carries a device name, which *suggests* it is +> scoped to this device, but nothing here has confirmed that. Until both are established, prefer +> attempting `signout-icloud` alone, report what was attempted rather than what was achieved, +> and point the user at their device list. + +### The failure path is the one that matters + +A crash, a lost connection or a force-quit during sign-out leaves the registration behind. That +is the path back to an accumulating device list, so: + +- Removal must be **idempotent and retried** — attempt it again on next launch if the last + sign-out did not complete. +- The app should be able to say *"an earlier sign-out may not have finished; a device called X + may still be listed"* and link the user to their device list, rather than staying silent. +- **Never regenerate the identity to recover from a failed removal.** A fresh identity does not + clean up the old entry, it adds a second one — which is exactly how twenty of them appear. + +### The escrow record follows the same logic, in reverse + +Stage 3 creates an escrow record whose purpose is to let the client *re-establish* keychain trust +if it loses its local state. So the two artefacts have opposite lifetimes: the device registration +is what keeps ordinary access working and is kept, while the escrow record is only insurance and +can be deleted once trust is established — at the cost of needing the user's device passcode +again should local state ever be lost. Stage 3 specifies that trade properly. + +## 8. Open questions + +**Answered by the runs of 2026-08-13:** + +1. ~~Does the legacy request form still work?~~ **Yes.** This was the load-bearing unknown for the + whole project — the basis for authenticating without an attestation blob. Apple has not + withdrawn it. +2. ~~What is the full token set?~~ **Eleven names**, listed in §5. +3. ~~Which configuration key carries the CloudKit endpoint?~~ **None — no configuration is + returned at all.** It comes from opening the container instead; see §6. +4. ~~Is the configuration nested under a `com.apple.mobileme` key?~~ **No**, and there is no such + key. + +**Still open:** + +5. **Is `UNAUTHORIZED` really always an expired PET?** It is the obvious cause and the useful + default, but treating every rejection as "retry Stage 1" risks an infinite loop against an + account that is genuinely refused. Bound the retry. +6. **Does requesting only `com.apple.mobileme` return the same service data as requesting it + alongside the IDS delegate?** Assumed yes; unverified. +7. **What do the terms endpoints return for an account with nothing outstanding?** The acceptance + flow of §5.2 must not fire spuriously, so the no-terms-pending response needs to be + distinguishable from one carrying terms. + +**Open, and only relevant if the device-naming call of §7 is implemented:** + +8. **Does it rename an existing entry, or only name a new one?** If only at registration, calling + it later is useless and the serial of Stage 1 §2.2 is the only lever. +9. **Does the sign-out sequence remove the device from the account, or only sign services out?** + §7.1 makes removal at sign-out the policy, and this decides whether that promise can be kept. + Directly testable: run it and watch the device list. +10. **Does re-registering after a removal create a fresh entry or restore the old one?** Bears on + whether removal at sign-out costs a two-factor prompt on the next run — see the README. + diff --git a/docs/findmy-export/03-keychain-trust.md b/docs/findmy-export/03-keychain-trust.md new file mode 100644 index 00000000..362a1c9f --- /dev/null +++ b/docs/findmy-export/03-keychain-trust.md @@ -0,0 +1,2033 @@ +# Stage 3 — iCloud Keychain trust circle + +Specification of how a client joins the user's iCloud Keychain trust circle, which is what makes +the PCS keys of Stage 5 reachable and therefore what makes the accessory records of Stage 4 +readable. + +Read [README.md](./README.md) first. In particular: **implement from this document alone.** + +> **The read-only half is verified against a live account on 2026-08-13** — host derivation, +> transport, error envelope, the record schema of §5, and the escrow-to-Cuttlefish join, by two +> independent implementations written from this document. Values from that run are marked **[observed]**, and several corrected what reading +> the reference had suggested. +> +> **[observed] Deletion (§7.1) is verified** against a live account, including the corrected +> label addressing. +> +> **Recovery, joining the circle and enrolling remain unverified** — §6 onward. The sequence, +> cryptography and message layouts are specified; none has been run. Those are derived by reading implementations whose authors report them working. +> +> It is also the stage that **needs the user's device passcode** and **writes to their account**. +> Everything before it was read-only; this is not. Treat the residue rules of §7 as part of the +> specification rather than as advice. + +--- + +## 1. What this stage is for + +Find My accessory records are encrypted. Their keys live in the user's iCloud Keychain, which is +shared between the devices the user has explicitly trusted. To read the keys, this client has to +become one of those trusted devices. + +Apple's mechanism for admitting a device that cannot be approved by tapping "Allow" on an +existing one is **escrow recovery**: the user's keychain material is escrowed in a record +protected by a device's screen-lock passcode, and anyone who knows that passcode can recover it. + +That is why this feature needs a passcode it never uses for anything else, and it is the honest +answer to "does this remove the need for an Apple device": it removes the need to *have* one, not +the need for one to have *existed*. An Apple ID whose keychain circle has never contained a +device has nothing to recover from. + +## 2. Two services, joined by an identifier + +This stage talks to two unrelated-looking services, and understanding the split makes everything +else legible: + +| Service | Transport | Holds | +| --- | --- | --- | +| **Escrow proxy** | plain HTTPS, plist bodies | the escrow *records* — the passcode-protected material and its metadata | +| **Cuttlefish** | CloudKit function invocation | the trust *circle* — peers, bottles, keys, who trusts whom | + +"Cuttlefish" is the server side of Octagon, Apple's trust-circle protocol. "Bottle" is its name +for a sealed piece of key material a peer can be reconstituted from. + +Listing what the account has requires **both**: the escrow proxy knows the human-readable +metadata, Cuttlefish knows which bottles are actually usable, and they are joined on the bottle +identifier. Neither alone answers "what can I recover from". + +### 2.1 Cuttlefish rides on CloudKit + +Cuttlefish is not a REST service. It is invoked as a **server-side function through a CloudKit +container**: + +| Property | Value | +| --- | --- | +| Container id | `com.apple.security.keychain` | +| Bundle id | `com.apple.security.cuttlefish` | +| Database | private | +| Environment | `Production` | + +Calls are made with the function-invoke operation — field `1101` of `RequestOperation`, per +[Stage 4 §3.2](./04-cloudkit.md) — naming the target `Cuttlefish`, the method, and a protobuf +body. The response body is protobuf too. + +### 2.2 The invoke message + +Field `1101` of `RequestOperation` carries a `FunctionInvokeRequest`: + +| # | Field | Type | Value | +| --- | --- | --- | --- | +| 1 | `service` | string | `Cuttlefish` | +| 2 | `name` | string | the method, e.g. `fetchViableBottles` | +| 3 | `parameters` | bytes | **the method's own request message, serialised** | + +The response arrives at field `1101` of `ResponseOperation` as a `FunctionInvokeResponse`: + +| # | Field | Type | +| --- | --- | --- | +| 1 | `serializedResult` | bytes — **the method's own response message, serialised** | + +So there are **two layers of protobuf**: the CloudKit envelope, and inside `parameters` and +`serializedResult` an entirely separate message that CloudKit neither parses nor validates. A +malformed inner message therefore produces a successful CloudKit operation carrying a payload that +fails to decode, not a CloudKit error. + +### 2.3 `fetchViableBottles` + +The method [§5](#5-listing-what-the-account-has--read-only) step 2 needs. + +**Request:** + +| # | Field | Type | Value | +| --- | --- | --- | --- | +| 1 | `filter` | uint32 | `1` | +| 2 | `metrics` | bytes | empty | + +**Response:** + +| # | Field | Type | Meaning | +| --- | --- | --- | --- | +| 1 | `valid` | repeated `EscrowData` | **the usable bottles** | +| 2 | — | — | a legacy repeat of the same type; ignore | +| 3 | `partial` | repeated `EscrowMeta` | bottles that are not fully usable | + +**`EscrowData`:** + +| # | Field | Type | Meaning | +| --- | --- | --- | --- | +| 1 | `id` | string | **the join key** — matches an escrow record's `label` (§5 step 3) | +| 2 | `bottle` | message | the sealed material | +| 4 | `meta` | message | metadata | + +> Note field 3 is skipped, and that `EscrowMeta` is an **empty message** in every schema examined +> — an individual `partial` entry carries nothing at all. +> +> **But the count is meaningful.** [observed] A live call returned 3 `valid` and 8 `partial`, and +> the join of §5 produced exactly 3 recoverable records and 8 that had metadata but no viable +> bottle. So `partial` is not noise to discard: its length should equal the number of described- +> but-unrecoverable records, and a discrepancy means the join has gone wrong. **Use it as a +> cross-check**, and log both counts. + +**The endpoint is a different service from the record database**: + +``` +POST https://gateway.icloud.com/ckcoderouter/api/client/code/invoke +``` + +derived from `cloudKitCodeGatewayUrl` rather than `cloudKitDatabaseGatewayUrl`, both of which +[Stage 4 §2.3](./04-cloudkit.md) returns. The CloudKit headers, envelope and authentication are +otherwise identical to any other operation. + +> **So Stage 4 is a hard prerequisite for Stage 3, not merely a convenient one.** The CloudKit +> transport, its container-open step, its headers and its envelope are all needed here. The +> ordering in this document set — specifying and verifying Stage 4 before writing Stage 3 — was +> the correct dependency order and not just risk management. + +A second container, `com.apple.securityd` over the same `com.apple.security.keychain` container +id, exists for keychain item operations proper. + +## 3. Prerequisites + +| Value | From | +| --- | --- | +| **PET** | Stage 1. Used as the escrow proxy's HTTP Basic password — see the warning below. | +| The account's email | Stage 1 `acname`, the Basic username | +| A working CloudKit container | Stage 4 §2, for every Cuttlefish call | +| `com.apple.gs.icloud.escrow.auth` | Stage 1 issues this token; whether this stage needs it, or only the PET, is unestablished | +| **The user's device passcode** | the user, at the moment of recovery. Never stored. | +| Anisette headers | the same identity as every other stage | + +> **The PET is spent by Stage 2, and this is an ordering constraint on the whole flow.** Stage 2 +> uses it as its HTTP Basic password and then replaces it with the service tokens it returns, so +> an implementation that follows Stage 2 as written has **no PET left** by the time this stage +> begins — and it expires in about five minutes regardless. +> +> So Stage 3 needs either a **second, fresh authentication** whose PET is retained rather than +> spent, or Stage 2 amended to keep a copy. This is not a detail of Stage 3; it changes how +> Stages 1 to 3 fit together, and an implementation that discovers it late will have to restructure. + +## 4. The escrow proxy + +``` +POST /escrowproxy/api/ +``` + +> **[observed] The derivation below is confirmed.** `p-escrowproxy.icloud.com` answered a +> request from an account on partition 24 with a well-formed service response. The escrow proxy +> is partitioned like every other per-account host, and deriving the host from the CloudKit +> partition works. + +**Deriving the host is the first problem, and the protocol does not hand it over.** + +The intended source appears to be a key named `escrowProxyUrl`, nested somewhere inside the +MobileMe delegate's configuration — the reference searches its whole configuration tree +recursively for it, which is itself a sign that its position is not fixed. + +> **[observed] It is not there.** The delegate response for this client contains no configuration +> at all beyond a `protocolVersion` — see [Stage 2 §6](./02-mobileme-delegate.md). Independently, +> the reference exporter reports the same thing and falls back to a **hardcoded** +> `p97-escrowproxy.icloud.com`. + +That hardcoded value is a bug waiting to happen, and the reason is visible in +[Stage 4 §2.3](./04-cloudkit.md): `p97` is a **partition prefix**. iCloud shards accounts across +numbered partitions, every per-account service host is named `p-.icloud.com`, and the +account observed for these documents is on a different partition entirely. + +**So derive it.** Opening the CloudKit container returns URLs such as +`p-ckdatabase.icloud.com`; take the partition number from any of them and construct: + +``` +https://p-escrowproxy.icloud.com:443 +``` + +**[observed]** This works. A client should still fail legibly rather than mysteriously if the +host does not answer, since the derivation is a pattern rather than a documented rule. Whether +another partition's proxy would also serve the account is untested and does not need to be. + +### 4.1 Errors + +A failure returns a plist with `success`, `errorCode` and `errorMessage`. **[observed]** The +messages are specific and worth surfacing verbatim — `Wrong command sent: get_records` named the +exact mistake. Check `success` before reading anything else. + +### 4.2 Transport + +| Aspect | Value | +| --- | --- | +| Authentication | HTTP Basic: the account **email**, and the **PET** as password | +| `Content-Type` | `application/x-apple-plst` | +| Body | an XML property list | +| `User-Agent` | `com.apple.sbd/638.100.48 CFNetwork/ Darwin/` | +| `X-Mme-Client-Info` | the client-info string of Stage 1 §2.2, bundle `com.apple.AuthKit/1 (com.apple.sbd/638.100.48)` | +| `Accept` | `*/*` | +| `Accept-Language` | `en-US,en;q=0.9` | +| `X-Apple-I-Locale` | `en_US` | +| `x-apple-i-device-type` | `1` | + +Plus all the Anisette headers. + +`sbd` is *secure backup daemon* — Apple's internal name for this mechanism, and the reason +escrow records are called secure-backup records throughout. Note the unusual content type: +`application/x-apple-plst`, not the `text/x-xml-plist` used by Grand Slam. + +### 4.3 Commands + +> **Each command is named twice, in two different spellings, and both must be right.** The URL's +> last path segment is lower_snake_case; the body's `command` field is SCREAMING_SNAKE_CASE, and +> it is *not* simply the uppercase of the path. Sending the path spelling in the body is rejected +> with `Wrong command sent: ` — **[observed]**, and it is the failure this document +> originally caused by describing the two as one thing. + +| Command | URL path segment | Body `command` value | Purpose | +| --- | --- | --- | --- | +| Get records | `get_records` | `GETRECORDS` | **list escrow records** — read-only | +| Get club certificate | `get_club_cert` | `GETCLUB` | fetch the certificate chain used when enrolling | +| SRP init | `srp_init` | `SRP_INIT` | begin passcode-authenticated recovery | +| Recover | `recover` | `RECOVER` | complete it and obtain the escrowed material | +| Enroll | `enroll` | `ENROLL` | **create** an escrow record | +| Delete | `delete` | `DELETE` | **destroy** an escrow record | + +Note the irregularity: `get_club_cert` becomes `GETCLUB`, dropping a word, while `srp_init` +becomes `SRP_INIT`, keeping its separator, and `get_records` becomes `GETRECORDS`, losing its. +There is no rule to derive one from the other — the pairs must be tabulated. + +Only `get_records` is free of consequences. `enroll` and `delete` both modify the account, and +`delete` is destructive to something a real device may depend on. + +### 4.4 Request body + +A dictionary. Fields vary by command; the ones common to all: + +| Key | Type | Meaning | +| --- | --- | --- | +| `command` | string | the command's **body spelling** — see the table above, not the URL segment | +| `label` | string | which record class is being addressed — see below | +| `transactionUUID` | string | a fresh uppercase v4 UUID per request | +| `userActionLabel` | string | a free-text description of *why*, for Apple's logs | +| `version` | integer | `1` | + +and, per command: `blob`, `blobDigest`, `metadata`, `dsid`, `silentAttempt`, +`baseRootCertVersions`, `trustedRootCertVersions`. + +**`label` is overloaded**, and which meaning applies depends on the command: + +| Command | `label` means | +| --- | --- | +| `get_records` | a **record class** — `com.apple.securebackup.record` | +| `get_club_cert` | a record class — `com.apple.icdp.record` | +| `srp_init`, `recover`, `enroll`, `delete` | **one specific record**, identified individually | + +A specific record's label has the form: + +``` +com.apple.icdp.record.SHA256: +``` + +the class, then the identifier of the peer the record belongs to. **[observed] That peer +identifier is a `SHA256:`-prefixed base64 digest** — not a UUID, not a serial number, and not +anything a client chooses. It is the same value `CuttlefishPeer.hash` carries (§6.9), which is why +a peer is addressed by a hash rather than a name. + +Two consequences: a label cannot be constructed from anything the user can see, so a record must +always be located by listing rather than by building its label; and **two records for the same +physical device will have entirely different labels** if the client regenerated its peer identity +between them. + +> **A record's label is not its `bottleID`, and the two must not be substituted for each other.** +> The label identifies the *escrow record*; `bottleID`, found inside the record's metadata, is a +> separate **UUID** identifying the bottle. They are different strings with different shapes. +> Everything addressed to the escrow proxy takes the **label**. + +### 4.5 Enrolling a record + +**This is the write, and it is what makes an escrowed bottle recoverable at all.** A bottle sent in +`joinWithVoucher` whose entropy was never enrolled produces a peer that is in the circle and can +**never** be recovered — permanent, invisible in every Apple interface, and precisely the residue +§7's rules exist to prevent. Enrol before joining, not after. + +It is two requests sharing one `transactionUUID`. + +**Step 1 — `get_club_cert`.** Label `com.apple.icdp.record` — the *class*, not a specific record. +Send `baseRootCertVersions` and `trustedRootCertVersions` as the same list of acceptable versions, +**[observed]** `[101, 500, 103, 102]`. The response carries `clubCert`, base64 of a DER +certificate. + +> **[observed] The reply is `clubCert` and four scalars, and carries no chain.** So the issuing +> root cannot be fetched here and fingerprint-checked against the table below — a client has to +> carry the roots. Worth stating because it is the obvious idea, it would be strictly better than +> bundling, and it does not work. + +> **Verify that certificate against a pinned set of roots before using it.** It is the key the +> user's escrow blob is encrypted to, so accepting an unverified one hands the material to +> whoever supplied it. A chain that does not verify is a refusal, not a warning — and the +> verification belongs **before** anything is encrypted to that key, not after. + +**The four roots**, which a client carries rather than fetches. All are self-signed, all share the +subject `CN=Escrow Service Root CA, OU=Apple Certification Authority, O=Apple Inc., C=US`, and the +version number is the certificate's **serial number**: + +| Version | Not before | SHA-256 fingerprint | +| --- | --- | --- | +| 101 | 2015-05-16 | `5644C142208DD4BF7AD770902F70D6730B8571164FD874D9AE5168807B32F766` | +| 102 | 2016-05-13 | `D4EAA88170B8AEE18FCEFF68698310D4C2AB4CEB48179D99206A53AC73E8A4EB` | +| 103 | 2021-05-13 | `FDBA6F3365D617B9EB799D2F48851B1A85C62CF366C36797E539B3C4C2C3BDC6` | +| 500 | 2022-12-09 | `654BFB656D80A715559034C7FE08E637B335E770C84EC05CD90475E8BBC59EB0` | + +> **The four are carried alongside this document**, in [`escrow-roots/`](./escrow-roots/), as DER +> (`.crt`, what a loader wants) and PEM (`.pem`, for reading). Each has been checked against the +> fingerprint above. +> +> **The fingerprints are the useful half, and that is what makes the delivery route irrelevant.** A +> root is trustworthy once it matches one of these and not before, so it needs no trusted channel — +> and equally, a pinning set assembled without checking is pinning to whatever arrived. Keep the +> check on the bundled copies too: a bundled certificate that fails its own fingerprint is a +> corrupted install, and should refuse exactly as loudly as a wrong one from a caller. +> +> **This is library data, not user input.** A client that asks whoever installs it to go and find +> four certificates gets them pasted in unverified, which is worse than shipping them, because it +> looks like someone made a security decision when nobody did. +> +> ### [observed] 500 is the one in use, and it expires in 2032 +> +> On a real account, the club certificate verified against **root 500 and none of the other +> three**, and its issuer DN carries `2.5.4.5=500` — the X.520 `serialNumber` attribute, which is +> what makes an issuer match select one of four otherwise identical subjects. All four share +> `CN=Escrow Service Root CA`, so a matcher that ignores that attribute has four candidates and +> no way to choose. +> +> **That turns 500's expiry into this specification's deadline.** 101, 102 and 103 run to 2049 and +> are apparently not what Apple issues against; 500 runs out in **December 2032**, so Apple moves +> to a root outside this set before then. A pinning set with a known end date is something to be +> told in advance rather than discovered, because the way it surfaces is enrolment refusing with a +> certificate error, months after anyone could have shipped a fix. +> +> **So warn early.** A client should say something when the root it is verifying against is within +> a year of expiry, and treat a club certificate that verifies against none of the four as "Apple +> has rotated" rather than as a broken account. +> +> One account, one call — this is where the set stood when it was looked at, not a promise about +> what Apple issues tomorrow. Carry all four regardless: the cost is four kilobytes, and the +> alternative is a client that breaks on a rotation it could have absorbed. + +> Whichever set is carried, a client should say clearly when a chain fails to verify rather than +> falling back to the system trust store — the system store holds Apple's public roots, and these +> are not those. + +**Step 2 — `enroll`.** Label is the **specific record's** label, per §4.4. Fields: + +| Key | Content | +| --- | --- | +| `blob` | base64 of the escrow blob, §4.5.1 | +| `blobDigest` | base64 of the blob's **SHA-1** — not SHA-256 | +| `dsid` | the numeric account id | +| `metadata` | base64 of a **binary** property list, §4.5.2 | + +> **The certificate versions go on `get_club_cert` only.** `enroll` does not carry +> `baseRootCertVersions` or `trustedRootCertVersions`; they asked which roots the *client* would +> accept, and that question was answered by the first request. Sending them again is sending a +> field to a write that a working client does not send. + +#### 4.5.1 The escrow blob + +Two nested layers, both in the KeyVault framing of §6.1.1. + +**The inner message** — header `unk1=160`, `unk2=0`, `rounds=10000`, `unk3=10`, then six sections +in this order: + +| # | Section | Note | +| --- | --- | --- | +| 1 | the **dsid**, as ASCII | padded to a 16-byte footprint | +| 2 | a random **64-byte salt** | | +| 3 | the **SRP verifier** | computed over the dsid as identity, the password, and that salt, using the 2048-bit group and SHA-256. **Zero-pad it to 256 bytes** | +| 4 | the **encrypted record** | §4.5.3, AES-128-CBC under PBKDF2-HMAC-SHA256(password, salt, **10000** iterations, 16 bytes), IV = the salt's **first 16 bytes** | +| 5 | the **label**, as ASCII | padded to an 80-byte footprint | +| 6 | the **timestamp**, as ASCII | `YYYY-MM-DD HH:MM:SS` — a space rather than a `T`, and no zone. Padded to a 24-byte footprint | + +> The verifier is what later makes `srp_init` and `recover` (§6) work with this password. The +> encrypted record and the verifier are derived from the **same** password and the **same** salt, +> by two different constructions — do not reuse one derivation for the other. + +**The outer message** — header `unk1=161`, `unk2=1`, `unk3=0`, `unk4=0`, `unk5=10`, then: + +| # | Section | +| --- | --- | +| 1 | HMAC-SHA-256 of section 2, under a fresh random 32-byte key | +| 2 | a random 16-byte IV, followed by the inner message encrypted with **AES-256-CBC** under a fresh random 32-byte key | +| 3 | **RSA-OAEP** of those two random keys concatenated — the AES key then the HMAC key — to the club certificate's public key, with **SHA-1** as both the OAEP and MGF1 digest | +| 4 | SHA-256 of the certificate's public key, as **PKCS#1** DER | +| 5 | SHA-256 of the **inner message** | + +> Note the digest asymmetry, all three of which are load-bearing and none derivable: the blob is +> announced with **SHA-1**, the RSA padding uses **SHA-1**, and the two integrity hashes inside are +> **SHA-256**. + +#### 4.5.2 The metadata + +A binary property list, base64-encoded: + +> **Correction. These key names disagreed with §5.1, and §5.1 was right.** Three of them were +> written here as an implementation's internal field names rather than as what goes on the wire. +> The two sections describe one plist, the service stores it verbatim, and a record written under +> the wrong spellings is one §5.1's listing cannot describe — no name, no serial, no date — which +> is the only place these records are ever seen. + +| Key | Content | +| --- | --- | +| `serial`, `build` | the client's own, as in [Stage 1 §2.2](./01-authentication.md) | +| `passcodeGeneration` | **[observed]** `13` | +| `bottleID` | the bottle's **UUID** — see §4.4 on why this is not the label | +| `escrowedSPKI` | the escrowed **signing** public key | +| **`com.apple.securebackup.timestamp`** | the same string as the blob's section 6, in the same `YYYY-MM-DD HH:MM:SS` form | +| **`ClientMetadata`** | a nested dictionary, below. **Capital C** | +| **`SecureBackupUsesMultipleiCSCs`** | `true`. Note the lower-case `i` in `iCSCs` | + +> Only three keys are ordinary camelCase. The rest are a reverse-DNS key, a PascalCase one and a +> `SecureBackup`-prefixed one with irregular capitalisation — the same +> [strata](./README.md) pattern as everywhere else here. Copy them exactly; none is derivable. + +`clientMetadata` describes the device to a human reading the record listing of §5 — which is the +only place any of it is ever seen: + +| Key | | +| --- | --- | +| `device_name`, `device_model`, `device_model_version`, `device_model_class`, `device_platform` | what §5.1's listing shows the user | +| `device_mid` | the machine identifier | +| `SecureBackupMetadataTimestamp` | the timestamp again | +| `SecureBackupUsesNumericPassphrase` | whether the password is all digits | +| `SecureBackupNumericPassphraseLength` | its length if so, otherwise `0` | +| `SecureBackupUsesComplexPassphrase` | `1` | + +> **This metadata is how a user recognises their own record**, so it is worth filling honestly. +> §5.1 records that escrow records outlive the devices that made them and appear in no Apple +> interface; a record labelled with an empty device name is one the user cannot identify when +> deciding what to delete. + +#### 4.5.3 What the encrypted record holds + +Section 4 of the inner message is a **binary property list** before encryption, with three keys: + +| Key | Content | +| --- | --- | +| `BottledPeerEntropy` | **72 random bytes**. Everything the bottle yields derives from this | +| `com.apple.securebackup.timestamp` | the same timestamp again | +| `BackupVersion` | the string `1` | + +> **`BottledPeerEntropy` is the whole point of the record**, and §6.7's recovery reads exactly this +> key back out. A record enrolled without it recovers successfully and yields nothing — which is +> worse than failing, because the escrow proxy will report a usable record for as long as the +> account exists. +> +> The entropy is **generated, not derived**: it is fresh randomness this client keeps nowhere else, +> and enrolling is what makes it recoverable at all. + +> ### Generate it outside enrolment, not inside +> +> **The same bytes must seal the bottle that `joinWithVoucher` carries.** Enrolment makes the +> entropy recoverable by passcode; the bottle is what that entropy opens. A record escrowing +> different entropy from the bottle sent alongside it enrols cleanly, joins cleanly, and recovers +> months later to **a peer that does not exist** — with no step in between that could notice. +> +> So the entropy is an input to enrolment, never something enrolment invents. A function that +> generates its own is one a caller cannot make agree with the bottle, and the failure is +> unobservable until someone needs the recovery. + +> **[observed] The other fields recovered material carries are not required.** §6.7 reports +> `SecureBackupIDMSData`, a `DoubleEnrollmentPassword` and version, a `BackupBagPassword`, a backup +> version and a timestamp coming back from real records. A record enrolled with the three keys +> above is recoverable and yields its entropy — so those are what an Apple client happens to +> include, not what the service or the recovery requires. Do not synthesise them. + +> **PascalCase and a reverse-DNS key in the same three-key dictionary**, and the timestamp key is +> the same reverse-DNS spelling as §4.5.2's. Two of the three do not follow the convention of +> either neighbouring structure. + +> **`userActionLabel` is a free-text string that Apple keeps.** The reference sends descriptions +> of the operation being performed. It is not authentication and nothing checks it, but it is +> written into someone else's audit trail, so it should describe what is actually happening and +> should not impersonate a first-party Apple process. + +## 5. Listing what the account has — read-only + +This is the part worth building first: it shows the user what exists, creates nothing, and needs +no passcode. + +**Step 1 — escrow metadata.** `get_records` with label `com.apple.securebackup.record`. + +**[observed]** The response envelope carries `status`, `message`, `version`, `dsid` and +`metadataList`. Check `status` and surface `message`; the list is the payload. `metadataList` is +an array of: + +| Key | Meaning | +| --- | --- | +| `label` | **the record's identifier, and the join key** — see below | +| `metadata` | base64 of an XML plist: the record's descriptive fields | + +> **Join on `label`, not on `bottleID`.** The escrow proxy's `label` is the same value as the +> Cuttlefish bottle's `id` (§2.3), and that pairing is what step 3 matches. The `bottleID` inside +> the decoded metadata is a **different** value — a UUID for the bottle itself — and nothing +> requires it to equal the label. Reporting a mismatch between them is reasonable +> diagnostics; treating either as a substitute for the other is not. + +**Step 2 — viable bottles.** Cuttlefish `fetchViableBottles`, with a filter value of `1` and an +empty metrics list. The response carries a list of valid bottles, each with an id. + +**Step 3 — join them.** Match each bottle's id against a metadata entry's `label`. A bottle +without metadata cannot be described to the user; metadata without a viable bottle cannot be +recovered from. **Report both mismatches rather than silently dropping them** — they are the +signal that this specification has drifted from what Apple returns. + +### 5.1 What the metadata contains + +Decoding the base64 plist gives: + +| Key | Meaning | +| --- | --- | +| `ClientMetadata` | a dictionary holding `device_name`, `device_model`, `device_model_class` | +| `serial` | the device's serial number | +| `build` | its OS build | +| `com.apple.securebackup.timestamp` | when the record was escrowed | +| `bottleID` | the bottle's own identifier | +| `escrowedSPKI` | the escrowed public key information | +| `passcodeGeneration` | which generation of the device's passcode protects it. **[observed]** camelCase, and present on only the newest records — 2 of 12 on the observed account. | +| `SecureBackupUsesMultipleiCSCs` | a legacy flag, from the iCloud Security Code era | + +**This is a copy of a device's identity**, which is why there is no separate escrow UI in any +Apple interface — see the [README](./README.md). Present it to the user as device name, model, +serial and escrow date, because those are the fields they can match against their own device +list. + +### 5.2 Records count runs; serials count devices [observed] + +A live listing showed twelve records whose serials were **not** distinct: one serial appeared three +times, another twice, and two more once each. + +Three records for one serial is not a pair under any reading. What it fits is a client that +**regenerated its peer identity on every run while reusing a fixed serial** — the serial being the +thing the client declares about itself, the peer identity being the thing it was supposed to +persist. Each run minted a new peer, each new peer escrowed a new record, and the serial stayed +put. + +So when counting: + +| To count | Group by | +| --- | --- | +| escrow **runs** | the record, or its peer digest | +| claimed **devices** | the **serial** | + +Reporting a record count as a device count overstates the position, and grouping by peer identity +does not fix it — only the serial does. + +> **This is the residue rule failing, observed in the wild.** The +> [README](./README.md) says identity state must never be silently regenerated; these records are +> what happens when it is. And the escrow list is where the evidence accumulates rather than the +> device list, because **nothing ever removes an escrow record** — so it preserves a run-by-run +> history of a mistake that the device list would have partly hidden. + +> ### [observed] Escrow records outlive the devices that created them +> +> On the account these documents were written against, the device list showed **one** iMac Pro +> while the escrow proxy held **eight** `iMacPro1,1` records — phantoms left by an older +> macOS-VM export route whose device entries had since been removed. +> +> **Removing a device from the account does not remove its escrow record.** The device entry is +> the visible artefact; the escrow record is the durable one, and it accumulates in the one place +> Apple provides no interface to inspect. +> +> Two consequences, and they point in opposite directions: +> +> - **The residue rules matter more here than for the device registration**, not less. An +> uncleaned escrow record is permanent as far as the user is concerned, because they cannot +> see it to clean it up. +> - **The list a user is shown before any deletion will contain records they no longer recognise** +> — devices they have sold, wiped or removed years ago. That makes "delete the one you don't +> recognise" actively dangerous advice, and is why deletion must confirm against the serial of +> a specific record rather than inviting a judgement call. + +**Treat the schema as unstable — [observed], it genuinely is.** Of twelve records on one +account, eleven had the device shape above and one had a different shape entirely: +`BackupKeybagDigest`, `ClientMetadata` and a timestamp, with **no** serial, build, `bottleID` or +`escrowedSPKI`. `passcodeGeneration` appeared on only two. + +So there is more than one kind of record under this label, and a client that assumes the device +shape will fail on the others. Decode defensively, report what could not be understood, and never +discard a record silently. A record without a `bottleID` cannot be a recovery candidate and +should be filtered out rather than treated as broken. + +## 5.3 The peer directory + +Two verifications later in this stage need to look up a peer's public keys by its identifier, and +nothing described so far provides that. The directory comes from Cuttlefish: + +``` +Cuttlefish fetchChanges +``` + +| Message | # | Field | +| --- | --- | --- | +| Request | 1 | `syncToken` — omit on a first call | +| Response | 1 | `changes`: `syncToken` (1) and repeated `change` (2) | +| `change` | 3 | `add` — a `CuttlefishPeer` | + +A `change` may carry other kinds at other field numbers; **only `add` at field 3 builds the +directory**, and anything else is skipped. Each added peer carries its `hash` — the identifier used +everywhere else — and a `permanentInfo` whose `info` decodes to a `PeerPermanentInfo` (§6.9) +holding **`signingKey` at field 2** and `encryptionKey` at field 3. That signing key is what the +checks below verify against. + +> **This is a paging loop, not a single call.** One request does not return the circle. Loop: +> +> 1. Call `fetchChanges` with the current token — **absent** on the first call. +> 2. If `changes` is **empty**, store the returned token and **stop**. That is the only +> termination condition. +> 3. Otherwise add every peer found, take the returned `syncToken`, and go again. +> +> A client that issues one request and reads its result **will often see nothing**, because the +> first page need not carry peers. An empty directory after a single call is not evidence that the +> circle is empty — it is the expected outcome of not looping. + +**Keep the final `syncToken`** and pass it next time; the feed is incremental, and a client that +always starts from nothing re-reads the whole circle on every run. + +> **[observed] Exactly one peer in a circle carries no voucher**, and it is not an anomaly: the +> peer that **established** the circle joined nothing, so there was nobody to vouch for it. +> Thirteen peers with twelve vouchers is the expected shape, and thirteen with thirteen would be +> the surprise. +> +> Worth knowing before a check over the directory reports it as a gap. Count the unvouched rather +> than skipping them — one is right, none means the establishing peer has been removed, and more +> than one is worth looking at. + +**Handle a rejected token.** If a call fails with a CloudKit error whose description is +`.changeTokenExpired`, the stored token is no longer valid — someone has reset the circle. Discard +the token **and the accumulated directory**, then start again from no token. Treating this as a +fatal error strands a client that could simply resynchronise. + +> **This is a prerequisite, not an optimisation, and it is needed twice:** +> +> - [§6.7 step 3](#step-3--verify-before-trusting) verifies the **sponsoring peer's signature** +> over the recovered bottle. That peer is looked up here. +> - [§6.7.0 step 1](#670-fetching-the-recovered-peers-key-shares--and-why-joining-may-be-unnecessary) +> verifies each **key share's sender**. Likewise. +> +> A client without the directory cannot perform either check. Skipping them is not a small +> weakening: an unverified bottle is one an attacker who could serve a response may have chosen, +> and an unverified share is key material of unknown origin being trusted to decrypt the user's +> data. **Fetch the directory first**, and treat a peer that is genuinely absent from it as a +> reason to reject that share or bottle rather than to proceed. + +## 6. Recovering — the passcode step + +Recovery is an **SRP exchange in which the device's passcode is the password**. That is why a +passcode is required: it is not a credential Apple checks, it is the secret that decrypts the +escrowed material, and Apple cannot recover it either. + +**The passcode is used twice** — once as the SRP password, and again as a PBKDF2 input to unwrap +the innermost blob (§6.5). Both uses are essential; neither substitutes for the other. + +### 6.1 The KeyVault message framing + +Both escrow blobs use a framing that appears nowhere else in this protocol. Given a header length +*H* and a section count *S*: + +| Offset | Content | +| --- | --- | +| 0 | **the total message length**, 4 bytes big-endian | +| 4 | the header, *H* bytes | +| 4 + *H* | *S* + 1 offsets, each 4 bytes big-endian, **relative to the start of the body** | +| 4 + *H* + (*S*+1)×4 | the body | + +Each section lives at `body_start + offset` and begins with its own 4-byte big-endian length, +followed by that many bytes. + +> There is **one more offset than there are sections**, the last marking end-of-body. Computing +> the body's start from *S* rather than *S* + 1 puts every section four bytes out and produces +> garbage that looks like a decryption failure. + +**[observed]** A real `srp_init` response: prefix `00000180` = 384 bytes, *H* = 24, three sections +of 8, 64 and 256 bytes. The arithmetic checks: body starts at 24 + 4 + 16 = 44, the sections with +their length prefixes occupy (4+8) + (4+64) + (4+256) = 340, and 44 + 340 = **384**. If a decoder's +computed total does not equal the prefix, it has the framing wrong and should say so rather than +proceeding. + +### 6.1.1 Writing one + +Reading and writing are not symmetrical, and this is the step where that bites. + +1. Lay out the **body** first: each section is `be32(length) ‖ data`, concatenated. Record each + section's offset from the body's start as you go. +2. Append **one more offset**, equal to the finished body's length. +3. Emit: a 4-byte placeholder, the header, the *S* + 1 offsets, then the body. +4. **Overwrite the placeholder with the total length** of everything written. + +> **A section may be padded to a fixed footprint, and then its length prefix and its footprint +> disagree.** Where §6.3 asks for a section "sized to 20 bytes", it means: build +> `be32(length) ‖ data` as usual, then **zero-pad the result until it occupies exactly 20 bytes**. +> For an 8-byte value that is 4 + 8 = 12 bytes of content followed by 8 zero bytes. The padding is +> **appended**, never prepended, and the fixed footprint is a constant the service expects rather +> than anything derivable from the data. +> +> The declared length stays **8**, not 20 — a reader takes the length from the prefix and ignores +> the padding — but the **offsets must account for the full 20**, or every later section is +> misplaced. Sending the value unpadded, or padded without the inner length prefix, produces a +> message the server rejects without saying why. + +### 6.2 Begin the exchange + +Send `srp_init` (§4.3) with `label` set to the chosen record's **label** — its own identifier from +the `metadataList` entry, *not* its `bottleID` — and `blob` set to the base64 of the SRP client's +public value **A**. + +SRP parameters match [Stage 1 §4.1](./01-authentication.md): the RFC 5054 2048-bit group, +SHA-256, and a 32-byte client secret. + +The response carries `respBlob`, `dsid`, and an optional `clubTypeID`. Parse `respBlob` with +*H* = 24 and *S* = 3: + +| Section | Contents | +| --- | --- | +| 0 | an **exchange token** — echoed back in §6.3. **[observed] 8 bytes.** | +| 1 | the SRP **salt**. **[observed] 64 bytes.** | +| 2 | the server public value **B**. **[observed] 256 bytes**, matching the 2048-bit group. | + +The 24-byte header is two 32-bit values followed by a **16-byte transaction id**. + +> **There are two identifier-like values here and they are not interchangeable.** The header's +> 16-byte transaction id is carried back **unchanged, inside the header**. The 8-byte exchange +> token in section 0 is what §6.3 re-sends **as a section**. Padding 16 bytes to 20 looks like the +> more natural operation and is the wrong one; the value that gets padded is the 8-byte token. + +### 6.3 Compute the proof + +> **This is standard SRP-6a, unlike Stage 1.** Stage 1 §4.6 documents three deviations for Grand +> Slam; **none of them apply here.** The identity *is* included in the private-key hash, and the +> password is the passcode **directly** — not stretched through PBKDF2 first as `s2k` requires. +> The same SRP code serves both only if it is parameterised, and getting this backwards produces +> an authentication failure indistinguishable from a wrong passcode. + +The identity is the **`dsid` string from the `srp_init` response**, not the Apple ID and not the +`adsid` from Stage 1. + +Then build the response blob: take the 24-byte header from §6.2 and change two fields — set the +first 32-bit value to **165**, and the second to **2** if `clubTypeID` is 1, otherwise **0**. + +**[observed]** The value received in that first field is **164**, so the response is the request's +value plus one. Both readings — a literal 165, or an increment — fit what has been seen; the +reference sends a literal 165. +Frame it as a KeyVault message (§6.1.1) with **exactly two sections, and no others**: + +| Section | Contents | +| --- | --- | +| 0 | the **8-byte exchange token** from §6.2 section 0, in a **20-byte footprint** | +| 1 | the SRP proof **M1**, an ordinary section | + +The first is built as §6.1.1 describes: its own 4-byte length prefix holding **8**, then the eight +bytes, then **eight zero bytes appended** to bring the footprint to twenty. The declared length +stays 8. Why twenty is not established — it is simply the size the service expects. + +The header is the same 24 bytes received in §6.2 with only its two 32-bit fields changed; **the +16-byte transaction id is carried back untouched.** + +Send that as `blob` with the `recover` command, same `label` and same transaction id. + +### 6.4 Unwrap the outer blob + +Parse the response `respBlob` with *S* = 3 and *H* = **40 if `clubTypeID` is 1, otherwise 24**. + +Verify the server's proof **M2** against section 0 before going further. + +The version is a 4-byte big-endian integer at **offset 4 within the header**, and it selects the +cipher: + +| Version | Cipher | Key | IV / nonce | Ciphertext | +| --- | --- | --- | --- | --- | +| 0 | AES-256-CBC | the SRP session key | section 1 | section 2 | +| 2 | AES-256-GCM, **16-byte nonce** | the SRP session key | section 1 | section 2 | +| 1 | **unknown** — refuse rather than guess | | | | + +### 6.5 Unwrap the inner blob + +The plaintext from §6.4 is itself a KeyVault message, with *H* = 16 and *S* = 6. Its header is +four 32-bit values, of which **the third is a PBKDF2 iteration count**. + +``` +derived = PBKDF2-HMAC-SHA256(passcode, salt = section 1, iterations = header[2], length = 16) +material = AES-128-CBC-decrypt(derived, iv = first 16 bytes of section 1, data = section 3) +``` + +Note that section 1 serves as **both** the PBKDF2 salt and, in its first 16 bytes, the CBC +initialisation vector. The result is the escrowed keychain material. + +> **Sections 0, 2, 4 and 5 are unaccounted for.** Six sections are framed; three are used. What +> the others carry is unestablished. + +### 6.6 Certificate pinning + +The escrow service is authenticated against a **pinned set of certificates**, referenced by the +`baseRootCertVersions` and `trustedRootCertVersions` fields of §4.4 — the reference offers +versions 101, 102, 103 and 500. The certificates themselves must be obtained; this document does +not carry them. + +**This matters more here than elsewhere.** The material crossing this exchange is the user's +entire keychain, and the passcode protecting it is short enough to brute-force offline if an +attacker obtains the blob. Do not disable verification, and do not fall back to the system trust +store because pinning was inconvenient. + +### 6.7 From recovered material to a trusted peer + +The §6.5 plaintext is not the keychain. It is a **bottled peer** — the sealed identity of a +device that was already in the circle. Recovering it lets this client impersonate that device +just long enough for it to vouch for a *new* identity of our own, which is what actually joins. + +That indirection is the whole design, and it is why the sequence has five steps rather than one. + +**Step 1 — parse the recovered blob.** It is a property list with one field that matters: a +**bottled peer entropy** blob. Everything below derives from it plus the account's `adsid`. + +**Step 2 — derive the bottle's three keys.** All by **HKDF-SHA384**, with the **`adsid` as salt** +and the entropy as the input keying material: + +| Key | HKDF info string | Output | +| --- | --- | --- | +| Symmetric | `Escrow Symmetric Key` | 32 bytes | +| Signing | `Escrow Signing Private Key` | 56 bytes → an EC scalar | +| Encryption | `Escrow Encryption Private Key` | 56 bytes → an EC scalar | + +The two EC keys are on **P-384**, and the 56-byte output is converted to a scalar by **FIPS 186-4 +B.5.1**, the extra-random-bits method — generate more bits than the order needs and reduce, rather +than retrying. + +> Note the salt: the account identifier, not a random value. Two accounts recovering the same +> entropy would derive different keys, which is the point. + +**Step 3 — verify before trusting.** The bottle carries public keys and signatures, and all four +checks should pass before anything is decrypted: + +- the derived encryption key's public part equals the bottle's escrowed encryption key +- the derived signing key's public part equals the bottle's escrowed signing key +- `escrowedKeySignature` verifies under the **escrowed signing key**, **ECDSA over SHA-384** +- `peerKeySignature` verifies under the **sponsoring peer's signing key** from the directory of + §5.3, also **SHA-384** + +> **Both signatures cover the same data: the raw serialised `OTBottle` bytes** — the value of the +> `bottle` field of §6.9, exactly as received, with nothing prepended and no re-encoding. This is +> the simplest signed-data construction in the stage, and the only one that is just a field's +> bytes. + +A mismatch on either of the first two means the passcode produced the wrong entropy. The third and +fourth mean the bottle is not what it claims. + +**Step 4 — open the bottle.** **AES-256-GCM**, keyed by the symmetric key from step 2, with the +IV and authentication tag carried alongside the ciphertext in the bottle structure. The plaintext +holds the sponsoring peer's **signing and encryption private keys**, again P-384. + +At this point the client holds a second peer's identity. It has not joined anything. + +**Step 5 — vouch, then join.** + +1. Generate a **new identity of our own** — a fresh peer, with its own keys. +2. Using the recovered peer, **sign a voucher** for the new identity. This is the recovered + device saying "I trust this one". +3. **Fetch the recovered peer's key shares** — the TLK shares that carry the actual keychain view + keys. **If it has none, stop**: joining would succeed and yield no keys, which is a worse + outcome than failing. +4. **Create a bottle for the new identity**, sealed under the *device passcode*, so that this + client is itself recoverable later. This is the escrow record §7 discusses. +5. Call Cuttlefish **`joinWithVoucher`** with the new peer, its bottle, the voucher's sponsor, and + the re-shared keys. + +The response carries trust changes to apply to local state. A client with no voucher would call +**`establish`** instead, creating a new circle rather than joining one — **not what this project +wants**, and calling it by mistake resets the account's trust rather than joining it. + +> **`establish` and `joinWithVoucher` differ by one branch and are catastrophically different.** +> The first is for a device forming a circle where none exists; the second for joining one that +> does. An implementation that falls back from the second to the first on error will silently +> destroy the user's existing trust circle. There is no reason this project should ever call +> `establish`. + +**Step 6 — sync the views.** With trust established, sync the `Manatee` and `ProtectedCloudStorage` +views (§6.8). Only then does [Stage 5](./05-pcs-decryption.md) have keys. + +> **Passcode handling.** Read it, use it twice, discard it. Never written to disk, never logged, +> never in diagnostics, and not retained in memory past the exchange. It is the most sensitive +> value this project touches: it is the key to the user's whole keychain, and unlike an Apple ID +> password it cannot be rotated without physical access to the device. + +### 6.8.2 Building the identity a join sends + +Three values a joining client must produce for itself, none of which is derivable from the message +layouts of §6.9. Each fails **after** `joinWithVoucher` has been sent. + +#### The peer identifier + +``` +peerId = "SHA256:" ‖ base64( SHA-256( permanentInfo.info ‖ permanentInfo.signature ) ) +``` + +where both parts come from the **signed `PeerPermanentInfo`** of §6.9 — its serialised payload +bytes followed by its signature bytes, concatenated, and nothing else. + +> **The `SHA256:` prefix is part of the identifier**, not decoration on the escrow label. It is +> what `CuttlefishPeer.hash` carries and what `Voucher.beneficiary` must equal, and it is the same +> string that appears in §5.1's escrow labels. +> +> **This is checkable before anything is sent.** A client can recompute it for every peer already +> in the directory and confirm each matches the `hash` that peer reports. If they do, the +> derivation is right; if not, nothing has been written. Do that first — it converts the one +> irreversible call in this stage into an ordinary one. + +#### `PeerStableInfo` + +Eighteen fields, all optional on the wire, and an incomplete one is **admitted and then behaves +oddly** rather than refused. What a real client sends: + +| Field | Value | +| --- | --- | +| `clock` | **the highest `clock` among all peers in the circle, plus one** — so a first peer sends 1, not 0 | +| `frozenPolicyVersion` | `5` | +| `frozenPolicyHash` | `SHA256:O/ECQlWhvNlLmlDNh2+nal/yekUC87bXpV3k+6kznSo=` | +| `flexiblePolicyVersion` | `20` | +| `flexiblePolicyHash` | `SHA256:OIzjC3WyLGrM8GAd/EyIfVzTJdYmcGoKPFdQeWeRZTY=` | +| `secrets` | empty | +| `osVersion` | this client's OS string | +| `deviceName` | may be empty | +| `serialNumber` | the client's serial, as in [Stage 1 §2.2](./01-authentication.md) | +| `userControllableViewStatus` | `1` | +| `isInheritedAccount` | `false` | + +Everything else is omitted. + +> **The two policy hashes are constants, not something to compute.** They are the digests of +> Apple's own trust policy documents, and a client asserts which policy version it is speaking +> rather than deriving anything. Send them verbatim; a wrong value here is not detectable locally. + +#### `PeerDynamicInfo` + +> **This section replaces an earlier answer that was wrong.** It previously said a joining peer +> sends `clock: 0` and nothing else, and that trust is asserted afterwards by `updateTrust`. That +> is the **`establish`** path — the one this project must never take — and sending it on a join +> produces a peer that is admitted while claiming to trust nobody, which is precisely the +> "accepted, then behaves oddly" failure this section exists to prevent. What follows is the join. + +A joining peer **inherits its sponsor's trust and then adds itself**. It does not invent a trust +set, and it does not send an empty one. + +1. **Sync first.** The peer directory of §5.3 must be current, because everything below reads it. +2. **Copy the sponsor's dynamic info** — the recovered peer that signed the voucher. Take its + `includeds`, its `excludeds` and its `clock` verbatim as the starting point. +3. **Fast-forward over any peer with a higher `clock`.** Take them in ascending `clock` order, and + for each one: adopt every id in its `includeds` not already held, then apply its `excludeds` — + removing each from `includeds` if present before adding it — and take its `clock` as the + current one. +4. **Add this peer's own id to `includeds`** if it is not already there. +5. **Increment `clock` by one**, so the info sent is newer than anything it was derived from. + +> **A peer offering a trust update is not automatically believed.** In step 3, a peer that is not +> already in `includeds` is only adopted if it carries a voucher whose sponsor this peer already +> trusts, whose signature verifies under that sponsor, whose `beneficiary` is the peer presenting +> it, and whose beneficiary is not in `excludeds`. Otherwise its update is ignored. That check is +> what stops an untrusted peer writing itself into the circle by asserting it belongs. + +> **`clock: 0` with everything cleared is still correct in one place**: a client that finds itself +> **not** in the circle resets to it, rather than keeping what it last asserted. It is the reset, +> not the join. + +> **`updateTrust` is a different call for a different purpose.** It is how an established member +> re-states its trust or hands view keys to *another* peer later — not the second half of a join, +> and a join that relies on it to assert trust asserts none. +> +> | # | Field | Type | +> | --- | --- | --- | +> | 1 | `restorePoint` | string | +> | 2 | `peerId` | string | +> | 3 | `stableInfo` | `SignedInfo` | +> | 4 | `dynamicInfo` | `SignedInfo` | +> | 5 | `tlkshares` | repeated `TlkShare` | +> | 6 | `viewKeys` | repeated `ViewKeys` | +> +> Handing a newly-vouched peer its keys is this call with `peerId` and `tlkshares` set and the two +> signed blobs omitted. Its response is the same `CuttlefishChanges` as §6.9.5. + +### 6.9 The messages `joinWithVoucher` carries + +All are protobuf, and all travel inside the invoke envelope of §2.2 — so they are serialised into +a `bytes` field that CloudKit does not inspect. + +**`CuttlefishJoinWithVoucherRequest`** — the request itself: + +| # | Field | Type | +| --- | --- | --- | +| 1 | `restorePoint` | string — the client's sync token, if it has one | +| 2 | `peer` | `CuttlefishPeer` | +| 3 | `bottle` | `Bottle` — the escrow record being created for the new identity | +| 4 | `shares` | repeated `TlkShare` | +| 5 | `keys` | repeated `ViewKeys` | + +`CuttlefishEstablishRequest` is the same message with the fields in a **different order** — peer 1, +bottle 2, keys 3, shares 4 — and no restore point. Another reason not to confuse the two (§6.7). + +**`CuttlefishPeer`** — everything except `hash` is a signed blob: + +| # | Field | Type | +| --- | --- | --- | +| 1 | `hash` | string — the peer's identifier | +| 2 | `permanentInfo` | `SignedInfo` | +| 3 | `stableInfo` | `SignedInfo` | +| 4 | `dynamicInfo` | `SignedInfo` | +| 5 | `voucher` | `SignedInfo` | + +**`SignedInfo`** is the wrapper that makes this work — field 1 `info` (bytes, a serialised message) +and field 2 `signature` (bytes). + +**The signature does not cover `info` alone.** It covers a **type string prepended to it**: + +``` +signed data = ‖ info +``` + +| Blob | Type string | +| --- | --- | +| `permanentInfo` | `TPPB.PeerPermanentInfo` | +| `stableInfo` | `TPPB.PeerStableInfo` | +| `dynamicInfo` | `TPPB.PeerDynamicInfo` | +| `voucher` | `TPPB.Voucher` | + +No separator, no length, no terminator — the ASCII bytes immediately followed by the serialised +message. The prefix is what stops a blob of one kind being presented as another, so omitting it +does not merely fail verification, it removes a protection. + +**And the signature covers the serialised bytes, not the parsed message**, so a client must keep +the exact bytes it signed and send those; re-encoding before sending invalidates the signature +even when the content is identical. + +**`PeerPermanentInfo`** — inside `permanentInfo.info`: + +| # | Field | +| --- | --- | +| 1 | `epoch` (uint64) | +| 2 | `signingKey` (bytes) | +| 3 | `encryptionKey` (bytes) | +| 4 | `machineId` (string) | +| 5 | `modelId` (string) | +| 6 | `creationTime` (uint64) | + +**`PeerStableInfo`** — inside `stableInfo.info`. Larger, and mostly optional: + +| # | Field | | # | Field | +| --- | --- | --- | --- | --- | +| 1 | `clock` | | 10 | `flexiblePolicyVersion` | +| 2 | `frozenPolicyVersion` | | 11 | `flexiblePolicyHash` | +| 3 | `frozenPolicyHash` | | 12 | `userControllableViewStatus` | +| 4 | `secrets` (repeated bytes) | | 13 | `custodianRecoveryKeys` (repeated) | +| 5 | `osVersion` | | 14 | `secureElementIdentity` | +| 6 | `deviceName` | | 15 | `walrus` | +| 7 | `recoverySigningPublicKey` | | 16 | `webAccess` | +| 8 | `recoveryEncryptionPublicKey` | | 18 | `isInheritedAccount` (bool) | +| 9 | `serialNumber` | | | | + +Note field 17 is absent. `deviceName` and `serialNumber` are how this peer appears to the user, so +the labelling rules of the [README](./README.md) apply here as they do to the device registration. + +**`PeerDynamicInfo`** — inside `dynamicInfo.info`, and the actual statement of who trusts whom: + +| # | Field | Type | +| --- | --- | --- | +| 1 | `clock` | uint64 | +| 2 | `includeds` | repeated string — peer ids this peer trusts | +| 3 | `excludeds` | repeated string — peer ids it has removed | +| 4 | `dispositions` | repeated `PeerDisposition` | +| 5 | `preapprovals` | repeated string | + +**`Voucher`** — inside `voucher.info`, and remarkably small for what it does: + +| # | Field | Type | +| --- | --- | --- | +| 1 | `reason` | uint32 | +| 2 | `beneficiary` | string — the peer being vouched **for** | +| 3 | `sponsor` | string — the peer vouching | + +Three fields. The signature on the enclosing `SignedInfo`, made with the recovered peer's signing +key, is the entire weight of the claim. + +**`Bottle`** — the escrow record created for the new identity: + +| # | Field | Type | +| --- | --- | --- | +| 2 | `bottle` | bytes — a serialised `OTBottle` | +| 3 | `escrowedSigningKey` | bytes | +| 4 | `escrowedKeySignature` | bytes | +| 5 | `peerKeySignature` | bytes | +| 6 | `peerID` | string | +| 7 | `bottleID` | string | + +Field 1 is absent. `escrowedKeySignature` and `peerKeySignature` are the two signatures §6.7 step 3 +verifies when recovering. + +**`OTBottle`** — inside `bottle.bottle`: + +| # | Field | Type | +| --- | --- | --- | +| 1 | `peerID` | string | +| 2 | `bottleID` | string — a UUID | +| 8 | `escrowedSigningKey` | bytes | +| 9 | `escrowedEncryptionKey` | bytes | +| 10 | `peerSigningKey` | bytes | +| 11 | `peerEncryptionKey` | bytes | +| 12 | `ciphertext` | message: 1 `ciphertext`, 2 `authenticationCode`, 3 `initializationVector` | + +**Fields 3 to 7 are reserved and must not be used.** The AES-256-GCM unsealing of §6.7 step 4 +takes its three inputs from field 12; note the tag is carried separately from the ciphertext and +must be appended before decryption, or supplied to a detached-tag interface. + +The plaintext is an **`OTInternalBottle`**, which is two fields and both are at surprising numbers: +`signingKey` at **3** and `encryptionKey` at **4**, each an `OTPrivateKey` of `keyType` (1) and +`keyData` (2). There is no field 1 or 2. + +**`TlkShare`** — one per view key being handed to the new peer. Note that almost everything is a +**string**, including things that look like binary: + +| # | Field | Type | | # | Field | Type | +| --- | --- | --- | --- | --- | --- | --- | +| 1 | `service` | string | | 7 | `receiverPublicEncryptionKey` | **string** | +| 2 | `curve` | uint64 | | 8 | `sender` | string | +| 3 | `epoch` | uint64 | | 9 | `signature` | **string** | +| 4 | `keyId` | string | | 10 | `version` | uint32 | +| 5 | `poisoned` | uint32 | | 11 | `wrappedKey` | **string** | +| 6 | `receiver` | string | | | | | + +**`ViewKeys`** — sent when *establishing* keys rather than receiving them, so this project sends an +empty list: `service` (1, string), then `topLevelKey` (2), `classA` (3), `classC` (4) and +`oldTopLevelKey` (5), each a nested `ViewKey`: + +| # | Field | Type | +| --- | --- | --- | +| 1 | `keyId` | string | +| 2 | `topLevelKeyId` | string | +| 3 | `keyNumber` | uint32 | +| 4 | `key` | **string** | +| 5 | `harware` | string — the name is misspelled in every schema examined, and that spelling is the one to use | + +> **Nothing in this project writes one**, so unlike every other table here this one has never been +> exercised in the direction that would check it. `key` being a `string` follows the pattern of +> `TlkShare` and is stated on the same authority as the rest, but the README's own test applies: +> a transcription no writer has ever constructed is weaker than one that has. + +### 6.9.1 Signing, and what the new identity's `PeerPermanentInfo` says + +**Every `SignedInfo` in §6.9 is ECDSA over SHA-384**, made with the signing key of the peer the +blob belongs to, over the type-prefixed bytes above. The one exception in this stage is the share +signature of §6.7.0, which is **SHA-256** — a different construction with a different digest, in +the same message. + +Both of a peer's keys are **P-384**, generated fresh, and both public halves travel as **DER +SPKI**, not as raw points. That matters twice over: `PeerPermanentInfo` carries them, and the +identifier of §6.8.2 is a digest of the signed blob that contains them, so a different encoding is +a different peer. + +| Field | Value | +| --- | --- | +| `epoch` | `1` | +| `signingKey`, `encryptionKey` | the two public keys, **DER SPKI** | +| `machineId` | the Anisette **`X-Apple-I-MD-M`** header — the same machine identity Stage 1 binds the session to | +| `modelId` | the client's hardware model string, as in [Stage 1 §2.2](./01-authentication.md) | +| `creationTime` | **milliseconds** since the Unix epoch, not seconds | + +> **`machineId` ties the peer to the Anisette in use.** Rule 4 of the repo's `AGENTS.md` applies +> here with more force than anywhere else: a peer created under local Anisette and a peer created +> under a server carry different machine ids, permanently, because this blob is signed at creation +> and never rewritten. + +**The permanent info is signed once, when the identity is generated**, and the signed bytes are +kept. Everything afterwards — the identifier, the peer message, the bottle — refers to those +bytes. Re-encoding the message later gives a different signature and a different identifier. + +### 6.9.2 Creating a share + +The inverse of §6.7.0, and the direction a joining peer actually needs. The plaintext is the same +four-field message §6.7.0 step 3 recovers — `uuid`, `zoneName`, `keyclass`, `key` — serialised. + +**A joining peer shares to itself.** Both `sender` and `receiver` are the new peer's identifier, +and the wrapping key is the new peer's own encryption public key. That reads like a no-op and is +not: the keys were recovered from a *different* peer's shares, and Cuttlefish only recognises a +peer as holding a view key when a share addressed to that peer says so. The join is where the +keys are re-addressed from the recovered identity to this one. + +**Wrapping** — SFIES, mirroring §6.7.0 step 3: + +| Step | | +| --- | --- | +| Ephemeral key | a fresh P-384 key pair; its **uncompressed point** is what gets archived | +| Agreement | ECDH with the receiver's public key | +| Derivation | ANSI X9.63, SHA-256, shared info = that same 97-byte point, output 48 bytes | +| Key / nonce | first 32 bytes, then the next 16 — a **16-byte** GCM nonce | +| Cipher | AES-256-GCM, **no AAD** | + +Then split the output: the last 16 bytes are the tag and go in `SFIESAuthenticationCode`; the rest +is the ciphertext. + +> ### Reproduce the overrun, with zeros +> +> §6.7.0 describes `SFCiphertext` carrying 113 bytes past the end of the ciphertext, and a reader +> trimming them. **A writer must produce them**, padding the ciphertext by `len(point) + len(tag)` +> — the same quantity, derived the same way. +> +> This is not politeness towards a quirk. Apple's reader trims by that amount unconditionally, +> because Apple's writer always produces it; a tightly-sized ciphertext would have its last 113 +> real bytes cut off and would fail authentication. +> +> **Pad with zeros.** The bytes Apple leaks there are uninitialised heap, and there is no reason +> to reproduce *that* part of the behaviour. + +Archive the three members under the names of §6.7.0 — including the misspelled one — as a **binary +property list**, and base64 that for `wrappedKey`. + +> **The archived object is a plain dictionary, not a class.** The top level is an ordinary keyed +> archive of a three-entry dictionary; nothing has to declare `$class` naming `SFIESCiphertext` or +> reconstruct its class hierarchy. The only class that matters is the `NSMutableData` on the +> ephemeral point, and that is a property of the member rather than of the envelope. +> +> Apple's side finds the members by name, which is why this works and why the misspelling is fatal +> while the missing class is not. + +**The message's fields**, for a share this client creates: + +| Field | Value | +| --- | --- | +| `service` | the key's zone name | +| `keyId` | the key's UUID — present on the message, absent from the record form | +| `curve` | `4` — P-384 | +| `epoch` | `1` | +| `receiver`, `sender` | the new peer's identifier, both | +| `receiverPublicEncryptionKey` | base64 of the receiver's **uncompressed point**, not its SPKI | +| `wrappedKey` | base64 of the archive | +| `poisoned`, `version` | **omitted** | +| `signature` | base64 of ECDSA-SHA-256 over §6.7.0's concatenation | + +> **`poisoned` and `version` are omitted from the message and signed as zero.** The signature input +> of §6.7.0 has seven parts and none of them may be skipped, so both contribute eight zero bytes +> each to the digest of a message that does not carry them. Building the signed data by walking the +> populated fields produces a signature over five parts, which verifies nowhere. + +### 6.9.3 Building the bottle for the new identity + +Step 5.4's bottle, inverting §6.7 steps 1 to 4. **Fresh entropy** — the 72 bytes of §4.5.3 — with +the three keys derived from it exactly as §6.7 step 2 derives them, `adsid` as salt. + +| Field | Value | +| --- | --- | +| `OTInternalBottle.signingKey`, `.encryptionKey` | the **new peer's private keys**, each an `OTPrivateKey` with `keyType` `1` | +| `keyData` | the **uncompressed public point followed by the private scalar** — 97 + 48 bytes on P-384 | +| `OTBottle.peerID` | the new peer's identifier | +| `.bottleID` | a **v4 UUID, upper-case** | +| `.escrowedSigningKey`, `.escrowedEncryptionKey` | the public halves of the two HKDF-derived keys, **DER SPKI** | +| `.peerSigningKey`, `.peerEncryptionKey` | the new peer's own public keys, DER SPKI | +| `.ciphertext` | the sealed `OTInternalBottle` | + +**The seal is AES-256-GCM under the derived symmetric key, with a 32-byte IV.** Not 12, not 16 — +32 random bytes, carried in `initializationVector`, with the tag split off into +`authenticationCode` as §6.9 describes. + +The outer `Bottle` then carries the serialised `OTBottle` bytes and two signatures over exactly +those bytes, both **ECDSA-SHA-384**, as §6.7 step 3 verifies them: + +- `escrowedKeySignature` — by the **derived escrow signing key** +- `peerKeySignature` — by the **new peer's own signing key** + +> §6.7 step 3 describes the second as verifying "under the sponsoring peer's signing key". For a +> bottle a peer creates for itself those are the same key, and the general rule is the one to +> implement: **the signature is by the peer the bottle belongs to**, which for every bottle this +> project creates is this client. + +**The escrow label is `com.apple.icdp.record.`** — the peer's full identifier, `SHA256:` +prefix and base64 included, appended to the prefix. §5.1's listing shows the same shape from the +read side. + +> **§4.5.2's `escrowedSPKI` is the same value as `OTBottle.escrowedSigningKey`** — the DER SPKI of +> the HKDF-derived escrow *signing* key, not the encryption one and not the peer's own. That +> identity is what ties an escrow record to the bottle it accompanies, and it is why the record +> and the bottle have to be built from **one** derivation rather than from two calls that each +> generate entropy. + +### 6.9.4 The order these happen in, which is not the order they are described in + +**Enrol the escrow record before calling `joinWithVoucher`, not after.** The two failure modes are +not comparable: + +- enrolment fails after joining → a peer in the circle whose entropy was never escrowed. **Nobody + can ever recover it**, there is no listing that shows it, and §7's deletion does not reach it. +- joining fails after enrolment → an escrow record for a peer that does not exist. Visible in the + listing of §5, and **deletable** by §7.1. + +One is permanent and invisible; the other is tidy-up. So the sequence is: + +1. Recover the bottle (§6.7 steps 1–4) → the recovered peer's identity. +2. Generate the new identity (§6.9.1). **This clears any keys and items held under a previous + identity** — they were addressed to a peer that is being replaced. +3. Sign the voucher with the recovered peer, `reason` `1`, `beneficiary` the new peer. +4. Fetch the recovered peer's shares (§6.7.0). **If empty, stop** — §6.7 step 5.3. +5. Build the dynamic info (§6.8.2), which syncs the directory as its first act. +6. Create the bottle **and enrol it** (§6.9.3, §4.5). +7. Re-share the keys to the new peer (§6.9.2). +8. Call `joinWithVoucher` with `restorePoint` set to the sync token if one is held, the peer, the + bottle, the shares, and **`keys` empty** — this project never establishes view keys. +9. Apply the returned changes, persist the dynamic info as sent, and store the keys. + +> ### Never retry `joinWithVoucher` +> +> **A response that fails to decode is not a call that failed.** The peer is in the circle and the +> escrow record exists; what was lost is the change set and the sync token, and both come back +> from re-reading the directory (§5.3). +> +> The reflex on a decode error is to retry, and retrying this one **joins twice** — a second peer, +> a second bottle and a second escrow record, all permanent, none of which the first attempt's +> failure gives any reason to want. Treat any response at all as the join having happened, and +> recover by syncing rather than by repeating. The same goes for a timeout, where it is not even +> known whether a response was sent. + +> **Enrolment can fail because a record already exists under that label**, which is the one case +> worth handling rather than reporting: delete the record at that label and enrol again. Do that +> only for an error the escrow service *reports* — a transport failure has not established that +> anything exists, and deleting on one would remove a record this client cannot see the contents +> of. §7.1 covers the deletion and its residue rules. + +### 6.9.5 What `joinWithVoucher` returns, and the token it carries + +**The response is a defined message, not an opaque blob**, and it is one a client that has synced +the peer directory already decodes: + +| Message | # | Field | +| --- | --- | --- | +| `CuttlefishJoinWithVoucherResponse` | 1 | `changes` — a `CuttlefishChanges` | +| `CuttlefishChanges` | 1 | `syncToken` — **string** | +| | 2 | `changes` — repeated `CuttlefishChange` | + +This is the **same `CuttlefishChanges` that `fetchChanges` returns**, so the join's result goes +through the existing path: apply the changes to the peer directory, and take `syncToken` as the +new sync token. + +> **Leaving it undecoded is a reasonable instinct applied to the wrong case.** Decoding as the +> wrong message yields every field empty rather than an error — the §6.7.0 trap — but that argues +> for decoding as a message that is *specified*, which this one is. Discarding it silently drops +> both the trust changes the server is reporting and the token below. + +**`restorePoint` is that `syncToken`, and it is a string on both sides.** The field is declared +`string` on the request and `string` on the response that produces it, so there is nothing to +encode and nothing to guess: send back exactly what Cuttlefish last sent. A client holding it as +bytes has converted it locally and should stop. + +> Sending it is optional and omitting it costs at most a fuller response — but the reason to hold +> the token is not this call. It is what makes later syncs incremental +> ([§6.7.1](#671-the-steady-state--what-the-passcode-buys)), and the join's response is where a +> new peer first receives one. + +### 6.7.0 Fetching the recovered peer's key shares — and why joining may be unnecessary + +Step 5.3 above says to fetch the recovered peer's key shares. The method is: + +``` +Cuttlefish fetchRecoverableTLKShares +``` + +| Message | # | Field | +| --- | --- | --- | +| Request | 1 | `forPeer` — the **recovered peer's** identifier, which §4.3 shows is already in the record's label | +| Response | 1 | `shares` — repeated `RecoverableTlkShare` | + +**`RecoverableTlkShare`:** + +| # | Field | Type | +| --- | --- | --- | +| 1 | `service` | string — the view this share is for, e.g. `Manatee` | +| 2 | `viewkeys` | `RecoverableViewKeys` — `tlk` (1), `classA` (2), `classB` (3) | +| 3 | `share` | the share itself | + +Each of those four members is a wrapper whose **field 2** holds a **CloudKit `Record`** — the same +`Record` message as [Stage 4 §3.6](./04-cloudkit.md), field numbers and all. + +> ### The share is a CloudKit record, not a protobuf message +> +> **This is the mistake to avoid, and it fails silently.** §6.9 specifies a `TlkShare` *message* +> with numbered fields. That message is what a client **sends** when joining. What it **receives** +> here is entirely different: a CloudKit record of type `tlkshare`, whose values live in +> **named fields** — the `recordField` list of Stage 4 §3.6, each with an `identifier.name` and a +> typed value. +> +> Decoding the record's bytes as a `TlkShare` message does not error. Protobuf skips fields whose +> numbers it does not recognise, so it yields a message with **every field empty** — which looks +> like a share with no sender rather than a parse that found nothing. Forty-one identical empty +> results is the signature of this mistake. +> +> **Read the fields by name.** + +**Record type `tlkshare`** — the share: + +| Field name | Type | +| --- | --- | +| `sender`, `receiver` | string | +| `receiverPublicEncryptionKey` | string | +| `wrappedkey` | string — base64 | +| `signature` | string — base64 | +| `curve`, `epoch`, `poisoned`, `version` | int64 | +| `parentkeyref` | reference | + +**Record type `synckey`** — each of `tlk`, `classA` and `classB`: + +| Field name | Type | +| --- | --- | +| `wrappedkey` | string — base64 | +| `class` | string | +| `uploadver` | string | +| `parentkeyref` | reference | + +> **The record carries no `service` and no `keyId`.** Both exist on the `TlkShare` *message* and +> neither is on the record — the view name comes from `RecoverableTlkShare.service` at field 1, one +> level up, and the record's own identity comes from its `recordIdentifier`. Looking for them among +> the record's fields finds nothing, correctly. + +**The important property: a share is wrapped to the *receiving* peer's encryption key.** Escrow +recovery (§6.7 step 4) yields exactly that key for the recovered peer — so **the shares it can +receive are shares this client can unwrap**, with nothing further required. + +Unwrapping one: + +1. **Verify the sender.** Each share carries a signature over a concatenation of its own fields, + in this order and no other: + + | Order | Field | Encoding | + | --- | --- | --- | + | 1 | `version` | **64-bit little-endian** | + | 2 | `receiver` | UTF-8 bytes | + | 3 | `sender` | UTF-8 bytes | + | 4 | `wrappedKey` | the **base64-decoded** bytes, not the string | + | 5 | `curve` | **64-bit little-endian** | + | 6 | `epoch` | **64-bit little-endian** | + | 7 | `poisoned` | **64-bit little-endian** | + + > **All four integers are eight bytes**, `version` and `poisoned` included — even though the + > §6.9 *message* declares those two as `uint32` and the CloudKit record carries them as its own + > integer type. The signed form is not either of those forms. + > + > Earlier revisions of this table said 32-bit for those two. That is wrong in the way that costs + > most: both are zero on real shares, so the digest is taken over four extra zero bytes, the + > check fails, and step 1 **skips the share**. The symptom is a peer that appears to have no + > shares to give, not a verification error. + > + > **The integers are little-endian.** Everything else in this protocol — CloudKit, the KeyVault + > framing, the PCS structures — is big-endian. This one construction is not, and getting it + > wrong produces a verification failure with no other symptom. + + Verify with **ECDSA over SHA-256** against the sender's signing key from the directory of §5.3. + A share whose sender is not in the directory is to be skipped, not trusted. +2. **Decode the wrapped key.** It is base64 of an **`NSKeyedArchiver` archive**, which expands to + an ECIES ciphertext structure. Another encoding appearing nowhere else in this protocol. + + > **[observed] The archive holds the three parts separately, under named members** — a reader + > expecting one concatenated blob finds nothing to slice. On a real share: + > + > | Member | Size | What it is | + > | --- | --- | --- | + > | `SFEphemeralSenderPublicKeyExternaRepresentation` | 97 B | the uncompressed P-384 point | + > | `SFCiphertext` | 230 B | the body | + > | `SFIESAuthenticationCode` | 16 B | the tag | + > + > **The first name is misspelled, and that spelling is the wire format** — `Externa`, no final + > `l`. Apple's own writer produced it and Apple's own reader looks for it. A reader can be + > forgiving and match on the fragment `EphemeralSenderPublicKey`; **a writer cannot**, because + > the correctly-spelled name is one Apple's unarchiver will not find the ephemeral key under, + > and the failure arrives as a share that authenticates against nothing. + > + > That member is archived as **`NSMutableData`**; the other two are plain `NSData`. + > + > **These are Apple's `SFIESCiphertext` from SecurityFoundation**, so the construction is a + > named framework class rather than anything bespoke to this protocol. **No initialisation + > vector is archived** — it is derived, as §3 below sets out. + + ### The ciphertext member is longer than the ciphertext + > + > **This is the whole trap, and no amount of trying cipher parameters reaches it.** + > + > `SFCiphertext` carries the ciphertext followed by **113 bytes that are not part of it** — + > 97 + 16, the sizes of the ephemeral point and the authentication code. The producer sizes + > that buffer to hold all three parts, writes only the ciphertext into it, and archives the + > whole buffer without trimming it back. + > + > **So the ciphertext is the member with its last 113 bytes discarded.** For the 230-byte + > member observed above, 117 bytes are real. Passing all 230 to the cipher fails + > authentication exactly as a wrong key or a wrong parameter would, which is why a search + > over parameters can run to any size and never succeed. + > + > Derive the 113 from the sizes of the other two members rather than writing the constant — + > it is a point length plus a tag length, and a curve other than P-384 changes it. + > + > The trailing bytes are **uninitialised heap**, so they are whatever was in that memory. + > Do not read them, do not log them, and do not treat them as a field. + +3. **Decrypt with the recovered peer's encryption private key.** + + | Step | | + | --- | --- | + | Agreement | ECDH between the recovered encryption private key and the archived ephemeral point. **Plain, not cofactor** — P-384's cofactor is 1 | + | Derivation | **ANSI X9.63**, **SHA-256**, 4-byte big-endian counter from 1, **shared info = the 97-byte ephemeral point** exactly as archived. Output **48 bytes** | + | Key | the **first 32** bytes — AES-256 | + | Nonce | the **next 16** bytes. **A 16-byte GCM nonce, not 12** | + | AAD | **none** | + | Cipher | AES-256-GCM over the trimmed ciphertext, with `SFIESAuthenticationCode` as the tag | + + The plaintext is a protobuf message: + + | # | Field | Type | + | --- | --- | --- | + | 1 | `uuid` | string | + | 2 | `zoneName` | string | + | 3 | `keyclass` | string | + | 4 | `key` | **bytes** — the key material, and the thing this whole stage is for | + + > **`key` is the one `bytes` field**, against the pattern of §6.9 where binary-looking values + > are declared `string`. The three around it are strings. + > + > A writer should not need this table: §6.9.2 re-shares this message **byte for byte** as it was + > recovered, reading only `zoneName` and `uuid` out of it. Re-encoding would change the bytes the + > receiver's key is bound to even when every field is identical, so pass-through is the rule + > regardless of what the types are. + + > **This message is the view's top-level key.** It is not a container the real key is nested + > inside, and `key` is **symmetric key material, not an EC private key** — trying to read it as + > one fails for a reason nothing announces. + +4. **Unwrap the class keys the entry carries.** `viewkeys` holds `classA` and `classB` as `synckey` + records, and their `wrappedkey` is wrapped **under the key from step 3**. + + > **These are not ECIES**, and this is the second place a reader reasonably assumes the + > construction carries over from the step before. They are **AES-256-CMAC-SIV** + > ([RFC 5297](https://www.rfc-editor.org/rfc/rfc5297), CMAC rather than PMAC), keyed with the + > `key` bytes from step 3, with **no associated data** — an empty vector of headers, which is + > not the same as one empty header. + > + > **`tlk` is not unwrapped here.** Step 3's plaintext *is* the top-level key, so an entry's + > `tlk` member is not a fourth thing to decrypt. Keep the step 3 message alongside the two + > class keys. + + Each class key becomes a key of the same shape as step 3's message: the `synckey` record's own + `recordIdentifier` as the UUID, the zone name carried over from step 3, the record's `class`, + and the unwrapped bytes. + + Keep all three. The top-level key alone is not what Stage 5 matches against. + +> ### This may remove every write from the flow +> +> §6.7 presents joining as the way to obtain keys, and for a device that wants to *be* in the +> circle it is. But this project only wants to **read**, and the keys arrive at step 5.3 — +> **before** any peer is created, any voucher signed, any bottle enrolled, or anything sent to +> Cuttlefish. +> +> If the shares unwrap and yield the `Manatee` view keys, then steps 5.4 and 5.5 and everything in +> §6.9 are **unnecessary for reading**, and the flow becomes: recover the bottle, unwrap the +> shares, decrypt. **No peer, no escrow record, nothing written to the user's account.** +> +> That is a materially better position than this document has assumed throughout — it would make +> the whole feature read-only apart from the device registration of +> [Stage 1 §13](./01-authentication.md), and it removes the artefact the residue rules of §7 exist +> to manage. +> +> **It is not yet established**, and the cheap way to find out is the order above: fetch the +> shares, unwrap them, and see whether the keys they yield satisfy §6.8's contract. If they do, +> stop there. **Nothing in steps 1 to 5.3 writes anything**, so trying costs a passcode and no +> account change. +> +> **What would make joining necessary anyway:** wanting to *stay* in the circle across key +> rotations. A non-member receives no new shares, so a client that never joins may find its keys +> going stale and have to recover again. That is a real cost, but it is a later one, and it trades +> against never writing to the account at all. + +### 6.7.1 The steady state — what the passcode buys + +**The device passcode is needed once, not per fetch.** Everything this stage establishes persists, +and ordinary use afterwards never touches it again. That is what makes the feature a connection +rather than a one-time import, and it is what lets a newly-paired accessory be noticed later. + +**What persists locally, and must be stored as carefully as any key:** + +| | | +| --- | --- | +| the **new peer identity** — its signing and encryption private keys | this client's membership of the circle | +| the **keychain view keys** synced from `Manatee` and `ProtectedCloudStorage` | what [Stage 5](./05-pcs-decryption.md) decrypts with | +| the trust **sync token** | so later syncs are incremental | +| the CloudKit **continuation token** | so later fetches are incremental — see [Stage 4 §3.5](./04-cloudkit.md) | + +**What persists on the account:** the device registration of [Stage 1 §13](./01-authentication.md), +and the escrow record this stage creates for the new identity. + +**What an ordinary later run does:** + +1. Stage 1 — re-authenticate. **Silent**: no second factor, because the machine is registered. +2. Stage 2 — refresh the service tokens if they have aged out, roughly weekly. +3. Stage 4 — fetch changes from the continuation token. +4. Stage 5 — decrypt with the keys already held. + +No passcode, no escrow call, no join. **Stage 3 does not run at all.** + +**What forces a re-sync, but not a passcode:** a record whose protection structure names a key not +held. Keys roll — that is what `rollCount` and the fallback signature in +[Stage 5 §3.1](./05-pcs-decryption.md) are for — so sync the two views again and retry once before +concluding anything is wrong. This is a cache miss, not a failure. + +**What forces the whole stage again:** losing the local state above, or the user removing this +client from the circle. Both are recoverable; neither is silent. + +### 6.7.2 Losing local state need not cost another passcode + +The bottle this client creates for itself (§6.7 step 5) is sealed under a **password the client +chooses**, not under the user's device passcode. + +So if local state is lost and that password was kept, the client recovers **from its own bottle** +and the user is never asked for anything. If the password was not kept, or the record was deleted, +recovery falls back to a first-party device's bottle and the user's device passcode again. + +> ### A generated password stored locally is not a recovery path +> +> The obvious implementation — generate a random password, keep it with the peer identity — reads +> as self-recovery and **is not**. The case it exists for is *losing local state*, and the password +> is in that state. Where the client still holds the password it also still holds the identity, so +> there is nothing to recover; where it has lost the identity it has lost the password too. +> +> **Nothing here is wasted, but it is worth being clear what it buys.** Enrolment is what stops +> the peer being permanently unrecoverable by *anyone* — §4.5 — so the record earns its place +> regardless. What it does not do is spare the user a passcode after a reinstall. +> +> A password that would survive has to come from outside that state: shown once as a recovery code +> the user keeps, or derived from something they can re-supply. Both are real options and both cost +> the user something, so **treat the device passcode as the fallback and say so**, rather than +> implying a self-recovery that only works in the case where it is not needed. +> +> [Stage 1 §13](./01-authentication.md)'s device registration is the reason the fallback is +> tolerable: re-authenticating is silent, so recovering after a reinstall costs a passcode and not +> a second-factor round trip. + +> **This sharpens the trade in §7.** Deleting the client's own escrow record removes the artefact +> from the user's account and removes the client's ability to recover itself. Keeping it means one +> more record on an account that already accumulates them. There is no free option, and the choice +> should be the user's rather than a default nobody explained. +> +> **The chosen password must be stored as securely as the keys it protects**, because it is +> equivalent to them: anyone holding it and the record can reconstitute this client's membership of +> the keychain circle. + +### 6.8 What this stage must actually deliver + +[Stage 5 §2](./05-pcs-decryption.md) makes the output contract concrete, and it is narrower than +"join the circle": + +| Required | Detail | +| --- | --- | +| The **`Manatee`** keychain view, synced | the view holding Find My's Protected Cloud Storage keys | +| The **`ProtectedCloudStorage`** view, synced | synced alongside it | +| The service key within `Manatee` | the item labelled `com.apple.ProtectedCloudStorage-com.apple.icloud.searchparty` | +| Items readable as dictionaries | key lookup matches on the `acct` attribute, which holds base64 of a compressed EC public key | + +So success for this stage is not "we are in the circle" but "we can enumerate `Manatee` and find +an item whose `acct` matches a key referenced by a record's protection structure". That is the +thing to test against. + +### 6.8.1 From a view's keys to its items + +§6.7.0 ends holding three keys per view. This is the step that turns them into the dictionary +§6.8 asks for, and **none of it is a search** — every unknown below is a name or a field number. + +**The container is not the one the shares came from.** + +| | | +| --- | --- | +| Container | `com.apple.security.keychain` — the same as Cuttlefish | +| Bundle | **`com.apple.securityd`** — *not* `com.apple.security.cuttlefish` | +| Database | private | +| Environment | production | + +> Same container, different bundle. A client that reuses the Cuttlefish container here is asking +> the wrong service. + +**The zone is the view name.** `Manatee` is a private zone in that container, as is +`ProtectedCloudStorage`. Enumerate it with the same `RetrieveChanges` paging as +[Stage 4 §3.7](./04-cloudkit.md), keeping the continuation token per zone. + +Two record types matter, and the rest can be ignored: + +| Type | What it is | +| --- | --- | +| `item` | one encrypted keychain item | +| `currentitem` | a **named pointer** to an `item` | + +#### The pointer is how the Find My key is found + +A `currentitem` record carries a single field, `item` — a **weak reference** whose record +identifier names an `item` record. Its own record identifier is the **tag**, and the tag for this +project's key is: + +``` +com.apple.ProtectedCloudStorage-com.apple.icloud.searchparty +``` + +**So the service key is a direct lookup, not a scan**: read that `currentitem` in the `Manatee` +zone, follow its reference, decrypt that `item`. The `acct` match of §6.8 is the *other* way in — +used when a record's protection structure names a key and the item holding it must be found — and +both are worth having, because they fail differently. + +#### The `item` record + +| Field | Type | | +| --- | --- | --- | +| `data` | bytes | the encrypted item | +| `wrappedkey` | string | base64 — this item's own key, wrapped | +| `parentkeyref` | reference | **which key unwraps it** | +| `encver` | int64 | 1 or 2, and it changes the additional data | +| `gen` | int64 | | +| `uploadver` | string | | +| `pcsservice` | int64 | optional | +| `pcspublickey` | bytes | optional — compressed | +| `pcspublicidentity` | bytes | optional | +| `server_wascurrent`, `server_suggestDeletion` | int64 | optional, server-set | + +**`parentkeyref` answers "which of the three keys".** Its record identifier is a key **UUID**, +matched against the `uuid` of the keys §6.7.0 produced — the share's own key and the two class +keys. It is not a class name, so `classA`/`classB` are not what to match on. + +#### Decrypting one + +1. **Unwrap the item's own key.** AES-256-CMAC-SIV, keyed with the §6.7.0 key that `parentkeyref` + names, over the base64-decoded `wrappedkey`, **no associated data**. Yields 64 bytes. +2. **Split `data`.** The **first 16 bytes are an initialisation vector**; the ciphertext is the + rest. +3. **Decrypt** with AES-256-CMAC-SIV under the key from step 1. SIV takes a *vector of headers*, + and here it is **the IV first, then the additional data values** of the next section, in order. +4. **Strip the padding.** The plaintext is padded to a multiple of **20 bytes**: walk backwards + over trailing zero bytes to the first **`0x80`**, and truncate there. That byte is the marker, + not data. Anything else encountered before it means the decryption is wrong. +5. **Parse** what remains as a **binary property list**. The result is the item dictionary. + +#### The additional data, which is where this goes wrong + +A sorted map of name to bytes; **only the values are passed**, in order of their names. + +For `encver` **1**, exactly four entries. For anything else, those four plus the optional PCS +fields that are present, plus **every other field on the record**: + +| Name | Value | +| --- | --- | +| `UUID` | the record's own identifier, as UTF-8 | +| `encver` | **64-bit little-endian** | +| `gen` | **64-bit little-endian** | +| `wrappedkey` | **the parent key's UUID as UTF-8** — *not* the `wrappedkey` field's value | +| `pcsservice` | 64-bit little-endian, if present | +| `pcspublicidentity`, `pcspublickey` | as-is, if present | + +> **`wrappedkey` in the additional data is not the `wrappedkey` field.** It is the identifier +> `parentkeyref` points at. The name is reused for two different things one line apart, and using +> the wrapped key itself produces an authentication failure with nothing to say why. + +Every remaining field on the record joins the map under its own name, **except** the ten already +spoken for — `gen`, `pcspublickey`, `UUID`, `data`, `pcsservice`, `pcspublicidentity`, +`parentkeyref`, `uploadver`, `wrappedkey`, `encver` — and **except anything beginning with +`server_`**. Their values are rendered by type: + +| Type | Rendering | +| --- | --- | +| string | UTF-8 | +| bytes | as-is | +| int64 | **64-bit little-endian** | +| double | **cast to an integer, then 64-bit little-endian** | +| date | **RFC 3339, whole seconds, `Z`** — e.g. `2026-08-14T09:23:41Z` | + +> Little-endian again, as in §6.7.0's signature, and against everything else in this protocol. +> +> **The map is sorted by name and the names are then discarded.** Only the values are passed, so +> the sort is the only thing that puts them in the right order — and a map that preserves +> insertion order instead produces a wrong order that is stable, repeatable, and wrong on every +> item. + +#### What the dictionary holds + +Ordinary keychain attributes — `class`, `acct`, `agrp`, `vwht`, `pdmn`, `atyp`, `labl`, `srvr`, +`cdat`, `mdat`, `sha1`, `musr`, `tomb` — and **`v_Data`**, which is the payload. + +For the service key, `acct` is base64 of the **compressed** public key, `agrp` is +`com.apple.ProtectedCloudStorage`, `vwht` is `Manatee`, and `v_Data` is a **DER-encoded private +key structure** — the thing [Stage 5](./05-pcs-decryption.md) needs. + +**That structure is a CHOICE of two forms**, and Find My's service uses the second: + +| Form | Encoding | +| --- | --- | +| V1 | a sequence: the key octets, and optionally the public structure | +| V2 | **`[APPLICATION 5]`**, wrapping a protobuf carrying an **encryption key** and a **signing key**, each as compressed private key bytes with an optional DER public structure | + +**Find My's service key is V2.** Its `com.apple.icloud.searchparty` service is type **82**, view +hint and zone both `Manatee`. + +> ### The key octets are 64 bytes, and the public half comes first +> +> **This layout is shared by both forms** — V1's octets and each of V2's protobuf key fields are +> written the same way, so a single reader serves both: +> +> | Bytes | Content | +> | --- | --- | +> | 0–31 | the **public x coordinate** | +> | 32–63 | the **private scalar** | +> +> **Public first.** Reading the leading 32 bytes as the scalar does not fail — it yields a +> perfectly valid key on the curve, with a public x that matches nothing. Everything downstream +> then reports "no key held" rather than "wrong key", because the two are indistinguishable once +> the wrong key is a valid one. +> +> The check that catches it costs nothing and is worth doing at the point of parse: **derive the +> public x from the scalar and confirm it equals bytes 0–31**. The blob carries both halves +> precisely so that this is possible. +> +> **A 32-byte blob also occurs**, and is the scalar alone with no public half — so dispatch on +> length rather than assuming either. At 32 bytes there is nothing to check against. + +> **This is the join Stage 5 was missing.** A 64-byte AES-SIV view key never becomes an EC private +> key, and no derivation turns one into the other. The view key decrypts an *item*; the item's +> `v_Data` **contains** the EC private key. Two different things, one step apart. + +> **Passcode handling.** It must be read, used, and discarded. Never written to disk, never +> logged, never included in diagnostics, and not retained in memory past the exchange. It is the +> single most sensitive value this project ever touches: it is the key to the user's entire +> keychain, and unlike an Apple ID password it cannot be rotated without physical access to the +> device. + +## 7. What this stage creates, and the rules for it + +Joining the circle enrolls **an escrow record of this client's own**, via `enroll`. That is a +second artefact on the user's account, after the device registration of Stage 1 §13. + +The [README](./README.md) sets out the standing rules. Applied here: + +- **One record, created once.** Never a new one per run. +- **It inherits the labelling for free.** A record created with the identity of Stage 1 §2.2 + carries the same recognisable serial, so it identifies itself by the same means the device + entry does — provided the identity is consistent, which it must be anyway. +- **It is insurance, not access.** Its purpose is to let the client re-establish trust if it + loses local state. Once trust is established and the peer identity is stored locally, the + record can be deleted — at the cost of needing the user's passcode again should that local + state ever be lost. This is the opposite lifetime to the device registration, which is kept + precisely because it is what keeps ordinary access working. +- **Deletion must confirm against the serial.** The list contains the user's *real* devices, and + every entry looks structurally alike. Destroying the wrong record destroys that device's + ability to recover its keychain. Require the user to re-enter the serial of the record being + deleted; never accept a list position. +- **Never delete without listing first**, and never offer a "clean up all" action. + +## 7.1 Deleting an escrow record + +Deletion is **two calls**, both `delete`, in this order: + +| Order | `label` | +| --- | --- | +| 1 | `