diff --git a/.claude/skills/build-insight-dashboard/SKILL.md b/.claude/skills/build-insight-dashboard/SKILL.md index e4e1f80c..85b23077 100644 --- a/.claude/skills/build-insight-dashboard/SKILL.md +++ b/.claude/skills/build-insight-dashboard/SKILL.md @@ -140,7 +140,7 @@ One round: the first suspect is the cache and the first move is to clear it — before re-reading the config, before doubting the change, before restarting anything. Runs lose whole stretches to this by debugging the edit instead, - and it is the same symptom as the un-restarted server (`rounds.md`): what you + and it is the same symptom as the un-restarted server (`scaffold-workflow-app/rounds.md`): what you see is the previous configuration, faithfully served. 2. **Resolve every candidate to its edit before offering it.** In thinking, name the file and the change that implements it. A candidate whose edit you cannot @@ -292,7 +292,7 @@ print(page.frame_locator("iframe").first.locator("body") Only then shorten headers if the total overflows. Three builds on two notebooks had the 45px figure and the observed panel fits and still converged by trial across four and five gate cycles, because nothing here turned a panel width into a column width. A fourth had the formula and converged anyway, because the formula is only the numerator. -**And do not buy the fit by dropping a column.** The same build removed one to make the sum work and disclosed it in its closing turn, which is the one place `handover.md` says a decision must never first appear. A column that does not fit is a question, not a silent edit. +**And do not buy the fit by dropping a column.** The same build removed one to make the sum work and disclosed it in its closing turn, which is the one place `scaffold-workflow-app/handover.md` says a decision must never first appear. A column that does not fit is a question, not a silent edit. **The gate does not see the wrap.** A build reported *"875 wrapped onto a phantom row, 800 clipped 'Score', 815 fit"* and added that only the screenshot showed it — the gate returned 11/11 on the wrapped page. So a green gate is not evidence the table fits; the screenshot is. Read it before believing the count, and never treat a passing gate as the check for this. This rule used to name a fixed figure for a 1280px window until two builds on two notebooks sized their columns against it and wrapped anyway: one measured ~830px at a 1280px window, the other 864px with the wrap at a 865px sum. Both then re-derived the budget from a screenshot and repeated gate runs, which is the cost of a number that was one layout's measurement rather than something to compute. `verify-webapp-usability` measures the real property and fails on it; shorten a long header rather than widening its column | | **Normalise per side before a mirror plot** | The halves share one symmetric axis; raw counts (~7e4) against theoretical intensities (~1) flatten one onto the baseline | diff --git a/.claude/skills/interview-parameters/probe.py b/.claude/skills/interview-parameters/probe.py index e5a4fc89..e1208bb6 100644 --- a/.claude/skills/interview-parameters/probe.py +++ b/.claude/skills/interview-parameters/probe.py @@ -294,7 +294,7 @@ def main() -> int: if ( re_eff > MATERIAL_THRESHOLD and re_eff >= MASKING_RATIO * max(eff, EFFECT_THRESHOLD) - and (best_unmask is None or re_eff > best_unmask[1]) + and (best_unmask is None or re_eff > best_unmask[1]) # pylint: disable=unsubscriptable-object ): best_unmask = (other["key"], re_eff) if best_unmask: diff --git a/.claude/skills/notebook-to-webapp/SKILL.md b/.claude/skills/notebook-to-webapp/SKILL.md index 02c74351..03c50cdd 100644 --- a/.claude/skills/notebook-to-webapp/SKILL.md +++ b/.claude/skills/notebook-to-webapp/SKILL.md @@ -77,7 +77,7 @@ Four lines. The same build narrated step by step runs to eight: a line for the extracted file, one for the golden test, one for the workflow class and its pages, one for registering them, one for the template cleanup, one for the test lists. Each is a step inside a stage, and the last of them reads a phrase -straight out of `cleanup.md` — a step that only exists because this framework +straight out of `scaffold-workflow-app/cleanup.md` — a step that only exists because this framework has a rule about it is not a step the user is waiting on. **The bad version is described here and not written out, deliberately.** It used diff --git a/.github/scripts/ci-assertions.sh b/.github/scripts/ci-assertions.sh index aeb9d15a..22e6fa2a 100644 --- a/.github/scripts/ci-assertions.sh +++ b/.github/scripts/ci-assertions.sh @@ -916,6 +916,7 @@ assert_doc_links_resolve() { # `docs/a16-storage-runbook.md`, while a sibling reference inside docs/ # may reasonably write just `build_app.md`. _ci_dangling="" + _ci_dangling_at="" _ci_checked=0 for _ci_ref in $_ci_refs; do _ci_checked=$((_ci_checked + 1)) @@ -925,14 +926,36 @@ assert_doc_links_resolve() { if git ls-files --error-unmatch "docs/$_ci_ref" >/dev/null 2>&1; then continue fi + # Skills cite their own supporting files by bare name (`cleanup.md`) + # or relative to the skills root (`scaffold-workflow-app/handover.md`). + if git ls-files --error-unmatch ".claude/skills/$_ci_ref" >/dev/null 2>&1; then + continue + fi + # Last, relative to the directory of the Markdown file citing it. Every + # citing file has to resolve it on its own: one file with a sibling of + # that name says nothing about another file citing the same name. + _ci_unresolved="" + for _ci_citer in $(git ls-files '*.md' -z 2>/dev/null \ + | xargs -0 grep -l -- "\`$_ci_ref\`" 2>/dev/null); do + if ! git ls-files --error-unmatch "$(dirname "$_ci_citer")/$_ci_ref" >/dev/null 2>&1; then + _ci_unresolved="$_ci_unresolved $_ci_citer" + fi + done + if [ -z "$_ci_unresolved" ]; then + continue + fi _ci_dangling="$_ci_dangling $_ci_ref" + for _ci_citer in $_ci_unresolved; do + _ci_dangling_at="$_ci_dangling_at $_ci_citer:$_ci_ref" + done done if [ -n "$_ci_dangling" ]; then _ci_fail "tracked Markdown cites .md paths that do not exist:$_ci_dangling" - printf '%s\n' "$_ci_dangling" | tr ' ' '\n' | grep -v '^$' | while read -r _ci_d; do - git ls-files '*.md' -z 2>/dev/null \ - | xargs -0 grep -n -- "\`$_ci_d\`" 2>/dev/null | sed 's|^| |' >&2 || true + # Only the citing files that cannot resolve the reference. + for _ci_d in $_ci_dangling_at; do + grep -n -- "\`${_ci_d#*:}\`" "${_ci_d%%:*}" 2>/dev/null \ + | sed "s|^| ${_ci_d%%:*}:|" >&2 || true done _ci_rc=1 fi diff --git a/.github/workflows/build-windows-executable-app.yaml b/.github/workflows/build-windows-executable-app.yaml index 03748875..92326306 100644 --- a/.github/workflows/build-windows-executable-app.yaml +++ b/.github/workflows/build-windows-executable-app.yaml @@ -13,10 +13,6 @@ permissions: contents: write env: - OPENMS_VERSION: 3.5.0 - # Override to pin contrib to a specific release when OPENMS_VERSION is a non-release branch (e.g. "develop"). - # Leave empty to use OPENMS_VERSION. - OPENMS_CONTRIB_VERSION: "" PYTHON_VERSION: 3.11.0 # Name of the installer APP_NAME: OpenMS-StreamlitTemplateApp @@ -26,160 +22,28 @@ env: TOPP_TOOLS: "FeatureFinderMetabo FeatureLinkerUnlabeledKD SiriusExport" jobs: - build-openms: - runs-on: windows-2022 - + # `with:` of a reusable-workflow call cannot read `env`, so this job hands + # TOPP_TOOLS over as an output and it stays defined in one place. + config: + runs-on: ubuntu-latest + outputs: + topp-tools: ${{ steps.out.outputs.topp-tools }} steps: - - name: Checkout - uses: actions/checkout@v4 - with: - repository: OpenMS/OpenMS - ref: release/${{ env.OPENMS_VERSION }} - path: 'OpenMS' - - - name: Install Qt (Windows) - uses: jurplel/install-qt-action@v4 - with: - version: '6.8.3' ## Note this version is build with win64_msvc2022_64 and should always match what we use - arch: 'win64_msvc2022_64' - cache: 'false' - archives: 'qtsvg qtimageformats qtbase' - - # https://github.com/marketplace/actions/visual-studio-shell - - name: Set up Visual Studio shell - uses: egor-tensin/vs-shell@v2 - with: - arch: x64 + - id: out + run: echo "topp-tools=$TOPP_TOOLS" >> "$GITHUB_OUTPUT" - - name: Setup build tools - shell: bash - run: | - choco install ccache ninja -y --no-progress - choco install cmake --version=3.31.12 -y --no-progress --force - ## GH CLI "SHOULD BE" installed. Sometimes I had to manually install nonetheless. Super weird. - # https://github.com/actions/runner-images/blob/main/images/win/scripts/Installers/Install-GitHub-CLI.ps1 - echo "C:/Program Files (x86)/GitHub CLI" >> $GITHUB_PATH - - - name: Extract branch/PR infos - shell: bash - run: | - cd OpenMS - RUN_NAME_LOCAL=$(echo ${GITHUB_REF#refs/heads/} | tr / -) - echo "RUN_NAME=${RUN_NAME_LOCAL}" >> $GITHUB_ENV - echo "BASE_REF=$(gh pr view --json baseRefName -q .baseRefName || echo ${RUN_NAME_LOCAL})" >> $GITHUB_ENV - id: extract_branch - env: - GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} - - - name: Cache contrib - id: cache-contrib-win - uses: actions/cache@v4 - with: - path: ${{ github.workspace }}/OpenMS/contrib - key: ${{ runner.os }}-contrib-${{ env.OPENMS_CONTRIB_VERSION || env.OPENMS_VERSION }} - - - name: Load contrib build - if: steps.cache-contrib-win.outputs.cache-hit != 'true' - env: - GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} - run: | - cd OpenMS/contrib - # Download the file using the URL fetched from GitHub - gh release download release/${{ env.OPENMS_CONTRIB_VERSION || env.OPENMS_VERSION }} -R OpenMS/contrib --pattern 'contrib_build-Windows.tar.gz' - # Extract the archive - 7z x -so contrib_build-Windows.tar.gz | 7z x -si -ttar - rm contrib_build-Windows.tar.gz - ls - - - name: Add contrib to PATH - shell: bash - run: | - # Add contrib library path for runtime DLL resolution - echo "${{ github.workspace }}/OpenMS/contrib/lib" >> $GITHUB_PATH - - - name: Setup ccache cache - uses: actions/cache@v4 - with: - path: .ccache - key: ${{ runner.os }}-ccache-${{ env.RUN_NAME }}-${{ github.run_number }} - # Restoring: From current branch, otherwise from base branch, otherwise from any branch. - restore-keys: | - ${{ runner.os }}-ccache-${{ env.RUN_NAME }} - ${{ runner.os }}-ccache-${{ env.BASE_REF }} - ${{ runner.os }}-ccache- - - - name: Add THIRDPARTY - shell: bash - run: | - # initialize THIRDPARTY - cd OpenMS - git submodule update --init THIRDPARTY - cd .. - # add third-party binaries to PATH - # use flat THIRDPARTY structure - mkdir -p _thirdparty - cp -R OpenMS/THIRDPARTY/Windows/x86_64/* _thirdparty/ - cp -R OpenMS/THIRDPARTY/All/* _thirdparty/ - # add third-party binaries to PATH - for thirdpartytool in ${{ github.workspace }}/_thirdparty/* - do - echo $thirdpartytool >> $GITHUB_PATH - done - - - name: Build Windows - shell: bash - run: | - mkdir $GITHUB_WORKSPACE/OpenMS/bld/ - bash OpenMS/tools/ci/capture-env.sh -v $GITHUB_WORKSPACE/OpenMS/bld/CMakeCache.txt - ctest --output-on-failure -V -S $GITHUB_WORKSPACE/OpenMS/tools/ci/cibuild.cmake - env: - OPENMS_CONTRIB_LIBS: "${{ github.workspace }}/OpenMS/contrib" - CI_PROVIDER: "GitHub-Actions" - CMAKE_GENERATOR: "Ninja" - SOURCE_DIRECTORY: "${{ github.workspace }}/OpenMS" - BUILD_NAME: "${{ env.RUN_NAME }}-Win64-class-topp-${{ github.run_number }}" - ENABLE_STYLE_TESTING: "OFF" - ENABLE_TOPP_TESTING: "ON" - ENABLE_CLASS_TESTING: "ON" - WITH_GUI: "OFF" - WITH_PARQUET: "OFF" - ADDRESS_SANITIZER: "Off" - BUILD_TYPE: "Release" - OPENMP: "Off" - USE_STATIC_BOOST: "On" - # BUILD_FLAGS: "-p:CL_MPCount=2" # For VS Generator and MSBuild - BUILD_FLAGS: "-j2" # Ninja will otherwise use all cores (doesn't go well in GHA) - CMAKE_CCACHE_EXE: "ccache" - CCACHE_BASEDIR: ${{ github.workspace }} - CCACHE_DIR: ${{ github.workspace }}/.ccache - CCACHE_COMPRESS: true - CCACHE_COMPRESSLEVEL: 12 - CCACHE_MAXSIZE: 400M - - - name: Test Windows - shell: bash - run: ctest --output-on-failure -V -S $GITHUB_WORKSPACE/OpenMS/tools/ci/citest.cmake - env: - SOURCE_DIRECTORY: "${{ github.workspace }}/OpenMS" - CI_PROVIDER: "GitHub-Actions" - BUILD_NAME: "${{ env.RUN_NAME }}-Win64-class-topp-${{ github.run_number }}" - - - name: Package - shell: bash - run: | - ctest --output-on-failure -V -S $GITHUB_WORKSPACE/OpenMS/tools/ci/cipackage.cmake - env: - SOURCE_DIRECTORY: "${{ github.workspace }}/OpenMS" - PACKAGE_TYPE: zip - SEARCH_ENGINES_DIRECTORY: "${{ github.workspace }}/_thirdparty" - CI_PROVIDER: "GitHub-Actions" - CPACK_PACKAGE_FILE_NAME: "openms-package" - - - name: Upload package as artifact - uses: actions/upload-artifact@v4 - with: - name: openms-package - path: ${{ github.workspace }}/OpenMS/bld/*.zip + # OpenMS TOPP tools, DLLs and share/ from the official Windows installer. + # See docs/openms-windows-workflow.md for the inputs, including `mode: source` + # for building a branch or fork. App repositories call the same workflow as + # OpenMS/streamlit-template/.github/workflows/openms-windows.yml@. + build-openms: + needs: config + uses: ./.github/workflows/openms-windows.yml + with: + openms-version: "3.5.0" + mode: installer + topp-tools: ${{ needs.config.outputs.topp-tools }} + artifact-name: openms-package build-executable: runs-on: windows-2022 diff --git a/.github/workflows/openms-windows.yml b/.github/workflows/openms-windows.yml new file mode 100644 index 00000000..9aed74e1 --- /dev/null +++ b/.github/workflows/openms-windows.yml @@ -0,0 +1,484 @@ +# Reusable workflow: OpenMS TOPP tools for Windows, packaged for a web app installer. +# +# Produces the artifact that the `build-executable` job of +# build-windows-executable-app.yaml consumes: a single zip whose top level is +# +# openms-package/bin/ TOPP tool .exe files and every DLL they need +# openms-package/share/ share/OpenMS, including share/OpenMS/THIRDPARTY/ +# +# Two ways to get there: +# +# mode: installer (default) +# Takes the official Windows installer attached to the OpenMS GitHub release +# of `openms-version`, installs it silently and repackages bin/ and share/. +# No compiler, no contrib. The official installers are tested upstream, so no +# OpenMS tests run here -- only a `--help` smoke test of the kept tools. +# +# mode: source +# Builds `openms-repository`@`openms-ref` with the vcpkg CMake presets +# (`windows-x64-release`) and packages it with CPack's ZIP generator. Needs +# OpenMS sources that carry the vcpkg presets: OpenMS 3.6 and later, and +# develop from about 2026-08 on. A fork of an older develop must rebase first. +# vcpkg binaries and ccache are cached, but a cold build takes hours. +# +# Usage from an app repository: see docs/openms-windows-workflow.md. + +name: OpenMS for Windows + +on: + workflow_call: + inputs: + openms-version: + description: >- + OpenMS release to package, e.g. "3.5.0". In installer mode this selects + the GitHub release; in source mode it only supplies the default ref + (release/). + type: string + default: "3.5.0" + mode: + description: '"installer" (repackage the official installer) or "source" (build with vcpkg).' + type: string + default: installer + openms-repository: + description: Repository whose releases (installer mode) or sources (source mode) are used. + type: string + default: OpenMS/OpenMS + openms-ref: + description: Branch, tag or SHA to build in source mode. Empty means release/. + type: string + default: "" + topp-tools: + description: >- + Space-separated TOPP tools to keep, e.g. "FeatureFinderMetabo SiriusExport". + Every listed tool must exist. Empty keeps all tools. DLLs and share/ are + always kept in full. + type: string + default: "" + artifact-name: + description: Name of the uploaded artifact. + type: string + default: openms-package + runs-on: + description: Windows runner label. + type: string + default: windows-2025 + outputs: + artifact-name: + description: Name of the uploaded artifact. + value: ${{ jobs.openms.outputs.artifact-name }} + +jobs: + openms: + name: OpenMS (${{ inputs.mode }}) + runs-on: ${{ inputs.runs-on }} + # A cold source build compiles the vcpkg dependencies (Arrow, Boost, ...) and + # OpenMS itself; installer mode finishes in minutes. + timeout-minutes: ${{ inputs.mode == 'source' && 360 || 60 }} + permissions: + contents: read + outputs: + artifact-name: ${{ inputs.artifact-name }} + env: + MODE: ${{ inputs.mode }} + OPENMS_VERSION: ${{ inputs.openms-version }} + OPENMS_REPOSITORY: ${{ inputs.openms-repository }} + TOPP_TOOLS: ${{ inputs.topp-tools }} + + steps: + - name: Validate inputs + shell: bash + run: | + case "$MODE" in + installer|source) ;; + *) echo "::error::mode must be 'installer' or 'source', got '$MODE'"; exit 1 ;; + esac + if [[ "$MODE" == installer && -z "$OPENMS_VERSION" ]]; then + echo "::error::installer mode needs openms-version"; exit 1 + fi + + ############################################################################ + # installer mode + ############################################################################ + - name: Download the official installer + if: inputs.mode == 'installer' + id: download + shell: bash + env: + GH_TOKEN: ${{ github.token }} + run: | + set -euo pipefail + # Tag schemes, newest first: v3.6.0 (release.yml pushes v* tags from 3.6 + # on), release/3.5.0 ... release/3.2.0, Release3.0.0. + tag="" + for candidate in "v${OPENMS_VERSION}" "release/${OPENMS_VERSION}" "Release${OPENMS_VERSION}"; do + if gh release view "$candidate" -R "$OPENMS_REPOSITORY" --json tagName >/dev/null 2>&1; then + tag="$candidate" + break + fi + done + if [[ -z "$tag" ]]; then + echo "::error::No release for OpenMS ${OPENMS_VERSION} in ${OPENMS_REPOSITORY} (tried v${OPENMS_VERSION}, release/${OPENMS_VERSION}, Release${OPENMS_VERSION})" + exit 1 + fi + + # The release carries the installers of every platform; the Windows one is + # the only .exe (CPack names it OpenMS--Win64.exe). + mapfile -t exes < <(gh release view "$tag" -R "$OPENMS_REPOSITORY" --json assets -q '.assets[].name' | grep -i '\.exe$' || true) + asset="" + if [[ ${#exes[@]} -eq 1 ]]; then + asset="${exes[0]}" + else + for e in "${exes[@]}"; do + if [[ "$e" =~ [Ww]in(dows)?[-_]?(64|x64|x86_64) ]]; then asset="$e"; break; fi + done + fi + if [[ -z "$asset" ]]; then + echo "::error::Could not pick the Windows installer of release '$tag'. .exe assets: ${exes[*]:-none}" + exit 1 + fi + + mkdir -p "$RUNNER_TEMP/installer" + gh release download "$tag" -R "$OPENMS_REPOSITORY" --pattern "$asset" --dir "$RUNNER_TEMP/installer" --clobber + echo "Release: $tag, asset: $asset" + { + echo "tag=$tag" + echo "asset=$asset" + echo "path=$(cygpath -w "$RUNNER_TEMP/installer/$asset")" + } >> "$GITHUB_OUTPUT" + + - name: Install silently (7-Zip extraction as fallback) + if: inputs.mode == 'installer' + shell: pwsh + env: + INSTALLER: ${{ steps.download.outputs.path }} + run: | + $ErrorActionPreference = 'Stop' + # A short path without spaces: NSIS takes /D= unquoted and as the last argument. + $dir = 'C:\OpenMS-install' + if (Test-Path $dir) { Remove-Item -Recurse -Force $dir } + + $installed = $false + # /allusers is required by the OpenMS installer in silent mode. + $p = Start-Process -FilePath $env:INSTALLER -ArgumentList '/S', '/allusers', "/D=$dir" -PassThru + if ($p.WaitForExit(15 * 60 * 1000)) { + Write-Host "Installer exit code: $($p.ExitCode)" + $tools = @(Get-ChildItem -Path "$dir\bin" -Filter *.exe -ErrorAction SilentlyContinue) + if ($p.ExitCode -eq 0 -and $tools.Count -gt 0) { $installed = $true } + } else { + Write-Host "::warning::Silent install did not finish within 15 minutes; killing it." + $p | Stop-Process -Force -ErrorAction SilentlyContinue + } + + $root = $dir + if (-not $installed) { + Write-Host "::warning::Silent install failed; extracting the installer with 7-Zip instead." + $x = 'C:\OpenMS-extract' + if (Test-Path $x) { Remove-Item -Recurse -Force $x } + 7z x $env:INSTALLER "-o$x" -y | Out-Null + if ($LASTEXITCODE -ne 0) { throw "7-Zip extraction failed ($LASTEXITCODE)" } + # Depending on the 7-Zip version, NSIS $INSTDIR content lands at the top or under '$INSTDIR'. + $root = if (Test-Path "$x\`$INSTDIR\bin") { "$x\`$INSTDIR" } else { $x } + } + if (-not (Test-Path "$root\bin")) { throw "No bin\ directory under $root" } + "RAW_ROOT=$root" | Out-File -FilePath $env:GITHUB_ENV -Append -Encoding utf8 + Get-ChildItem $root | Format-Table -AutoSize | Out-String | Write-Host + + ############################################################################ + # source mode + ############################################################################ + - name: Resolve the ref + if: inputs.mode == 'source' + id: ref + shell: bash + env: + OPENMS_REF: ${{ inputs.openms-ref }} + run: echo "ref=${OPENMS_REF:-release/${OPENMS_VERSION}}" >> "$GITHUB_OUTPUT" + + # Checked out at the workspace root, as OpenMS's own CI does: vcpkg's build + # trees are deep, and Windows paths are limited to 260 characters. + - name: Checkout OpenMS + if: inputs.mode == 'source' + uses: actions/checkout@v4 + with: + repository: ${{ inputs.openms-repository }} + ref: ${{ steps.ref.outputs.ref }} + persist-credentials: false + + - name: Check for the vcpkg presets + if: inputs.mode == 'source' + shell: bash + env: + OPENMS_REF: ${{ steps.ref.outputs.ref }} + run: | + if ! grep -q '"windows-x64-release"' CMakePresets.json 2>/dev/null || [[ ! -f vcpkg.json ]]; then + echo "::error::${OPENMS_REPOSITORY}@${OPENMS_REF} has no vcpkg CMake presets (CMakePresets.json with windows-x64-release, vcpkg.json)." + echo "::error::Source mode needs OpenMS 3.6+ or a develop from about 2026-08 on. Rebase the fork, or use installer mode for a released version." + exit 1 + fi + + # Only these two submodules: contrib is what this workflow replaces. + - name: Fetch the vcpkg and THIRDPARTY submodules + if: inputs.mode == 'source' + id: submodules + shell: bash + run: | + set -euo pipefail + git submodule update --init --depth 1 THIRDPARTY + # vcpkg resolves its baseline from the submodule's git history, so no --depth. + git submodule update --init vcpkg + echo "vcpkg=$(git rev-parse HEAD:vcpkg)" >> "$GITHUB_OUTPUT" + + - name: Flatten THIRDPARTY for packaging + if: inputs.mode == 'source' + shell: bash + run: | + mkdir -p _thirdparty + cp -R THIRDPARTY/Windows/x86_64/* _thirdparty/ + cp -R THIRDPARTY/All/* _thirdparty/ + ls _thirdparty + + - name: Set up a Visual Studio shell + if: inputs.mode == 'source' + uses: egor-tensin/vs-shell@v2 + with: + arch: x64 + + - name: Install build tools + if: inputs.mode == 'source' + shell: bash + run: | + choco install -y --no-progress ninja ccache + cmake --version + ninja --version + ccache --version + + - name: Restore the vcpkg binary cache + if: inputs.mode == 'source' + id: vcpkg-cache + uses: actions/cache/restore@v4 + with: + path: ${{ github.workspace }}/.vcpkg-cache + key: openms-win-vcpkg-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('vcpkg.json', 'vcpkg-configuration.json', 'vcpkg-overlays/**') }}-${{ steps.submodules.outputs.vcpkg }} + # The files provider keys every package by its ABI hash, so an older cache + # is safe to start from: whatever still matches is reused. + restore-keys: | + openms-win-vcpkg-${{ runner.os }}-${{ runner.arch }}- + + - name: Restore the ccache cache + if: inputs.mode == 'source' + id: ccache-cache + uses: actions/cache/restore@v4 + with: + path: ${{ github.workspace }}/.ccache + key: openms-win-ccache-${{ inputs.openms-repository }}-${{ steps.ref.outputs.ref }}-${{ github.run_id }}-${{ github.run_attempt }} + restore-keys: | + openms-win-ccache-${{ inputs.openms-repository }}-${{ steps.ref.outputs.ref }}- + openms-win-ccache-${{ inputs.openms-repository }}- + openms-win-ccache- + + - name: Configure + if: inputs.mode == 'source' + id: configure + shell: bash + run: | + set -euxo pipefail + mkdir -p "$GITHUB_WORKSPACE/.vcpkg-cache" "$GITHUB_WORKSPACE/.ccache" + { + echo "VCPKG_BINARY_SOURCES=clear;files,$GITHUB_WORKSPACE/.vcpkg-cache,readwrite" + echo "CCACHE_DIR=$GITHUB_WORKSPACE/.ccache" + echo "CCACHE_BASEDIR=$GITHUB_WORKSPACE" + echo "CCACHE_COMPRESS=true" + echo "CCACHE_MAXSIZE=2G" + } >> "$GITHUB_ENV" + export VCPKG_BINARY_SOURCES="clear;files,$GITHUB_WORKSPACE/.vcpkg-cache,readwrite" + export CCACHE_DIR="$GITHUB_WORKSPACE/.ccache" CCACHE_BASEDIR="$GITHUB_WORKSPACE" + + ./vcpkg/bootstrap-vcpkg.bat -disableMetrics + + # A TOPP-only build. The optional dependencies match OpenMS's windows-x64-ci + # preset, i.e. what the official installer ships, minus GUI, docs and tests. + cmake --preset windows-x64-release \ + -DCMAKE_C_COMPILER_LAUNCHER=ccache \ + -DCMAKE_CXX_COMPILER_LAUNCHER=ccache \ + -DWITH_GUI=OFF \ + -DENABLE_DOCS=OFF \ + -DENABLE_TUTORIALS=OFF \ + -DHAS_XSERVER=OFF \ + -DPYOPENMS=OFF \ + -DWITH_HDF5=OFF \ + -DWITH_ONNX=OFF \ + -DWITH_WNETALIGN=OFF \ + -DWITH_THERMO_RAW=ON \ + -DWITH_OPENTIMS=ON \ + -DUSE_EXTERNAL_JSON=ON \ + -DUSE_EXTERNAL_SQLITECPP=ON \ + -DUSE_EXTERNAL_SIMDE=ON \ + -DUSE_EXTERNAL_ISOSPEC=ON \ + -DUSE_EXTERNAL_EOLBSPLINE=ON \ + "-DVCPKG_MANIFEST_FEATURES=isospec;eol-bspline;simde;nlohmann-json;sqlitecpp;openms-thermo-bridge;opentims" \ + -DENABLE_CLASS_TESTING=OFF \ + -DENABLE_TOPP_TESTING=OFF \ + -DENABLE_PIPELINE_TESTING=OFF + + # Saved as soon as configure (which runs `vcpkg install`) succeeded, so a + # failing compile does not throw away hours of dependency builds. + - name: Save the vcpkg binary cache + if: always() && inputs.mode == 'source' && steps.configure.outcome == 'success' && steps.vcpkg-cache.outputs.cache-hit != 'true' + uses: actions/cache/save@v4 + with: + path: ${{ github.workspace }}/.vcpkg-cache + key: ${{ steps.vcpkg-cache.outputs.cache-primary-key }} + + - name: Build + if: inputs.mode == 'source' + id: build + shell: bash + run: | + set -euxo pipefail + ccache --zero-stats + # Two jobs keep the MSVC compiler within the hosted runner's memory. + cmake --build --preset windows-x64-release --parallel 2 + ccache --show-stats || true + + - name: Save the ccache cache + if: always() && inputs.mode == 'source' && steps.build.outcome == 'success' + uses: actions/cache/save@v4 + with: + path: ${{ github.workspace }}/.ccache + key: ${{ steps.ccache-cache.outputs.cache-primary-key }} + + - name: Package with CPack (ZIP) + if: inputs.mode == 'source' + shell: bash + env: + # Names the zip and its top-level folder (cmake/package_nsis.cmake). + CPACK_PACKAGE_FILE_NAME: openms-package + run: | + set -euxo pipefail + # Stage in a short directory: CPack's default under the build tree can + # exceed MAX_PATH (see OpenMS's release.yml). + rm -rf /c/cpk && mkdir -p /c/cpk + cmake -S . -B build/windows-x64-release \ + -DPACKAGE_TYPE=zip \ + "-DSEARCH_ENGINES_DIRECTORY=$GITHUB_WORKSPACE/_thirdparty" \ + -DCPACK_PACKAGE_DIRECTORY=C:/cpk + cmake --build build/windows-x64-release --target dist + rm -rf "$RUNNER_TEMP/cpack-out" && mkdir -p "$RUNNER_TEMP/cpack-out" + 7z x /c/cpk/openms-package.zip -o"$RUNNER_TEMP/cpack-out" -y > /dev/null + echo "RAW_ROOT=$(cygpath -m "$RUNNER_TEMP/cpack-out/openms-package")" >> "$GITHUB_ENV" + + ############################################################################ + # both modes: repackage, check, smoke test, upload + ############################################################################ + - name: Repackage bin/ and share/ + shell: bash + # Passed through the environment, never expanded into the script: a Git + # ref or a release asset name may contain `$(...)` or backticks. + env: + OPENMS_REF: ${{ steps.ref.outputs.ref }} + RELEASE_TAG: ${{ steps.download.outputs.tag }} + RELEASE_ASSET: ${{ steps.download.outputs.asset }} + run: | + set -euo pipefail + raw="$(cygpath -u "$RAW_ROOT")" + pkg="$RUNNER_TEMP/staging/openms-package" + rm -rf "$RUNNER_TEMP/staging" && mkdir -p "$pkg/bin" + + [[ -d "$raw/bin" ]] || { echo "::error::$RAW_ROOT has no bin/"; exit 1; } + [[ -d "$raw/share/OpenMS" ]] || { echo "::error::$RAW_ROOT has no share/OpenMS/"; exit 1; } + + cp "$raw"/bin/*.dll "$pkg/bin/" + # Qt plugins (e.g. the SQLite driver) where the build ships them. + if [[ -d "$raw/bin/plugins" ]]; then cp -R "$raw/bin/plugins" "$pkg/bin/"; fi + + if [[ -n "${TOPP_TOOLS// /}" ]]; then + missing=() + for tool in $TOPP_TOOLS; do + if [[ -f "$raw/bin/$tool.exe" ]]; then cp "$raw/bin/$tool.exe" "$pkg/bin/"; else missing+=("$tool"); fi + done + if [[ ${#missing[@]} -gt 0 ]]; then + echo "::error::TOPP tools not in this OpenMS package: ${missing[*]}" + exit 1 + fi + else + cp "$raw"/bin/*.exe "$pkg/bin/" + fi + + cp -R "$raw/share" "$pkg/share" + # HTML documentation is not needed by the tools and only bloats the app installer. + rm -rf "$pkg/share/doc" + + # The .bat of the app installer puts every share\OpenMS\THIRDPARTY\* folder on PATH. + shopt -s nullglob + engines=("$pkg"/share/OpenMS/THIRDPARTY/*/) + if [[ ${#engines[@]} -eq 0 ]]; then + echo "::error::share/OpenMS/THIRDPARTY is missing or empty; the search engine adapters would not work" + exit 1 + fi + + # The MSVC and OpenMP runtimes. OpenMS packages normally carry them + # (InstallRequiredSystemLibraries); if one does not, take the runner's copy + # so the app does not depend on a VC++ redistributable on the user's machine. + for dll in vcruntime140.dll vcruntime140_1.dll msvcp140.dll msvcp140_1.dll msvcp140_2.dll concrt140.dll vcomp140.dll; do + if [[ ! -f "$pkg/bin/$dll" && -f "/c/Windows/System32/$dll" ]]; then + cp "/c/Windows/System32/$dll" "$pkg/bin/" + echo "::notice::Added $dll from the runner (the OpenMS package did not ship it)" + fi + done + + n_exe=$(ls "$pkg"/bin/*.exe | wc -l) + n_dll=$(ls "$pkg"/bin/*.dll | wc -l) + { + echo "### OpenMS for Windows ($MODE)" + if [[ "$MODE" == installer ]]; then + echo "- Source: \`${OPENMS_REPOSITORY}\` release \`${RELEASE_TAG}\`, asset \`${RELEASE_ASSET}\`" + else + echo "- Source: \`${OPENMS_REPOSITORY}@${OPENMS_REF}\` (\`$(git rev-parse HEAD 2>/dev/null || echo ?)\`)" + fi + echo "- ${n_exe} TOPP tools, ${n_dll} DLLs" + echo "- THIRDPARTY: $(for e in "${engines[@]}"; do basename "$e"; done | tr '\n' ' ')" + } >> "$GITHUB_STEP_SUMMARY" + cat "$GITHUB_STEP_SUMMARY" + + - name: Smoke-test the TOPP tools + shell: bash + run: | + set -euo pipefail + pkg="$RUNNER_TEMP/staging/openms-package" + tools="${TOPP_TOOLS// /}" + if [[ -n "$tools" ]]; then + tools="$TOPP_TOOLS" + elif [[ -f "$pkg/bin/FeatureFinderMetabo.exe" ]]; then + tools="FeatureFinderMetabo" + else + tools="$(basename "$(ls "$pkg"/bin/*.exe | head -1)" .exe)" + fi + export OPENMS_DATA_PATH="$(cygpath -w "$pkg/share/OpenMS")" + for tool in $tools; do + echo "::group::$tool --help" + # A bare system PATH: every DLL must come from the package itself, not + # from the install directory or anything else on the runner. + if ! PATH="/c/Windows/System32:/c/Windows" "$pkg/bin/$tool.exe" --help; then + echo "::endgroup::" + echo "::error::$tool --help failed; a DLL is probably missing from bin/" + exit 1 + fi + echo "::endgroup::" + done + + - name: Zip the package + shell: bash + run: | + set -euo pipefail + cd "$RUNNER_TEMP/staging" + # build-executable unzips "*.zip" and expects openms-package/{bin,share} inside. + 7z a -tzip -mx=5 "$RUNNER_TEMP/openms-package.zip" openms-package > /dev/null + ls -l "$RUNNER_TEMP/openms-package.zip" + + - name: Upload the package + uses: actions/upload-artifact@v4 + with: + name: ${{ inputs.artifact-name }} + path: ${{ runner.temp }}/openms-package.zip + compression-level: 0 + if-no-files-found: error diff --git a/CLAUDE.md b/CLAUDE.md index 37e4c048..c9af905b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -221,6 +221,7 @@ Downstream apps also use **OpenMS-Insight** (`Table`, `LinePlot`, `Heatmap`, `Vo - `docs/*.md` is **single-sourced**: `content/documentation.py` reads those files and renders them in-app. Editing a doc changes the page. `docs/toppframework.py` is executable doc content — its *code samples* stay current because they are pulled from live source via `getsource()`/`st.help()`, but its surrounding prose does not (see gotchas). - `test_gui.py::test_documentation` parametrizes over the exact selectbox labels in `documentation.py` — **renaming a doc chapter breaks CI** unless both are updated. +- Windows installer: `build-windows-executable-app.yaml` gets the TOPP tools from the reusable `.github/workflows/openms-windows.yml` (`workflow_call`; `installer` mode repackages the official OpenMS installer, `source` mode builds with the vcpkg presets). Its artifact `openms-package` is a zip of `openms-package/{bin,share}`. The six app repos call it too, so its inputs and artifact layout are a public interface — see `docs/openms-windows-workflow.md`. - Four Dockerfiles: `Dockerfile` (full, builds OpenMS `release/3.5.0` + TOPP tools from source) and `Dockerfile_simple` (pyOpenMS via pip only), each with an `.arm` variant. All use `docker/entrypoint.sh`. - Kubernetes: `k8s/base` → `k8s/components/memory-tier-{low,high}` → `k8s/overlays/prod` (sets `namePrefix`, GHCR image, Traefik hosts, `REDIS_URL`), plus the separate `k8s/storage/` root. The `memory-tier-*` components are **pod sizing only** — a tier is how big a worker is, not which node it runs on, and `requests == limits` there is what makes the worker Guaranteed QoS. **Nothing in `k8s/` pins a pod to a node**: no `nodeSelector`, no `nodeName`, no `nodeAffinity`, no `openms.de/memory-tier` labels. The scheduler places pods, and `rq-worker` runs a fixed replica count spread over `kubernetes.io/hostname` with `maxSkew: 1`. CI validates with kubeconform, asserts those invariants statically (`.github/scripts/ci-assertions.sh`), and runs kind integration tests against both nginx and Traefik ingress; the kind jobs apply `k8s/overlays/ci/`, which is `prod` with the worker shrunk to fit a runner. - Env/secrets: `WORKSPACES_DIR`, `REDIS_URL`, `STREAMLIT_SERVER_COUNT` (>1 puts nginx in front of N Streamlit instances), and `st.secrets["admin"]["password"]` from `.streamlit/secrets.toml` (mounted at `/app/admin-secrets/secrets.toml` in k8s) gating save-as-demo. @@ -276,7 +277,7 @@ Two carry runnable tools: `interview-parameters/probe.py` (measures what each pa **The skills never name the evaluation corpus.** Rules are earned by running three specific notebooks, so worked examples drift toward quoting them — at which point an agent handed one of those notebooks can read its expected answer out of the skill it is being measured against. Measured values are written as placeholders (` of spectra`, `df.head()`, `cell `) and filled from the run at hand. `eval/generality.py` fails a skill for naming a corpus notebook, an identifier lifted from one, or an outcome measured on one. -Design and rationale: `docs/notebook-to-webapp-design.md`, `docs/adr/0001..0005`, glossary in `CONTEXT.md`. The user-facing walkthrough is `docs/notebook_to_webapp.md`, rendered in-app. `eval/` holds the self-improvement loop (`eval/README.md`), the hand-built baseline evidence (`eval/baseline-task2.md`), artifact scoring (`run_eval.py`), blind pairwise guidance judging (`judge.py`), the user-facing-turn check (`register.py`), the corpus-leak check (`generality.py`) and the check that generated apps obey the docs (`conformance.py`). Artifacts are the regression guard and are saturated at 1.00; guidance is what a tick is decided on. +Design and rationale: `docs/notebook-to-webapp-design.md`, `docs/adr/0001..0005`, glossary in `CONTEXT.md`. The user-facing walkthrough is `docs/notebook_to_webapp.md`, rendered in-app. `eval/` holds the self-improvement loop, the hand-built baseline evidence, artifact scoring (`run_eval.py`), blind pairwise guidance judging (`judge.py`), the user-facing-turn check (`register.py`), the corpus-leak check (`generality.py`) and the check that generated apps obey the docs (`conformance.py`). Artifacts are the regression guard and are saturated at 1.00; guidance is what a tick is decided on. `eval/` is not committed to this repository, so none of these files are present in a checkout. ## Conventions and gotchas diff --git a/README.md b/README.md index 5a1db4b2..b8808ac6 100644 --- a/README.md +++ b/README.md @@ -10,7 +10,7 @@ This repository contains a template app for OpenMS workflows in a web applicatio - Persistent parameters and input files within a workspace - local and online mode - Captcha control -- Packaged executables for Windows +- Packaged executables for Windows (OpenMS TOPP tools via the reusable [`openms-windows.yml`](docs/openms-windows-workflow.md) workflow) - framework for workflows with OpenMS TOPP tools - Deployment [with docker-compose](https://github.com/OpenMS/streamlit-deployment) diff --git a/docs/notebook-to-webapp-design.md b/docs/notebook-to-webapp-design.md index 5e739199..5b8353d2 100644 --- a/docs/notebook-to-webapp-design.md +++ b/docs/notebook-to-webapp-design.md @@ -286,8 +286,9 @@ This rule exists because the naive version fails on the hero example. Task 2's `absolute_tolerance` moves candidate counts by 0.7% as shipped — because `relative_tolerance=0.1` opens a ±169 Da window that swamps it — yet it is the most important parameter in the notebook. Inert-in-isolation and genuinely-inert -are different findings and must not share a recommendation. Evidence: -[`eval/baseline-task2.md`](../eval/baseline-task2.md). +are different findings and must not share a recommendation. The evidence is +the hand-built task 2 baseline in the `eval/` loop, which is not committed to +this repository. Each row resolves to one of: **shown**, **advanced** (`"advanced": True`), **hardcoded** (absent from `DEFAULTS`), or **dashboard control** (a display-time diff --git a/docs/openms-windows-workflow.md b/docs/openms-windows-workflow.md new file mode 100644 index 00000000..69a8ebd5 --- /dev/null +++ b/docs/openms-windows-workflow.md @@ -0,0 +1,68 @@ +## OpenMS for Windows: the reusable workflow + +`.github/workflows/openms-windows.yml` is a [reusable workflow](https://docs.github.com/en/actions/sharing-automations/reusing-workflows) that provides the OpenMS TOPP tools for a Windows app installer. It uploads one artifact (default name `openms-package`) holding a zip whose top level is: + +``` +openms-package/bin/ TOPP tool .exe files and every DLL they need +openms-package/share/ share/OpenMS, including share/OpenMS/THIRDPARTY/ +``` + +This is the layout the `build-executable` job of `build-windows-executable-app.yaml` has always consumed, so an app switching to the reusable workflow keeps that job unchanged. + +It replaces the old `build-openms` job, which compiled OpenMS against the `contrib` dependency archive with `tools/ci/cibuild.cmake`. Neither exists any more for OpenMS 3.6 and develop: OpenMS builds its dependencies with vcpkg (OpenMS/OpenMS#10327). + +### Calling it from an app + +```yaml +jobs: + build-openms: + uses: OpenMS/streamlit-template/.github/workflows/openms-windows.yml@ + with: + openms-version: "3.5.0" + topp-tools: "FeatureFinderMetabo FeatureLinkerUnlabeledKD SiriusExport" + + build-executable: + runs-on: windows-2022 + needs: build-openms + steps: + # ... unchanged: download the `openms-package` artifact, unzip it, copy + # openms-package/bin and openms-package/share into the installer. +``` + +`with:` of a reusable-workflow call cannot read the workflow's `env`. To keep `TOPP_TOOLS` defined once in `env`, pass it through a small job output, as this repository's `build-windows-executable-app.yaml` does. + +A reusable workflow gets the caller's `GITHUB_TOKEN`; it only needs `contents: read`. + +### Inputs + +| input | default | meaning | +|---|---|---| +| `openms-version` | `3.5.0` | Release to package. Installer mode looks up the GitHub release; source mode uses it only for the default ref. | +| `mode` | `installer` | `installer`: repackage the official Windows installer. `source`: build OpenMS with vcpkg. | +| `openms-repository` | `OpenMS/OpenMS` | Repository whose releases (installer) or sources (source) are used. | +| `openms-ref` | `release/` | Branch, tag or SHA to build in source mode. | +| `topp-tools` | *(all)* | Space-separated TOPP tools to keep. Each must exist, or the job fails. DLLs and `share/` are always kept whole. | +| `artifact-name` | `openms-package` | Name of the uploaded artifact (also an output). | +| `runs-on` | `windows-2025` | Windows runner label. | + +### Installer mode (default) + +Downloads the Windows installer (`*.exe`) from the OpenMS GitHub release and installs it silently (`/S /allusers /D=...`); if that fails, it extracts the installer with 7-Zip instead. Releases are found under the tags `v` (3.6 on), `release/` (3.2.0 to 3.5.0) and `Release` (3.0.0). + +Nothing is compiled, so this takes minutes. It runs no OpenMS tests, because the official installers are tested upstream; it does run ` --help` for every kept tool (or `FeatureFinderMetabo` when all are kept) with a bare system `PATH`, which catches a DLL missing from `bin/`. + +Use it for every app that ships a released OpenMS. + +### Source mode + +For apps that ship a branch or a fork, for example `mode: source`, `openms-repository: /OpenMS`, `openms-ref: my-feature`. + +It checks out the ref with the `vcpkg` and `THIRDPARTY` submodules (not `contrib`), configures with `cmake --preset windows-x64-release` plus a TOPP-only set of options (`WITH_GUI=OFF`, `ENABLE_DOCS=OFF`, no tests; the optional dependencies match the official installer's), builds, and packages with CPack's `ZIP` generator, with `THIRDPARTY` as `SEARCH_ENGINES_DIRECTORY`. + +**Source mode needs OpenMS sources with the vcpkg CMake presets**: OpenMS 3.6 and later, and develop from about 2026-08 on. A fork based on an older develop must rebase first; the job stops with that message if `CMakePresets.json` has no `windows-x64-release` preset. + +The vcpkg binaries (a `files` binary source, keyed on `vcpkg.json`, `vcpkg-configuration.json`, the overlays and the vcpkg commit) and ccache are cached with `actions/cache`, in the calling repository's cache. Measured on `windows-2025` against OpenMS develop: a cold run (empty caches) takes about 2 h 50 min, 1 h 23 min of it building the vcpkg dependencies and 1 h 24 min compiling OpenMS; a warm run takes about 8 min. The job allows 6 h, so the first run has headroom; the caches are per repository and branch scope, so each app pays the cold run once. + +### Which `` to use + +Pin a ref that does not move under you: a commit SHA of `OpenMS/streamlit-template`, or a tag, if the maintainers publish one for this workflow. `@main` works, but picks up every change to the workflow the moment it is merged. Dependabot's `github-actions` ecosystem keeps a SHA or tag pin up to date.