FunVIP 1.0.0 merge

.github/workflows/build-ete4-windows-wheels.yml

@@ -0,0 +1,80 @@
+name: build-ete4-windows-wheels
+
+# Build Windows (win_amd64) wheels for ete4, which upstream does NOT ship for
+# Windows (PyPI has an sdist only; conda-forge has no win-64 build). ete4 is a
+# hard FunVIP dependency, so without a Windows wheel FunVIP cannot be pip/conda
+# installed on Windows at all. This applies the small etetoolkit/ete PR #783
+# path-separator fix (tools/ete4-windows/ete4-windows.patch) and builds wheels
+# with cibuildwheel.
+#
+# Trigger manually: Actions tab -> "build-ete4-windows-wheels" -> "Run workflow".
+# The built wheels are uploaded as a downloadable artifact.
+#
+# ete4 is GPL-3.0-or-later; if you redistribute these wheels, also publish the
+# corresponding (patched) source and keep ete4's LICENSE. See
+# tools/ete4-windows/README.md.
+
+on:
+  workflow_dispatch:
+    inputs:
+      ete4_version:
+        description: "ete4 version to build Windows wheels for"
+        required: true
+        default: "4.4.0"
+
+jobs:
+  build-windows-wheels:
+    name: ete4 ${{ github.event.inputs.ete4_version }} win_amd64
+    runs-on: windows-latest
+    steps:
+      - name: Checkout FunVIP (for the patch)
+        uses: actions/checkout@v4
+
+      - name: Set up host Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      - name: Download and unpack the ete4 source
+        shell: bash
+        run: |
+          set -euxo pipefail
+          VER="${{ github.event.inputs.ete4_version }}"
+          python -m pip install --upgrade pip
+          python -m pip download "ete4==${VER}" --no-deps --no-binary :all: -d sdist
+          tar -xzf "sdist/ete4-${VER}.tar.gz"
+          echo "ETE4_SRC=ete4-${VER}" >> "$GITHUB_ENV"
+
+      - name: Apply the Windows build patch (etetoolkit/ete PR #783)
+        shell: bash
+        run: |
+          set -euxo pipefail
+          cd "$ETE4_SRC"
+          git init -q                     # give `git apply` a worktree to operate in
+          git apply --verbose "../tools/ete4-windows/ete4-windows.patch"
+          echo "--- patched lines ---"
+          grep -n "replace(sep" setup.py
+          grep -n "expanduser" ete4/config.py
+
+      - name: Build wheels with cibuildwheel
+        shell: bash
+        env:
+          # FunVIP-supported CPythons; narrow this to speed up a first "does it
+          # compile?" check (e.g. just cp312-win_amd64).
+          CIBW_BUILD: "cp310-win_amd64 cp311-win_amd64 cp312-win_amd64 cp313-win_amd64"
+          CIBW_BUILD_VERBOSITY: "1"
+          # Import test loads the compiled Cython .pyd -- proves the wheel actually
+          # works on Windows, not merely that it compiled. Single-quoted Python
+          # strings keep this robust under Windows cmd.
+          CIBW_TEST_COMMAND: python -c "import ete4; from ete4 import Tree; print('ete4', ete4.__version__)"
+        run: |
+          set -euxo pipefail
+          python -m pip install cibuildwheel
+          python -m cibuildwheel --output-dir wheelhouse "$ETE4_SRC"
+
+      - name: Upload wheels
+        uses: actions/upload-artifact@v4
+        with:
+          name: ete4-${{ github.event.inputs.ete4_version }}-windows-wheels
+          path: wheelhouse/*.whl
+          if-no-files-found: error

.github/workflows/ci.yml

@@ -0,0 +1,96 @@
+name: CI
+
+on:
+  push:
+    branches: [main, development]
+  pull_request:
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  # Fast, broad matrix. The unit tests import only lightweight deps (pandas,
+  # numpy, matplotlib, biopython, pyyaml, psutil, unidecode) -- NOT ete4 -- so
+  # they run across Python versions and OSes without ete4's Cython build hurdle.
+  unit:
+    name: unit / py${{ matrix.python }} / ${{ matrix.os }}
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, macos-latest]
+        python: ["3.10", "3.11", "3.12", "3.13"]
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python }}
+      - name: Install unit-test dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install pytest "pandas>=2.2.2,<4.0.0" "numpy<2" matplotlib biopython pyyaml psutil unidecode tabulate
+      - name: Run unit tests
+        run: pytest -q
+        env:
+          PYTHONPATH: .
+
+  # One combo that exercises the real Linux install path: a fresh conda env, then a
+  # pip install of the package. ete4 is pulled and built from PyPI by pip (not conda:
+  # conda-forge's ete4 lags PyPI and does not have the pinned version), matching the
+  # documented install (conda for the external tools, pip for FunVIP + ete4). Then a
+  # full import + unit-test smoke test.
+  install-smoke:
+    name: full install (conda env + pip) / ubuntu / py3.12
+    runs-on: ubuntu-latest
+    defaults:
+      run:
+        shell: bash -el {0}
+    steps:
+      - uses: actions/checkout@v4
+      - uses: conda-incubator/setup-miniconda@v3
+        with:
+          python-version: "3.12"
+          channels: conda-forge,bioconda
+          channel-priority: flexible
+          auto-activate-base: false
+          activate-environment: funvip-ci
+      - name: Install FunVIP with test extras (pip pulls + builds ete4 from PyPI)
+        run: pip install -e ".[test]"
+      - name: Smoke test (package imports) + unit tests
+        run: |
+          # The install step above is the real check: it proves ete4 builds from
+          # its PyPI sdist and the whole dependency set resolves in a fresh env.
+          # We intentionally do NOT import ete4 here -- ete4 eagerly imports PyQt6
+          # (GUI), which needs Qt system libraries (libGL, libEGL, libX11, ...) that
+          # a headless runner may lack. That is a runtime/deployment concern, not an
+          # install one, and real users have those libs. funvip's package init is
+          # import-safe (no ete4/Qt), so this check stays reliable.
+          python -c "import funvip; print('funvip imports OK')"
+          pytest -q
+
+  # Same as install-smoke, on Apple Silicon macOS. The macOS-specific value is
+  # confirming ete4 compiles from its PyPI sdist on osx-arm64 (conda-forge lags the
+  # pinned version, so ete4 comes via pip here too). Qt-free smoke, as on Linux
+  # (macOS provides Qt at runtime via conda's pyqt in the documented recipe).
+  install-smoke-macos:
+    name: full install (conda env + pip) / macos-arm64 / py3.12
+    runs-on: macos-latest
+    defaults:
+      run:
+        shell: bash -el {0}
+    steps:
+      - uses: actions/checkout@v4
+      - uses: conda-incubator/setup-miniconda@v3
+        with:
+          python-version: "3.12"
+          channels: conda-forge,bioconda
+          channel-priority: flexible
+          auto-activate-base: false
+          activate-environment: funvip-ci
+      - name: Install FunVIP with test extras (pip pulls + builds ete4 from PyPI)
+        run: pip install -e ".[test]"
+      - name: Smoke test (package imports) + unit tests
+        run: |
+          python -c "import funvip; print('funvip imports OK')"
+          pytest -q

.gitignore

@@ -12,13 +12,11 @@
 /*.fasta
 /*.nwk
 /funid/external/MAFFT_Windows/
-!/funid/external/mmseqs_Windows/bin/*
-!/funid/external/mmseqs_Windows/bin/*.exe
-!/funid/external/mmseqs_Windows/bin/*.dll
-!/funip/external/mmseqs_Windows/bin/*
-!/funip/external/mmseqs_Windows/bin/*.exe
-!/funip/external/mmseqs_Windows/bin/*.dll
-!/funvip/external/mmseqs_Windows/bin/*
+/funvip/external/MAFFT_Windows/
+# mmseqs' bundled busybox self-installs a symlink per POSIX tool name (ls, cat,
+# grep, ...) into this dir at runtime; only the real committed binaries should
+# be tracked.
+/funvip/external/mmseqs_Windows/bin/*
 !/funvip/external/mmseqs_Windows/bin/*.exe
 !/funvip/external/mmseqs_Windows/bin/*.dll
 /funid/__pycache__/*.pyc
@@ -44,6 +42,10 @@
 /DB/blast/*/*.nsq
 /DB/mmseqs/*/*
 !/DB/mmseqs/*/*.gitignore
+
+# Runtime BLAST/mmseqs db cache, built next to the installed package (see
+# funvip/src/initialize.py: self.in_db)
+/funvip/db/
 /FunID-win32-x64/*
 /FunIP-win32-x64/*
 /FunVIP-win32-x64/*
@@ -67,3 +69,15 @@ DB.fasta
 /funvip/FunVIP.egg-info/*
 CLAUDE.md
 .claude/
+
+# Python + pytest caches
+__pycache__/
+*.pyc
+.pytest_cache/
+.coverage
+htmlcov/
+
+# Virtual environments
+.venv/
+.venv*/
+venv/

CHANGELOG.md

@@ -0,0 +1,71 @@
+# Changelog
+
+All notable changes to FunVIP are documented here. This project adheres to
+[Semantic Versioning](https://semver.org/).
+
+## [1.0.0] - 2026-07-09
+
+First stable release. The pipeline was ported from ete3 to ete4, made
+substantially faster, hardened across the board, and given native Windows support.
+
+Upgrading from 0.5.x: 0.5.x is built on ete3 and 1.0 on ete4 (different packages),
+so rebuild the conda environment instead of `pip install --upgrade`. See
+`tutorial/installation.md` (Migrating from 0.5.x to 1.0).
+
+### Major
+- **ete4 engine.** Migrated the whole tree layer from ete3 to ete4. Supported
+  Python is now 3.9-3.13.
+- **Native Windows support.** ete4 has no Windows wheel on PyPI; FunVIP now bundles
+  prebuilt ete4 wheels (`funvip/_vendor/ete4_wheels`) and installs the matching one
+  on first run, falling back to building ete4 from source (patched for Windows) when
+  no wheel matches. The Windows install is now the same `pip install FunVIP` as
+  Linux/macOS.
+
+### Performance
+- Tree interpretation roughly 13x faster (memoized tree search, per-tree set-cache
+  in `decide_type`, numpy `calculate_zero`, within-cluster `diff_min` instead of an
+  O(n^2) distance matrix).
+- `seperate_clade` no longer re-deep-copies the resolved comb at every level
+  (dropped an O(depth^2) copy).
+- Vectorized group clustering; `generate_dataset` pre-indexes sequences by group
+  (was O(groups x genes x N)).
+- Pool workers share the sequence universe via fork copy-on-write instead of
+  per-task pickling.
+- `concatenate` frees per-gene search tables after building the concatenated table
+  and dictionary-encodes its string columns.
+
+### Fixed
+- **5.8S / conserved-gene interpretation crash.** The conserved-gene tree step used
+  to fail silently (RecursionError / pickle failure); it now interprets correctly.
+- **Result-affecting preset bug.** An explicitly given cluster cutoff was discarded,
+  so runs used the wrong cutoff; presets and CLI cutoffs are now applied correctly.
+- **TCS (t-coffee) no longer eats all system memory.** The bundled t-coffee
+  crash-loops on modern kernels (PID-indexed static array vs. large
+  `kernel.pid_max`); FunVIP no longer executes t-coffee just to detect it, caps and
+  times out the TCS run, and skips TCS cleanly on failure. See `tools/tcoffee/` for
+  a recipe to build a working t-coffee.
+- Numerous correctness and robustness fixes: external-tool exit codes are checked,
+  option/preset validation bugs, several always-wrong guards, a list-mutation bug,
+  `--all` handling, `homogenize`, and the GenMine executable resolution.
+
+### Added
+- Typed exception hierarchy with a top-level handler, plus a logging overhaul across
+  the core modules.
+- Method-aware external-tool preflight (checks only the tools the chosen methods
+  need) and a more robust `Version` probe.
+- Unit-test suite and CI (pip matrix on Python 3.10-3.13 + a conda install smoke
+  test), and a marked terrei end-to-end integration test.
+- `environment.yml` and a `Dockerfile` for reproducible installs.
+- Self-contained HTML report and per-group statistics.
+- Parameter reference (`docs/parameters.md`), refreshed README and installation
+  docs, and build recipes under `tools/` (working t-coffee for TCS; ete4 Windows
+  wheels).
+
+### Changed (behavior)
+- Cluster e-value default is now `1e-4` (a single clean preset key).
+- Group-assignment ties are broken by top-bitscore majority.
+- Missing per-gene bitscores in the concatenated table are filled by projecting onto
+  the fitted regression line.
+- Removed confirmed-dead / broken code paths.
+
+[1.0.0]: https://github.com/Changwanseo/FunVIP/releases/tag/v1.0.0

Dockerfile

@@ -0,0 +1,19 @@
+# FunVIP in a self-contained conda environment (bundles the external tools).
+# Useful on Windows/WSL or anywhere a native install is awkward.
+#
+#   docker build -t funvip .
+#   docker run --rm -v "$PWD:/data" funvip --test terrei --email you@example.com --outdir /data/out
+#
+FROM condaforge/miniforge3:latest
+
+# Build the environment from environment.yml (external tools via conda + FunVIP
+# via pip). Clean caches to keep the image small.
+COPY environment.yml /tmp/environment.yml
+RUN mamba env create -f /tmp/environment.yml && mamba clean -a -y
+
+# Headless Qt for ete4 tree rendering
+ENV QT_QPA_PLATFORM=offscreen
+
+# Run FunVIP inside the env by default; arguments passed to `docker run` go to FunVIP.
+ENTRYPOINT ["conda", "run", "--no-capture-output", "-n", "funvip", "FunVIP"]
+CMD ["--help"]

README.md

@@ -25,15 +25,14 @@ Bug reports are always welcomed
 
 
 
-#### IMPORTANT NOTICE: The python dependency for Linux platform has changed from 3.12 to 3.11 for TCS inclusion. Please remake conda environment for FunVIP 0.3.25 update
-
 ## Tutorial
 * [Part 1 - Getting started!](https://github.com/Changwanseo/FunVIP/blob/main/tutorial/tutorial.md)
 * [Part 2 - Preparing database and query](https://github.com/Changwanseo/FunVIP/blob/main/tutorial/tutorial2.md)
 * [Advanced tips](https://github.com/Changwanseo/FunVIP/blob/main/tutorial/advanced.md)
 <br><br/>
 ## Documentation
 * See [Documentation](https://github.com/Changwanseo/FunVIP/blob/main/Documentation.md) for advanced usage !
+* See the [command-line parameter reference](docs/parameters.md) for every option, or run `FunVIP --help`
 <br><br/>
 ## Requirements
 - Conda or Mamba environment
@@ -43,17 +42,34 @@ Bug reports are always welcomed
 \* Recently, Mamba is a lot faster than conda. See [here](https://github.com/conda-forge/miniforge?tab=readme-ov-file) to how to install mamba environment
 <br><br/>
 ## Installation
-* [Windows](https://github.com/Changwanseo/FunVIP/blob/main/tutorial/installation.md##Windows)
-* [Mac - apple silicon](https://github.com/Changwanseo/FunVIP/blob/main/tutorial/installation.md##Apple )
-* [Linux](https://github.com/Changwanseo/FunVIP/blob/main/tutorial/installation.md##Linux)
-* [from source](https://github.com/Changwanseo/FunVIP/blob/main/tutorial/installation.md##Installation)
+* [Windows](https://github.com/Changwanseo/FunVIP/blob/main/tutorial/installation.md#windows)
+* [Mac - apple silicon](https://github.com/Changwanseo/FunVIP/blob/main/tutorial/installation.md#apple-silicon-mac)
+* [Linux](https://github.com/Changwanseo/FunVIP/blob/main/tutorial/installation.md#linux)
+* [from source](https://github.com/Changwanseo/FunVIP/blob/main/tutorial/installation.md#installation-from-source-for-developers)
+
+### Quick install (conda / mamba)
+The external aligners and tree tools install via conda; FunVIP and its Python dependencies (including ete4) install via pip (which works on Linux/macOS).
+```bash
+conda env create -f environment.yml   # or: mamba env create -f environment.yml
+conda activate funvip
+FunVIP --test terrei --email you@example.com
+```
+Or run everything in a container (no local install; handy on Windows/WSL):
+```bash
+docker build -t funvip .
+docker run --rm -v "$PWD:/data" funvip --test terrei --email you@example.com --outdir /data/out
+```
 <br><br/>
 ## Usage
 ```FunVIP --db {Your database file} --query {Your query file} --email {Your email} --gene {Your genes} --preset {fast or accurate}```
 <br><br/>
 ### Example
 ```FunVIP --db Penicillium.xlsx --query Query.xlsx --email {Your email} --thread 8 --gene ITS BenA RPB2 CaM --preset fast```
 
+### Quick check
+Verify your installation on a bundled dataset (needs an e-mail for the GenBank accessions):
+```FunVIP --test terrei --email {Your email}```
+
 \* See documentation for detailed usage
 <br><br/>
 
@@ -88,11 +104,12 @@ tabular (```.xlsx```, ```.csv```,  ```.parquet```, ```.ftr```) form
 * TREE_METHOD : fasttree is fastest, but least accurate (However, still a lot accurate than NJ tree). It is treated that iqtree is faster but slightly less accurate than raxml, but iqtree requires at least 1000 bootstrap. So in case of speed, raxml could be a little bit faster when low bootstrap selected-->
 
 ## Results
-* ```Section Assignment.xlsx``` : Your clustering result is here. You can find which of your sequences are clustered to which section 
-* ```Identification_result.xlsx``` : Your final identification result. Shows how your sequences were assigned to species level through tree-based identification
-* ```report.xlsx``` : overall statistics about the tree. If your find taxon ends with numbers, these taxon are found to be paraphyletic, so should be checked
-* ```/Tree/{section}_{gene}.svg``` : Final collapsed tree in svg format. Can be edited in vector graphics programs, or in powerpoint (by ungroup)
-* ```/Tree/{section}_{gene}_original.svg ``` : Uncollapsed tree for inspection
+Outputs are written to `{outdir}/{runname}/`:
+* ```{runname}.result.csv``` : Final identification result: how each query was assigned to the group and species level through tree-based identification
+* ```{runname}.dataset.csv``` : Clustering result: which group (e.g. genus/section) each sequence was assigned to
+* ```{runname}.report.txt``` : Run report with the options used and a per-step summary. Taxa ending with a number are paraphyletic and should be checked
+* ```07_Tree/{runname}_{group}_{gene}.svg``` : Final collapsed tree (SVG), editable in vector-graphics programs or PowerPoint (ungroup)
+* ```07_Tree/{runname}_{group}_{gene}_original.svg``` : Uncollapsed tree for inspection
 
 * Example output tree of FunVIP
 ![image](https://github.com/user-attachments/assets/7291c990-62d0-4579-8ae7-adc5d39a7fed)
@@ -106,6 +123,9 @@ Will be tested by our lab memebers to fix bugs and advance features~~
 Will be tested by peer taxonomists
 3. Stable release (ver 1.0)
 
+## Running the tests
+```pytest``` runs the fast unit tests (no external tools needed). ```pytest -m integration``` runs the end-to-end terrei test (needs the external tools on PATH and ```FUNVIP_TEST_EMAIL``` set). See [tests/README.md](tests/README.md).
+
 ## License
 [GPL 3.0](https://github.com/Changwanseo/FunVIP/blob/main/LICENSE)