FunVIP 1.0.0 merge#70
Merged
Merged
Conversation
ete3 is abandonware broken on Python 3.13 (removed cgi module). ete4 is
the maintained successor with Cython traversal and Python 3.13 support.
Key API changes applied:
- Import paths: ete4 + ete4.treeview for TreeStyle/NodeStyle/faces
- iter_leaves() → leaves(), get_common_ancestor() → common_ancestor()
- ladderize(direction=1) → ladderize(reverse=True)
- Tree(file, format=N) → Tree(file) (ete4 autodetects newick)
- write(format=0, outfile=...) → write(outfile=...)
- Guard all node.support and node.dist comparisons against None
(ete4 sets dist=None for root nodes with no explicit branch length,
and support=None for leaf nodes — ete3 defaulted to 0.0)
- Normalize dist=None → 0.0 after tree load and copy("newick")
Also updates: PyQt5 → PyQt6, requires-python <3.13 → <3.14
Validated: full FunVIP --test Penicillium run completed end-to-end
in 866s with no errors through tree interpretation and visualization.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
main.py imported PyQt5 to initialize Qt before setting QT_QPA_PLATFORM=offscreen. This was missed when migrating from ete3/PyQt5 to ete4/PyQt6, causing ModuleNotFoundError on envs that only have PyQt6 (e.g. Python 3.13). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
ete4's resolve_polytomy() assigns support=0 to newly created internal nodes, whereas ete3 assigned DEFAULT_SUPPORT=1.0. Set support=1.0 on new nodes immediately after resolve_polytomy() so the NWK output writes 1 instead of 0, matching ete3 behavior. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Reflects the ete3→ete4 migration and associated bug fixes landed since 0.5.9. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Also fixes the gene-clobber bug where the concatenated per-partition overlap was only applied to the first pair. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Cluster leaves by <= zero_init branches and scan only within-cluster pairs for diff_min; when diff_min < zero_init the answer is max(diff_min - 1e-8, floor) regardless of max_identical, so the O(n^2) phylogenetic distance matrix and the all-pairs scan are skipped. Falls back to the full scan only for well-separated data. Byte-identical self.zero, ~13x faster overall, and far less RAM. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A near-zero branch is clamped to the engine minimum and then Brent's optimizer stops within its absolute tolerance of it, so identical sequences land anywhere in [min, min+tol] (FastTree emits both 5e-9 and 6e-9). Floor at the top of that band so noise-split identical clusters flatten. Verified no divergent over-merge on Lactarius. fasttree 5e-9->6e-9, iqtree/raxml 1e-6->2e-6. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Highly conserved genes (5.8S) leave a giant star polytomy that reroot_outgroup's resolve_polytomy() combs into a ~3681-deep chain, overflowing cpickle's recursion inside the worker. Three coupled fixes: - seperate_clade: copy the input via deepcopy, not the default cpickle copy, which RecursionErrors on the deep comb. - concat_all: rebuild same-taxon groups as a BALANCED (log-depth) tree instead of a left-deep comb, so the reconstructed result stays shallow enough to pickle back to the parent (also drops the O(n^2) build). - pipe: clear the stale tree_info.outgroup_clade after reconstruct; it still referenced the deep pre-reconstruct tree and re-inflated it when the worker pickled its result on multiprocessing return. Lactarius test: 5.8S now interprets (122/122 query ASVs assigned, was 54/122); species identification unchanged (only within-species clade renumbering on db rows from the reshaped same-taxon clusters). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
seperate_clade deep-copied every child upfront, so the recursive descent of the resolve_polytomy comb (~D deep for conserved genes like 5.8S) re-copied the whole remaining subtree at each level -> O(D^2). Only copy what is kept (a single tip appended to clade_list, or the reconstruct input) and descend the zero-length comb in place; seperate_clade and reconstruct never mutate their input. Byte-identical result.csv on the Lactarius test (same md5); interpretation 9m55s -> 7m33s. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two changes to the cluster step:
- Efficiency: replace cluster()'s per-FI scan of the whole concatenated
search table (V_cSR[V_cSR["qseqid"] == FI.hash], O(N_FI * N_rows), with the
entire table pickled to every mp worker) with a single in-process
groupby("qseqid"). Benchmarks ~6x at 1k queries to ~79x at 6k and growing
(per-FI is quadratic, groupby linear); the mp.Pool is removed.
append_query_group's per-row .apply becomes a vectorized .map.
- Behavior: when a query's cutoff set spans >=2 groups, assign by group
plurality among the hits TIED at the best bitscore, descending a bitscore
level only if that count ties (was: the single top-bitscore hit's group,
tie-broken by pandas' unstable quicksort). Deterministic and resistant to
database-composition bias. On the Lactarius test it moves 6/122 queries:
4 arbitrarily-Russula ASVs -> Lactarius (the plurality of their equally-best
hits), 2 -> Inonotus (Inonotus dominates even their best hits).
Verified: the new cluster() reproduces the intended assignments on the real
search table (exactly the 6 changes); the pipeline runs end-to-end.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Foundation for cleaner errors/logging, plus cluster.py as the first module: - New funvip/src/exceptions.py: a FunVIPError hierarchy (Config/Input/Search/ Cluster/Dataset/Tree/ExternalTool) so callers (and main) can tell an expected, explained failure from a bug. - main(): wrap the pipeline (now _run_funvip()) in a top-level handler that reports a FunVIPError cleanly, logs an unexpected crash with a full traceback, and no longer swallows KeyboardInterrupt. - cluster.py: context-free `raise Exception` -> ClusterError carrying the offending FI/group/gene; bare `except:` -> the specific exception (KeyError/IndexError/ValueError) with a reason; diagnostic print() -> logging. Behavior-preserving: compiles, imports, FunVIP --help works, and the full cluster step runs unchanged (verified on the Lactarius session). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the context-free "[DEVELOPMENTAL ERROR]" raise in combine_alignment with a DatasetError naming the group and the genes actually present, so a group that reaches concatenation without a real gene dataset fails with an actionable message. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- exist_dataset: bare `except:` -> `except KeyError:` (the only expected miss). - Replace five context-free "[DEVELOPMENTAL ERROR]" raises with typed errors carrying the offending values: an unexpected --level -> ConfigError; the list_FI/dict_hash_FI conflicts (final_species, adjusted_group, bygene_species) and the --terminate alignment-validation stop -> DatasetError. Behavior-preserving: cluster step (generate_dataset/remove_invalid_dataset) runs unchanged on the Lactarius session. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace nine context-free raises with typed, actionable errors: unknown search method (build/run) and fragment-merge inconsistencies -> SearchError; a missing database file or a gene present in the query but absent from the database -> InputError; no valid gene in the database -> ConfigError. Narrow the bare `except:` guarding the (possibly deleted) df_search to `except NameError:`. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Tree import: bare `except:` -> `except Exception as e: raise TreeError(...) from e` preserving the parse cause; missing tree file and duplicated tree_info -> TreeError with the offending path / group+gene. - Species-label parse: bare `except:` -> `except (ValueError, IndexError):` (label without a trailing integer). - The _safe per-item guards now log the full traceback (not just repr) on a failed (group, gene) interpretation/visualization, so a skipped item is debuggable. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Bring the module onto the logging/exception hierarchy (it previously used only
print and bare raise):
- Add `import logging`; route the 38 diagnostic/error/warning print()s (many
ANSI-colored manual logging) to logging.debug/info/warning.
- Replace all 16 context-free `raise Exception` with TreeError carrying the
offending clade/leaf/group/tree. In reroot_outgroup the control-flow
raise->fallback still works (TreeError subclasses Exception, so the fallback
`except` catches it).
- Narrow the 7 bare `except:` to the specific exception (KeyError/ValueError/
TypeError/Exception) with a reason; the reroot and viz type-detection
fallbacks are preserved and Ctrl-C is no longer swallowed.
- Fixes a latent NameError ({resety} typo) in a dead error path.
Verified result-preserving: `--continue --step visualize` on the Lactarius
session (same cached trees) gives a byte-identical result.csv, terrei runs
clean, and worker-process logging works with no deadlock.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two statically-wrong conditionals that silently degrade results: - trim.py: `if trimming_result is not ((-1,-1))` compared a tuple by identity (always true) -> a failed/over-aggressive trim slipped through and produced a one-column alignment. Compare by value. - tree.py: `if list_og_FI == 0` compared a list to 0 (always false) -> an outgroup-less dataset was never skipped. Use `len(...) == 0`. ext.py: check every external tool's exit code and raise SearchError/ ExternalToolError instead of feeding empty/stale output forward (blast, mmseqs, makeblastdb, makemmseqsdb, mafft, trimal, fasttree, iqtree); guard Gblocks parsing against a missing/format-drifted .gb.txt (was UnboundLocalError/ FileNotFoundError); wrap RAxML's os.chdir in try/finally so cwd is always restored; quote trimal's Linux paths; drop leaked os.devnull fds (use subprocess.DEVNULL); typed raises; remove debug prints. trim.py: guard the empty trimmed-FASTA read (IndexError); modeltest.py: drop the undefined `model_dict["BIC"]` line that NameError'd into a misleading "cannot parse" warning every run. Verified: terrei smoke test runs clean (exit-code checks do not false-positive). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
validate_option.py:
- RESULT-AFFECTING: `CLUSTER-CUTOFF` preset key was written to cluster.evalue,
not cluster.cutoff, so the shipped presets' 0.97 was silently discarded and
clustering ran at the 0.95 default. Now applies (verified via a preset harness
and a terrei run). This changes group-clustering results for --preset runs.
- Systemic: `key.lower() in ("string")` in update_from_preset was a substring
test, not tuple membership (missing comma) -> preset keys collided (search->
nosearchresult, trim->allow-innertrimming, solveflat->nosolveflat, cachedb->
nocachedb). Converted all 51 single-string tests to 1-tuples and added the
missing positive solveflat/cachedb branches.
- `~` (bitwise-not) on bools -> `not`; typos parser_dicy/parse_dict -> parser_dict;
preset outgroupoffset wrote to wordsize -> outgroupoffset.
- CLI: `--all` was `~parser.all` (a no-op) -> False; `--outgroupoffset` read the
nonexistent parser.cluster_outgroupoffset (AttributeError swallowed) -> read
parser.outgroupoffset.
- validate(): method dict lookups `X_adjust[key]` KeyError'd on invalid input ->
`.get(key, key)`; undefined `color` in the --highlight/--outgroupcolor error ->
the actual attribute; max_target_seqs<=0 reset cluster.outgroupoffset ->
max_target_seqs; `list_warning(...)` called a list -> .append; impossible
`gt < 0 and gt >= 1` -> `or`.
validate_input.py: query count comprehension counted all seqs (no filter) ->
sum with condition; query tables saved to 01_DB -> 02_Query by datatype;
`opt.gene = list(set(...))` reordered genes nondeterministically -> dict.fromkeys
(preserve CLI order); `not x == None` -> `is not None`.
preset/accurate.yaml: `CONFIDENT: fale` typo -> false.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- reporter.py: the report.txt [OPTION] block printed the DB and QUERY paths swapped; the per-gene result guard tested `gene in FI.bygene_species` but then indexed `FI.seq[gene]` -> a latent KeyError that would abort result.csv generation if the two dicts diverged (now also requires `gene in FI.seq`). - save.py: save_session had no per-key guard, so a single unpicklable object aborted the whole run at a checkpoint -> log a warning and continue; the xlsx size-limit fallback wrote CSV bytes to a `.xlsx`-named file (now writes a `.csv`) and the column guard was off-by-one (16383 -> 16384). - tool.py: get_genus_species had a mutable default list under @lru_cache -> tuple. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
cazlculation->calculation, peformed->performed, stasifies->satisfies, Pararellize->Parallelize, reoprt->report, delimitaion->delimitation, unexpectively->unexpectedly, simultaniously->simultaneously. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
decode() compiled `"|".join(hash_dict.keys())` -- a regex with one alternative per hash -- and re-escaped every match to look it up, giving O(text * N_hashes) behaviour that is a hotspot at metabarcoding scale. Every hash has the fixed, self-delimiting form HS<number>HE, so match that single token `HS\d+HE` once and dict-look-up each occurrence (a token absent from the dict is left unchanged). re.escape was a no-op for these alphanumeric tokens, so dropping it (here and in decode_df, which also loses an unnecessary deepcopy) changes nothing. Verified byte-identical to the previous implementation across all newick/svg combinations, decode_df, and edge cases (adjacent hashes, prefix-overlapping HS1HE/HS12HE, tokens not in the dict, unicode/special-char values); a terrei run produces a fully decoded tree and a result.csv identical to the pre-change run. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…alized get_genus_species(genus_list=None) read the module global `genus_file` (path.genusdb, ~3094 lines) with `open(...)` on EVERY call, and validate_input calls it ~15x per sequence -> the 37KB genus DB was re-opened and re-parsed tens of thousands of times at metabarcoding scale. Cache the parsed list keyed by path (read once). If the global was never set by initialize_path() the open raised a cryptic `NameError: genus_file`; now raise an actionable ValueError. get_genus_species is @lru_cache'd, so this pairs with the earlier endwords list->tuple fix (hashable args); pipeline callers already pass genus_list as a tuple. Verified: uninitialized -> clear ValueError, results identical to passing genus_list explicitly, and a terrei run produces a byte-identical result.csv. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Runs the full pipeline on the bundled terrei dataset and asserts stable invariants (79 queries, all Aspergillus, all species assigned) rather than an exact byte match, since mmseqs/mafft/fasttree are nondeterministic across runs. Marked `integration`; skipped unless external tools are on PATH and FUNVIP_TEST_EMAIL is set. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
command.py: correct help strings whose stated defaults no longer matched the code
or the default fast preset (--search mmseqs->blast, --mode validation->
identification, --fsize 10->14, --maxoutgroup 1->3, --cluster-evalue 1e-7->1e-4,
--backgroundcolor colours, --collapsebscutoff/--bootstrap notes); drop the [WIP]
placeholder on --step and describe it; add missing type= on --maxoutgroup (int)
and --mafft-algorithm/--trimal-algorithm (str); add defaults to --trimal-gt and
--criterion; lowercase the --test dataset names.
docs/: add gen_param_reference.py (introspects the live argparse parser) and the
generated docs/parameters.md, so the parameter table can't drift from command.py.
README.md: fix the Results section to the actual outputs ({runname}.result.csv /
.dataset.csv / .report.txt and 07_Tree/*.svg, not the old xlsx names); drop the
stale 0.3.25 python-version notice; link the parameter reference; add a
`--test terrei` quick-check and a "Running the tests" section.
Verified: FunVIP --help renders, terrei runs clean, explicit new-type args parse.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
version.py probed all 8 external tools unconditionally at startup with no error handling, so a user who has only the tools for their chosen methods (e.g. the fast preset's blast/mafft/fasttree) crashed on the first unrelated missing tool (mmseqs) with a bare FileNotFoundError. - Version now probes via a robust helper that returns '' for any missing or unparseable tool instead of crashing, and collapses the three near-duplicate per-platform branches. - New preflight(opt): checks only the tools the selected --search/--trim/ --modeltest/--tree (+ TCS) will actually invoke, logs a "tools found" summary, warns-and-skips for optional TCS (generalizing the existing graceful-degrade), and raises a clear ConfigError listing the missing required tools and how to install them. Wired into main.py right after logging setup, so a missing tool fails fast with an actionable message instead of mid-run. Verified: terrei runs clean and logs the summary; the preflight raises for a missing required tool, warns for missing optional TCS, and is method-aware (terrei's TRIM=none correctly omits trimAl). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The reliable install path is: external tools via conda (bioconda/conda-forge), FunVIP + its Python deps (incl. ete4) via pip -- ete4 4.4.0 pip-installs cleanly on Linux/macOS (the Cython issue is Windows-specific). - environment.yml: one-command conda/mamba env with the 8 tools (+ optional t-coffee) and FunVIP from PyPI; a comment explains the from-source variant. - Dockerfile: miniforge-based image that bundles everything, with a headless-Qt env var and FunVIP as the entrypoint -- for Windows/WSL or awkward installs. - README: a self-contained "Quick install (conda / mamba)" + Docker section, so users are not solely dependent on the external tutorial links. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Statistics: re-implement the per-group identification summary (it was dead
code, and the removed version even counted STATUS strings that no longer exist).
update_statistics() aggregates the real STATUS values (match/assigned ->
IDENTIFIED, new species -> NEW SPECIES CANDIDATE, conflict -> MISIDENTIFIED,
failed -> UNDETERMINED) per GROUP_ASSIGNED with a grand-total row, written to
{runname}.statistics.csv and a [STATISTICS] block in report.txt. Verified the
counts match the raw STATUS column exactly.
- HTML report: implement the report_html stub (it also had a self-less
signature) -> a self-contained {runname}.report.html with inline CSS, the
statistics and identification tables, and relative links to the collapsed tree
SVGs; wired into the report step. Verified well-formed on terrei.
(Provenance was already covered: report.txt carries [VERSIONS] with FunVIP +
external-tool versions, [COMMAND], and [OPTION].)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Unit test for Report.update_statistics: verifies STATUS values map to the right categories per group with a correct grand-total row, and that an empty result is handled. Add tabulate to the CI unit-test deps (reporter imports it; it does not pull in ete4, so the tests stay lightweight). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- installation.md: standardize on python 3.12 everywhere (Linux said 3.11, a leftover from the old TCS constraint); add the missing external-tool install step to the from-source recipe; rewrite the Windows section to conda/WSL/Docker since a bare `pip install` fails on ete4's Cython there; add the recommended `environment.yml` one-liner; note that TCS/t-coffee is optional and `--test` is case-insensitive. - development.md: drop the two references to genus_count (removed as dead code). - tutorial.md: point the install links at installation.md with valid anchors (they pointed at nonexistent README anchors). - README.md: fix the installation.md links (double-## -> single-# lowercase anchors). - tutorial2.md / advanced.md: typo fixes (sequneces, sould, specie). Verified against the working env: the Linux/from-source tool list and the pip+ conda split match what actually installs; `--test Terrei` works because the code lowercases it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reproduced on a fresh env: the bioconda t-coffee build leaks memory unboundedly
even on `t_coffee -help`/`-version` (consumed ~140 GB before being killed), and
hangs on the real `-evaluate` call. FunVIP made this catastrophic because:
- the TCS availability probe ran `t_coffee -help` via subprocess with NO
timeout, so option validation hung and ate all RAM; and
- the actual TCS call ran t-coffee with no memory or time bound.
Fixes:
- validate_option.py: check t-coffee with shutil.which (presence only) instead of
executing `t_coffee -help` -- never run t-coffee just to detect it.
- ext.py TCS: run t-coffee under an 8 GB address-space cap (RLIMIT_AS) in its own
session, with a 300 s timeout that kills the whole process group -- a leak is
capped, a hang is bounded, no orphaned t-coffee.
- dataset.py: make the TCS step non-fatal (log and skip on failure) and guard the
.tcs score-file consumption (skip missing/unparseable files) so a failed TCS
does not crash the run at the reader.
- environment.yml: stop installing t-coffee by default (commented, with a note),
so the recommended install cannot trigger this. TCS stays opt-in; FunVIP skips
it when t-coffee is absent.
Verified on a fresh conda env WITH the leaky t-coffee: run completes (result.csv
produced), TCS times out -> killed -> skipped, peak memory bounded to ~4.5 GB
(vs ~140 GB before). With t-coffee absent, terrei is byte-identical.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Diagnosed why t-coffee "leaks all memory" (it is not a leak). The bioconda t-coffee indexes a static array by the raw OS PID but is compiled with MAX_N_PID=260000. Modern kernels default kernel.pid_max=4194304, so t-coffee's own PID overflows that array on essentially every run -> out-of-bounds -> SIGSEGV -> its signal handler re-runs the faulting instruction forever, growing the heap until all RAM is gone. It is a crash-loop, fires on ANY invocation (even `t_coffee -version`), and is independent of MAX_N_PID_4_TCOFFEE, which is not compiled into the binary (only the "Recompile" error string is present). Evidence: strace shows a flood of identical SIGSEGV(SEGV_MAPERR) at one address after "--ERROR: MAX_N_PID exceded (current: 260000 Requested: <pid>)"; peak RSS is identical for MAX_N_PID_4_TCOFFEE unset/1000/4194304. No runtime userspace fix works here: unprivileged user namespaces are denied (uid_map EPERM) and t-coffee reads the PID via a direct syscall (LD_PRELOAD getpid() shim bypassed). Working TCS requires a t-coffee built with MAX_N_PID >= pid_max, or a host with pid_max < 260000 -- neither of which FunVIP can do at runtime, so the cap + timeout + skip added earlier is the correct handling. This commit only corrects the code comments and the environment.yml note to state the true root cause. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
TCS (T-Coffee's alignment-consistency score) is an optional FunVIP feature, but the prebuilt bioconda t-coffee crash-loops and can consume all system memory on modern kernels: it indexes a table by raw PID, sized by a compile-time MAX_N_PID=260000, and kernel.pid_max=4194304 overflows it (SIGSEGV handler re-faults forever). Setting MAX_N_PID_4_TCOFFEE is not enough -- the table is allocated with the compile-time constant regardless -- so the only real fix is to recompile with MAX_N_PID >= pid_max. Because T-Coffee is GPL with an added "non-commercial pipelines only" caveat, FunVIP does not redistribute a t-coffee binary. Instead this adds a build recipe the user runs themselves (accepting T-Coffee's license directly): - tools/tcoffee/build_tcoffee_for_tcs.sh: downloads the pinned, checksum-verified T-Coffee source, applies the one-line MAX_N_PID patch, builds only the t_coffee binary with modern-g++ flags, self-tests it (TCS on a tiny alignment; fails if it still crashes), and installs it. Verified end-to-end: t_coffee -version and a real TCS run go from a ~3 GB crash-loop to 20 MB / 0.5 GB and a valid .tcs. - tools/tcoffee/max_n_pid.patch: the exact one-line diff (documents the change). - tools/tcoffee/README.md: the root cause, usage, and the T-Coffee license notice. - installation.md: stop telling users to `conda install t-coffee` (both recipes); explain TCS is optional and point to the build script. - validate_option.py: the "t-coffee not found" warning now points to the recipe. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ete4 is a hard FunVIP dependency but upstream ships no Windows build: PyPI has an
sdist only (so pip compiles, which fails on Windows) and conda-forge has no win-64.
That is the real blocker to a pip/conda FunVIP install on Windows (t-coffee/TCS is
Linux-only and unrelated). ete4 is plain GPL-3.0-or-later, so redistributing a
self-built Windows wheel is allowed (offer source, ship the patch, keep LICENSE).
Adds a manual (workflow_dispatch) workflow that downloads the pinned ete4 sdist,
applies the small etetoolkit/ete PR #783 path-separator fix, builds win_amd64
wheels for CPython 3.10-3.13 with cibuildwheel, import-tests the compiled
extension, and uploads them as an artifact.
- tools/ete4-windows/ete4-windows.patch: the 3-line PR #783 diff (setup.py uses
os.path.sep for Cython module names; config.py uses expanduser('~') not
os.environ['HOME']). Verified: applies cleanly to 4.4.0 and is a no-op on Linux
(the patched source still builds a normal Linux wheel).
- tools/ete4-windows/README.md: rationale, how to run, how to host/use the wheels,
the "upstream it" option, and the ete4 GPL notice.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…n build) The intro wrongly listed conda as a Windows install option. Neither pip nor conda can install FunVIP on Windows because ete4 ships no Windows build (PyPI is source-only; conda-forge has no win-64). Correct the intro and the Windows section to say WSL/Docker only, and point to the in-progress tools/ete4-windows/ wheel recipe for future native support. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Match the project's no-em-dash convention; replace with commas, colons, semicolons, or parentheses. No content change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ete4 has no Windows wheel on PyPI and cannot compile there, which blocked any pip/conda install of FunVIP on Windows. Instead of asking Windows users to fetch a wheel manually, FunVIP now carries prebuilt ete4 Windows wheels and installs the right one automatically on first run, so the Windows recipe is the same four lines as the other platforms (conda create / activate / pip install FunVIP / run). - pyproject.toml: ete4 is now a dependency only off Windows (`platform_system != 'Windows'`), so `pip install FunVIP` no longer tries to compile ete4 on Windows. ete4's own runtime deps (bottle/cheroot/brotli/requests) are added as Windows-only deps so the bundled wheel installs with --no-deps and stays fully offline. funvip/_vendor/ete4_wheels is packaged as data. - funvip/main.py: `_ensure_ete4()` runs at the start of `_run_funvip()` (before any ete4 import). On Windows, if ete4 is missing it pip-installs the bundled wheel matching the interpreter (cp310-cp313), with a clear error if none is bundled. No-op on Linux/macOS and once ete4 is importable. - funvip/_vendor/ete4_wheels/: drop-in location for the wheels (built by the build-ete4-windows-wheels workflow), with ete4's GPLv3 LICENSE and a README. The wheels themselves are added at release time. - installation.md: Windows section is now the same recipe as Linux/macOS. - tools/ete4-windows/README.md: document dropping the wheels into _vendor. Verified on Linux: main.py compiles; _ensure_ete4() is a clean no-op when ete4 is present; the ete4 marker requires ete4 on Linux but not Windows; the wheel-selection glob picks the right file; and a built FunVIP wheel ships funvip/_vendor/ete4_wheels. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Trim the ete4/bundling and t-coffee internals from the user-facing install docs (they confused readers). Windows is now the same four-step recipe as Linux/macOS; the TCS note is a short pointer. The technical details still live in tools/ete4-windows/ and tools/tcoffee/. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
These 25 stale .pyc files were committed before __pycache__/ was added to .gitignore; untrack them (they can shadow current source and bloat the repo). .gitignore already excludes __pycache__/ and *.pyc, so they will not return. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Previously _ensure_ete4 errored out on Windows if no prebuilt wheel matched the interpreter. Now it falls back to building ete4 from source: download the sdist, apply the Windows patch (etetoolkit/ete PR #783, applied in-process so no git/patch tool is needed), and pip-install it. This needs a C/C++ compiler and internet and runs at most once; a clear message (including `conda install -c conda-forge cxx-compiler`) is shown if it fails. Bundling wheels is still preferred so ordinary Windows users need no compiler. Verified end-to-end on Linux: download + in-process patch + compile + install produces a working ete4 (~50s); patch helper is idempotent and raises on upstream text drift; the no-op path (ete4 already importable) is unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The source-build path used `pip download --no-binary :all:`, which forced ALL packages (including Cython) to compile from source; building Cython on Windows failed with a linker error (LNK1104). Scope it to `--no-binary ete4` so Cython and setuptools install as wheels and only ete4 is built. Verified end-to-end on Linux. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ete4 auto-build was broken for every fresh Windows install: it fetched the sdist via `pip download --no-binary ete4`, which runs the sdist's own (unpatched) setup.py through pip's build-hook resolution before we ever get a chance to apply the Windows path-separator patch, so it always failed with "'ete4\core\operations' is not a valid module name". Fetch the raw sdist tarball via the PyPI JSON API instead, bypassing pip's build backend until after patching. Separately, validate_input.py invoked GenMine as a bare shell command, so a stale/incompatible GenMine elsewhere on PATH could silently shadow the one installed alongside FunVIP. Its older output format doesn't match what FunVIP parses, so accessions were never replaced by sequences and the run failed downstream with "no valid gene found in the database" -- with no indication of the real cause. Resolve GenMine relative to sys.executable instead, and normalize download_dict keys to match the version-stripped lookup. Also fix .gitignore gaps left over from the funid/funip -> funvip rename (MAFFT_Windows, mmseqs busybox symlinks, the runtime db cache) and ignore local .venv* test environments. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
_resolve_genmine used os.path.dirname(sys.executable), which on Windows is the env
root, not Scripts\, so GenMine.exe was never found there and it fell back to a bare
PATH lookup -- defeating the anti-shadowing purpose on the platform it targets. Use
sysconfig.get_path("scripts") (bin/ on POSIX, Scripts\ on Windows). Verified the
resolver still finds GenMine on Linux.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Explain that 0.5.x (ete3) and 1.0 (ete4) are different packages, so rebuild the conda environment instead of pip-upgrading across the jump. Corrects the earlier draft's overstated "FunVIP fails to import" wording (with the Windows bundled-ete4 marker a pip upgrade would not fail that way; the real issue is stale mixed deps). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Built by reusing the download+patch steps from _build_ete4_from_source in funvip/main.py (fetch the PyPI sdist, apply the PR #783 Windows path-separator patch), then `pip wheel . --no-deps` instead of `pip install`. Verified MSVC's cl.exe only warns on the upstream `-O3` flag (D9002: unknown option, ignored -- compiles at /O2 instead) rather than failing, so the build succeeds unmodified on Windows with VS 2022 installed. With this wheel present, Windows users on Python 3.12 get ete4 via `_ensure_ete4()`'s bundled-wheel path instead of compiling from source, so no C/C++ compiler is required. cp310/cp311/cp313 wheels are left for the build-ete4-windows-wheels GitHub Action to produce on a clean runner. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Bump version 0.6.0 -> 1.0.0 (single source of truth in pyproject.toml; FunVIP --version reads it via importlib.metadata). Add CHANGELOG.md (the ete3->ete4 port, performance, fixes, Windows support) and RELEASING.md (maintainer checklist: bundle ete4 wheels, verify on all platforms, build/publish, tag). Verified on Linux: FunVIP --version -> 1.0.0, 28 unit tests pass, terrei integration passes end-to-end (~69s). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
conda-forge's ete4 lags PyPI (max 4.3.0) and has no 4.4.0, so `conda install "ete4>=4.4.0,<4.5.0"` had no solve candidate and the job failed. ete4 is not reliably a conda package; the documented install uses conda for the external tools and pip for FunVIP + ete4. Drop the conda-install-ete4 step and let `pip install -e ".[test]"` pull and build ete4 from the PyPI sdist (as the unit matrix and local builds already do). Unit jobs were unaffected (they install no ete4). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The install-smoke job's `python -c "import funvip.main"` failed: importing
funvip.main runs _ensure_deterministic_hash, which re-execs the interpreter to set
PYTHONHASHSEED, and for `python -c` the re-exec drops the code string and errors
("Argument expected for the -c option", exit 2). Use `FunVIP --version` instead --
it re-execs cleanly (console-script argv survives) and still imports the whole
chain including ete4. The pip-install step (ete4 via pip) already passed; this was
the only remaining failure.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The preset referenced FunVIP_Sanghaungporus_db.xlsx (transposed "au"), but the bundled file is FunVIP_Sanghuangporus_db.xlsx, so `--test sanghuangporus` could not find its database and failed. Corrected the reference. Verified: the run now completes (exit 0) and writes a 274-row result CSV (single-gene ITS with CONCATENATE:true works). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The install-smoke job kept failing on the import/entry-point step because ete4 imports PyQt6 (QtGui/QtWidgets) at import time, and PyQt6's manylinux wheel bundles only ICU + Qt6 -- it needs libGL/libEGL/libX11/libxcb/libxkbcommon/libdbus/glib/... from the system, a large runner-dependent set. Rather than chase those libs, keep the smoke reliable: the install step already proves ete4 builds from its PyPI sdist and the whole dependency set resolves in a fresh env; the smoke then verifies only that funvip's (import-safe) package imports and the unit tests pass. The full entry-point/GUI path is exercised by the terrei integration test and real runs, not by CI. Verified locally: `import funvip` pulls neither ete4 nor PyQt6; pytest passes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Paste-ready notes (highlights, install/upgrade, full grouped changes, known issues) derived from CHANGELOG.md, for the v1.0.0 GitHub release. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds an install-smoke job on macos-latest (arm64) mirroring the Linux one: fresh conda env, pip install FunVIP (which builds ete4 from its PyPI sdist -- conda-forge lags the pinned version), then the Qt-free smoke + unit tests. The macOS-specific value is confirming ete4 compiles on osx-arm64. Lets the Mac install be verified on GitHub's macOS runners without owning a Mac. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
FunVIP 1.0.0 merge