Repository navigation
metadata: read only the proteins a search hits - #35
Merged
Merged
Conversation
Since IEDB#25, hits get their protein metadata from one join against a per-protein table instead of a copy in every hit row, but that table holds the whole proteome: 82k proteins with nine text fields each for human, read from the index and converted however few proteins the search hit. On small searches that cost more than the search itself (an exact search of 1,000 MHC ligands took 0.29 s, 0.26 s of it building the table). rs_metadata now takes an optional list of protein numbers and returns just those rows, ascending and deduplicated, raising ValueError for a number the index doesn't have; without the list it returns the whole table as before. _to_dataframe passes the distinct proteins in the hits, and the join asks for the hits' row order (maintain_order='left') instead of relying on polars' unspecified default. A hit on a protein the index doesn't have means the search read stale .pepidx files, so that error now says to rebuild them. Output is unchanged. test_metadata_rows.py builds each mode's frame twice from the same raw hits, once reading only the hit proteins and once reading every protein, and requires the two to be identical; it also covers duplicate, unsorted, empty and out-of-range requests, the column types of a search that hits nothing, and a stale index. The new file has its CI matrix row.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #25. After a search finds its hits, each hit only has a protein number, so we look up that protein's details (ID, name, gene, species...) and join them on. Right now that lookup reads the details for every protein in the proteome (82k for human) on every search, and the join then throws away every protein that wasn't hit. On small searches that's most of the runtime (eg. an exact search of 1,000 MHC ligands spends 0.26 of its 0.29 s on it).
This PR only looks up the proteins the search hit, then does the same join as before. For a search with 3 hits on 2 proteins:
So it keeps what #25 fixed (no repeated copies) without reading the whole proteome.
In the code, rs_metadata takes an optional list of protein numbers (without one it still returns the whole table), and _to_dataframe passes the distinct proteins from the hits. I also set maintain_order='left' on the join since polars doesn't guarantee join order by default. Same order as now, it's just guaranteed.
That exact search goes from 0.29 s to 0.04 s, and mismatch, discontinuous and indel searches are 16 to 32% faster. Searches that hit most of the proteome take the same time as before.
Output is unchanged. The proteins it skips are the ones the join was already throwing away. The new test_metadata_rows.py checks this for every mode: it builds each result from the same raw hits twice, once reading only the hit proteins and once reading every protein, and makes sure they're identical. Added its CI matrix row. I also ran 28 searches against the human proteome on master and on this branch (every mode, every output format, the CLI) and got the same results, apart from row order in exact search and ties in best match, which already change from run to run on master. Full suite passes on Python 3.9 to 3.13 and polars 1.31 (our minimum) through 2.0.
One behavior change: indel batches that mix short and long queries, and best match without k, read hits from one .pepidx and metadata from another. If those were built from different versions of the FASTA, a hit can point to a protein the metadata index doesn't have. Master returns that hit with a blank Protein ID (and best match can drop the query). Now it raises an error that says to rebuild the .pepidx files.
Found a few unrelated bugs while testing this, I'll put those in a separate issue.