Repository navigation
indel annotation: annotate each distinct match once, and find indel positions without rebuilding strings - #34
Merged
Conversation
The Indel Positions annotation depends only on the query and the matched peptide, and the same matched peptide recurs across many proteins, so most hit rows repeat a pair. Compute each distinct pair's annotation once and reuse it. Output is unchanged.
_indel_placements tried every combination of edit positions and rebuilt the string for each, about L^n combinations per hit (544 us for a 2-deletion 25-mer). It now reads the valid positions off precomputed agreement runs between the query and the matched peptide, with identical placements in identical order. The original implementation is kept in test_indel_positions_equivalence.py as the reference, checked exhaustively over small repeat-rich alphabets, on unrelated and empty-string pairs, and on random peptides; the new file has its CI matrix row.
Collaborator
|
Very nice, I checked the speed locally myself and it checks out. Approved. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Indel search was spending most of its time building the Indel Positions column, not in the Rust search. Two reasons. The annotation ran once per hit row even though it only depends on the query and the matched peptide, and the same matched peptide comes back from a lot of proteins, so most rows were recomputing an annotation we already had. And _indel_placements found the edit positions by trying every combination of positions and rebuilding the string for each one (eg. a 25-mer with 2 deletions tries every pair of positions and rebuilds a 23-residue string for each pair), which gets expensive fast with longer queries and 2 indels.
The first commit annotates each distinct (query, matched) pair once and reuses it for every row that has it. The second replaces the combination search: it compares the two strings at each shift (0 to n residues), records how far they agree from each position, and reads the valid placements straight off those runs. Placements come out identical and in the same order as before, which matters since _indel_edits reads placements[0]. No Rust changes and no change to the output format, and which matches get reported is untouched since this only runs after the search.
Output is unchanged. I kept the original _indel_placements in the new test_indel_positions_equivalence.py as the reference and check the new one against it (placements and the formatted text) on every 1 and 2 edit pair over {A,B} and {A,B,C} up to small lengths, which covers homopolymers and periodic repeats, plus unrelated and empty string pairs, random 6 to 25-mers, and the documented examples. Added its CI matrix row. Full suite passes on 3.11 and 3.13. I also ran the old and new matcher.py against the same engine build on all of our benchmark sets on the cluster, and every full output table was byte-identical (Indel Positions column included) with 100% recall for both.
The gain is biggest on 2-indel searches and large batches, where the annotation was most of the time. Small batches gain less since they're mostly fixed per-call cost.