perf: cut analysis hot-path overhead without changing output - #299
Conversation
|
Pushed Neither defect could move a fixture hash, so the report-identity check the earlier commits rest on could not see either. Reproductions, before and after the fix Explicit OS name, with two configured ApiScout databases where one file is missing: Blockrefs normalization, Validation
Performance impact of the fix Restoring The absolute difference between the two fast-path variants is 0.17 ms across an entire report, against multi-second analyses, and The two uncovered diff lines are in the AArch64 top-byte prefilter from an earlier commit, not in this one: the arm taken when a word's top byte matches but its full encoding does not, and a prologue rejected by the code filter. Both are genuine residual-verification paths that no bundled fixture reaches. Happy to add coverage for them if you would prefer the gate at 100%. |
bb420ed to
66cf167
Compare
|
Pushed That commit re-keyed Measured on a 32-bit MSVC PE cross-compiled from Rust (hyperfine 1.19.0,
That is 1002 functions, or 16%. The rewrite was also not faster: Worth flagging separately: the same commit added This is the third defect the report-identity check did not catch, and the first that is not explained by an unreached path. None of the ten bundled fixtures is gap-dominated, so a 16% recall loss could not move a single hash. Validation on the pushed head: the full suite reports 1847 passed, 1 skipped and 2584 subtests; lint and format checks are clean; the type checker exits 0 with no error-level diagnostics; and diff coverage against the base is 99% overall with 100% on the lines changed by the revert. |
5478945 to
3455546
Compare
|
Same pass as #300 (comment), and this one inherits that PR's position because it contains it — the branch carries #300's Cleared:
|
3455546 to
6cae1c5
Compare
…iew replies for posting Not part of either PR branch. These are the comment bodies and the description edits produced by the rebase-and-re-measure pass, parked on this scratch branch so a session with API access to the upstream repo can fetch and post them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PRvm5qFMW52aWnmVgR5vkD
|
Rebased, in the order you asked for: #300 first onto What is in it nowTwenty-one commits — #300's three plus the test-repair commit its rebase needed, then the sixteen of this branch's own, then one more: the docstring you asked for on Four of the sixteen needed real resolution rather than a textual merge:
The report hashes, regenerated against the tree this lands onYou are right that the old claim had gone stale, and for the reason you named rather than for a rebase reason. Regenerated: Ten bundled fixtures,
This branch is byte-for-byte equal to the tree it lands on, on all ten. That is the claim the PR makes and it is now made against the right baseline rather than against v4.5.0. The two that move do so between master and #300, not here: And with your point taken about what this proves: it proves these sixteen commits change nothing on these ten images. It is not evidence about
|
The instruction loop re-resolved disassembly.code_map, data_map, ins2fn, binary_info.binary, the backend handler and state.processed_bytes for every decoded instruction (~180k attribute chains on a 29k-instruction sample). All of these are only mutated in place during analysis, so binding them to locals at loop entry is identity-safe. Validation: report-identity hashes across 10 bundled intel/aarch64/dalvik/cil fixtures unchanged; make lint, make test. (cherry picked from commit fc8d2fe)
Three pieces of per-function rework in report construction: - getPicHashSequence now reuses escaped sequences through a size-capped cache keyed on (escaper, bytes, mnemonic, operands, flag, bounds), the complete input set of escapeBinary, so hits are byte-identical by construction. Real samples repeat 30-50% of their instructions (98% in dalvik classes dumps), and the cap bounds retained strings. - getNormalizedBlockRefs rebuilt a sorted copy of what getBlockRefs had just produced as sorted, deduplicated lists whenever a function carries no try_ranges - which is every function on intel and aarch64. The fast path returns that same content directly. - get_nesting_depth ran fix_graph() on a graph build_dominator_tree() had already fixed; the expansion only adds empty successor lists, which can never contribute to significant_nodes. Validation: report-identity hashes across 10 bundled fixtures unchanged; make lint, make test. (cherry picked from commit af75c2a)
The BL and prologue discovery passes tested all ~37k aligned words per
scan in Python. Every BL encoding (mask 0xFC000000) has a top byte in
0x94..0x97 and every recognized prologue pattern has one in {0xA9, 0xD5,
0xF8} - verified exhaustively over each pattern's free bits - so one
bytes.translate over the top-byte lane skips non-matching words at C
speed before any Python runs. Full verification of each hit is kept.
Timeout polling stays on the same 4096-word boundaries, so candidate
sets under the budget guards are unchanged. Prologue scan drops from
4.14ms to 0.74ms and BL from 1.17ms to 0.59ms on the bundled static
ARM64 sample; scans are O(image bytes), so multi-MB ARM64 inputs save
proportionally more.
Validation: report-identity hashes across 10 bundled fixtures unchanged;
tests/testAArch64Disassembler.py green; make lint, make test.
(cherry picked from commit 88e6cb0)
hasUnprocessedBlocks() allocated queued_blocks - processed_blocks on every call. chooseNextBlock() moves each pop straight into processed_blocks and addBlockToQueue() refuses processed blocks, so the two sets are always disjoint and the difference is just the queue itself. On a function with thousands of blocks the per-block set copies added up to quadratic allocation; the dalvik twin of this method got the same fix. Validation: report-identity hashes across 10 bundled fixtures unchanged; make lint, make test. (cherry picked from commit cfa6c64)
The CFG walk allocated an identical resolver lambda for every decoded instruction (~9.8k allocations on the bundled classes sample). One closure per analysis pass behaves identically. Validation: report-identity hashes across 10 bundled fixtures unchanged; tests/testDalvikDisassembler.py green. (cherry picked from commit 4ad6c0f)
Constructing a WinApiResolver parsed every configured ApiScout JSON database eagerly: ~26ms for the bundled XP profile, paid on every intel/ aarch64 analysis even when no API is ever resolved. Cached databases are still attached at construction so os-name selection stays cheap, while uncached ones load on the first getApi() call, after which the process- wide cache keeps later resolvers free. Adds a regression test covering the deferred path: nothing parsed at construction, first lookup parses and pins the os name. Validation: tests/testPeSymbolProvider.py green (31 passed); make lint, make test. (cherry picked from commit 62cabfb)
Two follow-ups from automated review of the perf branch: - the escape-cache limit was only checked when entering a function, so a single function with more distinct instructions than the cap could grow the process-global cache past it; enforce at insertion instead - the AArch64 top-byte prefilter built its slice and translate over the whole image before the first timeout poll, delaying an expiry that the old word loop would have caught within one 4096-word step; build the filter per polling chunk so discovery stops on the same boundaries as before Validation: report-identity hashes across 10 bundled fixtures unchanged; tests/testAArch64Disassembler.py, testCandidateSafeguards, testAnalysisBudgetBounds, testEscaper green; make lint. (cherry picked from commit 50c1f7d)
…fs fast path Two defects introduced by the preceding performance commits on this branch. Neither can move a fixture hash, so the report-identity validation those commits rest on could not see either one. Deferring ApiScout database parsing moved _resolveOsName() out of __init__ and into the first getApi() call. There it ran unconditionally, overwriting a name an intervening setOsName() had pinned; before the deferral the inference could only run while no caller had yet had the chance to set one. It now returns early when a name is already set, matching _ensureHashes(), which fills only fields that are still unset. _deferred_dbs is initialized on both construction paths as well, so reading it needs no getattr fallback and the non-deferred path no longer raises AttributeError. The getNormalizedBlockRefs() fast path returned each successor list unchanged, on the grounds that getBlockRefs() already emits them sorted and deduplicated. That holds for analysis-produced data, but the method is also the normalizer on the fromDict() path, where blockrefs come from a report this code did not necessarily write; there it became a no-op and duplicates reached the SCC and dominator-tree passes. Restoring sorted(set(...)) keeps what the fast path actually saves - it still skips the per-block try-range overlap test and the intermediate dict - and it stays 1.85x faster than the slow path it bypasses, measured interleaved over 319 functions from two bundled fixtures. Both fixes carry a regression test that fails on the pre-fix source. A whole-tree sweep for each root-cause signature found no further sites. (cherry picked from commit 35f3624)
The candidate queue re-pushed the same FunctionCandidate on every score change and compared live getScore() inside __lt__, so a large AArch64 image paid millions of heap comparisons and hundreds of full heapify()s. Each heap item now stores (-score, addr) at push time, add() is a no-op for a resident candidate, and next() skips stale items or re-pushes the current score. Validation: tests/testBracketQueue.py; libcrypto report identity unchanged; make lint. (cherry picked from commit 64464be)
bytearray([0] * size) built a Python list of zeros the size of the image before copying it. bytearray(n) is the same zero-fill without that temporary. Validation: tests/testPeFileLoader.py, testFileFormatParsers; make lint. (cherry picked from commit fbbd70f)
Several inner-loop structures paid per-instruction or per-byte work that did not change the recovered CFG: - FunctionAnalysisState kept a (from, to) pair-set beside the from/to maps. The pair-set is gone; .code_refs is a derived view. The cil and dalvik copies of the same bookkeeping match. - isInCodeAreas and _passesCodeFilter walked every range; they now merge and bisect. - gap construction walked every code_map byte key; it now merges recovered instruction intervals, with the byte walk only when functions is empty. - candidate_queue.update() heapified the whole heap after each dirty candidate; it now updates that candidate. - SmdaInstruction.from_tuple avoids str() of already-str fields, and getOutRefs walks each function once. Validation: targeted common/cil/dalvik/report tests; make lint; libcrypto and asprox report identity unchanged aside from execution_time. (cherry picked from commit f165d4e)
BL/prologue discovery now jumps to the next matching top byte with bytes.find instead of enumerating a mostly-zero translate buffer. Jump-table bounds use a sorted candidate/border cache and bisect instead of min() over the whole dict each time. _recordDataRefs still falls back to a detailed capstone decode, but adrp, adr, add/sub immediate, and movz/movn/movk immediates are taken from the instruction word. Those encodings match capstone's ARM64_OP_IMM values on libcrypto (zero mismatches after the #0 add/sub fallback). Interleaved A/B on libcrypto.3.dylib: faster in all three pairs, report hash 2a36c361b5381474 unchanged. Validation: tests/testAArch64Disassembler.py, testAArch64JumpTableLsl.py; make lint. (cherry picked from commit 62d153c)
Jump-table and stub/PLT scans used '.' with re.DOTALL over the mapped image. In a bytes pattern that is the slow spelling of [\x00-\xff]; the charset form matches a 0x0A displacement the same way and does not need the flag. The gap scanner also ran a 15-byte disasm_lite at every non-padding byte just to ask whether the mnemonic is nop. Bytes that cannot start a nop skip that decode and still go through the effective-NOP tables. Validation: tests/testJumpTableAnalyzer.py, tests/testIntelDisassembler.py stub-chain and gap tests; make lint. (cherry picked from commit a504f9d)
Fuzzing replayed a report whose xmetadata.imported_functions was a string. _getImportMap treated any truthy value as a dict and crashed on .items(). A non-dict (or a dict whose values are not name pairs) is now ignored, matching how fromDict already requires xmetadata itself to be a dictionary. Validation: tests/testSynthesis.py; make lint, make test. (cherry picked from commit 9081492)
Unsigned, pre-index, unscaled, and register-offset loads and signed-offset ldp/stp cannot produce IMM or absolute-MEM data refs, so returning None made every matching instruction pay for a detail decode that recorded nothing. Return an empty immediate tuple instead, and keep add/sub #0 on the word path. Post-index writeback still falls back: capstone 5 emits a separate IMM operand for those forms. (cherry picked from commit e832b38)
2e476ce re-keyed updateFunctionGaps() from a walk over code_map byte keys to merging the instruction intervals of disassembly.functions, leaving the byte walk reachable only when functions is empty. The two are not interchangeable. code_map records every decoded byte; disassembly.functions records only finalized functions. Deriving gaps from the latter drops the decoded-but-unattributed regions out of the gap scan, and with them every candidate the scan would have seeded there. On a 32-bit MSVC PE cross-compiled from Rust (hyperfine 1.19.0, i686-pc-windows-msvc), recovery falls from 6249 functions to 5247 -- a loss of 1002, or 16%. Measured across the range, every revision before 2e476ce reports 6249 and every revision after reports 5247. The rewrite did not pay for itself either. updateFunctionGaps() runs once per analysis, and the interval merge costs more than the byte walk it replaced: 0.0410s against 0.0365s cumulative under cProfile. Reverting restores 6249 at unchanged wall-clock. Only that hunk is reverted. The rest of 2e476ce keeps its wins -- the _passesCodeFilter bisect index, the candidate-queue update, the reference-pair bookkeeping removal, and the SmdaInstruction.from_tuple and getOutRefs work are untouched. 2e476ce also added test_interior_holes_match_the_covered_byte_walk, which passed an empty code_map and asserted a gap length derived from function intervals -- pinning the new behaviour under a name describing the old one, so the suite could not object. It now exercises code_map, and a second test asserts that gaps follow code_map coverage when both sources are populated; that test fails on the pre-revert source. The original change was validated on report identity for two binaries. Neither is gap-dominated, so a 16% recall loss was invisible to it. (cherry picked from commit 3ff05a8)
The name describes the covered byte walk, and an earlier version of the test asserted against an empty code_map with the length derived from function extents instead -- which pinned the behaviour that broke the contract under a name describing the contract. Recording in the docstring why both sources now carry something, so the next edit has to argue with it rather than quietly reintroduce the same shape. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PRvm5qFMW52aWnmVgR5vkD
…ilter
`_NOP_START_BYTES` gates the 15-byte disasm_lite probe on the theory that
anything outside it cannot be a nop. In 64-bit that is not true of 0x40-0x4F:
every REX prefix can open an instruction capstone spells `nop` -- `40 90`,
`48 0f 1f 00`, and `41 0f 1f 00` which it renders `nop dword ptr [r8]`.
Without them the gap scan stops recognising REX-padded alignment filler and
books a candidate on the padding. Driving nextGapCandidate() over a gap that
opens with each form, the first address it offers:
gap opens with before this branch with the filter as pushed
0f 1f 00 0x4 0x4
48 0f 1f 00 0x5 0x1
41 0f 1f 00 0x5 0x1
40 90 0x3 0x1
0x1 is the nop's own address. The plain multi-byte nop is the control.
The range goes in a separate 64-bit set rather than the shared one, because at
32 bits 0x40-0x4F are inc/dec and never a nop, so admitting them there would
spend a 15-byte disassembly on every inc or dec that opens a gap. Which bytes
each mode needs was swept against capstone over all 256 first bytes: 32-bit
needs nothing beyond the existing set, 64-bit needs exactly the REX range.
Both directions are pinned. The three encodings above fail on the pre-fix
source (4096 != 4100, 4096 != 4100, 4096 != 4098) and a 32-bit test asserts
disasm_lite is never reached for a REX byte.
No bundled fixture is REX-padded in a gap, which is why the branch's ten report
hashes could not see this -- the point its own third-defect note makes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014qdSgwKDgNvjtoDotuf8Mg
6cae1c5 to
b833695
Compare
|
Reviewed. #300 is in as Pre-push head The rebase itself was mechanical:
One bookkeeping note before anything else: your ten report hashes all change on this rebase, because master carries v4.5.1's version bump and your old base did not. The one thing I had to fix:
|
| gap opens with | master | this branch as pushed |
|---|---|---|
0f 1f 00 (plain multi-byte nop) |
0x4 |
0x4 |
48 0f 1f 00 (REX.W) |
0x5 |
0x1 |
41 0f 1f 00 (REX.B) |
0x5 |
0x1 |
40 90 (REX) |
0x3 |
0x1 |
0x1 is the nop's own address: the scan stops recognising REX-padded alignment filler and books a candidate on the padding instead of stepping over it. The plain-nop row is the control.
Fixed in b833695 as a separate 64-bit set rather than by widening the shared one, because at 32 bits 0x40–0x4F are inc/dec and never a nop — admitting them unconditionally would spend a 15-byte disassembly on every inc or dec that opens a gap, which in a perf change is the wrong direction. Selected once per nextGapCandidate() call by bitness.
Two tests, both directions: the three encodings fail on the pre-fix source (4096 != 4100, 4096 != 4100, 4096 != 4098) and pass after, and a 32-bit test asserts disasm_lite is never reached for a REX byte, so the mode split cannot quietly collapse later.
No bundled fixture is REX-padded in a gap, which is exactly why the ten report hashes could not see it. That is your own third-defect note arriving a second time, from a different direction, and it is the reason I went looking at the byte level rather than trusting the hashes — so the generalisation earned its keep within one review of being written down.
What I checked and what held
- PriorityQueue pop order is preserved exactly. This was the one I most expected to find something in: the rewrite swaps
other.element < self.elementfor a(neg_score, addr)min-heap, andFunctionCandidate.__lt__reverses on address when scores tie, so it looks like the tie-break flips. It does not — the two reversals cancel. Property-tested against master's_MaxHeapItemover 2,000 random heaps: 0 mismatches. - Report identity: 13 of 13 fixtures identical, under both candidate queues.
BracketQueueis not the default and is not in your ten, and it is the queue whoseupdate()call site the branch changes, so it is worth having on the record. (smda_versionexcluded, per the note above.) - The escape cache does not leak across reports. The key holds the escaper class, bytes, mnemonic, operands and the bounds, but
escapeBinaryalso reachesins.getDetailed(), which decodes under the report's capstone engine — a mode that is not in the key. I could not construct a collision, and 3,988 functions across 12 fixtures hash identically whether analysed in one shared process or in fresh ones. Recording it because onlylower_addr/upper_addrkeep a 32-bit and a 64-bit image apart today, and that is accidental rather than designed; architecture and bitness in the key would make it deliberate. - The AArch64 top-byte filter tables are exactly the achievable sets — not supersets. Derived from each pattern's mask (BL's
0xFCtop mask gives exactly0x94–0x97; all three prologue masks pin bits 31–24 outright, giving{0xA9, 0xD5, 0xF8}), then cross-checked by sampling ~2M words with rejected top bytes for zero misses._wordsViewindexing lines up withbinary[4i+3]on both host byte orders, including the big-endian materialised path. getNormalizedBlockRefs' fast path preserves key order. I built a function whoseself.blocksis deliberately out of ascending order to check — the slow path sorts at the end too, sosorted(self.blocks)matches by construction.- Dropping
fix_graphfromget_nesting_depthis sound: it only adds empty successor lists, which cannot reachlen(v) > 1. There.DOTALL→[\x00-\xff]rewrites are set-equivalent with no stray.left behind.
Two notes, neither blocking, both ours
finalizeAnalysisnow creates empty successor entries. Iteratingcode_refs_from.items()rather than pairs means anaddr_fromwhose destinations were all removed writesdisassembly.code_refs_from[addr] = set(), where the old pair loop wrote nothing. Reachable — 30 occurrences over 2,839 calls on five fixtures — and two membership tests inDisassemblyResult(lines 131 and 280) can see it, though no bundled fixture's report changes. Anif dests:guard makes it exact.- Two
getattr(self, "_code_area_index", None)reads where__init__could define the attribute, which is the pattern refactor(common): type the attributes the dropped annotations rest on #320 removed elsewhere. The index is keyed on list identity, so an in-place mutation ofcode_areaswould go unnoticed; nothing mutates it today. Related: theraw_binaryhoist inanalyzeFunctionis safe only because the one in-analysis rebinder ofbinary_info.binary— the Delphi relocation pass — runs in the candidate phase. Your comment covers the dicts and sets; that one is worth naming beside them.
Gates on the pushed head
python -m pytest tests/: 2094 passed, 2 skipped, 2596 subtestsruff check .andruff format --check .: cleanmake typecheck: exit 0, 0 error-level diagnostics
Merging once CI is green. Thanks for the third-defect writeup in particular — it is the reason this review looked where it did.
The changelog entry for what has landed since v4.5.1 (4868e19): #300, #325 and #299. Ten engine changes under feat(core) plus a hot-path pass is a minor-bump shape, which is why #299 and #300 were re-scoped off v4.5.1 rather than squeezed into it. VERSION in src/smda/SmdaConfig.py and __version__ in src/smda/__init__.py bumped together with the entry, per AGENTS.md. No escaper output changed anywhere in this set. The only escaping-related edit memoises escapeBinary results behind a cache keyed on its complete input tuple and no escaper module is touched, so ESCAPER_DOWNWARD_COMPATIBILITY stays at 4.4.5 and INTEL_PIC_HASH_ESCAPE_VERSION at 4.3.5, and no report needs reprocessing. Figures are the contributor's own except the ARM64 Mach-O row and the ReadyToRun result, which were reproduced during the #300 review; the entry says which is which rather than presenting one table as equally checked. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The changelog entry for what has landed since v4.5.1 (4868e19): #300, #325 and #299. Ten engine changes under feat(core) plus a hot-path pass is a minor-bump shape, which is why #299 and #300 were re-scoped off v4.5.1 rather than squeezed into it. VERSION in src/smda/SmdaConfig.py and __version__ in src/smda/__init__.py bumped together with the entry, per AGENTS.md. Deliberately still the current changelog format rather than keep-a-changelog. #323 proposed adopting against an empty Unreleased so the first entries it holds are written by their authors at merge time; v4.6.0 cannot be that, since #299's and #300's entries are written here at release time either way. Adopting after this release lets the inaugural section fill itself from the seven PRs currently open. What this entry does adopt is the compromise: each topic keeps the mechanism and its headline figure with the cost, and links the PR carrying the full measurement and the dead ends -- ~2,200 words against v4.5.1's ~2,790, and the first entry in the file to use links at all. No escaper output changed anywhere in this set. The only escaping-related edit memoises escapeBinary results behind a cache keyed on its complete input tuple and no escaper module is touched, so ESCAPER_DOWNWARD_COMPATIBILITY stays at 4.4.5 and INTEL_PIC_HASH_ESCAPE_VERSION at 4.3.5, and no report needs reprocessing. Figures are the contributor's own except the ARM64 Mach-O row and the ReadyToRun result, which were reproduced during the #300 review; the entry says which is which rather than presenting one table as equally checked. Also fixes AGENTS.md:117, which had prescribed a one-line changelog entry since before v4.5.1 while v4.5.1 and v4.6.0 both use the nested **Topic:** list. Independent of #323, which would replace that line again on adoption. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Summary
Six focused changes that cut measurable overhead from the analysis pipeline, plus a follow-up commit fixing two defects that self-review found in two of them.
analyzeFunctioncode_map/data_map/ins2fn/buffer/bound methods to locals in the instruction loop (~180k attribute chains per 29k-instruction sample); containers are only mutated in place, so aliasing is safeescapeBinary;getNormalizedBlockRefsfast path for functions without try_ranges; drop a duplicatefix_graph()in the nesting-depth passbytes.translateover the top-byte lane skips non-matching words at C speed; full per-hit verification kept, timeout polling boundaries unchangedhasUnprocessedBlocks()allocated a set difference on every call although the queue and processed sets are disjoint by construction; O(blocks^2) allocation on wide functionsEvery change was benchmarked before and after, and full serialized reports hash byte-for-byte equal to master across ten bundled fixtures covering all four backends (intel x86/x86-64 memory dumps, aarch64 static + Mach-O, dalvik DEX classes, CIL), with volatile fields (
timestamp,execution_time) excluded from the comparison.Interleaved median wall-clock vs master on this machine: aarch64_static -9 to -10%, komplex -6 to -7%, blockblast/asprox -2 to -4%, cutwail/njrat neutral within noise.
Follow-up fixes
Self-review of the six commits above found two defects. Both live on paths that no bundled fixture reaches, which is exactly why the report-identity check could not see either one: neither can move a fixture hash. Both are fixed in the final commit, each with a regression test that fails on the pre-fix source.
An explicit ApiScout OS name was silently discarded. Deferring database parsing moved the os-name inference out of
WinApiResolver.__init__and into the firstgetApi()call, where it ran unconditionally and overwrote a name an interveningsetOsName()had pinned. Before the deferral, the inference could only run while no caller had yet had the chance to set one, so the setter always won. It now returns early when a name is already set, matchingDisassembler._ensureHashes, which fills only fields that are still unset._deferred_dbsis initialized on both construction paths as well, so reading it needs nogetattrfallback and the non-deferred path no longer raisesAttributeError.The
getNormalizedBlockRefs()fast path stopped normalizing. It returned each successor list unchanged, on the grounds thatgetBlockRefs()already emits them sorted and deduplicated. That holds for analysis-produced data, but the same method is also the normalizer on thefromDict()path, where blockrefs come from a report this code did not necessarily write; there it became a no-op, and duplicates reached the SCC and dominator-tree passes. Restoringsorted(set(...))keeps what the fast path actually saves - it still skips the per-block try-range overlap test and the intermediate dict - and it stays 1.85x faster than the slow path it bypasses, measured interleaved over 319 functions harvested from two bundled fixtures. Equivalence with the slow path was then re-checked over 3000 randomized inputs, including blockrefs carrying keys absent fromblocks: values, key set and key order all match.A whole-tree sweep for each root-cause signature found no further sites in
src/smda/.A third defect, and what it says about the validation above
The report-identity check missed a third defect, and this one is worse than the two above: the path was not unreached, it was under-represented.
perf(common): drop redundant ref bookkeeping and speed hot lookupsre-keyedupdateFunctionGaps()from a walk overcode_mapbyte keys to merging the instruction intervals ofdisassembly.functions, leaving the byte walk reachable only whenfunctionsis empty.The two are not interchangeable.
code_maprecords every decoded byte;disassembly.functionsrecords only finalized functions. Deriving gaps from the latter drops the decoded-but-unattributed regions out of the gap scan, and with them every candidate the scan would have seeded there. On a 32-bit MSVC PE cross-compiled from Rust (hyperfine 1.19.0,i686-pc-windows-msvc), recovery falls from 6249 functions to 5247 - a loss of 1002, or 16%. Measured revision by revision across this branch, everything before that commit reports 6249 and everything after reports 5247.The rewrite did not pay for itself either.
updateFunctionGaps()runs once per analysis, and the interval merge costs more than the byte walk it replaced: 0.0410s against 0.0365s cumulative under cProfile. Reverting restores 6249 at unchanged wall-clock, so the speedups claimed above are unaffected.Only that hunk is reverted. The rest of the commit keeps its wins - the
_passesCodeFilterbisect index, the candidate-queue update, the reference-pair bookkeeping removal, and theSmdaInstruction.from_tupleandgetOutRefswork are untouched.The same commit also added
test_interior_holes_match_the_covered_byte_walk, which passed an emptycode_mapand asserted a gap length derived from function intervals - pinning the new behaviour under a name describing the old contract, so the suite could not object to it. It now exercisescode_map, and a second test asserts that gaps followcode_mapcoverage when both sources are populated; that test fails on the pre-revert source.None of the ten bundled fixtures is gap-dominated, which is why a 16% recall loss could not move a single hash. The lesson from the two defects above generalizes further than it was first written: a report-identity comparison is evidence about the fixtures, not about the change, and it is only ever as strong as the fixtures are representative.
Changed behavior
Reports serialize identically on all fixtures listed above, and the golden corpus expectations are untouched. That statement is about the fixtures and not about every binary: as described in the section above, one commit did change recovery on a 32-bit MSVC PE outside the fixture set, and the revert restores it. Three behavioral nuances are intentional. An uncached ApiScout database is now parsed at the first
getApi()call rather than at resolver construction, so the missing-file log timing moves with it. An explicitsetOsName()outranks the os-name inference regardless of when the deferred load fires, restoring the pre-deferral ordering. AndgetNormalizedBlockRefs()still deduplicates and sorts successor lists on thefromDict()path, as it did before the fast path was introduced.Approaches that were tried and rejected after measurement, for the record: enlarging Capstone disassembly windows (slower despite fewer calls - object construction dominates), batch-decoding instruction details for string extraction (slower for the same reason), fusing the intel candidate-scan regexes (measured 1.0x), interval-backed
processed_bytes(no gain).Validation
python -m pytest tests/: 1847 passed, 1 skipped, 2584 subtestsmake lintandruff format --check .: cleanmake typecheck: exit 0, diagnostic count unchanged against the same tree without these changestoDict()SHA256, volatile fields scrubbed) equal to master across 10 fixtures spanning intel/aarch64/dalvik/cil, default config and WITH_STRINGS=Truestatus: ok, analysis ~1.8s