Skip to content

perf: cut analysis hot-path overhead without changing output - #299

Merged
danielplohmann merged 18 commits into
danielplohmann:masterfrom
r0ny123:perf/analysis-hot-path-overhead
Sep 10, 2026
Merged

perf: cut analysis hot-path overhead without changing output#299
danielplohmann merged 18 commits into
danielplohmann:masterfrom
r0ny123:perf/analysis-hot-path-overhead

Conversation

@r0ny123

@r0ny123 r0ny123 commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Summary

Six focused changes that cut measurable overhead from the analysis pipeline, plus a follow-up commit fixing two defects that self-review found in two of them.

Commit What it does Indicative effect
hoist per-instruction lookups out of analyzeFunction binds code_map/data_map/ins2fn/buffer/bound methods to locals in the instruction loop (~180k attribute chains per 29k-instruction sample); containers are only mutated in place, so aliasing is safe ~1-3% end-to-end
reuse escape results + skip redundant report-build work PIC-hash escaping memoized behind a size-capped cache keyed on the complete input tuple of escapeBinary; getNormalizedBlockRefs fast path for functions without try_ranges; drop a duplicate fix_graph() in the nesting-depth pass real binaries repeat 30-50% of instructions (98% in a DEX classes dump)
prefilter AArch64 candidate scans by top byte BL encodings have top byte 0x94-0x97 and prologues {0xA9, 0xD5, 0xF8} - verified exhaustively over each pattern's free bits - so one bytes.translate over the top-byte lane skips non-matching words at C speed; full per-hit verification kept, timeout polling boundaries unchanged prologue scan 4.14ms -> 0.74ms, BL 1.17ms -> 0.59ms; scans are O(image bytes), so large ARM64 dumps save proportionally more
trust the block queue as the emptiness signal hasUnprocessedBlocks() allocated a set difference on every call although the queue and processed sets are disjoint by construction; O(blocks^2) allocation on wide functions removes the quadratic trap
hoist Dalvik resolver closure one resolver closure per analysis pass instead of one per decoded instruction (~9.8k allocations avoided) small but free
defer ApiScout DB parsing to first API lookup uncached databases cost ~26ms at backend construction even when no API is ever resolved; cached ones still attach eagerly so os-name selection stays cheap cold-start only when API files are configured

Every change was benchmarked before and after, and full serialized reports hash byte-for-byte equal to master across ten bundled fixtures covering all four backends (intel x86/x86-64 memory dumps, aarch64 static + Mach-O, dalvik DEX classes, CIL), with volatile fields (timestamp, execution_time) excluded from the comparison.

Interleaved median wall-clock vs master on this machine: aarch64_static -9 to -10%, komplex -6 to -7%, blockblast/asprox -2 to -4%, cutwail/njrat neutral within noise.

Follow-up fixes

Self-review of the six commits above found two defects. Both live on paths that no bundled fixture reaches, which is exactly why the report-identity check could not see either one: neither can move a fixture hash. Both are fixed in the final commit, each with a regression test that fails on the pre-fix source.

An explicit ApiScout OS name was silently discarded. Deferring database parsing moved the os-name inference out of WinApiResolver.__init__ and into the first getApi() call, where it ran unconditionally and overwrote a name an intervening setOsName() had pinned. Before the deferral, the inference could only run while no caller had yet had the chance to set one, so the setter always won. It now returns early when a name is already set, matching Disassembler._ensureHashes, which fills only fields that are still unset. _deferred_dbs is initialized on both construction paths as well, so reading it needs no getattr fallback and the non-deferred path no longer raises AttributeError.

The getNormalizedBlockRefs() fast path stopped normalizing. It returned each successor list unchanged, on the grounds that getBlockRefs() already emits them sorted and deduplicated. That holds for analysis-produced data, but the same method is also the normalizer on the fromDict() path, where blockrefs come from a report this code did not necessarily write; there it became a no-op, and duplicates reached the SCC and dominator-tree passes. Restoring sorted(set(...)) keeps what the fast path actually saves - it still skips the per-block try-range overlap test and the intermediate dict - and it stays 1.85x faster than the slow path it bypasses, measured interleaved over 319 functions harvested from two bundled fixtures. Equivalence with the slow path was then re-checked over 3000 randomized inputs, including blockrefs carrying keys absent from blocks: values, key set and key order all match.

A whole-tree sweep for each root-cause signature found no further sites in src/smda/.

A third defect, and what it says about the validation above

The report-identity check missed a third defect, and this one is worse than the two above: the path was not unreached, it was under-represented. perf(common): drop redundant ref bookkeeping and speed hot lookups re-keyed updateFunctionGaps() from a walk over code_map byte keys to merging the instruction intervals of disassembly.functions, leaving the byte walk reachable only when functions is empty.

The two are not interchangeable. code_map records every decoded byte; disassembly.functions records only finalized functions. Deriving gaps from the latter drops the decoded-but-unattributed regions out of the gap scan, and with them every candidate the scan would have seeded there. On a 32-bit MSVC PE cross-compiled from Rust (hyperfine 1.19.0, i686-pc-windows-msvc), recovery falls from 6249 functions to 5247 - a loss of 1002, or 16%. Measured revision by revision across this branch, everything before that commit reports 6249 and everything after reports 5247.

The rewrite did not pay for itself either. updateFunctionGaps() runs once per analysis, and the interval merge costs more than the byte walk it replaced: 0.0410s against 0.0365s cumulative under cProfile. Reverting restores 6249 at unchanged wall-clock, so the speedups claimed above are unaffected.

Only that hunk is reverted. The rest of the commit keeps its wins - the _passesCodeFilter bisect index, the candidate-queue update, the reference-pair bookkeeping removal, and the SmdaInstruction.from_tuple and getOutRefs work are untouched.

The same commit also added test_interior_holes_match_the_covered_byte_walk, which passed an empty code_map and asserted a gap length derived from function intervals - pinning the new behaviour under a name describing the old contract, so the suite could not object to it. It now exercises code_map, and a second test asserts that gaps follow code_map coverage when both sources are populated; that test fails on the pre-revert source.

None of the ten bundled fixtures is gap-dominated, which is why a 16% recall loss could not move a single hash. The lesson from the two defects above generalizes further than it was first written: a report-identity comparison is evidence about the fixtures, not about the change, and it is only ever as strong as the fixtures are representative.

Changed behavior

Reports serialize identically on all fixtures listed above, and the golden corpus expectations are untouched. That statement is about the fixtures and not about every binary: as described in the section above, one commit did change recovery on a 32-bit MSVC PE outside the fixture set, and the revert restores it. Three behavioral nuances are intentional. An uncached ApiScout database is now parsed at the first getApi() call rather than at resolver construction, so the missing-file log timing moves with it. An explicit setOsName() outranks the os-name inference regardless of when the deferred load fires, restoring the pre-deferral ordering. And getNormalizedBlockRefs() still deduplicates and sorts successor lists on the fromDict() path, as it did before the fast path was introduced.

Approaches that were tried and rejected after measurement, for the record: enlarging Capstone disassembly windows (slower despite fewer calls - object construction dominates), batch-decoding instruction details for string extraction (slower for the same reason), fusing the intel candidate-scan regexes (measured 1.0x), interval-backed processed_bytes (no gain).

Validation

  • python -m pytest tests/ : 1847 passed, 1 skipped, 2584 subtests
  • make lint and ruff format --check .: clean
  • make typecheck: exit 0, diagnostic count unchanged against the same tree without these changes
  • Report-identity hashes (serialized toDict() SHA256, volatile fields scrubbed) equal to master across 10 fixtures spanning intel/aarch64/dalvik/cil, default config and WITH_STRINGS=True
  • Function recovery on the 32-bit MSVC PE described above: 5247 -> 6249, status: ok, analysis ~1.8s
  • Diff coverage against the base branch: 99% overall, 100% on the lines changed by the revert. The two uncovered lines are the residual per-hit verification arms of the AArch64 top-byte prefilter (a word whose top byte matches but whose full encoding does not, and a prologue rejected by the code filter); neither is reached by a bundled fixture today.

@r0ny123

r0ny123 commented Aug 27, 2026

Copy link
Copy Markdown
Contributor Author

Pushed f8c3e57, which fixes two defects that self-review found in the earlier commits on this branch. Body updated with the details; the numbers below are the final ones for the branch as it now stands.

Neither defect could move a fixture hash, so the report-identity check the earlier commits rest on could not see either. setOsName() is never called anywhere in the analysis pipeline, and the blockrefs divergence needs input that this code does not itself produce — it only surfaces on the fromDict() path.

Reproductions, before and after the fix

Explicit OS name, with two configured ApiScout databases where one file is missing:

before  os_name='good'     getApi(0x70001000) -> ('good.dll', 'GoodApi')
after   os_name='absent'   getApi(0x70001000) -> (None, None)      # matches master

Blockrefs normalization, blockrefs = {0x10: [0x20, 0x20, 0x08]}:

before  {16: [32, 32, 8]}
after   {16: [8, 32]}                                              # matches master

Validation

Gate Result
full test suite 1817 passed, 1 skipped, exit 0
lint and format check clean
typecheck exit 0, diagnostic count identical to the same tree without these changes
both new tests against the pre-fix source 2 failed — the tests are non-vacuous
fast-path vs slow-path equivalence, 3000 randomized inputs 0 mismatches in values, key set and key order
whole-tree sweep, both root-cause signatures clean, 0 siblings outstanding
diff coverage vs base 98% overall, 100% on both files this commit touches

Performance impact of the fix

Restoring sorted(set(...)) gives back part of the fast path's saving but keeps most of it. Measured interleaved in one process over 319 functions (2545 blocks, 3342 edges) harvested from komplex and asprox:

slow path (master)              0.68 ms   1.00x
fast path, list()               0.20 ms   0.30x
fast path, sorted(set())        0.37 ms   0.54x

The absolute difference between the two fast-path variants is 0.17 ms across an entire report, against multi-second analyses, and getNormalizedBlockRefs() is memoized per function. This line was never load-bearing for the headline numbers in either direction.

The two uncovered diff lines are in the AArch64 top-byte prefilter from an earlier commit, not in this one: the arm taken when a word's top byte matches but its full encoding does not, and a prologue rejected by the code filter. Both are genuine residual-verification paths that no bundled fixture reaches. Happy to add coverage for them if you would prefer the gate at 100%.

@cursor
cursor Bot force-pushed the perf/analysis-hot-path-overhead branch from bb420ed to 66cf167 Compare August 28, 2026 09:56
@r0ny123

r0ny123 commented Aug 28, 2026

Copy link
Copy Markdown
Contributor Author

Pushed 999fe74, which reverts one hunk of perf(common): drop redundant ref bookkeeping and speed hot lookups and adds the regression tests for it. The PR body now carries the detail; the short version and the numbers are here.

That commit re-keyed updateFunctionGaps() from a walk over code_map byte keys to merging the instruction intervals of disassembly.functions. code_map records every decoded byte and disassembly.functions only finalized functions, so the new source silently drops decoded-but-unattributed regions out of the gap scan along with every candidate the scan would have seeded there.

Measured on a 32-bit MSVC PE cross-compiled from Rust (hyperfine 1.19.0, i686-pc-windows-msvc), disassembled with each revision of this branch in turn:

revision functions status
every revision before the commit 6249 ok
the commit, and every revision after it 5247 ok
after 999fe74 6249 ok

That is 1002 functions, or 16%. The rewrite was also not faster: updateFunctionGaps() runs once per analysis, and the interval merge costs 0.0410s cumulative under cProfile against 0.0365s for the byte walk it replaced, so the revert restores recovery at unchanged wall-clock and none of the speedups this PR claims are affected. Only that hunk is reverted; the _passesCodeFilter bisect index, the candidate-queue update, the reference-pair bookkeeping removal and the SmdaInstruction.from_tuple and getOutRefs work are untouched.

Worth flagging separately: the same commit added test_interior_holes_match_the_covered_byte_walk, which passed an empty code_map and asserted a gap length derived from function intervals. It pinned the new behaviour under a name describing the old contract, so the suite could not object. It now exercises code_map, and the added test_gaps_follow_code_map_coverage_not_recovered_function_extents asserts that gaps follow code_map coverage when both sources are populated; it fails on the pre-revert source.

This is the third defect the report-identity check did not catch, and the first that is not explained by an unreached path. None of the ten bundled fixtures is gap-dominated, so a 16% recall loss could not move a single hash.

Validation on the pushed head: the full suite reports 1847 passed, 1 skipped and 2584 subtests; lint and format checks are clean; the type checker exits 0 with no error-level diagnostics; and diff coverage against the base is 99% overall with 100% on the lines changed by the revert.

@danielplohmann

Copy link
Copy Markdown
Owner

Same pass as #300 (comment), and this one inherits that PR's position because it contains it — the branch carries #300's 456b855 plus sixteen of its own perf and fix commits on top.

Cleared: fuzz_buffer was infra, not a hang

Yesterday's open question on this PR was whether the fuzz_buffer job failing at 55 minutes against a timeout-minutes: 45 cap was an infra timeout or a hang the perf changes introduced. The logs had expired, so it needed a re-run to tell those apart.

It has since re-run green: Fuzz (fuzz_buffer) passes in 2m45s, along with all five other fuzz targets. Nothing here hangs. That item is closed and does not need to come back.

The rest is blocked behind #300

Both of this PR's problems are #300's problems, arriving one layer down:

What I have read so far, and liked

I have not reviewed the diff line by line yet, for the same reason as #300 — it is not readable until rebased. But the PR body is, and two things in it are worth saying out loud now rather than at merge time.

The third defect writeup is the most useful thing in either PR. Finding that updateFunctionGaps() had been re-keyed from code_map to disassembly.functions, that this silently dropped decoded-but-unattributed regions out of the gap scan, that it cost 1,002 functions (16%) on a 32-bit MSVC PE, and that no bundled fixture is gap-dominated enough to notice — and then reverting only that hunk while keeping the rest of the commit's wins — is a better outcome than the change never having been made. The generalisation you drew is the right one and I would like it on the record beyond this PR: a report-identity comparison is evidence about the fixtures, not about the change. That is precisely the failure mode that let it through, and it is the same shape as the two figures we shipped in v4.5.1 marked as unverifiable because no bundled fixture reaches them (#322).

Pinning a new behaviour under a name describing the old contracttest_interior_holes_match_the_covered_byte_walk passing an empty code_map and asserting a gap length derived from function intervals — is the sharpest single example of a test that cannot fail I have seen in this repo. Worth keeping the corrected version's docstring explicit about why it now populates both sources.

Also noting for the record that you list the approaches you measured and rejected — larger capstone windows, batch detail decoding, fused candidate regexes, interval-backed processed_bytes. That saves the next person from re-deriving four dead ends, and it is unusual to bother writing down.

Order of operations

  1. feat(core): function-boundary accuracy from what the image declares #300 rebases onto current master and re-measures.
  2. This rebases on top and regenerates the ten report hashes against the same tree.
  3. Then I review both properly.

No urgency — neither is going into v4.5.1, and we are travelling shortly. Better measured than fast.

@r0ny123
r0ny123 force-pushed the perf/analysis-hot-path-overhead branch from 3455546 to 6cae1c5 Compare September 9, 2026 02:08
r0ny123 pushed a commit to r0ny123/smda that referenced this pull request Sep 9, 2026
…iew replies for posting

Not part of either PR branch. These are the comment bodies and the
description edits produced by the rebase-and-re-measure pass, parked on
this scratch branch so a session with API access to the upstream repo can
fetch and post them.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PRvm5qFMW52aWnmVgR5vkD
@r0ny123

r0ny123 commented Sep 9, 2026

Copy link
Copy Markdown
Contributor Author

Rebased, in the order you asked for: #300 first onto 6240b74, then this on top of it.

What is in it now

Twenty-one commits — #300's three plus the test-repair commit its rebase needed, then the sixteen of this branch's own, then one more: the docstring you asked for on test_interior_holes_match_the_covered_byte_walk. The pre-squash copies of #302, #306, #307, #308, #309 and #310 are gone, including the two on this branch that were #310's series (resume past a refused landing pad's block and stop the refused-pad walk at a trap); master carries both as 7d01480.

Four of the sixteen needed real resolution rather than a textual merge:

The report hashes, regenerated against the tree this lands on

You are right that the old claim had gone stale, and for the reason you named rather than for a rebase reason. Regenerated:

Ten bundled fixtures, computeReportIdentityHash over the full serialized report, timestamp and execution_time excluded. Three trees: current master 6240b74, #300 rebased onto it, and this branch rebased onto that.

fixture backend master 6240b74 #300 rebased this branch
asprox intel d961211b d961211b d961211b
cutwail intel e201853a e201853a e201853a
bashlite intel be88d8f5 be88d8f5 be88d8f5
blockblast dalvik 767ebb8e 767ebb8e 767ebb8e
komplex intel 7fce68c9 7fce68c9 7fce68c9
njrat cil cc2a2ea3 cc2a2ea3 cc2a2ea3
pe_export intel 6d4b219a 6d4b219a 6d4b219a
aarch64_static aarch64 1f415f37 3613e186 3613e186
aarch64_switch_macho_O0 aarch64 418c8419 418c8419 418c8419
dotnet_readytorun intel 713c981f 9c651848 9c651848

This branch is byte-for-byte equal to the tree it lands on, on all ten. That is the claim the PR makes and it is now made against the right baseline rather than against v4.5.0.

The two that move do so between master and #300, not here: aarch64_static 278 → 276 functions and dotnet_readytorun 419 → 626, both #300's business and both explained over there.

And with your point taken about what this proves: it proves these sixteen commits change nothing on these ten images. It is not evidence about updateFunctionGaps, because none of the ten is gap-dominated — which is the whole reason 3ff05a8 exists. The gap-scan behaviour is pinned by test_gaps_follow_code_map_coverage_not_recovered_function_extents, which fails on the pre-revert source, rather than by any hash.

fuzz_buffer

Noted, and thanks for re-running it rather than leaving it as an open question on the branch. Nothing here holds a lock or loops on buffer content, so I had no better explanation than "infra" either — good to have it settled by a green run rather than by argument.

On the third defect

Taking the generalisation as a standing rule rather than a one-off, since you asked for it on the record: a report-identity comparison is evidence about the fixtures, not about the change. Concretely: an identity hash is a gate only for paths a fixture actually reaches. Everywhere else it reports coverage and reads as behaviour, which is exactly how the 16% gap-scan recall loss got through — none of the ten bundled fixtures is gap-dominated, so no hash could have moved whatever the change did.

I have made the corrected test_interior_holes_match_the_covered_byte_walk docstring say why it populates both sources, per your note.

Gates

Full suite on the rebased branch: 2092 passed, 2 skipped, 2593 subtests. ruff check and ruff format --check clean, make typecheck exit 0 with no error-level diagnostics. (#300's own tree, underneath this one, is 2059 passed, 2 skipped, 2593 subtests.)

The instruction loop re-resolved disassembly.code_map, data_map, ins2fn,
binary_info.binary, the backend handler and state.processed_bytes for every
decoded instruction (~180k attribute chains on a 29k-instruction sample).
All of these are only mutated in place during analysis, so binding them to
locals at loop entry is identity-safe.

Validation: report-identity hashes across 10 bundled intel/aarch64/dalvik/cil
fixtures unchanged; make lint, make test.

(cherry picked from commit fc8d2fe)
Three pieces of per-function rework in report construction:

- getPicHashSequence now reuses escaped sequences through a size-capped
  cache keyed on (escaper, bytes, mnemonic, operands, flag, bounds), the
  complete input set of escapeBinary, so hits are byte-identical by
  construction. Real samples repeat 30-50% of their instructions
  (98% in dalvik classes dumps), and the cap bounds retained strings.
- getNormalizedBlockRefs rebuilt a sorted copy of what getBlockRefs had
  just produced as sorted, deduplicated lists whenever a function carries
  no try_ranges - which is every function on intel and aarch64. The fast
  path returns that same content directly.
- get_nesting_depth ran fix_graph() on a graph build_dominator_tree() had
  already fixed; the expansion only adds empty successor lists, which can
  never contribute to significant_nodes.

Validation: report-identity hashes across 10 bundled fixtures unchanged;
make lint, make test.

(cherry picked from commit af75c2a)
The BL and prologue discovery passes tested all ~37k aligned words per
scan in Python. Every BL encoding (mask 0xFC000000) has a top byte in
0x94..0x97 and every recognized prologue pattern has one in {0xA9, 0xD5,
0xF8} - verified exhaustively over each pattern's free bits - so one
bytes.translate over the top-byte lane skips non-matching words at C
speed before any Python runs. Full verification of each hit is kept.

Timeout polling stays on the same 4096-word boundaries, so candidate
sets under the budget guards are unchanged. Prologue scan drops from
4.14ms to 0.74ms and BL from 1.17ms to 0.59ms on the bundled static
ARM64 sample; scans are O(image bytes), so multi-MB ARM64 inputs save
proportionally more.

Validation: report-identity hashes across 10 bundled fixtures unchanged;
tests/testAArch64Disassembler.py green; make lint, make test.

(cherry picked from commit 88e6cb0)
hasUnprocessedBlocks() allocated queued_blocks - processed_blocks on
every call. chooseNextBlock() moves each pop straight into
processed_blocks and addBlockToQueue() refuses processed blocks, so the
two sets are always disjoint and the difference is just the queue itself.
On a function with thousands of blocks the per-block set copies added up
to quadratic allocation; the dalvik twin of this method got the same fix.

Validation: report-identity hashes across 10 bundled fixtures unchanged;
make lint, make test.

(cherry picked from commit cfa6c64)
The CFG walk allocated an identical resolver lambda for every decoded
instruction (~9.8k allocations on the bundled classes sample). One closure
per analysis pass behaves identically.

Validation: report-identity hashes across 10 bundled fixtures unchanged;
tests/testDalvikDisassembler.py green.

(cherry picked from commit 4ad6c0f)
Constructing a WinApiResolver parsed every configured ApiScout JSON
database eagerly: ~26ms for the bundled XP profile, paid on every intel/
aarch64 analysis even when no API is ever resolved. Cached databases are
still attached at construction so os-name selection stays cheap, while
uncached ones load on the first getApi() call, after which the process-
wide cache keeps later resolvers free.

Adds a regression test covering the deferred path: nothing parsed at
construction, first lookup parses and pins the os name.

Validation: tests/testPeSymbolProvider.py green (31 passed); make lint,
make test.

(cherry picked from commit 62cabfb)
Two follow-ups from automated review of the perf branch:

- the escape-cache limit was only checked when entering a function, so a
  single function with more distinct instructions than the cap could grow
  the process-global cache past it; enforce at insertion instead
- the AArch64 top-byte prefilter built its slice and translate over the
  whole image before the first timeout poll, delaying an expiry that the
  old word loop would have caught within one 4096-word step; build the
  filter per polling chunk so discovery stops on the same boundaries as
  before

Validation: report-identity hashes across 10 bundled fixtures unchanged;
tests/testAArch64Disassembler.py, testCandidateSafeguards,
testAnalysisBudgetBounds, testEscaper green; make lint.

(cherry picked from commit 50c1f7d)
…fs fast path

Two defects introduced by the preceding performance commits on this branch.
Neither can move a fixture hash, so the report-identity validation those
commits rest on could not see either one.

Deferring ApiScout database parsing moved _resolveOsName() out of __init__
and into the first getApi() call. There it ran unconditionally, overwriting
a name an intervening setOsName() had pinned; before the deferral the
inference could only run while no caller had yet had the chance to set one.
It now returns early when a name is already set, matching _ensureHashes(),
which fills only fields that are still unset. _deferred_dbs is initialized
on both construction paths as well, so reading it needs no getattr fallback
and the non-deferred path no longer raises AttributeError.

The getNormalizedBlockRefs() fast path returned each successor list
unchanged, on the grounds that getBlockRefs() already emits them sorted and
deduplicated. That holds for analysis-produced data, but the method is also
the normalizer on the fromDict() path, where blockrefs come from a report
this code did not necessarily write; there it became a no-op and duplicates
reached the SCC and dominator-tree passes. Restoring sorted(set(...)) keeps
what the fast path actually saves - it still skips the per-block try-range
overlap test and the intermediate dict - and it stays 1.85x faster than the
slow path it bypasses, measured interleaved over 319 functions from two
bundled fixtures.

Both fixes carry a regression test that fails on the pre-fix source. A
whole-tree sweep for each root-cause signature found no further sites.

(cherry picked from commit 35f3624)
The candidate queue re-pushed the same FunctionCandidate on every
score change and compared live getScore() inside __lt__, so a large
AArch64 image paid millions of heap comparisons and hundreds of
full heapify()s. Each heap item now stores (-score, addr) at push
time, add() is a no-op for a resident candidate, and next() skips
stale items or re-pushes the current score.

Validation: tests/testBracketQueue.py; libcrypto report identity
unchanged; make lint.

(cherry picked from commit 64464be)
bytearray([0] * size) built a Python list of zeros the size of the
image before copying it. bytearray(n) is the same zero-fill without
that temporary.

Validation: tests/testPeFileLoader.py, testFileFormatParsers; make lint.
(cherry picked from commit fbbd70f)
Several inner-loop structures paid per-instruction or per-byte work
that did not change the recovered CFG:

- FunctionAnalysisState kept a (from, to) pair-set beside the
  from/to maps. The pair-set is gone; .code_refs is a derived view.
  The cil and dalvik copies of the same bookkeeping match.
- isInCodeAreas and _passesCodeFilter walked every range; they now
  merge and bisect.
- gap construction walked every code_map byte key; it now merges
  recovered instruction intervals, with the byte walk only when
  functions is empty.
- candidate_queue.update() heapified the whole heap after each
  dirty candidate; it now updates that candidate.
- SmdaInstruction.from_tuple avoids str() of already-str fields,
  and getOutRefs walks each function once.

Validation: targeted common/cil/dalvik/report tests; make lint;
libcrypto and asprox report identity unchanged aside from
execution_time.

(cherry picked from commit f165d4e)
r0ny123 and others added 7 commits September 10, 2026 17:40
BL/prologue discovery now jumps to the next matching top byte with
bytes.find instead of enumerating a mostly-zero translate buffer.
Jump-table bounds use a sorted candidate/border cache and bisect
instead of min() over the whole dict each time.

_recordDataRefs still falls back to a detailed capstone decode, but
adrp, adr, add/sub immediate, and movz/movn/movk immediates are
taken from the instruction word. Those encodings match capstone's
ARM64_OP_IMM values on libcrypto (zero mismatches after the #0
add/sub fallback). Interleaved A/B on libcrypto.3.dylib: faster in
all three pairs, report hash 2a36c361b5381474 unchanged.

Validation: tests/testAArch64Disassembler.py,
testAArch64JumpTableLsl.py; make lint.

(cherry picked from commit 62d153c)
Jump-table and stub/PLT scans used '.' with re.DOTALL over the mapped
image. In a bytes pattern that is the slow spelling of [\x00-\xff];
the charset form matches a 0x0A displacement the same way and does
not need the flag.

The gap scanner also ran a 15-byte disasm_lite at every non-padding
byte just to ask whether the mnemonic is nop. Bytes that cannot start
a nop skip that decode and still go through the effective-NOP tables.

Validation: tests/testJumpTableAnalyzer.py,
tests/testIntelDisassembler.py stub-chain and gap tests; make lint.

(cherry picked from commit a504f9d)
Fuzzing replayed a report whose xmetadata.imported_functions was a
string. _getImportMap treated any truthy value as a dict and crashed
on .items(). A non-dict (or a dict whose values are not name pairs)
is now ignored, matching how fromDict already requires xmetadata
itself to be a dictionary.

Validation: tests/testSynthesis.py; make lint, make test.
(cherry picked from commit 9081492)
Unsigned, pre-index, unscaled, and register-offset loads and signed-offset
ldp/stp cannot produce IMM or absolute-MEM data refs, so returning None
made every matching instruction pay for a detail decode that recorded
nothing. Return an empty immediate tuple instead, and keep add/sub #0 on
the word path.

Post-index writeback still falls back: capstone 5 emits a separate IMM
operand for those forms.

(cherry picked from commit e832b38)
2e476ce re-keyed updateFunctionGaps() from a walk over code_map byte
keys to merging the instruction intervals of disassembly.functions,
leaving the byte walk reachable only when functions is empty.

The two are not interchangeable. code_map records every decoded byte;
disassembly.functions records only finalized functions. Deriving gaps
from the latter drops the decoded-but-unattributed regions out of the
gap scan, and with them every candidate the scan would have seeded
there. On a 32-bit MSVC PE cross-compiled from Rust (hyperfine 1.19.0,
i686-pc-windows-msvc), recovery falls from 6249 functions to 5247 -- a
loss of 1002, or 16%. Measured across the range, every revision before
2e476ce reports 6249 and every revision after reports 5247.

The rewrite did not pay for itself either. updateFunctionGaps() runs
once per analysis, and the interval merge costs more than the byte walk
it replaced: 0.0410s against 0.0365s cumulative under cProfile.
Reverting restores 6249 at unchanged wall-clock.

Only that hunk is reverted. The rest of 2e476ce keeps its wins -- the
_passesCodeFilter bisect index, the candidate-queue update, the
reference-pair bookkeeping removal, and the SmdaInstruction.from_tuple
and getOutRefs work are untouched.

2e476ce also added test_interior_holes_match_the_covered_byte_walk,
which passed an empty code_map and asserted a gap length derived from
function intervals -- pinning the new behaviour under a name describing
the old one, so the suite could not object. It now exercises code_map,
and a second test asserts that gaps follow code_map coverage when both
sources are populated; that test fails on the pre-revert source.

The original change was validated on report identity for two binaries.
Neither is gap-dominated, so a 16% recall loss was invisible to it.

(cherry picked from commit 3ff05a8)
The name describes the covered byte walk, and an earlier version of the test
asserted against an empty code_map with the length derived from function
extents instead -- which pinned the behaviour that broke the contract under
a name describing the contract. Recording in the docstring why both sources
now carry something, so the next edit has to argue with it rather than
quietly reintroduce the same shape.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PRvm5qFMW52aWnmVgR5vkD
…ilter

`_NOP_START_BYTES` gates the 15-byte disasm_lite probe on the theory that
anything outside it cannot be a nop. In 64-bit that is not true of 0x40-0x4F:
every REX prefix can open an instruction capstone spells `nop` -- `40 90`,
`48 0f 1f 00`, and `41 0f 1f 00` which it renders `nop dword ptr [r8]`.

Without them the gap scan stops recognising REX-padded alignment filler and
books a candidate on the padding. Driving nextGapCandidate() over a gap that
opens with each form, the first address it offers:

    gap opens with     before this branch    with the filter as pushed
    0f 1f 00           0x4                   0x4
    48 0f 1f 00        0x5                   0x1
    41 0f 1f 00        0x5                   0x1
    40 90              0x3                   0x1

0x1 is the nop's own address. The plain multi-byte nop is the control.

The range goes in a separate 64-bit set rather than the shared one, because at
32 bits 0x40-0x4F are inc/dec and never a nop, so admitting them there would
spend a 15-byte disassembly on every inc or dec that opens a gap. Which bytes
each mode needs was swept against capstone over all 256 first bytes: 32-bit
needs nothing beyond the existing set, 64-bit needs exactly the REX range.

Both directions are pinned. The three encodings above fail on the pre-fix
source (4096 != 4100, 4096 != 4100, 4096 != 4098) and a 32-bit test asserts
disasm_lite is never reached for a REX byte.

No bundled fixture is REX-padded in a gap, which is why the branch's ten report
hashes could not see this -- the point its own third-defect note makes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014qdSgwKDgNvjtoDotuf8Mg
@danielplohmann
danielplohmann force-pushed the perf/analysis-hot-path-overhead branch from 6cae1c5 to b833695 Compare September 10, 2026 19:24
@danielplohmann

Copy link
Copy Markdown
Owner

Reviewed. #300 is in as 50d3289, so I have rebased this onto current master and pushed it, along with one fix — details below, and the pre-push head is recorded so nothing is lost.

Pre-push head 6cae1c5aa2a9f6c3b1e6a73fea8a07e9d959038d, new head b833695e51fe3ad59c0ae933b208a69f2d127a4d.

The rebase itself was mechanical: --onto master 68b2626, all 17 commits applied clean, no conflict. A test merge did conflict on one line — self._code_area_index = None in the init() reset block, which git could not place because the squash reordered its neighbours — but a rebase applies each commit three-way and resolved it without asking. Verified content-neutral two ways rather than assumed:

One bookkeeping note before anything else: your ten report hashes all change on this rebase, because master carries v4.5.1's version bump and your old base did not. smda_version is inside toDict(). Excluding it, I re-ran the comparison and it still holds — see below. Nothing is wrong when you see that table move.


The one thing I had to fix: _NOP_START_BYTES omits the REX prefixes

perf(intel): match any-byte regexes without DOTALL and skip gap disasm gates the 15-byte disasm_lite probe on a first-byte set, with the comment "anything else cannot be a nop". In 64-bit that is not true of 0x40–0x4F. Every REX prefix can open an instruction capstone spells nop:

40 90        -> nop
48 0f 1f 00  -> nop
41 0f 1f 00  -> nop dword ptr [r8]

Swept against capstone over all 256 first bytes in both modes: 32-bit needs nothing beyond the set you had, 64-bit needs exactly the REX range on top of it.

The effect, driving nextGapCandidate() over a gap that opens with each form — the first address the scan offers:

gap opens with master this branch as pushed
0f 1f 00 (plain multi-byte nop) 0x4 0x4
48 0f 1f 00 (REX.W) 0x5 0x1
41 0f 1f 00 (REX.B) 0x5 0x1
40 90 (REX) 0x3 0x1

0x1 is the nop's own address: the scan stops recognising REX-padded alignment filler and books a candidate on the padding instead of stepping over it. The plain-nop row is the control.

Fixed in b833695 as a separate 64-bit set rather than by widening the shared one, because at 32 bits 0x400x4F are inc/dec and never a nop — admitting them unconditionally would spend a 15-byte disassembly on every inc or dec that opens a gap, which in a perf change is the wrong direction. Selected once per nextGapCandidate() call by bitness.

Two tests, both directions: the three encodings fail on the pre-fix source (4096 != 4100, 4096 != 4100, 4096 != 4098) and pass after, and a 32-bit test asserts disasm_lite is never reached for a REX byte, so the mode split cannot quietly collapse later.

No bundled fixture is REX-padded in a gap, which is exactly why the ten report hashes could not see it. That is your own third-defect note arriving a second time, from a different direction, and it is the reason I went looking at the byte level rather than trusting the hashes — so the generalisation earned its keep within one review of being written down.

What I checked and what held

  • PriorityQueue pop order is preserved exactly. This was the one I most expected to find something in: the rewrite swaps other.element < self.element for a (neg_score, addr) min-heap, and FunctionCandidate.__lt__ reverses on address when scores tie, so it looks like the tie-break flips. It does not — the two reversals cancel. Property-tested against master's _MaxHeapItem over 2,000 random heaps: 0 mismatches.
  • Report identity: 13 of 13 fixtures identical, under both candidate queues. BracketQueue is not the default and is not in your ten, and it is the queue whose update() call site the branch changes, so it is worth having on the record. (smda_version excluded, per the note above.)
  • The escape cache does not leak across reports. The key holds the escaper class, bytes, mnemonic, operands and the bounds, but escapeBinary also reaches ins.getDetailed(), which decodes under the report's capstone engine — a mode that is not in the key. I could not construct a collision, and 3,988 functions across 12 fixtures hash identically whether analysed in one shared process or in fresh ones. Recording it because only lower_addr/upper_addr keep a 32-bit and a 64-bit image apart today, and that is accidental rather than designed; architecture and bitness in the key would make it deliberate.
  • The AArch64 top-byte filter tables are exactly the achievable sets — not supersets. Derived from each pattern's mask (BL's 0xFC top mask gives exactly 0x940x97; all three prologue masks pin bits 31–24 outright, giving {0xA9, 0xD5, 0xF8}), then cross-checked by sampling ~2M words with rejected top bytes for zero misses. _wordsView indexing lines up with binary[4i+3] on both host byte orders, including the big-endian materialised path.
  • getNormalizedBlockRefs' fast path preserves key order. I built a function whose self.blocks is deliberately out of ascending order to check — the slow path sorts at the end too, so sorted(self.blocks) matches by construction.
  • Dropping fix_graph from get_nesting_depth is sound: it only adds empty successor lists, which cannot reach len(v) > 1. The re.DOTALL[\x00-\xff] rewrites are set-equivalent with no stray . left behind.

Two notes, neither blocking, both ours

  • finalizeAnalysis now creates empty successor entries. Iterating code_refs_from.items() rather than pairs means an addr_from whose destinations were all removed writes disassembly.code_refs_from[addr] = set(), where the old pair loop wrote nothing. Reachable — 30 occurrences over 2,839 calls on five fixtures — and two membership tests in DisassemblyResult (lines 131 and 280) can see it, though no bundled fixture's report changes. An if dests: guard makes it exact.
  • Two getattr(self, "_code_area_index", None) reads where __init__ could define the attribute, which is the pattern refactor(common): type the attributes the dropped annotations rest on #320 removed elsewhere. The index is keyed on list identity, so an in-place mutation of code_areas would go unnoticed; nothing mutates it today. Related: the raw_binary hoist in analyzeFunction is safe only because the one in-analysis rebinder of binary_info.binary — the Delphi relocation pass — runs in the candidate phase. Your comment covers the dicts and sets; that one is worth naming beside them.

Gates on the pushed head

  • python -m pytest tests/: 2094 passed, 2 skipped, 2596 subtests
  • ruff check . and ruff format --check .: clean
  • make typecheck: exit 0, 0 error-level diagnostics

Merging once CI is green. Thanks for the third-defect writeup in particular — it is the reason this review looked where it did.

@danielplohmann
danielplohmann merged commit 46adf4b into danielplohmann:master Sep 10, 2026
25 checks passed
@danielplohmann danielplohmann mentioned this pull request Sep 10, 2026
danielplohmann added a commit that referenced this pull request Sep 10, 2026
The changelog entry for what has landed since v4.5.1 (4868e19): #300, #325 and
#299. Ten engine changes under feat(core) plus a hot-path pass is a minor-bump
shape, which is why #299 and #300 were re-scoped off v4.5.1 rather than squeezed
into it.

VERSION in src/smda/SmdaConfig.py and __version__ in src/smda/__init__.py bumped
together with the entry, per AGENTS.md.

No escaper output changed anywhere in this set. The only escaping-related edit
memoises escapeBinary results behind a cache keyed on its complete input tuple
and no escaper module is touched, so ESCAPER_DOWNWARD_COMPATIBILITY stays at
4.4.5 and INTEL_PIC_HASH_ESCAPE_VERSION at 4.3.5, and no report needs
reprocessing.

Figures are the contributor's own except the ARM64 Mach-O row and the
ReadyToRun result, which were reproduced during the #300 review; the entry says
which is which rather than presenting one table as equally checked.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
danielplohmann added a commit that referenced this pull request Sep 10, 2026
The changelog entry for what has landed since v4.5.1 (4868e19): #300, #325 and
#299. Ten engine changes under feat(core) plus a hot-path pass is a minor-bump
shape, which is why #299 and #300 were re-scoped off v4.5.1 rather than squeezed
into it.

VERSION in src/smda/SmdaConfig.py and __version__ in src/smda/__init__.py bumped
together with the entry, per AGENTS.md.

Deliberately still the current changelog format rather than keep-a-changelog.
#323 proposed adopting against an empty Unreleased so the first entries it holds
are written by their authors at merge time; v4.6.0 cannot be that, since #299's
and #300's entries are written here at release time either way. Adopting after
this release lets the inaugural section fill itself from the seven PRs currently
open. What this entry does adopt is the compromise: each topic keeps the
mechanism and its headline figure with the cost, and links the PR carrying the
full measurement and the dead ends -- ~2,200 words against v4.5.1's ~2,790, and
the first entry in the file to use links at all.

No escaper output changed anywhere in this set. The only escaping-related edit
memoises escapeBinary results behind a cache keyed on its complete input tuple
and no escaper module is touched, so ESCAPER_DOWNWARD_COMPATIBILITY stays at
4.4.5 and INTEL_PIC_HASH_ESCAPE_VERSION at 4.3.5, and no report needs
reprocessing.

Figures are the contributor's own except the ARM64 Mach-O row and the
ReadyToRun result, which were reproduced during the #300 review; the entry says
which is which rather than presenting one table as equally checked.

Also fixes AGENTS.md:117, which had prescribed a one-line changelog entry since
before v4.5.1 while v4.5.1 and v4.6.0 both use the nested **Topic:** list.
Independent of #323, which would replace that line again on adoption.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants